Information processing method, information processing device, and computer program
The information processing method for drones allows for efficient, accurate, and safe return to a desired position by generating a return route based on the drone's traveled path, addressing inefficiencies in current manual return processes.
Patent Information
- Application Number
- PCT/JP2024/015233
- Authority / Receiving Office
- WO · WO
- Patent Type
- Applications
- Current Assignee / Owner
- Priority Date
- 2023-06-01
- Filing Date
- 2024-04-17
- Publication Date
- 2024-12-05
AI Technical Summary
Current methods for manually returning drones to a desired position are time-consuming, lack accuracy, and are inefficient, especially when transitioning from autonomous or automatic flight modes.
An information processing method and device that records the drone's traveled route, determines a return specified position, and generates a return route based on the drone's current position on the traveled route, allowing the drone to safely and accurately return to a desired position by retracing a portion of its previous path.
Enables the drone to be returned to a specified position quickly, accurately, and safely, improving operational efficiency and reducing the time required for manual repositioning.
Smart Images

Figure JP2024015233_05122024_PF_FP_ABST
Abstract
Description
Information processing method, information processing device, and computer program
[0001] The present disclosure relates to an information processing method, an information processing device, and a computer program.
[0002] Autonomous mobile vehicles such as drones are being used in a variety of fields. Drones are already being used for aerial photography, surveying, inspection, photogrammetry (3D model generation), disaster relief, and other applications.
[0003] Drones can fly manually, automatically, or autonomously. In manual operation, the user controls the movement of the drone by operating, for example, an operation terminal. In automatic or autonomous flight, the user's operation is not generally involved in controlling the movement of the drone. Here, automatic flight means, for example, that the drone automatically flies along a predetermined flight plan route. Autonomous flight means, for example, that the drone flies along a predetermined flight plan route while recognizing the space, obstacles, etc. Autonomous flight can, for example, avoid collisions between the drone and obstacles, etc.
[0004] A typical drone can capture images during flight and transmit the captured images to an external device. For example, drone-based inspections can be performed based on images captured by the drone during manual, automatic, or autonomous flight.
[0005] For example, in the case of an inspection using a drone as described above, a situation may arise in which a location suspected of damage appears in an image captured by the drone during flight, making it desirable to capture the location again. In such a case, currently, the pilot either continues to operate the drone manually if it is in manual operation, or switches to manual operation if it is in automatic or autonomous flight, and then returns the drone to the location where it is desired to capture the image again.
[0006] However, manually returning a drone to a desired position is time-consuming, can be imprecise, and is inefficient.
[0007] Japanese Patent Application Laid-Open No. 2020-191523
[0008] The present disclosure has been made in consideration of the above circumstances, and provides an information processing method, an information processing method, and a computer program that can easily, accurately, and safely return a moving object that has moved to a certain position to a desired position.
[0009] The information processing method disclosed herein comprises a travel path recording step for recording the traveled path of a moving body up to its current position; a designated return position determination step for determining a designated return position between the starting point of the traveled path and the current position of the moving body; and a return path generation step for generating a return path for moving the moving body along the portion of the traveled path from the return reference position to the designated return position, using the current position of the moving body on the traveled path as a return reference position.
[0010] The information processing device of the present disclosure includes a moving path recording unit that records the path traveled by a moving body up to its current position, a designated return position determination unit that determines a designated return position between the starting point of the moved path and the current position of the moving body, and a return path generation unit that sets the current position of the moving body on the moved path as a return reference position and generates a return path for moving the portion of the moved path from the return reference position to the designated return position to the moving body.
[0011] The computer program of the present disclosure causes a computer to execute a traveled path recording step for recording the traveled path of a moving body up to its current position; a designated return position determination step for determining a designated return position between the starting point of the traveled path and the current position of the moving body; and a return path generation step for generating a return path for moving the moving body along the portion of the traveled path from the return reference position to the designated return position, using the current position of the moving body on the traveled path as a return reference position.
[0012] 5 is a diagram schematically illustrating an information processing system according to an embodiment, including a drone as a moving object. FIG. 6 is a block diagram illustrating the functional configuration of a drone, a controller, and a cloud server constituting the information processing system according to an embodiment. FIG. 7 is a diagram illustrating the detailed functional configuration of a path planning unit in the controller shown in FIG. 2. FIG. 8 is a diagram conceptually illustrating the relationship between images captured by the drone and recorded in the information processing system and the path traveled by the drone. FIG. 9 is a flowchart illustrating an example of processing by an information processing system according to an embodiment. FIG. 5 is a diagram illustrating an example of an operating state of a drone controlled by the processing of FIG. 5, where (A) is a bird's-eye view of the operating state of the drone, and (B) is a diagram illustrating the operating state of the drone from above. FIG. 6 is a diagram illustrating an example of an operating state of a drone controlled by the processing of FIG. 5, where (A) is a bird's-eye view of the operating state of the drone, and (B) is a diagram illustrating the operating state of the drone from above. FIG. 7 is a diagram illustrating an example of an operating state of a drone controlled by the processing of FIG. 5, where (A) is a bird's-eye view of the operating state of the drone, and (B) is a diagram illustrating the operating state of the drone from above. FIG. 8 is a diagram illustrating an example of an interface used by a user for part of the processing of FIG. 5 (step S508: operation to determine a designated return position). A diagram showing a state in which the drone has been moved from the state of the drone in Fig. 8. A diagram explaining an example of an operation of returning the drone in the state shown in Fig. 10 to a predetermined route. A diagram explaining an example of an operation of returning the drone in the state shown in Fig. 10 to a predetermined route. A diagram explaining an example of an operation of returning the drone in the state shown in Fig. 10 to a predetermined route. A diagram explaining an example of an operation of returning the drone in the state shown in Fig. 10 to a predetermined route. A diagram showing an example of the hardware configuration of an information processing device that can constitute a drone and a controller that constitute an information processing system according to an embodiment.
[0013] Hereinafter, an embodiment of the present disclosure will be described with reference to the drawings.
[0014] 1 is a schematic diagram of an information processing system S according to one embodiment. In this example, the information processing system S includes a drone 10 as a mobile object, a controller 20 operated by a user U to control the drone 10, and a cloud server 30. Note that when simply referring to a drone, this term refers to an unmanned aerial vehicle.
[0015] The drone 10 includes an airframe 11 that holds various control components and instruments, an imaging device 12, multiple motors 13, and multiple propellers 14. The airframe 11 supports the imaging device 12, the motors 13, and the propellers 14. The imaging device 12 is supported on the airframe 11 via a gimbal 12A. The imaging device 12 captures images of the surroundings of the drone 10, and can change the imaging direction by controlling the gimbal 12A. The motor 13 drives the propellers 14.
[0016] In the information processing system S, the drone 10 and the controller 20 are capable of two-way wireless communication. In addition, in the information processing system S, the drone 10 and the cloud server 30 are also capable of two-way wireless communication.
[0017] The drone 10 can be manually controlled by the controller 20. In manual operation, the drone 10 receives control commands based on the user's intentions transmitted from the controller 20, for example, and is controlled based on the control commands. The control commands are generated, for example, by the user directly operating the controller 20. For example, if the control command is a flight command, the drone 10 controls the drive of the propellers 14 via the motors 13 based on the control commands.
[0018] The drone 10 is also capable of automatic flight (automatic movement) and autonomous flight (autonomous movement). In automatic flight, the drone 10, for example, generates a flight plan route (planned movement route) by itself or receives a flight plan route from the controller 20, and flies according to the flight plan route. In autonomous flight, the drone 10, for example, generates a flight plan route (planned movement route) by itself or receives a flight plan route from the controller 20, and flies autonomously according to the flight plan route.
[0019] The above-mentioned "flying autonomously" means, for example, that the drone 10 recognizes the space, obstacles, subjects, etc., avoids collisions, maintains the relative distance to the subject within specified conditions, and flies according to a flight plan route or generates a flight plan route.
[0020] The drone 10 can capture images using the imaging device 12 during various flights as described above. At this time, the imaging direction of the imaging device 12 can be controlled by controlling the gimbal 12A using a control command from the controller 20. In this embodiment, the imaging device 12 also has a thermal camera function, which allows the drone 10 to obtain temperature information of the area within the image.
[0021] The drone 10 can then transmit the captured image, temperature information of the area within the image, its current position, and the like to the controller 20. As described above, the drone 10 and the cloud server 30 are capable of two-way wireless communication. This allows the drone 10 to transmit, for example, the captured image, temperature information of the area within the image, its current position, and the like to the cloud server 30 as well.
[0022] Figure 1 shows multiple base stations 40. When the drone 10 and the cloud server 30 communicate wirelessly in both directions, one or more base stations 40 are involved, as indicated by the two-dot chain arrow in Figure 1. Furthermore, the controller 20 can also transmit control commands to the drone 10 via one or more base stations 40, as indicated by the two-dot chain arrow in Figure 1. When operating the drone 10 from a remote location or when flying the drone 10 beyond visual line of sight, the drone 10 may be controlled via a base station 40.
[0023] The controller 20 in this embodiment is a dedicated operation device that is held by the user U and includes an antenna that transmits control commands. Reference numeral 21 in FIG. 1 denotes a display 21 provided on the controller 20. The display 21 can display, for example, images captured by the drone 10. The controller 20 may also be configured as, for example, a tablet 20′ (including a smartphone) or a personal computer 20″ shown in FIG. 1 . For example, when the controller 20 communicates with the drone 10 via a base station 40, or when the tablet 20′ (including a smartphone) or the personal computer 20″ functions as the controller 20 and communicates with the drone 10, the two-way communication may be performed via a network such as the Internet using a communication method such as Bluetooth (registered trademark) or Wi-Fi.
[0024] In this specification, expressions such as "sending or receiving an image" or "a location" accurately mean sending or receiving image information or location information as electronic information. In this specification, information sent and received between the drone 10 and the controller 20, information sent and received between the drone 10 and the cloud server 30, and information exchanged within the drone 10 or the controller 20 refer to electronic information even if they are not specified by expressions including the term "information."
[0025] 2 is a block diagram showing the functional configuration of the drone 10, the controller 20, and the cloud server 30. The drone 10, the controller 20, and the cloud server 30 will be described in detail below.
[0026] (Drone) The drone 10 has a communication unit 101, a gimbal imaging control unit 102, an image recording unit 103, an image analysis unit 104, a self-position recognition unit 105, a distance measurement unit 106, an obstacle recognition unit 107, a movement path recording unit 108, and a movement control unit 109.
[0027] The communication unit 101 communicates with the controller 20 and / or the cloud server 30 via an antenna. More specifically, communication between the communication unit 101 and the cloud server 30 is performed via a base station 40. The communication unit 101 transmits, for example, the position of the drone 10 and images captured by the drone 10 with the imaging device 12 to the controller 20 and / or the cloud server 30. The position of the drone 10 is information calculated by a self-position recognition unit 105 (described later) and includes at least the direction of travel, latitude, longitude, and altitude of the drone 10.
[0028] On the other hand, the communication unit 101 receives control commands related to flight and imaging, a flight plan route for automatic flight or autonomous flight, etc. from the controller 20. Note that the control commands and the flight plan route from the controller 20 can also be transmitted to the drone 10 via the cloud server 30.
[0029] The communication unit 101 may use any communication method. For example, the communication method may be based on the IEEE 802.11 standard, the IEEE 802.15.1 standard (Bluetooth (registered trademark)), an OFDM modulation method, or other standards. The frequency band used for wireless communication may be, for example, the 2.4 GHz band, the 5 GHz band, or other frequency bands.
[0030] The gimbal imaging control unit 102 controls the imaging device 12 and the gimbal 12A. Specifically, the gimbal imaging control unit 102 controls the imaging conditions (zoom, focus, exposure, shutter speed, white balance, ISO, aperture, etc.) and imaging direction of the imaging device 12 to desired conditions. The gimbal imaging control unit 102 may control the imaging device 12 and the gimbal 12A based on a control command transmitted from the controller 20 or the like, or may autonomously control the imaging device 12 and the gimbal 12A.
[0031] The gimbal imaging control unit 102 provides the image captured by the imaging device 12 and angle information (pitch / yaw / roll) of the gimbal 12A when the image was captured to the communication unit 101 and / or image recording unit 103. The communication unit 101 transmits the provided image and angle information of the gimbal 12A to the controller 20 and / or cloud server 30. Note that the imaging device 12 basically captures moving images at a predetermined frame rate. Moving images include multiple images captured continuously. Note that when simply referred to as an image, this means a still image.
[0032] In this embodiment, as described above, the image capturing device 12 also has a thermal camera function, and temperature information of the area within the image can be obtained. This temperature information is also provided to the communication unit 101 and / or the image recording unit 103.
[0033] The image recording unit 103 records the image provided by the gimbal imaging control unit 102 and the angle information of the gimbal 12A linked to the image. In addition to recording the image, the image recording unit 103 may also record temperature information of a region within the image linked to the image. The image recording unit 103 then provides at least the image provided by the gimbal imaging control unit 102 and the temperature information of the region of the image to the image analysis unit 104.
[0034] The image analysis unit 104 detects errors in the images provided by the image recording unit 103. Specifically, the image analysis unit 104 detects an error when an image that does not satisfy predetermined imaging conditions occurs among multiple consecutively captured images. Image errors include, for example, images that are not captured with proper focus, proper exposure, or improper imaging. Each error is determined by comparing the image with a reference value. The image analysis unit 104 also detects whether a predetermined detection target is present in the multiple consecutively captured images. The detection target is predetermined by the user and may be, for example, rust or scratches on the structure being captured. The image analysis unit 104 can also detect abnormal temperature areas above a predetermined temperature based on temperature information from the images.
[0035] As described above, when the image analysis unit 104 detects at least one of an image error, a detection target, and an abnormal temperature area, in this embodiment, the image analysis unit 104 generates a stop signal and provides it to the movement control unit 109, which will be described later. In response to this, the movement control unit 109 controls the drone 10 to a temporary stop state (hovering state). Furthermore, information on at least one of the image error, detection target, and abnormal temperature area detected by the image analysis unit 104 is transmitted to the controller 20 via the communication unit 101. As will be described later, the controller 20 issues a notification urging the user to return the drone 10 to the location where the image error or the like occurred, based on the information from the image analysis unit 104. When the drone 10 is returned, triggered by this, the image that should be acquired can be properly acquired, for example.
[0036] The self-location recognition unit 105 includes various sensors for detecting the position of the drone 10. In this embodiment, the self-location recognition unit 105 includes, for example, a geomagnetic sensor, a barometric pressure sensor, and a GNSS receiver as the various sensors. The self-location recognition unit 105 recognizes the direction of travel, latitude, longitude, and altitude of the drone 10 as the position of the drone 10, and provides this information to the communication unit 101, the obstacle recognition unit 107, the movement path recording unit 108, and the movement control unit 109.
[0037] The geomagnetic sensor measures the geomagnetism to determine the direction of travel of the drone 10. The barometric pressure sensor measures air pressure. Air pressure varies depending on the height above the ground. Therefore, the altitude of the drone 10 can be calculated based on the air pressure measured by the barometric pressure sensor. The GNSS receiver receives signals from GPS satellites and other satellites (e.g., Galileo and QZSS) and detects the current position (latitude, longitude, and altitude) of the drone 10 based on the received signals.
[0038] The distance measurement unit 106 includes any one or a combination of two or more of a stereo camera, a Lidar sensor, a ToF sensor, an infrared sensor, an ultrasonic sensor, and a monocular camera for detecting obstacles around the drone 10. The distance measurement unit 106 may also be configured by incorporating an IMU (Inertial Measurement Unit) into any one or a combination of these sensors. The distance measurement unit 106 provides information on the distance between the drone 10 and obstacles present around the drone 10 to the obstacle recognition unit 107. Note that an obstacle refers to anything that could hinder the flight of the drone 10, such as an aircraft other than the drone 10, a building, a tree, a person, a vehicle, the ground, a mountain surface, or the surface of water.
[0039] A stereo camera captures images of the drone 10's surroundings using two or more imaging units and acquires two or more different image signals. The two or more image signals are used to detect the distance from the drone 10 to an object in the captured image. A LiDAR (Light Detection and Ranging), TOF (Time Of Flight) sensor, and infrared sensor measure the distance from the drone 10 to an obstacle by emitting light and measuring the time it takes for the reflected light to return. An ultrasonic sensor generates ultrasonic waves and measures the distance from the drone 10 to an obstacle by measuring the time it takes for the reflected ultrasonic waves to return. A monocular camera measures the distance from the drone 10 to an obstacle based on factors such as the focus shift between subjects.
[0040] The IMU is configured to include, for example, a three-axis gyro sensor and a three-directional acceleration sensor, and measures the angular velocity and acceleration of the drone 10. Based on the angular velocity and acceleration information of the drone 10 measured by the IMU, the accuracy of distance measurement can be improved by correcting, for example, the light irradiation angle of the LiDAR. The information detected by the IMU is also provided to the movement control unit 109.
[0041] The obstacle recognition unit 107 determines the relative positional relationship between the drone 10 and the obstacle based on information on the distance from the drone 10 to the obstacle provided by the distance measurement unit 106. The obstacle recognition unit 107 provides the movement control unit 109 with the relative positional relationship between the drone 10 and the obstacle.
[0042] The movement path recording unit 108 records the direction of travel, latitude, longitude, and altitude of the drone 10 provided by the self-position recognition unit 105. The position information of the drone 10 recorded in the movement path recording unit 108 can record and identify the movement path of the drone 10 from its departure to its current position. The point from which the drone 10 departs corresponds to the start point of the movement path.
[0043] The movement control unit 109 is a so-called flight controller, and controls the flight state of the drone 10 by controlling the motor 13 and the propeller 14. For example, when the drone 10 is manually operated, the movement control unit 109 receives control commands transmitted from, for example, the controller 20 and controls the motor 13 and the propeller 14. The movement control unit 109 generates a flight plan route itself or receives a flight plan route from, for example, the controller 20, and, when performing automatic or autonomous flight, controls the motor 13 and the propeller 14 so that the drone 10 flies automatically or autonomously according to the flight plan route. When performing autonomous flight, the movement control unit 109 autonomously controls the drone 10 to prevent collision with an obstacle based on the relative positional relationship between the drone 10 and the obstacle provided by the obstacle recognition unit 107.
[0044] Furthermore, for example, when a flight plan route for automatic flight or autonomous flight is predetermined, the movement control unit 109 detects whether the drone 10 has deviated from the predetermined flight plan route by a predetermined distance or more. Then, when the movement control unit 109 detects that the drone 10 has deviated from the flight plan route by a predetermined distance or more, it controls the drone 10 to a temporary stop state (hovering state). Then, information about the deviation from the flight plan route detected by the movement control unit 109 is transmitted to the controller 20 via the communication unit 101. As will be described later, the controller 20 in this embodiment notifies the user, based on the information from the movement control unit 109, to prompt the user to return the drone 10 to the position before the deviation.
[0045] Furthermore, the drone 10 can perform autonomous flight, for example, by recognizing an object, such as a subject, designated by a user, maintaining a relative distance to the object within a predetermined condition, and flying according to a predetermined flight plan path or by generating a flight plan path. In such an operating state, the movement control unit 109 detects whether the relative distance between the drone 10 and the object deviates from the predetermined condition, and when this state is detected, controls the drone 10 to a temporary stop state (hovering state). Information about an abnormality in the relative distance to the object detected by the movement control unit 109 is also transmitted to the controller 20 via the communication unit 101. In this embodiment, when the controller 20 receives information about the abnormality in the relative distance to the object from the movement control unit 109, it notifies the user to return the drone 10 to a position where the relative distance to the object was appropriate.
[0046] (Controller) The controller 20 includes a communication unit 201 , an input unit 202 , a display unit 203 , an image recording unit 204 , a travel path recording unit 205 , a path planning unit 206 , and a notification unit 207 .
[0047] The communication unit 201 communicates with the drone 10 via an antenna or via an antenna and the base station 40. The communication unit 201 transmits, for example, control commands to the drone 10. From the drone 10, the communication unit 201 receives, for example, the position of the drone 10, images captured by the drone 10 with the imaging device 12, angle information (pitch / yaw / roll) of the gimbal 12A when the images were captured, and temperature information of areas within the images. The communication unit 201 also receives information on image errors, detected objects, and abnormal temperature areas detected by the image analysis unit 104 of the drone 10, information on deviations from the flight plan route detected by the movement control unit 109, and information on abnormalities in the relative distance to the object.
[0048] The communication method of the communication unit 201 may be any method, and as long as it can communicate properly with the drone 10, it may be the same as the method of the communication unit 101 of the drone 10 or a different method.
[0049] The input unit 202 is connected to a physical operation unit (such as a switch, a stick, or a touch panel on the display 21) on the controller 20 (not shown), and detects a user operation via the operation unit. The input unit 202 then generates a command corresponding to the detected user operation and provides it to an appropriate location.
[0050] For example, when a user operates an operation unit to manually operate the drone 10, the input unit 202 generates a control command corresponding to the user's operation and transmits the control command from the communication unit 201 to the drone 10. In addition, in this embodiment, the user can generate, for example, a flight plan route for automatic flight or autonomous flight by using a switch, stick, or touch panel while visually checking an interface displayed on the display 21 (not shown). At this time, the input unit 202 detects the user's operation and provides a command corresponding to the operation to the route planning unit 206 (described later) to generate a flight plan route.
[0051] The display unit 203 displays images captured by the drone 10 using the imaging device 12 and map images on the display 21. The display unit 203 can also display the current position of the drone 10 and the route traveled so far superimposed on the map image. The display unit 203 can also display various types of information such as warnings.
[0052] The image recording unit 204 records the image captured by the imaging device 12 of the drone 10 received by the communication unit 201, and the angle information (pitch / yaw / roll) of the gimbal 12A when the image was captured. Furthermore, when the communication unit 201 receives temperature information of an area within the image, the image recording unit 204 may record the temperature information of the area within the image in association with the image.
[0053] The movement path recording unit 205 records the direction of travel, latitude, longitude, and altitude of the drone 10 received by the communication unit 201. The position information of the drone 10 recorded in the movement path recording unit 108 can record and identify the movement path of the drone 10 from its departure to its current position.
[0054] The route planning unit 206 generates a flight plan route for automatic or autonomous flight of the drone 10. As described above, when a user generates a flight plan route for automatic or autonomous flight on the controller 20, the input unit 202 detects an operation from the user and provides a command corresponding to the operation to the route planning unit 206. The route planning unit 206 generates a flight plan route in response to, for example, the specification of a destination position or a command related to route generation. The route planning unit 206 transmits the generated flight plan route to the drone 10 from the communication unit 201.
[0055] Furthermore, in this embodiment, when a user wishes to return the drone 10 from its current position to a certain position, the path planning unit 206 generates a path (hereinafter referred to as a return path) for returning the drone 10 from its current position to that position based on an instruction from the user. As described above, the communication unit 201 also receives information on image errors, detected objects, and abnormal temperature areas detected by the image analysis unit 104 of the drone 10, as well as information on deviations from the flight plan path detected by the movement control unit 109 and information on abnormalities in the relative distance to the object. Based on this information, the controller 20 issues a notification via the notification unit 207. This notification prompts the user to return the drone 10 to the position where the above information was detected. After receiving this notification, when the user instructs the drone 10 to return from its current position to the position where the above information was detected, the path planning unit 206 generates the above return path.
[0056] 3 shows detailed functional blocks of the route planning unit 206. The route planning unit 206 includes a command receiving unit 211, an automatic autonomous route generating unit 212, a designated return position determining unit 213, and a return route generating unit 214.
[0057] The command receiving unit 211 receives commands and information for generating a flight plan route for automatic flight or autonomous flight from the input unit 202. The command receiving unit 211 also receives commands and information for generating a return route from the input unit 202.
[0058] The automatic autonomous route generation unit 212 generates a flight plan route for automatic flight or autonomous flight based on the command and information for generating a flight plan route for automatic flight or autonomous flight provided to the command reception unit 211. In this case, the automatic autonomous route generation unit 212 may acquire three-dimensional map information from an external source and generate a flight plan route that allows the drone 10 to avoid buildings and the like.
[0059] The designated return position determination unit 213 and the return path generation unit 214 generate a return path for returning the drone 10 from its current position to a certain position, based on the command and information for generating the return path provided to the command reception unit 211. The return path is generated based on the return reference position, which is the current position of the drone 10, the position to which the user wants to return the drone 10 (hereinafter referred to as the designated return position), and the traveled path recorded in the travel path recording unit 205.
[0060] In this embodiment, as an example, the display 21 is provided with an interface that displays the traveled route recorded in the traveled route recording unit 205 superimposed on a map image, allowing the user to specify the above-mentioned designated return position from the traveled route between the starting point of the traveled route and the current position of the drone 10, and an interface that displays the images recorded in the image recording unit 204 in a manner that allows rewinding and fast forwarding, allowing the user to specify the above-mentioned designated return position by selecting a specific image.
[0061] The designated return position determination unit 213 determines the designated return position based on the position of the traveled route on the map image or the selection of an image specified by the user in the two interfaces. The reason the designated return position is determined by the selection of an image is because the image recorded in the image recording unit 204 is linked to the position where the image was captured. When the designated return position is determined by the selection of an image, the image was captured at a position somewhere on the traveled route, so the determined designated return position is a position on the traveled route that is upstream (towards the starting point) of the current position of the drone 10.
[0062] Then, when the designated return position is determined by the designated return position determination unit 213, the return path generation unit 214 generates a return path from the return reference position, the designated return position, and the already-traveled path recorded in the travel path recording unit 205. In particular, the return path generation unit 214 generates a return path from the portion of the already-traveled path from the return reference position to the designated return position. In other words, the generated return path is formed by a portion that is cut out from the already-traveled path. Note that the return reference position is identified from the current position of the drone 10, but this is not limited to this.
[0063] The return route generated by the return route generation unit 214 is transmitted to the drone 10 as a flight plan route for automatic or autonomous flight. The drone 10 that has received the return route is controlled by the movement control unit 109 to move to the specified return position according to the return route.
[0064] Here, after the drone 10 has been moved to the specified return position as described above, in this embodiment, the command receiving unit 211 receives an instruction to move the drone 10 to the return reference position. The instruction to move the drone 10 to the return reference position may be input to the command receiving unit 211 by the user operating a movement instruction button displayed on the display 21, for example.
[0065] In this embodiment, when an instruction to move the drone 10 to the above-mentioned return reference position is issued, the return path generation unit 214 generates a return path. At this time, in this embodiment, the return path generation unit 214 displays multiple return path pattern candidates on the display 21, generates a return path corresponding to the return path pattern selected by the user on the display 21, and transmits it to the drone 10. Having received the return path, the drone 10 is controlled by the movement control unit 109 to move to the return reference position.
[0066] Examples of the return path patterns described above include a pattern in which the drone 10 moves linearly to a return reference position, a pattern in which the drone 10 moves linearly to a specified return position, and the like. Details of these patterns will be described later. The return reference position that serves as the end point of the return path does not necessarily have to coincide with the return reference position when the return path was generated. For example, the return reference position when generating the return path is the most recent position of the drone 10 when generating the return path, but if image capture is stopped before generating the return path, the position where image capture was stopped may be set as the return reference position that serves as the end point of the return path.
[0067] 2 , when the communication unit 201 receives information on an image error, a detected object, or an abnormal temperature area detected by the image analysis unit 104 of the drone 10, or information on a deviation from the flight plan route detected by the movement control unit 109, or information on an abnormality in the relative distance to the object, the notification unit 207 is provided with one or more of these pieces of information. Based on the information provided by the communication unit 201, the notification unit 207 issues a notification to prompt the user to return the drone 10 to the position where the detection of the information occurred.
[0068] For example, the notification unit 207 may provide a command to the display unit 203 to display a warning, and display the warning on the display 21. The warning may be displayed by light emission or by displaying text. When displaying text, it may be displayed to the effect that the drone 10 has temporarily stopped (e.g., "The drone has temporarily stopped") or that it is recommended to return the drone 10 (e.g., "It is recommended to return the drone"). The notification unit 207 may also provide notification by generating a warning sound or vibration.
[0069] (Cloud Server) The cloud server 30 includes a communication unit 301 , an image recording unit 302 , a travel path recording unit 303 , and an image analysis unit 304 .
[0070] The communication unit 301 communicates with the drone 10 and / or the controller 20. Depending on the situation, the communication unit 301 transmits to the drone 10 control commands generated by the controller 20, a flight plan route, and the like. From the drone 10, the communication unit 301 receives, for example, the position of the drone 10, images captured by the imaging device 12 of the drone 10, angle information of the gimbal 12A when the images were captured, and temperature information of areas within the images.
[0071] The communication method of the communication unit 301 may be any method, and as long as it can communicate properly with the drone 10 and / or the controller 20, it may be the same as the method of the communication unit 101 of the drone 10 or a different method.
[0072] The image recording unit 302 records the image captured by the imaging device 12 of the drone 10 and the angle information of the gimbal 12A when the image was captured, which is received by the communication unit 301. Furthermore, when the communication unit 301 receives temperature information of an area within the image, the image recording unit 302 may record the temperature information of the area within the image in association with the image.
[0073] The movement path recording unit 303 records the direction of travel, latitude, longitude, and altitude of the drone 10 received by the communication unit 301. The information on the position of the drone 10 recorded in the movement path recording unit 303 can record and identify the path traveled by the drone 10 from its departure to its current position.
[0074] The image analysis unit 304 functions similarly to the image analysis unit 104 in the drone 10, and detects errors in the image provided by the image recording unit 302, whether a specified detection target is present in the image, and abnormal temperature areas in the image. When the image analysis unit 304 detects at least one of an image error, a detection target, and an abnormal temperature area, the cloud server 30 may send a notification to the drone 10 and the controller 20. The drone 10 may temporarily suspend operation in response to this notification, and the controller 20 may send a notification urging the user to return the drone 10. When the image analysis unit 304 is provided in the cloud server 30, the image analysis unit 104 may not be provided in the drone 10. By providing the image analysis unit 304 in the cloud server 30, the processing load on the drone 10 can be reduced.
[0075] (Status of Recorded Images and Traveled Routes) The status of information recorded in the information processing system S will be described in detail below. As shown in FIG. 2 , the drone 10, controller 20, and cloud server 30 in the information processing system S each have an image recording unit (103, 204, 302) and a traveled route recording unit (108, 205, 303). The images recorded in these image recording units (103, 204, 302) and the position of the drone 10 recorded in the traveled route recording unit (108, 205, 303) are linked at the same time. In other words, any of the multiple images recorded in the image recording units (103, 204, 302) and a position on the traveled route of the drone 10 recorded in the traveled route recording unit (108, 205, 303) are recorded in a state where the timelines are synchronized.
[0076] FIG. 4 conceptually illustrates the relationship between images captured by the drone 10 and the route traveled by the drone 10, which are recorded in the information processing system S. The horizontal axis in FIG. 4 represents the time axis. In FIG. 4, the time axis identifies the time at which the images recorded in the image recording units (103, 204, 302) were captured. The position of the drone 10 recorded in the travel route recording units (108, 205, 303) is linked to the time at which the drone 10 flew through that position. Therefore, for example, by identifying the position of the drone 10 indicated by reference symbol Pn in FIG. 4, the image Im captured at position Pn can be identified. In addition, the image recording units (103, 204, 302) record angle information of the gimbal 12A associated with the image along with the image. Therefore, when the image Im is identified, the angle information of the gimbal 12A at that time is also identified, and the imaging conditions and imaging direction are identified.
[0077] In the controller 20, an interface is provided on the display 21 that displays images recorded in the image recording unit 204 in a manner that allows rewinding and fast-forwarding, and allows the user to specify the specified return position by selecting a specific image, as described above. Such an interface is realized by recording images and previously traveled paths in a synchronized timeline, as described above. Furthermore, in this interface, by specifying a position on the previously traveled path, an image corresponding to the specified position can be displayed. This makes it possible to rewind and fast-forward the displayed image by tracing the previously traveled path on the interface. Note that the processing by the travel path recording unit (108, 205, 303) corresponds to an example of the processing of the travel path recording step.
[0078] <Information Processing Method> Next, an example of an information processing method performed by the information processing system S will be described with reference to the flowchart shown in Fig. 5. As an example, the process shown in Fig. 5 begins when the drone 10 starts flying and capturing images according to a flight plan route for autonomous flight. However, the process shown in Fig. 5 can also be applied when the drone 10 starts flying and capturing images according to a flight plan route for automatic flight, or when a user starts capturing images while manually flying the drone 10.
[0079] 5 is performed, as an example, between the drone 10 and the controller 20. However, the processing described below may be performed only in the controller 20, or between the drone 10 and the cloud server 30.
[0080] After the process starts, first, in step S500, it is determined whether or not the user has manually instructed the drone 10 to temporarily suspend its flight. If the instruction to temporarily suspend its flight is not confirmed in step S500, the process proceeds to step S501. On the other hand, if the instruction to temporarily suspend its flight is confirmed, the process proceeds to step S505, where the flight of the drone 10 is temporarily suspend- ed.
[0081] In step S501, it is determined whether the drone 10 has deviated from the set flight plan route by a predetermined distance or more. If deviation from the flight plan route is not confirmed in step S501, the process proceeds to step S502. On the other hand, if deviation from the flight plan route is confirmed, the process proceeds to step S505, where the flight of the drone 10 is temporarily suspended.
[0082] In step S502, it is determined whether the relative distance between the drone 10 and an object designated by the user, in this case, the object designated for imaging by the imaging device 12, has deviated from a predetermined condition. The user can specify that the drone 10 be flown so as to maintain a constant distance from the object, such as the object, during flight. In step S502, when such a specification is made, it is detected that the specified state is no longer being maintained. If it is not confirmed in step S502 that the relative distance between the drone 10 and the object has deviated from the predetermined condition, the process proceeds to step S503. On the other hand, if it is confirmed that the relative distance has deviated from the predetermined condition, the process proceeds to step S505, where the flight of the drone 10 is temporarily suspended.
[0083] In step S503, it is determined whether an error has occurred in the image captured by the imaging device 12. As described above, image errors include, for example, an image not captured with proper focus, an image not captured with proper exposure, or an image not captured properly. If no image error is confirmed in step S503, the process proceeds to step S504. On the other hand, if an image error is confirmed, the process proceeds to step S505, where the flight of the drone 10 is temporarily stopped.
[0084] In step S504, it is determined whether a pre-specified detection target is present in the image captured by the imaging device 12. As described above, the detection target is predetermined by the user, such as rust or scratches on the structure that is the subject of the image being captured. If no image error is confirmed in step S503, the process returns to step S500. On the other hand, if an image error is confirmed, the process proceeds to step S505, where the flight of the drone 10 is temporarily suspended.
[0085] In this embodiment, the processing of steps S501 and S502 is performed by the movement control unit 109 of the drone 10, and the processing of steps S503 and S504 is performed by the image analysis unit 104 of the drone 10. However, the processing of steps S501 to S504 may be performed by the controller 20 or the cloud server 30. In this case, when moving from any of steps S501 to S504 to S505, a temporary suspension command is transmitted from the controller 20 or the cloud server 30 to the drone 10.
[0086] The flight of the drone 10 is temporarily stopped in step S505 by the movement control unit 109.
[0087] Next, in step S506, a notification is issued to prompt the user to return the drone 10 to a certain position. If the user issues an instruction to pause in step S500 (if "YES"), the occurrence of this instruction is provided to the notification unit 207. If the occurrence of a predetermined event is detected in steps S501 to S504 (if "YES"), information indicating the occurrence of the predetermined event is transmitted from the drone 10 to the controller 20 and provided to the notification unit 207. Then, the notification processing in step S506 is performed by the notification unit 207, triggered by the notification unit 207 receiving information indicating the issuance of the instruction or the occurrence of the predetermined event. The notification processing in step S506 is performed, for example, by displaying a warning on the display 21, as described above.
[0088] Next, in step S507, it is determined whether or not the user has instructed the drone 10 to return. Whether or not the user has instructed the drone 10 to return is determined based on whether or not the command receiving unit 211 in the route planning unit 206 of the drone 10 has received commands and information for generating a return route from the input unit 202 in response to a user operation.
[0089] The information for generating the return path provided by the command receiving unit 211 includes information on a designated return position designated by the user, which is the destination of the drone 10 to return from the current position. In this embodiment, the display 21 provides an interface that displays the traveled path recorded in the travel path recording unit 205 superimposed on a map image as described above, allowing the user to designate a designated return position from the traveled path, and an interface that displays images recorded in the image recording unit 204 in a rewindable and fast-forwardable manner, allowing the user to designate a designated return position by selecting a specific image.
[0090] In this embodiment, for example, when the user specifies a designated return position using the above two interfaces, a designated return position and an instruction to generate a return path are generated and provided to the command receiving unit 211. As a result, in step S507, it is confirmed that the user has instructed the drone 10 to return, and the process proceeds to step S508.
[0091] On the other hand, in this embodiment, as an example, in addition to providing the above two interfaces, a cursor that issues an instruction to resume movement without performing an operation to return the drone 10 is displayed on the display 21, and when this cursor is operated, it is determined that the user does not want to return the drone 10 ("NO" in step S507). In this case, the process proceeds to step S511, and the drone 10 resumes flying according to the flight plan route. Then, the process returns to step S500.
[0092] Then, in step S508, after the instruction to return the drone 10 is confirmed in step S507, the designated return position designated by the user is determined. When the user designates the designated return position by selecting a point from the traveled route displayed superimposed on the map image, the position corresponding to the point selected by the user is determined as the designated return position. When the user designates the designated return position by selecting a specific image, the position on the traveled route at which the designated image was captured is determined as the designated return position. The processing of step S508 corresponds to an example of a designated return position determination step.
[0093] Next, in step S509, a return route is generated by the return route generation unit 214 in the route planning unit 206. The return route generation unit 214 generates a return route from the portion of the already-traveled route from the return reference position to the specified return position. In other words, the generated return route is formed by a cut-out portion of the already-traveled route. The return route is a route for moving the drone 10 along the portion of the already-traveled route from the return reference position to the specified return position. The generated return route is transmitted from the controller 20 to the drone 10. Then, the drone 10 is moved to the specified return position under the control of the movement control unit 109. The processing of step S509 corresponds to an example of a return route generation step and a movement control step.
[0094] Then, in step S510, information regarding the selection of the restart method is confirmed in the controller 20. Then, the return path generation unit 214 generates a return path according to the selected restart method and transmits it to the drone 10. In this embodiment, after the drone 10 has been moved to the specified return position as described above (after step S509), the command reception unit 211 receives an instruction to move the drone 10 to the return reference position. As a prerequisite for receiving this instruction, the return path generation unit 214 displays multiple return path pattern (i.e., restart method) candidates on the display 21. Then, the return path generation unit 214 generates a return path corresponding to the return path pattern selected by the user on the display 21 and transmits it to the drone 10.
[0095] Then, in step S511, the flight of the drone 10 is resumed. As described above, if it is determined in step S507 that the user does not want to return the drone 10, the drone 10 resumes flight according to the flight plan route. On the other hand, if the return route generated in step S510 is transmitted to the drone 10 and then step S511 is proceeded to, the drone 10 is moved to the return reference position according to the return route under the control of the movement control unit 109.
[0096] After the process of step S511, when the drone 10 resumes flying from the return reference position, the process returns to step S500. Note that the processes of steps S510 and S511 are processes corresponding to an example of a restart step.
[0097] Below, we will explain in detail an example of the operation of the drone 10 controlled by the information processing method explained in Fig. 5 using Fig. 6 to Fig. 17. In Fig. 6 to Fig. 8, (A) is a bird's-eye view of the operating state of the drone 10, and (B) is a view of the operating state of the drone 10 seen from above.
[0098] Fig. 6 shows the drone 10 flying according to a predetermined flight plan route Rp. In this example, the drone 10 flies around a building BL according to the flight plan route Rp to inspect defects in the building BL. Although only a portion of the flight plan route Rp is shown in Fig. 6, Fig. 6 shows a portion of the flight plan route Rp that moves the drone 10 from the front to the side of the building BL.
[0099] During the flight shown in Fig. 6, the processes of steps S501 to S504 in Fig. 5 are repeated. Fig. 7 shows a state in which a predetermined state is detected in any of the processes of steps S501 to S504, and the flight of the drone 10 is temporarily stopped. As an example, Fig. 7 shows a state in which a defect Ac as a detection target is detected in an image captured by the imaging device 12 in step S504.
[0100] 7, when the flight of the drone 10 is temporarily stopped, a notification is sent to the user urging the user to return the drone 10 (step S506). D By specifying R A return path R is generated. R is the return reference position P S and the return position P D and the traveled path recorded in the travel path recording unit 205. S is determined from the current position of the drone 10.
[0101] FIG. 8 shows the return route R R Return to the reference position P S At this time, the drone 10 moves backward along a part of the route it actually took.
[0102] As described above, the image recorded in the image recording unit 204 is linked to the position on the traveled path where the image was captured. The image recording unit 204 also records the recorded image and the angle information of the gimbal 12A when the image was captured. Therefore, the return reference position P S When the return reference position P S The angle information of the gimbal 12A corresponding to the return reference position P S When the gimbal 12A moves to the return reference position P S 1. The drone 10 is controlled to match the angle information of the gimbal 12A corresponding to the return reference position P S When the image pickup device 12 moves to the position shown in FIG.
[0103] Here, Fig. 9 shows a specific example of an interface used by the user to specify the above-mentioned "specified return position." In Fig. 9, a first interface U1, a second interface U2, and a third interface U3 are shown on the display 21.
[0104] The first interface U1 is an interface that displays the traveled route recorded in the traveled route recording unit 205 superimposed on a map image, and allows the user to specify a return position from the traveled route. In FIG. 9, the traveled route R when the drone 10 flies around the building BL shown in FIGS. 6 to 8 is F The first interface U1 is displayed on the display 21 when the drone 10 is temporarily stopped in response to a user instruction or when a predetermined state is detected in steps S501 to S504 described above.
[0105] In the first interface U1, the traveled route R F The user can specify the designated return position by touching any of the above positions with, for example, a finger Fg. D In the first interface U1, when a predetermined state is detected in the above steps S501 to S504, the traveled route R on which the predetermined state is detected is displayed.F The top position may be displayed differently than the other positions.
[0106] The second interface U2 displays images recorded in the image recording unit 204 in a rewind and fast-forward manner, and allows the user to select a specific image to specify a return position. In the second interface U2, when the user performs a flick operation with, for example, a finger Fg in the direction indicated by the arrow α, the display of multiple recorded images, shown within the range X for ease of explanation, switches in the direction indicated by the arrow β. In this example, when a specific image is displayed, the user selects the image by touching the image with, for example, the finger Fg. By selecting the image in this manner, a return position is determined based on the image.
[0107] The third interface U3 is an interface that displays the current position (heading, latitude, longitude, altitude) of the drone 10 as numerical values, graphs, etc. The third interface U3 also includes operation components for rewinding and fast-forwarding the image displayed on the second interface U2. This allows the user to rewind and fast-forward the image displayed on the second interface U2 by operating the third interface U3 with, for example, the finger Fg.
[0108] 10 shows how the drone 10 is moved closer to the defect Ac from the state shown in FIG. 8 to acquire a detailed image of the defect Ac. R Return to the specified position P D After moving the drone 10 to the designated return position P, the user may operate the drone 10 arbitrarily to move the drone 10 to a position outside the route already traveled. DAfter the drone 10 has been moved from the target position, a situation arises in which it is desirable to return the drone 10 to the original flight plan route Rp. In this embodiment, an instruction to move the drone 10 to the return reference position is accepted as described above. Then, as a prerequisite for accepting this instruction, the return route generation unit 214 displays multiple return route pattern (i.e., restart method) candidates on the display 21. Then, the return route generation unit 214 generates a return route corresponding to the return route pattern selected by the user on the display 21 and transmits the generated return route to the drone 10. Thereafter, the drone 10 resumes flight according to the flight plan route Rp.
[0109] 11 to 14 each show how the drone 10 in the state shown in FIG. 10 is moved along a flight plan route Rp as a predetermined route.
[0110] FIG. 11 shows the drone 10 returning to the designated position P D After moving to the R Return along the reference position P S The return path pattern shown in FIG.
[0111] 12 and 13 show the generated return route R R By changing (offsetting) the position of B (see FIG. 13 ), and the drone 10 is guided along the return path R B Return along the reference position P S The return path pattern shown in FIG.
[0112] The return route R shown in FIG. B is the generated return route R R The upstream end of the traveled route R F The return route R R Return reference position P S The return path R intersects with R is generated by moving (offsetting) it from its original position.
[0113] FIG. 14 shows the drone 10 traveling along a return route R RThe return reference position P S 10 shows a return path pattern in which the robot moves linearly to the target position.
[0114] For example, the specified return position P D After the drone 10 is moved from the target location, multiple return path patterns (i.e., restart methods) such as arrow γ1 shown in Fig. 11, arrow γ2 shown in Fig. 13, and arrow γ3 shown in Fig. 14 may be displayed on the display 21 in response to a predetermined operation, prompting the user to select a return path pattern. Providing such an interface makes it easier to restart the flight.
[0115] The information processing system S according to the present embodiment described above performs a travel path recording step for recording the travel path of the drone 10 up to its current position, a designated return position determination step for determining a designated return position between the start point of the travel path and the current position of the drone 10, and a return path generation step for generating a return path for the drone 10 to move the portion of the travel path from the return reference position to the designated return position, using the current position of the drone 10 on the travel path as the return reference position.
[0116] This allows the drone 10, which has moved to a certain position, to be easily, accurately, and safely returned to the desired position. In other words, by following the steps described above, the drone 10 can be returned to a designated return position, which is a desired position upstream of the current position, simply by specifying the designated return position. Therefore, the operation to return the drone 10 is not time-consuming, and because the target position (designated return position) is clearly specified, the drone 10 can be accurately moved to the designated return position. The return path for returning the drone 10 to the designated return position is formed from the path the drone 10 has already traveled. Therefore, the return path is a path that has already been flown safely without coming into contact with obstacles, etc., and the drone 10 can be safely moved to the designated return position. Therefore, the drone 10 can be easily, accurately, and safely returned to the desired position.
[0117] In this embodiment, the designated return position can be determined from a position designated on the traveled route displayed on the map image. Alternatively, one of the recorded images can be selected, and the position on the traveled route where the selected image was captured can be determined as the designated return position. This facilitates the user's operation when returning the drone 10 to a desired position, improving convenience.
[0118] Furthermore, in this embodiment, if the drone 10, which is moving according to a predetermined flight plan route, deviates from the flight plan route by more than a predetermined distance, a notification is issued urging the user to specify a designated return position. Furthermore, if the relative distance between the drone 10 and a predetermined subject (target object) deviates from a predetermined range, a notification is issued urging the user to specify a designated return position. Furthermore, if an image that does not satisfy predetermined imaging conditions is generated, a notification is issued urging the user to specify a designated return position. Furthermore, if a predetermined detection target is detected, a notification is issued urging the user to specify a designated return position. By issuing such notifications, it becomes possible to properly perform operations such as imaging and flight that were planned to be performed by the drone 10, thereby improving convenience. For example, the accuracy of inspections using the drone 10 can be improved.
[0119] Furthermore, in this embodiment, when the drone 10 is moved to a specified return position and then moves to a position deviating from the previously traveled path, in order to move the drone 10 to the return reference position, the drone 10 can be moved to the specified return position and then moved along the return path to the return reference position in response to a user instruction, as shown in FIG. 11 . Furthermore, as shown in FIGS. 12 and 13 , a return path is generated by aligning the upstream end of the return path with the position of the drone 10 that deviated from the previously traveled path and moving the return path so that the return path intersects with the return reference position, and the drone 10 can be moved along the return path to the return reference position. Furthermore, as shown in FIG. 14 , the drone 10 can be moved linearly to the return reference position. This makes it possible to easily resume operations such as image capture and flight that were originally planned and were interrupted during the movement to the specified return position, thereby improving convenience.
[0120] <Hardware Configuration> Fig. 15 shows an example of the hardware configuration of the drone 10 and controller 20 as information processing devices. The drone 10 and controller 20 can be configured by a computer device 500 as shown in Fig. 15. The computer device 500 includes a CPU 501, an input interface 502, an external interface 503, a communication device 504, a main memory device 505, and an external memory device 506, which are interconnected by a bus. The drone 10 and controller 20 may not include at least one of these elements.
[0121] The CPU (Central Processing Unit) 501 executes a computer program on the main memory device 505. The computer program is, for example, a program that realizes each of the above-mentioned functional configurations of the drone 10. The computer program may be realized not as a single program, but as a combination of multiple programs and scripts. Each functional configuration is realized by the CPU 501 executing the computer program.
[0122] The input interface 502 is a circuit for inputting operation signals from input devices such as a keyboard, mouse, and touch panel to the drone 10 and the controller 20.
[0123] The external interface 503 displays, on a display device, for example, data stored in the drone 10 and the controller 20 or data calculated by the drone 10 and the controller 20. The external interface 503 may be connected to, for example, an LCD (liquid crystal display), an organic electroluminescence display, or a PDP (plasma display).
[0124] The communication device 504 is a circuit that allows the drone 10 and the controller 20 to communicate with an external device wirelessly or via a wired connection. Data used by the drone 10 and the controller 20 can be input from the external device via the communication device 504. The communication device 504 may include an antenna. Data input from the external device can be stored in the main memory device 505 or the external memory device 506. The communication device 504 may be connected to another communication device, such as a smartphone or a tablet, via a wired or wireless connection.
[0125] The main memory device 505 stores data necessary for executing a computer program, data generated by the execution of the computer program, and the like. The computer program is deployed and executed on the main memory device 505. The main memory device 505 is, for example, a RAM, a DRAM, or an SRAM, but is not limited to these. The information and data storage unit of the communication device 504 may be constructed on the main memory device 505.
[0126] The external storage device 506 stores data necessary for executing a computer program, data generated by the execution of the computer program, etc. These computer programs and data are read into the main storage device 505 when the computer program is executed. The external storage device 506 is, for example, but is not limited to, a hard disk, an optical disk, a flash memory, or a magnetic tape.
[0127] The computer program may be pre-installed in the computer device 500, or may be stored on a storage medium such as a CD-ROM. The computer program may also be uploaded to the Internet.
[0128] Furthermore, the computer device 500 may be configured as a single device, or may be configured as a system made up of multiple computer devices connected to each other.
[0129] The above-described embodiment is merely an example for realizing the present disclosure, and the present disclosure can be implemented in various other forms. For example, various modifications, substitutions, omissions, or combinations thereof are possible without departing from the spirit of the present disclosure. Such modifications, substitutions, omissions, etc. are also included within the scope of the present disclosure, as well as within the scope of the inventions described in the claims and their equivalents. For example, in the above-described embodiment, an example of a drone, which is an unmanned aerial vehicle, is described as an example of a moving object. However, the technology of the present disclosure can also be applied to underwater drones, etc.
[0130] Furthermore, the effects of the present disclosure described in this specification are merely examples, and other effects may also be present.
[0131] The present disclosure can also be configured as follows. [Item 1] An information processing method comprising: a traveled path recording step of recording a traveled path of a moving body up to its current position; a designated return position determination step of determining a designated return position between a starting point of the traveled path and the current position of the moving body; and a return path generation step of setting the current position of the moving body on the traveled path as a return reference position and generating a return path for moving the moving body along a portion of the traveled path from the return reference position to the designated return position. [Item 2] The information processing method described in [1], further comprising a return movement control step of moving the moving body from the return reference position to the designated return position along the return path. [Item 3] The information processing method described in [1] or [2], wherein in the designated return position determination step, a position specified on the traveled path displayed on a map image is determined as the designated return position. [Item 4] The information processing method according to any one of [1] to [3], further comprising an image recording step of recording a plurality of images captured consecutively as the moving object moves along the previously traveled path, wherein in the designated return position determination step, one of the plurality of images recorded in the image recording step is selected, and a position on the previously traveled path at which the selected image was captured is determined as the designated return position. [Item 5] The information processing method according to any one of [1] to [4], further comprising a restart step of moving the moving object to the designated return position and then moving it along the return path to the return reference position when the moving object is to be moved to the previously traveled path in a situation where the moving object has moved to a position off the previously traveled path after being moved to the designated return position.[Item 6] The information processing method according to any one of [1] to [5], further comprising a restart step of, when moving the moving body to the return reference position in a situation where the moving body has moved to a position off the previously traveled path after being moved to the specified return position, generating a return path by moving the return path so that the return path intersects with the return reference position while aligning an upstream end of the return path with the position of the moving body that has deviated from the previously traveled path, and moving the moving body along the return path to the return reference position. [Item 7] The information processing method according to any one of [1] to [6], further comprising a restart step of, when moving the moving body to the return reference position in a situation where the moving body has moved to a position off the previously traveled path after being moved to the specified return position, moving the moving body linearly to the return reference position. [Item 8] The information processing method according to any one of [1] to [7], further comprising a notification step of issuing a notification prompting the user to specify the designated return position when the moving object, moving according to a predetermined planned route, deviates from the planned route by a predetermined distance or more. [Item 9] The information processing method according to any one of [1] to [8], further comprising a notification step of issuing a notification prompting the user to specify the designated return position when the moving object is moving so that a relative distance between the moving object and a predetermined object surrounding the moving object falls outside a predetermined range. [Item 10] The information processing method according to any one of [1] to [9], further comprising a notification step of issuing a notification prompting the user to specify the designated return position when an image that does not satisfy predetermined imaging conditions occurs in a plurality of images continuously captured by the moving object. [Item 11] The information processing method according to any one of [1] to
[10] , further comprising a notification step of issuing a notification prompting the user to specify the designated return position when a predetermined detection object is detected in a plurality of images continuously captured by the moving object.[Item 12] An information processing device comprising: a traveled path recording unit that records a traveled path of a moving body up to its current position, a designated return position determination unit that determines a designated return position between a start point of the traveled path and the current position of the moving body, and a return path generation unit that sets the current position of the moving body on the traveled path as a return reference position and generates a return path for moving the moving body along a portion of the traveled path from the return reference position to the designated return position. [Item 13] A computer program that causes a computer to execute the following steps: a traveled path recording step that records a traveled path of a moving body up to its current position, a designated return position determination step that determines a designated return position between the start point of the traveled path and the current position of the moving body, and a return path generation step that sets the current position of the moving body on the traveled path as a return reference position and generates a return path for moving the moving body along a portion of the traveled path from the return reference position to the designated return position.
[0132] S...Information processing system 10...Drone (moving object) 11...Airframe 12...Imaging device 12A...Gimbal 13...Motor 14...Propeller 101...Communication unit 102...Gimbal imaging control unit 103...Image recording unit 104...Image analysis unit 105...Self-position recognition unit 106...Distance measurement unit 107...Obstacle recognition unit 108...Movement path recording unit 109...Movement control unit 20...Controller 20'...Tablet 20''...Personal computer 21...Display 201...Communication unit 202...Input unit 203...Display unit 204...Image recording unit 205...Movement path recording unit 206...Path planning unit 207...Notification unit 211...Command reception unit 212...Automatic autonomous path generation unit 213...Return designated position determination unit 214...Return path generation unit 30...Cloud server 301...Communication unit 302: Image recording unit 303: Travel route recording unit 304: Image analysis unit 40: Base station U1: First interface U2: Second interface U3: Third interface P S … Return reference position P D …Return position R R …Return route R F …Traveled route R B…Return route
Claims
1. An information processing method comprising: a traveled path recording step for recording a traveled path of a moving body up to its current position; a designated return position determination step for determining a designated return position between a starting point of the traveled path and the current position of the moving body; and a return path generation step for generating a return path for moving the moving body along the portion of the traveled path from the return reference position to the designated return position, the portion being determined as a return reference position by the current position of the moving body on the traveled path.
2. The information processing method according to claim 1, further comprising a return movement control step of moving said moving object from said return reference position to said specified return position along said return path.
3. An information processing method according to claim 1, wherein in said designated return position determination step, a position designated on said already traveled route displayed on a map image is determined as said designated return position.
4. An information processing method as described in claim 1, further comprising an image recording step of recording a plurality of images taken successively as the moving body moves along the traveled path, and in the designated return position determination step, one of the plurality of images recorded in the image recording step is selected, and the position on the traveled path at which the selected image was taken is determined as the designated return position.
5. An information processing method as described in claim 1, further comprising a restart step of moving the moving body to the return reference position after the moving body has been moved to the specified return position and the moving body has moved to a position off the previously moved path, and then moving the moving body to the specified return position and then moving the moving body along the return path to the return reference position.
6. The information processing method of claim 1, further comprising a resume step of, when the moving body is moved to the specified return position and then to a position off the already-traveled path, generating a return path by moving the return path so that the return path intersects with the return reference position while aligning the upstream end of the return path with the position of the moving body that has deviated from the already-traveled path, and moving the moving body along the return path to the return reference position.
7. An information processing method as described in claim 1, further comprising a restart step of moving the moving body in a straight line to the return reference position when the moving body is moved to a position outside the traveled path after the moving body has been moved to the specified return position and the moving body is then moved to the return reference position.
8. An information processing method according to claim 1, further comprising a notification step of issuing a notification to prompt the user to specify the designated return position when the mobile object, moving according to a predetermined planned route, deviates from the planned route by a predetermined distance or more.
9. An information processing method as described in claim 1, further comprising a notification step of issuing a notification to prompt the user to specify the return position when the relative distance between the moving body and a predetermined object surrounding the moving body falls outside a predetermined range while the moving body is moving so that the relative distance falls within the predetermined range.
10. The information processing method according to claim 1, further comprising a notification step of issuing a notification to prompt the user to specify the return position when an image that does not satisfy predetermined imaging conditions occurs among the multiple images captured continuously while the moving body is moving.
11. The information processing method according to claim 1, further comprising a notification step of issuing a notification urging the user to specify the return position when a predetermined detection target is detected in a plurality of images captured continuously while the moving body is moving.
12. An information processing device comprising: a traveled path recording unit that records a traveled path of a moving body up to its current position; a designated return position determination unit that determines a designated return position between a starting point of the traveled path and the current position of the moving body; and a return path generation unit that sets the current position of the moving body on the traveled path as a return reference position and generates a return path for moving the portion of the traveled path from the return reference position to the designated return position to the moving body.
13. A computer program for causing a computer to execute the following steps: a traveled path recording step for recording the traveled path of a moving body up to its current position; a designated return position determination step for determining a designated return position between the starting point of the traveled path and the current position of the moving body; and a return path generation step for generating a return path for moving the moving body along the portion of the traveled path from the return reference position to the designated return position, using the current position of the moving body on the traveled path as a return reference position.
Citation Information
Patent Citations
Information processing system, control method of the same, and program
JP2019010943A
Control system for movable body
JP2022065749A
Information processor, moving apparatus, and computer program
JP2022181736A
Information processing device and information processing method
WO2020189492A1
