Information processing method, information processing device, information processing program, and information processing system
By creating a three-dimensional movement area with geofences and determining optimal routes for drones, the method addresses the challenges of inspecting complex structures, improving the reliability and quality of image capture and simplifying drone operation.
Patent Information
- Application Number
- PCT/JP2024/012161
- Authority / Receiving Office
- WO · WO
- Patent Type
- Applications
- Current Assignee / Owner
- Priority Date
- 2023-05-15
- Filing Date
- 2024-03-27
- Publication Date
- 2024-11-21
AI Technical Summary
Current methods for flying objects, such as drones, to inspect three-dimensional objects like buildings and tunnels require manual operation and struggle with automatically navigating complex structures, maintaining focus, and ensuring accurate image capture, especially in areas difficult for humans to access.
The method generates a three-dimensional movement area using first and second geofences around the object, allowing the drone to move within a recommended area, which includes determining optimal movement routes and distances based on camera parameters and objectives like depth of field and resolution, enabling automatic flight paths and improved image quality.
This approach enhances the reliability and ease of maneuvering for drones by reducing the complexity of manual operation, ensuring appropriate image capture and focus, and allowing for automatic flight over complex structures while maintaining image quality.
Smart Images

Figure JP2024012161_21112024_PF_FP_ABST
Abstract
Description
Information processing method, information processing device, information processing program, and information processing system
[0001] The present disclosure relates to an information processing method, an information processing device, an information processing program, and an information processing system used for moving a moving body around an object, specifically for flying a flying body around a three-dimensional object.
[0002] A technique for inspecting three-dimensional objects such as tunnels and buildings using an aircraft equipped with a camera is known (Patent Document 1).
[0003] Japanese Patent Application Laid-Open No. 2022-106904
[0004] By using an aircraft equipped with such a camera, it is possible to effectively capture images of places that are difficult for humans to enter, and it is expected that this will be useful for inspecting objects that are difficult for humans to approach, such as social infrastructure such as bridges over rivers and seas, tunnels, dams, and roads, and industrial infrastructure such as airports, buildings, warehouses, factories, and plants.
[0005] In view of the above circumstances, an object of the present disclosure is to provide an information processing method, an information processing device, an information processing program, and an information processing system that are more appropriately used for moving bodies to move around objects, specifically for flying bodies to fly around three-dimensional objects.
[0006] An information processing method according to one embodiment of the present disclosure includes generating a first geofence at a distance from an object, generating a second geofence different from the first geofence at a distance from the object, and generating a movement area in which a moving object can move using the first geofence and the second geofence.
[0007] According to this embodiment, the model of the object to be photographed is enlarged or reduced, the area bounded by the multiple models is defined as a three-dimensional closed space, and the moving object moves within this movement area. Therefore, the amount of information processing during movement is relatively small, and the reliability of obtaining appropriate images safely is increased.
[0008] The first geofence may be generated at a distance away from the object using a first predetermined distance, and the second geofence may be generated at a distance away from the object using a second predetermined distance, the first predetermined distance being greater than the second predetermined distance.
[0009] The second geofence may be generated between the object and the first geofence.
[0010] A recommended movement area may be generated within the movement area, in which the moving object is recommended to move. According to this embodiment, the moving object only needs to move within the recommended movement area within the movement area, so the amount of information processing required for controlling the moving object is relatively small, and the reliability of safely obtaining appropriate images is improved.
[0011] A travel path of the moving object may be generated within the recommended travel area. According to this embodiment, the moving object only needs to travel on a travel path within the recommended travel area, so the amount of information processing required for controlling the moving object is relatively small, and the reliability of safely obtaining appropriate images is improved.
[0012] A movement path of the moving body may be generated within the recommended movement area by slicing the recommended movement area to generate multiple slices, connecting the multiple slices with a single line, and converting the multiple slices and the single line into a single stroke.
[0013] For example, the slice interval and / or slice direction may be automatically determined based on battery life, short total flight time, resistance to wind influence, etc. This allows an optimal movement path to be generated.
[0014] The travel route may be output to the moving body.
[0015] According to this embodiment, the moving object can automatically move along the moving route.
[0016] The first geofence, the second geofence, and the recommended movement area may be generated by acquiring an object model that is a model of the object, and enlarging or reducing the object model.
[0017] This allows appropriate models to be generated for each object.
[0018] The object model, the first geofence, or the second geofence may be a 3D model.
[0019] This allows the original data such as CAD data to be utilized.
[0020] The moving body has an imaging device, and the first geofence, the second geofence and the recommended movement area are each, when the object is 3D scanned, composed of a point cloud equidistant from a reference plane, with the surface of the object model as the reference plane, and are similar to the object model; and, when the object is orthophotographed, composed of a point cloud equidistant from a reference plane, with the surface of a reference model including at least a portion of the point cloud on the surface of the object model as the reference plane, and may be dissimilar to the object model.
[0021] According to this embodiment, for example, different movement areas and movement paths can be generated for orthophotography and 3D scan photography, appropriate movement areas and movement paths can be generated to suit the purpose, and photography can be performed appropriately according to the purpose.
[0022] The recommended movement area may be generated to include at least a portion of a cloud of points that are spaced a recommended movement distance from a reference plane based on the object; the second geofence may be generated to include at least a portion of a cloud of points that are spaced a minimum distance from the reference plane; and the first geofence may be generated to include at least a portion of a cloud of points that are spaced a maximum distance from the reference plane.
[0023] For 3D scanning, three types of models are generated that consist only of point clouds equidistant from the point cloud that constitutes the surface (reference surface) of the model of the object. For orthophotography, a reference model is generated that includes at least a portion of the point cloud that constitutes the surface of the model of the object, and three types of models are generated that consist only of point clouds equidistant from the point cloud that constitutes the surface (reference surface) of the reference model. This allows for the generation of appropriate movement areas and movement paths that suit the purpose, and for appropriate execution of photography according to the purpose.
[0024] The moving body may have an imaging device, and the subject distance at which the imaging device is in focus when photographing the object may be determined as the recommended moving distance, the minimum separation distance may be calculated by subtracting the forward depth of field of the imaging device from the subject distance, and the maximum separation distance may be calculated by adding the rear depth of field of the imaging device to the subject distance.
[0025] This allows the moving object to move within the range (depth of field) in which the imaging device is in focus.
[0026] The moving body may have an imaging device, and the subject distance at which the imaging device is in focus when photographing the object may be determined as the recommended moving distance, the subject distance may be determined as the minimum separation distance, and the maximum separation distance may be calculated by adding the rear depth of field of the imaging device to the subject distance.
[0027] This allows the moving body to be brought as close as possible to the object to photograph it, and also allows the imaging device to be in focus even when the moving body moves away from the object.
[0028] The minimum distance that the moving body can move from the object may be determined as the minimum separation distance, the maximum distance that the moving body can move from the object may be determined as the maximum separation distance, and the average of the minimum separation distance and the maximum separation distance may be determined as the recommended movement distance.
[0029] This allows the moving body to move within the maximum range where no physical obstacles exist and within the maximum range where an acceptable resolution can be obtained.
[0030] The moving body may have an imaging device, the minimum separation distance may be calculated based on the hyperfocal distance of the imaging device, the distance at which the imaging device has the minimum resolution may be determined as the maximum separation distance, and the average of the minimum separation distance and the maximum separation distance may be determined as the recommended movement distance.
[0031] This allows the moving object to be moved within a range that does not fall below the minimum resolution while focusing at infinity.
[0032] An information processing device according to one embodiment of the present disclosure includes an arithmetic device that generates a first geofence at a distance from an object, generates a second geofence different from the first geofence at a distance from the object, and generates a movement area in which a moving object can move using the first geofence and the second geofence.
[0033] An information processing program according to one embodiment of the present disclosure causes a calculation device of an information processing device to operate in the following manner: generate a first geofence at a distance from an object; generate a second geofence different from the first geofence at a distance from the object; and generate a movement area in which a moving object can move using the first geofence and the second geofence.
[0034] An information processing system according to one embodiment of the present disclosure includes a moving body; and an information processing device having a calculation device that generates a first geofence at a distance from an object, generates a second geofence different from the first geofence at a distance from the object, generates a movement area in which the moving body can move using the first geofence and the second geofence, generates a recommended movement area within the movement area in which the moving body is recommended to move, generates a movement path for the moving body within the recommended movement area, and outputs the movement path to the moving body.
[0035] 1 shows a functional configuration of an information processing system according to an embodiment of the present disclosure; 2 shows an operation flow of the information processing system; 3 shows the relative position (inside, outside) of a movement area with respect to an object; 4 shows an object, a movement area around the object, and a recommended movement area within the movement area; 5 shows a case where a depth of field is specified; 6 shows a case where a separation distance is specified; 7 shows a case where a minimum distance and a maximum distance (or a maximum resolution and a minimum resolution) are specified; 8 shows a case where a minimum resolution is specified; 9 shows a case where 3D scanning and orthophotography are specified; 10 shows a case where a movement path generation method is specified;
[0036] 1. Overview of this embodiment
[0037] The use of drones and other robotics for inspection and surveying is increasing. There is a need to be able to inspect the walls and other surfaces of objects by taking photos with drones.
[0038] When flying a drone around a three-dimensional object (building, tunnel, etc.) to scan its surface, manual flight is often required due to the drone's limited versatility, and depends on the pilot's experience.Automatic flight path creation systems also mainly specify a linear flight path or a planar area and control the one-dimensional distance to the structure.
[0039] In addition, to utilize the photographic data, especially for inspection purposes, it is necessary to narrow the focus range appropriately. The main methods for doing this are to use the camera's autofocus function or fix the focus at infinity.
[0040] For example, currently, it is practical to inspect the surface condition of an object by having a person look at each of a large number of photographs taken. For this task, it is best to blur the background to reduce the cognitive load on the human eye. In other words, it is best to narrow the depth of field (the width of the depth direction in which the image is in focus) and focus only on scratches and the surface of the object.
[0041] Furthermore, even if you look at a partial photograph of an object, it is difficult to determine which part of the object it is and its location. For this reason, it is desirable to generate 3D data to grasp the entire object. When creating a 3D image, it is desirable to blur the background so that it is easy for people to understand visually and for machines to automatically separate the background.
[0042] Also, if you are combining multiple photos for 3D purposes, it is best to maintain the same focal length. Orthophotography requires focusing at infinity, so you need to take photos from a certain distance. Shooting at infinity is difficult when scanning small or detailed objects. This requires the user to manually control the distance between the object and the aircraft. It is also difficult to ensure the shooting distance at the edges.
[0043] The main parameters for depth of field are the focal length of the lens, the F-number (aperture value that indicates the brightness of the lens), and the distance from the subject. There are restrictions on the parameters of cameras mounted on aircraft. For example, only relatively short lenses with focal lengths of 28mm or 35mm can be mounted. The F-number is determined by the influence of weather and shadows. A short shutter speed is required. To ensure brightness and to photograph using natural light so that it can be used even on cloudy days, the F-number is often around 2.8 or 5.6. Because camera parameters are restricted, the only way to ensure the quality of the photo is to adjust the distance between the subject and the aircraft, but precise adjustment of the distance is difficult and depends on the skill of the pilot who adjusts it manually.
[0044] Furthermore, it is difficult to automatically fly an aircraft around complex structures, such as industrial plants. A drone piloting expert flies the aircraft while visually monitoring it to ensure it does not collide with objects or other obstacles. For example, a 3D route created on a computer is rotated in 3D to visually check whether the aircraft will collide with objects. Furthermore, there are cases where the aircraft is flown inside an internal structure, such as when flying an aircraft inside a tunnel to inspect the inner walls of the tunnel. In such cases, it is difficult to determine collisions in advance in 3D using original data such as CAD. For this reason, even if CAD data is available, it cannot be used for automatic flight of the aircraft.
[0045] However, when operating a drone, the shooting position may deviate from the plan due to wind or errors in the Global Navigation Satellite System (GNSS) data. To check if there are any problems with the data, the shooting data itself must be visually checked, and all of the shooting data must also be checked on a separate system, which takes time.
[0046] In view of the above circumstances, according to this embodiment, a movement area is generated as a three-dimensional closed space around an object (typically, a three-dimensional object; the same applies hereinafter) in which a moving object (typically, an aircraft; the same applies hereinafter) can move. The movement area is a three-dimensional area in the air around the object in which the aircraft can photograph the object and obtain appropriate images. Even when a pilot manually flies the aircraft, the pilot only needs to fly the aircraft within this movement area, which increases the ease of control and the reliability of obtaining appropriate images.
[0047] According to this embodiment, a movement path of the moving object is further generated within the movement area. By generating a movement path within the movement area, the movement path can be easily and appropriately generated, and automatic flight of the flying object can be achieved.
[0048] According to this embodiment, a moving area and a moving path are generated more appropriately to appropriately perform imaging according to the purpose. For example, different moving areas and moving paths are generated for orthophotography and 3D scanning imaging, and an appropriate moving area and moving path suited to the purpose are generated.
[0049] Hereinafter, embodiments of the present disclosure will be described with reference to the drawings.
[0050] 2. Functional configuration of the information processing system
[0051] FIG. 1 shows a functional configuration of an information processing system according to an embodiment of the present disclosure.
[0052] The information processing system 1 includes a mobile object 100, an information processing device 200, and a server device 300. The mobile object 100, the information processing device 200, and the server device 300 are connected to each other via a network such as the Internet so as to be able to wirelessly communicate with each other.
[0053] The mobile object 100 is typically an air vehicle (drone) capable of moving (flying) in three dimensions, and will be described as an air vehicle in the following embodiments. However, the mobile object 100 may also be a mobile object capable of traveling in two dimensions, such as an automatic guided vehicle (AGV), an autonomous collaborative robot (AMR), or an autonomously driven vehicle.
[0054] The mobile object 100 includes a communication device 101, an imaging device 104 including a lens 102 and a camera 103 (image sensor), a GNSS unit 105, a ranging sensor 106, and a flight control device 107 including a flight mechanism and a control circuit. The communication device 101 is an interface for communicating with the information processing device 200. The imaging device 104 captures an object while the mobile object 100 is flying and generates a captured image. The GNSS unit 105 acquires current location information based on position information, time information, etc. transmitted from GNSS satellites. The ranging sensor 106 is an active sensor that outputs signals such as electromagnetic waves, light, or sound to the environment and receives reflected waves. For example, the ranging sensor 106 may be a time-of-flight (ToF) sensor, LiDAR, millimeter-wave radar, and / or ultrasonic sonar. The flight control device 107 controls the flight mechanism via a control circuit to fly the mobile object 100.
[0055] The information processing device 200 is a terminal device such as a personal computer or a tablet computer. Typically, the information processing device 200 is used by users such as an operator who wants to inspect an object using the mobile body 100 or a pilot expert who flies the mobile body 100. The information processing device 200 generates a movement range for the mobile body 100 to move (fly) around an object to photograph the object, and generates a movement path within the movement range. In this specification, the "object" typically refers to a three-dimensional object such as a building or a tunnel, but may also be a flat plot of land such as a field or an object with almost no height that is laid on the land, such as a pipeline running across the land. In this specification, "photographing" includes all of the following: capturing still image data and storing it in a recording medium, photographing a video and displaying the video data on a display device in real time, and storing the video data in a recording medium.
[0056] The server device 300 stores at least a portion of data necessary for the information processing device 200 to generate a travel range and a travel route, and supplies the data to the information processing device 200. Specifically, the server device 300 stores 2D and / or 3D map data 301 and 2D and / or 3D models 302 of the object to be photographed, and supplies the data to the information processing device 200. There may be one or more server devices 300.
[0057] For example, in the case of an artificial three-dimensional structure such as a bridge, building, tunnel, or factory plant, the object model 302 may be 3D CAD data used for design, etc. Alternatively, the object model 302 may be a texture-free 3D model roughly scanned using a SLAM or ToF sensor, or a simple three-dimensional object such as a cylinder or rectangular parallelepiped. When flying the aircraft 100 automatically or manually to scan the periphery of a three-dimensional structure, the object model 302 must be used. The object model 302 may be a 2D model in the case of a flat land plot such as a field or an object with almost no height laid on the land such as a pipeline running across the land. Furthermore, when inspecting or surveying a flat land plot such as a field, the object model 302 is not necessarily required, and only the map data 301 may be used. In this embodiment, the object model 302 is described as a 3D model.
[0058] The information processing device 200 includes a communication device 201, an input device 202, a calculation device 203, a display device 204, a file output device 205, and a storage device 206. The communication device 201 is an interface for communicating with the mobile object 100. The input device 202 acquires data input from the server device 300 and parameters input by the user. The calculation device 203 calculates a recommended movement area 211, an outer geofence 212 (first geofence), an inner geofence 213 (second geofence), and a movement route 214 for the mobile object 100, and generates a movement range file 210 including these (the method thereof will be described in detail later). The display device 204 displays the movement range 210 and the movement route 214 generated by the calculation device 203 so that the user can recognize them. The file output device 205 records the movement range file 210 generated by the calculation device 203 in the storage device 206 and transmits it to the mobile object 100 via the communication device 201.
[0059] Note that the above system configuration is merely an example. For example, the data of the server device 300 may be stored in the mobile object 100 and / or the information processing device 200. Instead of the information processing device 200, the server device 300 and / or the mobile object 100 may generate the movement range and / or the movement route. The lens 102 and the camera 103 may communicate with the mobile object 100 without being built into the mobile object 100.
[0060] 3. Operational flow of the information processing system
[0061] FIG. 2 shows the operation flow of the information processing system.
[0062] The input device 202 of the information processing device 200 acquires map data 301 and an object model 302 stored in the server device 300. The calculation device 203 draws the object model 302 superimposed on a map drawn using the map data 301, and displays it on the display device 204 for viewing by a user (step S1). The user may be a pilot in a manual flight or a supervisor in an automatic flight.
[0063] Next, the user uses the input device 202 to input parameters for generating a movement area in which the moving body 100 can move. Alternatively, the parameters may be input automatically (step S2). The parameters include a designation mode 21, a relative position 22 of the movement area with respect to the object, parameters 23 of the image capture device 104, a preferred movement path 24, a use 25, an overlap rate 26, and other options 27.
[0064] The designation mode 21 is a target that the user wants to prioritize, specifically relating to distance or resolution. The designation mode 21 is, for example, one of depth of field A, separation distance B from the object, minimum and maximum distances (or maximum and minimum resolutions) C, minimum resolution D, and recommended resolution E. Specifically, if the user wants to move the moving object 100 within a range in which the imaging device 104 can focus, the user designates the value of depth of field A as a parameter. If the user wants to capture an image of the moving object 100 as close as possible to the object and also wants the imaging device 104 to focus even when the moving object 100 is farther away from the object, the user designates the value of separation distance B as a parameter. If the user wants to move the moving object 100 within the maximum range in which no physical obstacles exist or within the maximum range in which an acceptable resolution can be obtained, the user designates the values of minimum and maximum distances (or maximum and minimum resolutions) C as parameters. When the user wants to focus at infinity and move the moving object 100 within a range that does not fall below the minimum resolution, the user specifies the value of the minimum resolution D as a parameter. When the user specifies the recommended resolution E, the calculation device 203 specifies one of the parameters A) through (D) as the specified mode 21 in accordance with the various parameters. The processing related to these parameters A) through (D will be explained in more detail later.
[0065] FIG. 3 shows the relative positions (inside, outside) of the moving area with respect to the object.
[0066] The relative position 22 of the moving area with respect to the object 31 indicates the inside and / or outside of the object 31. For example, a tunnel, a building, a gas tank, or the like has an inside and an outside of the object 31, i.e., an inner wall surface 32 and an outer wall surface 33. The user may specify whether to photograph the inner wall surface 32 or the outer wall surface 33. Alternatively, the user may specify whether to photograph both the inner wall surface 32 and the outer wall surface 33 in a single flight. Alternatively, particularly when photographing both the inner wall surface 32 and the outer wall surface 33 of a tunnel or the like in a single flight, or when only the outer wall surface 33 exists, such as a wind turbine or a steel tower, the relative position 22 does not need to be input. As shown in (A), examples of when the outside is specified as the relative position 22 include photographing the inner wall surface 32 of a tunnel or a building, or calculating the volume of a building or a tank. In this case, the moving area 36 around the outer wall surface 33 is moved to photograph the outer wall surface 33 37. As shown in (B), examples of cases where the inside is specified as the relative position 22 include photographing the surface of a wind turbine, a steel tower, etc., and calculating the volume of an embankment at a construction site. In this case, the inner wall surface 32 is photographed 34 by moving within a moving area 35 around the inner wall surface 32.
[0067] The parameters 23 of the imaging device 104 include the focal length of the lens, the allowable circle of confusion diameter, the F-number (aperture value indicating the brightness of the lens), the image sensor size, etc. The parameters 23 of the imaging device 104 may be automatically input from the moving body 100.
[0068] The priority movement path 24 is a path (e.g., vertical or horizontal) that should be prioritized during automatic flight.
[0069] Use 25 is the use of photography, such as 3D scan photography or ortho photography.
[0070] The overlap rate 26 refers to the rate at which captured images overlap (overlap with adjacent images) during automatic flight. A high overlap rate is expected to enable the creation of an accurate 3D model, for example, when creating a 3D model from multiple still images. Furthermore, when the moving body 100 captures video while flying automatically and displays the video data in real time on the display device 204, the rate at which adjacent images overlap in a direction perpendicular to the direction of movement and the direction relative to the object can be specified as the overlap rate 26. In other words, the overlap rate 26 can also be considered the density of the movement path (movement density).
[0071] The other options 27 may be, for example, an upper speed limit, a maximum flight time, upper and lower limits of flight altitude, a shooting interval, and a shooting direction (the direction of the lens 102, the attitude of the moving body 100, etc.).
[0072] Next, the arithmetic unit 203 creates a movement area in which the moving body 100 can move and a movement route (step S3). The movement area and the movement route will now be described.
[0073] FIG. 4 schematically shows an object, a movement area around the object, and a recommended movement area within the movement area.
[0074] The movement area 215 is a model of a closed space area in which the mobile body 100 can move, partitioned by an outer geofence 212 (first geofence) and an inner geofence 213 (second geofence). The outer geofence 212 is generated at a distance from the object model 214 using a first predetermined distance and covers the object model 214 in the air. The inner geofence 213 is generated between the object model 214 and the outer geofence 212 at a distance from the object model 214 using a second predetermined distance and covers the object model 214 in the air. The first predetermined distance is longer than the second predetermined distance. The recommended movement area 211 is generated within the movement area 215, covers the object model 214 in the air, and is an area in which the mobile body 100 is recommended to move.
[0075] The calculation device 203 determines a maximum separation distance r_max from the object to the moving body 100 and a minimum separation distance r_min from the object to the moving body 100 based on various parameters and specified conditions. The calculation device 203 generates an outer geofence 212 (first geofence) that at least partially includes a point cloud that constitutes the maximum separation distance r_max, and an inner geofence 213 (second geofence) that at least partially includes a point cloud that constitutes the minimum separation distance r_min. The calculation device 203 generates a closed space formed by the outer geofence 212 and the inner geofence 213 as a movement region 215 for the moving body 100. The calculation device 203 determines a recommended movement distance r_flight from the object to the moving body 100 within the movement region 215 based on various parameters and specified conditions. The calculation device 203 generates a recommended movement region 211 within the movement region 215 that at least partially includes a point cloud that constitutes the recommended movement distance r_flight. In other words, the computing device 203 enlarges or reduces the 3D model of the object (input in step S1) or a model based on the 3D model of the object, thereby generating a model of the outer geofence 212, a model of the inner geofence 213, and the recommended movement area 211. The generation method will be described in more detail later (step S33).
[0076] Returning to the explanation of the flowchart, the operation of the arithmetic device 203 to create the movement area and movement route (step S3) will be described in detail.
[0077] First, the computing device 203 determines whether to create a model of the moving object's moving area inside or outside the 3D model of the object (input in step S1) based on the relative position 22 of the moving area with respect to the object (input in step S2), or whether to create a model of the moving object's moving area continuously inside and outside (step S31). "Continuously inside and outside" means, for example, moving (flying) continuously between the outside and inside of a tunnel. When creating a moving area outside the object (e.g., flying outside a tunnel), the order is from the object model to the outside, an inner geofence (second geofence), and then, at the outermost, an outer geofence (first geofence). On the other hand, when creating a moving area inside the object (e.g., flying inside a tunnel), the order is from the object model to the inside, an inner geofence (second geofence), and then, at the innermost, an outer geofence (first geofence).
[0078] Next, the calculation device 203 generates an outer geofence model, an inner geofence model, a recommended movement area, and a movement area (step S32).
[0079] Here, the arithmetic device 203 generates different outer geofence models and inner geofence models depending on the designation mode 21 (input in step S2). Therefore, the arithmetic device 203 generates different movement areas and recommended movement areas depending on the designation mode 21 (input in step S2).
[0080] FIG. 5 shows a schematic diagram of a case where the depth of field is specified.
[0081] The depth of field A1 is a parameter designated when moving the moving object 100 within the range (depth of field) within which the imaging device 104 is focused. For example, as in the example (A), when the user wants to focus on a narrow depth range including the surface of the building that is the inspection target A4 and blur the building behind it, the user designates a narrow depth distance range (e.g., 4 m) as the depth of field A1 parameter. On the other hand, as in the example (B), when the user wants to focus on a wide depth range including the entire building that is the inspection target A4 and blur the clouds and sky in the background, the user designates a wide depth distance range (e.g., 25 m) as the depth of field A1 parameter. As in the example (A), the narrower the depth of field A1, the shorter the subject distance A2 and the shorter the movable distance A3 + A5. As in the example of (B), the wider the depth of field A1, the longer the subject distance A2 and the longer the movable distance A3+A5.
[0082] The calculation device 203 uses the following simultaneous equations (1) to calculate the subject distance A2 from the imaging device 104 to the object A4, the forward depth of field A5, and the rear depth of field A3 from the focal length of the lens 102, the F-number (aperture value), the allowable circle of confusion diameter (e.g., 0.03 mm), and the depth of field A1.
[0083] Front depth of field A5 = (allowable circle of confusion diameter x aperture value x subject distance 2 ) / (focal length 2 + Allowable circle of confusion diameter x aperture value x subject distance) Rear depth of field A3 = (Allowable circle of confusion diameter x aperture value x subject distance) 2 ) / (focal length 2 - allowable circle of confusion diameter x aperture value x subject distance) Depth of field = front depth of field A5 + rear depth of field A3 ... (1)
[0084] The calculation device 203 determines the subject distance A2 as the recommended travel distance r_flight. The calculation device 203 adds the rear depth of field A3 of the image capture device 104 to the subject distance A2 (recommended travel distance r_flight) to calculate the maximum separation distance r_max. The calculation device 203 subtracts the front depth of field A5 of the image capture device 104 from the subject distance A2 (recommended travel distance r_flight) to calculate the minimum separation distance r_min (step S32A). This can be summarized as follows:
[0085] r_min = Subject distance A2 - Front depth of field A5 r_flight = Subject distance A2 r_max = Subject distance A2 + Back depth of field A3
[0086] For example, as in the example (A), when the depth of field A1 is 4 m, the calculated subject distance A2 is 5 m. As in the example (B), when the depth of field A1 is 25 m, the calculated subject distance A2 is 10 m.
[0087] FIG. 6 shows a schematic diagram of a case where a separation distance is specified.
[0088] The separation distance B2 is a parameter that is specified when the user wants to bring the moving body 100 as close as possible to the object B4 to capture an image and to keep the imaging device 104 in focus even when the moving body 100 moves away from the object B4. The user specifies the value of the separation distance B2 (i.e., the subject distance) from the object B4 to the moving body 100 as a parameter. For example, as in the example (A), when the user wants to focus on a narrow depth range including the surface of the building that is the inspection object B4 and blur the building in the background (the depth of field B1 is narrow), the user specifies the separation distance B2 from the object B4 to the moving body 100 to be close (for example, 5 m). On the other hand, as in the example (B), if the user wants to focus on a wide depth range including the entire building, which is the inspection object B4 (wide depth of field B1), and blur the clouds and sky in the background, the user specifies a long distance B2 from the object B4 to the moving body 100 (for example, 10 m).
[0089] The calculation device 203 uses the above simultaneous equation (1) to calculate the rear depth of field B3 from the focal length of the lens 102, the F-number (aperture value), the allowable circle of confusion diameter (e.g., 0.03 mm), and the subject distance (separation distance B2).
[0090] The calculation device 203 determines the subject distance (separation distance B2) as the recommended travel distance r_flight and the minimum separation distance r_min. The calculation device 203 adds the rear depth of field B3 of the image capture device 104 to the subject distance (separation distance B2) to calculate the maximum separation distance r_max (step S32B). This can be summarized as follows:
[0091] r_min = Subject distance B2 r_flight = Subject distance B2 r_max = Subject distance B2 + Rear depth of field B3
[0092] For example, as in the example (A), when the separation distance B2 (subject distance) is 5 m, the rear depth of field B3 is 2.6 m, and the maximum separation distance r_max is calculated to be 7.6 m. As in the example (B), when the separation distance B2 (subject distance) is 10 m, the rear depth of field B3 is 21 m, and the maximum separation distance r_max is calculated to be 31 m.
[0093] FIG. 7 shows a schematic diagram of a case where the minimum distance and the maximum distance (or the maximum resolution and the minimum resolution) are specified.
[0094] If the user wants to move the moving body 100 within the maximum range where there are no physical obstacles or where an acceptable resolution is obtained, the user specifies the minimum distance C1 and the maximum distance C2 (or the maximum resolution and the minimum resolution) as parameters.
[0095] The calculation device 203 determines the minimum distance C1 over which the moving body 100 can move from the object C4 as the minimum separation distance r_min. The calculation device 203 determines the maximum distance C2 over which the moving body 100 can move from the object C4 as the maximum separation distance r_max. The calculation device 203 determines the average (r_min + r_max) / 2 of the minimum separation distance r_min and the maximum separation distance r_max as the recommended travel distance r_flight (step S32C). This can be summarized as follows:
[0096] r_min=Minimum distance C1 r_flight=(C1+C2) / 2=(r_min+r_max) / 2 r_max=Maximum distance C2
[0097] For example, if the minimum distance C1 = 5 m and the maximum distance C2 = 25 m are input as parameters, the minimum separation distance r_min = 5 m, the maximum separation distance r_max = 25 m, and the recommended movement distance r_flight = 15 m are calculated. The movable range C3 is the maximum distance C2 - minimum distance C1 = 25 m - 5 m = 20 m.
[0098] FIG. 8 shows a schematic diagram of a case where the minimum resolution is specified.
[0099] When the user wants to focus on infinity and move the moving body 100 within a range that does not fall below the minimum resolution, the user specifies the value of the minimum resolution as a parameter.
[0100] The calculation device 203 calculates the minimum separation distance r_min based on the hyperfocal distance of the image capture device 104. Specifically, the calculation device 203 calculates the value of (hyperfocal distance / 2)D1 as the minimum separation distance r_min. The hyperfocal distance is a distance beyond which the focus can be considered to be in focus at infinity. The hyperfocal distance is calculated using the following equation (3).
[0101] Hyperfocal distance = focal length 2 × F-number × allowable circle of confusion diameter (3)
[0102] For example, assume that the focal length is 35 mm, the F-number is 2.8, and the allowable circle of confusion diameter is 0.03 mm. 2 × 2.8 × 0.03) / 2 = 6 m.
[0103] The calculation device 203 determines the distance D2 at which the image capture device 104 has the minimum resolution as the maximum separation distance r_max based on the following equation (4).
[0104] r_max = (resolution (mm / px) x focal length) / (sensor area / total number of pixels) 1/2 ...(4)
[0105] The calculation device 203 determines the average (r_min+r_max) / 2 of the minimum separation distance r_min and the maximum separation distance r_max as the recommended travel distance r_flight (step S32D).
[0106] r_min = (hyperfocal distance / 2) D1 r_flight = (D1 + D2) / 2 = (r_min + r_max) / 2 r_max = distance D2 at which minimum resolution is achieved
[0107] For example, resolution 1 mm / px, focal length 35 mm, sensor area 35.7 × 23.8 mm 2 Assume that the total number of pixels is 61 million. Maximum separation distance r_max = (1 x 35) / (35.7 x 23.8 / 61 million)1/2 = 10 m. Since the minimum separation distance r_min = 6 m and the maximum separation distance r_max = 10 m, the recommended movement distance r_flight is calculated as r_flight = (6 + 10) / 2 = 8 m. The movable range D3 is r_max - r_min = 10 m - 6 m = 4 m.
[0108] On the other hand, if recommendation E is specified by the user, the arithmetic device 203 specifies one of parameters A) to (D) as the specification mode 21 according to various parameters. The various parameters include, for example, the focal length of the lens 102, the F-number (aperture value), the three-dimensional sizes of the 3D model of the object, the shallowness or depth of change in the surface structure of the object, the presence or absence of RTK (Real-Time Kinematic) (C does not require RTK), the safe distance from the object to the moving body 100, the allowable amount of blur (A>B>C>D: easy to blur, A>B>C>D: difficult to blur), and the size of the object (small objects can be photographed, A<B<C<D = small objects of 6m or less are difficult to photograph). The arithmetic device 203 calculates the maximum separation distance r_max, the minimum separation distance r_min, and the recommended travel distance r_flight according to the specified parameters A) to (D) (step S32E).
[0109] Next, the calculation device 203 determines whether the purpose 25 (input in step S2) is 3D scan photography or orthophotography (step S33).
[0110] FIG. 9 shows a schematic diagram of 3D scanning photography and orthophotography.
[0111] As shown in (A), in the case of 3D scanning, the computing device 203 generates three types of models composed only of point clouds at an equal distance r from the point cloud constituting the surface (reference plane) of the 3D model 302 of the object (input in step S1). Specifically, the computing device 203 generates an outer geofence model composed only of point clouds at an equal distance (r_max) from the point cloud constituting the surface (reference plane) of the 3D model 302 of the object (input in step S1). The computing device 203 generates an inner geofence model composed only of point clouds at an equal distance (r_min) from each point on the surface (reference plane) of the 3D model 302 of the object. The computing device 203 generates a recommended travel area composed only of point clouds at an equal distance (r_flight) from each point on the surface (reference plane) of the 3D model 302 of the object. The outer geofence model, inner geofence model, and recommended travel area are similar to the 3D model 302 of the object (step S33s). The triangle in the figure schematically indicates the field of view of a moving object.
[0112] As shown in (B), in the case of orthophotography, the calculation device 203 generates a reference model including at least a portion of the point cloud constituting the surface of the 3D model 302 of the object (input in step S1). The reference model is, for example, a rectangular parallelepiped (w × d × h) in the direction perpendicular to gravity. The calculation device 203 enlarges (when photographing the outside of the object) or reduces (when photographing the inside of the object) the rectangular parallelepiped (w × d × h) reference model using the ground as a reference. Specifically, the calculation device 203 substitutes r_max, r_min, and r_flight for the variable r in the following equation (5) to generate an outer geofence model, an inner geofence model, and a recommended travel area. The outer geofence model, the inner geofence model, and the recommended travel area are composed of point clouds equidistant from the surface (reference plane) of the reference model and are dissimilar to the 3D model 302 of the object (step S33o). The overlapping of the outer geofence model, inner geofence model, and recommended movement area can be performed based on the center of gravity of each model, or alternatively, other methods can be used, such as the intersection of the center of the ground plane polygon and the midpoint in the height direction, user input, or horizontal movement to the ground plane.
[0113] (w±2r)×(d±2r)×(h±r) (5)
[0114] The calculation device 203 superimposes the outer geofence model, inner geofence model, and recommended movement area on the object model 302 (displayed in step S1) and displays it on the display device 204 for viewing by the user (step S33).
[0115] FIG. 10 shows a schematic diagram of a method for generating a travel route.
[0116] The arithmetic device 203 slices the recommended movement region 211 (created by r_flight) shown in (A) into parallel slices at equal intervals to generate multiple slices S shown in (B). The arithmetic device 203 determines the slice interval based on, for example, the overlap rate 26 (input in step S2). The arithmetic device 203 determines the slice direction (e.g., vertical or horizontal) to be parallel to the preferred movement path 24 (e.g., vertical or horizontal) (input in step S2). The arithmetic device 203 may automatically determine the slice interval and / or slice direction based on, for example, battery life, short total flight time, low susceptibility to wind, etc. This allows the generation of an optimal movement path.
[0117] As shown in (C), the arithmetic device 203 connects the surfaces of the multiple slices S with a single line L (e.g., a perpendicular line). The arithmetic device 203 may automatically determine the position of the single line L connecting the multiple slices S, or may determine it by user specification. The arithmetic device 203 may determine the position of the single line L, for example, as the position closest to the flight start point or the position that results in the shortest total flight time. When determining it by user specification, the user may specify a connection point on the screen.
[0118] As shown in (D), the arithmetic device 203 converts the plurality of slices S and the single line L into a single stroke, thereby generating a movement path 214 of the moving body 100 within the recommended movement area 211. The arithmetic device 203 draws the movement path 214 superimposed on the object model 302 (displayed in step S1), the outer geofence model, the inner geofence model, and the recommended movement area (displayed in step S33), and displays it on the display device 204 for viewing by the user (step S34).
[0119] The user checks the travel route 214 displayed on the display device 204. If the user wishes to change the travel route 214 or if the travel route 214 cannot be created according to the parameters, the process returns to step S2, where the user re-inputs the parameters. If the user decides to confirm the travel route 214, the process proceeds to step S5 via the input device 202 (step S4).
[0120] The calculation device 203 generates a determined movement range file 210. The movement range file 210 includes a recommended movement area 211, an outer geofence 212, an inner geofence 213, and a movement route 214. The file output device 205 records the movement range file 210, which includes the recommended movement area 211, the outer geofence 212, the inner geofence 213, and the movement route 214 generated by the calculation device 203, in the storage device 206 (step S5).
[0121] The file output device 205 transmits the movement range file 210 to the mobile object 100 via the communication device 201. The mobile object 100 automatically flies along the movement route 214 included in the received movement range file 210, and moves within a movement area, which is a closed space partitioned by an outer geofence 212 and an inner geofence 213. If the mobile object 100 deviates from the movement route 214 due to the influence of a strong wind or the like, it moves within the movement area, which is a closed space, to return to the movement route 214 (step S6).
[0122] 5. Conclusion
[0123] According to this embodiment, the model of the object to be photographed is enlarged or reduced, the range defined by the multiple models is defined as a three-dimensional closed space, and a travel route that suits the user's purpose is created within that range. Since the mobile object only needs to move along the travel route within this travel area, the amount of information processing required for controlling the mobile object is relatively small, and the reliability of obtaining appropriate images safely is improved.
[0124] It is possible to use an appropriate scaled 3D model based on the scanning purpose (object surface scan, elevation ortho) and camera orientation (inside, outside), and obtain an appropriate flight range by specifying the desired focus depth (depth of field), resolution, etc. Furthermore, by specifying several parameters within that flight range, such as the overlap rate, it is possible to automatically generate a flight path and shooting plan.
[0125] As a modified example, the recommended movement area 211 and the movement path 214 may not be generated, and the moving body 100 may be moved by manual operation of the user within the movement area, which is a closed space partitioned by the outer geofence 212 and the inner geofence 213. Even when the pilot manually flies the aircraft, the pilot only needs to fly the aircraft within this movement area, which increases the ease of operation and the reliability of obtaining appropriate images.
[0126] The present disclosure may have the following configurations.
[0127] (1) An information processing method comprising: generating a first geofence at a distance from an object; generating a second geofence different from the first geofence at a distance from the object; and generating a movement area in which a moving object can move by using the first geofence and the second geofence. (2) The information processing method described in (1) above, wherein the first geofence is generated at a distance from the object using a first predetermined distance; and the second geofence is generated at a distance from the object using a second predetermined distance, wherein the first predetermined distance is longer than the second predetermined distance. (3) The information processing method described in (2) above, wherein the second geofence is generated between the object and the first geofence. (4) The information processing method described in any one of (1) to (3) above, wherein a recommended movement area in which the moving object is recommended to move is generated within the movement area. (5) The information processing method according to any one of (1) to (4) above, comprising generating a movement path of the moving body within the recommended movement area. (6) The information processing method according to (4) or (5) above, comprising: slicing the recommended movement area to generate a plurality of slices, connecting the plurality of slices with a single line, and converting the plurality of slices and the single line into a single stroke, thereby generating a movement path of the moving body within the recommended movement area. (7) The information processing method according to (5) or (6) above, comprising outputting the movement path to the moving body. (8) The information processing method according to any one of (4) to (7), comprising acquiring an object model that is a model of the object, and generating the first geofence, the second geofence, and the recommended movement area by enlarging or reducing the object model. (9) The information processing method according to (8), wherein the object model, the first geofence, or the second geofence is a 3D model.(10) The information processing method according to (8) or (9), wherein the moving body has an imaging device, and the first geofence, the second geofence, and the recommended movement area are each, when the object is 3D scanned, composed of point clouds equidistant from a reference plane, with the surface of the object model as a reference plane, and are similar to the object model, and when the object is orthophotographed, composed of point clouds equidistant from a reference plane, with the surface of a reference model including at least a portion of the point cloud on the surface of the object model as a reference plane, and are dissimilar to the object model. (11) The information processing method according to any one of (4) to (10) above, wherein the recommended movement area is generated to include at least a portion of a point cloud that is separated from a reference plane based on the object by a recommended movement distance, the second geofence is generated to include at least a portion of a point cloud that is separated from the reference plane by a minimum separation distance, and the first geofence is generated to include at least a portion of a point cloud that is separated from the reference plane by a maximum separation distance. (12) The information processing method according to (11) above, wherein the moving object has an imaging device, and the information processing method includes determining, as the recommended movement distance, a subject distance at which the imaging device is in focus when photographing the object, subtracting a forward depth of field of the imaging device from the subject distance to calculate the minimum separation distance, and adding a rear depth of field of the imaging device to the subject distance to calculate the maximum separation distance. (13) The information processing method according to (11) above, wherein the moving body has an imaging device, and a subject distance at which the imaging device is in focus when photographing the object is determined as the recommended movement distance, the subject distance is determined as the minimum separation distance, and the maximum separation distance is calculated by adding a rear depth of field of the imaging device to the subject distance. (14) The information processing method according to (11) above, wherein a minimum distance from the object that the moving body can move over is determined as the minimum separation distance, a maximum distance from the object that the moving body can move over is determined as the maximum separation distance, and an average of the minimum separation distance and the maximum separation distance is determined as the recommended movement distance.(15) The information processing method according to (11), wherein the moving object has an imaging device, calculates the minimum separation distance based on a hyperfocal distance of the imaging device, determines the distance at which the imaging device has the minimum resolution as the maximum separation distance, and determines the average of the minimum separation distance and the maximum separation distance as the recommended movement distance. (16) An information processing device comprising: a computing device that generates a first geofence by moving away from an object, generates a second geofence different from the first geofence by moving away from the object, and generates a movement area in which the moving object can move by using the first geofence and the second geofence. (17) An information processing program that causes the computing device of the information processing device to operate in the following manner: generate a first geofence by moving away from an object, generate a second geofence different from the first geofence by moving away from the object, and generate a movement area in which the moving object can move by using the first geofence and the second geofence. (18) An information processing system comprising: a moving body; and an information processing device having a calculation device that generates a first geofence at a distance from an object, generates a second geofence different from the first geofence at a distance from the object, generates a movement area in which the moving body can move by using the first geofence and the second geofence, generates a recommended movement area in which the moving body is recommended to move within the movement area, generates a movement path for the moving body within the recommended movement area, and outputs the movement path to the moving body. (19) A non-transitory computer-readable recording medium having recorded thereon an information processing program that causes the calculation device of the information processing device to operate to generate a first geofence at a distance from an object, generate a second geofence different from the first geofence at a distance from the object, and generate a movement area in which the moving body can move by using the first geofence and the second geofence.
[0128] Although the embodiments and modified examples of the present technology have been described above, the present technology is not limited to the above-described embodiments, and it goes without saying that various modifications can be made within the scope of the gist of the present technology.
[0129] REFERENCE SIGNS LIST 1 Information processing system 100 Mobile body 101 Communication device 102 Lens 103 Camera 104 Imaging device 105 GNSS unit 106 Distance measurement sensor 107 Flight control device 200 Information processing device 201 Communication device 202 Input device 203 Arithmetic device 204 Display device 205 File output device 206 Storage device 300 Server device 301 Map data 302 Object model
Claims
1. An information processing method comprising: generating a first geofence at a distance from an object; generating a second geofence different from the first geofence at a distance from the object; and generating a movement area in which a moving object can move using the first geofence and the second geofence.
2. An information processing method as described in claim 1, wherein the first geofence is generated at a distance away from the object using a first predetermined distance, and the second geofence is generated at a distance away from the object using a second predetermined distance, and the first predetermined distance is longer than the second predetermined distance.
3. The information processing method according to claim 2, further comprising generating the second geofence between the object and the first geofence.
4. An information processing method according to claim 1, further comprising the step of generating, within the movement area, a recommended movement area in which the moving object is recommended to move.
5. An information processing method according to claim 4, further comprising the step of generating a movement route for the moving object within the recommended movement area.
6. An information processing method according to claim 4, further comprising: slicing the recommended movement area to generate a plurality of slices; connecting the plurality of slices with a single line; and converting the plurality of slices and the single line into a single stroke, thereby generating a movement path for the moving body within the recommended movement area.
7. An information processing method according to claim 5, further comprising the step of outputting the travel route to the moving body.
8. An information processing method according to claim 4, comprising: acquiring an object model which is a model of the object; and generating the first geofence, the second geofence and the recommended movement area by enlarging or reducing the object model.
9. The information processing method according to claim 8, wherein the object model, the first geofence or the second geofence is a 3D model.
10. An information processing method as described in claim 8, wherein the moving body has an imaging device, and the first geofence, the second geofence and the recommended movement area are respectively: when the object is 3D scanned, the surface of the object model is used as a reference plane, and the first geofence, the second geofence and the recommended movement area are composed of point clouds equidistant from the reference plane, and are similar to the object model; when the object is ortho-photographed, the surface of a reference model including at least a portion of the point cloud of the surface of the object model is used as the reference plane, and the first geofence, the second geofence and the recommended movement area are dissimilar to the object model.
11. An information processing method as described in claim 4, comprising: generating the recommended movement area including at least a portion of a point cloud that is spaced a recommended movement distance from a reference plane based on the object; generating the second geofence including at least a portion of a point cloud that is spaced a minimum distance from the reference plane; and generating the first geofence including at least a portion of a point cloud that is spaced a maximum distance from the reference plane.
12. An information processing method as described in claim 11, wherein the moving object has an imaging device, and a subject distance at which the imaging device is in focus when photographing the target object is determined as the recommended movement distance, the minimum separation distance is calculated by subtracting a forward depth of field of the imaging device from the subject distance, and the maximum separation distance is calculated by adding a rear depth of field of the imaging device to the subject distance.
13. An information processing method according to claim 11, wherein the moving object has an imaging device, and a subject distance at which the imaging device is in focus when photographing the target object is determined as the recommended movement distance, the subject distance is determined as the minimum separation distance, and the maximum separation distance is calculated by adding a rear depth of field of the imaging device to the subject distance.
14. An information processing method as described in claim 11, further comprising: determining a minimum distance from the object that the moving body can move to as the minimum separation distance; determining a maximum distance from the object that the moving body can move to as the maximum separation distance; and determining an average of the minimum separation distance and the maximum separation distance as the recommended movement distance.
15. An information processing method as described in claim 11, wherein the moving object has an imaging device, the minimum separation distance is calculated based on the hyperfocal distance of the imaging device, the distance at which the imaging device has the minimum resolution is determined as the maximum separation distance, and the average of the minimum separation distance and the maximum separation distance is determined as the recommended movement distance.
16. An information processing device comprising: a computing device that generates a first geofence by moving away from an object; generates a second geofence different from the first geofence by moving away from the object; and generates a movement area in which a moving object can move using the first geofence and the second geofence.
17. An information processing program that causes a calculation device of an information processing device to operate in the following manner: generate a first geofence by moving away from an object; generate a second geofence different from the first geofence by moving away from the object; and generate a movement area in which a moving object can move using the first geofence and the second geofence.
18. An information processing system comprising: a moving body; and an information processing device having a calculation device that generates a first geofence by moving away from an object, generates a second geofence different from the first geofence by moving away from the object, generates a movement area in which the moving body can move using the first geofence and the second geofence, generates a recommended movement area within the movement area in which the moving body is recommended to move, generates a movement path for the moving body within the recommended movement area, and outputs the movement path to the moving body.
Citation Information
Patent Citations
Robot control device
JP2006185438A
Information processing device, information processing system, and control method of information processing device
JP2020147140A
Methods and apparatus of tracking moving targets from air vehicles
US20170038781A1
Unmanned aerial vehicle inspection system
US20170192418A1
Interfaces And Control Of Aerial Vehicle For Automated Multidimensional Volume Scanning
US20220390940A1
