Information processing method, information processing device, and program

Abstract

The present invention calculates three-dimensional data for an object on the basis of a plurality of images of the object that have been captured using an imaging device, calculates the positions and postures of the imaging device at the capture times of the plurality of images, determines, from among the calculated positions and postures of the imaging device, the position and posture of the imaging device at the capture time of a point in need of inspection on the object that has been identified on the basis of the images captured of the object and the three-dimensional data, and uses the determined position and posture to determine a target position and a target posture for a mobile body that includes an imaging device that is for capturing an image of the point in need of inspection for subsequent inspection.

Description

Information Processing Method, Information Processing Apparatus, and Program

[0001] The present invention relates to an information processing method, an information processing apparatus, and a program for determining a target position and a target attitude of a moving body including an imaging device.

[0002] Conventionally, in order to inspect whether there are defective parts such as cracks and chips in structures such as buildings and bridges, a method of taking pictures using a camera mounted on a robot or a drone and grasping the state of the object from the taken pictures is known. In Patent Document 1, in order to link a recorded video and a shooting position, a method of obtaining the position of a camera using SfM (Structure from Motion) technology and then displaying a captured image when a user designates a position on a reference map is disclosed.

[0003] Japanese Unexamined Patent Application Publication No. 2024-9427

[0004] When a defective part is discovered by inspecting a structure, reinspection may be necessary to grasp the progress of the defect thereafter. Conventionally, in order to photograph the same defective part during reinspection, shooting may be performed after setting a rough position of the position where the camera should shoot. Then, since the shooting conditions such as the exact position of the camera and the angle of the camera at the time of shooting differ depending on the person operating the camera or for each inspection time, images in which the appearance of the defective part is different are obtained. And when comparing the taken image with a past image, it is difficult to accurately grasp the change of the defective part when comparing images with different appearances of the defective part.

[0005] Patent Document 1 does not describe a specific method for reinspecting a defective part of a structure or a technique for setting shooting conditions of a camera during reinspection.

[0006] Therefore, an object of the present invention is to provide a technique for determining a target position and a target attitude of a moving body including an imaging device for effectively performing reinspection of an object.

[0007] An information processing method, as one aspect of the present invention for solving the above problems, is an information processing method for determining a target position and target orientation for a mobile body including an imaging device for inspecting an object, characterized in that a computer calculates three-dimensional data relating to the object based on a plurality of images obtained by photographing the object using the imaging device, calculates the position and orientation of the imaging device when the plurality of images were taken, determines the position and orientation of the imaging device when it photographs a part of the object that needs to be inspected, which has been identified based on the images obtained by photographing the object and the three-dimensional data, from among the calculated position and orientation of the imaging device, and uses the determined position and orientation to determine a target position and target orientation for the mobile body including the imaging device for photographing the part that needs to be inspected for subsequent inspections.

[0008] According to the present invention, it is possible to provide a technique for determining the target position and target orientation of a moving body, including an imaging device, for effectively re-inspecting an object.

[0009] A diagram showing the system configuration. A diagram showing the configuration of the information processing device. A diagram showing a flowchart of the information processing method.

[0010] Embodiments of the present invention include, for example, the following configurations.

[0011] [Item 1] An information processing method for determining a target position and target orientation for a mobile body including an imaging device for inspecting an object, characterized in that a computer calculates three-dimensional data relating to the object based on a plurality of images obtained by photographing the object using the imaging device, calculates the position and orientation of the imaging device when the object was photographed, determines the position and orientation of the imaging device when a part of the object requiring inspection, identified based on the images obtained by photographing the object and the three-dimensional data, is photographed from among the calculated position and orientation of the imaging device, and uses the determined position and orientation to determine a target position and target orientation for a mobile body including the imaging device for photographing the part requiring inspection for subsequent inspections. [Item 2] The information processing method according to Item 1, characterized in that the changes in the part requiring inspection are evaluated from the results of photographing the part requiring inspection using the mobile body controlled based on the created target position and target orientation. [Item 3] The information processing method according to Item 1 or 2, characterized in that a movement path of the mobile body connecting a plurality of target positions for photographing each of the plurality of parts requiring inspection is determined. [Item 4] A program for causing a computer to execute the information processing method described in any one of Items 1 to 3. [Item 5] An information processing device for determining a target position and target orientation of a mobile body including an imaging device for inspecting an object, wherein the processing device has a processing unit, the processing unit calculates three-dimensional data relating to the object based on a plurality of images obtained by photographing the object using the imaging device, calculates the position and orientation of the imaging device when the object is photographed, determines the position and orientation of the imaging device when a part of the object requiring inspection, identified based on the images obtained by photographing the object and the three-dimensional data, is photographed from among the calculated position and orientation of the imaging device, and determines a target position and target orientation of the mobile body including the imaging device for photographing the part requiring inspection for subsequent inspections using the determined position and orientation.

[0012] Preferred embodiments of this disclosure will be described in detail below with reference to the attached drawings. In this specification and the drawings, components having substantially the same functional configuration are denoted by the same reference numerals, and redundant descriptions will be omitted.

[0013] <Details of the Embodiment> This embodiment describes a technique for inspecting buildings, structures, etc., for the presence and extent of damage such as cracks and chips. Examples of objects to be inspected include buildings and structures, skyscrapers, bridges, dams, wind power generation facilities, factories, power transmission towers, communication towers, agricultural facilities, and forests. Examples of objects to be inspected include cracks, fissures, and leaks in the exterior walls and roofs of buildings, concrete deterioration, rebar rust, sagging of power lines, loose connections, deterioration of metal fittings, cracks, deterioration, and wear of wind turbine blades, cracks, corrosion, and peeling paint on bridge girders, surface deterioration, corrosion, and leaks in chimneys and tanks, damage to piping in chemical plants, inspection of hazardous areas, structural damage, corrosion, and loose bolts in communication towers and power transmission towers, the condition of cables, the health of trees and damage caused by pests, deterioration of agricultural facilities, damage after disasters, and damage to facilities after typhoons and floods.

[0014] Figure 1 shows a diagram of the system configuration in this embodiment. This system comprises a mobile unit 1, a control device 2, an information processing device 3, and a server (information processing device) 4. These are connected by a network 9.

[0015] Mobile body (mobile device) 1 is a device that moves on its own, such as a mobile robot, drone (unmanned aerial vehicle), or vehicle. Mobile body 1 includes a camera (imaging device) that photographs an object to be inspected in order to check whether there are any defects or other problems with the object to be inspected. The camera has an image sensor that includes a photoelectric conversion element and may be equipped with a lens. In addition to the function of acquiring still images by shooting, the camera may be equipped with various functions such as a video camera with a video shooting function, a variable aperture (brightness) function, a zoom (variable focal length) function, and a sensitivity adjustment function. The camera may be fixed to the structure of the mobile body or may be configured to be movable. If it is configured to be movable, the relative position and relative angle (attitude) of the camera with respect to mobile body 1 may be configured to be controllable. The position and attitude of the camera in three-dimensional space are defined relative to the position and attitude (roll, pitch, yaw) of mobile body 1 and may be set to be synonymous with the position and attitude of mobile body 1. The camera's orientation can be defined by angles around three orthogonal axes, but if the camera is movable relative to a moving object, it can be set according to the number of movable axes of the camera, and if there is one movable axis, the camera's angle parameter for that axis is set. The current positions of the moving object 1 and the camera can be measured using a GPS module or an accelerometer, or they can be determined using self-localization techniques such as V SLAM or Lidar SLAM from a reference point, sound, Wi-Fi, or Bluetooth.

[0016] The control device 2 is a device for controlling the mobile object 1 by communicating with it. The control device 2 communicates with the mobile object 1 with control signals to control the position, angle, and camera of the mobile object 1, and also communicates image data captured by the camera. The control device 2 may be located near the site where the mobile object 1 is deployed and communicate with the mobile object 1 using a specific wireless communication method, or it may be installed in a remote location far from the site where the mobile object 1 is deployed and communicate with the mobile object 1 via a base station, master unit, or mobile network. The control device 2 may also be equipped with a pilot that is operated by a human.

[0017] The information processing device 3 is a device for determining (creating) the target position and target orientation of a mobile body, including a camera for taking photographs of areas requiring inspection for inspection purposes.

[0018] Server 4 is a device for performing Structure from Motion (SfM), a technique that calculates a 3D point cloud based on images taken from multiple viewpoints. Server 4 acquires image data such as video and multiple still images taken by the camera of the mobile object 1, and constructs a 3D point cloud of the object from that image data.

[0019] The information processing device 3 communicates with the control device 2 and the server 4 via the network 9, sends data to the control device 2 to control the mobile device 1, and receives data acquired by the mobile device 1 via the control device 2. The information processing device 3 also sends data such as images to the server 4 and receives the results of data processing performed by the server 4.

[0020] The configuration of the information processing device 3 will now be described. The information processing device 3 has a computer that can be installed in a general-purpose computer such as a personal computer, or in a device such as a smartphone or tablet. The information processing device 3 can determine the movement path of the camera of the mobile device 1 using installed software or applications (programs).

[0021] Figure 2 shows the configuration diagram of the information processing device 3. The information processing device 3 includes a processing unit 11, a memory 12, a storage 13, a communication unit 14, an input unit 15, and a display unit 16. These are electrically connected to each other via a bus 17.

[0022] The processing unit 11 is a computing device that controls the operation of the entire information processing device 3, controls the transmission and reception of data between each part, and performs information processing necessary for program execution and authentication processing. The processing unit 11 includes a computing device such as a CPU, GPU, or FPGA, and executes programs stored in the storage 13 and loaded into the memory 12 to perform various information processing described later.

[0023] The memory 12 (storage unit) includes a main memory composed of a volatile storage device such as DRAM, and an auxiliary memory composed of a non-volatile storage device such as flash memory or HDD. The memory 12 is used as a work area for the processing unit 11, and also stores the BIOS and various setting information that are executed when the information processing device 3 is started up.

[0024] The storage 13 (storage medium) includes storage devices such as HDDs and SSDs, and stores various programs such as applications and programs. In addition, a database containing data used for each process is built on the storage 13.

[0025] The communication unit 14 connects the information processing device 3 to the network 9. The communication unit 14 communicates with external devices directly or via a network access point using methods such as wired LAN, wireless LAN, Wi-Fi (Wireless Fidelity, registered trademark), infrared communication, Bluetooth (registered trademark), short-range or contactless communication. The input unit 15 is an information input device such as a keyboard, mouse, or touch panel.

[0026] The display unit (display device) 16 includes a display that shows information calculated by the processing unit 11 or information received from an external source by the communication unit 14. The display unit 16 provides a graphical user interface (GUI) that displays various types of information on its screen. The display unit 16 is not limited to being integrated with the information processing device 3, but may also be a display (display device) provided separately from the information processing device 3 and connected to the information processing device 3.

[0027] Bus 17 is connected in common to all of the above components and transmits, for example, address signals, data signals, and various control signals.

[0028] Next, the configuration of Server 4 will be described. Server 4 has a processing unit, memory (storage unit), storage (storage medium), and communication unit. These are electrically connected to each other via a bus. The processing unit is a computing device that controls the operation of the entire Server 4, controls the transmission and reception of data between each unit, and performs information processing necessary for program execution and authentication processing. The processing unit includes a computing device such as a CPU, GPU, or FPGA, and executes programs stored in storage and loaded into memory to perform various information processing described later.

[0029] The storage of server 4 stores various programs such as applications and other programs. A database containing data used for each process is built in this storage. Software programs for performing SfM are stored in the storage or memory. The communication unit connects server 4 to network 9. This communication unit communicates with control device 2 or information processing device 3 via network 9. Received information is stored in memory and used for information processing performed by the processing unit, or stored in storage.

[0030] Next, we will explain the method using the system described above. Figure 3 shows a flowchart of this method.

[0031] First, the mobile device 1 is used to photograph the object and acquire image data (S1). The camera on the mobile device 1 photographs the inspection target, such as a building or structure, from various angles. The photography can be done by recording a video of the inspection target using the camera on the mobile device 1, or by taking multiple still images. Also, maintaining a certain distance between the object and the camera during photography allows for more accurate reconstruction of the three-dimensional (3D) information. The mobile device 1 transmits the captured images to the control device 2. In the case of video recording, the mobile device 1 can transmit images to the control device 2 while moving and taking pictures. The control device 2 acquires the image data transmitted from the mobile device 1. Multiple image acquisition can be done by extracting multiple images from the video data when the inspection target is recorded using the camera on the mobile device 1, or by taking multiple still images.

[0032] When taking photographs, it is preferable to photograph the object from multiple angles, ensuring that there is 30-80% overlap between each of the multiple images taken. Furthermore, the photographs may be taken by the mobile body 1 moving autonomously, by the control device 2 automatically moving the mobile body 1, or by a person controlling the position and orientation of the mobile body 1 by operating the control device 2.

[0033] When moving the mobile body 1 automatically, the control device 2 sets waypoints that include points along the path the mobile body 1 will travel, and controls the position of the mobile body 1 so that it moves according to the waypoints. The control device 2 transmits the waypoints of the mobile body 1 as target positions to the mobile body 1 and performs position control on the mobile body 1, for example, according to the difference between the target position and the measured current position. The control device 2 can also control shooting conditions such as the angle (direction), focal position, brightness, and sensitivity of the camera of the mobile body 1 when shooting video.

[0034] Waypoints can be set using pre-prepared 3D maps or map data of the shooting area within the designated software. For example, if the mobile object 1 is a drone, the user sets the points the drone should pass through on the GUI map to reach its destination. This includes the drone's flight path and shooting spots. Altitude, latitude, and longitude location information can also be set for each waypoint. It is also possible to specify the time the drone hovers at a waypoint and the camera's attitude. The speed at which the drone moves between waypoints can also be set. Based on these set waypoints, the precise route of the mobile object 1 is programmed using GPS and an inertial navigation system (INS).

[0035] For example, when photographing a building, waypoints are set along the exterior wall of the building, and the drone takes pictures with its camera at each waypoint. When photographing a bridge, waypoints are set to photograph the bridge piers and the underside of the bridge, and the drone moves around the bridge.

[0036] Furthermore, if necessary, preprocessing can be performed on the images obtained from S1 before inputting the data into the SfM algorithm. For example, unnecessary images can be removed, or the brightness and contrast of the images can be adjusted.

[0037] Next, the server 4 performs SfM based on the multiple images obtained in S1 (S2). The server 4 acquires the image data transmitted from the control device 2 or the information processing device 3. The information processing device 3 may instruct the control device 2 to send the captured image data to the server 4, or the control device 2 may, in response to a request from the information processing device 3, send the acquired image data to the information processing device 3, and the information processing device 3 may then send the image data to the server 4.

[0038] The information processing method performed by the processing unit of Server 4 will now be explained. First, the processing unit of Server 4 detects feature points in the images using the multiple images obtained in S1, or pre-processed images. Feature points are unique points in an image that are easy to match from different viewpoints, such as the edges and corners of an object. Common algorithms include SIFT (Scale-Invariant Feature Transform) and SURF (Speeded Up Robust Features).

[0039] The processing unit of server 4 maps the feature points detected from each image across multiple images. It matches common points in images taken from different viewpoints to identify which parts of which images represent the same points. Feature-based methods (such as feature descriptors like SIFT or SURF) or nearest neighbor search algorithms are used for matching.

[0040] The processing unit of server 4 estimates the camera's position and orientation (angle, direction) at the time each image was taken, based on the positional information (2D coordinates) of feature points matched in the two images. For example, using a mathematical model called epipolar geometry, the relative position and orientation of the cameras can be calculated by integrating information from multiple viewpoints based on parallax. In this case, a base matrix and an essential matrix are calculated to estimate the relative position and orientation between cameras. Alternatively, the camera's position and orientation can be calculated using AI such as Gaussian Splatting. In this way, the camera's position and orientation in 3D space are determined, and the image and the camera's position and orientation at the time the image was taken can be associated and stored in a storage unit such as a database.

[0041] The processing unit of Server 4 constructs a 3D point cloud using feature points matched from each viewpoint. Specifically, based on the estimated camera position and orientation, it calculates the coordinates of corresponding 3D points from the 2D feature point coordinates that match across multiple images. This process is based on triangulation. This method allows us to determine where the features of an object seen in a 2D image are located in 3D space. The 3D point cloud is represented by 3D coordinates, with each point representing a position corresponding to the surface of the object. This allows us to grasp the approximate shape of the object. If necessary, by simultaneously adjusting the positions of all cameras and the 3D point cloud and performing bundle adjustments to minimize errors, more accurate camera positions and 3D point clouds can be obtained.

[0042] Since the 3D point cloud obtained at this stage consists of sparse data, in order to obtain a dense point cloud, it is necessary to find more corresponding points and add them. For this, algorithms such as stereo matching and optical flow can be used. In stereo matching, two or more image pairs are used to find corresponding points for each pixel and generate a dense point cloud. In optical flow, the pixel movement between temporally consecutive images is calculated to generate a more detailed point cloud. Also, a dense 3D point cloud can be obtained using methods such as Patch-based Multi-view Stereo (PMVS) that estimates the position of each point using the pixel information of the image. Further, if the point cloud contains noise such as outliers, noise removal and optimization may be performed.

[0043] Note that since the camera positions and 3D point clouds calculated by SfM are relatively estimated, in order to obtain actual sizes and distances, there is a method of using external scale information (e.g., dimensions of known objects, GPS data). For example, prior to shooting S1, set up known survey points or targets at the inspection site in advance, and after SfM is completed, input the coordinate values and perform fitting. Alternatively, during the SfM process, place a target such as an AR marker for which arbitrary coordinates can be known in the image data, and by giving the coordinate values during the SfM process and performing fitting, the output of the SfM process can be represented in the actual coordinate system.

[0044] Based on the constructed point cloud data, the processing unit of server 4 generates a mesh model (3D model) by connecting point to point. A mesh is a step for polygonizing a 3D point cloud to smoothly represent the shape, and the detailed shape of the object becomes clearer.

[0045] Server 4 can output the obtained 3D point cloud and 3D model to information processing device 3 in various formats.

[0046] Next, the processing unit of the information processing apparatus 3 superimposes the obtained image on the obtained 3D model or 3D point cloud data (S3). As a result, a texture (surface color and pattern) can be applied to the 3D data, making it closer to the appearance of the actual object. This can be done, for example, using a 3D viewer displayed on the display (display device) of the information processing apparatus 3. Specifically, the position of the viewer's eyes is set at the position of the camera obtained in S2, and the viewpoint is adjusted along the direction of the camera (line of sight) obtained in S2. Then, a plate-shaped object model is generated in the forward direction of the line of sight from the position of the camera, and the 2D image and 3D data can be superimposed by arranging the image on the object model.

[0047] Next, the user views the image superimposed on the 3D data displayed on the viewer screen and identifies the locations to be inspected (inspection locations) such as the damaged parts of the object (S4). The user confirms the inspection target while viewing the image displayed on the viewer (screen of the display device), specifies the position on the 3D data, and inputs the inspection result (evaluation) at that position, enabling the information processing apparatus 3 to store these data.

[0048] The information processing apparatus 3 determines the inspection locations that require reinspection (key inspection locations), such as the locations selected from the inspection locations specified in S4 or the locations that require future follow-up observation (S5). The determination of the key inspection locations can be made, for example, by the user specifying them using the viewer as in S4, or based on the evaluation results when evaluating the inspection locations.

[0049] Next, the information processing apparatus 3 determines the conditions for photographing the key inspection locations during reinspection (S6). First, based on the position information of the key inspection locations determined in S5, an image suitable for evaluating the key inspection locations is determined from the images obtained in the S1 photographing that include the key inspection locations. Suitable images are, for example, images in which the states of defective parts such as cracks, defects, and scratches on the inspection target are well displayed and suitable for the user to observe.

[0050] Then, using the database stored in S2 that associates images with camera position and orientation, information on the camera's position and orientation when an image suitable for evaluating the areas requiring inspection is captured is extracted. If there are multiple images suitable for evaluating the areas requiring inspection, the above database is used to extract information on the camera's position and orientation when each image was captured. Furthermore, if there are multiple areas requiring inspection determined in S5, for each area requiring inspection, an image suitable for evaluating that area is determined, and the above database is used to extract information on the camera's position and orientation when the image of that area was captured.

[0051] In this way, the position and orientation of the camera for photographing the areas requiring inspection are determined, and the target position and orientation of the mobile body including the camera when photographing the areas requiring inspection during re-inspection are determined using the determined camera position and orientation information. If the camera is fixed to the mobile body, the camera's position and orientation can be controlled by setting the camera's position and orientation to be synonymous with the mobile body's position and orientation, and controlling the mobile body's position and orientation as control parameters. Therefore, the target position and orientation of mobile body 1 or the camera are determined as the target position and orientation when photographing the areas requiring inspection during re-inspection. Alternatively, the target position and orientation of the mobile body may be created by incorporating an offset value into the camera's target position and orientation, and the mobile body's position and orientation may be controlled. Furthermore, if the camera is movable relative to the mobile body, the relative position and orientation of the camera are controlled along with the position and orientation of the mobile body, so the target position and orientation of the mobile body and the relative target position and orientation of the camera are determined. If the camera position and orientation calculated by S2's SfM, and the camera position and orientation (angle) stored in S2, are relative values ​​in the computational coordinate system, they should be converted to the actual coordinate system in which the moving object and camera are located, as described above. For example, if the moving object 1 is moving in WGS84 (the coordinate system of GNSS (Global Navigation Satellite System)), the target position and target orientation should be converted to the GNSS coordinate system.

[0052] If there are multiple target positions and orientations for the camera to photograph areas requiring inspection, these multiple positions can be set as waypoints (point information on the path), and the path connecting these multiple positions can be set as the camera's movement path. Similarly, if there are multiple areas requiring inspection, the path connecting the target positions of the cameras photographing each area can be set as the camera's movement path. The camera's movement path may be determined by minimizing the length of the path passing through each target position, or by specifying the order in which the camera passes through each target position. Furthermore, setting the movement path may include the movement speed, acceleration / deceleration information, and stopping time of the moving object.

[0053] In addition to setting conditions for photographing areas requiring inspection during a re-inspection, it is also possible to set conditions for various camera functions, such as camera zoom. Furthermore, it is possible to set the camera to photograph areas requiring inspection at a higher magnification than the previous time.

[0054] The conditions determined in S6 are transmitted to the control device 2 via the network 9 for re-inspection. Furthermore, the conditions determined in S6, such as the target position and orientation of the camera for photographing the areas requiring inspection, and the movement path of the camera (mobile body 1), may be displayed on the screen of the display device for visual confirmation by the user.

[0055] Next, the camera on the mobile body 1 is used to take a re-inspection photograph (S7). During the re-inspection photograph, the control device 2 controls the mobile body 1 based on the conditions determined in S6. The control of the mobile body 1 can be automated by the control device 2. The information processing device 3 transmits the conditions for photographing the areas requiring inspection determined in S6 to the control device 2 and commands the control device 2 to control the mobile body 1. Specifically, the control device 2 moves the mobile body 1 based on waypoints so that it reaches the target position and target orientation of the camera for photographing the areas requiring inspection determined in S6 in the actual three-dimensional coordinate system, and controls the position and orientation of the camera on the mobile body 1. Then, when the mobile body 1 reaches the target position and target orientation of the camera for photographing the areas requiring inspection determined in S6, it photographs the areas requiring inspection with its camera. The mobile body 1 may stop when it reaches the target position and target orientation of the camera for photographing the areas requiring inspection determined in S6, or it may decelerate just before reaching the position of the camera for photographing the areas requiring inspection determined in S6, and photograph the areas requiring inspection with its camera. If there are multiple areas requiring inspection, the camera's position and orientation are controlled to achieve the target position and orientation for each area requiring inspection, and then photography is performed. The images captured by the camera are transmitted to the control device 2, which then transmits the image data to the information processing device 3 via the network 9. In practice, control errors are involved in controlling the position and orientation of the mobile body 1 and the camera. Alternatively, the camera may be controlled to photograph the areas requiring inspection only when its position and orientation fall within a predetermined range that includes the target position and orientation determined in S6. This predetermined range can be determined based on an acceptable range in which captured images can be identified as identical.

[0056] Next, the information processing device 3 evaluates the areas requiring inspection using the re-inspection imaging results (S8). The information processing device 3 displays the image data for re-inspection acquired in S7 on a viewer. At this time, the image of the areas requiring inspection captured in S1 may be displayed simultaneously on the same screen, or it may be displayed on a separate screen at the user's discretion. This makes it possible to compare the image from the re-inspection with the image obtained from the previous imaging. The viewer that displays the images may either display the 2D images as they are, or it may be the 3D viewer used in S3 to S5. The user evaluates the changes, progress, and status of the areas requiring inspection by comparing the image from the re-inspection displayed on the screen with the image obtained from the previous imaging. The user can input the evaluation results, and the input results are saved in memory or elsewhere, corresponding to the areas requiring inspection (3D position). In addition, the image data for re-inspection can also be stored in a storage device, corresponding to the areas requiring inspection (3D position), and can be read out and displayed on the display device as appropriate, according to the user's selection.

[0057] The information processing device 3 may output the results of a comparison between the image data for re-inspection acquired in S7 and the image taken in S1, using an image processing algorithm or artificial intelligence (AI) or other computational processing, for the same location of the area requiring inspection. The output comparison results may include, for example, a display that highlights the differences between the images, or output as indicators, numerical values, or evaluation comments.

[0058] According to this embodiment, the position and orientation of the camera that captures images of areas requiring inspection during re-inspection are the same as the position and orientation of the camera that captures images suitable for evaluating the areas requiring inspection. Therefore, by comparing these images, changes, progression, and progress in the condition of the areas requiring inspection can be grasped more accurately. Conversely, if the position and orientation of the camera when photographing the areas requiring inspection during re-inspection differ from those of the previous time, the appearance of the areas requiring inspection in the resulting images will change, and this difference in appearance may lead to an incorrect evaluation of the condition of the areas requiring inspection. In reality, control errors are included in the control of the position and orientation of the mobile body 1 and the camera, so differences due to these control errors will appear in images taken multiple times by the camera, but these can be defined as being within the range of the same image.

[0059] Subsequently, if further inspections of areas requiring inspection are conducted periodically or at any time, steps S7 and S8 are repeated. This allows for obtaining images of the areas requiring inspection taken from the same position and in the same posture with each inspection, enabling continuous monitoring of the progress of the condition of the areas requiring inspection over a relatively long period.

[0060] As described above, the information processing device 3 and server 4 calculate three-dimensional data about an object based on multiple images obtained by photographing the object using the imaging device, calculate the position and orientation of the imaging device when multiple images were taken, determine the areas of the object that need inspection based on the images and three-dimensional data obtained by photographing the object, determine the position and orientation of the imaging device when the imaging device photographed the areas that need inspection from among the calculated position and orientation of the imaging device, and use the determined position and orientation of the imaging device to determine the target position and target orientation of a mobile body including the imaging device for photographing the areas that need inspection for subsequent inspections such as re-inspection.

[0061] Preferred embodiments of this disclosure have been described in detail above with reference to the attached drawings, but the technical scope of this disclosure is not limited to these examples. Furthermore, not all components shown in the embodiments are essential components of this disclosure. Also, features shown in each embodiment are applicable to other embodiments insofar as they do not contradict each other.

[0062] The information processing device 3 and the server 4 may be configured as the same device, or they may be configured as multiple distributed information processing devices, or they may be configured as virtual servers or containers in the cloud.

[0063] The information processing device 3 or server 4 may further perform SfM using the data of multiple images and videos taken during the re-inspection to construct the latest 3D data. Alternatively, the 3D data obtained during the re-inspection may be used to update the 3D point cloud and shape information of the object. Furthermore, the camera position and orientation calculated by SfM during the re-inspection may be used to correct the target position and orientation of the moving object 1 for the next inspection. This updated and corrected information may also be displayed on the display device.

[0064] During re-inspection, if an obstacle is found on the movement path when the mobile body is moved according to the determined target position and orientation, the waypoint may be modified by creating a new route to avoid the obstacle. Alternatively, a safe route may be created to avoid unfavorable conditions depending on the weather conditions (wind speed, rain, etc.) and GPS signal strength of the space where the mobile body is located. The surrounding environment of the mobile body can be measured using various sensors such as cameras, distance sensors, anemometers, and barometers. The information obtained from these sensors can be fed back to adjust the waypoints. This allows for flexible control of the mobile body in response to changes in the surrounding environment, suppressing damage to the mobile body and accidents, and ensuring reliable inspection of the target object.

[0065] Furthermore, the number of mobile units 1 is not limited to one; multiple units may be prepared and multiple mobile units may be controlled. The entire created travel path may be divided into the number of units, each mobile unit may be assigned to one of the divided travel paths on a one-to-one basis, and each mobile unit may be controlled to move through each of the divided waypoints. When controlling multiple mobile units simultaneously, each mobile unit may communicate with each other to perform autonomous control.

[0066] This disclosure relates to an information processing method, information processing device, and program for determining the target position and target orientation of a moving object including an imaging device, and has industrial applicability.

[0067] 1. Mobile device 2. Control device 3. Information processing device 4. Server

Claims

1. An information processing method for determining a target position and target orientation for a mobile body including an imaging device for inspecting an object, characterized in that a computer calculates three-dimensional data relating to the object based on a plurality of images obtained by photographing the object using the imaging device, calculates the position and orientation of the imaging device when the plurality of images were taken, determines the position and orientation of the imaging device when it photographs a part of the object that needs inspection, which has been identified based on the images obtained by photographing the object and the three-dimensional data, from among the calculated position and orientation of the imaging device, and uses the determined position and orientation to determine a target position and target orientation for the mobile body including the imaging device for photographing the part that needs inspection for subsequent inspections.

2. The information processing method according to claim 1, characterized in that changes in the area to be inspected are evaluated from the results of photographing the area using the moving body controlled based on the created target position and target orientation.

3. The information processing method according to claim 1, characterized in that waypoints for the moving body are determined, which include a plurality of target positions for photographing each of the plurality of locations requiring inspection.

4. A program for causing a computer to execute the information processing method described in any one of claims 1 to 3.

5. An information processing device for determining a target position and target orientation of a mobile body including an imaging device for inspecting an object, wherein the processing device comprises a processing unit, the processing unit calculates three-dimensional data relating to the object based on a plurality of images obtained by photographing the object using the imaging device, calculates the position and orientation of the imaging device when the plurality of images were taken, determines the position and orientation of the imaging device when a part of the object requiring inspection, identified based on the images obtained by photographing the object and the three-dimensional data, is photographed from among the calculated position and orientation of the imaging device, and determines a target position and target orientation of the mobile body including the imaging device for photographing the part requiring inspection for subsequent inspections using the determined position and orientation of the imaging device.

Images