Information processing method, computer program, and information processing system

Abstract

[Problem] To provide an information processing method, a computer program, and an information processing system which enable a user to easily specify a target region in an inspection object. [Solution] An information processing method according to the present disclosure includes: photographing an inspection object by a camera provided on a moving body; detecting a target region in the inspection object on the basis of image data photographed by the camera; generating position information of the target region on the basis of position information of the moving body at the time of photographing the image data; and displaying visible information indicating the target region in the target region or in a region optically corresponding to the target region, on the basis of the position information of the target region.

Description

Information processing method, computer program, and information processing system

[0001] The present disclosure relates to an information processing method, a computer program, and an information processing system.

[0002] In recent years, drones have been used for a variety of purposes. For example, drone-based inspection methods have been proposed for inspecting objects such as utility poles. Drone-based inspection methods are effective in environments where human inspection or visual inspection is difficult, such as the inspection of steel towers, dams, and bridges. When an abnormality is detected in an inspected object by a drone, it may be necessary to confirm the abnormality visually by a human (for example, by visually confirming and repairing as necessary). In such cases, a human must identify the abnormality in the inspected object on-site, but identifying the abnormality is often difficult.

[0003] The following Patent Document 1 discloses a technology that uses images captured by a drone to construct a model in a virtual space, display content, and evaluate the condition of each location in the virtual space. Furthermore, by repeatedly capturing images, it is possible to display cumulative information about the content in the virtual space (e.g., to see changes over time).

[0004] However, even when the technology of Patent Document 1 is used to display the location of an abnormality, it only realizes a system that can confirm the location of the abnormality within a model in a virtual space, and even if this technology is used, it is difficult to identify the abnormality at the actual site.

[0005] Japanese Patent Application Laid-Open No. 2002-21465

[0006] The present disclosure provides an information processing method, a computer program, and an information processing system that enable a user to easily identify a target area detected from an inspection object by a moving body.

[0007] The information processing method disclosed herein photographs an object to be inspected using a camera mounted on a mobile body, detects a target area in the object to be inspected based on image data captured by the camera, generates positional information for the target area based on positional information of the mobile body at the time the image data was captured, and displays visible information indicating the target area in the target area or in an area optically corresponding to the target area based on the positional information for the target area.

[0008] 1 is a diagram showing an example of the overall configuration of an information processing system according to a first embodiment of the present disclosure. FIG. 1 is a block diagram showing an example of a mobile body and an information processing device. FIG. 1 is a diagram showing a first example of projecting visible information onto an abnormal location in an inspection object. FIG. 2 is a diagram showing an example of capturing an image of an area illuminated by a projector with a camera and displaying the image on a display unit of an operation device. FIG. 2 is a diagram showing a second example of projecting visible information onto an abnormal location. FIG. 3 is a diagram showing a third example of projecting visible information onto an abnormal location. FIG. 4 is a diagram explaining another example of displaying priority. FIG. 5 is a diagram showing a fourth example of projecting visible information onto an abnormal location. FIG. 6 is a diagram showing a sixth example of projecting visible information onto an abnormal location. A flowchart showing an example of operation of an information processing system in a shooting mode. A flowchart showing an example of operation of an information processing system in a projection display mode. A flowchart of an example of operation when camera shooting and projector projection are performed simultaneously in parallel. A diagram showing an example of the configuration of a mobile body equipped with the functions of an information processing device. A diagram showing an example of the configuration of an operation device equipped with the functions of an information processing device. A block diagram showing an example of a mobile body, an information processing device, and a user terminal according to a second embodiment. A diagram showing a state in which a user is pointing the camera of a user terminal towards an object to be inspected. A diagram showing an example of a display on a display unit of a user terminal. A flowchart showing an example of the operation of an information processing device and a user terminal in a through image display mode. A block diagram showing an example of a moving body, an information processing device, and an AR device in an information processing system according to a third embodiment. A diagram showing a state in which a user wearing an AR device is pointing towards an object to be inspected. A diagram showing an example of a display on a display unit (lens unit) of the AR device. A flowchart showing an example of the operation of an information processing device and an AR device in an AR display mode.

[0009] Hereinafter, embodiments of an information processing method, a computer program, and an information processing system will be described with reference to the drawings. The following description will focus on the main components of the information processing method, the computer program, and the information processing system, but the information processing method, the computer program, and the information processing system may include components and functions that are not shown or described. The following description does not exclude components and functions that are not shown or described.

[0010] First Embodiment FIG. 1 is a diagram illustrating an example of the overall configuration of an information processing system 1 according to a first embodiment of the present disclosure. The information processing system 1 of FIG. 1 includes a mobile object 100 capable of moving in a three-dimensional space, an information processing device 200 capable of wireless or wired communication with the mobile object 100, and an operation device 300 operated by a user 400 as an operator to provide various instructions to the mobile object 100 or the information processing device 200. In this embodiment, the mobile object 100 is a drone capable of flying in any environment, such as a forest, a factory, or a town. The mobile object 100 flies in space by driving a rotor 101 with a drive system such as a motor. The flight route is instructed by the user 400 or specified in advance. The mobile object according to this embodiment is not limited to a drone and may be a manned flying object such as a helicopter. Furthermore, the mobile object according to this embodiment is not limited to an flying object and may be a vehicle or a robot capable of moving on the ground.

[0011] The operation device 300 is operated by the user 400. The operation device 300 provides various instructions to the mobile object 100 or performs various settings for the mobile object 100. The operation device 300 is, for example, an operation tablet or a remote control system. The operation device 300 includes a display unit 320 that displays images and an instruction unit 310 through which the user 400 inputs various instructions. The display unit 320 may display interface images (e.g., various menu screens) for the user 400 to provide various instructions. The display unit 320 may also display map data showing the real-time environment during the flight of the mobile object 100, location information showing the current location of the mobile object 100, video data (images) captured by the mobile object 100, and the like. The operation device 300 may be capable of communicating with the information processing device 200 to provide various instructions.

[0012] The moving body 100 may fly autonomously in three-dimensional space in accordance with instruction information from the operation device 300. Alternatively, the flight of the moving body 100 may be controlled in real time by manual operation by the user 400. The moving body 100 is equipped with a projector 110 and a camera 120. The camera 120 captures images of the surrounding environment while the moving body 100 is flying. The projector 110 is also capable of projecting information or images onto any target. The positions or orientations (attitudes) of the projector 110 and the camera 120 may be controllable by the operation device 300. The camera 120 may also be zoomable by the operation device 300.

[0013] Communication between the mobile object 100 and the operation device 300 may be performed directly, or a base station may be interposed between the mobile object 100 and the operation device 300. In this case, communication between the operation device 300 and the mobile object 100 is performed via the base station. Some of the functions of the operation device 300 may be provided in the base station or the information processing device 200.

[0014] The information processing device 200 is a device that communicates wirelessly or wired with the mobile object 100 and performs information processing according to this embodiment based on image data captured by the mobile object 100 and the position and posture of the mobile object 100. The information processing device 200 may communicate with the mobile object 100 via, for example, a mobile communication network such as 5G, a wireless LAN such as Wi-Fi, or Bluetooth. The information processing device 200 is a computer device equipped with a processor such as a CPU, and is configured, for example, by a personal computer (PC) or a server. The information processing performed by the information processing device 200 will be described later.

[0015] 2 is a block diagram showing an example of a moving object 100 and an information processing device 200. The moving object 100 includes a projector 110, a camera 120, a sensor unit 130, a position / posture estimation unit 140, a control unit 150, and a storage unit 160. The information processing device 200 includes an image analysis unit 210, a storage unit 220, and an image creation unit 230.

[0016] The mobile object 100 and the information processing device 200 are each equipped with a communication unit for wireless or wired communication, and transmit and receive data via the communication unit. Fig. 2 shows only the blocks necessary for implementing the information processing according to this embodiment, and for example, in the mobile object 100, a driving unit such as a motor, a communication unit, and the like are not shown.

[0017] [Mobile Object 100] The storage unit 160 stores various information related to the mobile object 100. For example, information such as various specifications and parameters of the mobile object 100, such as information related to the parameters of the camera 120 (e.g., its position and attitude) and the parameters of the projector 110 (e.g., its position and attitude), is stored. Furthermore, map data of the environment in which the mobile object 100 moves and information necessary for the operation of the control unit 150 are stored. If the control unit 150 is a CPU, the information necessary for the operation of the control unit 150 may include an application (program) to be executed by the CPU. Some of the information stored in the storage unit 160 may be stored in the storage unit 220 of the information processing device 200. For example, information related to the parameters of the projector 110 and the camera 120 may be stored in the storage unit 220. The storage unit 160 may be configured, for example, by a non-volatile memory such as a flash memory, or a volatile memory such as a dynamic random access memory (DRAM) or a static random access memory (SRAM). The storage unit may be configured to use a configuration other than memory, such as an SSD (Solid State Drive), a hard disk, or an optical disk.

[0018] The camera 120 captures an image of the surrounding environment of the moving object 100 and generates image data. The image capture is performed, for example, at regular sampling intervals. The camera 120 has an angle of view and captures the environment within the angle of view. The camera 120 transmits the captured image data to the image analysis unit 210 of the information processing device 200. The image data may include the time when the image data was acquired. The camera 120 may be any camera, such as an RGB camera, a monochrome camera, an infrared camera, a stereo camera, or a depth camera, as long as it can capture an image of the surrounding environment.

[0019] The sensor unit 130 includes a GPS (Global Positioning System) 131 and an IMU (Inertial Measurement Unit) 132 .

[0020] The GPS 131 detects position data indicating the position of the mobile object 100. The detected position data indicates a position in a world coordinate system, specifically, a three-dimensional position including longitude, latitude, and altitude, or a two-dimensional position including longitude and latitude. However, the position data is not limited to longitude, latitude, and altitude, and may be information indicating a position in an arbitrarily defined coordinate system as long as it is possible to identify a position within the flight environment. The GPS 131 detects position data at regular time intervals and outputs the detected position data to the position / attitude estimation unit 140. The GPS 131 is an example of a position detection unit that detects the position of the mobile object 100, and position detection may be performed using a sensor other than the GPS 131 (e.g., a vision sensor).

[0021] The IMU 132 detects IMU data indicating the three-axis acceleration and three-axis angular acceleration of the moving body 100. The IMU 132 performs detection at regular time intervals and outputs the detected IMU data to the position / posture estimation unit 140. A gyro sensor and an acceleration sensor may be used as the IMU 132.

[0022] The sensor unit 130 may further include sensors other than the GPS 131 and the IMU 132, such as an infrared sensor, a geomagnetic sensor, an air pressure sensor, and a temperature sensor.

[0023] The position / orientation estimation unit 140 estimates the position and orientation of the moving body 100 based on the position data and IMU data input from the sensor unit 130. The position / orientation estimation unit 140 transmits position / orientation data indicating the estimated position and orientation to the information processing device 200. The position / orientation data may include the estimated time. The estimated time may be a time corresponding to the time when the position data and IMU data were acquired. The position / orientation data includes position information indicating the position of the moving body 100 and orientation information indicating the orientation.

[0024] The projector 110 acquires visible information generated by the information processing device 200 and projects the acquired visible information. The projection is performed within a range (projection range) according to the angle of view of the projector 110. Projecting visible information corresponds to an example of irradiating visible light based on the visible information. Information regarding the position and angle (posture) of the projector 110 may be stored in the storage unit 160 in the moving body 100 and the storage unit 220 in the information processing device 200. The projector 110 corresponds to an example of an output unit that displays visible information in a target region of the inspection object.

[0025] Projection by projector 110 and capture by camera 120 may be performed simultaneously in parallel. For example, camera 120 may be able to capture images while projecting by projector 110. The angle of view of camera 120 and the angle of view of projector 110 may at least partially overlap, and the angle of view of camera 120 may include the angle of view of projector 110.

[0026] The control unit 150 controls the operation of the moving object 100, and controls, for example, the projector 110, the camera 120, the sensor unit 130, the position / posture estimation unit 140, and other driving units (not shown). The control unit 150 is a processor such as a CPU, an ASIC, an FPGA, or a combination of these.

[0027] The information processing system 1 can execute a capture mode and a display mode. As an example, the capture mode and the projection display mode are executed during different periods, in which case the projection display mode is executed after the capture mode. For example, the projection display mode and the capture mode are executed on different days, or at different times on the same day. Alternatively, as another example, the capture mode and the projection display mode may be executed simultaneously in parallel.

[0028] In the imaging mode, the mobile object 100 performs an image capture of the object to be inspected. More specifically, the user 400 flies the mobile object 100 toward the object to be inspected and captures the object from one or more various directions to acquire image data. The image capture range may be the entire object to be inspected, or a region including a pre-specified portion of the object to be inspected. By capturing images of the object to be inspected from various directions, the position and three-dimensional shape of the object to be inspected can be determined based on the position of the mobile object 100 and the principle of triangulation, even if the camera is a monocular camera such as an RGB camera. Note that the image capture may not only capture the object to be inspected, but may also include capturing images of the surrounding environment from the departure of the mobile object 100 while flying toward the object to be inspected. Images may also be captured during the flight from the departure of the mobile object 100 to the arrival point after the image capture. In the imaging mode, the information processing device 200 performs an image capture of an abnormality (target area) in the object to be inspected based on image data acquired by the mobile object 100, as described below.

[0029] In the projection display mode, the mobile object 100 operates by using the projector 110 to project visible information indicating an abnormality onto an abnormality (target area) detected by the information processing device 200. The visible information is generated by the information processing device 200. For example, when a user 400 checks for an abnormality on-site, the projector 110 projects the visible information onto the abnormality, thereby supporting the user 400 in identifying the abnormality. For example, the visible information is projected so as to be superimposed on the abnormality using projection mapping. Projection mapping is the projection of an image onto a real object having a three-dimensional shape (e.g., the surface of a three-dimensional object). The user 400 can easily identify the abnormality from the projected visible information. Furthermore, if the visible information includes information such as text relating to the attributes of the abnormality (e.g., the type of abnormality), the user 400 can also easily identify the attributes of the abnormality from the visible information. This supports the user 400 in inspecting the abnormality.

[0030] [Information Processing Device 200] In imaging mode, the image analysis unit 210 of the information processing device 200 receives image data and position / orientation data from the mobile object 100 at regular time intervals and performs image analysis based on the received image data. In the image analysis, abnormalities (target areas) are detected for the object under inspection. The image analysis unit 210 calculates position information for the detected abnormal areas based on the position / orientation data for the mobile object 100 at the time of imaging. The position information for the abnormal areas is information indicating the position or range of the abnormal area (the position or range of the target area) in the world coordinate system. The image analysis unit 210 also detects the attributes of the abnormalities (e.g., rust, paint chipping, etc.) in the abnormal areas. When the abnormal area has multiple areas with different attributes, the image analysis unit 210 detects the position information and attributes for each area. The detection of abnormal areas and attributes may be performed by inputting image data or the analysis results of the image data into, for example, a machine-learned model or AI. Alternatively, segmentation techniques such as semantic segmentation may be used. The attributes of the detected abnormal area may be acquired based on information input by the user 400. In this case, the user 400 determines the attributes by looking at an image of the abnormal area. The image analysis unit 210 corresponds to a detection unit that detects a target area (abnormal area) in an inspection object based on image data captured by the camera 120 of the mobile object 100, and generates position information of the target area based on position information of the mobile object 100 at the time the image data was captured.

[0031] The image analysis unit 210 may also determine the priority of inspection for an abnormality based on the detected attributes. For example, the priority corresponding to an attribute may be determined based on attribute-priority correspondence data that defines a priority for each attribute. The priority is also referred to as the urgency of inspection for an abnormality. The image analysis unit 210 may also determine the priority based on information input by the user 400. In this case, the user 400 determines the priority by looking at an image of the abnormality.

[0032] The image analysis unit 210 generates content information including information about the detected abnormal area (target area), associates the content information with a data ID, and stores the content information in the storage unit 220. The content information includes at least location information of the abnormal area, and may further include at least one of an attribute and a priority. Examples of content information are shown below.

[0033] [Example of including only the position information of the abnormality location in the content information] When the abnormality location is specified by a region of one plane, the position information of the abnormality location can be specified by a plurality of coordinates indicating the range of the region. For example, when the region of the abnormality location is approximated by a polygon, the content information including the position information of the abnormality location can be expressed in the following format (1). In this example, the abnormality location is approximated by an m-sided plane (plane m) and expressed by m coordinates. The polygon may be any shape, such as a triangle, a rectangle, a pentagon, or a hexagon. {(x １ , y １ , z １ ), (x ２ , y ２ , z ２ ), ... (x ｍ , y ｍ , z ｍ ) (1)

[0034] When an abnormality is identified in two or more planar regions, the position information of the abnormality can be expressed by specifying the range of each region using multiple coordinates. For example, when an abnormality is composed of two planar regions, the content information can be expressed in the following format (2). In this example, the abnormality is expressed by a set of an m-gonal plane (plane m) and an n-gonal plane (plane n). {[(x １ , y １ , z １ ), (x ２ , y ２ , z ２ ), ... (x ｍ , y ｍ , z ｍ )], [(x １ , y １ , z １ ), (x ２ , y ２ , z ２ ), ... (x ｎ, y ｎ , z ｎ ) ] (1-1)

[0035] [Example of Including Position Information and Attributes of Anomaly Locations in Content Information] When the attribute on plane m is E1 and the attribute on plane n is E2, the content information can be expressed in the following format (2): {[E1, (x １ , y １ , z １ ), (x ２ , y ２ , z ２ ), ... (x ｍ , y ｍ , z ｍ ) ], [E2, (x １ , y １ , z １ ), (x ２ , y ２ , z ２ ), ... (x ｎ , y ｎ , z ｎ ) )]} (2) As a specific example, if the attribute E1 is rust and the attribute E2 is paint peeling, it will be as shown in the following (2-1): {[rust, (x １ , y １ , z １ ), (x ２ , y ２ , z ２ ), ... (x ｍ , y ｍ , z ｍ ) ], [Paint peeling, (x １ , y １ , z １ ), (x ２ , y ２ , z ２ ), ... (x ｎ , y ｎ , z ｎ ) ] (2-1)

[0036] When both plane m and plane n have the attribute E1, the content information may be expressed in the following format (2-2): {E1, ((x１ , y １ , z １ ), (x ２ , y ２ , z ２ ), ... (x ｍ , y ｍ , z ｍ )), ((x １ , y １ , z １ ), (x ２ , y ２ , z ２ ), ... (x ｎ , y ｎ , z ｎ )) (2-2)

[0037] When the priority in plane m is P1 and the priority in plane n is P2, the content information can be expressed in the following format (3): {[P1, (x １ , y １ , z １ ), (x ２ , y ２ , z ２ ), ... (x ｍ , y ｍ , z ｍ ) ], [P2, (x １ , y １ , z １ ), (x ２ , y ２ , z ２ ), ... (x ｎ , y ｎ , z ｎ ) )]} (3) As a specific example, if priority P1 is priority "1" and priority P2 is priority "7", it will be as shown in the following (3-1). In this example, the smaller the number, the higher the priority, but it can also be the other way around. {[1, (x １ , y １ , z １ ), (x ２ , y ２ , z ２ ), ... (x ｍ , y ｍ , z ｍ ) ], [7, (x１ , y １ , z １ ), (x ２ , y ２ , z ２ ), ... (x ｎ , y ｎ , z ｎ ) ] (3-1)

[0038] When the content information includes both the attribute and the priority, it can be expressed as in the following format (4): {[E1, P1, (x １ , y １ , z １ ), (x ２ , y ２ , z ２ ), ... (x ｍ , y ｍ , z ｍ )], [E2, P2, (x １ , y １ , z １ ), (x ２ , y ２ , z ２ ), ... (x ｎ , y ｎ , z ｎ ) ] (4)

[0039] In the example of content information described above, when the abnormal area is composed of multiple planes, information for each plane (position, attributes, priority) is stored together in one piece of content information, but separate content information may be created for each plane.

[0040] The above-described format of the content information is an example, and any format may be used as long as it can include the location information, attributes, or priority. For example, the location information may be approximated or specified by any shape other than a polygon, such as a circle, sphere, or cylinder, or a part of these shapes (e.g., a part of the side surface of a cylinder).

[0041] The content information may also include information other than the location information, attribute, or priority. For example, it may include image data including the detected abnormality. In this case, information indicating the abnormality may be added to the image data. The content information may be transmitted to the operation device 300 or another terminal in response to an instruction from the user 400, so that the user 400 can visually recognize the location of the abnormality in the image.

[0042] The storage unit 220 stores the content information generated by the image analysis unit 210. When the mobile body 100 photographs multiple inspection objects, the content information for each of the multiple inspection objects is stored in the storage unit 220. When multiple abnormalities are present at separate locations on the same inspection object, content information may be stored for each abnormality. When multiple abnormalities fit within the angle of view of the projector 110, information about these multiple abnormalities may be stored together as a single piece of content information. The storage unit 220 is configured, for example, by a non-volatile memory such as a flash memory, or a volatile memory such as a DRAM or SRAM. Configurations other than memory, such as an SSD, a hard disk, or an optical disk, may also be used as the storage unit.

[0043] In the projection display mode, the image creation unit 230 generates visible information to be projected from the flying mobile object 100 onto the abnormality location based on the content information. The visible information is display information that visually displays the location or range of the abnormality location to the user 400. For example, the image creation unit 230 detects content information from the storage unit 220, including information about the abnormality location within a range that can be irradiated by the projector 110 from the mobile object 100, and generates visible information to project onto the abnormality location. Projection mapping may be used as a method for generating and projecting the visible information. For example, the image creation unit 230 determines the three-dimensional shape of the abnormality location from multiple coordinates included in the position information in the content information, generates an image corresponding to the determined three-dimensional shape, and geometrically corrects the generated image according to the position and orientation of the projector 110 in the world coordinate system to generate the visible information for projection. The visible information may include information indicating at least one of the attributes and priority included in the content information. The information may be represented by color, text, icons, symbols, or the like.

[0044] The image creation unit 230 transmits the generated visual information to the mobile object 100 and causes the projector 110 to project the visual information onto the abnormality location. This allows the visual information to be displayed so as to overlap the abnormality location, allowing the user 400 to easily identify the location of the abnormality. Furthermore, by including at least one of the attributes and priority of the abnormality location in the visual information, the user can easily grasp not only the location of the abnormality location but also at least one of the attributes and priority of the abnormality location. The image creation unit 230 sequentially performs geometric correction on the generated image based on the position and orientation of the mobile object 100 at regular time intervals to generate and transmit visual information. This allows the projected visual information to be displayed with minimal blurring as viewed by the user, regardless of the position and orientation of the mobile object 100. In other words, even when the projection is performed while the mobile object 100 is moving or changing its orientation, the display can be displayed with minimal blurring as viewed by the user. The image creation unit 230 corresponds to a generation unit that generates visual information indicating the abnormality location (target area) based on the position information of the abnormality location (target area).

[0045] The user 400 may use the operation device 300 to specify content information to be projected and displayed in the storage unit 220. In this case, a route for flying the mobile object 100 toward the inspection target may be generated based on the location information included in the specified content information, and the mobile object 100 may be flown.

[0046] FIG. 3 is a diagram showing a first example of projecting visible information onto an abnormality 510 in an inspection object 501. It is assumed that the following content information is stored in the storage unit 220: {[rust, (x １ , y １ , z １ ), (x ２ , y ２ , z ２ ), (x ３ , y ３ , z ３ ), (x ４ , y ４ , z ４ ) ], [Paint peeling, (x ３ , y ３ , z ３ ), (x ４ , y ４ , z ４ ), (x ５ , y ５ , z ５ ), (x ６ , y ６ , z ６ ) ]} Also, assume a situation in which the user 400 is in front of the inspection target object 501 .

[0047] The image creation unit 230 of the information processing device 200 detects, based on the position and posture of the moving body 100, that the abnormal area 510 specified by the position information included in the content information is included in the projection range of the projector 110. In this example, the abnormal area 510 is part of an inspection object 501 (e.g., a steel frame) having a rectangular parallelepiped shape, and includes two planar areas 511 and 512. The area 511 is located within the above (x １ , y １ , z １ ) ~ (x ４ , y ４ , z ４ ) and the area 512 is the area specified by the above (x ３ , y３ , z ３ ) ~ (x ６ , y ６ , z ６ ) from the content information. The image creation unit 230 identifies that the attribute of the region 511 is "rust" and the attribute of the region 512 is "paint peeling." The image creation unit 230 generates visible information including a first color for the region 511 with the "rust" attribute and a second color different from the first color for the region 512 with the "paint peeling" attribute. For example, the first color is red and the second color is blue. The image creation unit 230 transmits the generated visible information to the mobile object 100, and causes the projector 110 to project the visible information onto the abnormality location 510. That is, the projector 110 irradiates the abnormality location 510 with visible light 170 including the visible information. The user 400 can recognize the location onto which the visible information is projected as the abnormality location 510. Furthermore, the user 400 can easily determine from the colors projected onto the areas 511 and 512 that the area 511 is an area with rust and the area 512 is an area with peeling paint.

[0048] It is also possible to use the camera 120 provided on the moving body 100 to capture images of the areas 511 and 512 illuminated by the projector 110 , and transmit the captured image data to the operation device 300 to display on the display unit 320 .

[0049] FIG. 4 shows an example in which areas 511 and 512 illuminated by the projector 110 are captured by the camera 120 and displayed on the display unit 320 of the operation device 300. The user 400 is not near the site where the inspection target 501 is located, but holds the operation device 300 at a remote location. The user 400 can remotely inspect the abnormality 510 by visually checking the inspection target 501 and the abnormality area 510 (areas 511 and 512) displayed on the display unit 320. At this time, the user 400 may zoom in on the screen by pinching out or the like to check the areas 511 and 512 in more detail. A configuration similar to that of FIG. 4 can also be applied to the examples of FIGS. 5 to 7 described below.

[0050] FIG. 5 is a diagram showing a second example of projecting visible information onto an abnormality location. In the above-described FIG. 3, different colors were projected onto each region depending on the attribute. In the example of FIG. 5, however, characters indicating the attribute are also projected onto the regions. The characters "rust" are projected onto region 511, and the characters "paint peeling" are projected onto region 512. This allows user 400 to more easily determine the attribute of each region. Here, characters indicating the attribute are projected, but icons evocative of the attribute, such as an icon evocative of rust or an icon evocative of paint peeling, may also be projected. Furthermore, when characters are projected, the colors of each region may be the same because the attributes can be distinguished by the characters.

[0051] FIG. 6 is a diagram showing a third example of projecting visible information onto an abnormality location. As a premise, the following content information is stored in the storage unit 220. Priority for each area is further added to the content information of the first example shown in FIG. 3 above. The priority of area 511 is "1", and the priority of area 512 is "7". {[1, rust, (x １ , y １ , z １ ), (x ２ , y ２ , z ２ ), (x ３ , y ３ , z ３ ), (x ４ , y ４ , z ４ ) ], [7, Paint peeling, (x ３ , y ３ , z ３ ), (x ４ , y ４ , z ４ ), (x ５ , y ５ , z ５ ), (x ６ , y ６ , z ６ ) ]

[0052] A first color indicating "rust" and a "1" indicating priority are projected onto the area 511. A second color indicating "paint chipping" and a "7" indicating priority are projected onto the area 512. By looking at the priority, the user 400 can determine that the inspection of the abnormality in the area 511 needs to be performed with the highest priority (over the area 512).

[0053] In the third example shown in FIG. 6, the priority is displayed numerically, but an example of displaying the priority in a way other than numerically will be described with reference to FIG.

[0054] 7 illustrates another example of displaying priority. For example, as shown in FIG. 7A, priority may be displayed by color. For example, circles filled with different colors are displayed from left to right, with red, beige, orange, yellow, and blue from the right. The rightmost color has the highest priority, and the priority decreases as you move to the left. By displaying the area of ​​the abnormality in one of these colors, the priority is notified to the user 400.

[0055] The priority may be displayed using animation, as shown in Fig. 7B. For example, the fastest blinking speed indicates the highest priority, and the slower the blinking speed, the lower the priority. By displaying the abnormal area as blinking at the speed corresponding to the priority, the priority can be notified to the user 400.

[0056] The priority may be displayed using icons as shown in Fig. 7(C). Three icons are shown in Fig. 7(C), with the rightmost icon having the highest priority and the leftmost icon having the lowest priority. The number of icons is three, but it may be two, four, or more. By projecting an image of the icon of the priority corresponding to the abnormality area, the priority can be notified to the user 400.

[0057] 3 to 6, the region 511 and the region 512 are continuous regions, but the multiple regions that make up the abnormality may be regions that are located apart from each other. An example of this case will be described with reference to FIG.

[0058] 8A and 8B are diagrams showing a fourth example of projecting visible information onto an abnormality. Fig. 8A shows an inspection target 501 and its surrounding environment in an image closer to the actual environment. Note that the mobile object 100, the user 400, and the information processing device 200 are not shown.

[0059] In Fig. 8A, the inspection target object 501 is a utility pole, and an abnormality area 510 is shown as an area 541 with a rust attribute and an area 542 with a paint peeling attribute. Fig. 8B is a partial enlarged view of area 542. Visible information including the katakana character "rust" is projected onto area 541, and visible information including the character "paint peeling" is projected onto area 542. By visually recognizing this visible information, user 400 can easily grasp the position and attribute of each area, even if the areas are located far apart.

[0060] In the examples of FIGS. 3 to 8, the visibility information includes colors for identifying regions for each attribute in the abnormal portion, but it is also possible to identify regions using other methods.

[0061] 9 shows a fifth example of projecting visible information onto an abnormality. In this example, information including a line L that highlights the outline of the area for each attribute in the abnormality is projected. The user can easily identify the area surrounded by the line L as the area for each attribute.

[0062] FIG. 10 shows a sixth example of projecting visible information onto an abnormality location. In this example, information including arrows (symbols) indicating the area of ​​each attribute at the abnormality location is projected. By looking at the arrows, a user can easily identify the area of ​​each attribute at the abnormality location. The arrows are just an example, and other symbols may be used.

[0063] 11 is a flowchart showing an example of the operation of the information processing system 1 in the imaging mode. The mobile object 100 flies along a pre-specified route in accordance with instructions from the user 400 via the operation device 300, captures images of the inspection target 501 from various directions with the camera 120, and acquires position and orientation data of the mobile object 100 with the position / orientation estimation unit 140 in synchronization with the image capture (S11). The mobile object 100 transmits the captured image data and position and orientation data to the information processing device 200.

[0064] The information processing device 200 analyzes the image data using the image analysis unit 210 based on the position and orientation data of the moving object 100 (S12). If the image analysis unit 210 detects an abnormality (target area) in the inspection target (Yes in S13), it acquires information about the detected abnormality and stores it as content information in the storage unit 220 (S14). For example, the information about the abnormality may include location information and attributes of the abnormality, and the inspection priority may be further determined based on the acquired attributes. The location information about the abnormality may include, for example, multiple coordinates that identify the range of the abnormality. These multiple coordinates can identify the shape, position, and orientation of the abnormality. Note that the format of the location information about the abnormality is not limited to multiple coordinates. For example, a shape that most closely resembles the abnormality may be identified from predetermined 3D shape candidates, and the identified shape, position, and orientation of the shape may be used as the location information about the abnormality. In this case, if the shape is reduced or enlarged, the reduction or enlargement rate may also be included.

[0065] After storing the information about the abnormality, or if no abnormality is detected in step S13 (No in S13), the mobile body 100 or the information processing device 200 determines whether to continue flying, and if not, flies the specified route and lands at a specified point (S16). If continuing flying, the process returns to step S11. As a determination of whether to continue flying, for example, if the photographing of the inspection target 501 has been completed, it may be determined not to continue flying, or if the photographing has not been completed, it may be determined to continue flying. The determination of whether photographing has been completed may be made by the user 400, or may be made by the mobile body 100 or the information processing device 200.

[0066] FIG. 12 is a flowchart showing an example of the operation of the information processing system 1 in the projection display mode. The operation of this flowchart supports the user 400 when the user 400 inspects (e.g., visually inspects) an abnormality. The moving object 100 flies along a pre-specified route in accordance with instructions from the user 400 via the operation device 300, and acquires position and orientation data at regular time intervals using the position / orientation estimation unit 140. The moving object 100 transmits the acquired position and orientation data to the information processing device 200. The image creation unit 230 of the information processing device 200 acquires the position and orientation data from the moving object 100 and determines whether the abnormality is included within the angle of view of the projector 110 of the moving object 100, i.e., within the range that can be projected by the projector 110 (S21). Specifically, the image creation unit 230 determines whether the area (abnormality) identified by the position information included in the content information stored in the storage unit 220 is at least partially included within the range that can be projected. If a plurality of pieces of content information exist in the storage unit 220, the determination is made for each piece of content information. If no abnormality is included in the projectable range (No in S21), the process proceeds to step S26.

[0067] If an abnormality is found within the projection range, the image creation unit 230 reads the corresponding content information from the storage unit 220 (S22). The image creation unit 230 creates a content image for projection mapping based on the position information, attributes, and priority included in the read content information, and geometrically corrects the created content image based on the position and orientation of the projector 110 (S24). The image creation unit 230 transmits visual information, which is the corrected content image, to the mobile object 100 and projects it from the projector 110 (S25). This displays the visual information at the abnormality. By viewing the displayed visual information, the user 400 can easily understand the location, attributes, and priority of the abnormality. If the mobile object 100 continues flying (Yes in S26), the process returns to step S21. If not, the mobile object 100 flies along the specified route and lands at the specified location (S27).

[0068] In the above-described embodiment, an example was shown in which the shooting mode of FIG. 11 and the projection display mode of FIG. 12 are executed at separate, independent timings, but it is also possible to execute shooting and display (projection) simultaneously in parallel.

[0069] FIG. 13 shows a flowchart of an example of the operation when capturing images with the camera 120 and projecting images with the projector 110 are performed simultaneously in parallel. Steps that are the same as those in FIGS. 11 and 12 are given the same reference numerals, and descriptions thereof will be omitted. Starting from step S21, steps S21 to S25 are the same as steps S21 to S25 in FIG. 12. If it is determined in step S21 that no abnormality is included in the projection range of the projector, or after step S25, the process proceeds to step S11. Steps S11 to S16 are the same as steps S11 to S16 in FIG. 11. However, if it is determined in step S15 that flight should continue, the process returns to step S21 instead of step S11.

[0070] (Alternative Configuration Example 1) It is also possible to add the functions of the information processing device 200 to the mobile object 100 and integrate the mobile object 100 and the information processing device 200 together.

[0071] 14 shows an example of the configuration of a mobile object 100A equipped with the functions of the information processing device 200. Elements with the same names as those in FIG. 2 are assigned the same reference numerals. By providing the functions of the information processing device 200 in the mobile object 100A in this way, the following effects can be obtained.

[0072] That is, poor communication conditions between the mobile object 100 and the information processing device 200 may prevent or reduce the efficiency of at least one of the image capture mode and the projection display mode. In contrast, by providing the functions of the information processing device 200 to the mobile object 100A, the image capture mode and the projection display mode can be executed more reliably regardless of the radio wave environment.

[0073] (Other Configuration Example 2) In the above-described embodiment, an image is projected by the projector 110 as visible information. However, as a modified example, a beam from a laser or the like may be irradiated onto the abnormal area. In this case, a light source such as a laser may be mounted on the mobile object 100. In this case, the beam may be moved within the area of ​​the abnormal area or along the periphery of the area. This allows the user 400 to easily grasp the location of the abnormal area.

[0074] (Other Configuration Example 3) It is also possible to add the functions of the information processing device 200 to the operation device 300 and integrate the operation device 300 and the information processing device 200 into one unit.

[0075] FIG. 15 shows a configuration example of an operation device 300A equipped with the functions of the information processing device 200. Elements with the same names as those in FIGS. 1 and 2 are assigned the same reference numerals. The operation device 300A includes a control unit 340. The control unit 340 controls the display unit 320, operation unit 310, image analysis unit 210, and image creation unit 230. The display unit 320 acquires image data captured by the camera 120 transmitted from the mobile object 100B and displays it on a screen. When visible information is projected onto the abnormality location 510 from the projector 110 of the mobile object 100, the user 400 can visually confirm the visible information at the site and also on the screen of the display unit 320. This allows the user 400 to visually identify the position of the abnormality location 510 with high accuracy and to zoom in on the screen to confirm the condition of the abnormality location 510 in detail. Furthermore, by providing the functions of the information processing device 200 to the operation device 300A, the weight of the moving object 100 can be reduced compared to the configuration in Fig. 14 described above, thereby enabling power saving of the moving object 100. Furthermore, by positioning the moving object 100 and the operation device 300A in close proximity, high-quality communication is possible, and operations in the shooting mode and the projection display mode can be reliably executed.

[0076] Second Embodiment In the first embodiment, visible information was projected onto the abnormality location in the projection display mode. In the second embodiment, however, visible information is displayed in an area optically corresponding to the abnormality location. More specifically, in a situation where an object to be inspected is photographed (video-captured) with a user terminal, a through-image display mode is performed in which visible information is superimposed on an area corresponding to the abnormality location in the image data of the video image and the image is displayed on the user terminal. The user 400 points the camera of the user terminal at the object to be inspected, captures video in real time, and displays the visual information superimposed on an area corresponding to the abnormality location while viewing the object to be inspected through the screen. As with the photographing mode and display mode in the first embodiment, the photographing mode and through-image display mode may be executed independently at different times, or may be executed simultaneously in parallel. The through-image display mode allows the user 400 to easily grasp the location, attributes, priority, etc. of the abnormality location through the screen of the user terminal.

[0077] 16 is a block diagram showing an example of a moving object 100B, an information processing device 200A, and a user terminal 500 according to the second embodiment. The user terminal 500 may additionally include the functions of the operation device 300 in FIG.

[0078] The moving body 100B basically has the same configuration as the moving body 100 of the first embodiment, but differs in that it does not include the projector 110. The information processing device 200A basically has the same configuration as the information processing device 200 of the first embodiment, but the functions of the image creation unit 230A are partially changed or expanded.

[0079] The user terminal 500 is a device held by a user, such as a tablet device or a smartphone. The user terminal 500 is capable of wireless or wired communication with the information processing device 200A. The user terminal 500 includes a camera 520, an image synthesis unit 550, a display unit 560, a sensor unit 530, and a position / posture estimation unit 540. It is assumed that the user is a human, but the user may also be a robot.

[0080] The camera 520 captures video of the environment surrounding the user terminal 500 at regular sampling intervals and generates image data. The camera 520 has a field of view and captures the environment within the field of view. The camera 520 sends the captured image data to the image synthesis unit 550.

[0081] 17 shows a state in which a user 400 points a camera 520 of a user terminal 500 toward an inspection object 501. The user 400 takes a photograph of the inspection object 501 so that all or part of the inspection object 501 is within the angle of view of the camera 520.

[0082] The camera 520 may be any camera that can capture the surrounding environment, such as an RGB camera, a monochrome camera, an infrared camera, a stereo camera, a depth camera, etc. In this embodiment, an RGB camera is assumed.

[0083] Information regarding the position and angle (posture) of the camera 520 may be stored in a storage unit (not shown) in the user terminal 500. In addition, the information may be stored in the storage unit 220 in the information processing device 200A.

[0084] The sensor unit 530 includes a GPS 531 and an inertial measurement unit (IMU) 532. The GPS 531 and the IMU 532 have the same functions as the GPS 131 and the IMU 132 included in the moving body 100B. The GPS 531 and the IMU 532 send the detected position data and IMU data, respectively, to the position / attitude estimation unit 540.

[0085] The position / orientation estimation unit 540 estimates the position and orientation of the user terminal 500 (or the camera 520) at regular time intervals based on the position data and IMU data input from the sensor unit 530. The position / orientation estimation unit 540 transmits position / orientation data indicating the estimated position and orientation to the information processing device 200A. The position / orientation data includes position information indicating the position of the user terminal 500 (or the camera 520) and orientation information including the orientation.

[0086] The image creation unit 230A of the information processing device 200A determines whether at least a portion of the abnormality in the object being inspected is included within the field of view of the camera 520 based on the position and orientation data. If included, the image creation unit 230A generates visible information to be superimposed on the abnormality in the image data captured by the camera 520. For example, the image creation unit 230A detects content information in the storage unit 220, including information about the abnormality included within the field of view of the camera 520, and generates visible information to be superimposed on the abnormality based on the position and orientation of the user terminal 500 and the position information of the abnormality. For example, the image creation unit 230A determines the three-dimensional shape of the abnormality from multiple coordinates included in the position information, generates a content image that matches the determined three-dimensional shape, and geometrically corrects the generated content image according to the position and orientation of the user terminal 500. The content image may include at least one of attribute and priority information included in the content information using color, text, icons, symbols, or the like. The image creation unit 230 transmits the geometrically corrected content image to the user terminal 500 as visible information.

[0087] The image synthesis unit 550 of the user terminal 500 superimposes the visible information from the information processing device 200A on the abnormality in the image data from the camera 520, and sends the image data on which the visible information has been superimposed to the display unit 560. The display unit 560 displays an image based on the image data on which the visible information has been superimposed. When no visible information has been input, the image synthesis unit 55 sends the image data to the display unit 560 without superimposing the visible information. By visually checking the displayed image, the user 400 can easily confirm the position, attribute, priority, etc. of the abnormality in the inspection object viewed through the camera 520. The image synthesis unit 550 or the display unit 560 corresponds to an example of an output unit that displays visible information in an area optically corresponding to the target area.

[0088] 18 shows an example of a display on the display unit 560 of the user terminal 500. The left diagram shows an example of a display before the visual information is superimposed, and the right diagram shows an example of a display after the visual information is superimposed.

[0089] 19 is a flowchart showing an example of the operation of the information processing device 200A and the user terminal 500 in the through image display mode. The user 400 moves close to the inspection target 501 in advance and points the camera of the user terminal 500 toward the inspection target 501. The image creation unit 230A of the information processing device 200A receives position and orientation data from the user terminal 500 at regular time intervals and determines whether the angle of view of the camera 520 includes an abnormality (S51). Specifically, the image creation unit 230A determines whether the area (abnormality) identified by the position information included in the content information stored in the storage unit 220 is at least partially included in the angle of view of the camera 520. If the angle of view of the camera 520 does not include an abnormality (No in S51), the process proceeds to step S56.

[0090] If the camera 520's angle of view includes an abnormality, the image creation unit 230A reads the corresponding content information from the storage unit 220 (S52). The image creation unit 230A then creates a content image based on the position information, attributes, and priority included in the content information, using parameters such as the position and orientation of the user terminal 500 (or the camera 520) and the camera 520's angle of view, and performs geometric correction on the created content image based on the position and orientation of the user terminal 500 (S53). The image creation unit 230 transmits visual information, which is the corrected content image, to the mobile object 100, and the image synthesis unit 550 superimposes the visual information on the abnormality in the image data captured by the camera 520 (S54). The display unit 560 of the user terminal 500 displays an image based on the superimposed image data. By viewing the image displayed on the display unit 560, the user 400 can easily grasp the position, attributes, and priority of the abnormality by viewing the image. If the display is to be continued (Yes in S56), the process returns to step S51; if not, the process ends. The determination as to whether to continue the display is made, for example, based on whether the user 400 has input an instruction to end the display.

[0091] As described above, according to this embodiment, since there is no need to mount a projector on the mobile body 100B, the configuration of the mobile body 100B can be simplified. This allows, for example, the weight of the mobile body 100B to be reduced. Furthermore, since there is no need for the mobile body 100B to accompany the user 400 when inspecting an abnormality in an inspection target, the user 400 can easily inspect the abnormality even in a situation where the user 400 cannot accompany the mobile body 100B (for example, when the mobile body 100B is performing other work).

[0092] (Third Embodiment) In this third embodiment, instead of the projection display mode in the first embodiment, an AR display mode is executed in which visible information is projected onto an area (an area optically corresponding to the abnormality) corresponding to the abnormality visible through the lens unit of an AR device such as AR (Augmented Reality) glasses. This allows a user 400 wearing the AR device to easily grasp the position, attributes, priority, etc. of the abnormality based on the visible information projected onto the lens unit while viewing the inspection target through the lens unit. As with the imaging mode and display mode in the first embodiment, the imaging mode and AR display mode may be executed independently at different times, or may be executed simultaneously in parallel.

[0093] 20 is a block diagram showing an example of a moving object 100B, an information processing device 200B, and an AR device 600 in an information processing system according to the third embodiment. The AR device 600 may additionally include the functions of the operation device 300 in FIG. 1 .

[0094] The moving body 100B is the same as the moving body 100B of the second embodiment. That is, the moving body 100B does not include the projector 110. The information processing device 200B basically has the same configuration as the information processing device of the first embodiment, but the functions of the image creation unit 230B are partially changed or expanded.

[0095] The AR device 600 is a device worn on the user's head, such as AR glasses. The AR device 600 is capable of wireless or wired communication with the information processing device 200B. The AR device 600 includes a display unit 660, a sensor unit 630, and a position / posture estimation unit 640.

[0096] The display unit 660 is a part that presents an image to the user 400. The display unit 660 includes a projector 661 and a lens unit 662. The lens unit 662 is a display (light-transmitting unit) made of a transparent material such as resin or glass, and the user wearing the AR device 600 recognizes the surrounding environment through the lens unit 662.

[0097] FIG. 21 shows a state in which a user 400 wearing an AR device 600 faces an object to be inspected 501.

[0098] The projector 661 is provided on a frame portion that supports the lens portion 662, and projects an image (visible information) onto the surface of the lens portion 662 as described below, thereby allowing the user 400 to view the visible information.

[0099] The sensor unit 630 includes a GPS 631 and an inertial measurement unit (IMU) 632. The GPS 631 and the IMU 632 have the same functions as the GPS 131 and the IMU 132 included in the moving body 100B. The GPS 631 and the IMU 632 send the detected position data and IMU data, respectively, to the position / attitude estimation unit 640.

[0100] The position / orientation estimation unit 640 estimates the position and orientation of the AR device 600 (or the display unit 660) at regular time intervals based on the position data and IMU data input from the sensor unit 630. The position / orientation estimation unit 640 transmits position and orientation data indicating the estimated position and orientation to the information processing device 200B. The position and orientation data includes position information indicating the position and orientation of the AR device 600 (or the display unit 660).

[0101] The image creation unit 230B of the information processing device 200B determines, based on the position and orientation data, whether at least a portion of the abnormality of the inspection target is included within the field of view of the lens unit 662. If included, the image creation unit 230B generates visible information to be displayed superimposed on an area of ​​the lens unit 662 that optically corresponds to the abnormality. For example, the image creation unit 230B detects content information containing information about the abnormality in the storage unit 220, and generates a content image based on the position and orientation of the AR device 600 and the position information of the abnormality. For example, the image creation unit 230B determines the three-dimensional shape of the abnormality from multiple coordinates included in the position information, and generates a content image that matches the determined three-dimensional shape. The image creation unit 230B geometrically corrects the generated content image according to the position and orientation of the AR device 600. The content image may include at least one of attribute and priority information included in the content information using color, text, icons, symbols, or the like. The image creation unit 230B transmits the visible information, which is the geometrically corrected content image, to the AR device 600.

[0102] A projector 661 of a display unit 660 in the AR device 600 projects visible information from the information processing device 200B onto an area in a lens unit 662 that optically corresponds to the abnormal area. By visually checking the visible information displayed in the area corresponding to the abnormal area visible through the lens unit 662, the user 400 can easily confirm the position, attributes, priority, etc. of the abnormal area through the lens unit 662. The projector 661 or the lens unit 662 corresponds to an example of an output unit that displays visible information in an area optically corresponding to the target area.

[0103] FIG. 22 shows an example of a display on the display unit 660 (lens unit 662) of the AR device 600. The inspection target 501 is within the angle of view of the display unit 660 (angle of view of the lens unit 662). The upper diagram shows an example of a display before visible information is projected, and the inspection target 501 is visible to the user 400 through the lens unit 662. The lower diagram shows an example of a display after the visible information is projected, and the visible information is projected from the projector 661 onto the inside of the lens unit 662. Specifically, visible light 670 containing visible information is irradiated onto an area optically corresponding to the abnormal part in the inspection target 501. The user 400 can easily identify the position, attributes, etc. of the abnormal part based on the position and content of the projected visible information.

[0104] FIG. 23 is a flowchart illustrating an example of the operation of the information processing device 200B and the AR device 600 in the AR display mode. Assume a situation in which the user 400 moves close to an object to be inspected, wears the AR device 600 on their head, and faces the object. The image creation unit 230B of the information processing device 200B receives position and orientation data from the AR device 600 at regular time intervals and determines whether the angle of view of the display unit 660 (the angle of view of the lens unit 662) contains an abnormality (S61). Specifically, the image creation unit 230B determines whether the area identified by the position information contained in the content information stored in the storage unit 220 is at least partially contained within the angle of view of the display unit 660. If the angle of view of the display unit 660 does not contain an abnormality (No in S61), the process proceeds to step S66.

[0105] If the angle of view of the display unit 660 includes an abnormality, the image creation unit 230B reads the corresponding content information from the storage unit 220 (S62). The image creation unit 230B then creates a content image based on the position information, attributes, and priority included in the content information, using parameters such as the position and orientation of the AR device 600 (or the display unit 660) and the angle of view of the display unit 660 (S63). The created content image is geometrically corrected based on the position and orientation of the AR device 600 (S64). The image creation unit 230 transmits visible information, which is the geometrically corrected content image, to the AR device 600, and the projector 661 of the display unit 660 projects the visible information onto an area of ​​the lens unit 662 that optically corresponds to the abnormality (S65). The user 400 can easily grasp the position, attributes, and priority of the abnormality by viewing the visible information projected onto the lens unit 662. If the display is to be continued (Yes in S66), the process returns to step S61, and if not, the process ends. The determination of whether to continue the display is made based on, for example, whether the user 400 has input an instruction to end the display.

[0106] As described above, according to this embodiment, since there is no need to mount a projector on the mobile body 100B, the configuration of the mobile body 100B can be simplified. This allows, for example, the weight of the mobile body 100B to be reduced. Furthermore, since there is no need for the mobile body 100B to accompany the user 400 when inspecting an abnormality in an inspection target, the user 400 can easily inspect the abnormality even in a situation where the user 400 cannot accompany the mobile body 100B (for example, when the mobile body 100B is performing other work).

[0107] The present technology can be configured as follows. [Item 1] An information processing method comprising: photographing an object to be inspected with a camera mounted on a mobile body; detecting a target area in the object to be inspected based on image data captured by the camera; generating position information for the target area based on position information for the mobile body at the time the image data was captured; and displaying visible information indicating the target area in the target area or in an area optically corresponding to the target area based on the position information for the target area. [Item 2] The information processing method according to Item 1, wherein attributes of the target area are acquired based on the image data or information input by a user, and the visible information includes information indicating the attributes of the target area. [Item 3] The information processing method according to Item 2, wherein the visible information includes different colors depending on the attributes of the target area. [Item 4] The information processing method according to Item 2 or 3, wherein the visible information includes characters indicating the attributes of the target area. [Item 5] The information processing method according to any one of Items 1 to 4, wherein a priority for the target area is acquired based on the image data or information input by a user, and the visible information includes information indicating the priority. [Item 6] The information processing method according to Item 5, wherein the visible information includes a number representing the priority. [Item 7] The information processing method according to Item 5 or 6, wherein the visible information includes a different color depending on the priority. [Item 8] The information processing method according to any one of Items 1 to 7, wherein the visible information includes a symbol indicating the position of the target area. [Item 9] The information processing method according to any one of Items 1 to 8, wherein the generated position information of the target area is stored, the target area existing within an angle of view of a projector of the moving object is detected based on the stored position information of the target area and position information of the moving object, and the visible information is projected onto the detected target area based on the position information of the detected target area. [Item 10] The information processing method according to Item 9, wherein the position information of the target area includes a plurality of coordinates that specify a shape of the target area, the image corresponding to the shape of the target area is generated based on the plurality of coordinates and the position information of the moving object, and the image is projected onto the target area by the projector.[Item 11] The information processing method according to any one of items 1 to 10, comprising storing position information of the generated target area, detecting the target area that exists within an angle of view of a light-transmitting portion of the AR glasses based on the stored position information of the target area and the position of the AR glasses worn by a user, and projecting the visible information onto an area of ​​the light-transmitting portion that optically corresponds to the detected target area. [Item 12] The information processing method according to item 11, wherein the position information of the target area includes a plurality of coordinates that specify a shape of the target area, generating an image according to the shape of the target area based on the plurality of coordinates and the position of the moving object, and projecting the image onto the area of ​​the light-transmitting portion that optically corresponds to the detected target area. [Item 13] The information processing method according to any one of items 1 to 12, comprising: storing position information of the generated target area; detecting the target area that exists within the angle of view of a camera included in a device held by a user and the stored position information of the target area; arranging the visible information in an area that optically corresponds to the target area in image data captured by the camera of the device; and displaying the image data with the visual information arranged on a display unit included in the device. [Item 14] The information processing method according to item 13, wherein the position information of the target area includes a plurality of coordinates that specify a shape of the target area; generating an image corresponding to the shape of the target area based on the plurality of coordinates and the position information of the moving object; and overlaying the image on the target area in the image data. [Item 15] A computer program for causing a computer to execute the following: photographing an object to be inspected with a camera mounted on a mobile body; detecting a target area in the object to be inspected based on image data photographed by the camera; generating position information of the target area based on position information of the mobile body at the time the image data was photographed; and displaying visible information indicating the target area in the target area or in an area optically corresponding to the target area based on the position information of the target area.[Item 16] An information processing system comprising: a mobile body including a camera that captures an image of an object to be inspected; a detection unit that detects a target area in the object to be inspected based on image data captured by the camera of the mobile body and generates position information of the target area based on position information of the mobile body at the time the image data was captured; a generation unit that generates visible information indicating the target area based on the position information of the target area; and an output unit that displays the visible information on the target area or an area optically corresponding to the target area.

[0108] The aspects of the present disclosure are not limited to the individual embodiments described above, but include various modifications that may be conceived by those skilled in the art, and the effects of the present disclosure are not limited to the above-described contents. In other words, various additions, modifications, and partial deletions are possible within the scope of the conceptual idea and spirit of the present disclosure, which is derived from the contents defined in the claims and their equivalents.

[0109] 1 Information processing system 100 Mobile body 100A Mobile body 100B Mobile body 101 Rotor 110 Projector 120 Camera 130 Sensor unit 131 GPS 132 Inertial measurement unit (IMU) 140 Attitude estimation unit 150 Control unit 160 Memory unit 170 Visible light 200 Information processing device 200A Information processing device 200B Information processing device 210 Image analysis unit 220 Memory unit 230 Image creation unit 230A Image creation unit 230B Image creation unit 300 Operation device 310 Instruction unit 320 Display unit 400 User 500 User terminal 501 Inspection object 510 Abnormal part 511 Area 512 Area 520 Camera 530 Sensor unit 531 GPS 532 Inertial measurement unit (IMU) 540 Attitude estimation unit 541 Area 542 Area 550 Image synthesis unit 560 Display unit 600 AR device 630 Sensor unit 631 GPS 632 Inertial measurement unit (IMU) 640 Attitude estimation unit 660 Display unit 661 Projector 662 Lens unit

Claims

The object to be inspected is photographed by a camera installed on the moving object, Detecting a target area in the inspection object based on image data captured by the camera; generating position information of the target area based on position information of the moving object at the time the image data was captured; based on the position information of the target area, visual information indicating the target area is displayed in the target area or in an area optically corresponding to the target area; Information processing methods. 　 acquiring attributes of the target region based on the image data or user input information; the visibility information includes information indicating attributes of the target region; The information processing method according to claim 1 . 　 the visual information includes different colors depending on the attributes of the target region; The information processing method according to claim 2 . 　 the visible information includes characters representing attributes of the target region; The information processing method according to claim 2 . 　 obtaining a priority for the target region based on the image data or user input information; the visibility information includes information indicating the priority; The information processing method according to claim 1 . 　 the visible information includes a number representing the priority; The information processing method according to claim 5 . 　 the visual information includes different colors depending on the priority; The information processing method according to claim 5 . 　 the visible information includes a symbol indicating the location of the target area; The information processing method according to claim 1 . 　 storing the generated position information of the target area; detecting the target area present within an angle of view of a projector of the moving object based on the stored position information of the target area and position information of the moving object; Projecting the visible information onto the detected target area based on the position information of the detected target area. The information processing method according to claim 1 . 　 the position information of the target area includes a plurality of coordinates that identify a shape of the target area; generating an image according to the shape of the target area based on the plurality of coordinates and the position information of the moving object; projecting the image onto the target area with the projector; The information processing method according to claim 9. 　 storing the generated position information of the target area; Detecting the target area present within an angle of view of a light transmitting portion of the AR glasses based on the stored position information of the target area and a position of the AR glasses worn by the user; projecting the visible information onto an area of ​​the light-transmitting portion that optically corresponds to the detected target area; The information processing method according to claim 1 . 　 the position information of the target area includes a plurality of coordinates that identify a shape of the target area; generating an image according to the shape of the target area based on the plurality of coordinates and the position of the moving object; projecting the image onto the area of ​​the light transmissive portion that optically corresponds to the detected area of ​​interest; The information processing method according to claim 11. 　 storing the generated position information of the target area; Detecting the stored position information of the target area and the target area existing within the angle of view of a camera included in a device held by a user; arranging the visible information in an area optically corresponding to the target area in image data captured by the camera of the device, and displaying the image data in which the visible information is arranged on a display unit provided in the device; The information processing method according to claim 1 . 　 the position information of the target area includes a plurality of coordinates that identify a shape of the target area; generating an image according to the shape of the target area based on the plurality of coordinates and the position information of the moving object; overlaying the image onto the region of interest in the image data; The information processing method according to claim 13. 　 The object to be inspected is photographed by a camera installed on the moving object, Detecting a target area in the inspection object based on image data captured by the camera; generating position information of the target area based on position information of the moving object at the time the image data was captured; based on the position information of the target area, visual information indicating the target area is displayed in the target area or in an area optically corresponding to the target area; A computer program that is executed by a computer. 　 a moving body including a camera that captures an image of an object to be inspected; a detection unit that detects a target area in the inspection object based on image data captured by the camera of the moving body, and generates position information of the target area based on position information of the moving body at the time the image data was captured; a generating unit that generates visible information indicating the target area based on the position information of the target area; an output unit that displays the visible information in the target area or an area optically corresponding to the target area; An information processing system comprising:

Images