Information processing systems, information processing methods, and programs

Abstract

To provide an information processing system, information processing method, and program for supporting the management of image data. [Solution] An information processing system comprising: a 3D data storage unit that stores partial model information and structural model information; a virtual space generation unit that generates a virtual space in which a 3D shape model of a structure is placed based on the structural model information; an image information acquisition unit that acquires an image and shooting condition information associated with the image; a shooting target identification unit that identifies a shooting target model captured in an image from among the partial shape models included in the 3D shape model of the structure placed in the virtual space, based on the shooting condition information; and an image information writing unit that links attribute information associated with the partial shape model identified as the shooting target model to the image.

Description

Technical Field

[0001] The present disclosure relates to an information processing system, an information processing method, and a program related to route design of a moving object.

Background Art

[0002] As shown in Patent Document 1, a system for inspecting a predetermined structure by photographing it with a photographing device is known.

Prior Art Document

Patent Document

[0003]

Patent Document 1

Summary of the Invention

Problems to be Solved by the Invention

[0004] In a conventional inspection system, an operator identifies an inspection site of a photographing target by referring to a design drawing or the like on a terminal or a document, and records and manages information such as the identified inspection site and information obtained at the time of inspection execution such as inspection results in a field book or the like. In this case, after the inspection, it is necessary to perform an operation of manually associating the image photographed in the inspection with the inspection site by the operator by referring to the information described in the field book or the like.

[0005] In this operation, operation mistakes such as associating an image taken at another location with the inspection site are likely to occur. In particular, in a structure where similar structural patterns continuously exist, operation mistakes of misidentifying the inspection site are more likely to occur during recording in a field book or the like and / or during the operation of associating images. Therefore, a more accurate system for managing image data taken in inspections or the like is required.

[0006] The object of the exemplary embodiments of this disclosure is to provide an information processing system, an information processing method, and a program for assisting in the management of image data. [Means for solving the problem]

[0007] The information processing system relating to one aspect of this disclosure is: A 3D data storage unit that stores multiple partial model information for showing the 3D structure of a part of a structure, and structural model information for showing the 3D structure of a structure constructed by integrating the multiple partial model information, A virtual space generation unit generates a virtual space in which a three-dimensional shape model of the structure is placed based on the aforementioned structural model information, An image information acquisition unit acquires an image obtained by capturing a real space with a camera, and shooting condition information associated with that image. A target identification unit identifies a target model that is captured in the image based on the shooting condition information, from among the partial shape models included in the three-dimensional shape model of the structure placed in the virtual space, The system includes an image information writing unit that associates attribute information associated with the partial shape model identified as the target model for photography with the image.

[0008] Because the information processing system possesses the above characteristics, it can automatically record information identifying the subject of the image in the image, thereby preventing erroneous recording and optimizing the management of image data. [Brief explanation of the drawing]

[0009] [Figure 1] Figure 1 is a diagram illustrating the configuration of an information processing system according to one embodiment of the present disclosure. [Figure 2] Figure 2 is a block diagram showing the hardware configuration of the management server shown in Figure 1. [Figure 3] Figure 3 is a block diagram showing the hardware configuration of the user terminal shown in Figure 1. [Figure 4]Figure 4 is a block diagram showing the hardware configuration of the aircraft shown in Figure 1. [Figure 5] Figure 5 is a block diagram illustrating the functions of the management server. [Figure 6] Figure 6 is a conceptual diagram showing an example of the operation of the information processing system shown in Figure 1. [Figure 7] Figure 7 is a diagram illustrating the process involved in identifying the model to be photographed. [Figure 8] Figure 8 is a flowchart showing an example of information processing performed by the information processing system shown in Figure 1. [Modes for carrying out the invention]

[0010] The information processing method, information processing system, and program disclosed herein have, for example, the following configuration. [Item 1] A 3D data storage unit that stores multiple partial model information for showing the 3D structure of a part of a structure, and structural model information for showing the 3D structure of a structure constructed by integrating the multiple partial model information, A virtual space generation unit generates a virtual space in which a three-dimensional shape model of the structure is placed based on the aforementioned structural model information, An image information acquisition unit acquires an image obtained by capturing a real space with a camera, and shooting condition information associated with that image. A target identification unit identifies a target model that is captured in the image based on the shooting condition information, from among the partial shape models included in the three-dimensional shape model of the structure placed in the virtual space, An information processing system comprising: an image information writing unit that associates attribute information associated with a partial shape model identified as the target model for photography with the image. [Item 2] The image information acquisition unit acquires information as shooting condition information, which includes the actual shooting position of the camera in real space when the image was taken, and the actual shooting direction, which is the direction in real space to which the optical central axis of the camera points. The imaging target specifying unit obtains, in the virtual space, a virtual straight line extending from a virtual imaging position corresponding to the actual imaging position in the direction of a virtual imaging direction corresponding to the actual imaging direction, based on information indicating the actual imaging position and the actual imaging direction, and specifies a partial shape model that intersects with the virtual straight line as the imaging target model. The information processing system according to item 1. [Item 3] A part of the structure indicated by each of the partial shape models corresponds to a member included in the structure. The attribute information associated with each of the partial shape models includes the name of the member and section information indicating the section in which the member is arranged in the structure. The information processing system according to item 1 or 2. [Item 4] The image information writing unit writes the name of the member and the section information associated with the partial shape model specified as the imaging target model into the auxiliary information of the image. The information processing system according to item 3. [Item 5] Storing a plurality of partial model information for showing a three-dimensional structure of a part of a structure and structure model information for showing a three-dimensional structure of the structure constructed by integrating the plurality of partial model information. Generating a virtual space in which a three-dimensional shape model of the structure is arranged based on the structure model information. Obtaining an image acquired by photographing an actual space with a camera and photographing condition information associated with the image. [[ID=*********]] Specifying, from among the partial shape models included in the three-dimensional shape model of the structure arranged in the virtual space, an imaging target model imaged in the image based on the photographing condition information. An information processing method in which a computer associates attribute information associated with a partial shape model specified as the imaging target model with the image. [Item 6] Storing a plurality of partial model information for showing a three-dimensional structure of a part of a structure and structure model information for showing a three-dimensional structure of the structure constructed by integrating the plurality of partial model information. Generating a virtual space in which a three-dimensional shape model of the structure is arranged based on the structure model information. Obtaining an image acquired by photographing the real space with a camera and photographing condition information associated with the image. Identifying a photographed object model projected onto the image based on the photographing condition information from among partial shape models included in the three-dimensional shape model of the structure arranged in the virtual space. A program for causing a computer to associate attribute information associated with the partial shape model identified as the photographed object model with the image.

[0011] <Details of the Embodiment> An information processing system according to an embodiment of the present disclosure will be described while referring to the drawings. In each of the attached drawings, the same or similar elements are given the same or similar reference numerals and names, and redundant descriptions regarding the same or similar elements may be omitted in the description of the embodiment. Note that the content shown in each drawing is merely an example for explaining the present embodiment, and is only a schematic example shown for ease of explaining the present embodiment. The content of each drawing may be modified or changed within a range where no technical problems occur.

[0012] <System Overview> The information processing system according to this embodiment is a system that performs a process to automatically record images taken during inspection of a structure, associating them with information about the object depicted in the image. A structure is something composed of a combination of multiple materials and / or members (for example, a bridge, tunnel, dam, embankment, building, factory, or other structure), and the type of structure to be inspected is not particularly limited. The inspections targeted in the operation of the information processing system may be inspections during the construction of the structure, inspections upon completion of construction, or periodic inspections after construction, and the type of inspection is not particularly limited. Furthermore, the method of photographing the structure during inspection is not particularly limited, and for example, the structure may be photographed with a camera mounted on an unmanned mobile vehicle such as a drone, unmanned aerial vehicle (UAV), or unmanned ground vehicle (UGV). In addition, a person such as a worker may operate a photographic device (for example, a smartphone, tablet terminal, digital camera, or other type of camera) to photograph the structure, or the structure may be photographed with a camera mounted on a manned mobile vehicle.

[0013] In this embodiment, for the sake of brevity, the information processing performed by the information processing system will be described in detail using the example of inspecting a bridge using the mobile body 4. As shown in Figure 6, the mobile body 4 is flown around a bridge (a bridge under construction and / or after construction is completed) and a predetermined part of the bridge is photographed with a camera 42 mounted on the mobile body 4. The mobile body 4 may be flown by a user (worker, etc.) using a remote control, or it may be autonomously navigated based on a flight path in which waypoints have been set in advance. The model and specifications of the mobile body 4 are not particularly limited. For example, it is preferable that the mobile body 4 is a model that can recognize the position of the aircraft based on the starting position of movement even in a non-GNSS (Global Navigation Satellite System) environment, and it is preferable that it is a model that can record various shooting conditions such as the position of the aircraft at the time of shooting, the orientation of the aircraft, and the orientation of the gimbal as supplementary information to the captured image (e.g., Exif information (Exif; Exchangeable image file format)).

[0014] In bridge inspections like the one shown in Figure 6, conventional methods have involved managing photographed images using field notebooks or similar documents where workers manually entered information about the inspected parts or components on-site. For example, workers would refer to design drawings or other documents on a terminal or in writing, and record the inspected parts or components in their field notebooks. Later, they would manually determine which inspected part or component each photographed corresponds to based on the information recorded in the field notebook, and then manually associate the photographed images with the information recorded in the field notebook. This conventional recording method is prone to errors, such as misreporting an incorrect image of a different location to the wrong inspection site. In particular, bridges have the characteristic of having the same structure repeated at predetermined intervals, making it easy for errors such as recording the wrong inspection location on-site or associating the wrong image with the wrong information to occur.

[0015] In this embodiment of the information processing system, a virtual space is created in which a 3D shape model of a bridge (structure) constructed by combining multiple partial shape models is placed. In this virtual space, based on the shooting condition information associated with the captured image (e.g., shooting position, shooting direction, focal length, field of view, etc.), the partial shape model corresponding to the object captured in the captured image is identified as the target model. The information processing system then performs a process to link the attribute information associated with the identified target model to the captured image. In this embodiment of the information processing system, information identifying the target of the image can be automatically recorded for the image data, thereby preventing erroneous recording, optimizing image data management, and improving the efficiency of inspections.

[0016] <System Configuration> As shown in Figure 1, the information processing system of this embodiment may include a management server 1, one or more user terminals 2, and one or more mobile devices 4. The management server 1, the user terminals 2, and the mobile devices 4 are connected to each other via a network NW so that they can communicate with one another. Note that the illustrated configuration is just an example and is not limited thereto. For example, the mobile device 4 does not have to be connected to the network NW. In that case, the mobile device 4 may be operated by a transmitter (so-called remote control) operated by the user, or image data acquired by the camera of the mobile device 4 may be stored in an auxiliary storage device connected to the mobile device 4 (for example, a memory card such as an SD card and / or a USB memory), and subsequently read from the auxiliary storage device to the user terminal 2 and / or management server 1 and stored by the user. The mobile device 4 may be connected to the network NW only for the purpose of operation or for the purpose of storing image data.

[0017] <Management Server 1> Figure 2 shows the hardware configuration of management server 1. Note that the configuration shown is just one example, and management server 1 may have a different configuration.

[0018] The management server 1 may be a general-purpose computer such as a workstation or personal computer, or it may be logically implemented through cloud computing. The management server 1 includes at least a processor 10, memory 11, storage 12, a transceiver 13, an input / output unit 14, etc., which are electrically connected to each other via a bus 15.

[0019] The processor 10 is a computing unit that controls the operation of the entire management server 1, controls the transmission and reception of data between each element, and performs information processing necessary for application execution and authentication processing. For example, the processor 10 is a CPU (Central Processing Unit) and / or a GPU (Graphics Processing Unit), and executes programs stored in the storage 12 and loaded into the memory 11 to perform various information processing tasks.

[0020] Memory 11 includes main memory composed of volatile storage devices such as DRAM (Dynamic Random Access Memory) and auxiliary memory composed of non-volatile storage devices such as flash memory and HDD (Hard Disk Drive). Memory 11 is used as a work area for the processor 10 and also stores the BIOS (Basic Input / Output System) executed when the management server 1 starts up, as well as various configuration information.

[0021] Storage 12 stores various programs, such as application programs. A database containing data used for each process may be built in storage 12. For example, the memory unit 120 described later may be provided in a part of the storage area of ​​storage 12.

[0022] The transmitting / receiving unit 13 is a communication interface for the management server 1 to communicate with user terminals 2 and mobile devices 4, etc., via a communication network. The transmitting / receiving unit 13 may further include Bluetooth® and BLE (Bluetooth Low Energy) short-range communication interfaces and / or USB (Universal Serial Bus) terminals, etc.

[0023] The input / output section 14 consists of information input devices such as keyboards and mice, and output devices such as displays.

[0024] Bus 15 is connected in common to all of the above elements and transmits, for example, address signals, data signals, and various control signals.

[0025] <User Terminal 2> The user terminal 2 shown in Figure 3 also includes a processor 20, memory 21, storage 22, a transceiver 23, an input / output unit 24, etc., which are electrically connected to each other via a bus 25. The user terminal 2 may be, for example, a transmitter or other remote control, a general-purpose computer such as a workstation or personal computer, or a mobile terminal such as a smartphone or tablet. The functions of each element of the user terminal 2 can be configured in the same way as the management server 1 described above, and a detailed explanation of each element of the user terminal 2 is omitted.

[0026] <Mobile Unit 4> Figure 4 is a block diagram showing the hardware configuration of the mobile unit 4. The flight controller 41 may have one or more processors, such as a programmable processor (e.g., a central processing unit (CPU)).

[0027] The flight controller 41 may also have a memory 411, which is accessible. The memory 411 stores logic, code, and / or program instructions that the flight controller can execute to perform one or more steps. The flight controller 41 may also include sensors 412 such as inertial sensors (accelerometers, gyroscopes), GPS sensors, and proximity sensors (e.g., LiDAR).

[0028] The memory 411 may include, for example, a separable medium such as an SD card and random access memory (RAM), or an external storage device. Data acquired from the camera / sensors 42 may be directly transmitted to and stored in the memory 411. For example, still images and video data captured by the camera may be recorded in the internal memory or external memory, but is not limited to this; the data may also be recorded from the camera / sensors 42 or the internal memory via the network NW to at least one of the management server 1 or user terminal 2. The camera 42 is mounted on the mobile body 4 via a gimbal 43.

[0029] The flight controller 41 includes a control module (not shown) configured to control the state of the mobile body 4. For example, the control module has six degrees of freedom (translational motion x, y, and z, and rotational motion θ). x θ y and θ z To adjust the spatial position, speed, and / or acceleration of the aircraft having the following features, the propulsion mechanism of the aircraft (motor 45, etc.) is controlled via the ESC 44 (Electric Speed ​​Controller). The motor 45, powered by the battery 48, rotates the propeller 46, generating lift for the aircraft. The control module of the flight controller 41 may have functions to control the drive of the camera 42 and shooting conditions (e.g., sharpness, focus (focal length), exposure value, shutter speed, ISO sensitivity, lens aperture, etc.). The control module can also control one or more of the states of the onboard parts and sensors.

[0030] The flight controller 41 can communicate with a transceiver 47 configured to transmit and / or receive data from one or more external devices (e.g., a transmitter / receiver (RC) 49, a terminal, a display device, or other remote controller). The transceiver 49 may use any suitable means of communication, such as wired or wireless communication.

[0031] For example, the transmitting / receiving unit 47 may utilize one or more of the following: local area network (LAN), wide area network (WAN), infrared, wireless, Wi-Fi, point-to-point (P2P) network, telecommunications network, cloud communication, etc.

[0032] The transmitting / receiving unit 47 can transmit and / or receive one or more of the following: data acquired by cameras / sensors 42, processing results generated by the flight controller 41, predetermined control data, and user commands from a terminal or remote controller.

[0033] The camera / sensor array 42 may include inertial sensors (accelerometers, gyroscopes), GPS sensors, proximity sensors (e.g., LiDAR), or vision / image sensors (e.g., cameras).

[0034] <Functions of Management Server 1> Figure 5 is a block diagram illustrating the functions implemented in the management server 1. In this embodiment, the management server 1 may include a virtual space generation unit 101, an image information acquisition unit 102, a target identification unit 103, an image information writing unit 104, and a display control unit 105. Furthermore, the storage unit 120 of the management server 1 may include various databases such as a 3D data storage unit 121, a captured image data storage unit 122, and a movement information storage unit 123.

[0035] First, the various databases of the storage unit 120 will be described in detail. The 3D data storage unit 121 stores multiple partial model information for showing the 3D structure of a part of a structure, and structural model information for showing the 3D structure of a structure constructed by integrating the multiple partial model information. The data format of the 3D model data, such as the partial model information and structural model information, is not necessarily limited. For example, the 3D model data may be constructed using BIM (Building Information Modeling) data, CIM (Construction Information Modeling) data, CAD data, or GML data such as City-GML, and may be constructed using two or more data formats selected from these data formats. Among the above data formats, it is particularly preferable that the 3D model data be constructed using BIM data and / or CIM data that includes not only information showing the 3D shape but also attribute information. In other words, both the partial model information and the structural model information may include a 3D shape model (3D geometry) showing the 3D shape of the respective target object, and attribute information of the target object.

[0036] The "part of the structure" indicated by the partial model information may be a "member" included in the structure, or a cohesive "part" composed of multiple members. In other words, the part of the structure indicated by each partial model information corresponds to a small unit model that is isolated when the 3D model of the structure is subdivided, and this part may be defined by subdividing the structure according to the inspection items.

[0037] In the case of the bridge illustrated in this embodiment, the 3D data storage unit 121 may store partial model information corresponding to each member included in the bridge. Examples of bridge members include deck slabs, main girders, cross girders, longitudinal girders, upper chords, lower chords, diagonal members, vertical members, bearings, piers, gussets, curbs, and guardrails. A bridge includes multiple members of the same type with matching shapes, but even such members of the same type are arranged in different positions. Therefore, for each arrangement of members of the same type included in the bridge, partial model information corresponding to each arrangement is generated and stored in the 3D data storage unit 121.

[0038] Each of the partial model information described above includes a partial shape model showing the three-dimensional shape of the target member and attribute information of the member. The attribute information of the partial model information includes, for example, the name of the member and section information that identifies the section in the bridge where the member is located. A section is an area that divides a structure into coarser groups than members (for example, a floor, room, or building of the structure), and in the case of a bridge, a section may be an area divided at a predetermined periodic interval, such as a span, a gap, or the connection interval of the main girders. If multiple members of the same type are included in the same section, the attribute information of the partial model information may include identification information such as a number to uniquely identify each member. Furthermore, the partial model information may also include, in addition to the information described above, information indicating the arrangement of components (for example, 3D coordinates indicating the position of components in real space and / or virtual space), information related to the specifications of components such as material, color, dimensions, part number, and performance, information indicating the relationship with other adjacent components (for example, information indicating which components the target component is connected to, at what location, and how), and information related to procurement history such as the price of the components and the supplier.

[0039] Structural model information may include a three-dimensional shape model of a bridge constructed by combining partial shape models, and attribute information associated with the three-dimensional shape model. The attribute information of the structural model information may include, for example, the dimensions of the bridge in the assembled state, characteristics (e.g., design specifications), location information of the bridge (e.g., three-dimensional coordinates of multiple representative points set on the outer surface of the bridge), etc. In addition, structural model information may also include information on accessories installed on the bridge (road signs, lighting, etc.), information on the relationships between adjacent members (e.g., which main girders and which cross girders are connected at which positions), information on the surrounding environment of the bridge (e.g., information related to roads connected to the bridge, information showing the positional relationship with surrounding facilities, etc.), information on the virtual space in which the three-dimensional shape model is placed (e.g., reference coordinates specified by user operation, setting of a spatial area defined by a three-dimensional coordinate system, scale of the virtual space, etc.), and information on the construction history.

[0040] The partial model information and structural model information shown above are merely examples, and each model information may be associated with information accumulated throughout the entire cycle from design to construction and maintenance. The virtual space generation unit 101, described later, reads the partial model information, structural model information, etc., stored in the 3D data storage unit 121 and generates a virtual space based on this information.

[0041] The captured image data storage unit 122 stores the captured image data acquired by the mobile device 4's cameras / sensors 42 when it photographs a structure, and transmitted from the mobile device 4 to the management server 1. The captured image data acquired by the cameras / sensors 42 includes a captured image showing at least a part of the structure, and shooting condition information associated with the captured image.

[0042] The shooting condition information includes, for example, the actual shooting position of the camera 42 in real space at the time the image was taken, and the actual shooting direction, which is the direction in real space that the optical central axis of the camera 42 points at the time the image was taken. The actual shooting position is a position expressed in a three-dimensional coordinate system in real space. When the actual shooting position is obtained using a GNSS sensor, the actual shooting position may be expressed in three-dimensional coordinates of latitude, longitude, and absolute altitude. When the actual shooting position is obtained in a non-GNSS environment using an inertial sensor (accelerometer, gyroscope), the actual shooting position may be expressed in relative three-dimensional coordinates with the starting position of the mobile body 4 (starting position of the shooting operation) as the reference position (origin). The actual shooting direction may be determined based on, for example, the orientation of the aircraft or the orientation of the gimbal at the time of shooting. In addition to the above, the shooting condition information may also include information indicating the focal length, angle of view, zoom ratio, exposure amount, ISO sensitivity, and shooting date and time of the camera 42 at the time of shooting. This shooting condition information may be metadata attached to the captured image, or it may be stored as Exif information.

[0043] Each captured image data stored in the captured image data storage unit 122 may be associated with accompanying information indicating the object to be photographed located at the target location (capture point), which has been identified by the target object identification unit 103 (described later) and linked by the image information writing unit 104. The information indicating the object to be photographed may include, for example, identification information that uniquely identifies the main subject object, the name of the object, and area information indicating the area in which the object is located. In addition to the information indicating the main subject object to be photographed, the accompanying information associated with the captured image data may also include information indicating other objects that are captured within the shooting range.

[0044] The movement information storage unit 123 stores movement information used in movements for purposes such as photographing various objects in a structure (e.g., various members included in a bridge). The movement information is instruction information for executing the movement and may include, for example, movement route information (including waypoint information), movement speed, flight altitude, imaging conditions (shooting direction, imaging angle of view, imaging focal length, overlap rate of captured images, etc.). The movement information storage unit 123 may also store movement acquisition information acquired as actual results by executing the movement. Movement acquisition information may include the actual movement route, movement speed, flight altitude, captured images, and shooting condition information associated with said captured images.

[0045] Movement information as instruction information may be generated, for example, by setting parameters of various information included in the movement information on the management server 1 or user terminal 2. As for the movement path, for example, the location of the mobile body storage device may be used as the start and end points of the movement, and a movement path passing through each waypoint may be generated. Alternatively, without a mobile body storage device, the location where the user carries the mobile body 4 may be used as the start point of the movement (so-called home point), or the user may retrieve the mobile body 4 at the end point of the movement (maybe returning to the home point). Alternatively, the movement path may be generated based on information about the mobile body storage device managed by the management server 1 (for example, location information, storage status information, storage device information, etc.), including the location of the mobile body storage device selected as the start or end point of the movement.

[0046] Next, we will describe in detail each functional part of the processor 10.

[0047] The virtual space generation unit 101 generates a virtual space in which a 3D shape model of a structure is placed, based on the structure model information. Figure 7(b) is a diagram that simulates the generated virtual space, in which a bridge model 60, which is a 3D shape model of a bridge, is placed. The bridge model 60 shown in Figure 7(b) has sections (J21-J26) defined according to the connection interval of the main girders, and each section contains multiple partial shape models (for example, partial shape models corresponding to members 61-66 in Figure 7(a)). When generating the virtual space, the information of each partial model may also be taken into consideration, and the 3D shape model of the structure may be constructed by combining the partial shape models. In the 3D shape model of the structure placed in the virtual space, the parts corresponding to each part model information (for example, each member of the bridge) may be displayed in an identifiable manner, such as by color coding.

[0048] The virtual space generation unit 101 may perform a process to establish a correlation between the three-dimensional coordinate system of the real space and the three-dimensional coordinate system of the virtual space by associating the three-dimensional coordinate system of the real space with the three-dimensional coordinate system of the virtual space when generating the virtual space. As an example, the virtual space generation unit 101 establishes a correlation between the three-dimensional coordinate system of the real space and the three-dimensional coordinate system of the virtual space by aligning and associating a reference position in the three-dimensional coordinate system of the real space (for example, the starting position of the moving object 4) with a reference position 55 (indicated by 55 in Figure 7(b)) in the three-dimensional coordinate system of the virtual space that corresponds to that position. As a result, the virtual space generation unit 101 can convert and represent the position of the moving object 4 flying in the real space in the three-dimensional coordinate system of the real space to the position of the virtual moving object 57 (a virtual object corresponding to the moving object 4 in the real space) in the virtual space. Furthermore, if the scale of the 3D shape model in the virtual space differs from the scale of the corresponding structure in the real space, the scale of the 3D shape model may be adjusted in conjunction with the alignment.

[0049] As described above, by establishing a correlation between the 3D coordinate system of the real space and the 3D coordinate system of the virtual space, even when waypoints are set based on the 3D coordinate system of the virtual space displayed on the user terminal 2, it becomes possible to specify positions such as generating movement paths based on the 3D coordinate system of the moving object 4 in the real space. If the 3D model data includes dimensional information, it becomes possible to generate movement paths using the real-world scale, such as going straight for 10m and then turning right, based on the 3D coordinate system of the virtual space.

[0050] The virtual space generation unit 101 may generate a display screen showing the generated virtual space and send it to the user terminal 2. The display screen of the user terminal 2 may allow the user to enlarge, reduce, rotate, etc., the 3D shape model (bridge model 60) placed in the virtual space according to the user's operation, and the virtual space generation unit 101 renders an image of the virtual space according to the user's operation and outputs it to the user terminal 2. In addition, the virtual space generation unit 101 may accept selection operations by the user, such as clicking on a part of the 3D shape model (bridge model 60) of the structure, via the display screen of the user terminal 2 showing the virtual space, and may perform processes such as highlighting the partial shape model corresponding to the selected part (member) (e.g., displaying it in a complementary color, enlarging only the selected partial shape model, etc.) or displaying attribute information associated with the said partial shape model.

[0051] The image information acquisition unit 102 acquires a captured image obtained by photographing the real space with the camera 42, and shooting condition information associated with the captured image. The captured image and shooting condition information acquired by the image information acquisition unit 102 are used as input information in the process of identifying the target model by the target identification unit 103, which will be described later. As shooting condition information, the image information acquisition unit 102 may acquire, for example, the actual shooting position and the actual shooting direction, and may also acquire information such as the focal length and angle of view when the captured image was taken with the camera 42.

[0052] The timing at which the image information acquisition unit 102 acquires the captured image and shooting condition information is not particularly limited. For example, the image information acquisition unit 102 may receive the captured image and shooting condition information from the mobile body 4 in real time at the time the mobile body 4 performs the shooting. Alternatively, the unit may perform the process of acquiring the captured image and shooting condition information based on user instructions at any time during or after the shooting operation by the mobile body 4.

[0053] The target identification unit 103 performs a process to identify the target model captured in the captured image from among the partial shape models included in the three-dimensional shape model of the structure placed in the virtual space, based on the shooting condition information. Figure 7 is a diagram illustrating the process of identifying the target model performed by the target identification unit 103, showing how the target member 62 captured in the captured image 50 shown in (a) is identified in the virtual space as shown in (b). The target identification unit 103 may, for example, identify the target model located at the target location in the virtual space based on the center position of the field of view of the camera 42 when the captured image 50 was taken.

[0054] Specifically, the target identification unit 103 determines the virtual shooting direction in the virtual space corresponding to the actual shooting direction of the camera 42, based on the shooting condition information obtained by the image information acquisition unit 102, which includes information indicating the actual shooting position and actual shooting direction. The virtual shooting direction can be represented by a virtual straight line extending from the virtual shooting position in the virtual space corresponding to the actual shooting position of the camera 42 to the virtual shooting direction corresponding to the actual shooting direction of the camera 42. The virtual shooting position can be identified based on the correlation between the 3D coordinate system of the real space and the 3D coordinate system of the virtual space, and the virtual shooting direction corresponds to the center position of the field of view of the camera 42. In the virtual space, the target identification unit 103 determines a virtual straight line (indicated by a dashed line in Figure 7(b)) extending from the virtual shooting position corresponding to the actual shooting position (the position of the virtual moving object 57 in Figure 7(b)) to the virtual shooting direction corresponding to the actual shooting direction, and identifies the partial shape model that intersects this virtual straight line as the target model. In Figure 7(b), among the components included in the shooting range of the captured image 50 (for example, components 61-66), the 3D shape model of component 62 (the component shown in gray in Figure 7(a) with the name "Lighting Widening Section") located within section J24 intersects with the dashed-dotted arrow and is identified as the model to be photographed.

[0055] Furthermore, even if a component is included in the shooting range of the captured image 50, components that do not intersect with a virtual straight line extending in the virtual shooting direction (for example, components 61, 63-66, etc.) may be treated as excluded components in the processing performed by the image information writing unit 104 described later. Alternatively, a partial shape model (component 62) that intersects with the virtual straight line may be designated as the main target model, and partial shape models (components 61, 63-66, etc.) that are included in the shooting range but do not intersect with the virtual straight line may be designated as secondary target models.

[0056] In the latter case, the target identification unit 103 may identify the sub-target models described above by calculating a virtual shooting range corresponding to the actual shooting range in real space, based on, for example, the angle of view information and focal length information of the camera 42, which are shooting condition information, and virtual distance information indicating the virtual distance from the virtual shooting position to the target position. Specifically, based on the angle of view information, focal length information, and virtual distance information, the unit identifies the "near plane" and "far plane" of the viewing frustum in virtual space, and calculates the range from the "near plane" to the "far plane" as the virtual shooting range. Then, it identifies the partial shape models included in the virtual shooting range (excluding components that are in the blind spot of the camera 42 and are not captured in the image). From among the partial shape models included in the virtual shooting range, the target identification unit 103 identifies the partial shape models (main target models) that intersect with a virtual straight line, and identifies the other partial shape models that do not intersect with the virtual straight line as sub-target models.

[0057] The image information writing unit 104 performs a process of associating attribute information associated with the partial shape model identified as the model to be photographed with the captured image. The attribute information to be associated with the captured image may be identification information for uniquely identifying the model to be photographed. For example, the image information writing unit 104 may associate the name of the member and / or section information (information indicating the section in which members are arranged in a structure (bridge)) from the attribute information associated with the model to be photographed with the captured image.

[0058] Furthermore, the "process of associating attribute information" with the captured image, which is performed by the image information writing unit 104, may be a process in which attribute information such as the name of a component and / or section information is written to the supplementary information of the captured image, and the captured image data including the supplementary information is stored in the captured image data storage unit 122. The file format (data format) of the supplementary information is not particularly limited, and the image information writing unit 104 may also perform a process of writing the attribute information of the model to be photographed to the Exif information of the captured image (an example of supplementary information). In the example shown in Figure 7, the image information writing unit 104 performs a process of writing the name of the component 62 identified as the model to be photographed, "Lighting Widening Section," and the section number "J24" in which the component 62 is located, to the supplementary information of the captured image as information indicating the object to be photographed (capture point).

[0059] If the target identification unit 103 identifies a secondary target model that is included within the shooting range, the image information writing unit 104 may perform a process to write the attribute information of the secondary target model to the supplementary information of the captured image. In this case, the attribute information of the secondary target model may be written to the supplementary information as information indicating other reflected elements other than the main target object (object being photographed).

[0060] The display control unit 105 reads a predetermined captured image stored in the captured image data storage unit 122 based on a user request and outputs it to the display screen of the user terminal 2. For example, the display control unit 105 may receive an image output request from the user via the user terminal 2 and output a captured image corresponding to the request. When receiving an image output request, the display control unit 105 may accept the specification of output conditions such as the date and time of shooting, the target component, and the target area, and refer to the accompanying information of each captured image data to identify a captured image corresponding to the specified output conditions. If multiple captured images satisfy the output conditions specified by the user, the display control unit 105 may output a list of the multiple captured images that satisfy the output conditions (image list). When the display control unit 105 displays a captured image (a single image or an image list) specified by the user on the user terminal 2, it may display the captured image and information that identifies the object being photographed, which is written to the accompanying information by the image information writing unit 104 (for example, the name of the component that is the object being photographed, or area information indicating the area where the component is located), side by side.

[0061] In addition to the image search process described above, the display control unit 105 may also perform a filtering process to narrow down the list of captured images to those that meet the output conditions specified by the user, or a sorting process to rearrange the order of the captured images and output a list based on sorting conditions specified by the user. In this case as well, the display control unit 105 accepts the user's specifications for filtering conditions (narrowing conditions) or sorting conditions, such as the date and time of shooting, the target component, and the target area.

[0062] The conditions for outputting captured images (search conditions, filtering conditions, sorting conditions) may be entered in text format or via a predetermined selection form such as a pull-down list. The display control unit 105 may also accept the user's specification of a target member via a UI (user interface) that displays a 3D shape model of the structure on the user terminal 2. Specifically, the display control unit 105 may accept the user's specification of any one member or section included in the 3D shape model on the UI displaying the 3D shape model of the structure (for example, by clicking to select a desired partial shape model or section). The display control unit 105 then refers to the accompanying information of each captured image data stored in the captured image data storage unit 122 to identify the captured image in which the specified member or section is written in the accompanying information. In this case, the display control unit 105 may switch from the screen displaying the 3D shape model to the screen displaying the captured image and output the identified captured image (a single image or a list of images), or it may display the 3D shape model and the identified captured image side by side on the display screen of the user terminal 2.

[0063] The functions of the display control unit 105 described above may also be performed by the processor 20 of the user terminal 2. Furthermore, the processor 10 may have other functional units not illustrated in Figure 5. For example, the processor 10 may have a functional unit that controls the movement of the mobile body 4 for the purpose of capturing actual images of various objects inside and outside a structure, based on various movement information stored in the movement information storage unit 123.

[0064] <An example of an information processing method> Next, with reference to Figure 8, the information processing method (information processing method related to the management of captured image data) by the information processing system according to this embodiment will be described. Figure 8 is a flowchart illustrating the information processing method according to this embodiment.

[0065] First, the virtual space generation unit 101 in the information processing system reads out the structure model information (and partial model information) stored in the 3D data storage unit 121 (step SQ101 in Figure 8). Then, based on the read structure model information, the virtual space generation unit 101 generates a virtual space in which the 3D shape model of the structure, constructed by combining partial shape models, is placed (step SQ102). When generating the virtual space in step SQ102, the virtual space generation unit 101 may perform a process to establish a correlation between the 3D coordinate system of the real space and the 3D coordinate system of the virtual space by associating the 3D coordinate system of the real space with the 3D coordinate system of the virtual space.

[0066] Next, the image information acquisition unit 102 acquires the captured image taken by the camera 42 of the mobile body 4 and transmitted from the mobile body 4 to the management server 1, as well as the shooting condition information associated with the captured image (step SQ103). At this time, the image information acquisition unit 102 may acquire, for example, the actual shooting position of the camera 42 in real space when the captured image was taken, and information indicating the actual shooting direction, which is the direction in real space that the optical central axis of the camera 42 is pointing, as shooting condition information. In addition, it may acquire information indicating the focal length of the camera 42 when the captured image was taken, information indicating the angle of view, etc.

[0067] Next, the target identification unit 103 identifies the target model captured in the captured image from among the partial shape models included in the three-dimensional shape model of the structure placed in the virtual space, based on the shooting condition information (step SQ104). For example, based on information indicating the actual shooting position and actual shooting direction, the target identification unit 103 determines a virtual straight line in the virtual space that extends from the virtual shooting position corresponding to the actual shooting position to the virtual shooting direction corresponding to the actual shooting direction, and identifies the partial shape model that intersects with this virtual straight line as the target model (for example, Figure 7(b)).

[0068] Subsequently, the image information writing unit 104 associates the attribute information associated with the partial shape model identified as the target model by the target model identification unit 103 with the captured image (step SQ105). The attribute information associated with the partial shape model may include, for example, information indicating the name of a component, or section information indicating the section in which the component is located within a structure. In step SQ105, the image information writing unit 104 may also write the name of the component and the section information, which are associated with the target model as attribute information, to the supplementary information of the captured image (for example, Exif information).

[0069] As described above, in the information processing system according to this embodiment, attribute information such as the names of members corresponding to partial shape models included in the 3D shape model of a structure is centrally managed, and attribute information of the target model identified in the virtual space is automatically linked to the captured image. In other words, the information processing system can prevent the misrecording of information associated with the captured image when using captured image data for structural inspections, etc. Furthermore, as described above, information identifying the target in the image can be automatically recorded for the captured image data, thereby optimizing the management of captured image data. Based on these effects, improvements in the efficiency of inspection work and the assurance of accuracy (reliability) of inspection results can be expected.

[0070] The embodiments described above are merely illustrative to facilitate understanding of this disclosure and are not intended to limit it. This disclosure may be modified and improved without departing from its intent, and its equivalents are included.

[0071] For example, in the above embodiment, the information processing system related to the management of captured image data was described using an example of inspecting a bridge using an unmanned mobile device such as a drone. However, the application examples of the information processing system are not limited to bridge inspections. The object of inspection may be other structures such as tunnels or dams, and the photography during the inspection may be performed using a portable device such as a smartphone, tablet, or other camera held by a person, or it may be performed using a manned mobile device.

[0072] Furthermore, the mobile unit 4 used for photographing structures may also be equipped with devices, equipment, etc., used to inspect for the presence or absence of predetermined events inside and / or outside the structure, in addition to the cameras / sensors 42 described above. More specifically, all devices necessary to know the condition of the structure to be inspected may be adopted, such as imaging devices (visible light cameras, infrared cameras, metal detectors, ultrasonic measuring instruments, etc.), keying devices, detection devices (metal detectors), sound collection devices, odor measuring instruments, gas detectors, air pollution measuring instruments, detection devices (devices for detecting cosmic rays, radiation, electromagnetic waves, etc.). Attribute information (names of components, etc.) of the target model identified by the target identification unit 103 may be linked to inspection data acquired by the other inspection devices described above, similar to linking to the captured images.

[0073] Furthermore, the purpose of photography in the information processing system is not limited to inspection, but may also include security, infrastructure monitoring, surveying, disaster response, etc., and the information processing system disclosed herein may be applied to photographic image data acquired through photography for these purposes.

[0074] Some or all of the functional units, such as the virtual space generation unit 101, image information acquisition unit 102, target identification unit 103, image information writing unit 104, and display control unit 105 described in the above embodiment, may be executed not by the processor 10 of the management server 1, but by the processor 20 of the user terminal 2 or the flight controller 41 of the mobile unit 4. [Explanation of symbols]

[0075] 1. Management Server 2 User terminals 4 Mobile Units

Claims

1. A three-dimensional data storage unit that stores multiple partial model information for showing the three-dimensional structure of a part of a structure, and structural model information for showing the three-dimensional structure of a structure constructed by integrating the multiple partial model information, A virtual space generation unit generates a virtual space in which a three-dimensional shape model of the structure is placed based on the aforementioned structural model information, An image information acquisition unit acquires an image obtained by capturing a real space with a camera, and shooting condition information associated with that image. A target identification unit identifies a target model that is captured in the image based on the shooting condition information, from among the partial shape models included in the three-dimensional shape model of the structure placed in the virtual space, An information processing system comprising: an image information writing unit that associates attribute information associated with a partial shape model identified as the target model for photography with the image.

2. The image information acquisition unit acquires information as shooting condition information, which includes the actual shooting position of the camera in real space when the image was taken, and the actual shooting direction, which is the direction in real space to which the optical central axis of the camera points. The information processing system according to claim 1, wherein the object identification unit determines a virtual straight line in the virtual space that extends from a virtual shooting position corresponding to the actual shooting position to a virtual shooting direction corresponding to the actual shooting direction, based on information indicating the actual shooting position and the actual shooting direction, and identifies a partial shape model that intersects the virtual straight line as the object to be photographed.

3. Each of the aforementioned partial shape models represents a part of the structure that corresponds to a member included in the structure. The information processing system according to claim 1 or 2, wherein the attribute information associated with each of the partial shape models includes the name of the member and section information indicating the section in which the member is located within the structure.

4. The information processing system according to claim 3, wherein the image information writing unit writes the name of the member associated with the partial shape model identified as the target model for photography, and the section information, to the supplementary information of the image.

5. The system stores multiple partial model information for showing the three-dimensional structure of a part of a structure, and structural model information for showing the three-dimensional structure of the structure constructed by integrating the multiple partial model information. Based on the aforementioned structural model information, a virtual space is generated in which the three-dimensional shape model of the structure is placed. This involves obtaining images acquired by capturing real space with a camera, and the shooting condition information associated with those images. From among the partial shape models included in the three-dimensional shape model of the structure placed in the virtual space, the target model captured in the image is identified based on the shooting condition information. An information processing method performed by a computer, comprising: linking attribute information associated with a partial shape model identified as the target model to be photographed to the image; and performing the following steps:

6. The system stores multiple partial model information for showing the three-dimensional structure of a part of a structure, and structural model information for showing the three-dimensional structure of the structure constructed by integrating the multiple partial model information. Based on the aforementioned structural model information, a virtual space is generated in which the three-dimensional shape model of the structure is placed. This involves obtaining images acquired by capturing real space with a camera, and the shooting condition information associated with those images. From among the partial shape models included in the three-dimensional shape model of the structure placed in the virtual space, the target model captured in the image is identified based on the shooting condition information. A program to cause a computer to perform the following actions: linking attribute information associated with a partial shape model identified as the target model to the image; and causing a computer to perform these actions.

Images