Information Processing System, Information Processing Method, and Program

Abstract

To provide an information processing system, an information processing method, and a program for assisting the management of image data. 【Solution means】 An information processing system including: a three-dimensional data storage unit that stores partial model information and structure model information; a virtual space generation unit that generates a virtual space in which a three-dimensional shape model of a structure is arranged based on the structure model information; an image information acquisition unit that acquires an image and shooting condition information associated with the image; a shooting target identification unit that identifies a shooting target model projected onto the image from among partial shape models included in the three-dimensional shape model of the structure arranged in the virtual space based on the shooting condition information; and an image information writing unit that associates attribute information associated with the partial shape model identified as the shooting target model with the image.

Description

Technical Field

[0001] The present disclosure relates to an information processing system, an information processing method, and a program related to the route design of a moving body.

Background Art

[0002] As shown in Patent Document 1, a system for photographing and inspecting a predetermined structure with a photographing device is known.

Prior Art Documents

Patent Documents

[0003]

Patent Document 1

Summary of the Invention

Problems to be Solved by the Invention

[0004] In a conventional inspection system, an operator identifies an inspection site of a photographing target by referring to a design drawing or the like on a terminal or on paper, and records and manages information obtained at the time of inspection, such as the identified inspection site and inspection results, in a field book or the like. In this case, after the inspection, it is necessary to perform an operation of associating the image taken in the inspection with the inspection site by manual operation by the operator with reference to the information described in the field book or the like.

[0005] In this operation, operation errors such as associating an image taken at another location with the inspection site are likely to occur. In particular, in a structure where similar structural patterns continuously exist, operation errors of misidentifying the inspection site are more likely to occur during recording in a field book or the like and / or during the operation of associating images. Therefore, a more accurate system for managing image data taken in inspections and the like is required.

[0006] An object of an exemplary embodiment of the present disclosure is to provide an information processing system, an information processing method, and a program for assisting in the management of image data.

Means for Solving the Problem

[0007] An information processing system according to an aspect of the present disclosure includes a three-dimensional data storage unit that stores a plurality of partial model information for showing a three-dimensional structure of a part of a structure and structure model information for showing a three-dimensional structure of the structure constructed by integrating the plurality of partial model information; a virtual space generation unit that generates a virtual space in which a three-dimensional shape model of the structure is arranged based on the structure model information; an image information acquisition unit that acquires an image obtained by photographing a real space with a camera and photographing condition information associated with the image; a photographing target specifying unit that specifies a photographing target model projected onto the image based on the photographing condition information from among partial shape models included in the three-dimensional shape model of the structure arranged in the virtual space; an image information writing unit that associates attribute information associated with the partial shape model specified as the photographing target model with the image.

[0008] By having the above characteristics, the information processing system can automatically record information for specifying the photographing target of the image with respect to the image data, preventing misrecording and optimizing the management of the image data.

Brief Description of the Drawings

[0009]

Figure 1

Figure 2

Figure 3

Figure 4

Figure 5

Figure 6

Figure 7

Figure 8

Embodiments for Carrying Out the Invention

[0010] The information processing method, information processing system, and program of the present disclosure are configured as follows, for example. [Item 1] A three-dimensional data storage unit that stores a plurality of partial model information for showing a three-dimensional structure of a part of a structure and structure model information for showing a three-dimensional structure of the structure constructed by integrating the plurality of partial model information; A virtual space generation unit that generates a virtual space in which a three-dimensional shape model of the structure is arranged based on the structure model information; An image information acquisition unit that acquires an image obtained by photographing the real space with a camera and photographing condition information associated with the image; A photographed object specifying unit that specifies a photographed object model projected onto the image based on the photographing condition information from among the partial shape models included in the three-dimensional shape model of the structure arranged in the virtual space; An information processing system including an image information writing unit that associates attribute information associated with the partial shape model specified as the photographed object model with the image. [Item 2] The image information acquisition unit acquires, as the photographing condition information, information indicating the actual photographing position of the camera in the real space when the image is photographed and the actual photographing direction in the real space in which the optical central axis of the camera is directed. The imaging target specifying unit obtains, in the virtual space, a virtual straight line extending from a virtual imaging position corresponding to the actual imaging position in the direction of a virtual imaging direction corresponding to the actual imaging direction, based on information indicating the actual imaging position and the actual imaging direction, and specifies, as the imaging target model, a partial shape model that intersects with the virtual straight line. The information processing system according to item 1. [Item 3] A part of the structure indicated by each of the partial shape models corresponds to a member included in the structure. The attribute information associated with each of the partial shape models includes the name of the member and section information indicating the section in which the member is arranged in the structure. The information processing system according to item 1 or 2. [Item 4] The image information writing unit writes the name of the member and the section information associated with the partial shape model specified as the imaging target model into the auxiliary information of the image. The information processing system according to item 3. [Item 5] Storing a plurality of partial model information for showing the three-dimensional structure of a part of a structure and structure model information for showing the three-dimensional structure of the structure constructed by integrating the plurality of partial model information; Generating a virtual space in which a three-dimensional shape model of the structure is arranged based on the structure model information; Obtaining an image acquired by photographing an actual space with a camera and photographing condition information associated with the image; Specifying, from among the partial shape models included in the three-dimensional shape model of the structure arranged in the virtual space, an imaging target model projected onto the image based on the photographing condition information; An information processing method executed by a computer of associating attribute information associated with the partial shape model specified as the imaging target model with the image. [Item 6] Storing a plurality of partial model information for showing a three-dimensional structure of a part of a structure and structure model information for showing a three-dimensional structure of the structure constructed by integrating the plurality of partial model information. Generating a virtual space in which a three-dimensional shape model of the structure is arranged based on the structure model information. Obtaining an image acquired by photographing a real space with a camera and photographing condition information associated with the image. Identifying a photographed object model projected onto the image based on the photographing condition information from among partial shape models included in the three-dimensional shape model of the structure arranged in the virtual space. A program for causing a computer to associate attribute information associated with the partial shape model identified as the photographed object model with the image.

[0011] <Details of the Embodiment> An information processing system according to an embodiment of the present disclosure will be described with reference to the drawings. In each of the attached drawings, the same or similar elements are given the same or similar reference numerals and names, and duplicate descriptions regarding the same or similar elements may be omitted in the description of the embodiment. Note that the content shown in each drawing is merely an example for explaining the present embodiment, and is only a schematic example shown for ease of explaining the present embodiment. The content of each drawing may be modified or changed within a range where no technical problem occurs.

[0012] <System Outline> The information processing system according to this embodiment is a system that executes a process of automatically recording an image taken at the time of inspecting a structure in association with information about an object projected onto the image. A structure is composed of a combination of a plurality of materials and / or members (for example, bridges, tunnels, dams, levees, buildings, factories, and other buildings, etc.), and the type of structure to be inspected is not particularly limited. The inspection targeted in the operation of the information processing system may be an inspection during the construction of a structure, an inspection at the completion of construction, or a regular inspection after construction, etc., and the type of inspection is not particularly limited. Also, the method of photographing a structure at the time of inspection is not particularly limited. For example, a structure may be photographed with a camera mounted on an unmanned moving body such as a drone, an unmanned aerial vehicle (UAV), or an unmanned ground vehicle (UGV). Alternatively, a person such as an operator may operate a photographing device (for example, various cameras such as a smartphone, a tablet terminal, or a digital camera) to photograph the structure, or a structure may be photographed with a camera mounted on a manned moving body.

[0013] In this embodiment, for the sake of simplicity, the case of inspecting a bridge using the moving body 4 will be exemplified, and the information processing executed by the information processing system will be described in detail. As shown in FIG. 6, the moving body 4 is flown around a bridge (during construction and / or after construction), and a predetermined part of the bridge is photographed with the camera 42 mounted on the moving body 4. The moving body 4 may be flown by the operation of a user (such as an operator) using a prop, or may be autonomously navigated based on a flight path in which waypoints and the like are set in advance. The model and specifications of the moving body 4 are not particularly limited. For example, the moving body 4 is preferably a body with a specification that can recognize its body position based on the starting position of movement even in a non-GNSS (Global Navigation Satellite System) environment, and is preferably a body with a specification that can record various shooting conditions such as the body position, the orientation of the body, and the orientation of the gimbal at the time of shooting as attached information (e.g., Exif information (Exif: Exchangeable image file format)) of the photographed image.

[0014] In the inspection of a bridge as shown in FIG. 6, conventionally, the management of photographed images has been performed using a field book or the like in which an operator manually enters the inspected parts or members on site. For example, while referring to design drawings or the like on a terminal or on paper, the operator writes down the photographed inspection parts or members in a field book or the like. Then, at a later date, the operator manually determines which inspected part or member the photographed image corresponds to based on the information recorded in the field book or the like, and associates the photographed image with the information recorded in the field book or the like by manual operation. In such a conventional recording method, operational errors are likely to occur, such as misrecording by associating a different image photographed at a different location with the inspection location. In particular, since a bridge has the characteristic that the same structure continues at a predetermined interval, operational errors such as misrecording the inspection location on site or associating incorrect images with information are likely to occur.

[0015] In the information processing system of the present embodiment, after generating a virtual space in which a three-dimensional shape model of a bridge (structure) constructed by combining a plurality of partial shape models is arranged, in the virtual space, based on the photographing condition information (for example, photographing position, photographing direction, focal length, angle of view, etc.) associated with the photographed image, the partial shape model corresponding to the object shown in the photographed image is specified as the photographing target model. Then, the information processing system executes a process of associating the attribute information associated with the specified photographing target model with the photographed image. In the information processing system of the present embodiment, by the above-described information processing, it is possible to automatically record information for specifying the photographing target of the image for the image data, and to prevent misrecording, optimize the management of the image data, and improve the efficiency of inspection.

[0016] <System Configuration> As shown in FIG. 1, the information processing system of this embodiment may include a management server 1, one or more user terminals 2, and one or more mobile bodies 4. The management server 1, the user terminal 2, and the mobile body 4 are communicably connected to each other via a network NW. Note that the illustrated configuration is an example and is not limited thereto. For example, the mobile body 4 may not be connected to the network NW. In that case, the operation of the mobile body 4 may be performed by a transmitter (so-called prop) operated by the user, or the image data acquired by the camera of the mobile body 4 may be stored in an auxiliary storage device (for example, a memory card such as an SD card and / or a USB memory, etc.) connected to the mobile body 4, and read out from the auxiliary storage device to the user terminal 2 and / or the management server 1 by the user and stored afterwards. The mobile body 4 may be connected to the network NW only for either the purpose of operation or the purpose of storing image data.

[0017] <Management server 1> FIG. 2 is a diagram showing the hardware configuration of the management server 1. Note that the illustrated configuration is an example, and the management server 1 may have other configurations.

[0018] The management server 1 may be a general-purpose computer such as a workstation or a personal computer, or may be logically realized by cloud computing. The management server 1 includes at least a processor 10, a memory 11, a storage 12, a transmission / reception unit 13, an input / output unit 14, etc., and these are electrically connected to each other through a bus 15.

[0019] The processor 10 is an arithmetic device that controls the operation of the entire management server 1, controls the transmission and reception of data between each element, and performs information processing necessary for the execution and authentication processing of applications. For example, the processor 10 is a CPU (Central Processing Unit) and / or a GPU (Graphics Processing Unit), and executes programs stored in the storage 12 and expanded in the memory 11 to perform each information processing.

[0020] The memory 11 includes a main memory composed of a volatile storage device such as a DRAM (Dynamic Random Access Memory), and an auxiliary memory composed of a non-volatile storage device such as a flash memory or an HDD (Hard Disc Drive). The memory 11 is used as a work area of the processor 10 and stores a BIOS (Basic Input / Output System) executed when the management server 1 is started up, and various setting information and the like.

[0021] The storage 12 stores various programs such as application programs. A database storing data used for each process may be constructed in the storage 12. For example, the storage unit 120 described later may be provided in a part of the storage area of the storage 12.

[0022] The transmission / reception unit 13 is a communication interface for the management server 1 to communicate with the user terminal 2, the mobile body 4, etc. via a communication network. The transmission / reception unit 13 may further include a short-range communication interface for Bluetooth (registered trademark) and BLE (Bluetooth Low Energy) and / or a USB (Universal Serial Bus) terminal or the like.

[0023] The input / output unit 14 is an information input device such as a keyboard and a mouse, and an output device such as a display.

[0024] The bus 15 is commonly connected to the above elements and transmits, for example, an address signal, a data signal, and various control signals.

[0025] <User terminal 2> The user terminal 2 shown in FIG. 3 also includes a processor 20, a memory 21, a storage 22, a transceiver 23, an input / output unit 24, etc., which are electrically connected to each other through a bus 25. The user terminal 2 may be, for example, a prop such as a transmitter, or a general-purpose computer such as a workstation and a personal computer, or a mobile terminal such as a smartphone and a tablet terminal. The functions of the respective elements of the user terminal 2 can be configured in the same manner as those of the management server 1 described above, and detailed descriptions of the respective elements of the user terminal 2 are omitted.

[0026] <Mobile body 4> FIG. 4 is a block diagram showing the hardware configuration of the mobile body 4. The flight controller 41 may have one or more processors such as a programmable processor (for example, a central processing unit (CPU)).

[0027] Further, the flight controller 41 may have a memory 411 and be accessible to the memory. The memory 411 stores logic, code, and / or program instructions executable by the flight controller to perform one or more steps. Further, the flight controller 41 may include sensors 412 such as inertial sensors (acceleration sensors, gyro sensors), GPS sensors, proximity sensors (for example, lidar), etc.

[0028] The memory 411 may include a separable medium or an external storage device such as an SD card and a random access memory (RAM). The data acquired from the camera / sensors 42 may be directly transmitted to and stored in the memory 411. For example, still image / moving image data captured by a camera or the like may be recorded in the built-in memory or an external memory, but not limited thereto, and may be recorded in at least one of the management server 1 and the user terminal 2 via the network NW from the camera / sensor 42 or the built-in memory. The camera 42 is installed on the mobile body 4 via a gimbal 43.

[0029] The flight controller 41 includes a control module (not shown) configured to control the state of the mobile body 4. For example, the control module controls the propulsion mechanism (such as the motor 45) of the flying object via the ESC 44 (Electric Speed Controller) in order to adjust the spatial arrangement, speed, and / or acceleration of the flying object having six degrees of freedom (translational motions x, y, and z, and rotational motions θ x , θ y and θ z ). The propeller 46 is rotated by the motor 45 powered by the battery 48 to generate lift for the flying object. The control module of the flight controller 41 may have a function of controlling the driving of the camera 42 and shooting conditions (such as sharpness, focus (focal length), exposure value, shutter speed, ISO sensitivity, lens aperture, etc.). Further, the control module can control one or more of the mounting part and the states of the sensors.

[0030] The flight controller 41 is communicable with a transceiver unit 47 configured to transmit and / or receive data from one or more external devices (such as a transceiver (prop) 49, a terminal, a display device, or another remote controller). The transceiver 49 may use any suitable communication means such as wired communication or wireless communication.

[0031] For example, the transceiver unit 47 may utilize one or more of a local area network (LAN), a wide area network (WAN), infrared, wireless, WiFi, a point-to-point (P2P) network, a telecommunications network, cloud communication, etc.

[0032] The transceiver unit 47 can transmit and / or receive one or more of the data acquired by the camera / sensors 42, the processing results generated by the flight controller 41, predetermined control data, user commands from the terminal or the remote controller, etc.

[0033] The cameras / sensors 42 may include inertial sensors (acceleration sensors, gyro sensors), GPS sensors, proximity sensors (e.g., lidar), or vision / image sensors (e.g., cameras).

[0034] <Functions of Management Server 1> FIG. 5 is a block diagram illustrating the functions implemented in the management server 1. In the present embodiment, the management server 1 may include a virtual space generation unit 101, an image information acquisition unit 102, a shooting target identification unit 103, an image information writing unit 104, and a display control unit 105. Further, the storage unit 120 of the management server 1 may include various databases such as a three-dimensional data storage unit 121, a captured image data storage unit 122, and a movement information storage unit 123.

[0035] First, the various databases in the storage unit 120 will be described in detail. The three-dimensional data storage unit 121 stores a plurality of partial model information for showing the three-dimensional structure of a part of a structure and structure model information for showing the three-dimensional structure of the structure constructed by integrating the plurality of partial model information. The data format of three-dimensional model data such as partial model information and structure model information is not necessarily limited. For example, the three-dimensional model data may be constructed by BIM (Building Information Modeling) data, CIM (Construction Information Modeling) data, CAD data, or GML data such as City-GML, etc., or may be constructed by two or more data formats selected from these data formats. Among the data formats as described above, the three-dimensional model data is preferably constructed by BIM data and / or CIM data that include not only information showing a three-dimensional shape but also attribute information. That is, the partial model information and the structure model information may each include a three-dimensional shape model (three-dimensional geometry) showing the three-dimensional shape of the respective target object and the attribute information of the target object.

[0036] The "part of the structure" indicated by the partial model information may be a "member" included in the structure or a coherent "portion" composed of a plurality of members. In other words, the part of the structure indicated by each partial model information corresponds to a small unit model isolated when the three-dimensional model of the structure is subdivided, and the part may be set by subdividing the structure according to the inspection items.

[0037] In the case of the bridge exemplified in this embodiment, the three-dimensional data storage unit 121 may store partial model information corresponding to each member included in the bridge for each member. Examples of the members of the bridge include, for example, floor slabs, main girders, cross girders, longitudinal girders, upper chord members, lower chord members, diagonal members, vertical members, bearings, bridge piers, gussets, ground coverings, guardrails, and the like. The bridge includes a plurality of members of the same type with the same shape, but even such members of the same type are arranged at different positions. Therefore, for the members of the same type included in the bridge, partial model information corresponding to each arrangement is generated and stored in the three-dimensional data storage unit 121.

[0038] Each piece of the above-mentioned partial model information includes a partial shape model showing the three-dimensional shape of the target member and the attribute information of the member. The attribute information of the partial model information includes, for example, the name of the member and the section information specifying the section where the member is arranged in the bridge. A section is a region that divides a structure into coarser aggregates than members (for example, floors, rooms, buildings, etc. of a structure), and a section in a bridge may be a region divided at periodic predetermined intervals such as, for example, spans, bays, and connection intervals of main girders. In the case where a plurality of the same type of members are included in the same section, the attribute information of the partial model information may include identification information such as a number for uniquely identifying each member. In addition to the above information, the partial model information may also include information indicating the arrangement of the members (for example, three-dimensional coordinates indicating the position of the member in the real space and / or virtual space), information related to the specifications of the member such as the material, color, dimensions, part number, and performance of the member, information indicating the relevance to other adjacent members (for example, information indicating which member the target member is connected to, at which position, and how), information related to the procurement history such as the price of the member and the procurement source, and the like.

[0039] The structure model information may include a three-dimensional shape model of a bridge constructed by combining partial shape models and the attribute information associated with the three-dimensional shape model. The attribute information of the structure model information may include, for example, the dimensions of the bridge in the state where the members are combined, characteristics (for example, design specifications, etc.), the position information of the bridge (for example, the three-dimensional coordinates of a plurality of representative points set on the outer surface of the bridge, etc.). In addition, the structure model information may also include information about attachments installed on the bridge (road signs, lighting, etc.), information about the relevance between adjacent members (for example, which main girder is connected to which cross girder and at which position), information about the surrounding environment of the bridge (for example, information related to the road connected to the bridge, information indicating the positional relationship with surrounding facilities, etc.), information about the virtual space in which the three-dimensional shape model is arranged (for example, reference coordinates specified by user operation, setting of a spatial region defined in a three-dimensional coordinate system, scale of the virtual space, etc.), information related to the construction history, and the like.

[0040] Note that the partial model information and the structure model information shown above are merely examples, and each model information may be associated with information accumulated throughout the entire cycle from design to construction and maintenance management. The virtual space generation unit 101 described later reads the partial model information, the structure model information, etc. stored in the 3D data storage unit 121, and generates a virtual space based on this information.

[0041] The photographed image data storage unit 122 stores the photographed image data acquired by photographing the structure with the cameras / sensors 42 of the moving body 4 and transmitted from the moving body 4 to the management server 1. The photographed image data acquired by photographing with the cameras / sensors 42 includes a photographed image that captures at least a part of the structure and photographed condition information associated with the photographed image.

[0042] The photographed condition information includes, for example, the actual photographing position of the camera 42 in the real space at the time of taking the image, the actual photographing direction which is the direction in the real space where the optical central axis of the camera 42 points at the time of taking the image, etc. The actual photographing position is a position expressed in the 3D coordinate system of the real space. When acquiring the actual photographing position using a GNSS sensor or the like, the actual photographing position may be represented by the 3D coordinates of latitude, longitude, and absolute altitude. When acquiring the actual photographing position in a non-GNSS environment using an inertial sensor (acceleration sensor, gyro sensor), etc., the actual photographing position may be represented by relative 3D coordinates with the movement start position (photographing work start position) of the moving body 4 as the reference position (origin). The actual photographing direction may be specified based on, for example, the orientation of the aircraft at the time of shooting, the orientation of the gimbal, etc. In addition to the above, the photographed condition information may include information indicating the focal length, angle of view, zoom ratio, exposure amount, ISO sensitivity, date and time of shooting, etc. of the camera 42 at the time of shooting. These photographed condition information may be metadata attached to the photographed image and may be stored as Exif information.

[0043] Each piece of photographed image data stored in the photographed image data storage unit 122 may be associated, as additional information, with information indicating a photographed object existing at a photographed object position (capture point) specified by a photographed object specifying unit 103 described later and linked by an image information writing unit 104. The information indicating the photographed object may include, for example, identification information that uniquely identifies a target member that is the main subject, and may also include the name of the target member, section information indicating the section where the target member is arranged, and the like. In addition, the additional information associated with the photographed image data may include, in addition to the information indicating the photographed object that is the main subject, information indicating other members captured in the photographed range.

[0044] The movement information storage unit 123 stores movement information used, for example, in movements for the purpose of photographing various photographed objects (for example, each member included in a bridge) in a structure. The movement information is instruction information for executing the movement, and may include, for example, movement route information (including waypoint information), movement speed, flight altitude, imaging conditions (photographing direction, imaging angle of view, imaging focal length, overlap rate of the photographed image, etc.), and the like. In addition, the movement information storage unit 123 may store movement-acquired information obtained as a result of executing the movement. The movement-acquired information may include the actual movement route, movement speed, flight altitude, photographed image, and photographed condition information associated with the photographed image.

[0045] The movement information as the indication information may be generated, for example, by setting parameters of various information included in the movement information on the management server 1 or the user terminal 2. Note that as the movement route, for example, the position of the moving body storage device may be used as the movement start position and the movement end position, and a movement route passing through each waypoint may be generated. Conversely, without having a moving body storage device, the position where the moving body 4 is carried by the user may be used as the movement start position (so-called home point), or the user may collect the moving body 4 at the movement end position (it may return to the home point). Alternatively, based on the information of the moving body storage device (for example, position information, storage state information, storage machine information, etc.) managed by the management server 1, a configuration may be adopted in which a movement route including the position of the moving body storage device selected as the movement start position or the movement end position is generated.

[0046] Next, each functional unit of the processor 10 will be described in detail.

[0047] The virtual space generation unit 101 generates a virtual space in which a three-dimensional shape model of a structure is arranged based on the structure model information. Fig. 7(b) is a diagram simulating the generated virtual space, in which a bridge model 60, which is a three-dimensional shape model of a bridge, is arranged in the virtual space. Note that the bridge model 60 shown in Fig. 7(b) has sections (J21 - J26) determined according to the connection intervals of the main girders, and each section includes a plurality of partial shape models (for example, partial shape models corresponding to the members 61 - 66 in Fig. 7(a)). When generating the virtual space, each partial model information may also be taken into consideration, and a three-dimensional shape model of the structure may be constructed by combining the partial shape models. In the three-dimensional shape model of the structure arranged in the virtual space, parts corresponding to each partial model information (for example, each member of the bridge) may be displayed in a distinguishable manner, such as being color-coded.

[0048] When generating a virtual space, the virtual space generation unit 101 may execute a process of constructing a correlation relationship between the three-dimensional coordinate system of the real space and the three-dimensional coordinate system of the virtual space by associating the three-dimensional coordinate system of the real space with the three-dimensional coordinate system of the virtual space. As an example, the virtual space generation unit 101 aligns and associates a reference position in the three-dimensional coordinate system of the real space (for example, the movement start position of the moving body 4) with a reference position 55 (reference numeral 55 in FIG. 7(b)) in the three-dimensional coordinate system of the virtual space corresponding to that position, thereby constructing a correlation relationship between the three-dimensional coordinate system of the real space and the three-dimensional coordinate system of the virtual space. Thereby, the virtual space generation unit 101 can convert and represent the position of the moving body 4 in the three-dimensional coordinate system of the real space flying in the real space into the position in the three-dimensional coordinate system of the virtual moving body 57 (virtual object corresponding to the moving body 4 in the real space) in the virtual space. When the scale of the three-dimensional shape model in the virtual space is different from the scale of the corresponding structure in the real space, the scale of the three-dimensional shape model may be adjusted along with the alignment.

[0049] As described above, by constructing a correlation relationship between the three-dimensional coordinate system of the real space and the three-dimensional coordinate system of the virtual space, even when a waypoint is set based on the three-dimensional coordinate system of the virtual space displayed on the user terminal 2, for example, it is possible to specify a position such as generating a movement route based on the three-dimensional coordinate system of the moving body 4 in the real space. When the three-dimensional model data includes dimensional information, it is possible to generate a movement route using a real scale such as going straight ahead for 10 m and then turning right with reference to the three-dimensional coordinate system of the virtual space.

[0050] The virtual space generation unit 101 may execute a process of generating a display screen showing the generated virtual space and transmitting it to the user terminal 2. On the display screen of the user terminal 2, expansion, contraction, rotation, etc. of the three-dimensional shape model (bridge model 60) arranged in the virtual space may be possible according to the user's operation. The virtual space generation unit 101 renders an image of the virtual space according to the user operation and outputs it to the user terminal 2. Further, the virtual space generation unit 101 receives a selection operation by the user such as clicking on a part of the three-dimensional shape model (bridge model 60) of the structure via the display screen of the user terminal 2 on which the virtual space is displayed, and highlights the partial shape model corresponding to the selected part (member) (for example, displays it with a coordinated color tone, enlarges and displays only the selected partial shape model, etc.), or executes a process of displaying the attribute information associated with the partial shape model.

[0051] The image information acquisition unit 102 acquires a captured image obtained by photographing the real space with the camera 42 and photographing condition information associated with the captured image. The captured image and the photographing condition information acquired by the image information acquisition unit 102 are used as input information in the process of specifying the photographing target model by the photographing target specifying unit 103 described later. The image information acquisition unit 102 acquires, as the photographing condition information, for example, the actual photographing position and the actual photographing direction, and may also acquire information such as the focal length and the angle of view when the camera 42 captures the captured image.

[0052] Note that the timing at which the image information acquisition unit 102 acquires the captured image and the photographing condition information is not particularly limited. For example, the image information acquisition unit 102 may receive the captured image and the photographing condition information in real time from the moving body 4 at the timing when the moving body 4 executes photographing. Further, during the execution of the photographing operation by the moving body 4 or at an arbitrary timing after the completion of the photographing operation, a process of capturing the captured image and the photographing condition information may be executed based on an instruction by the user.

[0053] The imaging target specifying unit 103 executes a process of specifying an imaging target model imaged in a captured image based on imaging condition information from among partial shape models included in a three-dimensional shape model of a structure arranged in a virtual space. FIG. 7 is a diagram for explaining the process of specifying an imaging target model executed by the imaging target specifying unit 103, and shows a state of specifying a member 62 of an imaging target imaged in a captured image 50 shown in (a) on the virtual space as shown in (b). The imaging target specifying unit 103 may specify, for example, an imaging target model existing at an imaging target position in the virtual space based on the center position of the angle of view of the camera 42 when the captured image 50 is captured.

[0054] Specifically, the imaging target specifying unit 103 obtains a virtual imaging direction in the virtual space corresponding to the actual imaging direction of the camera 42 based on information indicating the actual imaging position and the actual imaging direction among the imaging condition information acquired by the image information acquisition unit 102. The virtual imaging direction can be represented by a virtual straight line extending from a virtual imaging position in the virtual space corresponding to the actual imaging position of the camera 42 to the virtual imaging direction corresponding to the actual imaging direction of the camera 42. Note that the virtual imaging position can be specified based on the correlation between the three-dimensional coordinate system of the real space and the three-dimensional coordinate system of the virtual space, and the virtual imaging direction corresponds to the center position of the angle of view of the camera 42. The imaging target specifying unit 103 obtains, in the virtual space, a virtual straight line (arrow indicated by a dashed line in FIG. 7(b)) extending from the virtual imaging position corresponding to the actual imaging position (position of the virtual moving body 57 in FIG. 7(b)) to the virtual imaging direction corresponding to the actual imaging direction, and specifies a partial shape model that intersects the virtual straight line as the imaging target model. In FIG. 7(b), among the members (for example, members 61-66) included in the imaging range of the captured image 50, the three-dimensional shape model of the member 62 (member “Name: Lighting Widening Section” shown in gray in FIG. 7(a)) arranged in the section J24 intersects the arrow of the dashed line and is specified as the imaging target model.

[0055] Even for members included in the shooting range of the captured image 50, members that do not intersect a virtual straight line extending in the virtual shooting direction (for example, members 61, 63-66, etc.) may be treated as non-target members in the process executed by the image information writing unit 104 described later. Alternatively, a partial shape model (member 62) that intersects the virtual straight line may be set as the main target model, and partial shape models (such as members 61, 63-66, etc.) that are included in the shooting range but do not intersect the virtual straight line may be specified as sub-target models.

[0056] In the latter case, the shooting target specifying unit 103 may calculate a virtual shooting range corresponding to the actual shooting range in the real space based on, for example, the angle-of-view information and focal length information of the camera 42, which are shooting condition information, and virtual distance information indicating the virtual distance from the virtual shooting position to the shooting target position, so as to specify the above-mentioned sub-target model. Specifically, based on the angle-of-view information, focal length information, and virtual distance information, the "near plane" and "far plane" of the frustum of a cone in the virtual space are specified, and the range from the "near plane" to the "far plane" is calculated as the virtual shooting range. Then, the partial shape models included in the virtual shooting range (excluding members that enter the blind spot of the camera 42 and are not shown in the image) are specified. The shooting target specifying unit 103 specifies a partial shape model (main target model) that intersects the virtual straight line from among the partial shape models included in the virtual shooting range, and also specifies other partial shape models that do not intersect the virtual straight line as sub-target models.

[0057] The image information writing unit 104 executes a process of associating the attribute information associated with the partial shape model specified as the shooting target model with the captured image. The attribute information to be associated with the captured image may be identification information for uniquely specifying the shooting target model. For example, the image information writing unit 104 may associate the name of the member and / or the section information (information indicating the section where the member is arranged in the structure (bridge)) among the attribute information associated with the shooting target model with the captured image.

[0058] Also, the "processing of associating attribute information with the captured image" executed by the image information writing unit 104 may be, for example, a process of writing attribute information such as the name of a member and / or section information into the attached information of the captured image and storing (storing) the captured image data including the attached information in the captured image data storage unit 122. The file format (data format) of the attached information is not particularly limited, and the image information writing unit 104 may execute a process of writing the attribute information of the imaging target model into the Exif information (an example of attached information) of the captured image. In the case of the example shown in FIG. 7, the image information writing unit 104 writes the name "Lighting Widening Section" of the member 62 specified as the imaging target model and the number "J24" of the section where the member 62 is arranged into the attached information of the captured image as information indicating the imaging target (capture point).

[0059] When a sub-target model included in the imaging range is specified by the imaging target specifying unit 103, the image information writing unit 104 may execute a process of writing the attribute information of the sub-target model into the attached information of the captured image. At this time, the attribute information of the sub-target model may be written into the attached information as information indicating other captured members other than the main target (imaging target).

[0060] The display control unit 105 executes a process of reading a predetermined captured image stored in the captured image data storage unit 122 based on a user's request and outputting it to the display screen of the user terminal 2. For example, the display control unit 105 may receive an image output request from the user via the user terminal 2 and output a captured image corresponding to the request. When receiving an image output request, the display control unit 105 may receive specifications of output conditions such as the capture date and time, the target member, and the target section, and identify a captured image corresponding to the specified output conditions by referring to the attached information of each captured image data. When a plurality of captured images satisfy the output conditions specified by the user, the display control unit 105 may output a list (image list) of the plurality of captured images that satisfy the output conditions. When the display control unit 105 causes the user terminal 2 to display a captured image (single image or image list) specified by the user, the captured image and information for identifying the captured object written in the attached information by the image information writing unit 104 (for example, the name of the member that is the captured object, section information indicating the section where the member is arranged, etc.) may be displayed side by side.

[0061] In addition to or instead of the above-described captured image search process, the display control unit 105 may execute a filtering process of narrowing down a captured image that meets the output conditions specified by the user from a list of captured images, a sorting process of sorting the list of captured images based on the sorting conditions specified by the user, and outputting the sorted list, and so on. Also in this case, the display control unit 105 receives specifications of condition items such as the capture date and time, the target member, and the target section desired by the user as filtering conditions (narrowing-down conditions) or sorting conditions.

[0062] The conditions for outputting the photographed images (search conditions, filtering conditions, sorting conditions) may be input in text format or via a predetermined selection form such as a pull-down list. Further, the display control unit 105 may accept designation of a target member by the user via a UI (user interface) that displays the three-dimensional shape model of the structure on the user terminal 2. Specifically, the display control unit 105 may accept designation of any one member or section included in the three-dimensional shape model (for example, clicking and selecting a desired partial shape model or section) on the UI on which the three-dimensional shape model of the structure is displayed. Then, the display control unit 105 refers to the attached information of each photographed image data stored in the photographed image data storage unit 122, and identifies the photographed image in which the designated member or section is written in the attached information. In this case, the display control unit 105 may switch from the screen for displaying the three-dimensional shape model to the screen for displaying the photographed image, and output the identified photographed image (single image or image list), or may display the three-dimensional shape model and the identified photographed image side by side on the display screen of the user terminal 2.

[0063] Note that the functions of the display control unit 105 as described above may be executed by the processor 20 of the user terminal 2. Further, the processor 10 may have other functional units not illustrated in FIG. 5. For example, the processor 10 may have a functional unit that controls the movement of the moving body 4 for the purpose of photographing real images of various photographing objects inside and outside the structure based on various movement information stored in the movement information storage unit 123.

[0064] <An example of an information processing method> Subsequently, with reference to FIG. 8, an information processing method (information processing method related to management of photographed image data) by the information processing system according to the present embodiment will be described. FIG. 8 is a flowchart illustrating the information processing method according to the present embodiment.

[0065] First, the virtual space generation unit 101 in the information processing system reads the structure model information (and partial model information) stored in the three-dimensional data storage unit 121 (step SQ101 in FIG. 8). Then, based on the read structure model information, the virtual space generation unit 101 generates a virtual space in which a three-dimensional shape model of a structure constructed by combining partial shape models is arranged (step SQ102). When generating the virtual space in this step SQ102, the virtual space generation unit 101 may execute a process of constructing a correlation relationship between the three-dimensional coordinate system of the real space and the three-dimensional coordinate system of the virtual space by associating the three-dimensional coordinate system of the real space with the three-dimensional coordinate system of the virtual space.

[0066] Next, the image information acquisition unit 102 acquires a captured image captured by the camera 42 of the moving body 4 and transmitted from the moving body 4 to the management server 1, and captured condition information associated with the captured image (step SQ103). At this time, as the captured condition information, the image information acquisition unit 102 may acquire information indicating, for example, the actual shooting position of the camera 42 in the real space when the captured image is captured, and the actual shooting direction, which is the direction in the real space in which the optical central axis of the camera 42 is directed. In addition, information indicating the focal length of the camera 42 when the captured image is captured, information indicating the angle of view, and the like may also be acquired.

[0067] Next, the shooting target identification unit 103 identifies a shooting target model projected onto the captured image based on the captured condition information from among the partial shape models included in the three-dimensional shape model of the structure arranged in the virtual space (step SQ104). For example, based on the information indicating the actual shooting position and the actual shooting direction, the shooting target identification unit 103 obtains a virtual straight line extending from the virtual shooting position corresponding to the actual shooting position in the virtual space in the virtual shooting direction corresponding to the actual shooting direction in the virtual space, and identifies the partial shape model intersecting the virtual straight line as the shooting target model (for example, (b) in FIG. 7).

[0068] Thereafter, the image information writing unit 104 associates the attribute information associated with the partial shape model specified as the imaging target model by the imaging target specifying unit 103 with the captured image (step SQ105). The attribute information associated with the partial shape model may include, for example, information indicating the name of the member, section information indicating the section where the member is arranged in the structure, and the like. In step SQ105, the image information writing unit 104 may execute a process of writing the name of the member and the section information, which are associated with the imaging target model as attribute information, into the attached information (for example, Exif information) of the captured image.

[0069] As described above, in the information processing system according to the present embodiment, on the three-dimensional shape model of the structure, attribute information such as the name of the member corresponding to the partial shape model included in the three-dimensional shape model is centrally managed, and the attribute information of the imaging target model specified in the virtual space is automatically associated with the captured image. That is, in the information processing system, when utilizing the captured image data in the inspection of the structure or the like, misrecording of the information associated with the captured image can be prevented. In addition, as described above, information for specifying the imaging target in the image can be automatically recorded for the captured image data, and the management of the captured image data can be optimized. Based on these effects, effects such as the improvement of the inspection work efficiency and the guarantee of the accuracy (certainty) of the inspection results can also be expected.

[0070] The above-described embodiments are merely examples for facilitating the understanding of the present disclosure, and are not intended to limit the interpretation of the present disclosure. It goes without saying that the present disclosure can be changed and improved without departing from its gist, and equivalents thereof are included in the present disclosure.

[0071] For example, in the above embodiment, mainly an example of inspecting a bridge using an unmanned moving body such as a drone was given to explain the information processing system related to the management of photographed image data. However, the application examples of the information processing system are not limited to bridge inspections. The inspection target may be other structures such as tunnels and dams. The photographing during inspection execution may be performed using a portable terminal such as a smartphone, tablet terminal, or other camera held by a person, or may be performed using a manned moving body.

[0072] In addition, the moving body 4 used for photographing the structure may further include, in addition to the cameras / sensors 42 as described above, devices, equipment, etc. used for inspecting the presence or absence of predetermined events inside and / or outside the structure. More specifically, imaging devices (visible light cameras, infrared cameras, metal detectors, ultrasonic measuring devices, etc.), keying devices, etc., detection devices (metal detectors), sound collection devices, odor measuring devices, gas detectors, air pollution measuring devices, detection devices (devices for detecting cosmic rays, radiation, electromagnetic waves, etc.), etc., all devices necessary for knowing the state of the inspection target structure can be adopted. The attribute information (such as the name of the member) of the photographed object model specified by the photographing target specifying unit 103 may be associated with the inspection data acquired by the other inspection devices as described above, similar to the association with the photographed image.

[0073] In addition, the purpose of photographing in the information processing system is not limited to inspection, and may be security, monitoring of infrastructure, etc., surveying, disaster response, etc. The information processing system of the present disclosure may be applied to the photographed image data acquired for these purposes.

[0074] Some or all of the functional units such as the virtual space generation unit 101, image information acquisition unit 102, photographing target specifying unit 103, image information writing unit 104, and display control unit 105 described in the above embodiment may be executed by the processor 20 of the user terminal 2 or the flight controller 41 of the moving body 4 instead of the processor 10 of the management server 1.

Explanation of Reference Numerals

[0075] 1 Management Server 2 User Terminal 4 Mobile Body

Claims

1. A three-dimensional data storage unit that stores a plurality of partial model information for showing the three-dimensional structure of each member included in a bridge, and bridge model information for showing the three-dimensional structure of the bridge constructed by integrating the plurality of partial model information; a virtual space generation unit that generates a virtual space in which a three-dimensional shape model of the bridge is arranged based on the bridge model information; an image information acquisition unit that acquires an image acquired by capturing an image of a real space with a camera and capturing condition information associated with the image; a photographing target specifying unit that specifies a photographing target model shown in the image based on the photographing condition information from among partial shape models included in the three-dimensional shape model of the bridge arranged in the virtual space; an image information writing unit that links attribute information associated with the partial shape model identified as the photographing target model to the image, the attribute information associated with each of the partial shape models includes a name of the component, and section information indicating a section in which the component is located among sections divided at predetermined periodic intervals on the bridge; An information processing system, wherein the image information writing unit writes the name of the component associated with the partial shape model identified as the model to be photographed, and the partition information, into auxiliary information of the image.

2. the image information acquisition unit acquires, as the photographing condition information, information indicating an actual photographing position in real space of the camera when the image was photographed, and an actual photographing direction, which is a direction in real space to which an optical center axis of the camera faces; 2. The information processing system of claim 1, wherein the shooting subject identification unit determines, based on information indicating the actual shooting position and the actual shooting direction, a virtual straight line extending in the virtual space from a virtual shooting position corresponding to the actual shooting position to a virtual shooting direction corresponding to the actual shooting direction, and identifies a partial shape model that intersects with the virtual straight line as the shooting subject model.

3. The photographing target identification unit is determining, in the virtual space, a virtual straight line extending from a virtual shooting position corresponding to an actual shooting position to a virtual shooting direction corresponding to an actual shooting direction based on the shooting condition information; The information processing system according to claim 1 or 2, wherein a partial shape model that intersects with the virtual straight line is identified as the photographed object model, and a partial shape model that is included in the photographed range but does not intersect with the virtual straight line is identified as a secondary object model.

4. Storing a plurality of partial model information for showing a three-dimensional structure of each member included in a bridge, and bridge model information for showing a three-dimensional structure of the bridge constructed by integrating the plurality of partial model information; generating a virtual space in which a three-dimensional shape model of the bridge is arranged based on the bridge model information; Acquiring an image acquired by photographing a real space with a camera and photographing condition information associated with the image; Identifying a photographing target model shown in the image based on the photographing condition information from among partial shape models included in the three-dimensional shape model of the bridge arranged in the virtual space; an image information writing unit links attribute information associated with the partial shape model identified as the photographing target model to the image; the attribute information associated with each of the partial shape models includes a name of the component, and section information indicating a section in which the component is located among sections divided at predetermined periodic intervals on the bridge; An information processing method, wherein the image information writing unit writes the name of the component associated with the partial shape model identified as the model to be photographed, and the partition information, into auxiliary information of the image.

5. Storing a plurality of partial model information for showing a three-dimensional structure of each member included in a bridge, and bridge model information for showing a three-dimensional structure of the bridge constructed by integrating the plurality of partial model information; generating a virtual space in which a three-dimensional shape model of the bridge is arranged based on the bridge model information; Acquiring an image acquired by photographing a real space with a camera and photographing condition information associated with the image; Identifying a photographing target model shown in the image based on the photographing condition information from among partial shape models included in the three-dimensional shape model of the bridge arranged in the virtual space; and linking, by an image information writing unit, attribute information associated with the partial shape model identified as the photographing target model to the image; the attribute information associated with each of the partial shape models includes a name of the component, and section information indicating a section in which the component is located among sections divided at predetermined periodic intervals on the bridge; A program for causing the computer to execute the following: the image information writing unit writes, into the auxiliary information of the image, the name of the component associated with the partial shape model identified as the model to be photographed, and the partition information.

Images