Information processing system and information processing method

Abstract

An information processing system and the like are provided that can easily manage images captured by a moving object in association with the location of the image being captured. [Solution] According to one embodiment of the present invention, an information processing system is provided that includes a real shooting position information acquisition unit 110 that acquires real shooting position information indicating a real shooting position in real space linked to the acquired captured image data, a virtual distance generation unit 120 that generates virtual distance information indicating a virtual distance from a virtual shooting position corresponding to the real shooting position to a shooting subject position in virtual space, and a size determination unit 130 that determines the size of image data at the shooting subject position based on angle of view information, focal length information, and virtual distance information when the captured image data was captured.

Description

[Technical field]

[0001] The present invention relates to an information processing system and an information processing method. [Background technology]

[0002] Conventionally, autonomously controllable mobile objects such as flying objects (hereinafter collectively referred to as "flying objects") such as drones and unmanned aerial vehicles (UAVs) and running objects such as unmanned ground vehicles (UGVs) have begun to be used in industry, and in particular, the interior and exterior of buildings are inspected using such mobile objects. Patent Document 1 discloses a system that inspects indoors using a mobile object and manages images of the inspection. [Prior art documents] [Patent documents]

[0003] [Patent Document 1] JP 2019-179422 A Summary of the Invention [Problem to be solved by the invention]

[0004] However, in the technology disclosed in Patent Document 1, it is unclear which position on a building the image captured by the moving object was captured from which direction, and it is also difficult to know the actual size of the object being captured from the image. Therefore, it is difficult to manage the images captured by the moving object in association with the position of the object being captured.

[0005] In addition, it is possible to generate a virtual space that mimics an actual site by pasting a photographed image onto a three-dimensional model, but if the size of the photographed object in the actual site is unknown as described above, it is unclear at what size the photographed image should be pasted onto the three-dimensional model. Therefore, the worker must manually adjust the size of the photographed image and the pasting position on the three-dimensional model using an image editing system or the like, which places a heavy burden on the worker, especially when pasting a large number of images onto a three-dimensional model.

[0006] The present invention has been made in consideration of the above background, and aims to provide an information processing system, etc. that can easily manage images taken by a moving object in association with the position and size of the subject being photographed. [Means for solving the problem]

[0007] The main invention of the present invention for solving the above problem is an information processing system comprising an actual shooting position information acquisition unit that acquires actual shooting position information indicating an actual shooting position in real space linked to the acquired captured image data, a virtual distance generation unit that generates virtual distance information indicating a virtual distance from a virtual shooting position corresponding to the actual shooting position to a shooting target position in a virtual space, and a size determination unit that determines the size of the image data at the shooting target position based on the angle of view information, focal length information, and virtual distance information when the captured image data was captured. Effect of the Invention

[0008] According to the present invention, it is possible to provide an information processing system and the like that can easily manage images captured by a moving object in association with the position and size of the captured image. [Brief description of the drawings]

[0009] [Figure 1] 1 is a diagram showing a configuration of an information processing system according to an embodiment of the present invention; [Diagram 2] 2 is a block diagram showing a hardware configuration of the management server of FIG. 1. [Diagram 3] 2 is a block diagram showing a hardware configuration of the user terminal of FIG. 1. [Figure 4] 2 is a block diagram showing a hardware configuration of the moving body of FIG. 1. [Diagram 5] FIG. 2 is a block diagram showing the functions of the management server of FIG. 1. [Figure 6] 11 is a diagram for explaining an example of a process in which a size specifying unit specifies the size of image data at a shooting target position; FIG. [Figure 7] 1 is a flowchart illustrating an information processing method according to the present embodiment. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0010] The contents of the embodiment of the present invention will be listed and described below. An information processing system according to the embodiment of the present invention has the following configuration. [Item 1] an actual photography position information acquisition unit that acquires actual photography position information indicating an actual photography position in a real space associated with the acquired photographed image data; a virtual distance generating unit that generates virtual distance information indicating a virtual distance from a virtual shooting position corresponding to the actual shooting position to a shooting target position in a virtual space; a size determination unit that determines a size of the image data at the position of the subject to be photographed based on information on an angle of view and focal length when the photographed image data was photographed and on the virtual distance information; An information processing system comprising: [Item 2] An image projection unit that projects the image data of the specified size onto a shooting target model that exists at the shooting target position in the virtual space, Item 1. An information processing system according to item 1. [Item 3] Further comprising a difference detection unit that detects a difference in shape between the object to be photographed shown in the projected image data and the object to be photographed model. Item 3. An information processing system according to item 2. [Item 4] and a shooting target specifying unit that specifies at least one of a shooting target model present at the shooting target position in the virtual space and a shooting target space based on a center position of the angle of view when the captured image data is captured. Item 1. An information processing system according to item 1. [Item 5] An image linking unit that links the image data to at least one of the shooting target model or the shooting target space identified by the shooting target identification unit and stores the image data. 5. An information processing system according to item 4. [Item 6] acquiring real photographing position information indicating a real photographing position in a real space associated with the acquired photographed image data; generating virtual distance information indicating a virtual distance from a virtual shooting position corresponding to the real shooting position to a shooting target position in a virtual space; specifying a size of the image data at the position of the subject to be photographed based on information on an angle of view and focal length when the photographed image data was photographed and on the virtual distance information; 2. An information processing method implemented by a computer, comprising: [Item 7] acquiring real photographing position information indicating a real photographing position in a real space associated with the acquired photographed image data; generating virtual distance information indicating a virtual distance from a virtual shooting position corresponding to the real shooting position to a shooting target position in a virtual space; specifying a size of the image data at the position of the subject to be photographed based on information on an angle of view and focal length when the photographed image data was photographed and on the virtual distance information; A program that causes a computer to execute the above.

[0011] <Details of the embodiment> Hereinafter, an information processing system according to an embodiment of the present invention will be described. In the accompanying drawings, identical or similar elements are given identical or similar reference symbols and names, and duplicated descriptions of identical or similar elements may be omitted in the description of each embodiment. Furthermore, features shown in each embodiment can be applied to other embodiments as long as they are not mutually inconsistent.

[0012] <Configuration> As shown in Fig. 1, the information processing system in this embodiment has a management server 1, one or more user terminals 2, one or more moving bodies 4 (e.g., flying bodies, running bodies, etc.), and one or more moving body storage devices 5. The management server 1, the user terminal 2, the moving bodies 4, and the moving body storage devices 5 are connected to each other so as to be able to communicate with each other via a network. Note that the illustrated configuration is one example, and is not limited thereto, and may be, for example, a configuration in which the moving body storage device 5 is not included and the moving body is carried by the user.

[0013] <Administration Server 1> 2 is a diagram showing a hardware configuration of the management server 1. Note that the illustrated configuration is just an example, and the management server 1 may have other configurations.

[0014] As shown in the figure, a management server 1 is connected to a user terminal 2, a mobile object 4, and a mobile object storage device 5, and constitutes a part of the present system. The management server 1 may be a general-purpose computer such as a workstation or a personal computer, or may be logically realized by cloud computing.

[0015] The management server 1 includes at least a processor 10, a memory 11, a storage 12, a transmission / reception unit 13, an input / output unit 14, etc., which are electrically connected to one another via a bus 15.

[0016] The processor 10 is a computing device that controls the operation of the entire management server 1, controls the transmission and reception of data between each element, and performs information processing required for application execution and authentication processing. For example, the processor 10 is a CPU (Central Processing Unit) and / or GPU (Graphics Processing Unit), and executes programs for this system stored in the storage 12 and deployed in the memory 11 to perform each information processing.

[0017] The memory 11 includes a main memory configured with a volatile storage device such as a DRAM (Dynamic Random Access Memory), and an auxiliary memory configured with a non-volatile storage device such as a flash memory, a HDD (Hard Disc Drive), etc. The memory 11 is used as a work area for the processor 10, and also stores a BIOS (Basic Input / Output System) that is executed when the management server 1 is started, various setting information, and the like.

[0018] The storage 12 stores various programs such as application programs. A database that stores data used in each process may be constructed in the storage 12.

[0019] The transmitting / receiving unit 13 connects the management server 1 to a network. The transmitting / receiving unit 13 may include a short-range communication interface for Bluetooth (registered trademark) and Bluetooth Low Energy (BLE).

[0020] The input / output unit 14 includes information input devices such as a keyboard and a mouse, and output devices such as a display.

[0021] A bus 15 is commonly connected to each of the above elements, and transmits, for example, address signals, data signals, and various control signals.

[0022] <User device 2> 3 also includes a processor 20, a memory 21, a storage 22, a transmission / reception unit 23, an input / output unit 24, etc., which are electrically connected to each other via a bus 25. The functions of each element can be configured in the same way as the above-mentioned management server 1, so detailed explanations of each element will be omitted.

[0023] <Mobile unit 4> The moving body 4 is a known moving body including an air vehicle such as a drone or an unmanned aerial vehicle, and a running body such as an unmanned ground vehicle, and in particular, a moving body that can be autonomously controlled. As a specific example of such a moving body, the moving body 4 will be described below. FIG. 4 is a block diagram showing a hardware configuration of the moving body 4. The flight controller 41 can have one or more processors such as a programmable processor (e.g., a central processing unit (CPU)).

[0024] The flight controller 41 also has and has access to a memory 411. The memory 411 stores logic, code, and / or program instructions that the flight controller can execute to perform one or more steps. The flight controller 41 may also include sensors 412, such as inertial sensors (accelerometers, gyro sensors), GPS sensors, and proximity sensors (e.g., lidar).

[0025] The memory 411 may include, for example, a separable medium such as an SD card or a random access memory (RAM) or an external storage device. Data acquired from the camera / sensors 42 may be directly transmitted to and stored in the memory 411. For example, still image / video data captured by a camera or the like may be recorded in an internal memory or an external memory, but is not limited thereto, and may be recorded in at least one of the management server 1, the user terminal 2, and the mobile object storage device 5 from the camera / sensors 42 or the internal memory via the network NW. The camera 42 is installed on the mobile object 4 via a gimbal 43.

[0026] The flight controller 41 includes a control module (not shown) configured to control the state of the moving object. For example, the control module has six degrees of freedom (translational motion x, y, and z, and rotational motion θ x , θ y and θ z In order to adjust the spatial arrangement, speed, and / or acceleration of the moving body having a rotor 44, the control module controls the propulsion mechanism (motor 45, etc.) of the moving body via an ESC 44 (Electric Speed ​​Controller). The propeller 46 rotates by the motor 45 powered by a battery 48, generating lift for the moving body. The control module can control one or more of the states of the mounted parts and sensors.

[0027] The flight controller 41 can communicate with a transceiver 47 configured to transmit and / or receive data from one or more external devices (e.g., a transceiver 49, a terminal, a display device, or other remote control). The transceiver 49 can use any suitable communication means, such as wired or wireless communication.

[0028] For example, the transceiver 47 may utilize one or more of a local area network (LAN), a wide area network (WAN), infrared, radio, WiFi, a point-to-point (P2P) network, a telecommunications network, cloud communications, and the like.

[0029] The transceiver unit 47 can transmit and / or receive one or more of the following: data acquired by the sensors 42, processing results generated by the flight controller 41, specified control data, user commands from a terminal or a remote controller, etc.

[0030] The sensors 42 according to the present embodiment may include an inertial sensor (acceleration sensor, gyro sensor), a GPS sensor, a proximity sensor (for example, LiDAR (Light Detection And Ranging)), or a vision / image sensor (for example, a camera). Hereinafter, the sensors 42 may be referred to as "camera 42". The sensors 42 according to the present embodiment are configured to acquire, when an image is captured by the camera 42 as a vision / image sensor, image capturing condition information related to the actual image capturing position in real space of the camera 42 at the time the image is captured, the actual image capturing direction which is the direction in real space to which the optical central axis of the camera 42 faces, the focal length and angle of view of the camera 42, and the like, in association with the image. Therefore, the image data of an image captured by the camera 42 according to the present embodiment includes not only the image data but also the image capturing condition information as described above in association with the image.

[0031] <Management server functions> 5 is a block diagram illustrating functions implemented in the management server 1. In the embodiment of the present invention, the management server 1 has various functional units for executing a process of arranging a captured image in a corresponding three-dimensional model based on an image captured by a camera included in the sensors 42 of the mobile object 4 and image capture condition information associated therewith, and three-dimensional model data indicating the arrangement of components (e.g., walls, columns, stairs, equipment, etc.) within a structure (e.g., a building such as a building).

[0032] In this embodiment, the management server 1 includes a processor 10, a transmission / reception unit 13, and a storage unit 200. The processor 10 includes an actual shooting position information acquisition unit 110, a virtual distance generation unit 120, a size identification unit 130, an image projection unit 140, a difference detection unit 150, a shooting target identification unit 160, an image linking unit 170, and a movement execution unit 180. The storage unit 200 includes various databases, such as a three-dimensional data storage unit 210, a photographed image data storage unit 220, and a movement information storage unit 230.

[0033] First, various databases of the storage unit 200 will be described. The three-dimensional data storage unit 210 stores three-dimensional model data indicating the configurations inside and outside the structure and their arrangement positions. The three-dimensional model data may be any data as long as it is three-dimensional model data (more preferably three-dimensional model data having dimensional information) indicating the arrangement of the configurations inside the structure, which is created based on data created by CAD (Computer-Aided Design) design software. For example, it may be three-dimensional model data reconstructed from BIM (Building Information Modeling) data, CIM (Construction Information Modeling) data, CAD data, BIM data, etc., or it may be three-dimensional model data obtained by generating a configuration having a predetermined height based on two-dimensional design drawing data. The generation of the three-dimensional model data, such as reconstruction, may be executed by the processor 10 of the management server 1, or the management server 1 may acquire three-dimensional model data generated by executing the process in an external device different from the management server 1. In addition, the three-dimensional model data indicating the configuration may be linked to information indicating the type, size, scale, etc. of the configuration. In addition, the three-dimensional data storage unit 210 may store information regarding the three-dimensional model data, such as information indicating a predetermined spatial region based on the three-dimensional model data (for example, a spatial region based on a component such as the spatial region between pillar A and pillar B, or a spatial region defined by a three-dimensional coordinate system in which the three-dimensional model data is placed (particularly, a spatial region based on coordinates specified by user operation)).

[0034] The processor 10 of the management server 1 generates a virtual space represented by the three-dimensional model data in a three-dimensional coordinate system based on the three-dimensional model data stored in the three-dimensional data storage unit 210. The placement information such as the placement position and placement orientation of each three-dimensional model in the virtual space thus generated is known.

[0035] The photographed image data storage unit 220 stores photographed image data photographed by the sensors 42 of the moving object 4 and transmitted from the moving object 4 to the management server 1. As described above, the image data photographed and acquired by the sensors 42 of the moving object 4 includes not only the data of the image, but also photographing condition information related to the actual photographing position of the camera 42 in real space at the time the image was photographed, the actual photographing direction which is the direction in real space to which the optical center axis of the camera 42 faces, and the focal length of the camera 42, etc., in association (linked) with the image. Therefore, the photographed image data stored in the photographed image data storage unit 220 includes image data and photographing condition information linked thereto. In the photographed image data, the photographing condition information may be metadata associated with the image data.

[0036] The photographed image data stored in the photographed image data storage unit 220 may further include information on at least one of a photographed object model present at a photographed object position in the virtual space, which is specified by a photographed object specifying unit 160 of the processor 10 described later and linked to the image linking unit 170, or the photographed object space. Details of the above-mentioned specification process and linking process by the photographed object specifying unit 160 and the image linking unit 170 will be described later. Furthermore, the photographed image data stored in the photographed image data storage unit 220 may be associated with information on the size of the image data at the photographed object position, which is the position of the photographed object in the virtual space, which is specified by a size specifying unit 130 described later.

[0037] The movement information storage unit 230 stores movement information used in movement for the purpose of taking real images of various objects inside and outside buildings, etc. The movement information includes, for example, movement route information (including waypoint information), movement speed, flight altitude, imaging conditions (imaging direction, imaging angle of view, imaging focal length, overlap rate of captured images, etc.), information acquired during movement (for example, image data and imaging condition information associated therewith, etc.), etc.

[0038] The movement information can be generated, for example, by setting parameters of various information included in the movement information on the management server 1 or the user terminal 2. The movement route may be generated by, for example, setting the position of the moving body storage device 5 as the movement start position and the movement end position, and passing through each waypoint. Conversely, without the moving body storage device 5, the position where the user carries the moving body 4 is set as the movement start position (so-called home point), and the user may collect the moving body 4 at the movement end position (which may be back to the home point). Alternatively, the movement route may be generated as a movement route including the position of the moving body storage device 5 selected as the movement start position or the movement end position based on information of the moving body storage device 5 managed by the management server 1 (for example, position information, storage state information, storage machine information, etc.).

[0039] Next, each functional unit of the processor 10 will be described. The actual shooting position information acquisition unit 110 acquires, as shooting condition information linked to captured image data captured by the camera 42 of the moving object 4 and transmitted from the moving object 4 to the management server 1, in particular, actual shooting position information indicating the actual shooting position of the camera 42 in real space at the time the image was captured. The actual shooting position information acquisition unit 110 may further acquire information regarding the actual shooting direction, which is the direction in real space to which the optical center axis of the camera 42 faces, and the focal length and angle of view of the camera 42, from the above-mentioned shooting condition information linked to the acquired image data.

[0040] The actual photographing position in the real space of the camera (sensors) 42 is a position expressed in a three-dimensional coordinate system in the real space. When the camera 42 of the moving body 4 includes a GNSS sensor (GPS sensor, etc.) and the GNSS sensor can acquire latitude and longitude information as position information of the sensors 42, the actual photographing position in the real space of the camera (sensors) 42 may be expressed in latitude and longitude. Alternatively, when the position information of the sensors 42 is acquired by a non-GNSS sensor such as an inertial sensor (acceleration sensor, gyro sensor), the actual photographing position in the real space of the camera (sensors) 42 may be expressed as a position in a three-dimensional coordinate system with the origin being a reference position in the real space (for example, the flight start position (home point) of the moving body 4).

[0041] When generating a virtual space represented by the three-dimensional model data in a three-dimensional coordinate system based on the three-dimensional model data stored in the three-dimensional data storage unit 210, the processor 10 of the management server 1 performs a process of constructing a correlation between the three-dimensional coordinate system of the real space and the three-dimensional coordinate system of the virtual space, for example, by associating the three-dimensional coordinate system of the real space with the three-dimensional coordinate system of the virtual space. As an example, the processor 10 aligns and associates a reference position in the three-dimensional coordinate system of the real space (for example, the flight start position of the moving body 4) with a reference position in the three-dimensional coordinate system of the virtual space corresponding to that position, thereby constructing a correlation between the three-dimensional coordinate system of the real space and the three-dimensional coordinate system of the virtual space. As a result, the processor 10 can convert and express the position in the three-dimensional coordinate system of the real space of the moving body 4 flying in the real space into the position in the three-dimensional coordinate system of the virtual moving body (a virtual object corresponding to the moving body 4 in the real space) in the virtual space. When the scale of the three-dimensional model data in the virtual space is different from the scale of the corresponding structure (composition) in the real space, the scale of the three-dimensional model data may be adjusted in conjunction with the alignment.

[0042] In this way, by constructing a correlation between the three-dimensional coordinate system of the real space and the three-dimensional coordinate system of the virtual space, it becomes possible to specify positions such as generating a movement path based on the three-dimensional coordinate system of the moving object 4 in the real space, even when a waypoint is set based on the three-dimensional coordinate system of the virtual space displayed on the user terminal 2. When the three-dimensional model data has dimensional information, it becomes possible to generate a movement path using a real scale, such as flying straight for 10 m and then turning right, based on the three-dimensional coordinate system of the virtual space.

[0043] The virtual distance generation unit 120 performs processing to generate virtual distance information indicating a virtual distance from a virtual shooting position corresponding to a real shooting position in the real space of the camera 42 to a shooting target position which is a position of a shooting target in the virtual space in the virtual space generated by the processor 10. The shooting target position in the virtual space may be, for example, coordinate information in the virtual space, a position of a three-dimensional model of the shooting target in the virtual space (virtual components such as walls, pillars, stairs, and equipment in a building in the virtual space), or an arbitrary spatial region position in the virtual space.

[0044] The virtual distance generating unit 120 performs a process of calculating a virtual distance, which is the distance from the virtual shooting position to the shooting target position, based on a virtual shooting position in the virtual space in the three-dimensional coordinate system of the virtual space and a shooting target position (the position of a three-dimensional model corresponding to a component of the shooting target), and generating virtual distance information. As an example, the virtual distance, which is the distance from the virtual shooting position to the shooting target position, can be obtained by determining the length of a line segment connecting the virtual shooting position and the shooting target position in the three-dimensional coordinate system of the virtual space along the shooting direction (particularly the optical axis direction).

[0045] The size determination unit 130 performs a process of determining the size of the image data at the shooting target position, which is the position of the shooting target within the virtual space, based on the shooting condition information, i.e., the angle of view information and focal length information, of the camera 42, contained in the captured image data acquired when the camera 42 captured an image, and the virtual distance information generated by the virtual distance generation unit 120.

[0046] Now, with reference to FIG. 6, an example of a process in which the size specifying section 130 specifies the size of the image data at the shooting target position will be described.

[0047] First, using the angle of view information and focal length information of the camera 42 linked to the captured image data, the size determination unit 130 calculates the size of a first plane (the "near plane" of the viewing frustum shown in FIG. 6) that is a focal length n away from the virtual capturing position C in the virtual space that corresponds to the actual capturing position of the camera 42. The size of the first plane corresponds to the size of the captured image data of the subject captured by the camera 42.

[0048] Next, the size determination unit 130 calculates the size of the image data in a second plane (the "far plane" of the viewing frustum shown in FIG. 6) that is a virtual distance f away from the virtual shooting position C, using the size of the first plane calculated as described above, the focal length information, and the virtual distance information generated by the virtual distance generation unit 120. The size of the second plane is a size obtained by enlarging the size of the first plane in accordance with the ratio of the focal length n to the virtual distance f, and is the size of the image data at the shooting target position. When the three-dimensional model data includes dimension information and scale information, the size determination unit 130 calculates the size of the image data at the shooting target position as described above, and it becomes possible to obtain the actual size of the component in real space corresponding to the three-dimensional model at the shooting target position from the size of the image data.

[0049] The image projection unit 140 performs a process of projecting image data of the size specified by the size specification unit 130 onto the imaging target model, which is a three-dimensional model of the imaging target present at the imaging target position in the virtual space. Through this process, an image of a real-space component corresponding to the imaging target model, which is a three-dimensional model of the imaging target present at the imaging target position, is pasted at a corresponding position on the imaging target model, which is a three-dimensional model of the imaging target present at the imaging target position, at a size matching the size of the imaging target model, and the texture of the real-space component (imaging target) is imparted to the surface of the imaging target model.

[0050] The difference detection section 150 performs a process of detecting a difference in shape between a subject that appears in image data projected in a virtual space by the image projection section 140 and a subject model that corresponds to the subject.

[0051] As an example of the difference detection process, an example will be described in which the object to be photographed is a cupboard having a substantially rectangular parallelepiped shape. The difference detection unit 150 extracts characteristic parts of the object to be photographed that appear in the projected image data (for example, the outline of the cupboard that is the object to be photographed and the corners where the outlines intersect with each other) and characteristic parts of the object to be photographed that appear in the model to be photographed onto which the image data is projected that correspond to the characteristic parts of the object to be photographed (for example, the ridges of the object to be photographed that is a three-dimensional model of the cupboard and the corners where the ridges intersect with each other), and detects the difference in shape or arrangement between the object to be photographed that appears in the projected image data and the object to be photographed model based on the positional relationship between these characteristic parts. The difference information acquired as a result of the detection process by the difference detection unit 150 can be used, for example, to correct the position, shape, size, etc. of the image data that the image projection unit 140 projects in the virtual space, and to arrange the image data so that it is appropriately superimposed on the object to be photographed model.

[0052] The shooting target identification unit 160 performs a process of identifying at least one of the shooting target model present at the shooting target position in the virtual space or the shooting target space based on the center position of the angle of view of the camera 42 when the camera 42 captured the captured image data.

[0053] More specifically, the imaging target specifying unit 160 first obtains a virtual imaging direction in the virtual space corresponding to the real imaging direction of the camera 42 based on information on the real imaging direction, which is the direction in the real space to which the optical central axis of the camera 42 faces at the time when the imaging data is captured, among the imaging condition information acquired when the camera 42 captures the imaging data and linked to the imaging data. The virtual imaging direction in the virtual space can be expressed by a virtual straight line extending from a virtual imaging position in the virtual space corresponding to the real imaging position of the camera 42 to a virtual imaging direction corresponding to the real imaging direction of the camera 42. The center position of the angle of view of the camera 42 can correspond to the virtual imaging position in the virtual space. The imaging target specifying unit 160 then specifies a imaging target model or imaging target space present at a imaging target position in the virtual space, where a virtual straight line extending from a virtual imaging position in the virtual space corresponding to the real imaging position of the camera 42 in the virtual space in the virtual space intersects.

[0054] Through this processing, the photographing subject identification unit 160 identifies at least one of the photographing subject model or the photographing subject space present at the photographing subject position in the virtual space based on the center position of the angle of view of the camera 42 when the camera 42 captured the photographed image data.

[0055] The image linking unit 170 links the image data to at least one of the shooting target model or shooting target space identified by the shooting target identification unit 160, and stores the image data in the storage unit 200. More specifically, based on the identification information acquired as a result of the identification process by the shooting target identification unit 160, the image linking unit 170 associates the shooting target model or shooting target space in the virtual space corresponding to the shooting target object in the real space reflected in the shooting image data with the shooting image data captured by the camera 42, and stores the shooting image data linked to the shooting target model or shooting target space in the corresponding virtual space in the shooting image data storage unit 220 of the storage unit 200.

[0056] The movement execution unit 180 executes the movement of the moving object 4 for the purpose of capturing actual images of various shooting objects inside and outside a building, based on various pieces of movement information stored in the movement information storage unit 230.

[0057] <An example of the information processing method according to this embodiment> Next, an information processing method according to this embodiment (particularly, a method for specifying the image data size on a three-dimensional model) will be described with reference to Fig. 7. Fig. 7 is a flowchart illustrating the information processing method according to this embodiment.

[0058] First, the actual shooting position information acquisition unit 110 in the information processing system of this embodiment acquires shooting condition information linked to the captured image data captured by the camera 42 of the moving body 4 and transmitted from the moving body 4 to the management server 1 (step S101).

[0059] The photographing condition information includes information about the actual photographing position of the camera 42 in real space at the time when the image was photographed, which is acquired when the camera (sensors) 42 serving as a vision / image sensor photographs an image, the actual photographing direction which is the direction in real space to which the optical central axis of the camera 42 faces, the focal length and angle of view of the camera 42, and the like, and is associated with photographed image data which is data of the photographed image. In the process of step S101, the actual photographing position information acquisition unit 110 particularly acquires information about the actual photographing position of the camera 42 in real space at the time when the image was photographed from the photographed image data.

[0060] Next, the virtual distance generation unit 120 in the information processing system generates virtual distance information indicating a virtual distance from a virtual shooting position corresponding to the real shooting position of the camera 42 in the real space to a shooting target position which is the position of the shooting target in the virtual space (step S102) in the virtual space generated by the processor 10. As an example, the virtual distance generation unit 120 can obtain the virtual distance to the shooting target position which is the position of the shooting target in the virtual space by calculating the length of a line segment connecting the virtual shooting position in the virtual space and the shooting target position (the position of a three-dimensional model corresponding to a component of the shooting target) in the three-dimensional coordinate system of the virtual space.

[0061] Next, the size determination unit 130 in the information processing system determines the size of the image data at the position of the subject to be photographed in the virtual space based on the angle of view information and focal length information of the camera 42, which are part of the shooting condition information contained in the captured image data acquired when the camera 42 captured an image, and the virtual distance information generated by the virtual distance generation unit 120 (step S103).

[0062] As explained with reference to FIG. 6 as an example, the size determination unit 130 calculates the size of a first plane (the "near plane" of the viewing frustum shown in FIG. 6) that is a focal distance away from the virtual shooting position in the virtual space corresponding to the actual shooting position of the camera 42, and then uses the size of the first plane, the focal length, and the virtual distance to calculate the size of a second plane that is a virtual distance away from the virtual shooting position, thereby determining the size of the image data at the shooting target position in the virtual space.

[0063] In this way, by specifying the size of the image data at the position of the subject in the virtual space, when the photographed image data is projected at the position of the subject in a later process (the position of the three-dimensional model corresponding to the constituent of the subject), it is possible to make the image data a size suitable for the position of the subject. Furthermore, when the three-dimensional model data includes dimensional information and scale information, it is possible to obtain the actual size of the constituent of the real space corresponding to the three-dimensional model at the position of the subject from the size of the image data.

[0064] Next, the image projection unit 140 in the information processing system projects the image data of the size specified by the size specification unit 130 onto the shooting target model, which is a three-dimensional model of the shooting target present at the shooting target position in the virtual space (step S104). As a result, an image of the real-space component corresponding to the shooting target model is pasted on the shooting target model of the shooting target present at the shooting target position at a size matching the size of the shooting target model, and the texture of the real-space component (shooting target) is added to the surface of the shooting target model.

[0065] Next, difference detection section 150 in the information processing system detects a difference in shape between the subject shown in the image data projected in the virtual space by image projection section 140 and the subject model corresponding to the subject (step S105). As an example, the difference detection unit 150 extracts characteristic parts of the object to be photographed reflected in the projected image data and corresponding characteristic parts of the object to be photographed model onto which the image data is projected, and detects the difference in shape or arrangement between the object to be photographed reflected in the projected image data and the object to be photographed model based on the positional relationship between those characteristic parts. By using the difference information thus acquired, for example, it becomes possible to correct the position, shape, size, etc. of the image data projected by the image projection unit 140 in the virtual space, and arrange the image data so that it is appropriately superimposed on the object to be photographed model.

[0066] Next, the photographing subject identification unit 160 in the information processing system identifies at least one of the photographing subject model existing at the photographing subject position in the virtual space or the photographing subject space based on the center position of the angle of view of the camera 42 when the camera 42 captured the photographed image data (step S106).

[0067] In the processing of step S106, the shooting subject identification unit 160 first determines a virtual shooting direction in the virtual space corresponding to the actual shooting direction of the camera 42 based on information regarding the actual shooting direction, which is the direction in real space in which the optical center axis of the camera 42 faces at the time the captured image data was captured, and then identifies the shooting subject model or shooting subject space present at the shooting subject position in the virtual space where a virtual straight line extending from the virtual shooting position in the virtual space corresponding to the actual shooting position of the camera 42 in the virtual space intersects.

[0068] Finally, the image linking unit 170 in the information processing system links the image data to at least one of the shooting target model or shooting target space identified by the shooting target identification unit 160 and stores the image data in the storage unit 200 (step S107). This makes it possible to search for the shooting target model or shooting target space corresponding to the image data stored in the shooting image data storage unit 220, or search for image data corresponding to the shooting target model or shooting target space stored in the three-dimensional data storage unit 210, and enables the image projection unit 140 to automatically identify the image data to be projected at the shooting target position in step S104.

[0069] Thus, according to this embodiment, an information processing system or the like is provided that can easily manage images captured by the moving body 4 in association with the position and size of the subject to be photographed. In particular, the information processing method according to this embodiment includes identifying the size of image data at the position of the subject to be photographed in the virtual space, and thus when the photographed image data is projected onto the position of the subject to be photographed (the position of the three-dimensional model corresponding to the constituent of the subject to be photographed), it is possible to make the size of the image data suitable for the position of the subject to be photographed, and further, when the three-dimensional model data includes dimensional information and scale information, it is possible to obtain the actual size of the constituent of the real space corresponding to the three-dimensional model at the position of the subject to be photographed from the size of the image data.

[0070] The mobile object 4 may further include devices, equipment, etc. that are used to inspect the presence or absence of a predetermined event on the inner wall and / or outer wall of the structure. More specifically, any device necessary to know the state of the inspected structure, such as an imaging device (visible light camera, infrared camera, metal detector, ultrasonic measuring device, etc.), a keying device, etc., a detecting device (metal detector), a sound collecting device, an odor measuring device, a gas detector, an air pollution measuring device, a detecting device (a device for detecting cosmic rays, radiation, electromagnetic waves, etc.), etc., may be employed.

[0071] Furthermore, the information processing method according to the present embodiment may be executed, for example, during security or surveillance within a structure, and may further include devices, equipment, etc. used for security or surveillance. More specifically, any device necessary for imaging or detecting abnormalities or intruders in a structure to be guarded or monitored, such as an imaging device (visible light camera, infrared camera, night vision camera, metal detector, ultrasonic measuring device, etc.) or a sensor device (motion sensor, infrared sensor, etc.), may be employed.

[0072] Furthermore, the mobile object 4 can be suitably used as a mobile object for photographing by mounting a camera or the like, and can also be used in various industries such as the security field, infrastructure monitoring, surveying, inspection inside buildings and structures such as sports venues, factories, and warehouses, and disaster response.

[0073] The above-described embodiment is merely an example for facilitating understanding of the present invention, and is not intended to limit the present invention. The present invention can be modified or improved without departing from the spirit of the present invention, and it goes without saying that the present invention includes equivalents thereof. [Explanation of symbols]

[0074] 1 Management Server 2. User terminal 4. Mobile 5 Mobile storage device

Claims

1. an actual photography position information acquisition unit that acquires actual photography position information indicating an actual photography position in a real space associated with the acquired photographed image data; a virtual distance generating unit that calculates a virtual distance from a virtual shooting position in a virtual space corresponding to the actual shooting position to a shooting target position in the virtual space, and generates virtual distance information; a size determination unit that determines a plane size of the captured image data on a plane in the virtual space that is separated from the virtual shooting position by the virtual distance based on information on an angle of view and focal length when the captured image data was captured and on the virtual distance information; An information processing system comprising:

2. An image projection unit projects the captured image data having the specified plane size onto a shooting target model present at the shooting target position in the virtual space. The information processing system according to claim 1 .

3. A difference detection unit detects a difference in shape between the photographed object reflected in the projected photographed image data and the photographed object model. The information processing system according to claim 2 .

4. and a shooting target specifying unit that specifies at least one of a shooting target model present at the shooting target position in the virtual space and a shooting target space based on a center position of the angle of view when the captured image data is captured. The information processing system according to claim 1 .

5. An image linking unit that links the photographed image data to at least one of the photographing target model or the photographing target space identified by the photographing target identification unit and stores the photographed image data.

5. The information processing system according to claim 4.

6. acquiring real photographing position information indicating a real photographing position in a real space associated with the acquired photographed image data; calculating a virtual distance from a virtual shooting position in a virtual space corresponding to the real shooting position to a shooting target position in the virtual space, and generating virtual distance information; specifying a plane size of the captured image data on a plane in the virtual space that is separated from the virtual shooting position by the virtual distance based on angle of view information and focal length information when the captured image data was captured and the virtual distance information; 2. An information processing method implemented by a computer, comprising:

7. acquiring real photographing position information indicating a real photographing position in a real space associated with the acquired photographed image data; calculating a virtual distance from a virtual shooting position in a virtual space corresponding to the real shooting position to a shooting target position in the virtual space, and generating virtual distance information; specifying a plane size of the captured image data on a plane in the virtual space that is separated from the virtual shooting position by the virtual distance based on angle of view information and focal length information when the captured image data was captured and the virtual distance information; A program that causes a computer to execute the above.

Images