Spatial visualization system and spatial visualization method

Abstract

To acquire a desired image by making troublesome operation unnecessary.SOLUTION: A spatial visualization system includes an arithmetic device for executing prescribed arithmetic processing, and a storage device accessible by the arithmetic device. The arithmetic device has a space structure recognition part which constructs a space structure from a plurality of images, the arithmetic device has an image semantic recognition part which detects an object included in each of the plurality of images, the arithmetic device has an image semantic and space structure fusion part which estimates a space position on the space structure of the detected object, and the storage device stores information of an image that has been a basis of forming the space structure, the formed space structure, and information of the detected object.SELECTED DRAWING: Figure 1

Description

Technical Field

[0001] The present invention relates to a spatial visualization system.

Background Art

[0002] Disasters due to global warming occur frequently, and the response delay due to a shortage of on-site personnel and the threat to life and property due to the spread of damage are increasing. At the time of a disaster, it is necessary to quickly grasp the wide-area damage situation and take measures to suppress the damage in a short time, such as guiding people to evacuation routes. Also, even in normal times, a wide-area inspection operation is required to detect abnormalities in infrastructure. In order to quickly grasp the situation in a wide area and detect abnormalities, automatic video analysis using artificial intelligence (AI) technology for videos taken not only by fixed surveillance cameras but also by mobile wearable cameras and unmanned aerial vehicles (UAVs) has attracted attention. In addition, methods for reconstructing a three-dimensional space structure from a plurality of images have been proposed by photogrammetry technology and the like, and it has become possible to grasp the overall situation from videos taken by mobile cameras.

[0003] As the background art in this technical field, there is the following prior art. In Patent Document 1 (Japanese Patent Application Laid-Open No. 2019-211257), an inspection system for inspecting an inspection object includes a three-dimensional model generation unit that generates a three-dimensional model of the inspection object based on a plurality of images taken by a flying device equipped with a camera, a shooting information acquisition unit that acquires the shooting position and the viewing axis direction of the camera for each of the plurality of images in a three-dimensional coordinate system, an abnormality detection unit that detects an abnormality of the inspection object based on the image for each of the plurality of images, an abnormality position specifying unit that specifies an abnormality position, which is a position in the three-dimensional coordinate system according to the shooting position and the viewing axis direction, for the detected abnormality, and a three-dimensional model display unit that displays a three-dimensional model in which the abnormality position is mapped. Thereby, an abnormal portion detected on the image can be easily specified on the three-dimensional model and provided to the user quickly and accurately.

Prior Art Documents

Patent Documents

[0004] [Patent Document 1] Japanese Patent Publication No. 2019-211257 [Overview of the project] [Problems that the invention aims to solve]

[0005] Patent Document 1 envisions application to an inspection system, where the distance between the imaging device and the object being inspected is relatively short and does not change significantly. Furthermore, it is sufficient if the image of an abnormal area specified by the user on a three-dimensional model can be pinpointed. On the other hand, when photographing numerous objects and events of various sizes scattered over a wide area, such as in a disaster situation, from various distances and angles, it is difficult to grasp the overall situation from the coordinates of objects indicated by pinpointing, as in Patent Document 1. In addition, when there are many detection points, acquiring images based solely on the user's coordinate specification on a three-dimensional model requires cumbersome operations from the user and makes it difficult to obtain the desired image. [Means for solving the problem]

[0006] A representative example of the invention disclosed in this application is as follows: a spatial visualization system comprising a computing device that performs predetermined calculation processing and a storage device accessible by the computing device, wherein the computing device has a spatial structure recognition unit that constructs a spatial structure from a plurality of images, an image semantic recognition unit that detects objects contained in each of the plurality of images, and an image semantic-spatial structure fusion unit that estimates the spatial position of the detected objects on the spatial structure, and the storage device stores information of the images that formed the spatial structure, the constructed spatial structure, and information of the detected objects. [Effects of the Invention]

[0007] According to one aspect of the present invention, a wide variety of objects and events of different sizes scattered over a wide area, such as in a disaster situation, can be appropriately visualized in terms of spatial structure, enabling a rapid understanding of the situation. Problems, configurations, and effects other than those mentioned above will be clarified by the following description of the embodiments. [Brief explanation of the drawing]

[0008] [Figure 1] This is a block diagram showing an example configuration of the spatial visualization system in Example 1. [Figure 2] This block diagram shows an example of the hardware configuration of the spatial visualization system in Example 1. [Figure 3A] This is an explanatory diagram showing an example configuration of the image database in Example 1. [Figure 3B] This is an explanatory diagram showing an example configuration of the image database in Example 1. [Figure 3C] This is an explanatory diagram showing an example configuration of the image database in Example 1. [Figure 4] This diagram illustrates the overview of the spatial structure configuration process performed by the spatial structure recognition unit in Example 1. [Figure 5] This diagram illustrates the overview of the image recognition process performed by the image semantic recognition unit in Example 1. [Figure 6] This diagram illustrates the overview of the processing performed by the image semantic and spatial structure fusion unit in Example 1. [Figure 7] This is a flowchart of the database registration process in Example 1. [Figure 8] This figure illustrates the spatial visualization and image search processing performed by the image search device of Example 1. [Figure 9] This is a flowchart of the spatial visualization and image search process using the image search device of Example 1. [Figure 10] This diagram illustrates the overview of the object information summarization process in Example 1. [Figure 11] This is a flowchart of the summarization process performed by the image search device in Example 2. [Figure 12] This figure illustrates image retrieval utilizing the context of the 3D viewer in Example 3. [Figure 13] It is a flowchart of the context utilization image search process of the image search device of Example 3. [Figure 14] It is a diagram showing a configuration example of an operation screen for visualizing spatial information and performing image search using the image search devices of Examples 1 to 3. [Figure 15] It is a sequence diagram explaining an example of the process of the spatial visualization system of Example 1.

Embodiments for Carrying Out the Invention

[0009] Hereinafter, embodiments of the present invention will be described with reference to the accompanying drawings. This embodiment is merely an example for realizing the present invention and does not limit the technical scope of the present invention. The same reference numerals are assigned to common configurations in each figure.

Examples

[0010] The image search device 104 of this embodiment analyzes the images acquired by a mobile imaging device to construct a spatial structure. Further, it detects the semantic information of the objects (objects and events) included in the images, estimates the positions and sizes of the detected objects on the spatial structure, and constructs an image database 110 that holds both spatial information and semantic information. Since the user can overview the detected objects on the spatial structure, the user can quickly grasp the events that occurred over a wide area without checking each captured image. In this example, unless otherwise particularly limited, "space" is used in the sense of a three-dimensional space.

[0011] Note that the objects to be detected in the image semantic recognition process may be objects such as people and cars whose clear boundaries with the background can be recognized, or events such as landslides, fires, and smoke that are not regular. However, since it is necessary to obtain the positions on the spatial structure, events that occur in areas where there is no structural information (for example, in the air) cannot be accurately handled. In that case, it can be used for use cases that visualize approximate positions.

[0012] FIG. 1 is a block diagram showing a configuration example of the spatial visualization system 100 of Example 1.

[0013] The following use cases can be considered for the spatial visualization system 100, but are not limited thereto. (1) Disaster situation awareness: By grasping the locations of landslides, floods, fires, people, vehicles, buildings, etc. that have occurred over a wide area, it can be utilized for rescue operations and reconstruction plans. (2) Infrastructure maintenance: By regularly inspecting the deterioration and damage of buildings, bridges, etc., collapse can be prevented in advance. (3) Inventory management: By quantifying the quantity of materials and assets stored outdoors or in large-scale warehouses, the supply chain can be optimized. Also, by detecting abnormalities early, inventory losses can be prevented. (4) Wide-area security: Detect the flow of people and vehicles, accidents, and incidents over a wide area that cannot be covered by fixed surveillance cameras, and conduct an overall surveillance.

[0014] Also, in this embodiment, a UAV (Unmanned Aerial Vehicle) is assumed as the imaging device. However, if the self-position and orientation of the imaging device can be acquired or estimated, the spatial visualization system 100 can be applied to the data acquired by any imaging device such as a wearable camera. The "self-position" is the three-dimensional coordinates of the imaging device in the real space, and in the case of a UAV, it can be acquired by a satellite positioning system (GNSS: Global Navigation Satellite System) or an altitude sensor. The "orientation" is the rotation information of the imaging device, and in the case of a UAV, it can be acquired by a gyro sensor.

[0015] Hereinafter, each component will be described by taking the disaster situation awareness application as an example.

[0016] The spatial visualization system 100 constructs an image database 110 by analyzing the moving images acquired by a mobile imaging device, and presents the object detection results arranged on the spatial structure to the user. The spatial visualization system 100 has an image storage device 101, an input device 102, a display device 103, and an image search device 104.

[0017] The image storage device 101 is a storage medium that stores still image and video image data and attribute information associated with the image data. It can be configured as a hard disk drive built into a computer or a storage device connected via a network (e.g., NAS (Network Attached Storage), SAN (Storage Area Network)). The image storage device 101 may also be a cache memory that temporarily holds data continuously input from the imaging device. The image storage device 101 may be included in the storage device 202.

[0018] The input device 102 is an input interface for transmitting user operations, such as a mouse, keyboard, or touch device, to the image search device 104. If the input device 102 is a device equipped with an accelerometer, such as a smartphone, tablet, or head-mounted display, the orientation information of the input device 102 can be input to the image search device 104. The display device 103 is an output interface, such as a liquid crystal display, and is used for displaying search results by the image search device 104 and for interactive operation with the user.

[0019] The image search device 104 is a device that extracts spatial information and image semantic information necessary for searching, performs a registration process to create a database, and performs spatial structure visualization and image search processing using the registered data. The registration process will be described below.

[0020] In the registration process, the image retrieval device 104 constructs a spatial structure from the images and attribute information stored in the image storage device 101, extracts image semantic information, and merges the image semantic information and spatial structure to register them in the image database 110. The spatial structure is represented as a set of points in three-dimensional space, and a mesh can be represented by describing the connections between the points. Furthermore, by adding image data corresponding to the mesh, a textured spatial structure can be represented. The image semantic information contains information about the type of object contained in the image and its position on the two-dimensional image. Image semantic information with spatial information also contains information about the three-dimensional position and size of the object on the spatial structure. Details of the registration process will be explained with reference to Figure 7.

[0021] In the spatial structure visualization and image search processing, the system uses search criteria specified by the user via the input device 102 to search the image database 110 for images that match the search criteria and executes a search process to present the information on the display device 103. The image search device 104 can provide the user with a three-dimensional viewer using the spatial structure read from the image database 110. Furthermore, it can display information about objects obtained by image recognition on the three-dimensionally displayed spatial structure using image semantic information with spatial information. This allows the user to intuitively grasp an overview of the spatial distribution of the image recognition results. In addition, image information corresponding to the area specified in the three-dimensional viewer can be easily obtained.

[0022] The image retrieval device 104 includes an image input unit 105, a shooting information input unit 106, a spatial structure recognition unit 107, an image semantic recognition unit 108, an image semantic / spatial structure fusion unit 109, an image database 110, a context-utilizing query generation unit 111, an image retrieval unit 112, an image semantic / spatial structure summarization unit 113, and a display unit 114.

[0023] The image input unit 105 receives still image data or video data from the image storage device 101 and converts it into a data format used internally by the image retrieval device 104. For example, if the data received by the image input unit 105 is video data, the image input unit 105 performs video decoding to decompose it into frames (still image data format).

[0024] The shooting information input unit 106 receives data from the image storage device 101 containing the position information and orientation information of the shooting device. The position information is the three-dimensional coordinates of the shooting device in real space, and the orientation information represents the rotation angle of the shooting device. The position information and orientation information are acquired for each image acquired by the image input unit 105. In addition, camera parameters such as the shooting time, movement speed, acceleration, field of view, focal length, and lens distortion may also be accepted.

[0025] The spatial structure recognition unit 107 constructs a spatial structure from multiple images acquired by the image input unit 105. Structure from Motion (SfM), Visual Simultaneous Localization and Mapping (vSLAM), and other known photogrammetry techniques can be used to construct a spatial structure from multiple images taken from different viewpoints. In this case, providing auxiliary information such as the real-world position and orientation, and camera parameters acquired by the image acquisition information input unit 106 enables a more accurate construction. If position and orientation information for an image is unavailable, estimated values ​​are obtained during the spatial structure construction process, and these estimated position and orientation information are used in subsequent processing.

[0026] The image semantic recognition unit 108 performs image recognition processing on each image acquired by the image input unit 105 to detect objects and events in the image. Known image classification methods, object detection methods, and region estimation methods can be used for image recognition processing. Many recent methods allow for the construction of recognition models using machine learning, and arbitrary targets can be detected by changing the model used. Either a single model capable of detecting multiple types of targets can be used, or multiple models can be used depending on the type. As a result of the image recognition processing, the two-dimensional coordinates of the objects in the image, the type of object, and the confidence level of the recognition result are obtained.

[0027] The image semantic and spatial structure fusion unit 109 uses the position and orientation information of the image acquired by the image information input unit 106 and the two-dimensional coordinates of the object obtained by the image semantic recognition unit 108 to determine the three-dimensional position and size of the object in the spatial structure constructed by the spatial structure recognition unit 107, and stores the information obtained from the series of processes described above in the image database 110. The three-dimensional position of the object can be estimated by using the image position and orientation information to determine the three-dimensional coordinates at which a straight line is drawn from the center coordinates of the object in the image along the optical axis and collides with the spatial structure. The size of the object may be determined using a value set in advance according to the type of object, or it may be calculated using the size of the object in the image and the distance to the collision point between the image and the spatial structure. The image semantic and spatial structure fusion unit 109 may also extract image features that quantify the visual characteristics of the image and store them in the image database 110. Image features are usually given as fixed-length vector data, and images with close distances between vectors have high visual similarity, so they can be used for similar image search as needed in the spatial visualization and image search process described later.

[0028] The image database 110 stores spatial structure information, image information, and object information obtained through the registration process. The image database 110 can respond to queries from various parts of the image search device 104 by searching for registered data that satisfies the query conditions or outputting data with a specified ID. Furthermore, it can output registered images similar to the query image by using image features. Details of the structure of the image database 110 will be described later with reference to Figures 3A to 3C.

[0029] The above describes the operation of each part in the registration process of the image search device 104. Next, the operation of each part in the spatial visualization and image search process of the image search device 104 will be explained. The details of the spatial visualization and image search process will be explained in the flowchart in Figure 9.

[0030] The context-utilizing query generation unit 111 receives user operation information from the input device 102 and the state of the spatial structure presented to the user from the display unit 114, and generates an image search query from this information. The query may be a spatial structure identifier, an object type, an object position, or other conditions, or it may be an image feature for similarity search. It may also be a combination of one or more conditions and image features, and these image features and conditions may be assigned priorities or weights.

[0031] The image search unit 112 searches for object information from the image database 110 using the query generated by the context-utilizing query generation unit 111. For example, if the query is given as a condition, it outputs registered data that matches the condition. If the query is given as image features represented as vector data, it calculates the distance between the vectors and outputs registered data in order of high similarity (close distance between vectors).

[0032] The image semantic and spatial structure summarization unit 113 summarizes and simplifies the data to be presented to the user based on the search results obtained by the image search unit 112. Alternatively, the summarization process may be skipped and all search results displayed according to the user's instructions. The summarization process may, for example, exclude objects of the same type and spatial location that are similar, or combine objects that are determined to be a single object in three dimensions.

[0033] The display unit 114 displays the spatial structure read from the image database 110 on a three-dimensional viewer and visualizes the spatial structure from a viewpoint specified by the user via the input device 102. It also overlays the image search results obtained from the image semantic / spatial structure summarization unit 113 onto the three-dimensional viewer. Since the objects obtained through image recognition during the registration process have spatial position information, they may, for example, be placed as icons on the spatial structure. Furthermore, if necessary, image data and attribute information may be read from the image database 110, processed, and displayed on the screen.

[0034] The operation of each part of the spatial visualization and image retrieval processing of the image retrieval device 104 has been described above. It is preferable that the registration process and spatial visualization / image processing of the image retrieval device 104 be executed simultaneously. For example, if enough images to constitute a spatial structure have been input, it can be applied to a real-time system in which the constructed spatial structure is displayed on the display device 103, objects are detected from newly input images, and the detected objects are sequentially added and displayed on the viewer. The method is limited to vSLAM, but by using a method that updates the spatial structure for sequentially input images, the latest spatial structure can be displayed on the viewer.

[0035] Figure 2 is a block diagram showing an example of the hardware configuration of the spatial visualization system 100 in this embodiment.

[0036] The image retrieval device 104 includes a processor 201, a storage device 202, and a network interface device (NIC) 204. The processor 201, storage device 202, and network interface device 204 are connected, for example, by a bus.

[0037] The storage device 202 is composed of any type of storage medium, for example, a combination of semiconductor memory and a hard disk drive. Functional units such as the image input unit 105, the image capture information input unit 106, the spatial structure recognition unit 107, the image semantic recognition unit 108, the image semantic / spatial structure fusion unit 109, the context utilization query generation unit 111, the image search unit 112, the image semantic / spatial structure summarization unit 113, and the display unit 114 shown in Figure 1 are realized by the processor 201 executing the processing program 203 stored in the storage device 202. In other words, the processing performed by each functional unit is executed by the processor 201 based on the procedures defined in the processing program 203. Furthermore, the data in the image database 110 is stored in the storage device 202. Note that if the spatial visualization system 100 is configured with multiple devices for purposes such as processing load balancing, the device containing the image database 110 and the device executing the processing program 203 may be physically different devices connected by a network.

[0038] The program executed by the processor 201 is provided to the image retrieval device 104 via removable media (such as a CD-ROM or flash memory) or a network, and stored in the non-volatile, non-temporary storage medium (e.g., a hard disk drive) of the storage device 202. For this reason, the spatial visualization system 100 may have an interface for reading data from the removable media.

[0039] The image retrieval device 104 is a computer system that consists of one physical computer or multiple logically or physically configured computers, and may operate on a virtual computer built on multiple physical computer resources. For example, the registration process and spatial visualization / image processing may operate on separate physical or logical computers, or on a single physical or logical computer.

[0040] Figures 3A, 3B, and 3C are explanatory diagrams showing an example configuration of the image database 110 in this embodiment.

[0041] In this embodiment, the information used by the image retrieval device 104 is independent of the data structure and can be represented in any data structure. Figures 3A, 3B, and 3C show examples of a table format, but for example, a data structure appropriately selected from tables, lists, databases, or queues can store the information.

[0042] The image database 110 includes, for example, a spatial structure table 300 (Figure 3A) that holds spatial structure, an image table 310 (Figure 3B) that holds image information, and an object table 320 (Figure 3C) that holds object information. The configuration of each table and the field configuration of each table are examples, and tables and fields may be added or deleted depending on the application, for example. Also, the table configuration may be changed as long as similar information is held. For example, the image table 310 and the object table 320 may be joined to form a single table.

[0043] The spatial structure table 300 shown in Figure 3A includes a spatial structure ID field 301 and a spatial structure data field 302.

[0044] The spatial structure ID field 301 holds unique identification information for each spatial structure. The spatial structure data field 302 holds the constructed spatial structure data. The spatial structure data consists of three-dimensional vertex coordinate points, mesh structure information connecting the vertex coordinate points, texture image data, etc. These are necessary for point cloud display, mesh display, and textured mesh display when displayed in a three-dimensional viewer, but they may be stored in any format as long as they are compatible. Also, if the type of display can be limited to point cloud only, mesh only, etc., only some of the data may be stored.

[0045] The image table 310 shown in Figure 3B includes an image ID field 311, a spatial structure ID field 312, an image data field 313, a position field 314, an orientation field 315, and an image feature field 316. Fields such as the image capture time may also be included as needed.

[0046] The image ID field 311 holds unique identification information for each image. The spatial structure ID field 312 is a reference to the space in which the image was taken and holds the spatial structure ID managed in the spatial structure table 300. The image data field 313 holds the image data used for screen display in binary format. The position field 314 holds the three-dimensional position in the space in which the image was taken. The three-dimensional position may be an absolute position represented by a coordinate system in real space, such as [latitude, longitude, altitude], or a coordinate system in the spatial structure.<x、y、z> A relative position like this is also acceptable. The attitude field 315 holds data representing the rotation angle of the imaging device. The rotation angle can be represented in various ways, as long as it can appropriately reproduce the attitude of the imaging device in the spatial structure when the attitude information is used in the image semantic / spatial structure fusion unit 109 or the display unit 114. For example, it may be represented as a three-dimensional vector of [roll, pitch, yaw], or as a four-dimensional vector such as a quaternion. The image feature field 316 holds numerical vectors representing the features of the entire image.

[0047] The object table 320 shown in Figure 3C includes an object ID field 321, an image ID field 322, an object type field 323, an image position field 324, a confidence field 325, a spatial position field 326, an orientation field 327, a distance field 328, and a size field 329.

[0048] The Object ID field 321 holds unique identification information for each object. The Image ID field 322 is a reference to the original image in which the object was detected and holds the Image ID managed in the Image Table 310. The Object Type field 323 holds the object type. The object type may be stored directly as a string, as shown in the figure, or as a numerical value corresponding to the type. The Image Position field 324 holds the object's position information within the image. For example, if the object's area is represented by a rectangle, it can be represented by a four-dimensional vector of [top-left x-coordinate, top-left y-coordinate, width w, height h]. The Confidence field 325 holds a numerical value representing the confidence level of the image recognition result. For example, it is a value in the range of 0.0 to 1.0, with 1.0 being the highest confidence level. The Spatial Position field 326 holds the coordinates of the object in three-dimensional space, calculated by the Image Semantic / Spatial Structure Fusion Unit 109. The Direction field 327 holds the orientation of the straight line connecting the imaging device and the object in three-dimensional space. From the direction value, it is possible to know from what angle the object was photographed. The distance field 328 holds the length of the straight line connecting the imaging device and the object in three-dimensional space. From the distance value, it is possible to know how far away the object was photographed. The size field 329 holds the size information of the object in three-dimensional space, calculated by the image semantic-spatial structure fusion unit 109. The size information may be, for example, the radius, the range of each of the x, y, and z axes, or mesh data surrounding the object. In the following examples, for simplicity, the radius will be used as the size information.

[0049] Figure 4 is a diagram illustrating the overview of the spatial structure configuration processing performed by the spatial structure recognition unit 107 in this embodiment.

[0050] Known methods such as SfM and vSLAM can be used to construct the spatial structure. Multiple images 402 from different viewpoints, acquired by the imaging device 401, are required for spatial structure construction. Numerous feature points 403 are extracted from each image, and feature point matching is performed across the images to find identical points appearing in multiple images. The position and orientation of the imaging device are estimated, and the three-dimensional position of the points is estimated based on the principle of triangulation. At this time, it is advisable to improve the accuracy of the position by referring to real-world position and orientation information corresponding to the images. By repeating this process, a collection of numerous three-dimensional points (point cloud) can be obtained. Then, adjacent points are connected to generate a mesh, and a texture is projected onto the mesh. The constructed spatial structure 404 can be displayed from various viewpoints using a three-dimensional viewer.

[0051] Figure 5 is a diagram illustrating the overview of the image recognition process performed by the image semantic recognition unit 108 in this embodiment.

[0052] In image recognition processing, objects such as people, cars, and buildings, as well as events such as landslides, fires, and smoke, are detected in the input image 501. For image recognition, known methods such as object detection methods and region detection methods using models that respond to object regions and are trained with deep learning can be used. As a result of the processing, image semantic information is obtained, for example, including a rectangle 502 surrounding the object region, the type of object 503, and the confidence level of recognition 504.

[0053] Figure 6 is a diagram illustrating the overview of the processing performed by the image semantic and spatial structure fusion unit 109 in this embodiment.

[0054] The image semantic and spatial structure fusion unit 109 determines the three-dimensional position and size of the object 601 detected by the image semantic recognition unit 108 in the spatial structure 404. First, the three-dimensional position and orientation of the image 602 containing the object are determined from the data recorded during shooting and the values ​​estimated by the spatial structure recognition unit 107. Next, a straight line 603 is extended along the optical axis from the position and orientation of the image to find the point 604 that collides with the spatial structure 404, and this is the three-dimensional position of the object. The size 605 of the object may be determined using a value predetermined from the type of object, or it may be a value obtained by scaling the rectangular size of the object in the image in proportion to the distance between the image and the object in three-dimensional space.

[0055] The input image recognition process and database registration process can be performed using any procedure once the database configuration example shown in Figures 3A to 3C has been accumulated. For example, the procedure shown in the flowchart in Figure 7 may be used.

[0056] Figure 7 is a flowchart of the database registration process. The steps in Figure 7 are explained below. The trigger for executing the data registration process is, for example, inputting a set of image data captured by the user into the system. Details of the triggers will be described later in Figure 15, which is an overall sequence diagram of the registration and search processes.

[0057] The image input unit 105 acquires image data from the image storage device 101 and converts the acquired image data into a format usable within the system as needed (S701). For example, when video data is received as input, the conversion process includes video decoding, which decomposes the video data into frames (still image data format).

[0058] The shooting information input unit 106 acquires the position and orientation data of the shooting device at the time of shooting, which is recorded in the image storage device 101, and transforms the coordinate system as necessary (S702).

[0059] The spatial structure recognition unit 107 constructs a spatial structure using the image set acquired in step S701 and the position and orientation data of the images acquired in step S702, and registers the spatial structure data in the image database 110 (S703). As a result, spatial structure data such as a point cloud, mesh, or textured mesh can be obtained.

[0060] The image retrieval device 104 performs the procedures from steps S705 to S709 for each of the images acquired in step S701 (S704).

[0061] The spatial structure recognition unit 107 adds three-dimensional position and orientation information to the image acquired in step S701, based on at least one of the information of the imaging device acquired in step S702 and the values ​​estimated during the spatial structure configuration processing in step S703, and registers the image information in the image database 110 (S705).

[0062] The image semantic recognition unit 108 detects objects from the image acquired in step S701 through image recognition processing (S706). As a result, the position and size of the object in the two-dimensional coordinate system within the image are obtained.

[0063] The image semantic and spatial structure fusion unit 109 executes the procedure in step S708 if an object is detected within a predetermined area in step S706, and proceeds to step S710 (step S707) if no object is detected within the predetermined area. Here, the predetermined area is an area that is not far from the optical axis of the imaging device (the perpendicular to the center position of the image). Objects located far from the optical axis can lead to large errors when determining their spatial position if the camera parameters are not accurately reflected, so it is advisable to exclude them as needed. If it is desirable to retain many object detection results, it is advisable to record the position estimation confidence, which is inversely proportional to the distance from the center coordinates, in the image database 110, and narrow down the objects to display according to the position estimation confidence.

[0064] The image semantic and spatial structure fusion unit 109 estimates the three-dimensional position and size of the object in the spatial structure from the two-dimensional position of the object in the image obtained in step S706, the three-dimensional position and orientation of the image obtained in step S705, and the spatial structure constructed in step S703 (S708). In this process, as described above in the explanation of Figure 6, the position and size of the object are determined by finding the point and distance where a straight line extended from the image along the optical axis collides with the spatial structure.

[0065] The image semantic and spatial structure fusion unit 109 registers the object information obtained in steps S706 to S708 in the image database 110 (S709). The image semantic and spatial structure fusion unit 109 may also calculate image features as needed and store them in the image feature field 316 of the image table 310 in the image database 110.

[0066] The image semantic recognition unit 108 terminates the registration process (S710) after processing all images. If new data is continuously recorded in the image storage device 101, the unit waits until the new data is stored, then returns to step S701 and repeats the registration process.

[0067] Figure 8 illustrates the spatial visualization and image search processing performed by the image search device 104 in this embodiment.

[0068] Spatial structure, image information, and object information stored in the image database 110 are displayed on the display device 103 in response to user operations from the input device 102. The user operates the user interface displayed on the screen using a mouse cursor 801 or the like. For example, when a spatial structure ID is entered into the spatial structure ID input form 802, the context utilization query generation unit 111 generates a query to obtain the spatial structure of the entered ID, and the image search unit 112 obtains information on the spatial structure and the objects contained in that space from the image database 110. The image meaning / spatial structure summarization unit 113 aggregates the object information obtained from the image database 110 as needed. For example, it groups objects that are close in spatial location. The display unit 114 displays the spatial structure 404 and object icons 803 on the display device 103. Also, when the user selects an object icon, it obtains detailed information of the original image in which the object was detected from the image database 110 and displays the image in the pop-up window 804.

[0069] Figure 9 is a flowchart of the spatial visualization and image search process performed by the image search device 104 in this embodiment. The steps in Figure 9 will be explained below.

[0070] The context-utilizing query generation unit 111 obtains user screen operations from the input device 102 and receives the spatial structure ID. The image search unit 112 obtains the spatial structure data for the specified ID from the image database 110 (S901).

[0071] The display unit 114 displays the spatial structure represented by the spatial structure data acquired in step S901 on the display device 103 using a three-dimensional viewer (S902).

[0072] The image search unit 112 retrieves a list of object IDs in the spatial structure for the specified ID from the image database 110 (S903).

[0073] The image search device 104 performs the procedures from steps S905 to S910 for each of the object IDs acquired in step S903 (S904).

[0074] The image semantic and spatial structure summarization unit 113 retrieves object information from the image database 110 (S905).

[0075] The image search device 104 executes step S908 if another object is already located near the spatial location of the object, and executes step S907 if no other object is located there (S906).

[0076] The display unit 114 places the object icons superimposed on the spatial structure displayed in step S902 (S907). The size and style of the icons, such as color, may be changed according to the estimated size of the objects.

[0077] The image search device 104 executes step S909 if the user selects an object icon, and executes step S911 if the user has not selected an object icon (S908). The image search unit 112 retrieves image information of the selected object from the image database 110 (S909).

[0078] The display unit 114 uses a three-dimensional viewer to superimpose the details of the image information acquired in step S909 onto the spatial structure displayed in step S902, and displays the details in a pop-up window (S910).

[0079] The image search device 104 terminates the visualization and image search process after processing all object IDs (S911). Steps S908 to S910 are executed as needed according to the user's screen operations.

[0080] Figure 15 is a sequence diagram illustrating an example of the processing of the spatial visualization system 100 of Embodiment 1. Specifically, Figure 15 shows the processing sequence between the user 1200, the image storage device 101, the computer 1540, and the image database 110 in the image registration process S15000 and the spatial visualization / image search process S1520 of the spatial visualization system 100 described above. Note that the computer 1540 is the computer that implements the image search device 104.

[0081] The registration process S1500 starts when user 1200 requests data registration from computer 1540 (S1501). The registration process S1500 corresponds to the process described in Figure 7 and is executed repeatedly until the processing of the image file specified by the user is completed. Computer 1540 requests image and location data from image storage device 101 (S1502) and obtains image and location data from image storage device 101 (S1503). Computer 1540 constructs a spatial structure using the multiple image and location data obtained (S1504), registers the constructed spatial structure in the image database 110 (S1505), and receives a spatial structure ID (S1506). A series of processes S1507 are performed for each image. In the series of processes S1507, image semantic information is recognized (S1508) and the spatial position and size of the object are estimated (S1509). Computer 1540 registers the estimated spatial position and size of the object in the image database 110 (S1510) and receives the spatial structure ID (S1511). Once all images have been processed, it notifies user 1200 that registration is complete (S1512).

[0082] The spatial visualization and image retrieval process S1520 corresponds to the process described in Figure 9, and is initiated when user 1200 requests information display from computer 1540 (S1521). Computer 1540 obtains the spatial information requested by the user from the image database 110 (S1522, S1523) and obtains image and object information associated with the spatial information (S1524, S1525). This information is appropriately processed for display (S1526) and presented to user 1200 through the display device 103 (S1527). When user 1200 gives instructions to computer 1540 for screen viewpoint operation through input device 102 (S1528), computer 1540 changes the pose information of the virtual shooting device in three-dimensional space and obtains the pose information (S1529). Then, it generates a query based on the gaze point from the virtual shooting device (S1530) and searches the image database 110 (S1531). The search results are displayed on the screen of the display device 103 and the information is presented to the user (S1532).

[0083] As described above, the spatial visualization system 100 of Example 1 can appropriately visualize a wide variety of objects and events of different sizes that exist over a wide area, such as disaster situations, on a spatial structure, allowing users to quickly grasp the situation. [Examples]

[0084] Next, Embodiment 2 of the present invention will be described. In Embodiment 1, object detection results are displayed as icons on the spatial structure, enabling quick understanding of the situation over a wide area and facilitating access to necessary image information. However, as the number of images increases and a wide variety of objects and events are detected, numerous icons are displayed on the screen, making it difficult to access the desired information. Embodiment 2 shows a method for summarizing information on a large number of objects over a wide area. In Embodiment 2, the explanation of the same processes and functions as in Embodiment 1 will be omitted, and the differences will be explained mainly.

[0085] Figure 10 is a diagram illustrating the overview of the object information summarization process.

[0086] Screen 1001 shows the spatial structure 404 and object icons 1002 from an overhead view. Part of the screen is filled with object icons, reducing visibility. In the summarization process, for example, a data clustering method is used to group the object information and display group icons 1003. In addition, the types of objects included in the group are displayed as string labels 1004. The display method may be changed depending on the combination of types. For example, if disaster-related types are included, they may be highlighted. For the clustering method, for example, the known K-means method can be used. The clustering process can use vector data indicating the spatial position extracted from each object's information. In addition, the object type, size, orientation of the image, and distance may be added to the vector.

[0087] Figure 11 is a flowchart of the summarization process performed by the image retrieval device 104 in Example 2. The steps in Figure 11 will be described below.

[0088] The image search unit 112 obtains a list of object information to be displayed on the spatial structure (S1101).

[0089] The image search device 104 determines whether summarization processing is necessary based on user input or the number of icons displayed on the screen. If it determines that summarization processing is necessary, it proceeds to step S1103. If it determines that summarization processing is not necessary, it displays the icons for each object according to the flowchart in Figure 9 and terminates the process.

[0090] The image semantic and spatial structure summarization unit 113 generates a vector set from the object information list acquired in step S1101 and groups the objects by clustering the vector set (S1103). In generating the vector set of object information, the spatial position of the objects and numerical values ​​of the object type can be used to vectorize using known methods such as the K-means method. Since the K-means method requires specifying the number of clusters, it is preferable to use a numerical value or predetermined value specified by the user. Alternatively, the X-means method, in which the number of clusters is automatically determined, may be used.

[0091] The image semantic and spatial structure summarization unit 113 performs steps S1105 to S1107 on the clusters obtained in step S1103 (S1104).

[0092] The image semantic and spatial structure summarization unit 113 obtains location and type information of objects included in the cluster from the image database 110 (S1105).

[0093] The image semantic / spatial structure summarization unit 113 generates a group icon 1003 that indicates the region containing the objects included in the cluster, and places the generated group icon 1103 on the spatial structure (S1106).

[0094] The image semantic and spatial structure summarization unit 113 aggregates the types of all objects included in the cluster, generates icon labels 1004, and displays the generated labels 1004 (S1107). The display manner of the labels 1004 may be changed depending on the combination of object types.

[0095] The image search device 104 terminates the summarization process once it has finished processing all clusters.

[0096] As described above, the spatial visualization system 100 of Example 2, in addition to the effects of Example 1, can summarize and display a large amount of image recognition results in space, allowing users to efficiently grasp a wide area. [Examples]

[0097] Next, Embodiment 3 of the present invention will be described. Embodiment 1 showed an example of a user interface in which detailed image information is displayed when the user selects an object icon. However, when the number of images or objects increases, the method of clicking each icon one by one to display the necessary information becomes inefficient. Embodiment 3 shows a method of automatically generating a search query and presenting detailed image information using contextual information obtained from the user's operation and the state of the 3D viewer. Note that in Embodiment 3, the explanation of the same processes and functions as in Embodiment 1 will be omitted, and the differences will be explained mainly.

[0098] Figure 12 illustrates image search utilizing the context of a 3D viewer.

[0099] In the spatial visualization system 100 of Embodiment 3, the user 1200 manipulates the spatial structure displayed on the display device 103 using the input device 102, viewing the spatial structure while changing the viewpoint. Even if the user 1200 freely changes the viewpoint, the center 1201 of the screen is highly likely to be the user's point of focus. Furthermore, the spatial structure displayed on the screen is considered to be an image captured from a virtual shooting device 1202 that can be freely moved by the user's operation in three-dimensional space. At this time, similar to the process of estimating the three-dimensional position from the two-dimensional position of an object in the image semantic-spatial structure fusion unit 109, the position of the user's point of focus in the three-dimensional structure can be estimated. Once the three-dimensional position 1203 of the point of focus is estimated, object information in the surrounding area can be automatically obtained from the image database 110. For example, objects included in a predetermined search distance range 1204 from the point of focus can be searched. At this time, the distance from the point of interest may be calculated considering not only the spatial position 1205 of the object but also its size 1206. In addition to distance, the direction of capture, distance from the shooting device, image similarity, and object type may also be added as search conditions. The list of 1208 search results is automatically presented to the user.

[0100] Figure 13 is a flowchart of the context-based image search process of the image search device 104 in Example 3. The steps in Figure 13 will be described below.

[0101] The display unit 114 changes the position and orientation of the virtual imaging device in three-dimensional space in response to user input and displays spatial information as seen from the virtual imaging device on the screen (S1301).

[0102] The context-utilizing query generation unit 111 acquires information on the position and orientation of the virtual imaging device from the display unit 114 (S1302). The acquired position and orientation of the virtual imaging device are the viewpoint position and line of sight direction.

[0103] The context-utilizing query generation unit 111 estimates the three-dimensional position of the user's gaze point from the position and orientation information obtained in step S1302 and generates a search query (S1303). In addition to the distance range condition from the gaze point, the search query can also include the direction of shooting, the distance from the shooting device, and the type of object. Alternatively, spatial information viewed from the virtual shooting device may be treated as an image, and image features of the image may be calculated to perform a similar image search.

[0104] The image search unit 112 retrieves information on images and objects that match the query generated in step S1303 from the image database 110 (S1304).

[0105] The image semantic and spatial structure summarization unit 113 summarizes the search results as necessary (S1305). The summarization process is the same as the method shown in Example 2.

[0106] The display unit 114 displays the image and object information obtained in step S1304 on the screen (S1306). The display method may be to display the search results list in a pop-up window or to display it directly in three-dimensional space.

[0107] The above processes may be triggered by user actions such as button clicks, or by changes in the position or orientation of the virtual imaging device.

[0108] As described above, the spatial visualization system 100 of Example 3 allows users to efficiently obtain necessary information by generating intuitive search queries linked to viewpoint manipulation of the 3D viewer, without having to select an object. In particular, when using a smartphone, tablet, or head-mounted display as the input device 102 and display device 103, it can be difficult to specify detailed conditions on the screen. Therefore, by linking the device's acceleration sensor with the viewpoint manipulation of the 3D viewer, the user can be presented with information on 3D structures from different viewpoints and corresponding images and objects simply by moving the device.

[0109] Furthermore, by combining Examples 2 and 3, for example, when the user zooms out to view the three-dimensional structure from a bird's-eye perspective, a summary display may be shown in cluster units, and when the user zooms in, icons for individual objects may be displayed.

[0110] Figure 14 shows an example of the configuration of an operation screen for visualizing spatial information and performing image retrieval using the image retrieval device 104 described in Examples 1 to 3 above.

[0111] The image search device 104 displays the processing results on the display device 103. The user inputs operation information to the image search device 104 using the input device 102, such as the mouse cursor 1401 displayed on the screen. The screen displays the spatial structure 401, detailed image information 1402, object icons or grouping icons 1403, image search conditions 1404, and image search results 1405. The image search results 1405 are preferably displayed in order of similarity of image features, but the display order may be changed. The example screen configuration is just one example, and these elements may be freely arranged to configure the screen.

[0112] It should be noted that the present invention is not limited to the embodiments described above, but includes various modifications and equivalent configurations within the spirit of the attached claims. For example, the embodiments described above are described in detail for the purpose of clearly illustrating the present invention, and the present invention is not necessarily limited to having all the described configurations. Furthermore, some of the configurations of one embodiment may be replaced with those of another embodiment. Furthermore, configurations of other embodiments may be added to the configuration of one embodiment. Furthermore, some of the configurations of each embodiment may be added, deleted, or replaced with those of other embodiments.

[0113] Furthermore, each of the aforementioned configurations, functions, processing units, and processing means may be implemented in hardware, for example, by designing them as integrated circuits, or they may be implemented in software by having a processor interpret and execute programs that realize each function.

[0114] Information such as programs, tables, and files that implement each function can be stored in memory, hard disks, SSDs (Solid State Drives), or other storage media such as IC cards, SD cards, and DVDs.

[0115] Furthermore, the control lines and information lines shown are those deemed necessary for explanation purposes and do not necessarily represent all control lines and information lines required for implementation. In reality, it can be assumed that almost all components are interconnected. [Explanation of Symbols]

[0116] 100 Spatial Visualization Systems 101 Image storage device 102 Input device 103 Display device 104 Image search device 105 Image Input Section 106 Shooting Information Input Section 107 Spatial structure recognition unit 108 Image semantic recognition unit 109 Spatial structure fusion part 110 Image Databases 111 Context-Based Query Generation Unit 112 Image Search Section 113 Spatial structure summary part 114 Display section 201 Processor 202 Storage device 203 Processing Program 204 Network Interface Device 300 Spatial Structure Table 310 Image Table 320 Object Tables

Claims

1. 1. A spatial visualization system, comprising: A computing device for executing a predetermined computation process; a storage device accessible by the computing device; The arithmetic device includes a spatial structure recognition unit that configures a spatial structure from a plurality of images; an image semantic recognition unit for detecting an object included in each of the plurality of images; the computing device has an image semantic / spatial structure fusion unit that estimates a spatial position of the detected object on the spatial structure; A spatial visualization system characterized in that the storage device stores information about the images that constitute the spatial structure, the constructed spatial structure, and information about the detected objects.

2. 2. The spatial visualization system of claim 1, A spatial visualization system, characterized in that the storage device stores information on the positions and sizes of the detected objects.

3. 3. The spatial visualization system of claim 2, The spatial visualization system is characterized in that the storage device stores information on the direction and distance of the detected object from the shooting position of the image in which the object was captured.

4. 2. The spatial visualization system of claim 1, The storage device stores feature quantities of images that constitute the spatial structure.

5. 3. The spatial visualization system of claim 2, The spatial structure recognition unit estimates a three-dimensional position and orientation of the image from at least one of information on a shooting position of the image and a value estimated in a process of processing a spatial structure configuration, The image semantic / spatial structure fusion unit estimates the three-dimensional position and size of the object in the spatial structure from the two-dimensional position of the detected object in the image, the three-dimensional position and orientation of the estimated image, and the constructed spatial structure, and stores the estimate in the storage device.

6. 6. The spatial visualization system of claim 5, A spatial visualization system characterized in that the image semantic / spatial structure fusion unit estimates the three-dimensional position and size in the spatial structure of objects detected within a predetermined range from a perpendicular line to the center position of the image.

7. 2. The spatial visualization system of claim 1, The image semantic / spatial structure summarization unit clusters the objects based on the similarity of the object information, and places a group icon indicating an area containing all of the objects included in the generated cluster on the spatial structure.

8. 2. The spatial visualization system of claim 1, The image semantic and spatial structure summarization unit displays icon labels whose display manner differs depending on the type of all objects included in the cluster.

9. 2. The spatial visualization system of claim 1, a context-utilizing query generation unit that acquires a viewpoint position and a line-of-sight direction of the displayed spatial information, and generates a search query including a three-dimensional position of a user's gaze point as a condition from the acquired position and orientation information; and an image search unit that retrieves images from the storage device using the generated search query.

10. 10. The spatial visualization system of claim 9, The context-utilizing query generation unit treats spatial information seen from the viewpoint position as an image, calculates image features of the image, and generates a search query that includes the calculated image features as conditions.

11. 1. A computer implemented spatial visualization method, comprising: The computer includes an arithmetic unit that executes a predetermined arithmetic process and a storage device that is accessible by the arithmetic unit; The spatial visualization method includes: a spatial structure recognition step in which the arithmetic device configures a spatial structure from a plurality of images; an image semantic recognition step in which the computing device detects objects contained in each of the plurality of images; an image semantic and spatial structure fusion procedure in which the computing device estimates a spatial position of the detected object on the spatial structure; A spatial visualization method characterized in that the calculation device includes a procedure for storing, in the storage device, information on the images from which the spatial structure is constructed, the constructed spatial structure, and information on the detected objects.

Images