Information processing apparatus, information processing system, information processing method, and program

Abstract

To provide an information processing apparatus, an information processing system, an information processing method, and a program configured to facilitate setting of an annotation region in a three-dimensional space and clarify association between annotation information and an object disposed in the three-dimensional space or a range.SOLUTION: In an information processing system, an information processing apparatus (server device 200) includes: a space processing unit which acquires first coordinates in a first image appearing in a first field of view from a first viewpoint and second coordinates in a second image appearing in a second field of view from a second viewpoint in a three-dimensional space, on the basis of space information representing a distribution of objects arranged in the three-dimensional space; a region setting unit which determines a first straight line directed in a first direction corresponding to the first coordinates starting from the first viewpoint, and a second straight line directed in a second direction corresponding to the second coordinates starting from the second viewpoint, and sets annotation region in the three-dimensional space; and an annotation management unit which assigns the annotation information to a partial region of the annotation region.SELECTED DRAWING: Figure 3

Description

[Technical field]

[0001] The present embodiment relates to an information processing device, an information processing system, an information processing method, and a program. [Background technology]

[0002] In depopulated areas, the closure of commercial facilities is remarkable due to population decline. Coupled with a lack of means of transportation, a social problem has arisen, such as an increase in residents (so-called "shopping refugees") whose daily lives are hindered. In this regard, a demonstration experiment was conducted on a remote purchasing service for residents living in areas far from stores. In the experiment, a camera was installed in the store and the captured images were sent to the user's terminal. If the camera was fixed, the same products would always be presented to the user, and the freedom of product selection was not exercised. In addition, it was considered to have the user wear AR (Augmented Reality) glasses, present images of the inside of the store, and change the direction of the camera according to the user's own line of sight, but when finding a desired product, it was difficult to operate, such as changing the direction of the camera.

[0003] Therefore, a system has been proposed in which a self-propelled robot is introduced into a store to be able to move around the store, and a camera installed on the robot is used to capture images in quasi-real time. In this system, a web application (hereinafter, "web app") with a walk-through function is implemented on a user terminal. The walk-through function is a function that presents images arranged in a three-dimensional (3D) space, and moves the viewpoint according to the accepted operation. In this system, a self-propelled robot equipped with a 360-degree camera and a smartphone-integrated camera is introduced. The 360-degree camera is used to obtain a panoramic image showing the space inside the store, and the smartphone-integrated camera is used to obtain images mainly showing the product shelves. By periodically presenting the captured images on the user terminal, a more realistic walk-through function is realized.

[0004] Annotation information about products may be presented at specific positions in three-dimensional space. The annotation information may be presented in the form of tags, icons, etc. For example, the annotation information may be a character string representing an image, price, volume, etc. of a specific product, or the URL of an online store. However, this structure is not suitable for partial and frequent information updates. For example, in a facility such as a supermarket, the scene of the sales floor changes partially and frequently depending on the arrangement of individual products, their selling prices, and other factors.

[0005] For example, the 3D tour additional information display system described in Patent Document 1 provides a user terminal with a 3D tour including 3D information of a specific space, receives coordinate information for the start and end points of an additional information signpost newly created on the 3D tour displayed on the user terminal, creates an additional information signpost connecting the viewpoint and the end point, and displays the created additional information signpost overlapped on the 3D tour.

[0006] In the information processing method described in Patent Document 2, a viewpoint position and a line of sight direction on a map are determined, and an annotation display position of the object is determined from the position of the object on the map, which is determined by the observation direction of the object in multiple panoramic images, and the viewpoint position and line of sight direction, and an annotation image is synthesized at the annotation display position on the actual image corresponding to the viewpoint position. [Prior art documents] [Patent documents]

[0007] [Patent Document 1] Patent Publication No. 2021-72125 [Patent Document 2] JP 2004-234455 A Summary of the Invention [Problem to be solved by the invention]

[0008] In general, when the object to be associated with the annotation information is an individual product that occupies a relatively small area, the user can easily identify the product that corresponds to the annotation information. However, when the object on which the annotation information is placed has a relatively large area (e.g., a product shelf), the user may not immediately understand what object the annotation information is associated with.

[0009] In addition, in a three-dimensional space that does not have a geometric model, it may be difficult for a user to specify coordinates when adding annotation information. For example, the method described in Patent Document 1 only indicates the start and end points of an additional information signpost, and does not present a spatial range corresponding to the additional information. This tends to make it inefficient to set multiple annotation information individually in a three-dimensional space according to user operations. [Means for solving the problem]

[0010] An information processing device according to a first aspect includes a spatial processing unit that provides a first image appearing in a first field of view from a first viewpoint in a three-dimensional space based on spatial information representing a distribution of objects placed in the three-dimensional space, acquires first coordinates that are coordinates within the first image, provides a second image that represents an image common to the first field of view and appears in a second field of view from a second viewpoint in the three-dimensional space, and acquires second coordinates that are coordinates within the second image, a region setting unit that identifies a first straight line in the three-dimensional space that is oriented in a first direction corresponding to the first coordinates starting from the first viewpoint, identifies a second straight line that is oriented in a second direction corresponding to the second coordinates starting from the second viewpoint, determines an intersection between the first straight line and the second straight line, and sets an annotation region in the three-dimensional space having a predetermined range that includes the intersection, and an annotation management unit that acquires annotation information and assigns the annotation information to a partial region of the annotation region.

[0011] In an information processing method according to a second aspect, an information processing device provides a first image appearing in a first field of view from a first viewpoint in a three-dimensional space based on spatial information representing a distribution of objects placed in the three-dimensional space, acquires first coordinates that are coordinates within the first image, provides a second image representing an image common to the first field of view and appearing in a second field of view from a second viewpoint in the three-dimensional space, acquires second coordinates that are coordinates within the second image, identifies a first straight line in the three-dimensional space that is oriented in a first direction corresponding to the first coordinates starting from the first viewpoint and identifies a second straight line that is oriented in a second direction corresponding to the second coordinates starting from the second viewpoint, determines an intersection between the first straight line and the second straight line, sets an annotation region having a predetermined range including the intersection in the three-dimensional space, acquires annotation information, and assigns the annotation information to a partial region of the annotation region. Effect of the Invention

[0012] According to the present application, it is possible to easily set an annotation area in a three-dimensional space, and to clarify the relationship between annotation information and an object or area arranged in the three-dimensional space by using the set annotation area. [Brief description of the drawings]

[0013] [Figure 1] 1 is a schematic block diagram illustrating a configuration example of an information processing system according to a first embodiment. [Diagram 2] 1 is a schematic block diagram illustrating an example of a system configuration of a user terminal device according to a first embodiment. [Diagram 3] 2 is a schematic block diagram illustrating an example of a system configuration of a server device according to the first embodiment. FIG. [Figure 4] FIG. 2 is an explanatory diagram for explaining the relationship between a virtual space and a viewpoint image. [Diagram 5] 1 is an explanatory diagram for explaining an intersection of a line passing from a viewpoint in a virtual space to a coordinate on a viewpoint image; [Figure 6] FIG. 13 is a diagram illustrating an example of an intersection in a virtual space. [Figure 7] 11 is a diagram showing an example of an annotation region having a width of a line segment whose both ends are intersections in a virtual space; FIG. [Figure 8] FIG. 13 is a diagram showing an example of an allocation region set in an annotation region. [Figure 9] FIG. 13 is a diagram showing an example of an annotation set in an annotation region. [Figure 10] FIG. 13 illustrates an example of an annotation editing screen. [Figure 11] 13A and 13B are diagrams illustrating other examples of annotations set in annotation regions. [Figure 12] 5 is a flowchart showing an example of annotation information processing according to the first embodiment. [Figure 13] 13 is a flowchart showing another example of the annotation information processing according to the first embodiment. [Figure 14] FIG. 11 is a schematic block diagram illustrating an example of a system configuration of a server device according to a second embodiment. [Figure 15] FIG. 13 is a diagram showing an example of editing an annotation region. [Figure 16] FIG. 13 is a diagram showing another example of editing an annotation region. [Figure 17] 13 is a flowchart showing an example of annotation information processing according to the second embodiment. [Figure 18] FIG. 13 is a diagram showing an example of an annotation region according to the third embodiment. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0014] <First embodiment> The embodiments of the present disclosure will be described with reference to the drawings. First, a first embodiment will be described. FIG. 1 is a schematic block diagram showing a configuration example of an information processing system 1 according to the present embodiment. The information processing system 1 according to the present embodiment includes one or more user terminal devices 40 and a server device 200. The user terminal device 40 is connected to the server device 200 via a network NW so that various data can be transmitted and received wired or wirelessly.

[0015] The user terminal device 40 is, for example, an information terminal device such as a personal computer (PC), a tablet terminal device, a mobile phone (including a so-called smartphone), etc. The user terminal device 40 is used by an individual user. The server device 200 is mainly managed by a service provider, such as a web server. The services provided are typically the display of products to be traded, business negotiations or transactions, and the display of handicrafts, artworks, etc. In the present application, the main service provided is retail, and an image showing a three-dimensional virtual space simulating the space inside a store as a facility is provided to the user terminal device 40.

[0016] The network NW may be any one of the Internet, a public land mobile network (PLMN), a local area network (LAN), a virtual personal network (VPN), etc., or any combination thereof. The number of user terminal devices 40 is generally unspecified and may be multiple. In the example of FIG. 1, the number of user terminal devices 40 is two. The multiple user terminal devices 40 are distinguished by using a sub-number such as "-1". In the present application, when there is a matter common to the multiple user terminal devices 40 or when there is no need to distinguish between the multiple user terminal devices 40, they are simply referred to as user terminal devices 40.

[0017] Next, an example of the system configuration of the user terminal device 40 according to this embodiment will be described. Fig. 2 is a schematic block diagram showing an example of the system configuration of the user terminal device 40 according to this embodiment. The user terminal device 40 includes a memory 110, a CPU 111, buses 120a and 120b, a bridge 130, an interface 140, an input unit 141, a display unit 142, a storage unit 143, and a communication unit 144.

[0018] The memory 110 is connected to the CPU 111 via a bus 120a so as to be able to input and output various types of data, and functions as a main memory that provides a working area for the CPU 111. The memory 110 stores various programs executed by the CPU 111, various types of data used in the processes executed by the CPU 111, and various types of data acquired by the CPU 111. The memory 110 includes, for example, a volatile memory such as a RAM (Random Access Memory) and a non-volatile memory such as a ROM (Read Only Memory).

[0019] The CPU 111 executes various programs and provides the overall functions of the user terminal device 40 in cooperation with the memory 110 and other devices. In this application, "executing a program" or "running a program" includes the meaning of executing a process instructed by an instruction written in the program. The CPU 111 executes programs such as system firmware and an operating system (OS). The CPU 111 also executes programs such as application programs (sometimes referred to as "apps" in this application), device drivers, and utilities on the OS. The CPU 111 is connected to the memory 110 and the bridge 130 by the bus 120a. The CPU 111 and the memory 110 are the minimum devices required to configure the main computer system of the user terminal device 40 (sometimes referred to as the "host system" in this application). This host system can also be considered as a control unit of the user terminal device 40. In this application, the host system of the user terminal device 40 is sometimes referred to as the "control unit."

[0020] The bus 120a connects the CPU 111 to the memory 110 and the bridge 130. The bus 120a functions as a CPU bus. The bridge 130 relays between the bus 120a and the bus 120b. The bridge 130 converts the data transmission method of the bus 120a to the data transmission method of the bus 120b. The bridge 130 also converts the data transmission method of the bus 120b to the data transmission method of the bus 120a.

[0021] The bus 120b is connected to an input unit 141, a display unit 142, a storage unit 143, and a communication unit 144 via an interface 140. The bus 120b functions as an I / O (Input / Output) bus. The interface 140 functions as an input / output interface that connects to each device so that various data can be input / output. The interface 140 includes a controller for controlling data input / output in an input / output format with the connected device.

[0022] The input unit 141 receives an operation from a user, and outputs operation information corresponding to the received operation to the host system via the interface 140. The input unit 141 includes, for example, any one of pointing devices such as a mouse, a touch sensor, or a joystick. The display unit 142 displays various display information indicated in the display data input from the host system. The display information may include, for example, a viewpoint image that appears in a predetermined field of view from a certain viewpoint. The display unit 142 may include, for example, any of a liquid crystal display (LCD) and an organic light emitting diode (OLED) display.

[0023] The storage unit 143 includes an auxiliary storage device that stores various data in a readable and writable manner on a continuous basis. The storage unit 143 includes, for example, any one of storage media such as a hard disk drive (HDD: Hard-disk Drive), a solid-state storage device (SSD: Solid State Drive), a flash memory, and an EEPROM (Electrically Erasable Programmable Read-only Memory), or any combination of these.

[0024] The communication unit 144 is connected to the network NW, and transmits and receives various data to and from other devices connected to the network NW in accordance with a predetermined communication method. The communication unit 144 transmits and receives various information to and from the server device 200 via the network NW. For example, the communication unit 144 transmits operation information acquired by the host system from the input unit 141 to the server device 200. The communication unit 144 receives display data representing image information as received information from the server device 200, and outputs the received information to the host system. The received information from the server device 200 may include annotation information and store information.

[0025] Next, a system configuration example of the server device 200 according to this embodiment will be described. Fig. 3 is a schematic block diagram showing a system configuration example of the server device 200 according to this embodiment. The server device 200 includes a communication unit 210, a storage unit 220, and a control unit 230. The communication unit 210 is connected to the network NW and transmits and receives various data to and from other devices connected to the network NW. The communication unit 210 transmits and receives various information to and from the user terminal device 40 via the network NW. For example, the communication unit 210 receives operation information from the user terminal device 40 and outputs the received operation information to the control unit 230. The communication unit 210 transmits image information shown in display data representing image information input from the control unit 230 as transmission information to the user terminal device 40. The transmission information may include any of annotation information, store information, and the like.

[0026] The storage unit 220 includes an auxiliary storage device that stores various data in a readable and writable manner. The storage unit 220 stores, for example, panoramic image data, map information, area information, annotation information, user information, etc. In the present application, the panoramic image data, map information, area information, and annotation information may be collectively referred to as virtual space information. The storage unit 220 may include, for example, any one or a combination of storage media such as a hard disk drive (HDD: Hard-disk Drive) or a solid-state storage device (SSD: Solid State Drive), as well as a flash memory, an EEPROM (Electrically Erasable Programmable Read-only Memory), etc.

[0027] The control unit 230 performs processing for providing the overall functions of the server device 200 and controls the processing. The functions of the control unit 230 are provided by a CPU (not shown) of the server device 200 executing a predetermined application and working in cooperation with a memory (not shown) and other devices. That is, the control unit 230 provides the functions as the main computer system of the server device 200. The control unit 230 has the functions of an operation acquisition unit 231, a panoramic image acquisition unit 232, a virtual space display unit 233, a region coordinate acquisition unit 234, a region calculation unit 235, a region generation unit 236, an annotation generation unit 237, and an annotation information acquisition unit 238.

[0028] The control unit 230 can be considered to include a spatial processing unit, a region setting unit, and an annotation management unit as its functional configuration (not shown). The spatial processing unit provides viewpoint images to the user terminal device 40 and acquires operation information from the user terminal device 40. The spatial processing unit includes the above-mentioned operation acquisition unit 231, panoramic image acquisition unit 232, and virtual space display unit 233. The region setting unit sets an annotation region based on the operation information. The region setting unit includes a region coordinate display unit 234, a region calculation unit 235, and a region generation unit 236. The annotation management unit is involved in editing and providing annotations. The annotation management unit includes an annotation generation unit 237 and an annotation information acquisition unit 238.

[0029] The operation acquisition unit 231 acquires operation information received from the user terminal device 40 using the communication unit 210. As the operation information, for example, user information at the time of login, a point ID (Identifier) ​​indicating a position in the virtual space, coordinates related to the setting of an annotation, detailed information, and the like are acquired.

[0030] The panoramic image acquisition unit 232 acquires panoramic image data indicating a panoramic image observed from a position corresponding to the updated point ID from the storage unit 220 every time the point ID acquired from the operation acquisition unit 231 is updated. The panoramic image acquisition unit 232 outputs the acquired panoramic image data to the virtual space display unit 233. The panoramic image represents a scene of the entire upper half plane in the virtual space that can be observed from a viewpoint indicated by the point ID. As the virtual space, a three-dimensional space artificially constructed by the server device 200 can be applied. The virtual space is, for example, a space in a store. The virtual space may be generated based on information about the real space, or may include a part of the real space. The panoramic image data corresponds to one form of spatial information.

[0031] The panoramic image acquisition unit 232 preliminarily sets a plurality of viewpoints in the virtual space, collates the positions of the point IDs indicating the individual viewpoints with the map information, and identifies the panoramic image data corresponding to the positions. The map information includes information indicating the distribution of objects, facilities, and passages in the virtual space. The viewpoints may be set within the areas that occupy the passages. The map information indicates the positions of the individual viewpoints, the point IDs, and the addresses of the panoramic image data relating to the viewpoints in association with each other. The map information may be associated with product information indicating the position, price, product name, manufacturer name, availability of stock, and the like of each product using the position. In this embodiment, each viewpoint is set so that a part of the panoramic image of the viewpoint overlaps a part of the panoramic image corresponding to an adjacent viewpoint. As data indicating spatial information, panoramic image data indicating the panoramic image that can be observed from each viewpoint is preliminarily acquired.

[0032] The virtual space display unit 233 controls the display of the virtual space shown in the omnidirectional image data acquired from the omnidirectional image acquisition unit 232 in response to the operation information acquired from the operation acquisition unit 231. For example, the virtual space display unit 233 extracts a portion included in a predetermined field of view based on the line of sight direction indicated by the operation information from the omnidirectional image shown in the omnidirectional image data as a viewpoint image. The virtual space display unit 233 transmits display data showing the extracted viewpoint image to the user terminal device 40 using the communication unit 210. The control unit of the user terminal device 40 causes the display unit 142 to display the viewpoint image shown in the display data received from the server device 200. The viewpoint image shows a scene in the virtual space. The virtual space display unit 233 may transmit store information related to the store shown in the omnidirectional image to the user terminal device 40 in association with the display data.

[0033] The area coordinate acquisition unit 234 acquires coordinates indicated by operation information acquired from the operation acquisition unit 231 as area coordinates related to the annotation area. FIG. 4 illustrates an example of a viewpoint image VL observed from a viewpoint VP. The viewpoint image VL represents an image of an object projected onto a plane perpendicular to a straight line starting from the viewpoint VP toward the line of sight direction OA in a virtual space. The viewpoint image VL is represented in a rectangular area defined by the viewing angles in the horizontal and vertical directions centered on the line of sight direction OA. PI illustrates an example of coordinates indicated in the viewpoint image VL.

[0034] The area coordinate acquisition unit 234 acquires coordinates representing a common image appearing in each of the viewpoint images from two or more different viewpoints. The area coordinate acquisition unit 234 notifies the area calculation unit 235 of area coordinate information indicating the acquired coordinates. In the following description, the viewpoints related to the individual viewpoint images are called the first viewpoint, the second viewpoint, etc., the field of view image observed from the first viewpoint is called the first image, the field of view image observed from the second viewpoint is called the second image, etc. Furthermore, the coordinates indicated on the screen of the first image are called the first coordinates, the coordinates indicated on the screen of the second image are called the second coordinates, etc.

[0035] Returning to FIG. 3, the region calculation unit 235 calculates coordinates related to the annotation region based on the region coordinate information acquired from the region coordinate acquisition unit 234. More specifically, the region calculation unit 235 specifies a first straight line starting from a first viewpoint in a three-dimensional space forming a virtual space and heading in a first direction corresponding to the first coordinate. The region calculation unit 235 specifies a second straight line starting from a second viewpoint and heading in a second direction corresponding to the second coordinate. The region calculation unit 235 determines the coordinates of the intersection between the first straight line and the second straight line. The region calculation unit 235 determines, as the annotation region, a region having a predetermined size and shape including the determined intersection as a reference point. The shape of the annotation region may be defined within a certain plane in the three-dimensional space. The region calculation unit 235 notifies the region generation unit 236 of the determined annotation region. The region calculation unit 235 stores the determined annotation region in the storage unit 220 as region coordinate information indicating coordinates representing the position of the determined annotation region. The coordinates of the determined intersection may be applied as the representative coordinates. Alternatively, the coordinates of each vertex constituting a polygon indicating the outer edge of the annotation region may be applied as the coordinates representing the position of the annotation region.

[0036] The area calculation unit 235 may determine two intersections according to the above method, and determine a rectangular area having a width equal to the line segment terminated by the two intersections as the annotation area. For example, the area calculation unit 235 determines a rectangular area having the line segment as the upper side and each intersection as a vertex. However, when determining the two intersections, the area coordinate acquisition unit 234 acquires two different first coordinates from the first image. The multiple first coordinates may be distinguished by being called a first type first coordinate, a second type first coordinate, etc.

[0037] In the example of Fig. 5, in the virtual space, a first type first straight line RY11 extending from a viewpoint VP01 to a first type first coordinate and a first type second straight line RY12 extending from a viewpoint VP02 to a first type second coordinate are used to determine an intersection point AP01 between them. A second type first straight line RY21 extending from a viewpoint VP01 to a second type first coordinate and a second type second straight line RY22 extending from a viewpoint VP02 to a second type second coordinate are used to determine an intersection point AP02 between them. In the examples of Figures 6 and 7, when intersection points AP03 and AP04 are determined in virtual space, a rectangular filled area having a width equal to the length of the line segment that connects intersection points AP03 and AP04 at both ends is determined as the annotation area ANA.

[0038] A feature point having a predetermined spatial characteristic in the viewpoint image may be selected. For example, an intersection or a turning point of two or more edges having a sharp spatial change in shade or color tone may be selected as a feature point. Such feature points tend to appear at the end points, vertices, and the like of an object. In the example of FIG. 5, coordinates indicating one vertex AP01 of a bulletin board PN covering a part of a back panel BP arranged at the back of a display stand DT installed in a virtual space are specified as the first type first coordinate and the first type second coordinate. Coordinates indicating another vertex AP02 of the bulletin board PN are specified as the second type first coordinate and the second type second coordinate.

[0039] In order to encourage the selection of a common feature point between the first image and the second image, the area coordinate acquisition unit 234 may acquire operation information indicating the two first coordinates, and then cause the virtual space display unit 233 to transmit display data indicating the second image and guidance information for guiding the user to specify the same position on the second image as the first coordinates to the user terminal device 40. The control unit of the user terminal device 40 causes the display unit 142 to display the second image and the guidance information in association with each other. As the guidance information, for example, a message such as "Please select the same two points again" may be used.

[0040] Furthermore, the area coordinate acquisition unit 234 or the control unit of the user terminal device 40 may execute a known feature point extraction process on the first image and the second image to identify the positions of the feature points. The control unit of the user terminal device 40 causes the display unit 142 to display a display screen obtained by superimposing a predetermined symbol on the identified positions. Then, as operation information indicating a position on the image from the area coordinate acquisition unit 234 or the control unit of the user terminal device 40 or the input unit 141, operation information indicating the position of any of the feature points may be adopted, and operation information indicating other positions may be discarded. In this way, the positions of the specified first coordinates and second coordinates are controlled to be any of the feature points.

[0041] The area generating unit 236 is notified of the annotation area by the area calculating unit 235. The area generating unit 236 generates a figure occupying the annotation area in three-dimensional space as an area panel, and superimposes the generated area panel on the panoramic image related to the viewpoint specified at that time. The area generating unit 236 extracts a viewpoint image that appears in the field of view based on the line of sight specified at that time from the panoramic image on which the area panel is superimposed, and transmits display data indicating the extracted viewpoint image to the user terminal device 40. The control unit of the user terminal device 40 outputs the display data received from the server device 200 to the display unit 142. The display unit 142 presents a viewpoint image on which the area panel is superimposed. By visually recognizing the viewpoint image representing the area panel, the user can grasp the annotation area having a predetermined range based on the coordinates specified on the viewpoint image.

[0042] The annotation generating unit 237 generates an annotation based on operation information acquired from the operation acquiring unit 231 when a viewpoint image including an area panel is provided. An annotation refers to expressing or expressing guidance information related to an associated object or area. An annotation is, for example, a label, a tag, a sticker, or a display simulating them. An annotation is configured to include one or more characters and / or an image as elements. An annotation may include any of numbers, symbols, and signs as characters. An annotation may include any of figures and patterns as images. An image may be either a still image or a moving image.

[0043] The annotation generating unit 237 determines, as an allocation area, a two-dimensional area that is a part of the annotation area based on the coordinates indicated by the operation information for the annotation to be generated. The annotation generating unit 237 composes an annotation including element information indicated by the operation information, and superimposes the generated annotation in the determined allocation area on the panoramic image. The annotation generating unit 237 assigns the composed annotation to the determined allocation area. Here, the annotation generating unit 237 superimposes the annotation in the allocation area of ​​the panoramic image. FIG. 8 illustrates an example of an allocation area AL assigned in the annotation area ANA. FIG. 9 illustrates an example of a character string "bargain" as an annotation arranged in the allocation area AL.

[0044] The annotation generating unit 237 extracts a viewpoint image appearing within a field of view based on a line of sight direction instructed at that time from the panoramic image on which the annotation is superimposed, and transmits display data indicating the extracted viewpoint image to the user terminal device 40. The control unit of the user terminal device 40 outputs the display data received from the server device 200 to the display unit 142. The display unit 142 presents the viewpoint image on which the generated annotation is superimposed. The user can check the generated annotation by visually checking the presented viewpoint image. The annotation generating unit 237 stores annotation information including element information indicating an element of the generated annotation and position information indicating an assigned area in the storage unit 220. Note that the annotation element is not limited to a character, and an image may be used.

[0045] For example, the annotation generating unit 237 may transmit display data showing an annotation editing screen to the user terminal device 40 in response to an annotation editing instruction by the operation information, and cause the display unit 142 to display the annotation editing screen. By displaying the annotation editing screen, acquisition of a character string and image data constituting an annotation is prompted. FIG. 10 shows an example of the annotation editing screen. The annotation editing screen shown in the figure has an input field for a character string (text), a selection field for image data, and an update button. The input field is an area for specifying a character string by operation information. By pressing the update button, the specified character string is transmitted to the annotation generating unit 237. The image data is an area for specifying the location of a data file stored in the storage unit 143 of the user terminal device 40 by operation information. By pressing the update button, the image data stored in the specified data file is transmitted to the annotation generating unit 237. An end button (not shown) may be further arranged on the annotation editing screen. The end button is a screen component for instructing the user to end the annotation setting by pressing the end button.

[0046] The annotation information acquisition unit 238 acquires annotation information to be superimposed on the panoramic image designated at that time from the storage unit 220. For example, the annotation information acquisition unit 238 reads annotation information including element information and position information related to an annotation including a part or the whole of an assigned area in a range that is visible from a viewpoint at that time from the storage unit 220. The annotation information acquisition unit 238 superimposes an annotation designated by the element information on the assigned area designated by the read position information on the panoramic image. The annotation generation unit 237 extracts a viewpoint image based on a line of sight direction designated from the panoramic image on which the annotation is superimposed, and transmits display data indicating the extracted viewpoint image to the user terminal device 40. The control unit of the user terminal device 40 outputs the display data received from the server device 200 to the display unit 142. The display unit 142 presents the viewpoint image on which the annotation is superimposed. Note that a plurality of annotations may be displayed in one annotation area or viewpoint image. In the example of FIG. 11, two annotations AN01 and AN02 are set in one annotation region ANA.

[0047] The annotation information acquisition unit 238 may superimpose an annotation area including an assigned area of ​​the acquired annotation information on the panoramic image. The superimposed panoramic image is reflected on the viewpoint image. At this time, the annotation information acquisition unit 238 refers to the storage unit 220, specifies area coordinate information indicating an annotation area including, as a part or all, an assigned area indicated by the acquired annotation information, and reads out the specified area coordinate information. The area generation unit 236 specifies an annotation area having a predetermined size and shape based on the read out area coordinate information. The annotation information acquisition unit 238 generates an area panel occupying the specified annotation area, and superimposes the generated area panel on the panoramic image related to the viewpoint indicated at that time. Thus, an image of a panel indicating the annotation area is displayed in the viewpoint image, and individual annotations are arranged on the panel.

[0048] The area generating unit 236 and the annotation information acquiring unit 238 may superimpose a predetermined symbol (hereinafter, may be referred to as "area coordinate symbol") at a position in three-dimensional space indicated by the area coordinates. In the viewpoint image, the area coordinate symbol appears at a position corresponding to the direction from the viewpoint indicated by the operation information to the position indicated by the area coordinates. The position corresponds to the first coordinates indicated on the first image or the second coordinates indicated on the second image by the user's operation. The number of area coordinate symbols appearing in one viewpoint image is not limited to one, and may be two or more. In the examples of Figs. 6 to 9, two circles are shown as area coordinate symbols.

[0049] Next, an example of annotation information processing according to this embodiment will be described below with reference to a flowchart of FIG 12 exemplifying the annotation information processing according to this embodiment. (Step S102) The control unit 230 of the server device 200 accepts an access request from the user terminal device 40. The host system of the user terminal device 40, for example, starts an application for a store related to a virtual space, and transmits an access request to the server device 200 immediately after the application starts. (Step S104) The panoramic image acquisition unit 232 acquires virtual space information from the storage unit 220. As the virtual space information, for example, panoramic image data corresponding to a predetermined viewpoint is read.

[0050] (Step S106) Virtual space display unit 233 extracts an image within a predetermined field of view from the panoramic image data as a viewpoint image, and transmits display data indicating the extracted viewpoint image to user terminal device 40. The control unit of user terminal device 40 displays, on display unit 142, the viewpoint image indicated in the display data received from server device 200 as a virtual space image. (Step S108) The virtual space display unit 233 repeats the process of transmitting display data indicating a viewpoint image extracted from the omnidirectional image data to the user terminal device 40 based on the viewpoint and line of sight direction specified by the operation information received from the user terminal device 40. The control unit of the user terminal device 40 outputs the display data received from the server device 200 to the display unit 142 every time one or both of the specified viewpoint and line of sight direction are updated. The display unit 142 displays a viewpoint image indicating a scene of the virtual space corresponding to the updated viewpoint and line of sight direction. This prompts the user to move the viewpoint specified by the operation information to a point where a region of interest in the virtual space in which the user is interested appears in the first image, which is the viewpoint image.

[0051] (Step S110) The area coordinate acquiring unit 234 acquires first coordinates related to the annotation area designated by the operation information received from the user terminal device 40. (Step S112) The area coordinate acquisition unit 234 changes the viewpoint designated at that time to a viewpoint (point) adjacent to the viewpoint. The area coordinate acquisition unit 234 selects, as the adjacent viewpoint, a viewpoint from which the image of the object designated by the first coordinates can be observed. The virtual space display unit 233 reads the omnidirectional image data corresponding to the changed viewpoint from the storage unit 220, and extracts an image within the field of view including the image of the object indicated by the first coordinates from the read omnidirectional image data as a new viewpoint image. The virtual space display unit 233 transmits display data showing the extracted viewpoint image as a second image to the user terminal device 40. The control unit of the user terminal device 40 displays the second image shown in the display data newly received from the server device 200 on the display unit 142 as a virtual space image.

[0052] (Step S114) The area coordinate acquiring unit 234 acquires second coordinates related to the annotation area designated by the operation information received from the user terminal device 40. (Step S116) The area calculation unit 235 determines the coordinates of the intersection between a first line having the first viewpoint as an origin and corresponding to the first coordinates, and a second line having the second viewpoint as an origin and corresponding to the second coordinates. The area calculation unit 235 sets an annotation area having a predetermined range based on the determined intersection. The area generation unit 236 generates an area panel representing the set annotation area, and superimposes it on the panoramic image in the virtual space. The area generating unit 236 extracts a viewpoint image from the panoramic image on which the area panel is superimposed, based on the line of sight direction indicated at that time, and transmits display data indicating the extracted viewpoint image to the user terminal device 40. The control unit of the user terminal device 40 outputs the display data received from the server device 200 to the display unit 142, causing the new viewpoint image to be displayed.

[0053] (Step S118) The annotation generating unit 237 transmits to the user terminal device 40 display data showing the annotation editing screen. The annotation generating unit 237 generates an annotation according to the operation information received from the user terminal device 40. The annotation generating unit 237 determines a part of the annotation area specified by the operation information as an assigned area, and adds the generated annotation to the assigned area. The annotation generating unit 237 extracts a viewpoint image from the full-sky image to which the annotation has been added, based on the line-of-sight direction specified at that time, and transmits display data indicating the extracted viewpoint image to the user terminal device 40. The control unit of the user terminal device 40 outputs the display data received from the server device 200 to the display unit 142, and causes the new viewpoint image to be displayed. (Step S120) The area calculation unit 235 stores area coordinate information indicating the coordinates of the determined intersection in the storage unit 220. The annotation generation unit 237 stores annotation information indicating the generated annotation in the storage unit 220. Thereafter, the process in FIG. 12 ends.

[0054] Next, an example of annotation setting processing according to the present embodiment will be described. Fig. 13 is a flowchart showing another example of annotation information processing according to the present embodiment. Fig. 13 shows an example in which an annotation area is set from two intersections in a virtual store configured by the server device 200, and one or more annotations are set in the set annotation area.

[0055] (Step S202) The control unit 230 of the server device 200 performs a login process in response to an access request from the user terminal device 40. In the login process, the control unit 230 transmits display data showing a login screen to the user terminal device 40 as a response to the access request. The control unit 230 acquires a user ID and authentication information from the user terminal device 40 that is the source of the access request, refers to the user information, and determines whether the user authentication has been successful or not based on whether the authentication information corresponding to the acquired user ID matches the acquired authentication information. The storage unit 220 stores in advance user information showing the user ID and the authentication information for each user. If the user authentication is successful, the control unit 230 proceeds to the process of step S204, and if the user authentication is unsuccessful, the control unit 230 interrupts the process of FIG. 13.

[0056] (Step S204) The virtual space display unit 233 transmits an updated viewpoint image to the user terminal device 40 in response to the update of the viewpoint or line of sight direction instructed by the operation information. The process of this step is similar to the process of step S108, and therefore the description thereof is incorporated herein. The process of this step allows the user to virtually experience movement in the virtual space.

[0057] (Step S206) The area coordinate acquisition unit 234 executes the process of step S110, and acquires two first coordinates as a first type first coordinate and a second type first coordinate based on the operation information while the first image, which is the latest viewpoint image, is provided. Thereafter, the area coordinate acquisition unit 234 executes the process of step S112, and changes the viewpoint at that time to an adjacent viewpoint. The virtual space display unit 233 acquires a second image based on the changed viewpoint, and transmits display data indicating the second image to the user terminal device 40. The area coordinate acquisition unit 234 executes the process of step S112, and acquires two second coordinates as a first type second coordinate and a second type second coordinate based on the operation information while the second image is provided.

[0058] (Step S208) The area calculation unit 235 determines the coordinates of a first intersection point, which is an intersection point between a first type first straight line having the first viewpoint as a starting point and corresponding to the first type first coordinate, and a first type second straight line having the second viewpoint as a starting point and corresponding to the first type second coordinate. The area calculation unit 235 also determines the coordinates of a second intersection point, which is an intersection point between a second type first straight line having the first viewpoint as a starting point and corresponding to the second type first coordinate, and a second type second straight line having the second viewpoint as a starting point and corresponding to the second type second coordinate. A rectangular area having a horizontal width and a predetermined height in the vertical direction, with the first intersection point and the second intersection point as both ends, and extending on a two-dimensional plane, is determined as the annotation area.

[0059] The area generating unit 236 generates an area panel representing the set annotation area, and superimposes it on the panoramic image in the virtual space. The area generating unit 236 extracts a viewpoint image from the panoramic image on which the area panel is superimposed, based on the line of sight direction indicated at that time, and transmits display data indicating the extracted viewpoint image to the user terminal device 40. The area calculation unit 235 stores area coordinate information indicating the coordinates of the first intersection and the second intersection in the storage unit 220. In addition, the annotation generation unit 237 transmits to the user terminal device 40 display data indicating the annotation editing screen.

[0060] (Step S210) The annotation generation unit 237 generates an annotation according to the operation information received from the user terminal device 40, determines a part of the annotation area as an assigned area, and adds the generated annotation to the assigned area. The annotation generation unit 237 extracts a viewpoint image from the full-sky image to which the annotation has been added, based on the line-of-sight direction indicated at that time, and transmits display data indicating the extracted viewpoint image to the user terminal device 40.

[0061] (Step S212) The annotation generating unit 237 stores in the storage unit 220 annotation information indicating the generated annotation. (Step S214) The annotation generation unit 237 judges whether or not to add a new annotation based on operation information from the user terminal device 40. For example, the annotation generation unit 237 judges to add a new annotation when an instruction to end annotation setting is not given and other element information is given. The annotation generation unit 237 judges not to add an annotation when an instruction to end annotation setting is given. When it is determined to add an annotation (step S214 YES), the process returns to step S210. When it is determined not to add an annotation (step S214 NO), the process of FIG. 13 ends.

[0062] <Second embodiment> Next, the second embodiment will be described, mainly focusing on the differences from the first embodiment. For the points in common with the first embodiment, the description of the first embodiment will be cited. 14 is a schematic block diagram showing an example of the system configuration of the server device 200 according to this embodiment. The server device 200 further includes a region editing unit 239 in the control unit 230. The region editing unit 239 belongs to the above-mentioned region setting unit.

[0063] The area editing unit 239 adjusts the position and shape of the annotation area based on the operation information acquired from the operation acquisition unit 231 . For example, when an annotation region to be edited is specified by operation information, the region editing unit 239 transmits display data indicating a region editing screen for editing the annotation region to the user terminal device 40. The control unit of the user terminal device 40 causes the display unit 142 to display the region editing screen. The region editing screen has input fields for inputting the vertical and horizontal enlargement rates of the annotation region by operation (not shown). The vertical enlargement rate indicates the ratio of the height after the change based on the vertical height of the annotation region when it is initially generated. The horizontal ratio indicates the ratio of the width after the change based on the horizontal width of the annotation region when it is initially generated. The control unit of the user terminal device 40 transmits operation information indicating the input vertical enlargement rate and horizontal enlargement rate to the server device 200.

[0064] The area editing unit 239 specifies the vertical magnification ratio and the horizontal magnification ratio notified by the operation information received from the user terminal device 40. The area editing unit 239 adjusts the horizontal coordinates of each vertex that defines the outer edge of the annotation area based on the notified horizontal ratio and a predetermined horizontal reference coordinate. The area editing unit 239 adjusts the vertical coordinates of each vertex based on the set vertical ratio and a predetermined vertical reference coordinate. The reference coordinates are coordinates of a reference point that does not move regardless of enlargement or reduction. As the reference point, for example, any one of the vertices or the center of gravity of the annotation area, or the area coordinates used to set the annotation area may be used.

[0065] The area editing unit 239 updates the area coordinate information stored in the storage unit 220 to area coordinate information indicating the coordinates of each vertex after the adjustment. The area editing unit 239 also generates an area panel indicating the annotation area defined by each vertex after the adjustment, and superimposes the generated area panel on the panoramic image related to the viewpoint indicated at that time. The area editing unit 239 extracts a viewpoint image that appears within the field of view based on the line of sight direction that is currently specified from the panoramic image on which the area panel is superimposed, and transmits display data indicating the extracted viewpoint image to the user terminal device 40. The control unit of the user terminal device 40 causes the display unit 142 to display a new viewpoint image in accordance with the display data received from the server device 200. The displayed viewpoint image shows an area panel indicating the annotation area after the adjustment.

[0066] The example of Fig. 15 shows the annotation area ANA adjusted when the horizontal magnification rate is set to 180% and the vertical magnification rate is set to 100%. However, the upper right end of the annotation area ANA is preset as the reference point. The annotation area ANA is expanded to the left so that the horizontal width becomes 1.8 times the original width while fixing the right side and maintaining the vertical height. That is, the coordinates of both ends of the right side and the vertical coordinates of both ends of the left side are maintained, and the displacements of the horizontal coordinates of both ends of the left side and the horizontal coordinates of both ends of the right side are each 1.8 times the original displacement.

[0067] The example in Fig. 16 shows the annotation area ANA adjusted when the horizontal magnification is set to 100% and the vertical magnification is set to 230%. The annotation area ANA is expanded downward so that the vertical height becomes 12.3 times the original height while keeping the horizontal width by fixing the top side. That is, the coordinates of both ends of the top side and the horizontal coordinates of both ends of the bottom side are maintained, and the displacements of the vertical coordinates of both ends of the bottom side and the vertical coordinates of both ends of the bottom side are each 2.3 times the original displacement.

[0068] Next, an example of the annotation setting process according to this embodiment will be described. Fig. 17 is a flowchart illustrating the annotation setting process according to this embodiment. The process in Fig. 17 includes step S222 in addition to the process in Fig. 13. In the process in Fig. 17, after the process in step S208 ends, the process proceeds to step S222.

[0069] (Step S222) The area editing unit 239 adjusts the shape of the annotation area generated in step S208 based on the operation information acquired from the operation acquisition unit 231. The area editing unit 239 transmits display data showing an area editing screen for editing the annotation area to the user terminal device 40. The control unit of the user terminal device 40 outputs the display data received from the server device 200 to the display unit 142, causing the area editing screen to be displayed.

[0070] The area editing unit 239 specifies the vertical magnification ratio and the horizontal magnification ratio notified by the operation information received from the user terminal device 40. The area editing unit 239 adjusts the horizontal coordinate of each vertex that defines the outer edge of the annotation area based on the notified horizontal ratio and predetermined horizontal reference coordinate. The area editing unit 239 adjusts the vertical coordinate of each vertex based on the set vertical ratio and predetermined vertical reference coordinate. The area editing unit 239 updates the original area coordinate information to area coordinate information indicating the coordinates of each vertex after the adjustment. The area editing unit 239 generates an area panel indicating the annotation area defined by each vertex after the adjustment, and superimposes the generated area panel on the panoramic image related to the viewpoint indicated at that time. Then, the process proceeds to step S210.

[0071] <Third embodiment> Next, the third embodiment will be described, mainly focusing on the differences from the first embodiment. For the points in common with the first embodiment, the description of the first embodiment will be cited. In the first embodiment, an example is shown of a case where, when the area coordinate acquisition unit 234 provides display data indicating a first image to the user terminal device 40, it acquires two first coordinates based on operation information received from the user terminal device 40, and, when the area coordinate acquisition unit 234 provides display data indicating a second image to the user terminal device 40, it acquires two second coordinates based on operation information received from the user terminal device 40.

[0072] In contrast, when the area coordinate acquisition unit 234 according to this embodiment provides display data indicating a first image to the user terminal device 40, it acquires N (N is an integer equal to or greater than 3) different first coordinates based on operation information received from the user terminal device 40, and when it provides display data indicating a second image to the user terminal device 40, it acquires N different second coordinates based on the operation information received from the user terminal device 40. The first type second coordinate to the Nth type second coordinate in the second image are parts having characteristics in common or similar to the first type first coordinate to the Nth type first coordinate in the first image, respectively.

[0073] The area calculation unit 235 determines the coordinates of the first intersection point to the Nth intersection point, which are the intersection points between the first kind first straight line to the Nth kind first straight line, which have the first viewpoint as the starting point and correspond to the first kind first coordinate to the Nth kind first coordinate, respectively, and the first kind second straight line to the Nth kind second straight line, which have the second viewpoint as the starting point and correspond to the first kind second coordinate to the Nth kind second coordinate, respectively. Then, the area calculation unit 235 determines a polygonal (i.e., N-sided) area having vertices at the first intersection point to the Nth intersection point, respectively, as the annotation area. The area calculation unit 235 determines the coordinates of the N intersection points as area coordinates, and stores them in the storage unit 220 as area coordinate information indicating the determined area coordinates.

[0074] 18 illustrates an example of a viewpoint image showing annotation area ANA generated when N is 4. The filled-in portion represents the area panel showing annotation area ANA. This annotation area ANA is a rectangular area having four vertices AP05, AP06, AP07, and AP08. In this manner, an annotation area having an arbitrary shape is set in the virtual space by the user specifying N vertices specified by the operation using different viewpoint images.

[0075] The server device 200 according to the present embodiment may include a region editing unit 239. The region editing unit 239 can update the shape and thus the position of the annotation region by moving each vertex of the polygon as an adjustment point based on operation information. When display data showing a viewpoint image is provided to the user terminal device 40, the area editing unit 239 specifies an adjustment point to be adjusted and the coordinates on a new viewpoint image related to the adjustment point based on operation information. The area editing unit 239 can, for example, detect a drag operation, determine the start point of a path obtained by the drag operation or the closest point on the outer edge as the adjustment point, and specify the coordinates of the end point of the path as the coordinates of the new adjustment point. A drag operation is an operation in which a position on the display screen is continuously moved for a certain period of time or more while being specified.

[0076] The area editing unit 239 can determine, in the virtual space, coordinates in a plane including the original annotation area, as new area coordinates, corresponding to the specified coordinates based on the viewpoint and field of view at that time. The area editing unit 239 can determine a new annotation area based on the new area coordinates. When determining an annotation area based on a plurality of area coordinates, the area editing unit 239 can determine a new annotation area according to the above-mentioned method based on one new area coordinate and the remaining area coordinates. The area editing unit 239 updates the original area coordinate information to area coordinate information indicating the new area coordinates, and stores the updated area coordinate information in the storage unit 220.

[0077] <Modification> The above embodiment may be modified as follows. In the above description, a case where a panoramic image for each viewpoint is used as an example of spatial information representing the distribution of objects arranged in a three-dimensional space is taken as an example, but is not limited to this. The space related to the spatial information is not limited to a virtual space constituting a virtual reality (VR), and may be a real space or an augmented reality (AR) space in which a real space and a virtual space are combined. The spatial information representing the real space may include image data showing an image actually taken in a specified facility. The spatial information may be represented using a still image or a moving image. Furthermore, the facilities to which the above-described embodiments are applicable are not limited to commercial facilities such as supermarkets, department stores, and shopping centers, but may also be art galleries, museums, aquariums, zoos, and other exhibition venues.

[0078] In the above description, the spatial information for each viewpoint is mainly a panoramic image showing a scene on an upper half plane centered on the viewpoint, but is not limited to this. The spatial information only needs to include information showing a scene within a field of view that can be specified by operation information for each viewpoint. For example, a surrounding image that covers the periphery of the viewpoint and has a region within a predetermined range from the apex of the upper half plane missing may be used instead of the panoramic image. Furthermore, the spatial information does not need to include an image showing a space that is not the subject of presentation, for example, a scene outside a store.

[0079] In the above description, the spatial processing unit acquires a viewpoint image corresponding to the line of sight from a panoramic image corresponding to a viewpoint designated by operation information among predetermined viewpoints, but the present invention is not limited to this. The spatial processing unit may use map information as spatial information representing the distribution of objects, and synthesize a viewpoint image appearing in a field of view corresponding to the viewpoint and line of sight designated by operation information by executing a known computer graphics (CG) process. Information indicating the position and form of each object in a three-dimensional space may be set in advance in the map information.

[0080] In the first embodiment, the region setting unit sets a rectangular region having both ends of two intersection points as one side as the annotation region, but this is not limited to the above. The region setting unit may set a region forming a figure having the distance between two intersection points as a characteristic size as the annotation region. The characteristic size refers to a size that characterizes the region. Such a figure may be, for example, a circle having the distance between the intersection points as its diameter, or a triangle having the distance between the intersection points as its base or top side and having a predetermined height.

[0081] In the third embodiment, the region setting unit sets a quadrangular region having four intersections as vertices as the annotation region, but the present invention is not limited to this. The region setting unit may set a planar figure having three or five or more vertices as the annotation region.

[0082] The area editing unit 239 according to the second embodiment may adjust the shape of the annotation area by updating the coordinates of an adjustment point on the annotation area, similarly to the area editing unit 239 according to the third embodiment. As the adjustment point, any of the area coordinates used for setting the annotation area, a point or a vertex on a side forming the outer edge of the annotation area, etc. may be used.

[0083] Two or more annotation regions may be set at different positions in the three-dimensional space. For example, in the processes of Fig. 13 and Fig. 17, the processes of steps S204 to S214 are repeated two or more times to set two or more annotation regions. One or more annotations can be set in each annotation region.

[0084] As described above, the information processing device (e.g., the server device 200) according to this embodiment includes a spatial processing unit (e.g., the operation acquisition unit 231, the panoramic image acquisition unit 232, the virtual space display unit 233) that provides a first image that appears in a first field of view from a first viewpoint in three-dimensional space based on spatial information (e.g., omnidirectional image data) that represents the distribution of objects arranged in three-dimensional space (e.g., virtual space), acquires first coordinates that are coordinates within the first image, provides a second image that represents an image common to the first field of view and appears in a second field of view from a second viewpoint in three-dimensional space, and acquires second coordinates that are coordinates within the second image, and in the three-dimensional space: The system includes an area setting unit (e.g., area coordinate display unit 234, area calculation unit 235, area generation unit 236) that identifies a first straight line oriented in a first direction corresponding to a first coordinate originating from a first viewpoint, identifies a second straight line oriented in a second direction corresponding to the second coordinate originating from a second viewpoint, determines an intersection point between the first straight line and the second straight line, and sets an annotation area having a predetermined range including the intersection point in three-dimensional space, and an annotation management unit (e.g., annotation generation unit 237, annotation information acquisition unit 238) that acquires annotation information and assigns the annotation information to a portion of the annotation area. According to this configuration, coordinates are acquired in each of two viewpoint images having different viewpoints that show an image of a common object in three-dimensional space, and an annotation area is set at a position based on an intersection point of straight lines based on the acquired coordinates. An annotation is assigned to a part of the set annotation area. The annotation area can be easily set in three-dimensional space while visually recognizing the two-dimensional viewpoint images. By assigning an annotation to a part of the annotation area, an object or area corresponding to the annotation information in three-dimensional space can be clarified.

[0085] The region setting unit may determine at least two intersections located at different positions in a three-dimensional space, and may set the annotation region having a characteristic size equal to the distance between the two intersections. For example, the annotation region may be set to a rectangular region having a width as a characteristic size. According to this configuration, the distance between two intersections in a three-dimensional space is specified as a characteristic size, and an annotation region having the specified characteristic size is set. Therefore, by setting two intersections, it is possible to determine an annotation region having a desired size according to the distance between the intersections.

[0086] The region setting unit may set at least three intersections in a three-dimensional space, and set a polygonal region having the determined intersections as vertices as the annotation region. According to this configuration, by setting at least three intersections in a three-dimensional space, a polygonal area having a desired size and shape can be defined as the annotation area.

[0087] When a ratio to a reference value of the characteristic size (eg, the characteristic size of the annotation region when it is initially generated) is specified, the region setting unit may adjust the characteristic size of the annotation region based on the specified ratio. According to this configuration, the characteristic size of the annotation region is updated based on the ratio to a specified reference value, and therefore the shape of the annotation region is adjusted to have a desired characteristic size.

[0088] Although the present embodiment has been described above in detail with reference to the drawings, the specific configuration is not limited to the above-mentioned configurations, and includes designs within the scope of the present embodiment. The above-mentioned configurations can be arbitrarily combined, and some of them may be omitted. [Explanation of symbols]

[0089] 1...information processing system, 40 (40-1, 40-2)...user terminal device, 110...memory, 111...CPU, 120a, 120b...bus, 130...bridge, 140...interface, 141...input unit, 142...display unit, 143...storage unit, 144...communication unit, 200...server device, 210...communication unit, 220...storage unit, 230...control unit, 231...operation acquisition unit, 232...panoramic image acquisition unit, 233...virtual space display unit, 234...area coordinate acquisition unit, 235...area calculation unit, 236...area generation unit, 237...annotation generation unit, 238...annotation information acquisition unit, 239...area editing unit

Claims

1. Based on spatial information that represents the distribution of objects in three-dimensional space, providing a first image appearing in a first field of view from a first viewpoint in the three-dimensional space; Obtaining a first coordinate that is a coordinate within the first image; providing a second image representing a common image with the first view and appearing in a second view from a second viewpoint in the three-dimensional space; a spatial processing unit that acquires second coordinates that are coordinates within the second image; identifying a first straight line in the three-dimensional space that is oriented in a first direction corresponding to the first coordinates originating from the first viewpoint; Identifying a second straight line oriented in a second direction corresponding to the second coordinates originating from the second viewpoint; determining an intersection point between the first straight line and the second straight line; a region setting unit that sets an annotation region having a predetermined range including the intersection point in the three-dimensional space; an annotation management unit that acquires annotation information and assigns the annotation information to a part of the annotation area; Information processing device.

2. The region setting unit is determining at least two of the intersections located at different positions in the three-dimensional space; The distance between the two intersections is set as the annotation region having a characteristic size. The information processing device according to claim 1 .

3. The region setting unit is A rectangular area having a width equal to the distance between the two intersection points is set as the annotation area. The information processing device according to claim 2 .

4. The region setting unit is determining at least three intersections in the three-dimensional space; A polygonal region having the intersection points as vertices is set as the annotation region. The information processing device according to claim 1 .

5. The region setting unit is When a ratio of a characteristic size to a reference value is specified, the characteristic size of the annotation region is adjusted based on the ratio. The information processing device according to claim 1 .

6. An information processing apparatus comprising: the information processing apparatus according to claim 1; and one or more user terminals. Information processing system.

7. To your computer A program for causing the information processing device according to claim 1 to function.

8. An information processing method in an information processing device, The information processing device, Based on spatial information that represents the distribution of objects in three-dimensional space, providing a first image appearing in a first field of view from a first viewpoint in the three-dimensional space; Obtaining a first coordinate that is a coordinate within the first image; providing a second image representing a common image with the first view and appearing in a second view from a second viewpoint in the three-dimensional space; obtaining second coordinates that are coordinates within the second image; identifying a first straight line in the three-dimensional space that is oriented in a first direction corresponding to the first coordinates originating from the first viewpoint; Identifying a second straight line oriented in a second direction corresponding to the second coordinates originating from the second viewpoint; determining an intersection point between the first straight line and the second straight line; setting an annotation area having a predetermined range including the intersection point in the three-dimensional space; Annotation information is acquired, and the annotation information is assigned to a portion of the annotation area. Information processing methods.

Images