Guide image management device

Abstract

This guide image management device comprises: a management unit for managing one or more guide images, which are obtained by imaging a location in a real space that includes a virtual object virtually disposed in the real space, in association with the virtual object; and a communication control unit for causing a communication device to transmit one first guide image from among the one or more guide images to a user device and causing the communication device to receive, from the user device, a captured image generated due to the user device executing imaging. The management unit manages the captured image as a new guide image if the captured image satisfies a registration condition.

Description

[Technical Field]

[0001] The present invention relates to a guide image management device. [Background technology]

[0002] Patent Document 1 discloses a guide providing system for guiding a user in a virtual space. This guide providing system generates a scene graph. In the scene graph, multiple objects in the virtual space are represented as nodes. The scene graph describes the hierarchical interrelationships between the objects. The guide providing system determines whether the difference between the current scene graph and a past scene graph is equal to or greater than a certain level. If the difference is equal to or greater than the certain level, the guide providing system stores the current scene graph. The guide providing system provides the user with recommended places to visit and route information based on the scene graph and the user's visit history. [Prior art documents] [Patent documents]

[0003] [Patent Document 1] Japanese Patent Application Laid-Open No. 2009-251831 Summary of the Invention [Problem to be solved by the invention]

[0004] Conventional guide providing systems provide recommended places to visit and route information, but are unable to provide guide images that indicate the locations of virtual objects placed in virtual space. Conventional guide providing systems have a problem in that they are unable to provide new guide images to users, particularly when the environment of the location where the virtual object is placed changes. [Means for solving the problem]

[0005] The guide image management device of the present disclosure includes a management unit that manages one or more guide images obtained by capturing an image of a location in real space that includes the position of a virtual object virtually placed in the real space, in association with the virtual object, and a communication control unit that causes a communication device to send a first guide image of the one or more guide images to a user device and causes the communication device to receive from the user device an imaged image generated by the user device performing imaging, and the management unit manages the imaged image as a new guide image if the imaged image satisfies registration conditions. [Effects of the Invention]

[0006] According to the present disclosure, when the environment where a virtual object is placed changes, a new guide image indicating the location of the virtual object can be provided. [Brief explanation of the drawings]

[0007] [Figure 1] 1 is a diagram showing the overall configuration of an information processing system 1 according to an embodiment. [Figure 2A] FIG. 2 is an explanatory diagram showing an example of a mixed reality space visually recognized by a user U[k] through XR glasses 30-k. [Figure 2B] FIG. 2B is an explanatory diagram showing an example of a guide image G1 corresponding to the virtual object VO shown in FIG. 2A. [Figure 2C] FIG. 10 is an explanatory diagram showing an example of a guide image G2. [Figure 2D] FIG. 10 is an explanatory diagram showing an example of a virtual object image. [Figure 3] FIG. 10 is an explanatory diagram showing an example of an interface image Gi displayed on a tablet terminal when a developer places a virtual object. [Figure 4] A perspective view showing the appearance of XR Glasses 30-k. [Figure 5] Block diagram showing an example configuration of XR Glasses 30-k. [Figure 6] FIG. 2 is a block diagram showing an example of the configuration of an object management server 50. [Figure 7]FIG. 2 is an explanatory diagram showing an example of the data structure of a first database DB1. [Figure 8] FIG. 4 is an explanatory diagram showing an example of the data structure of a second database DB2. [Figure 9] 10 is a flowchart showing the contents of a process in which the object management server 50 transmits a guide image. [Figure 10] 10 is a flowchart showing the contents of a process in which the object management server 50 updates a guide image. [Figure 11] FIG. 4 is an explanatory diagram showing an example of a captured image. [Figure 12] FIG. 10 is an explanatory diagram showing an example of the contents stored in the second database DB2 after updating. [Figure 13] FIG. 2 is a block diagram showing an example of the configuration of a terminal device 20-k. DETAILED DESCRIPTION OF THE INVENTION

[0008] 1: Embodiment The information processing system 1 will be described below with reference to FIGS.

[0009] 1.1: Configuration of the embodiment 1.1.1: Overall structure FIG. 1 is a block diagram showing the overall configuration of an information processing system 1. As shown in FIG. 1, the information processing system 1 includes user devices 10-1, 10-2, ... 10-k, ... 10-j, a location management server 40 that manages the locations of the user devices, an object management server 50 that manages virtual objects, and a structure management server 60 that manages spatial structure data. j is an arbitrary integer equal to or greater than 1. k is an arbitrary integer equal to or greater than 1 and equal to or less than j. The user device 10-k includes a terminal device 20-k and XR glasses 30-k. In this embodiment, the user devices 10-1, 10-2, ... 10-k, ... 10-j have the same configuration as each other. Therefore, the terminal devices 20-1, 20-2, ... 20-k, ... 20-j have the same configuration as each other, and the XR glasses 30-1, 30-2, ... 30-k, ... 30-j have the same configuration as each other. However, the information processing system 1 may include a terminal device that does not have the same configuration as other terminal devices, or XR glasses that do not have the same configuration as other XR glasses.

[0010] In the information processing system 1, the location management server 40 and the user device 10-k are communicatively connected to each other via a communication network NET. The object management server 50 and the user device 10-k are communicatively connected to each other via the communication network NET. The object management server 50 and the structure management server 60 are communicatively connected to each other via the communication network NET. Furthermore, the terminal device 20-k and the XR glasses 30-k are communicatively connected to each other. In FIG. 1, a user U[k] uses the user device 10-k. Users U[1], U[2], ..., U[k-1], U[k+1], ..., U[j] use the user device 10-1, the user device 10-2, ..., the user device 10-k-1, the user device 10-k+1, ..., the user device 10-j, respectively.

[0011] The terminal device 20-k functions as a relay device that relays communication between the XR glasses 30-k and the position management server 40 and communication between the XR glasses 30-k and the object management server 50. The terminal device 20-k is configured by, for example, a smartphone or a tablet terminal.

[0012] The XR glasses 30-k are worn on the head of the user U[k]. The XR glasses 30-k are see-through glasses capable of displaying virtual objects. The user U[k] views the virtual objects while viewing the real space through the XR glasses 30-k. The virtual objects are positioned in a virtual space at positions corresponding to positions in the real space. By using the XR glasses 30-k, the user U[k] recognizes a mixed reality space in which the real space and the virtual space are combined. FIG. 2A is an explanatory diagram showing an example of a mixed reality space viewed by the user U[k] through the XR glasses 30-k. The virtual object VO shown in FIG. 2A has a spherical shape. The virtual object VO may be represented in three dimensions or two dimensions. The two-dimensional virtual object VO may be represented, for example, as a still image or a video.

[0013] Incidentally, virtual objects may be placed permanently or may be placed for a limited period, such as during an event. Furthermore, the locations where virtual objects can be placed are limited. In a service using virtual objects, notifying user U[k] of the location where the virtual object will be placed contributes to improving the convenience of the service. Therefore, the object management server 50 transmits a guide image obtained by capturing an image of a real-space location including the location of the virtual object virtually placed in real space to the user device 10-k as a first guide image. FIG. 2B is an explanatory diagram showing a guide image G1, which is an example of a guide image. The guide image G1 corresponds to the virtual object VO shown in FIG. 2A. When this guide image G1 is displayed on the user device 10-k, the user U[k] can recognize that the virtual object VO is virtually placed at the entrance of Ikebukuro Station. Therefore, the user U[k] can use the guide image G1 as a clue to find the virtual object VO.

[0014] As described above, the guide image is obtained by capturing an image of a location in real space that includes the position of the virtual object. However, in real space, new structures may be placed around the virtual object, or structures may be removed. When the environment around the virtual object changes in this way, it becomes difficult for the user U[k] to find the virtual object VO using the guide image as a clue. Therefore, the information processing system 1 updates the guide image in accordance with changes in the environment around the virtual object. For example, if a new structure is installed around the entrance of Ikebukuro Station, the guide image G1 shown in FIG. 2B is updated to a new guide image G2 shown in FIG. 2C, for example. In this example, two posters are installed above the entrance. Because the two posters are captured in the new guide image G2, the user U[k] can find the virtual object VO using the new guide image G2 as a clue.

[0015] The XR glasses 30-k shown in FIG. 1 include an imaging device that captures images of the outside world. The imaging device generates a captured image Gk by capturing an image. The captured image Gk is transmitted to the location management server 40 via the terminal device 20-k. The location management server 40 receives the captured image Gk transmitted from the terminal device 20-k. The location management server 40 identifies the position and direction of the XR glasses 30-k based on the captured image Gk. The location management server 40 returns position information Pk indicating the identified position and direction information Dk indicating the identified direction to the terminal device 20-k.

[0016] The location management server 40 stores a feature point map M. The feature point map M is data that represents a plurality of feature points using a three-dimensional global coordinate system. The feature point map M is generated, for example, by extracting a plurality of feature points from an image obtained by a stereo camera capturing an image of the area around the location where the virtual object is to be placed. In the feature point map M, the position in real space is represented using a world coordinate system.

[0017] The location management server 40 extracts a plurality of feature points from the captured image Gk. The location management server 40 identifies the imaging position and imaging direction used to generate the captured image Gk by comparing the extracted plurality of feature points with a plurality of feature points stored in the feature point map M. The location management server 40 returns location information Pk indicating the imaging position and direction information Dk indicating the imaging direction to the terminal device 20-k.

[0018] The XR glasses 30-k periodically transmit captured images Gk to the position management server 40, thereby periodically acquiring position information Pk and direction information Dk. The XR glasses 30-k track the local coordinates of the XR glasses 30-k in real time. The XR glasses 30-k correct the position and direction of the XR glasses 30-k in real time using the position information Pk and direction information Dk acquired from the position management server 40. This correction allows the XR glasses 30-k to recognize the position and direction of the XR glasses 30-k expressed in the world coordinate system in real time. In the following description, information indicating the position generated by the correction will be referred to as position information Pck, and information indicating the direction generated by the correction will be referred to as direction information Dck.

[0019] When the object management server 50 receives the position information Pck and the direction information Dck from the user device 10-k, it renders the virtual object based on the position information Pck and the direction information Dck. The object management server 50 transmits a virtual object image representing the rendered virtual object to the user device 10-k. In this example, the virtual object image is a three-dimensional image. Upon receiving the virtual object image, the user device 10-k causes the XR glasses 30-k to display the virtual object image. FIG. 2D is an explanatory diagram showing an example of a virtual object image. The virtual object image shown in FIG. 2D is displayed on the XR glasses 30-k when the user U[k] views the mixed reality space shown in FIG. 2A through the XR glasses 30-k.

[0020] The object management server 50 also manages the virtual object in association with one or more guide images. When the user device 10-k approaches the position of the virtual object, the object management server 50 transmits a first guide image selected from the one or more guide images to the user device 10-k.

[0021] The structure management server 60 manages spatial structure data. The spatial structure data is data that represents real objects in real space using a mesh structure, which is a collection of surfaces. The spatial structure data is expressed in a global coordinate system.

[0022] Spatial structure data has two main uses. The first use is to represent physical phenomena such as occlusion and reflection of virtual objects. For example, if the virtual object is a ball, the spatial structure data is used to represent the ball bouncing off the wall when thrown at a wall. Furthermore, if there is an obstacle between the user and the virtual object, the spatial structure data is used to make the virtual object invisible. The second use is to improve visibility for service developers when considering placing a virtual object. Developers set a reference point in real space and place the virtual object based on the reference point. The reference point is sometimes called an anchor. The reference point is set on a plane in real space. Because the spatial structure data represents surfaces using a mesh structure, using the spatial structure data allows developers to set a reference point on the surface of a real object existing in real space.

[0023] FIG. 3 is an explanatory diagram showing an example of an interface image Gi displayed on a tablet device when a developer places a virtual object. In the example shown in FIG. 3, it is assumed that a virtual object VOx is placed in an elevator hall. A superimposed image is displayed on the developer's tablet device. In the superimposed image, a structure image Gc showing a mesh structure indicated by dotted lines, an anchor image Gak showing a reference point, and the virtual object VOx are superimposed on an image acquired by the tablet device performing image capture. The reference point is set by the developer. In this case, the tablet device transmits position information indicating the position of the tablet device, directional information indicating the direction of the tablet device, and the captured image to the object management server 50. Based on the position information and directional information, the object management server 50 acquires spatial structure data indicating the spatial structure near the tablet device. Furthermore, the object management server 50 generates a structure image Gc by rendering a mesh structure based on the acquired spatial structure data. Furthermore, the object management server 50 renders the virtual object VOx. The object management server 50 then generates an interface image Gi by superimposing the structure image Gc, anchor image Gak, and virtual object VOx on the captured image. The object management server 50 transmits the interface image Gi to the tablet terminal. The developer can adjust the position and orientation of the virtual object VO1 by operating the tablet terminal. When the setting of the virtual object VO1 is complete, the tablet terminal transmits a completion notification to the object management server 50. The object management server 50 stores the position of the reference point and additional data in association with the virtual object VOx. The additional data includes the interface image Gi.

[0024] 1.1.2: XR Glasses Configuration 4 is a perspective view showing the appearance of the XR glasses 30-k. As shown in FIG. 4, the appearance of the XR glasses 30-k is similar to that of ordinary eyeglasses, and includes temples 91 and 92, a bridge 93, frames 94 and 95, and lenses 90L and 90R.

[0025] The bridge 93 is provided with an imaging device 36. The imaging device 36 is, for example, a camera. The imaging device 36 captures an image of the outside world to generate a captured image Gk. The imaging device 36 also outputs the captured image Gk. Each of the lenses 90L and 90R is provided with a half mirror. The frame 94 is provided with a liquid crystal panel or organic EL panel for the left eye and an optical member that guides light emitted from the display panel for the left eye to the lens 90L. The liquid crystal panel or organic EL panel will hereinafter be collectively referred to as the display panel. The half mirror provided on the lens 90L transmits light from the outside world and guides the light to the left eye, and reflects the light guided by the optical member, after which the reflected light is incident on the left eye. The frame 95 is provided with a display panel for the right eye and an optical member that guides light emitted from the display panel for the right eye to the lens 90R. The half mirror provided on the lens 90R transmits external light and guides the external light to the right eye, and also reflects the light guided by the optical member, and then directs the reflected light into the right eye.

[0026] The display 38, which will be described later, includes a lens 90L, a display panel for the left eye, and an optical member for the left eye, as well as a lens 90R, a display panel for the right eye, and an optical member for the right eye.

[0027] In the above configuration, the user U[k] can observe the image displayed on the display panel in a see-through state, superimposed on the outside world. Furthermore, the XR glasses 30-k display the left-eye image on the left-eye display panel and the right-eye image on the right-eye display panel, among the binocular images with parallax. Therefore, the XR glasses 30-k allow the user U[k] to perceive the displayed image as if it had depth and a three-dimensional effect.

[0028] 5 is a block diagram showing an example configuration of the XR glasses 30-k. The XR glasses 30-k include a processing device 31, a storage device 32, a detection device 35, an imaging device 36, a communication device 37, and a display 38. The elements of the XR glasses 30-k are connected to one another by one or more buses for communicating information. Note that the term "device" in this specification may be replaced with other terms such as circuit, device, or unit.

[0029] The processing device 31 is a processor that controls the entire XR glasses 30-k. The processing device 31 is configured using, for example, one or more chips. The processing device 31 is also configured using, for example, a central processing unit (CPU) that includes an interface with peripheral devices, an arithmetic unit, and registers. Some or all of the functions of the processing device 31 may be realized by hardware such as a DSP (Digital Signal Processor), ASIC (Application Specific Integrated Circuit), PLD (Programmable Logic Device), or FPGA (Field Programmable Gate Array). The processing device 31 executes various processes in parallel or sequentially.

[0030] The storage device 32 is a recording medium that can be read from and written to by the processing device 31. The storage device 32 also stores a plurality of programs including a control program PR1 executed by the processing device 31. The storage device 32 functions as a work area for the processing device 31.

[0031] The detection device 35 detects the state of the XR glasses 30-k. The detection device 35 includes, for example, inertial sensors such as an acceleration sensor that detects acceleration and a gyro sensor that detects angular acceleration, as well as a geomagnetic sensor that detects orientation. The acceleration sensor detects acceleration along each of the orthogonal X, Y, and Z axes. The gyro sensor detects angular acceleration around each of the X, Y, and Z axes as the central axis of rotation. The detection device 35 can generate orientation information indicating the orientation of the XR glasses 30-k based on the output information of the gyro sensor. The motion information described below includes acceleration information indicating the acceleration along each of the three axes and angular acceleration information indicating the angular acceleration along each of the three axes. The detection device 35 outputs the orientation information indicating the orientation of the XR glasses 30-k, movement information related to the movement of the XR glasses 30-k, and orientation information indicating the orientation of the XR glasses 30-k to the processing device 31.

[0032] The imaging device 36 outputs a captured image Gk obtained by capturing an image of the outside world. The imaging device 36 also includes, for example, a lens, an imaging element, an amplifier, and an AD converter. Light collected through the lens is converted into an analog imaging signal by the imaging element. The amplifier amplifies the imaging signal and outputs the amplified imaging signal to the AD converter. The AD converter converts the amplified imaging signal, which is an analog signal, into imaging information, which is a digital signal. The imaging information is output to the processing device 31. The captured image Gk output to the processing device 31 is output to the terminal device 20-k via the communication device 37.

[0033] The communication device 37 is hardware serving as a transmitting / receiving device for communicating with other devices. The communication device 37 may also be called, for example, a network device, a network controller, a network card, or a communication module. The communication device 37 may include a connector for wired connection, and the object management server 50 corresponding to the connector may include a circuit. The communication device 37 may also include a wireless communication interface. Examples of connectors and interface circuits for wired connection include products that comply with wired LAN, IEEE1394, and USB. Examples of wireless communication interfaces include products that comply with wireless LAN, Bluetooth (registered trademark), etc.

[0034] The display 38 is a device for displaying images. The display 38 displays various images under the control of the processing unit 31.

[0035] In the above configuration, the processing device 31 reads the control program PR1 from the storage device 32. The processing device 31 functions as a communication control unit 311, an estimation unit 312, and a display control unit 313 by executing the control program PR1.

[0036] The communication control unit 311 causes the communication device 37 to transmit the captured image Gk to the position management server 40, and also causes the communication device 37 to receive the position information Pk and direction information Dk transmitted from the position management server 40. The communication control unit 311 also causes the communication device 37 to transmit imaging parameters including the captured image Gk, position information Pck, and direction information Dck to the object management server 50. The communication control unit 311 also causes the communication device 37 to receive the guide image and virtual object image transmitted from the object management server 50. Note that communication between the position management server 40, the object management server 50, and the XR glasses 30-k is executed via the terminal device 20-k.

[0037] The estimation unit 312 corrects the position information Pk and direction information Dk periodically received from the position management server 40 based on the posture information, movement information, and orientation information output from the detection device 35. Through this correction, the estimation unit 312 estimates, in real time, position information Pck indicating the position of the XR glasses 30-k and direction information Dck indicating the direction of the XR glasses 30-k.

[0038] The display control unit 313 causes the display 38 to display the guide image and the virtual object image.

[0039] 1.1.3: Object Management Server Configuration 6 is a block diagram showing an example of the configuration of the object management server 50. The object management server 50 includes a processing device 51, a storage device 52, a communication device 53, a display 54, and an input device 55. The elements of the object management server 50 are connected to each other by one or more buses for communicating information. The object management server 50 is an example of a guide image management device.

[0040] The processing device 51 is a processor that controls the entire object management server 50. The processing device 51 is configured, for example, using one or more chips. The processing device 51 is configured, for example, using a central processing unit (CPU) that includes an interface with peripheral devices, an arithmetic unit, registers, etc. Some or all of the functions of the processing device 51 may be realized by hardware such as a DSP, ASIC, PLD, or FPGA. The processing device 51 executes various processes in parallel or sequentially.

[0041] The storage device 52 is a recording medium that can be read from and written to by the processing device 51. The storage device 52 also stores a plurality of programs including a control program PR2 executed by the processing device 51, a first database DB1, and a second database DB2. The storage device 52 also functions as a work area for the processing device 51.

[0042] The first database DB1 is used by a developer to manage virtual objects virtually placed in real space. FIG. 7 is an explanatory diagram showing an example of the data structure of the first database DB1. The first database DB1 has multiple records. Each record corresponds to a virtual object ID, virtual object data, reference point coordinates, relative coordinates, and additional data. The virtual object ID is an identifier that uniquely identifies a virtual object. The virtual object data is data representing a three-dimensional virtual object. The reference point coordinates are coordinates of a reference point used to virtually place a virtual object in real space. The reference point coordinates are expressed in a world coordinate system. The relative coordinates are coordinates that represent the position of a virtual object relative to the reference point coordinates. By combining the reference point coordinates and the relative coordinates, the position of the virtual object can be specified in the world coordinate system. The additional data represents an image in which a mesh-structured structural image Gc and an anchor image Gak are superimposed on a captured image Gk when a developer places a virtual object. The additional data is, for example, data representing an interface image Gi shown in FIG. 3.

[0043] The second database DB2 is used to manage guide images. FIG. 8 is an explanatory diagram showing an example of the data structure of the second database DB2. The second database DB2 has multiple records. In one record, a guide image ID, a virtual object ID, a guide image, imaging parameters, transmission conditions, and acquisition date and time are associated with each other. The guide image ID is an identifier that uniquely identifies a guide image. The guide image is a two-dimensional image. The imaging parameters are information that indicates the imaging conditions. The imaging parameters include position information Pck that indicates the position where the guide image was captured and direction information Dck that indicates the imaging direction. The position information Pck indicates coordinates in a global coordinate system. The direction information indicates an azimuth angle and an elevation / depression angle. The azimuth angle is an angle that increases clockwise, with north being 0 degrees. The east azimuth angle is 90 degrees, and the south azimuth angle is 180 degrees. The elevation / depression angle is an angle in the up / down direction based on the horizontal. The transmission conditions indicate the conditions for transmitting a guide image to the user device 10-k. In this example, the presence of the user device 10-k within a predetermined range is used as a transmission condition for the guide image. The acquisition date and time is the date and time when the guide image was captured or the date and time when the guide image was registered in the second database DB2.

[0044] In the example shown in FIG. 8, a guide image ID "G001" and a guide image ID "G002" are associated with a virtual object ID "V001." Therefore, two guide images are associated with one virtual object. On the other hand, a guide image ID "G003" is associated with a virtual object ID "V002." Therefore, one guide image is associated with one virtual object. Furthermore, the condition for transmitting guide image "001.jpeg" or guide image "002.jpeg" is that the user device 10-k is located within a range of a radius of 50 m centered on the coordinates (x11, y11, z11).

[0045] The communication device 53 shown in Fig. 6 is hardware serving as a transmitting / receiving device for communicating with other devices. The communication device 53 is also called, for example, a network device, a network controller, a network card, a communication module, etc. The communication device 53 may include a connector for wired connection and an interface circuit corresponding to the connector. The communication device 53 may also include a wireless communication interface.

[0046] The display 54 is a device that displays images and text information. The input device 55 includes, for example, a keyboard, a touchpad, a touch panel, or a pointing device such as a mouse.

[0047] The processing device 51 reads the control program PR2 from the storage device 52 and executes the control program PR2. As a result, the processing device 51 functions as a communication control unit 511, a management unit 512, a selection unit 513, and a determination unit 514.

[0048] The communication control unit 511 causes the communication device 53 to transmit a first guide image among one or more guide images to the user device 10-k, and also causes the communication device 53 to receive from the user device 10-k an image capture and imaging parameters generated by the user device 10-k performing imaging.

[0049] The management unit 512 manages one or more guide images in association with virtual objects. Specifically, the management unit 512 manages a first database DB1 and a second database DB2. Each record in the first database DB1 and each record in the second database DB2 are linked using a virtual object ID as a key. The one or more guide images are obtained by capturing an image of a location in real space that includes the position of a virtual object that is virtually placed in real space.

[0050] Furthermore, when a captured image acquired from the user device 10-k satisfies the registration conditions, the management unit 512 manages the captured image as a new guide image. Specifically, when the captured image satisfies the registration conditions, the management unit 512 adds a new record corresponding to the new guide image to the second database DB2. The new record includes a guide image ID, a virtual object ID, a guide image, imaging parameters, and an acquisition date and time.

[0051] The registration condition may include a condition that the captured image Gk does not contain any part that infringes portrait rights. The administrator may determine whether the captured image Gk contains any part that infringes portrait rights by observing the captured image Gk. Alternatively, the management unit 512 may determine whether portrait rights have been infringed by performing the following process. First, the management unit 512 performs a recognition process to recognize a person's face by analyzing the captured image Gk. Second, the management unit 512 performs a calculation process to calculate the ratio of the area of ​​the image of the recognized person's face to the entire captured image Gk. Third, the management unit 512 performs a determination process by comparing the calculated ratio with a predetermined value. In the determination process, the management unit 512 determines that the captured image Gk does not contain any part that infringes portrait rights if the ratio is less than the predetermined value. In the determination process, the management unit 512 determines that the captured image Gk contains any part that infringes portrait rights if the ratio is equal to or greater than the predetermined value.

[0052] Furthermore, the management unit 512 may use a learning model that has already learned the relationship between the captured image Gk and the infringement of portrait rights to determine whether or not there is any part in the captured image Gk that infringes portrait rights. This learning model is generated in the learning phase by learning pairs of label data and the captured image Gk as learning data. The label data indicates the result of a determination made by an administrator based on observation of the captured image Gk as to whether or not there is a portrait right infringement. In the operation phase, when the captured image Gk is input to the learning model, output data indicating whether or not there is a portrait right infringement is output from the learning model. The management unit 512 uses the output data to determine whether or not there is any part in the captured image Gk that infringes portrait rights.

[0053] The registration condition may include a condition that the degree of similarity between the captured image Gk and a guide image to be compared among one or more guide images is equal to or less than a threshold. The management unit 512 determines whether the captured image Gk satisfies this registration condition based on the guide image to be compared and its imaging parameters, as well as the captured image, position information, and direction information acquired from the user device 10-k. Specifically, the management unit 512 converts the captured image Gk captured from the viewpoint of the XR glasses 30-k into a transformed image, which is an image viewed from the viewpoint of the guide image to be compared, using a projective transformation matrix. The management unit 512 calculates the projective transformation matrix based on the acquired position information and direction information and the imaging parameters of the guide image to be compared. Furthermore, the management unit 512 calculates a similarity indicating the degree of similarity between the transformed image and the guide image to be compared. In calculating the similarity, the management unit 512, for example, first detects feature points from the transformed image and then similarly detects feature points from the guide image to be compared. Subsequently, the management unit 512 may obtain the similarity by comparing the feature points with each other. After that, the management unit 512 compares the similarity with a threshold value to determine whether the captured image Gk satisfies the registration condition. Furthermore, when the management unit 512 registers a new guide image in the second database DB2, it deletes the record of the guide image to be compared from the second database DB2. By this deletion, the previous guide image is updated to the new guide image. For example, the guide image G1 shown in FIG. 2B is the previous guide image, and the guide image G2 shown in FIG. 2C corresponds to the new guide image.

[0054] In the above description, the management unit 512 transforms the captured image Gk by using a projective transformation matrix, but conversely, the management unit 512 may transform the guide image to be compared by using a projective transformation matrix. In this case, the management unit 512 may calculate the similarity between the transformed guide image and the captured image Gk.

[0055] The selection unit 513 shown in FIG. 6 selects, from one or more guide images, a guide image captured under imaging conditions that are closest to the imaging conditions indicated by the imaging parameters of the captured image, as the first guide image, based on the imaging parameters received from the user device 10-k and the imaging parameters managed by the management unit 512 in association with each of the one or more guide images managed by the management unit 512.

[0056] The determination unit 514 determines whether the position of the user device 10-k is within a predetermined range. The predetermined range includes a position in real space where a virtual object is virtually placed. The predetermined range may be set for each virtual object or for each guide image. Alternatively, the predetermined range may be set for each location where multiple virtual objects are placed. Specifically, the determination unit 514 determines whether the position indicated by the position information Pck received by the communication device 53 from the user device 10-k is within a predetermined range indicated by the transmission condition.

[0057] 1.2: Operation of the embodiment The following describes a transmission process in which the object management server 50 transmits the first guide image and an update process in which the object management server 50 updates the guide image.

[0058] 1.2.1: Sending process FIG. 9 is a flowchart showing the contents of a process in which the object management server 50 according to the embodiment transmits a first guide image.

[0059] In step S10, the processing device 51 determines whether a captured image and imaging parameters have been received from the user device 10-k. The imaging parameters include position information Pck and direction information Dck. The processing device 51 repeats the process of step S10 until the determination result of step S10 becomes positive.

[0060] If the determination result of step S10 is positive, the processing device 51 determines whether the position of the user device 10-k is within a predetermined range (step S11). Specifically, the processing device 51 determines, for each guide image ID, whether the position indicated by the position information Pck acquired via the communication device 53 satisfies the transmission condition stored in the second database DB2.

[0061] If the determination result in step S11 is negative, the processing device 51 ends the process. On the other hand, if the determination result in step S11 is positive, the processing device 51 selects a guide image (step S12). If there is one guide image that satisfies the transmission condition in step S11, the processing device 51 selects the guide image that satisfies the transmission condition as the first guide image. On the other hand, if there are two or more guide images that satisfy the transmission condition in step S11, the processing device 51 selects the first guide image from among the two or more guide images.

[0062] Specifically, the processing device 51 executes the following processes. In a first process, the processing device 51 reads out two or more imaging parameters corresponding to two or more guide image IDs that satisfy the transmission condition from the second database DB2. In a second process, the processing device 51 identifies an imaging condition that is most similar to the imaging condition of the imaging parameters received from the user device 10-k, among the imaging conditions indicated by the two or more read imaging parameters. In a third process, the processing device 51 identifies a guide image ID that corresponds to the most similar imaging condition. In a fourth process, the processing device 51 reads out a guide image corresponding to the identified guide image ID from the second database DB2 as a first guide image. The processing device 51 selects a first guide image from two or more guide images by executing the first process to the fourth process.

[0063] In step S13, the processing device 51 causes the communication device 53 to transmit the first guide image to the user device 10-k.

[0064] In the above transmission process, the processing device 51 functions as a communication control unit 511 in steps S10 and S13. The processing device 51 also functions as a determination unit 514 in step S11. The processing device 51 also functions as a selection unit 513 in step S12.

[0065] 1.2.2: Update process FIG. 10 is a flowchart showing the process of updating the guide image by the object management server 50.

[0066] In step S20, the processing device 51 determines whether a captured image Gk and imaging parameters have been received from the user device 10-k. The imaging parameters include position information Pck and direction information Dck. The processing device 51 repeats the process of step S20 until the determination result of step S20 is positive.

[0067] If the determination result in step S20 is positive, the processing device 51 determines whether or not a virtual object to be superimposed on the captured image Gk exists (step S21). Specifically, the processing device 51 determines whether or not a virtual object exists in the field of view of the user U[k] based on the position and direction information Dck indicated by the position information Pck acquired via the communication device 53. For example, assume that the captured image Gk shown in FIG. 11 has been acquired. In this captured image Gk, a virtual object VO is superimposed on the portion indicated by the dotted line in FIG. 11 (part of the field of view of the user U[k]). Therefore, if the captured image Gk to be determined in step S21 is the image shown in FIG. 11, the determination result in step S21 is positive.

[0068] If the determination result of step S21 is negative, the processing device 51 ends the process. On the other hand, if the determination result of step S21 is positive, the processing device 51 determines whether or not a portion that infringes portrait rights is present in the captured image Gk (step S22). If the determination result of step S22 is positive, the processing device 51 ends the process. Therefore, a captured image Gk in which a human face is captured to an extent that would infringe portrait rights is not adopted as a guide image.

[0069] On the other hand, if the determination result of step S22 is negative, the processing device 51 calculates the similarity between the captured image and the guide image (step S23). In this case, the processing device 51 uses the second database DB2 to extract a set of a guide image (a guide image to be compared) corresponding to the virtual object ID to be superimposed on the captured image and imaging parameters corresponding to the virtual object ID to be superimposed on the captured image. The processing device 51 performs projective transformation of the captured image into a transformed image viewed from the viewpoint of the guide image, based on the imaging parameters of the captured image and the imaging parameters of the guide image. The processing device 51 calculates the similarity based on the transformed image and the guide image.

[0070] In the extraction of the above-mentioned sets of guide images and imaging parameters, multiple sets may be extracted. For example, assume that the storage contents of the second database DB2 are the storage contents shown in FIG. 8 and the virtual object ID is "V001". In this case, a set of guide images and imaging parameters corresponding to guide image ID "G001" and a set of guide images and imaging parameters corresponding to guide image ID "G002" are extracted. When multiple sets of guide images and imaging parameters are extracted in this way, the processing device 51 specifies, as a guide image to be compared, a guide image having imaging parameters that are most similar to the imaging parameters of the captured image, from among the extracted multiple imaging parameters. The processing device 51 calculates the similarity between the specified guide image to be compared and the captured image.

[0071] After the similarity is calculated in step S23, the processor 51 determines whether the similarity is equal to or less than a threshold value (step S24). If the determination result in step S24 is negative, the processor 51 ends the update process.

[0072] On the other hand, if the determination result of step S24 is positive, the processing device 51 manages the captured image Gk as a new guide image (step S25). Specifically, the processing device 51 adds a new record to the second database DB2 and deletes the record corresponding to the previous guide image. The new record has a guide image ID, a virtual object ID, a guide image, imaging parameters, transmission conditions, and acquisition date and time.

[0073] For example, it is assumed that the stored contents of the second database DB2 are the stored contents shown in Fig. 8, and the guide image ID to be updated is "G003". Furthermore, in a new record R11, for example, the new guide image ID is "G011", the captured image is "011.jpeg", the imaging parameters are position (x11, y11, z11) and direction (a11, b11), the transmission conditions are position (x33, y33, z33) and radius 15 m, and the acquisition date and time is 2022 / 3 / 25_16:00. In this case, the processing device 51 deletes record R3 shown in Fig. 8 and adds record R11. As a result, the stored contents of the second database DB2 are updated to the stored contents shown in Fig. 12.

[0074] In the above update process, in step S20, the processing device 51 functions as a communication control unit 511. In addition, in steps S21 to S25, the processing device 51 functions as a management unit 512.

[0075] 1.3: Effects of the embodiment According to the above description, the object management server 50 includes a management unit 512 and a communication control unit 511. The management unit 512 manages one or more guide images obtained by capturing an image of a location in real space including the position of a virtual object virtually placed in real space, in association with the virtual object. The communication control unit 511 causes the communication device 53 to transmit a first guide image among the one or more guide images to the user device 10-k, and causes the communication device 53 to receive, from the user device 10-k, a captured image Gk generated by the user device 10-k performing imaging. Furthermore, when the captured image Gk satisfies the registration conditions, the management unit 512 manages the captured image Gk as a new guide image.

[0076] The object management server 50 has the above configuration, so that when the environment around the virtual object changes, the guide image can be updated using the captured image Gk. As a result, the object management server 50 can provide a new guide image to the user, so that the user can search for the virtual object using the new guide image as a clue.

[0077] Furthermore, the communication control unit 511 causes the communication device 53 to receive the captured image Gk and imaging parameters related to the imaging conditions of the captured image Gk from the user device 10-k. When managing the captured image Gk as a new guide image, the management unit 512 manages the new guide image in association with the imaging parameters. With the above configuration, the object management server 50 can manage the imaging conditions of the guide image.

[0078] The object management server 50 further includes a selection unit 513. When two or more guide images are managed by the management unit 512 in association with a virtual object, the selection unit 513 selects, as a first guide image, from among the two or more guide images, a guide image captured under imaging conditions that are most similar to the imaging conditions indicated by the imaging parameters of the captured image Gk, based on the imaging parameters received from the user device 10-k and the imaging parameters managed in association with each of the two or more guide images.

[0079] According to the above configuration, the object management server 50 transmits to the user device 10-k a first guide image captured under imaging conditions that are most similar to the imaging parameters received from the user device 10-k. Therefore, since a first guide image according to the situation of the user device 10-k is transmitted, the user U[k] can easily move to the location where the virtual object is virtually placed, compared to a configuration in which a guide image arbitrarily selected from two or more guide images is transmitted as the first guide image.

[0080] The object management server 50 further includes a determination unit 514 that determines whether the user device 10-k is located within a predetermined range that includes the position in real space where the virtual object is virtually placed. If the determination result of the determination unit 514 is positive, the communication control unit 511 causes the communication device 53 to transmit a first guide image to the user device 10-k, and if the determination result of the determination unit 514 is negative, causes the communication device 53 not to transmit the first guide image to the user device 10-k.

[0081] According to the above configuration, the object management server 50 transmits the first guide image to the user device 10-k only when the user device 10-k is located within a predetermined range, so the first guide image is transmitted when the user device 10-k approaches the virtual object. Therefore, the user U[k] can receive the first guide image when approaching the virtual object. This allows the user U[k] to receive the first guide image at a timing when guidance is required.

[0082] The object management server 50 manages the captured image Gk as a new guide image when the captured image Gk satisfies the registration conditions. The registration conditions include a condition that the captured image Gk does not contain any part that infringes portrait rights. Although a person's face may appear in the captured image Gk, the object management server 50 does not manage the captured image Gk that infringes portrait rights as a new guide image, thereby preventing portrait right infringement.

[0083] The object management server 50 manages the captured image Gk as a new guide image when the captured image Gk satisfies the registration conditions. The registration conditions include a condition that the degree of similarity between the captured image Gk and a guide image to be compared among one or more guide images is equal to or less than a threshold. According to the above configuration, when the degree of similarity between the captured image Gk and the guide image to be compared is equal to or less than a threshold, the captured image Gk is managed as a new guide image. Therefore, when a change in the environment around the virtual object is detected, the guide image can be updated using the captured image Gk.

[0084] 3: Variation The present disclosure is not limited to the above-described exemplary embodiments. Specific modified embodiments are exemplified below. Two or more embodiments selected arbitrarily from the following examples may be combined.

[0085] 3.1: Variation 1 The user device 10-k according to the embodiment includes a terminal device 20-k and XR glasses 30-k. The terminal device 20-k functions as a relay device that relays communication between the XR glasses 30-k and the location management server 40 and communication between the XR glasses 30-k and the object management server 50. The present disclosure is not limited to an aspect in which the user device 10-k includes the terminal device 20-k and the XR glasses 30-k. For example, the XR glasses 30-k may have a function for communicating with the location management server 40 and the object management server 50. In this case, the user device 10-k may be included in the XR glasses 30-k.

[0086] Alternatively, the terminal device 20-k may have the functions of the XR glasses 30-k. In this case, the user device 10-k is configured by the terminal device 20-k. However, the terminal device 20-k differs from the XR glasses 30-k in that it displays virtual objects as two-dimensional images. FIG. 13 is a block diagram showing an example configuration of the terminal device 20-k. As shown in FIG. 13, the terminal device 20-k includes a processing device 21, a storage device 22, an input device 24, a detection device 25, an imaging device 26, a communication device 27, and a display 28. The processing device 21, the storage device 22, the detection device 25, the imaging device 26, and the communication device 27 correspond to the processing device 31, the storage device 32, the detection device 35, the imaging device 36, and the communication device 37 in the XR glasses 30-k shown in FIG. 5. The terminal device 20-k has a flat plate-like shape. The imaging device 26 is provided on the surface opposite the display 28. The processing device 21 reads out the control program PR3 from the storage device 22. By executing the control program PR3, the processing device 21 functions as the communication control unit 311 and the estimation unit 312 described above. The processing device 21 also functions as a display control unit 213. The display control unit 213 causes the display 28 to display an image in which a virtual object image is superimposed on a captured image generated by the imaging device 26 performing imaging. The virtual object image is received from the object management server 50 via the communication device 27. The input device 24 is configured by, for example, a touch panel. In the above configuration, the terminal device 20-k causes the display 28 to display a first guide image transmitted from the object management server 50. Furthermore, the terminal device 20-k transmits the captured image and imaging parameters to the object management server 50. If the received captured image satisfies the registration conditions, the object management server 50 updates the guide image using the captured image.

[0087] 3.2: Variation 2 In the above-described embodiment, the condition for transmitting a first guide image is that the user device 10-k is located within a predetermined range. When this transmission condition is met, the object management server 50 transmits the first guide image to the user device 10-k. However, the transmission condition for a first guide image is not limited to the above-described transmission condition. For example, a map on which icons are placed for each location of a virtual object may be displayed on the user device 10-k, and when an icon is selected by the user, the object management server 50 may transmit a first guide image corresponding to the selected icon to the user device 10-k.

[0088] 3.3: Variation 3 In the above-described embodiment, the condition for registering the captured image Gk is that the captured image Gk does not contain any part that infringes portrait rights. However, the condition for registration may also be that the captured image Gk does not contain any part that infringes copyright. The administrator may determine whether or not a copyright has been infringed by observing the captured image Gk.

[0089] 4:Other (1) In the above-described embodiment, storage device 22, storage device 32, and storage device 52 are exemplified by ROM and RAM, but may be a flexible disk, a magneto-optical disk (e.g., a compact disk, a digital versatile disk, a Blu-ray (registered trademark) disk), a smart card, a flash memory device (e.g., a card, a stick, a key drive), a CD-ROM (Compact Disc-ROM), a register, a removable disk, a hard disk, a floppy (registered trademark) disk, a magnetic strip, a database, a server, or any other suitable storage medium. The program may also be transmitted from a network via a telecommunications line. The program may also be transmitted from a communications network NET via a telecommunications line.

[0090] (2) In the above-described embodiments, the described information, signals, etc. may be represented using any of a variety of different technologies. For example, data, instructions, commands, information, signals, bits, symbols, chips, etc. that may be referred to throughout the above description may be represented by voltages, currents, electromagnetic waves, magnetic fields or magnetic particles, optical fields or photons, or any combination thereof.

[0091] (3) In the above-described embodiment, input and output information may be stored in a specific location (for example, a memory) or may be managed using a management table. Input and output information may be overwritten, updated, or added to. Output information may be deleted. Input information may be transmitted to another device.

[0092] (4) In the above-described embodiment, the determination may be made by a value (0 or 1) represented using one bit, by a Boolean value (true or false), or by a comparison of numerical values ​​(e.g., comparison with a predetermined value).

[0093] (5) The order of the process procedures, sequences, flowcharts, etc. illustrated in the above-described embodiments may be rearranged unless inconsistent. For example, the methods described in this disclosure present elements of various steps using an example order, and are not limited to the particular order presented.

[0094] (6) Each function illustrated in Figures 1 to 13 is realized by any combination of at least one of hardware and software. Furthermore, the method for realizing each functional block is not particularly limited. That is, each functional block may be realized using a single device that is physically or logically coupled, or may be realized using two or more physically or logically separated devices that are directly or indirectly connected (for example, by wire, wirelessly, etc.) and these multiple devices. A functional block may also be realized by combining software with the single device or the multiple devices.

[0095] (7) The programs exemplified in the above-described embodiments should be broadly construed to mean instructions, instruction sets, code, code segments, program code, programs, subprograms, software modules, applications, software applications, software packages, routines, subroutines, objects, executable files, execution threads, procedures, functions, etc., regardless of whether they are called software, firmware, middleware, microcode, hardware description language, or by other names.

[0096] Software, instructions, information, etc. may also be transmitted or received over a transmission medium. For example, if software is transmitted from a website, server, or other remote source using wired technologies (such as coaxial cable, fiber optic cable, twisted pair, Digital Subscriber Line (DSL)), and / or wireless technologies (such as infrared, microwave), then these wired and / or wireless technologies are included within the definition of transmission media.

[0097] (8) In each of the foregoing embodiments, the terms "system" and "network" are used interchangeably.

[0098] (9) The information, parameters, etc. described in this disclosure may be expressed using absolute values, relative values ​​from a predetermined value, or corresponding other information.

[0099] (10) In the above-described embodiments, the user devices 10-1 to 10-j, the terminal devices 20-1 to 20-j, and the XR glasses 30-1 to 30-j may be mobile stations (MS). A mobile station may also be referred to by those skilled in the art as a subscriber station, a mobile unit, a subscriber unit, a wireless unit, a remote unit, a mobile device, a wireless device, a wireless communication device, a remote device, a mobile subscriber station, an access terminal, a mobile terminal, a wireless terminal, a remote terminal, a handset, a user agent, a mobile client, a client, or some other appropriate term. In addition, in the present disclosure, terms such as "mobile station," "user terminal," "user equipment (UE)," and "terminal" may be used interchangeably.

[0100] (11) In the above-described embodiments, the terms "connected," "coupled," or any variations thereof refer to any direct or indirect connection or coupling between two or more elements, and may include the presence of one or more intermediate elements between two elements that are "connected" or "coupled" to each other. The coupling or connection between elements may be a physical coupling or connection, a logical coupling or connection, or a combination thereof. For example, "connected" may be read as "access." As used in this disclosure, two elements may be considered to be "connected" or "coupled" to each other using at least one of one or more wires, cables, and printed electrical connections, as well as electromagnetic energy having wavelengths in the radio frequency range, microwave range, and optical (both visible and invisible) range, as some non-limiting and non-exhaustive examples.

[0101] (12) In the above embodiments, the phrase "based on" does not mean "based only on," unless otherwise specified. In other words, the phrase "based on" means both "based only on" and "based at least on."

[0102] (13) As used in this disclosure, the terms "determining" and "determining" may encompass a wide variety of actions. "Determining" and "determining" may include, for example, judging, calculating, computing, processing, deriving, investigating, looking up, searching, inquiring (e.g., searching in a table, database, or other data structure), ascertaining, and the like. "Determining" and "determining" may also include receiving (e.g., receiving information), transmitting (e.g., sending information), input, output, accessing (e.g., accessing data in memory), and the like. Furthermore, "judgment" and "decision" can include regarding resolving, selecting, choosing, establishing, comparing, etc. as having been "judged" or "decided." In other words, "judgment" and "decision" can include regarding some action as having been "judged" or "decided." Furthermore, "judgment (decision)" can be interpreted as "assuming," "expecting," "considering," etc.

[0103] (14) In the above embodiments, when "include," "including," and variations thereof are used, these terms are intended to be inclusive, similar to the term "comprising." Furthermore, the term "or" as used in this disclosure is not intended to be an exclusive or.

[0104] (15) In this disclosure, where articles are added by translation, such as a, an, and the in English, this disclosure may include the nouns following these articles being plural.

[0105] (16) In this disclosure, the term "A and B are different" may mean "A and B are different from each other." The term may also mean "A and B are each different from C." Terms such as "separate" and "combined" may also be interpreted in the same way as "different."

[0106] (17) Each aspect / embodiment described in this disclosure may be used alone, in combination, or switched depending on the implementation. Notification of predetermined information (e.g., notification that "X is true") is not limited to explicit notification, but may be implicit (e.g., not notifying the predetermined information).

[0107] Although the present disclosure has been described in detail above, it is clear to those skilled in the art that the present disclosure is not limited to the embodiments described herein. The present disclosure can be implemented in modified and altered forms without departing from the spirit and scope of the present disclosure as defined by the claims. Therefore, the description of the present disclosure is intended to be illustrative and does not have any limiting meaning on the present disclosure. [Explanation of symbols]

[0108] 1...information processing system, 10-1 to 10-j...user device, 11, 21, 51...processing device, 13, 23, 53...communication device, 20-1 to 20-j...terminal device, 30-1 to 30-j...XR glasses, 511...communication control unit, 512...management unit, 513...selection unit, 514...determination unit, Dck...direction information, Pck...position information, G1, G2...guide image, Gk...captured image, VO1, VOx...virtual objects.

Claims

1. a management unit that manages one or more guide images obtained by capturing an image of a location in real space including a position of a virtual object virtually placed in the real space, in association with the virtual object; a communication control unit that causes a communication device to transmit a first guide image among the one or more guide images to a user device and causes the communication device to receive, from the user device, an imaged image generated by the user device performing imaging; Including, the management unit manages the captured image as a new guide image when the captured image satisfies a registration condition. Guide image management device.

2. the communication control unit causes the communication device to receive the captured image and imaging parameters related to imaging conditions of the captured image from the user device; When managing the captured image as the new guide image, the management unit manages the new guide image in association with the imaging parameters. The guide image management device according to claim 1 .

3. and a selection unit configured to select, as the first guide image, a guide image captured under imaging conditions that are closest to imaging conditions indicated by the imaging parameters of the captured image from among the two or more guide images, based on imaging parameters received from the user device and imaging parameters managed in association with each of the two or more guide images, when the management unit manages two or more guide images in association with each other. The guide image management device according to claim 2 .

4. a determination unit that determines whether the user device is located within a predetermined range that includes a position in real space where the virtual object is virtually placed, the communication control unit causes the communication device to transmit the first guide image to the user device when the determination result of the determination unit is positive, and does not cause the communication device to transmit the first guide image to the user device when the determination result of the determination unit is negative. The guide image management device according to claim 1 or 3.

5. The registration conditions include a condition that the captured image does not contain any part that infringes portrait rights.

4. The guide image management device according to claim 1, 2, or 3.

6. the registration condition includes a condition that a degree of similarity between the captured image and a guide image to be compared among the one or more guide images is equal to or less than a threshold value; 4. The guide image management device according to claim 1, 2 or 3.

Images