Information processing device, information processing method, and program

Abstract

This allows users to experience AR content even if they are not in the location where the AR content is displayed. [Solution] An information processing device 1 that displays AR content according to location. The information processing device 1 includes a location information acquisition unit 13 that acquires location information indicating the location of the information processing device 1, a content acquisition unit 171 that acquires AR content determined according to the location indicated by the location information, and a display control unit 172 that overlays the content onto an image captured by an imaging unit 12 and displays it on a display unit, and controls the display mode of the AR content based on a magnification ratio which is the ratio of the field of view included in the image to a reference field of view of the imaging unit.

Description

Technical Field

[0001] The present invention relates to an information processing apparatus, an information processing method, and a program.

Background Art

[0002] A system for displaying AR (Augmented Reality) content based on position information is known (for example, see Patent Document 1).

Prior Art Documents

Patent Documents

[0003]

Patent Document 1

Summary of the Invention

Problems to be Solved by the Invention

[0004] In the prior art, when there is an obstacle for the user to move to an area where AR content can be displayed, there has been a problem that the user cannot experience the AR content.

[0005] Therefore, the present invention has been made in view of these points, and an object thereof is to enable the user to experience AR content even when the user is not at the position of the display target of the AR content based on the position information. [[ID=z40]]

Means for Solving the Problems

[0006] An information processing apparatus in a first aspect of the present invention is an information processing apparatus for displaying AR content according to location, comprising: an imaging unit; a location information acquisition unit for acquiring location information indicating the location of the information processing apparatus; a content acquisition unit for acquiring AR content according to the location indicated by the location information; and a display control unit for displaying the content superimposed on an image captured by the imaging unit on a display unit, the display control unit for controlling the display of the AR content based on a magnification ratio which is the ratio of the field of view included in the image to a reference field of view of the imaging unit.

[0007] The content acquisition unit further acquires a displayable area indicating the range of the location of the information processing device in which the AR content can be displayed, in association with the AR content, and further includes a virtual position calculation unit that calculates a virtual position, which is the virtual location of the information processing device, based on the location of the information processing device indicated by the location information and the magnification ratio, and the display control unit may display the AR content if the virtual position is included in the range in which the AR content can be displayed.

[0008] The content acquisition unit may further acquire the position for displaying the AR content in association with the AR content, and the virtual position calculation unit may calculate the virtual position based on the position for displaying the AR content, such that the larger the magnification, the closer the virtual position is to the position for displaying the AR content from the position indicated by the position information.

[0009] The virtual position calculation unit may determine the virtual position as a point on a straight line connecting the position indicated by the position information and the position where the AR content is displayed, where the image of the position where the AR content is displayed when photographed from the position indicated by the position information at the magnification and the image of the position where the AR content is displayed when photographed from the virtual position at the reference field of view of the imaging unit are captured at the same size.

[0010] The content acquisition unit acquires a plurality of AR contents and the displayable area of ​​each of the plurality of AR contents, and the display control unit may display the AR content that can be displayed at the virtual location from among the plurality of contents.

[0011] If, after displaying the first AR content among the plurality of AR content, the virtual position is located outside the range in which the first AR content can be displayed, the display control unit may display a second AR content, which is different from the first AR content, from among the plurality of AR content that can be displayed at the virtual position.

[0012] In a second aspect of the present invention, an information processing method is performed by a computer having an imaging unit to display AR content corresponding to the position of the computer, comprising: a position information acquisition step of acquiring position information indicating the position of the computer; a content acquisition step of acquiring AR content according to the position indicated by the position information; and a display control step of superimposing the content onto an image captured by the imaging unit and displaying it on a display unit, the display control step of controlling the display of the AR content based on a magnification ratio which is the ratio of the field of view included in the image to a reference field of view of the imaging unit.

[0013] In a program according to a third aspect of the present invention, a computer having an imaging unit is given a program for displaying AR content corresponding to the position of the computer, the program comprising: a position information acquisition step for acquiring position information indicating the position of the computer; a content acquisition step for acquiring AR content according to the position indicated by the position information; and a display control step for displaying the content superimposed on an image captured by the imaging unit on a display unit, the display control step for controlling the display of the AR content based on a magnification ratio which is the ratio of the field of view included in the image to a reference field of view of the imaging unit. [Effects of the Invention]

[0014] According to the present invention, it is possible to provide AR content that can be displayed when the magnification of the camera satisfies a predetermined condition. As a result, according to the present invention, even when the user is not at the position of the display target of the AR content based on the position information, the user can experience the AR content.

Brief Description of the Drawings

[0015] [Figure 1] It is a diagram for explaining the outline of the information processing system S according to the embodiment. [Figure 2] It is a diagram for explaining the outline of the information processing system S according to the embodiment. [Figure 3] It is a block diagram showing the configuration of the information processing apparatus 1. [Figure 4] It is a diagram showing an example of the content information managed by the content management apparatus 2. [Figure 5] It is a diagram showing an example of the content information managed by the content management apparatus 2. [Figure 6] It is a diagram showing an example of the data structure of the virtual position table stored in the storage unit 16. [Figure 7] It is a diagram showing an example of the processing of the virtual position calculation unit 173. [Figure 8] It is a diagram showing an example of the content information managed by the content management apparatus 2. [Figure 9] It is a diagram showing an example of the processing of the virtual position calculation unit 173. [Figure 10] It is a flowchart showing the processing flow in the information processing apparatus 1.

Modes for Carrying Out the Invention

[0016] [Outline of Information Processing System S] FIGS. 1 and 2 are diagrams for explaining the outline of the information processing system S according to the embodiment. The information processing system S is a system for displaying AR content. The information processing system S includes an information processing apparatus 1 and a content management apparatus 2.

[0017] The information processing device 1 is a personal computer, tablet, smartphone, smart glass, or head-mounted display used by the user U. The information processing device 1 has a camera and superimposes and displays predetermined AR content on an image captured by the camera.

[0018] The content management device 2 is a device that manages AR content and distributes displayable AR content in response to a request from the information processing device 1. The content management device 2 stores content information associating the AR content, the conditions for acquiring the AR content, and the conditions for displaying the AR content.

[0019] As an example, in the content information, the AR content, the area where the AR content can be acquired, and the magnification of the camera at which the AR content can be displayed are associated. In this specification, the magnification is the ratio of the angle of view included in the image captured by the camera to the reference angle of view in the camera included in the information processing device 1. The reference angle of view is, for example, the widest angle of view (the angle of view at the so-called wide-angle end) among the angles of view that the camera included in the information processing device 1 can capture. The reference angle of view may be predetermined for each camera included in the information processing device 1. As an example, the magnification is the magnification of the focal length when the focal length of the wide-angle end of the camera is set to 1 times.

[0020] The processing in the information processing system S will be described. The information processing device 1 acquires position information ((1) in FIG. 1). The information processing device 1 acquires content corresponding to the position indicated by the position information from the content management device 2 ((2) in FIG. 1). The information processing device 1 controls the display of the content based on the magnification of the camera ((3) in FIG. 1).

[0021] An example of content display control in the information processing system S will be explained with reference to Figure 2. In Figure 2, image P1, indicated by the symbol P1, shows a picture of the subject (a castle in Figure 2) taken with the camera of the information processing device 1 at a reference field of view. Also in Figure 2, image P2, indicated by the symbol P2, shows a picture of the subject taken with a magnification greater than the reference field of view. Here, if user U moves closer to the subject than the position where image P1 was taken and then takes another picture of the subject at the reference field of view, an image similar to the second image P2 can be captured. In other words, changing the camera's magnification can be considered as simulating a change in the position between the camera and the subject. The inventors of the present invention have come to recognize the possibility of using the camera's magnification for display control of AR content.

[0022] The information processing system S is configured to control the display of AR content according to the camera's magnification, allowing users to experience AR content even when they are not at the location where the AR content is displayed based on location information.

[0023] [Configuration of Information Processing Device 1] Figure 3 is a block diagram showing the configuration of the information processing device 1. The information processing device 1 includes a communication unit 11, an imaging unit 12, a location information acquisition unit 13, a display unit 14, an operation reception unit 15, a storage unit 16, and a control unit 17. The control unit 17 includes a content acquisition unit 171, a display control unit 172, and a virtual location calculation unit 173.

[0024] The communication unit 11 is a communication interface for sending and receiving data with other devices via a network. The imaging unit 12 is a camera. The imaging unit 12 has optical zoom or digital zoom functionality and can adjust the magnification according to the user U's operation. The imaging unit 12 determines the magnification based on the user U's operation received by the operation reception unit 15 and outputs the magnification to the display control unit 172. The imaging unit 12 may have a gyro sensor or the like and output information indicating the optical axis direction of the imaging unit 12 to the virtual position calculation unit 173.

[0025] The location information acquisition unit 13 is, for example, a GPS (Global Pointing System) sensor. The location information acquisition unit 13 acquires location information indicating the location of the information processing device 1.

[0026] The display unit 14 is a display. The display unit 14 displays an image based on control signals received from the display control unit 172. The operation reception unit 15 is a device for receiving operations from user U. The operation reception unit 15 is, for example, a touch panel display, a keyboard, a mouse, etc.

[0027] The memory unit 16 is a storage medium that includes ROM (Read Only Memory), RAM (Random Access Memory), SSD (Solid State Drive), hard disk drive, etc. The memory unit 16 pre-stores programs that the control unit 17 will execute.

[0028] The control unit 17 is, for example, a processor such as a CPU (Central Processing Unit). By executing a program stored in the storage unit 16, the control unit 17 functions as a content acquisition unit 171, a display control unit 172, and a virtual position calculation unit 173.

[0029] The content acquisition unit 171 acquires content information. The content acquisition unit 171 acquires AR content from the content management device 2 according to the location indicated by the location information. Figure 4 is a diagram showing an example of content information managed by the content management device 2. The content information is associated with "Content ID", "Acquisitionable Area", "Zoom Level", and "Content".

[0030] "Content ID" is an ID (Identification) used to identify the content. "Acquirable Area" indicates the range of locations from which the content information can be acquired. "Magnification" indicates the magnification at which the content can be displayed. "Magnification" may be defined for each imaging device. "Content" refers to the content to be displayed superimposed on the image captured by the imaging unit 12. "Content" may include the content itself, or it may include information indicating the resource for acquiring the content (e.g., URI (Universal Resource Identifier)).

[0031] Furthermore, if the functions of the imaging unit 12 differ for each model, the model of the information processing device 1 may also be associated with the content information. The content acquisition unit 171 can acquire content information that indicates the displayable conditions according to the functions of the imaging unit 12 by acquiring content information corresponding to the model of the information processing device 1.

[0032] The content acquisition unit 171 acquires content information corresponding to the location indicated by the location information acquired by the location information acquisition unit 13.

[0033] The display control unit 172 overlays the content onto the image captured by the imaging unit 12 and displays it on the display unit 14. The display control unit 172 controls the display of the AR content based on the magnification of the imaging unit 12. The display control unit 172 obtains the magnification from the imaging unit 12. The display control unit 172 compares the magnification associated with the obtained content information with the magnification obtained from the imaging unit 12 and determines whether the magnification of the imaging unit 12 satisfies the magnification associated with the content information. If the magnification of the imaging unit 12 satisfies the magnification associated with the content information, the display control unit 172 overlays the obtained content onto the image captured by the imaging unit 12 and displays it.

[0034] With the information processing device 1 configured in this way, users can experience AR content even when they are not at the location where the AR content is displayed based on location information.

[0035] The information processing device 1 is configured to calculate a virtual position assuming the user moves from their current location along the optical axis of the imaging unit 12, and to determine whether or not content can be displayed based on this virtual position. This makes it possible to manage the display of AR content in more detail. As a result, even a user U who has difficulty moving freely can experience location-based AR content.

[0036] The content acquisition unit 171 further acquires a displayable area, which indicates the range of locations where the information processing device 1 can display the AR content, in association with the AR content. Figure 5 shows an example of content information managed by the content management device 2 in this case. In the content information shown in Figure 5, the "displayable area" is further associated. The "displayable area" indicates the location where the associated content can be displayed when the information processing device 1 is located in that area or when the virtual location of the information processing device 1 is located in that area.

[0037] The virtual position calculation unit 173 calculates a virtual position, which is the virtual position of the information processing device 1, based on the position of the information processing device indicated by the position information and the magnification factor. The virtual position calculation unit 173 calculates the virtual position such that the larger the magnification factor of the imaging unit 12, the closer it is to the position where the AR content is displayed from the position indicated by the position information.

[0038] As an example, the storage unit 16 may store a virtual position table that associates the magnification of the imaging unit 12 with the virtual movement distance of the imaging unit 12 in the optical axis direction. Figure 6 shows the data structure of the virtual position table stored by the storage unit 16. In the virtual position table, "magnification" and "virtual movement distance" are associated. "Virtual movement distance" indicates the virtual movement distance set to calculate the virtual position corresponding to the magnification. In "virtual movement distance," it is set so that the larger the magnification, the larger the movement distance required to calculate the virtual position. The virtual position calculation unit 173 identifies the virtual movement distance corresponding to the magnification in the virtual position table and, using the position indicated by the position information as a reference, calculates the position obtained by moving the identified virtual movement distance in the optical axis direction to which the imaging unit 12 is pointed as the virtual position of the information processing device 1.

[0039] The display control unit 172 compares the virtual position calculated by the virtual position calculation unit 173 with the displayable area associated with the content information and determines whether the virtual position calculated by the virtual position calculation unit 173 is included in the displayable area associated with the content information. If the virtual position calculated by the virtual position calculation unit 173 is included in the range where AR content can be displayed, the display control unit 172 displays the content associated with the content information.

[0040] With the information processing device 1 configured in this way, it becomes possible to manage the display of location-based AR content with a simple configuration. As a result, even user U, who has difficulty moving freely, can experience location-based AR content.

[0041] The information processing device 1 may be configured to calculate a virtual position based on a predetermined reference field of view. In this case, the virtual position calculation unit 173 determines a virtual position at a point on the optical axis of the imaging unit 12, based on the position indicated by the position information, so as to satisfy the following conditions. The condition is that the virtual position is determined to be the position where the subject appears at the same size when the imaging unit 12 images the subject from the position indicated by the position information, and when the same subject is imaged from the virtual position at a predetermined distance with a reference field of view. As will be described later, the virtual position calculation unit 173 may also determine the virtual position based on a predetermined display position of the AR content.

[0042] Figure 7 shows an example of the processing performed by the virtual position calculation unit 173 in this case. In this case, the distance (R2) from the virtual position P3 to the content display position P2 and the reference field of view (Θ2) at the virtual position P3 are predetermined. As an example, the reference field of view Θ2 is the field of view at the wide-angle end of the imaging unit 12. The virtual position calculation unit 173 determines the field of view (Θ1) of the imaging unit 12 based on the magnification ratio of the imaging unit 12. The virtual position calculation unit 173 calculates the virtual travel distance D from the information processing device 1 using the following formula.

[0043] [Formula 1] JPEG0007833579000002.jpg12170

[0044] The virtual position calculation unit 173 calculates a virtual position for the information processing device 1 by moving the calculated distance D in the direction of the optical axis and the subject of the imaging unit 12, based on the position indicated by the position information.

[0045] In the above explanation, the virtual position was calculated using a fixed value for the distance from the virtual position in the reference field of view. However, the information processing device 1 may be configured to calculate the virtual position based on the distance between the predetermined content display position and the current position, and the field of view.

[0046] The content acquisition unit 171 further acquires the position where the AR content will be displayed, associating it with the AR content. Figure 8 shows an example of content information managed by the content management device 2. In the content information, the "display position" is further associated. The display position is information that indicates a predetermined position where the content will be displayed.

[0047] The virtual position calculation unit 173 calculates a virtual position based on the position where the AR content will be displayed, such that the larger the magnification, the closer the virtual position is to the position where the AR content will be displayed from the position indicated by the location information. More specifically, the virtual position calculation unit 173 determines the virtual position to be a point on the straight line connecting the position indicated by the location information and the position where the AR content will be displayed, where the position where the AR content will be displayed is captured at the same size when the position indicated by the location information is captured by the imaging unit 12 at its magnification, and the position where the AR content will be displayed is captured at the standard field of view of the imaging unit 12 from the virtual position.

[0048] Figure 9 shows an example of the processing performed by the virtual position calculation unit 173 in this case. In this case, a reference field of view Θ2 is predetermined. The virtual position calculation unit 173 calculates the distance R1 between the display position of the content and the information processing device 1 based on the display position of the content indicated by the content information and the position of the information processing device 1 indicated by the position information. The virtual position calculation unit 173 determines the field of view Θ1 based on the magnification ratio of the imaging unit 12. Based on the distance R1, field of view Θ1 and field of view Θ2, the virtual position calculation unit 173 calculates the virtual travel distance D from the information processing device 1 using the following formula.

[0049] [Formula 2] JPEG0007833579000003.jpg12170

[0050] Based on the position indicated by the position information, the information processing device 1 calculates a virtual position by moving the position by a distance D calculated in the direction of the optical axis captured by the imaging unit 12.

[0051] With the information processing device 1 configured in this way, it becomes possible to determine the virtual position in more detail, and to control the display position of the content more precisely.

[0052] Depending on the location of the information processing device 1 indicated by the location information, it may be possible to acquire multiple AR contents. If multiple AR contents are acquired, it is necessary to decide which AR content to display.

[0053] The content acquisition unit 171 acquires multiple AR content and area information for each of the multiple AR content. The display control unit 172 displays the AR content that can be displayed at the virtual location from among the multiple content. Based on the displayable location associated with each of the multiple AR content, the display control unit 1 identifies the AR content that can be displayed at the virtual location of the information processing device 1. The display control unit 172 displays the identified AR content superimposed on the image captured by the imaging unit 12.

[0054] With the display control unit 172 configured in this way, it becomes possible to display appropriate content according to the virtual position.

[0055] If it is possible to display two or more types of AR content simultaneously, the information processing device 1 may be configured, for example, to prioritize the display of the content that was displayed first. In this case, for example, if the information processing device 1 is located outside the displayable area of ​​the content that was displayed first, other content will be displayed.

[0056] In other words, if the virtual position moves outside the range in which the first AR content can be displayed after the display control unit 172 has displayed the first AR content among the multiple AR content, the display control unit 172 displays a second AR content that is displayable at the virtual position and is different from the first AR content.

[0057] This configuration allows for seamless content switching.

[0058] If multiple AR contents can be displayed simultaneously, the display control unit 172 may accept a selection of the content to be displayed from the user and display the content selected by the user.

[0059] [Processing flow in information processing device 1] Figure 10 is a flowchart showing the processing flow in the information processing device 1. The flowchart shown in Figure 10 starts from the point when location information is acquired. At the start of the flowchart shown in Figure 10, the imaging unit 12 is in a state where it can capture an image.

[0060] The location information acquisition unit 13 acquires location information indicating the location of the information processing device 1 (S01). The content acquisition unit 171 acquires AR content based on the acquired location information (S02).

[0061] The virtual position calculation unit 173 calculates the virtual position of the information processing device 1 (S03). The display control unit 172 determines whether or not the acquired content can be displayed based on the calculated virtual position (S04).

[0062] If the system determines that the content can be displayed based on the virtual position (YES in S04), the display control unit 172 overlays the acquired content onto the image captured by the imaging unit 12 and displays it (S05). Then, the information processing device 1 terminates the process.

[0063] If the display control unit 172 determines that the content cannot be displayed based on the virtual position (NO in S04), it displays that the content is outside the display area (S06). In this case, the display control unit 172 displays a screen indicating that the content can be displayed by, for example, moving to a different location, changing the camera direction, or changing the zoom level. Then, the information processing device 1 terminates processing.

[0064] [Effects of Information Processing Device 1] The information processing device 1 controls the display of content based on the magnification of the imaging unit 12, allowing users to experience AR content even when they are not at the location where the AR content is displayed based on location information.

[0065] Furthermore, this invention will make it possible to contribute to Goal 9 of the United Nations-led Sustainable Development Goals (SDGs), "Build resilient infrastructure, promote inclusive and sustainable industrialization and foster innovation."

[0066] Although the present invention has been described above using embodiments, the technical scope of the present invention is not limited to the scope described in the above embodiments, and various modifications and changes are possible within the scope of its gist. For example, all or part of the apparatus can be configured by functionally or physically distributing and integrating in any unit. Furthermore, new embodiments resulting from any combination of multiple embodiments are also included in the embodiments of the present invention. The effects of the new embodiments resulting from the combinations are combined with the effects of the original embodiments. [Explanation of Symbols]

[0067] 1. Information Processing Device 2. Content Management Device 11 Communications Department 12 Imaging Unit 13 Location information acquisition unit 14 Display section 15 Operation reception section 16 Memory section 17 Control Unit 171 Content Acquisition Section 172 Display Control Unit 173 Virtual position calculation unit

Claims

1. An information processing device that displays Augmented Reality (AR) content according to location, Imaging unit, A location information acquisition unit that acquires location information indicating the location of the information processing device, A content acquisition unit acquires AR content and a displayable area indicating the range of locations of the information processing device capable of displaying the AR content, in association with the location indicated by the aforementioned location information. A virtual position calculation unit calculates a virtual position, which is a hypothetical position of the information processing device assuming that it has moved from its current position to the optical axis direction of the imaging unit, based on the position of the information processing device indicated by the position information and the magnification ratio, which is the ratio of the field of view included in the image captured by the imaging unit to the reference field of view, which is the widest field of view among the field of view that the imaging unit can capture. If the virtual position is included in the range within which the AR content indicated by the displayable area can be displayed, a display control unit causes the AR content to be superimposed on the image and displayed on the display unit of the information processing device, An information processing device having

2. The content acquisition unit further acquires the position for displaying the AR content, in association with the AR content. The virtual position calculation unit calculates the virtual position based on the position where the AR content is displayed, such that the larger the magnification, the closer the virtual position is to the position where the AR content is displayed from the position indicated by the position information. The information processing apparatus according to claim 1.

3. The virtual position calculation unit determines the virtual position as a point on the straight line connecting the position indicated by the position information and the position where the AR content is displayed, where the image obtained by capturing the position where the AR content is displayed at the magnification from the position indicated by the position information and the image obtained by capturing the position where the AR content is displayed at the reference field of view of the imaging unit from the virtual position are captured at the same size. The information processing apparatus according to claim 2.

4. The content acquisition unit acquires a plurality of AR contents and the displayable area of ​​each of the plurality of AR contents. The display control unit displays the AR content that can be displayed at the virtual position from among the plurality of AR content. The information processing apparatus according to claim 1.

5. If, after displaying the first AR content among the plurality of AR content, the virtual position becomes located outside the range in which the first AR content can be displayed, the display control unit will display a second AR content from among the plurality of AR content that is displayable at the virtual position and is different from the first AR content. The information processing apparatus according to claim 3.

6. A method for displaying AR content corresponding to the location of a computer having an imaging unit, which is performed by the computer, A location information acquisition step to acquire location information indicating the location of the computer, A content acquisition step involves acquiring AR content and a displayable area indicating the range of locations on the computer where the AR content can be displayed, in association with the location indicated by the location information. A virtual position calculation step calculates a virtual position, which is a hypothetical position when the computer moves in the optical axis direction from its current position, based on the position of the computer indicated by the position information and the magnification ratio, which is the ratio of the field of view included in the image captured by the imaging unit to the reference field of view, which is the widest field of view among the fields of view that the imaging unit can capture. If the virtual position is included in the range in which the AR content indicated by the displayable area can be displayed, a display control step of superimposing the AR content onto the image and displaying it on the computer's display unit, An information processing method having

7. A program for displaying AR content corresponding to the location of a computer having an imaging unit, A location information acquisition step to acquire location information indicating the location of the computer, A content acquisition step involves acquiring AR content and a displayable area indicating the range of locations on the computer where the AR content can be displayed, in association with the location indicated by the location information. A virtual position calculation step calculates a virtual position, which is a hypothetical position when the computer moves in the optical axis direction from its current position, based on the position of the computer indicated by the position information and the magnification ratio, which is the ratio of the field of view included in the image captured by the imaging unit to the reference field of view, which is the widest field of view among the fields of view that the imaging unit can capture. If the virtual position is included in the range in which the AR content indicated by the displayable area can be displayed, a display control step of superimposing the AR content onto the image and displaying it on the computer's display unit, A program to execute.

Images