Image display method, information processing apparatus, and program

Abstract

To display an image in an appropriate position over an object placed in a real space.SOLUTION: An image display method includes: receiving, from a user, a setting operation of setting a first face of an object placed in a real space as a face on which a projection image is to be virtually displayed; and displaying a first superposition image resulting from superposing a set image indicative of a plane including the first face over a captured image obtained by capturing the object.SELECTED DRAWING: Figure 12

Description

Technical Field

[0001] The present invention relates to an image display method, an information processing apparatus, and a program.

Background Art

[0002] So-called AR (Augmented Reality) technology has been developed to assist in obtaining visual information of a user by superimposing CG (Computer Graphics) or the like on a captured image of the real space and displaying it. For example, Patent Document 1 discloses an image processing apparatus that identifies the position of an object based on an input image obtained by imaging an object in the real space using an imaging device and feature data representing the characteristics of the appearance of the object. The image processing apparatus determines the position of the image to be superimposed on the input image based on the identified position of the object.

Prior Art Documents

Patent Documents

[0003]

Patent Document 1

Summary of the Invention

Problems to be Solved by the Invention

[0004] In the image processing apparatus described in Patent Document 1, when the feature data regarding the object to be imaged is not stored, there is a problem that the position of the object cannot be identified, so that the superimposed image is not arranged at an appropriate position.

Means for Solving the Problems

[0005] One aspect of the image display method according to the present invention includes receiving from a user a setting operation for setting a first surface of an object located in the real space as a surface on which a projection image is virtually displayed, and displaying a first superimposed image in which a setting image indicating a plane including the first surface is superimposed on a captured image obtained by imaging the object.

[0006] One embodiment of the information processing apparatus according to the present invention comprises a processing apparatus which performs the following: receiving a setting operation from a user to set a first surface of an object located in real space as the surface on which a projected image is virtually displayed; and outputting a first superimposed image to a display device, in which a setting image showing a plane including the first surface is superimposed on an image obtained by imaging the object.

[0007] One aspect of the program according to the present invention causes a processing device to receive a setting operation from a user to set a first surface of an object located in real space as the surface on which a projected image is virtually displayed, and to output a first superimposed image to a display device, in which a setting image showing a plane including the first surface is superimposed on an image obtained by imaging the object. [Brief explanation of the drawing]

[0008] [Figure 1] This is a schematic diagram illustrating the space in which object Ob1 is placed. [Figure 2] This is a schematic diagram illustrating the first superimposed image GM11. [Figure 3] This is a schematic diagram illustrating the second superimposed image GM21. [Figure 4] This is a schematic diagram illustrating the first superimposed image GM12. [Figure 5] This is a schematic diagram illustrating the detection process. [Figure 6] This is a schematic diagram illustrating the setup process. [Figure 7] This is a schematic diagram illustrating the display of the first superimposed image GM11. [Figure 8] This is a schematic diagram illustrating the monitor image GC2. [Figure 9] This is a schematic diagram to explain the notification image GN1. [Figure 10] This is a block diagram showing the configuration of smartphone 1 according to the embodiment. [Figure 11]This is a block diagram showing the configuration of the storage device 10 according to the embodiment. [Figure 12] This is a flowchart illustrating the operation of smartphone 1 according to the embodiment. [Figure 13] This is a schematic diagram illustrating the second superimposed image GM23. [Figure 14] This is a schematic diagram illustrating the first superimposed image GM13. [Modes for carrying out the invention]

[0009] Hereinafter, preferred embodiments of the present invention will be described with reference to the attached drawings. Note that the dimensions and scale of each part in the drawings may differ from those of the actual parts, and some parts are shown schematically for ease of understanding. Furthermore, the scope of the present invention is not limited to these forms unless otherwise stated in the following description. In addition, in this specification and the claims, when a numerical range is expressed using "Φ~Ψ" (where Φ and Ψ are both numerical values), the range includes the numerical values ​​of the upper limit (Ψ) and the lower limit (Φ). Furthermore, the units of the upper limit (Ψ) and the lower limit (Φ) are the same.

[0010] 1. Embodiment In this embodiment, a smartphone is used as an example to illustrate the image display method, information processing device, and program according to the present invention, by superimposing an image showing a plane including a side surface onto an image obtained by imaging an object having a side surface set as a surface on which a predetermined image is virtually displayed.

[0011] 1.1. Overview of Smartphones The following describes the overview of the smartphone 1 according to this embodiment, with reference to Figures 1 to 4. Figure 1 is a schematic diagram illustrating the real space in which object Ob1 is placed. Figures 2 to 4 are schematic diagrams illustrating the images displayed on the touch panel 16 of the smartphone 1.

[0012] The object Ob1 is a rectangular parallelepiped object having a surface W2, a surface W3, and a surface W4. The surface W2 is one of the bottom surfaces of the object Ob1 and is a surface that does not contact the floor surface WF. The surface W3 is one of the side surfaces of the object Ob1 and is a surface parallel to the wall surface W1. The surface W4 is one of the side surfaces of the object Ob1 and is a surface perpendicular to the wall surface W1. Also, the object Ob1 is in contact with the wall surface W1 through the side surface opposite to the surface W3. The plane H1 is a conceptual plane including the surface W3. That is, since the plane H1 does not exist in the real space, it cannot be directly visually recognized by the user U. Note that the object Ob1 is not limited to a rectangular parallelepiped as long as it has at least one plane, and may have other shapes.

[0013] In the present embodiment, the wall surface W1, the surface W3, and the plane H1 are parallel to the Z-axis and the X-axis. Here, the Z-axis is an axis parallel to the vertical direction. Also, the Z-axis is an axis perpendicular to the floor surface WF. Among the directions parallel to the Z-axis, the vertical direction is the -Z direction, and the direction opposite to the -Z direction is the +Z direction. The X-axis is an axis perpendicular to the Z-axis and is an axis parallel to the wall surface W1, the surface W3, and the plane H1. Also, the X-axis is an axis perpendicular to the surface W4. Among the directions parallel to the X-axis, the direction from the surface W4 to the side surface opposite to the surface W4 is the -X direction, and the direction opposite to the -X direction is the +X direction. Also, an axis perpendicular to the Z-axis and the X-axis is referred to as the "Y-axis". The Y-axis is an axis perpendicular to the wall surface W1, the surface W3, and the plane H1. Among the directions parallel to the Y-axis, the direction from the surface W3 to the wall surface W1 is the +Y direction, and the direction opposite to the +Y direction is the -Y direction.

[0014] User U performs an operation of setting the surface W3 as a surface on which a projected image GP1 described later is virtually displayed, with the smartphone 1 in contact with the surface W3 of the object Ob1. Specifically, User U performs this operation with the imaging device 18 provided in the smartphone 1 in contact with the surface W3. In this embodiment, "virtually display" means displaying, on the smartphone 1, an image obtained by superimposing another image on an image of the real space captured by the imaging device 18 of the smartphone 1. In this embodiment, the operation of setting the surface W3 as a surface on which the projected image GP1 is virtually displayed may be referred to as a "setting operation". The smartphone 1 receives a setting operation from User U while being positioned on the surface W3 of the object Ob1. The setting operation may be, for example, an operation of long-pressing the touch panel 16 for a predetermined time or longer. The setting operation may also be an operation on a virtual operator indicating the setting operation, such as an image of a confirmation button displayed on the touch panel 16.

[0015] Figure 2 is a schematic diagram for explaining the first superimposed image GM11. The first superimposed image GM11 is an image displayed on the touch panel 16 when the smartphone 1 receives a setting operation from User U while being positioned on the surface W3 of the object Ob1. The first superimposed image GM11 is an image obtained by superimposing a setting image GV1 on an imaging image GC1 obtained by imaging the object Ob1 with the imaging device 18. The imaging image GC1 includes an image GOb1 and an image GWF. The image GOb1 includes an image GW3 and an image GW4. The image GW3 is an image showing the surface W3. The image GW4 is an image showing the surface W4. The image GWF is an image showing the floor surface WF. The setting image GV1 is an image showing a plane H1 including the surface W3. The setting image GV1 shows a virtual plane on which the projected image GP1 is virtually displayed.

[0016] Furthermore, since plane H1 is not directly visible to user U, it is preferable that part or all of the setting image GV1 is semi-transparent. This allows user U to compare their own field of view with the captured image GC1 included in the first superimposed image GM11. Also, setting image GV1 is an image that indicates the position of the surface on which the projected image GP1 is virtually displayed. In Figure 2, image GW3, which is an image showing surface W3, coincides with a part of setting image GV1. Specifically, one of the four edges of image GW3 coincides with a part of the bottom edge of setting image GV1. From this, user U can see that surface W3 is the surface on which the projected image GP1 is virtually displayed by checking setting image GV1.

[0017] After receiving a setting operation from user U, smartphone 1 acquires an image GC1 by capturing an image of object Ob1. Smartphone 1 then generates a first superimposed image GM11 by superimposing a setting image GV1 onto the captured image GC1. Finally, smartphone 1 displays the first superimposed image GM11 on the touch panel 16.

[0018] Figure 3 is a schematic diagram illustrating the second superimposed image GM21. The second superimposed image GM21 is an image in which the projected image GP1 is superimposed on the image GW3 contained in the captured image GC1.

[0019] When smartphone 1 receives an operation from user U to display the projected image GP1, it generates a second superimposed image GM21 by superimposing the projected image GP1 onto the captured image GC1. Then, smartphone 1 displays the second superimposed image GM21 on the touch panel 16. In other words, smartphone 1 can superimpose and display the projected image GP1 onto an image GW3 representing the surface W3. To put it another way, smartphone 1 can virtually display the projected image GP1 on the surface W3 using the touch panel 16. The second superimposed image GM21 may or may not include the setting image GV1. In this embodiment, the second superimposed image GM21 does not include the setting image GV1. If the second superimposed image GM21 includes the setting image GV1, user U can easily understand the positional relationship between the surface set by the setting operation and the projected image GP1. If the second superimposed image GM21 does not include the setting image GV1, the user U can be presented with a second superimposed image GM21 that is closer to how the image would actually appear when projected onto the surface W3. In this embodiment, the operation to display the projected image GP1 may be referred to as a "display operation." The display operation may be, for example, an operation to tap the setting image GV1 included in the first superimposed image GM11 displayed on the touch panel 16. The display operation may also be an operation to tap a virtual operator displayed on the touch panel 16 that accepts an instruction to display the projected image GP1.

[0020] Figure 4 is a schematic diagram illustrating the first superimposed image GM12. The first superimposed image GM12 is an image in which the setting image GV2 is superimposed on the captured image GC1. The setting image GV2 is an image that shows a plane different from plane H1, i.e., a plane that does not include plane W3. For example, if user U does not perform the setting operation correctly, specifically if the imaging device 18 of the smartphone 1 performs the setting operation at a position away from plane W3 in the -Y direction, the smartphone 1 will display the first superimposed image GM12 on the touch panel 16. That is, if the first superimposed image GM12, which includes the setting image GV2, is displayed on the touch panel 16, user U can confirm that the setting operation was not performed with the imaging device 18 positioned on plane W3. On the other hand, if the first superimposed image GM11, which includes the setting image GV1, is displayed on the touch panel 16 as shown in Figure 2, user U can confirm that the setting operation was performed with the imaging device 18 positioned on plane W3.

[0021] If the setting operation is not performed while the imaging device 18 is positioned on surface W3, surface W3 is not set as the surface on which the projected image GP1 will be virtually displayed. In other words, the smartphone 1 may not be able to superimpose the projected image GP1 onto the image GW3 representing surface W3. When the first superimposed image GM12 is displayed on the touch panel 16, the user U can superimpose the projected image GP1 onto the image GW3 by re-performing the setting operation while the smartphone 1 is in contact with surface W3 of object Ob1. In this embodiment, re-performing the setting operation may be referred to as a "re-setting operation".

[0022] 1.2. Flowchart for displaying the first superimposed image The following describes the process of displaying the first superimposed image GM11 on the touch panel 16, referring to Figures 5 to 9.

[0023] Figure 5 is a schematic diagram illustrating the detection operation. Smartphone 1 has a function to detect a flat surface of object Ob1 by using image processing to extract feature points from an image obtained by capturing a flat surface of object Ob1. When a surface of object Ob1 is detected, smartphone 1 sets the surface of object Ob1 as the surface on which the projected image GP1 will be virtually displayed. That is, when a surface W3 of object Ob1 is detected, smartphone 1 sets the surface W3 as the surface on which the projected image GP1 will be virtually displayed.

[0024] Smartphone 1 displays the captured image obtained by imaging object Ob1 on the touch panel 16 and receives an operation from user U to instruct it to detect the surface W3 of object Ob1. When an operation to instruct it to detect the surface W3 of object Ob1 is received, smartphone 1 attempts to detect the surface W3 based on the captured image. In this embodiment, the operation to instruct it to detect the surface W3 of object Ob1 may be referred to as a "detection operation". The detection operation may be, for example, an operation to tap the image showing object Ob1 included in the captured image displayed on the touch panel 16. The detection operation may also be an operation to tap a virtual operator displayed on the touch panel 16 that receives an instruction to execute a surface detection process.

[0025] Figure 8 is a schematic diagram illustrating the monitor image GC2. The monitor image GC2 is an image obtained by imaging object Ob1 with the imaging device 18. The monitor image GC2 includes image GOb1a and image GWFa. Image GOb1a includes image GW3a. Image GW3a is an image showing surface W3. Image GWFa is an image showing floor surface WF.

[0026] When extracting feature points from an image using image processing, generally, points whose position can be accurately determined are extracted as feature points. Specifically, feature points are preferably those where the amount of change in brightness values, etc., between adjacent pixels is large in both the vertical and horizontal directions of the image. Examples of areas where the amount of change in brightness values, etc., between adjacent pixels is large in both the vertical and horizontal directions of the image include corner-shaped parts in the image. As can be seen from Figure 8, since the monitor image GC2 does not contain corner-shaped parts, there is a high possibility that feature point extraction by image processing will not be performed accurately. Therefore, when the user U performs a detection operation while the monitor image GC2 is displayed on the touch panel 16, the smartphone 1 may not be able to detect the surface W3.

[0027] If surface W3 is not detected, smartphone 1 prompts user U to perform a setting operation. For example, smartphone 1 may display a message on the touch panel 16 to prompt user U to perform a setting operation. Figure 9 is a schematic diagram illustrating notification image GN1. Notification image GN1 is an image containing a message prompting user U to perform a setting operation. Displaying notification image GN1 on the touch panel 16 is, in other words, outputting a notification regarding the setting operation.

[0028] Figure 6 is a schematic diagram illustrating the setting operation. User U sets surface W3 as the surface on which the projected image GP1 will be virtually displayed by pressing and holding the touch panel 16 for a predetermined time or longer while the imaging device 18 of the smartphone 1 is in contact with surface W3. When the smartphone 1 receives the setting operation from user U, it sets surface W3 as the surface on which the projected image GP1 will be virtually displayed. The smartphone 1 also sets the position of the imaging device 18 at the time the setting operation is received as the reference position. The smartphone 1 also sets the optical axis of the imaging device 18 at the time the setting operation is received as the reference axis. Hereafter, the optical axis of the imaging device 18 at the time the smartphone 1 receives the setting operation will be referred to as "optical axis L11". Optical axis L11 is an axis perpendicular to surface W3 and parallel to the Y axis. The time when the smartphone 1 receives the setting operation will be referred to as "the first time point".

[0029] Figure 7 is a schematic diagram illustrating the display of the first superimposed image GM11. After receiving the setting operation, the smartphone 1 controls the imaging device 18 to capture an image of object Ob1 from a position away from the plane W3 in the -Y direction, thereby acquiring the captured image GC1. The point in time when the imaging device 18 acquires the captured image GC1 is referred to as the "second point in time". The displacement of the imaging device 18 between the first point in time and the second point in time is referred to as the "displacement DP1". Here, "displacement" refers to a vector representing the change in the position of the imaging device 18. The optical axis of the imaging device 18 at the second point in time is referred to as the "optical axis L12". Based on the displacement DP1 and the change in the orientation of the imaging device 18 between the first and second points in time, the smartphone 1 generates a setting image GV1. The smartphone 1 then generates the first superimposed image GM11 by superimposing the setting image GV1 onto the captured image GC1. The smartphone 1 then displays the first superimposed image GM11 on the touch panel 16.

[0030] The change in the orientation of the imaging device 18 is represented, for example, by a three-dimensional rotation angle with the imaging device 18 as the origin, the optical axis of the imaging device 18 as the roll axis, the axis parallel to the short side of the smartphone 1 as the pitch axis, and the axis parallel to the long side of the smartphone 1 as the yaw axis. Specifically, the change in the orientation of the imaging device 18 is represented by the roll angle, pitch angle, and yaw angle at the time the smartphone 1 images object Ob1, assuming that the roll angle, pitch angle, and yaw angle at the time the smartphone 1 accepts the setting operation are all set to 0°. In other words, the change in the orientation of the imaging device 18 between the first time point and the second time point is represented by the changes in the roll angle, pitch angle, and yaw angle when the optical axis of the imaging device 18 changes from optical axis L11 to optical axis L12.

[0031] 1.3. Smartphone Configuration and Functions The configuration and functions of the smartphone 1 according to this embodiment will be described below with reference to Figures 10 and 11.

[0032] Figure 10 is a block diagram showing the configuration of a smartphone 1 according to an embodiment. The smartphone 1 includes a storage device 10 for storing various information, a processing device 12 for controlling the operation of the smartphone 1, a sensor 14 for outputting signals based on changes in the position of the smartphone 1 and signals based on changes in the orientation of the smartphone 1, a touch panel 16 for displaying various images and accepting operations from user U, and an imaging device 18 for imaging an object Ob1. The processing device 12 has the functions of a display control unit 120, an imaging control unit 121, an input management unit 122, an image generation unit 123, a detection unit 124, and a spatial management unit 125. The touch panel 16 includes a display unit 160 and an input unit 161. The imaging device 18 includes an imaging lens 180 and an image sensor 181.

[0033] The storage device 10 is composed of, for example, volatile memory such as RAM and non-volatile memory such as ROM. Here, RAM is an abbreviation for Random Access Memory, and ROM is an abbreviation for Read Only Memory.

[0034] Figure 11 is a block diagram showing the configuration of a storage device 10 according to an embodiment. The non-volatile memory of the storage device 10 stores a program 100 that defines the operation of the smartphone 1, spatial information 101 including information representing the position and shape of a detected object, position information 102 based on changes in the position of the imaging device 18, posture information 103 based on changes in the posture of the imaging device 18, projection image information 104 representing the projection image GP1, imaging information 105 representing the captured image GC1, monitor information 106 representing the monitor image GC2, setting image information 107 representing the setting image GV1, notification information 108 representing the notification image GN1, and superimposed image information 110. The superimposed image information 110 includes first superimposed image information 111 representing the first superimposed image GM11 and second superimposed image information 112 representing the second superimposed image GM21.

[0035] Furthermore, the volatile memory of the storage device 10 is used by the processing unit 12 as a work area when executing the program 100.

[0036] Furthermore, part or all of the storage device 10 may be located on an external storage device or an external server. Also, part or all of the various information stored in the storage device 10 may be stored in the storage device 10 in advance, or it may be obtained from an external storage device or an external server.

[0037] The processing unit 12 is comprised of one or more CPUs. However, the processing unit 12 may also include a programmable logic device such as an FPGA instead of a CPU, or in addition to a CPU. Here, CPU is an abbreviation for Central Processing Unit, and FPGA is an abbreviation for Field-Programmable Gate Array.

[0038] The processing unit 12 functions as the display control unit 120, imaging control unit 121, input management unit 122, image generation unit 123, detection unit 124, and spatial management unit 125 shown in Figure 10, by having its CPU and other components execute program 100 and operating according to program 100.

[0039] The display control unit 120 controls the display unit 160 of the touch panel 16 to display various images. In other words, the display control unit 120 outputs various images to the display unit 160 of the touch panel 16.

[0040] The imaging control unit 121 controls the imaging device 18 to image object Ob1. The imaging control unit 121 then acquires information representing the imaging result of image capture of object Ob1 from the imaging device 18. The imaging control unit 121 also stores the information acquired from the imaging device 18 in the storage device 10. In this embodiment, the imaging control unit 121 acquires imaging information 105 and monitor information 106 as information representing the imaging result of image capture of object Ob1. The imaging control unit 121 then stores the acquired imaging information 105 and monitor information 106 in the storage device 10.

[0041] Furthermore, the imaging control unit 121 generates position information 102 and attitude information 103 based on the signal output from the sensor 14. In this embodiment, the position information 102 is information representing the displacement DP1. The attitude information 103 is information representing the change in the attitude of the imaging device 18 between a first time point and a second time point. The imaging control unit 121 stores the generated position information 102 and attitude information 103 in the storage device 10.

[0042] The input management unit 122 controls the input unit 161 of the touch panel 16 to acquire information representing the content of the operation received from the user U. The input management unit 122 also performs various judgments related to the operation received from the user U.

[0043] The detection unit 124 uses image processing to extract feature points from the captured image obtained by imaging the object Ob1, thereby detecting the flat surfaces of the object Ob1. In other words, the detection unit 124 determines whether or not the captured image contains an image showing the flat surfaces of the object Ob1. In this embodiment, the detection unit 124 determines whether or not the monitor image GC2 contains an image showing the surface W3 of the object Ob1. The captured image that is the subject of the determination of whether or not it contains an image showing the flat surfaces of the object Ob1 is sometimes referred to as the "monitor image".

[0044] In the function related to the extraction of feature points from captured images, known image processing techniques may be used. An example of a known image processing technique related to feature point extraction is the algorithm known as "AKAZE". Detailed technical explanations regarding feature point extraction are omitted in this specification.

[0045] The spatial management unit 125 manages spatial information 101, which includes information representing the position and shape of objects placed in real space. For example, the spatial management unit 125 updates the spatial information 101 based on the position and shape of the faces of the detected object Ob1.

[0046] Spatial information 101 includes information about the position where the smartphone 1 received the setting operation. Based on the signal output from the sensor 14, the spatial management unit 125 updates the spatial information 101 so that the position of the smartphone 1 at the first time point in time when the setting operation was received from the user U becomes the reference position. More specifically, the spatial management unit 125 updates the spatial information 101 so that the position of the imaging device 18 at the first time point becomes, for example, the origin. In addition, by updating the spatial information 101, the spatial management unit 125 sets the plane on which the projected image GP1 is virtually displayed so as to include the position of the imaging device 18 at the first time point. That is, if the imaging device 18 is located on the plane W3 of object Ob1 when the user U receives the setting operation, the spatial management unit 125 sets the plane W3 as the plane on which the projected image GP1 is virtually displayed. The plane on which the projected image GP1 is virtually displayed is set, for example, perpendicular to the optical axis L11 of the imaging device 18 at the first time point.

[0047] The image generation unit 123 generates setting image information 107 based on spatial information 101, position information 102, and orientation information 103. In other words, the image generation unit 123 generates setting image GV1 based on spatial information 101, position information 102, and orientation information 103. The image generation unit 123 also generates first superimposed image information 111 based on setting image information 107 and imaging information 105. In other words, the image generation unit 123 generates first superimposed image GM11 based on setting image GV1 and captured image GC1. The image generation unit 123 also generates second superimposed image information 112 based on projection image information 104 and imaging information 105. In other words, the image generation unit 123 generates second superimposed image GM21 based on projection image GP1 and captured image GC1.

[0048] Sensor 14 comprises an accelerometer and a gyroscope. Sensor 14 detects changes in the position of smartphone 1 and changes in the orientation of smartphone 1. Sensor 14 then outputs signals based on the changes in the position of smartphone 1 and signals based on the changes in the orientation of smartphone 1 to processing unit 12.

[0049] The touch panel 16 is a device that integrates a display unit 160 for displaying images and an input unit 161 for receiving input operations from a user U. The display unit 160 is a so-called display panel and is composed of, for example, a liquid crystal panel or an organic EL panel. Here, EL is an abbreviation for Electro-Luminescence. The display unit 160 displays various images under the control of the display control unit 120. The input unit 161 is composed of, for example, a transparent sheet-like contact sensor. The input unit 161 is provided so as to cover the display unit 160. The input unit 161 detects the touch position using an object in contact with the input unit 161 and the capacitance identified by the input unit 161, and outputs data indicating the detected touch position to the processing unit 12.

[0050] The imaging device 18 is a camera that includes, for example, an imaging lens 180 for focusing light and an image sensor 181 that converts the focused light into an electrical signal. Optical axes L11 and L12 are the optical axes of the imaging lens 180. The image sensor 181 is an image sensor, such as a CCD or CMOS. Here, CCD is an abbreviation for Charge Coupled Device, and CMOS is an abbreviation for Complementary Metal Oxide Semiconductor. The imaging device 18 acquires an image under the control of the imaging control unit 121. The imaging device 18 also outputs information representing the acquired image to the processing unit 12. For example, the imaging device 18 acquires an image GC1 and a monitor image GC2. The imaging device 18 also outputs imaging information 105 representing the image GC1 and monitor information 106 representing the monitor image GC2 to the processing unit 12.

[0051] 1.4. Smartphone operation Figure 12 is a flowchart illustrating the operation of the smartphone 1 according to this embodiment. The series of operations shown in the flowchart begin, for example, when the power of the smartphone 1 is turned ON and the touch panel 16 receives an input operation from user U related to the start of operation.

[0052] In step S101, the imaging control unit 121 controls the imaging device 18 to image object Ob1. The imaging control unit 121 then obtains monitor information 106 from the imaging device 18, which represents the imaging result of image capture of object Ob1. The imaging control unit 121 also stores the obtained monitor information 106 in the storage device 10.

[0053] In step S102, the display control unit 120 controls the display unit 160 of the touch panel 16 to display the monitor image GC2 indicated by the monitor information 106.

[0054] In step S103, the input management unit 122 determines whether the input unit 161 has received a detection operation from user U. If the input unit 161 has received a detection operation from user U, that is, if the result in step S103 is YES, the input management unit 122 proceeds to step S104. If the input unit 161 has not received a detection operation from user U, that is, if the result in step S103 is NO, the input management unit 122 proceeds to step S101.

[0055] If the input unit 161 has not received a detection operation from user U, the smartphone 1 continues to have the imaging device 18 capture an image of object Ob1 until the input unit 161 receives a detection operation from user U. The smartphone 1 then continues to update the monitor image displayed on the touch panel 16. In this embodiment, it is assumed that user U performs a detection operation when the monitor image GC2 is displayed on the touch panel 16.

[0056] In step S104, the detection unit 124 determines whether the monitor image GC2 contains an image showing the surface W3 of object Ob1. If it determines that the monitor image GC2 contains an image showing the surface W3 of object Ob1, that is, if the result in step S104 is YES, the detection unit 124 proceeds to step S107. If it determines that the monitor image GC2 does not contain an image showing the surface W3 of object Ob1, that is, if the result in step S104 is NO, the detection unit 124 proceeds to step S105.

[0057] The determination in step S104 is performed by the detection unit 124, which extracts multiple feature points from the monitor image, and is based on the number and position of the extracted feature points. Therefore, if feature points cannot be correctly extracted from the monitor image, the detection unit 124 may determine that the monitor image does not contain images showing flat surfaces, even if the monitor image contains such images.

[0058] In step S105, the display control unit 120 controls the display unit 160 of the touch panel 16 to display the notification image GN1 indicated by the notification information 108.

[0059] In step S106, the input management unit 122 determines whether the input unit 161 has received a setting operation from user U. If the input unit 161 has received a setting operation from user U, that is, if the result in step S106 is YES, the input management unit 122 proceeds to step S107. If the input unit 161 has not received a setting operation from user U, that is, if the result in step S106 is NO, the input management unit 122 repeats the process in step S106.

[0060] If the input unit 161 has not received a setting operation from user U, the input management unit 122 repeats the determination in step S106 until the input unit 161 receives a setting operation from user U. Preferably, the notification image GN1 continues to be displayed on the touch panel 16 until the input unit 161 receives a setting operation from user U. This allows user U to understand what action to take by checking the message contained in the notification image GN1.

[0061] In step S107, the spatial management unit 125 updates the spatial information 101 to set surface W3 as the surface on which the projected image GP1 is virtually displayed.

[0062] In step S108, the imaging control unit 121 controls the imaging device 18 to image object Ob1. The imaging control unit 121 then obtains imaging information 105 from the imaging device 18, which represents the imaging result of image capture of object Ob1. The imaging control unit 121 also stores the obtained imaging information 105 in the storage device 10.

[0063] In step S109, the image generation unit 123 generates setting image information 107 representing the setting image GV1 based on the spatial information 101, position information 102, and orientation information 103. The image generation unit 123 also generates first superimposed image information 111 representing the first superimposed image GM11 based on the setting image information 107 and the imaging information 105.

[0064] In step S110, the display control unit 120 controls the display unit 160 of the touch panel 16 to display the first superimposed image GM11 indicated by the first superimposed image information 111.

[0065] In step S111, the input management unit 122 determines whether the input unit 161 has received a reset operation from user U. If the input unit 161 has received a reset operation from user U, that is, if the result in step S111 is YES, the input management unit 122 proceeds to step S107. If the input unit 161 has not received a reset operation from user U, that is, if the result in step S111 is NO, the input management unit 122 proceeds to step S112.

[0066] If, during the setup operation in step S106, the smartphone 1 and the surface W3 of object Ob1 are separated, the process up to step S110 will be executed, and the first superimposed image GM12 will be displayed. If the image displayed in step S110 is the first superimposed image GM12, user U can understand that the surface set as the surface on which the projected image GP1 is virtually displayed is not surface W3. In such a case, user U can set surface W3 as the surface on which the projected image GP1 is virtually displayed by performing the setup operation with the smartphone 1 in contact with surface W3 of object Ob1.

[0067] In step S112, the input management unit 122 determines whether the input unit 161 has received a display operation from user U. If the input unit 161 has received a display operation from user U, that is, if the result in step S112 is YES, the input management unit 122 proceeds to step S113. If the input unit 161 has not received a display operation from user U, that is, if the result in step S112 is NO, the input management unit 122 proceeds to step S108.

[0068] If the input unit 161 has not received a display operation from user U, the smartphone 1 continues to have the imaging device 18 capture an image of object Ob1 until the input unit 161 receives a display operation from user U. The smartphone 1 then continues to update the image displayed on the touch panel 16. In this embodiment, it is assumed that user U performs a display operation when the first superimposed image GM11 is displayed on the touch panel 16.

[0069] In step S113, the image generation unit 123 generates second superimposed image information 112 representing the second superimposed image GM21 based on the projected image information 104 and the imaging information 105.

[0070] In step S114, the display control unit 120 controls the display unit 160 of the touch panel 16 to display the second superimposed image GM21 indicated by the second superimposed image information 112.

[0071] If user U performs a setting operation in step S107, the projected image GP1 is displayed based on the position of image GW3 corresponding to the location on surface W3 where the imaging device 18 was located at the time the smartphone 1 received the setting operation. If the detection unit 124 detects surface W3 in step S104, the smartphone 1 is set to virtually display the projected image GP1 on part or all of surface W3 based on the detected position and shape of surface W3.

[0072] After the processing in step S114 is completed, the processing unit 12 completes the series of operations shown in the flowchart of Figure 12.

[0073] Based on the above, according to the embodiment, the smartphone 1 can set surface W3 as the surface on which the projected image GP1 is virtually displayed by receiving a setting operation from the user U while positioned on surface W3 of object Ob1. In other words, even if the smartphone 1 cannot detect surface W3, it can superimpose and display the projected image GP1 on the image GW3 representing surface W3.

[0074] Furthermore, according to this embodiment, the smartphone 1 can indicate the position of the surface on which the projected image GP1 is virtually displayed by displaying the first superimposed image GM11 which includes the setting image GV1. In other words, the user U can confirm whether or not the setting operation was performed while the imaging device 18 was positioned on surface W3.

[0075] As described above, the image display method according to the embodiment includes receiving a setting operation from a user U to set the surface W3 of an object Ob1 located in real space as the surface on which the projected image GP1 is virtually displayed, and displaying a first superimposed image GM11 in which a setting image GV1 showing a plane H1 including surface W3 is superimposed on an captured image GC1 obtained by imaging the object Ob1.

[0076] Furthermore, the smartphone 1 according to this embodiment includes a processing unit 12, which receives a setting operation from user U to set the surface W3 of an object Ob1 located in real space as the surface on which the projected image GP1 is virtually displayed, and outputs a first superimposed image GM11 to the touch panel 16, in which a setting image GV1 showing a plane H1 including surface W3 is superimposed on an captured image GC1 obtained by imaging the object Ob1.

[0077] Furthermore, the program 100 according to the embodiment causes the processing unit 12 to receive a setting operation from user U to set the surface W3 of an object Ob1 located in real space as the surface on which the projected image GP1 is virtually displayed, and to output a first superimposed image GM11 to the touch panel 16, in which a setting image GV1 showing a plane H1 including surface W3 is superimposed on the captured image GC1 obtained by imaging the object Ob1.

[0078] In other words, the image display method, smartphone 1, and program 100 according to this embodiment can set the position on the surface where the projected image GP1 is virtually displayed, or in other words, the position on the captured image GC1 overlaid with the projected image GP1, based on the operation of user U. As a result, smartphone 1 can display the projected image GP1 at the position specified by user U without using feature data representing the external features of the object Ob1 that is the subject, thus enabling the projected image GP1 to be displayed in an appropriate position. Furthermore, the smartphone 1 according to this embodiment can indicate the position of the surface on which the projected image GP1 will be virtually displayed by displaying a first superimposed image GM11 that includes the setting image GV1. This allows the user U to confirm whether the setting operation was performed as intended before the projected image GP1 is displayed.

[0079] In this embodiment, smartphone 1 is an example of an "information processing device," program 100 is an example of a "program," object Ob1 is an example of an "object," surface W3 is an example of a "first surface," projected image GP1 is an example of a "projected image," user U is an example of a "user," captured image GC1 is an example of an "captured image," plane H1 is an example of a "plane including the first surface," setting image GV1 is an example of a "setting image," first superimposed image GM11 is an example of a "first superimposed image," processing device 12 is an example of an "processing device," and touch panel 16 is an example of a "display device."

[0080] Furthermore, in the image display method according to the embodiment, the captured image GC1 is acquired by the imaging device 18, the setting operation is performed with the imaging device 18 positioned on the surface W3 of the object Ob1, and the method further includes acquiring position information 102 representing the displacement DP1 of the imaging device 18 at a second time when the imaging device 18 acquired the captured image GC1, based on the position of the imaging device 18 at a first time when the setting operation is received, and posture information 103 representing the change in the posture of the imaging device 18 at a second time, based on the posture of the imaging device 18 at a first time, and the setting image GV1 is generated based on the position information 102 and posture information 103.

[0081] In other words, the smartphone 1 accepts setting operations from the user U while the imaging device 18 is positioned on the surface W3 of object Ob1. This allows the user U to easily and accurately set the surface on which the projected image GP1 is virtually displayed. Furthermore, the first superimposed image GM11 is generated by superimposing a setting image GV1, which is generated based on position information 102 and orientation information 103, onto the captured image GC1. In other words, the smartphone 1 can identify the position of a surface even when, for example, a surface from which it is difficult to extract feature points by image processing is set as the surface that the projected image GP1 virtually displays. As a result, the smartphone 1 can appropriately superimpose the setting image GV1 at a predetermined position.

[0082] In this embodiment, the imaging device 18 is an example of an "imaging device," the position information 102 is an example of "position information," the attitude information 103 is an example of "attitude information," and the displacement DP1 is an example of "displacement."

[0083] Furthermore, the image display method according to the embodiment further includes receiving a detection operation from user U instructing the detection of surface W3, and, if the detection operation is received and surface W3 is not detected, outputting a notification regarding setting operations.

[0084] In other words, user U can set the surface on which the projected image GP1 is virtually displayed by multiple means. This improves usability when setting the surface on which the projected image GP1 is virtually displayed.

[0085] In this embodiment, "outputting a notification regarding setting operations" is exemplified by displaying the notification image GN1 on the touch panel 16.

[0086] Furthermore, in the image display method according to the embodiment, when a detection operation is received, the method further includes determining whether or not the monitor image GC2 obtained by imaging the object Ob1 includes an image showing surface W3, and outputting a notification includes outputting a notification when it is determined that the monitor image GC2 does not include an image showing surface W3.

[0087] This allows user U to perform detection operations while confirming the surface on which they want to set the projected image GP1 as the surface on which it will be virtually displayed, using the monitor image.

[0088] In this embodiment, monitor image GC2 is an example of a "monitor image." Also, "images showing the first surface" include images GW3 and GW3a as examples.

[0089] Furthermore, the image display method according to the embodiment further includes receiving a display operation from user U to display the projected image GP1, and, when the display operation is received, displaying a second superimposed image GM21 in which the projected image GP1 is superimposed on a part or all of the area of ​​the image GW3 included in the captured image GC1.

[0090] When user U requests a display operation, smartphone 1 superimposes the projected image GP1 onto the captured image GC1. This allows user U to see how the projected image GP1 would appear if it were projected into real space, using the second superimposed image GM21.

[0091] In this embodiment, the second superimposed image GM21 is an example of a "second superimposed image".

[0092] 2. Variations The above embodiments can be modified in various ways. Specific examples of modifications are given below. Furthermore, two or more embodiments arbitrarily selected from the following examples can be combined as appropriate, within the bounds of mutual consistency. In the modified examples given below, for elements whose operation and function are equivalent to those in the above embodiments, the same reference numerals used in the above description will be reused, and detailed explanations of each will be omitted as appropriate.

[0093] 2.1. Variation 1 In the embodiments described above, an example was given in which a projected image is virtually displayed on a surface of an object. However, a virtual projector that projects the image may also be displayed along with the projected image.

[0094] Figure 13 is a schematic diagram illustrating the second superimposed image GM23. The second superimposed image GM23 is displayed on the touch panel 16 of the smartphone 1. The second superimposed image GM23 is an image obtained by superimposing the projection image GP1b and the projection device image GS1 onto the captured image GC3 obtained by capturing object Ob1 with the imaging device 18. The captured image GC3 includes images GOb1b and GWFb. Image GOb1b includes images GW3b and GW4b. Image GW3b is an image representing surface W3. Image GW4b is an image representing surface W4. Image GWFb is an image representing the floor surface WF. The projection device image GS1 is an image representing a virtual projector virtually placed in real space. The projection image GP1b is an image virtually projected onto surface W3 from the virtual projector represented by the projection device image GS1. In other words, the projected image GP1b is superimposed on the image GW3b representing surface W3. This allows for simulations of how the projected image would appear if, for example, a projector were placed in real space.

[0095] As explained above, in the image display method relating to Modification 1, the projected image GP1b is an image that is virtually projected from a virtual projector that is virtually placed in real space.

[0096] In other words, smartphone 1 can display a projected image GP1b that is virtually displayed on the surface of an object Ob1 placed in real space, and a virtual projector that virtually projects the projected image GP1b. This allows the user U of smartphone 1 to perform a simulation of how the projected image would be displayed if, for example, a projector were placed in real space.

[0097] In Modification Example 1, the projected image GP1b is an example of a "projected image".

[0098] 2.2. Variation Example 2 In the embodiments and modifications described above, examples were given in which the projected image is superimposed on an image representing surface W3, in other words, the projected image is virtually displayed on surface W3. However, the surface on which the projected image is virtually displayed may be a surface different from surface W3. That is, the smartphone 1 may accept setting operations from user U when the imaging device 18 is located on a surface different from surface W3. For example, the smartphone 1 may accept setting operations from user U when the imaging device 18 is located on surface W2 of object Ob1.

[0099] Figure 14 is a schematic diagram illustrating the first superimposed image GM13. The first superimposed image GM13 is the image displayed on the touch panel 16 when the smartphone 1 is positioned on the surface W2 of object Ob1 and a setting operation is received from user U. The first superimposed image GM13 is an image in which the setting image GV4 is superimposed on the captured image GC1. The setting image GV4 is an image showing the plane containing surface W2. Also, like the setting image GV1, the setting image GV4 is an image that indicates the position of the surface on which the projected image is virtually displayed. By checking the setting image GV4, user U can see that the surface on which the projected image is virtually displayed is surface W2. Note that since the captured image GC1 does not contain an image showing surface W2, it is preferable for user U to have the smartphone 1 acquire an captured image that contains an image showing surface W2 before performing the display operation. Furthermore, the object having the surface on which the projected image is virtually displayed may be, for example, the wall surface W1 itself. In this case, the smartphone 1 accepts setting operations from the user U while the imaging device 18 is positioned on the wall surface W1.

[0100] 2.3. Variation 3 In the embodiments and modifications described above, a smartphone 1 was used as an example of the information processing device according to the present invention. However, instead of a smartphone 1, a tablet terminal or a laptop computer with similar functions to a smartphone 1 may also be used.

[0101] 2.4. Variation 4 In the embodiments and modifications described above, an example was given in which, if surface W3 is not detected, a notification image GN1 containing a message prompting user U to perform a setting operation is displayed. However, the present invention is not limited to such embodiments. For example, instead of displaying a message, voice may be used to prompt user U to perform a setting operation. That is, a notification regarding the setting operation may be output using voice. [Explanation of symbols]

[0102] 1...Smartphone, 10...Storage device, 12...Processing device, 14...Sensor, 16...Touch panel, 18...Imaging device, 100...Program, 101...Spatial information, 102...Location information, 103...Orientation information, 104...Projection image information, 105...Imaging information, 106...Monitor information, 107...Setting image information, 110...Superimposed image information, 111...First superimposed image information, 112...Second superimposed image information, 120...Display control unit, 121...Imaging control unit, 122...Input management unit ,123...Image generation unit, 124...Detection unit, 125...Spatial management unit, 160...Display unit, 161...Input unit, 180...Imaging lens, 181...Image sensor, DP1...Displacement, GC1...Imagine image, GC2...Monitor image, GM11...First superimposed image, GM21...Second superimposed image, GN1...Notification image, GV1...Setting image, GP1...Projection image, GS1...Projection device image, L11...Optical axis, Ob1...Object, H1...Plane, U...User, W1...Wall surface, W3...Surface, WF...Floor surface.

Claims

1. receiving a setting operation from a user to set a first surface of an object located in real space as a surface on which a projected image is virtually displayed; displaying a first superimposed image in which a setting image showing a plane including the first surface is superimposed on a captured image obtained by capturing an image of the object; Including, How the image is displayed.

2. the captured image is acquired by an imaging device, the setting operation is performed in a state where the imaging device is located on the first surface of the object, acquiring position information representing a displacement of the imaging device at a second time point when the imaging device acquires the captured image, based on a position of the imaging device at a first time point when the setting operation is accepted, and acquiring orientation information representing a change in orientation of the imaging device at the second time point, based on an orientation of the imaging device at the first time point; the setting image is generated based on the position information and the posture information. The image display method according to claim 1 .

3. receiving a detection operation from the user instructing to detect the first surface; receiving the detection operation and, if the first surface is not detected, outputting a notification regarding the setting operation; Further comprising:

3. The image display method according to claim 1 or 2.

4. The method further includes, when the detection operation is received, determining whether or not an image showing the first surface is included in a monitor image obtained by capturing an image of the object; outputting the notification includes outputting the notification when it is determined that the monitor image does not include an image showing the first surface. The image display method according to claim 3.

5. receiving a display operation from the user to display the projection image; When the display operation is received, a second superimposed image in which the projected image is superimposed on a part or all of an area of ​​an image showing the first surface included in the captured image is displayed; Further comprising:

3. The image display method according to claim 1 or 2.

6. the projected image is an image virtually projected from a virtual projector that is virtually placed in the real space; The image display method according to claim 5.

7. a processing device; The processing device includes: receiving a setting operation from a user to set a first surface of an object located in real space as a surface on which a projected image is virtually displayed; outputting, to a display device, a first superimposed image in which a setting image indicating a plane including the first surface is superimposed on a captured image obtained by capturing an image of the object; To execute Information processing device.

8. The processing device receiving a setting operation from a user to set a first surface of an object located in real space as a surface on which a projected image is virtually displayed; outputting, to a display device, a first superimposed image in which a setting image indicating a plane including the first surface is superimposed on a captured image obtained by capturing an image of the object; Execute program.

Images