Information processing device, information processing method, and information processing program

Abstract

The present invention makes it possible to provide an information processing device, an information processing method, and an information processing program with which it is possible to improve convenience for a user in relation to a change in positioning of a virtual projection plane and / or a virtual projection device.　An information processing device provided with a processor, wherein the processor acquires first image data representing a first image obtained by image capture by an imaging device, acquires positioning data relating to the positioning of a virtual projection plane (204) and a virtual projection device (202) in a space indicated by the first image, acquires positioning change data relating to a change in positioning of the virtual projection plane (204) and / or the virtual projection device (202) in the first image, generates second image data representing a second image in which the virtual projection plane (204) and / or the virtual projection device (202) for which the positioning is changed on the basis of the positioning change data is displayed in the first image, and outputs the second image data to an output destination.

Description

Technical Field

[0001] The present invention relates to an information processing apparatus, an information processing method, and an information processing program.

Background Art

[0002] Patent Document 1 discloses a projection display device that visually and easily guides a user to an optimal installation position with respect to a screen. In the projection display device, a camera images in the projection direction, a screen determination unit detects a projection range of the screen based on the imaging result of the camera, and determines relative position information of the projection display device with respect to the screen. Further, an installation location guidance unit guides so that the projection display device can be installed at a position where a projection image can be projected within the projection plane of the screen, based on the position information determined by the screen determination unit. This guidance location installation unit is constituted by, for example, a plurality of LEDs and direction keys, and displays so that the current installation position of the projection display device can be visually recognized based on the installation position information determined by the screen determination unit. Then, while viewing this display, the user can easily perform position setting of the projection display device.

[0003] Patent Document 2 provides an augmented reality system that displays a virtual object input by a user himself / herself. When the user inputs a virtual object, the augmented reality system acquires information regarding the position and orientation of the augmented reality device, and records the acquired information regarding the position and orientation together with the information of the virtual object. When the user uses the augmented reality system and approaches a position and orientation regarding the virtual object input by himself / herself, the virtual object input by himself / herself is displayed superimposed on the real environment.

Prior Art Documents

Patent Documents

[0004]

Patent Document 1

Patent Document 2

[0005] One embodiment of the technology relating to the present disclosure provides an information processing device, an information processing method, and an information processing program that can improve user convenience regarding the reconfiguration of a virtual projection plane and / or a virtual projection device. [Means for solving the problem]

[0006] An information processing apparatus in one aspect of the present invention is an information processing apparatus comprising a processor, wherein the processor acquires first image data representing a first image obtained by imaging by an imaging device, acquires arrangement data relating to the arrangement of a virtual projection plane and a virtual projection device in the space shown by the first image, acquires arrangement change data relating to the arrangement change of the virtual projection plane and / or the virtual projection device in the first image, generates second image data representing a second image in which the virtual projection plane and / or the virtual projection device whose arrangement has been changed based on the arrangement change data is displayed in the first image, and outputs the second image data to an output destination.

[0007] An information processing method according to one aspect of the present invention is an information processing method using an information processing device, wherein the processor of the information processing device acquires first image data representing a first image obtained by imaging by an imaging device, acquires arrangement data relating to the arrangement of a virtual projection plane and a virtual projection device in the space shown by the first image, acquires arrangement change data relating to the arrangement change of the virtual projection plane and / or the virtual projection device in the first image, generates second image data representing a second image in which the virtual projection plane and / or the virtual projection device whose arrangement has been changed based on the arrangement change data is displayed in the first image, and outputs the second image data to an output destination.

[0008] An information processing program in one aspect of the present invention is an information processing program for an information processing device, which causes the processor of the information processing device to perform the following processes: acquire first image data representing a first image obtained by imaging by an imaging device; acquire arrangement data relating to the arrangement of a virtual projection plane and a virtual projection device in the space shown by the first image; acquire arrangement change data relating to the arrangement change of the virtual projection plane and / or the virtual projection device in the first image; generate second image data representing a second image in which the virtual projection plane and / or the virtual projection device, whose arrangement has been changed based on the arrangement change data, are displayed in the first image; and output the second image data to an output destination. [Effects of the Invention]

[0009] According to the present invention, it is possible to provide an information processing device, an information processing method, and an information processing program that can improve user convenience regarding the rearrangement of a virtual projection surface and / or a virtual projection device. [Brief explanation of the drawing]

[0010] [Figure 1] This is a schematic diagram showing the general configuration of the projection device 10, which is the target of installation support by the information processing device of the embodiment. [Figure 2] Figure 1 is a schematic diagram showing an example of the internal structure of the projection unit 1. [Figure 3] This is a schematic diagram showing the external configuration of the projection device 10. [Figure 4] Figure 3 is a schematic cross-sectional view of the optical unit 106 of the projection device 10 shown. [Figure 5] This figure shows an example of an information processing device 50 of the embodiment. [Figure 6] This figure shows an example of the hardware configuration of the information processing device 50 of the embodiment. [Figure 7] This diagram illustrates the projection device coordinate system CA, which is an example of the coordinate system of the virtual projection device 202. [Figure 8] This is another diagram illustrating the projection device coordinate system CA, which is an example of the coordinate system of the virtual projection device 202. [Figure 9] This is a diagram for explaining a projection plane coordinate system CB, which is an example of the coordinate system of the virtual projection plane 204. [Figure 10] This is a flowchart showing an example of the processing by the information processing apparatus 50 of the embodiment. [Figure 11] This is an example of an operation image displayed by the tablet, which is the information processing apparatus 50 of the embodiment. [Figure 12] This is another example of an operation image displayed by the smartphone, which is the information processing apparatus 50 of the embodiment. [Figure 13] This is another example of an operation image mainly using touch operations and displayed by the information processing apparatus 50 of the embodiment. [Figure 14] This is a diagram showing an operation of horizontally moving the virtual projection apparatus 202 in the operation image of Fig. 13(A). [Figure 15] This is a diagram showing an operation of vertically moving the virtual projection apparatus 202 in the operation image of Fig. 13(A). [Figure 16] This is a diagram showing an operation of rotating the virtual projection apparatus 202 in the operation image of Fig. 13(A). [Figure 17] This is a diagram showing an operation of rotating the virtual projection plane 204 in the operation image of Fig. 13(B). [Figure 18] This is a simulation diagram of the initial state in the virtual projection apparatus priority mode in the coordinate system of Fig. 7. [Figure 19] This is a diagram for explaining the movement of the virtual projection apparatus 202 in the left direction in Fig. 18. [Figure 20] This is a diagram for explaining the movement of the virtual projection apparatus 202 in the rear direction in Fig. 18. [Figure 21] This is a diagram for explaining the movement of the virtual projection apparatus 202 in the upper direction in Fig. 18. [Figure 22] This is a simulation diagram of the initial state in the virtual projection apparatus priority mode in the coordinate system of Fig. 8. [Figure 23] This is a simulation diagram of the initial state in the virtual projection apparatus priority mode in the coordinate system of Fig. 9. [Figure 24]Figure 23 illustrates the movement of the virtual projection plane 204 to the right. [Figure 25] This diagram illustrates the state of the virtual plane 201 on which the projection device is installed and the spatial coordinate system CC. [Figure 26] This is a diagram showing an image displayed on the touch panel 51, illustrating the virtual projection device 202 installed on the floor. [Figure 27] This diagram shows the virtual projection device 202 suspended from the ceiling, as displayed on the touch panel 51. [Figure 28] This figure shows the state with the shift range F1 displayed in virtual projection device priority mode. [Figure 29] Figure 28 shows a state in which the position of the virtual projection device 202 or virtual projection surface 204 is clipped at the edge of the lens shift range by the shift range F1, thereby restricting further movement. [Figure 30] This is a simulation diagram of the initial state in the virtual projection plane priority mode when the position of the virtual projection device 202 is not fixed. [Figure 31] Figure 30 illustrates the movement of the virtual projection plane 204 to the left. [Figure 32] This is a simulation diagram of the initial state when the position of the virtual projection device 202 is fixed in virtual projection plane priority mode. [Figure 33] Figure 32 illustrates the movement of the virtual projection plane 204 to the left. [Figure 34] This is a simulation diagram of the initial state in the virtual projection plane priority mode when the position of the virtual projection device 202 is not fixed. [Figure 35] Figure 34 is a diagram illustrating the enlargement of the virtual projection plane 204. [Figure 36] This is a simulation diagram of the initial state in the virtual projection plane priority mode when the position of the virtual projection device 202 is not fixed. [Figure 37] Figure 36 is a diagram illustrating the enlargement of the virtual projection plane 204. [Figure 38]This figure shows an example of a method for displaying the boundary of the space through which projected light passes. [Figure 39] This figure shows another example of how to display the boundary of the space through which projected light passes. [Figure 40] This diagram shows the first step of the installation assistance process. [Figure 41] This diagram shows the second step of the installation assistance process. [Figure 42] This diagram shows the third step of the installation assistance process. [Modes for carrying out the invention]

[0011] Hereinafter, an example of an embodiment of the present invention will be described with reference to the drawings.

[0012] <Schematic configuration of the projection device 10 to be rearranged by the information processing device 50 of the embodiment> Figure 1 is a schematic diagram showing the general configuration of the projection device 10, which is to be installed with the information processing device of the embodiment.

[0013] The information processing device of this embodiment can be used, for example, to assist in the placement of a projection device 10. The projection device 10 comprises a projection unit 1, a control device 4, and an operation reception unit 2. The projection unit 1 is composed of, for example, a liquid crystal projector or a projector using LCOS (Liquid Crystal On Silicon). In the following description, the projection unit 1 will be described as a liquid crystal projector.

[0014] The control device 4 is a control device that controls the projection by the projection device 10. The control device 4 is a device that includes a control unit composed of various processors, a communication interface (not shown) for communicating with each unit, and a memory 4a such as a hard disk, SSD (Solid State Drive), or ROM (Read Only Memory), and it comprehensively controls the projection unit 1.

[0015] The various processors in the control unit of the control device 4 include a CPU (Central Processing Unit), which is a general-purpose processor that executes programs and performs various processes; a Programmable Logic Device (PLD), such as an FPGA (Field Programmable Gate Array), which is a processor whose circuit configuration can be changed after manufacturing; or a dedicated electrical circuit, such as an ASIC (Application Specific Integrated Circuit), which has a circuit configuration specifically designed to perform a particular process.

[0016] The structure of these various processors is, more specifically, an electrical circuit formed by combining circuit elements such as semiconductor elements. The control unit of the control device 4 may be composed of one of these various processors, or it may be composed of a combination of two or more processors of the same or different types (for example, a combination of multiple FPGAs or a combination of a CPU and an FPGA).

[0017] The operation reception unit 2 detects user instructions by receiving various operations from the user. The operation reception unit 2 may be a button, key, joystick, etc., provided on the control device 4, or it may be a receiving unit, etc., that receives signals from a remote controller that remotely operates the control device 4.

[0018] The object to be projected onto 6 is an object such as a screen or wall that has a projection surface on which the projected image is displayed by the projection unit 1. In the example shown in Figure 1, the projection surface of the object to be projected onto 6 is a rectangular plane. The top, bottom, left, and right of the object to be projected onto 6 in Figure 1 are assumed to be the top, bottom, left, and right of the actual object to be projected onto 6.

[0019] The projection range 11, shown by the dashed line, is the area of ​​the object to be projected onto 6 that is illuminated by the projection unit 1. In the example shown in Figure 1, the projection range 11 is rectangular. The projection range 11 is part or all of the projectionable range that can be projected by the projection unit 1.

[0020] The projection unit 1, control device 4, and operation reception unit 2 may be implemented by a single device (see, for example, Figures 3 and 4). Alternatively, the projection unit 1, control device 4, and operation reception unit 2 may be separate devices that communicate with each other to cooperate.

[0021] <Internal structure of projection unit 1 shown in Figure 1> Figure 2 is a schematic diagram showing an example of the internal structure of the projection unit 1 shown in Figure 1.

[0022] As shown in Figure 2, the projection unit 1 comprises a light source 21, a light modulation unit 22, a projection optical system 23, and a control circuit 24.

[0023] The light source 21 includes a light-emitting element such as a laser or an LED (Light Emitting Diode), and emits, for example, white light.

[0024] The light modulation unit 22 is composed of three liquid crystal panels that modulate the red, blue, and green light emitted from the light source 21 and separated into three colors (red, blue, and green) by a color separation mechanism (not shown) based on image information to emit each color image. Alternatively, each of these three liquid crystal panels may be equipped with red, blue, and green filters, and the white light emitted from the light source 21 may be modulated by each liquid crystal panel to emit each color image.

[0025] The projection optical system 23 receives light from the light source 21 and the light modulation unit 22, and is composed of, for example, a relay optical system that includes at least one lens. The light that passes through the projection optical system 23 is projected onto the object to be projected 6.

[0026] Of the object to be projected 6, the area illuminated by light that passes through the entire range of the light modulation unit 22 becomes the projectable range that can be projected by the projection unit 1. Within this projectable range, the area illuminated by light that actually passes through the light modulation unit 22 becomes the projection range 11. For example, by controlling the size, position, and shape of the area through which light passes within the light modulation unit 22, the size, position, and shape of the projection range 11 within the projectable range can be changed.

[0027] The control circuit 24 controls the light source 21, the light modulation unit 22, and the projection optical system 23 based on the display data input from the control device 4, thereby projecting an image based on this display data onto the object to be projected onto 6. The display data input to the control circuit 24 consists of three parts: red display data, blue display data, and green display data.

[0028] Furthermore, the control circuit 24 expands or shrinks the projection range 11 (see Figure 1) of the projection unit 1 by changing the projection optical system 23 based on commands input from the control device 4. Alternatively, the control device 4 may move the projection range 11 of the projection unit 1 by changing the projection optical system 23 based on user operations received by the operation reception unit 2.

[0029] Furthermore, the projection device 10 includes a shift mechanism that mechanically or optically moves the projection range 11 while maintaining the image circle of the projection optical system 23. The image circle of the projection optical system 23 is the region in which the projected light incident on the projection optical system 23 passes through the projection optical system 23 appropriately in terms of light intensity reduction, color separation, and peripheral curvature.

[0030] The shift mechanism is implemented by at least one of the following: an optical shift mechanism that performs optical system shifting, and an electronic shift mechanism that performs electronic shifting.

[0031] The optical system shift mechanism is, for example, a mechanism that moves the projection optical system 23 in a direction perpendicular to the optical axis (see, for example, Figures 3 and 4), or a mechanism that moves the light modulation unit 22 in a direction perpendicular to the optical axis instead of moving the projection optical system 23. Alternatively, the optical system shift mechanism may be a combination of moving the projection optical system 23 and moving the light modulation unit 22.

[0032] The electronic shift mechanism is a mechanism that performs a pseudo-shift of the projection range 11 by changing the range through which light is transmitted in the optical modulation section 22.

[0033] Furthermore, the projection device 10 may include a projection direction changing mechanism that moves the projection range 11 together with the image circle of the projection optical system 23. The projection direction changing mechanism is a mechanism that changes the projection direction of the projection unit 1 by changing the orientation of the projection unit 1 through mechanical rotation (see, for example, Figures 3 and 4).

[0034] <Mechanical configuration of projection device 10> Figure 3 is a schematic diagram showing the external configuration of the projection device 10. Figure 4 is a schematic cross-sectional view of the optical unit 106 of the projection device 10 shown in Figure 3. Figure 4 shows a cross-section along the optical path of the light emitted from the main body 101 shown in Figure 3.

[0035] As shown in Figure 3, the projection device 10 comprises a main body 101 and an optical unit 106 that protrudes from the main body 101. In the configuration shown in Figure 3, the operation reception unit 2, the control device 4, the light source 21, the light modulation unit 22, and the control circuit 24 in the projection unit 1 are provided in the main body 101. The projection optical system 23 in the projection unit 1 is provided in the optical unit 106.

[0036] The optical unit 106 comprises a first member 102 supported by the main body 101 and a second member 103 supported by the first member 102.

[0037] The first member 102 and the second member 103 may be an integrated member. The optical unit 106 may be configured to be detachably attached to the main body 101 (in other words, to be replaceable).

[0038] The main body 101 has a housing 15 (see Figure 4) in which an opening 15a (see Figure 4) for allowing light to pass through is formed in the portion connected to the optical unit 106.

[0039] Inside the housing 15 of the main unit 101, as shown in Figure 3, there is a light source 21 and an optical modulation unit 12 which includes an optical modulation unit 22 (see Figure 2) that spatially modulates the light emitted from the light source 21 based on input image data to generate an image.

[0040] Light emitted from the light source 21 is incident on the light modulation section 22 of the light modulation unit 12, spatially modulated by the light modulation section 22, and then emitted.

[0041] As shown in Figure 4, the image formed by the light spatially modulated by the optical modulation unit 12 passes through the aperture 15a of the housing 15 and enters the optical unit 106, where it is projected onto the object to be projected, 6, making the image G1 visible to the observer.

[0042] As shown in Figure 4, the optical unit 106 comprises a first member 102 having a hollow section 2A connected to the inside of the main body 101, a second member 103 having a hollow section 3A connected to the hollow section 2A, a first optical system 121 and a reflective member 122 arranged in the hollow section 2A, a second optical system 31, a reflective member 32, a third optical system 33, and a lens 34 arranged in the hollow section 3A, a shift mechanism 105, and a projection direction changing mechanism 104.

[0043] The first member 102 is a member whose cross-sectional shape is, for example, rectangular, and has openings 2a and 2b formed on mutually perpendicular surfaces. The first member 102 is supported by the main body 101 with opening 2a positioned opposite opening 15a of the main body 101. Light emitted from the light modulation section 22 of the light modulation unit 12 of the main body 101 is incident on the hollow section 2A of the first member 102 through openings 15a and 2a.

[0044] The direction of incidence of light entering the hollow section 2A from the main body 101 is described as direction X1, the opposite direction of direction X1 is described as direction X2, and directions X1 and X2 are collectively referred to as direction X. In Figure 4, the direction from the front to the back of the paper and the opposite direction are described as direction Z. Of the directions Z, the direction from the front to the back of the paper is described as direction Z1, and the direction from the back to the front of the paper is described as direction Z2.

[0045] Furthermore, the direction perpendicular to directions X and Z is denoted as direction Y, and within direction Y, the upward direction in Figure 4 is denoted as direction Y1, and the downward direction in Figure 4 is denoted as direction Y2. In the example in Figure 4, the projection device 10 is positioned such that direction Y2 is the vertical direction.

[0046] The projection optical system 23 shown in Figure 2 consists of a first optical system 121, a reflector 122, a second optical system 31, a reflector 32, a third optical system 33, and a lens 34. Figure 4 shows the optical axis K of this projection optical system 23. The first optical system 121, the reflector 122, the second optical system 31, the reflector 32, the third optical system 33, and the lens 34 are arranged along the optical axis K in this order from the light modulation unit 22 side.

[0047] The first optical system 121 includes at least one lens and guides light traveling in direction X1 incident from the main body 101 to the first member 102 to the reflecting member 122.

[0048] The reflective member 122 reflects light incident from the first optical system 121 in direction Y1. The reflective member 122 is made of, for example, a mirror. The first member 102 has an aperture 2b formed in the optical path of the light reflected by the reflective member 122, and this reflected light passes through the aperture 2b and proceeds to the hollow portion 3A of the second member 103.

[0049] The second member 103 has a roughly T-shaped cross-section, with an opening 3a formed at a position facing the opening 2b of the first member 102. Light from the main body 101 that has passed through the opening 2b of the first member 102 enters the hollow portion 3A of the second member 103 through this opening 3a. Note that the cross-sectional shapes of the first member 102 and the second member 103 are arbitrary and not limited to those described above.

[0050] The second optical system 31 includes at least one lens and guides the light incident from the first member 102 to the reflecting member 32.

[0051] The reflective member 32 reflects the light incident from the second optical system 31 in direction X2 and guides it to the third optical system 33. The reflective member 32 is made up of, for example, a mirror.

[0052] The third optical system 33 includes at least one lens and guides the light reflected by the reflective member 32 to the lens 34.

[0053] The lens 34 is positioned at the end of the second member 103 so as to block the opening 3c formed at the end on the direction X2 side. The lens 34 projects light incident from the third optical system 33 onto the object to be projected 6.

[0054] The projection direction changing mechanism 104 is a rotation mechanism that rotatably connects the second member 103 to the first member 102. This projection direction changing mechanism 104 allows the second member 103 to rotate freely around a rotation axis (specifically, the optical axis K) extending in direction Y. The projection direction changing mechanism 104 only needs to be able to rotate the optical system and is not limited to the arrangement shown in Figure 4. Furthermore, the number of rotation mechanisms is not limited to one; multiple mechanisms may be provided.

[0055] The shift mechanism 105 is a mechanism for moving the optical axis K of the projection optical system (in other words, the optical unit 106) in a direction perpendicular to the optical axis K (direction Y in Figure 4). Specifically, the shift mechanism 105 is configured to change the position of the first member 102 in direction Y relative to the main body 101. The shift mechanism 105 may be used to move the first member 102 manually or electrically.

[0056] Figure 4 shows the state in which the first member 102 has been moved to its maximum extent toward direction Y1 by the shift mechanism 105. From the state shown in Figure 4, the first member 102 moves toward direction Y2 by the shift mechanism 105, which changes the relative position between the center of the image formed by the light modulation unit 22 (in other words, the center of the display surface) and the optical axis K, thereby shifting (translating) the image G1 projected onto the object to be projected 6 toward direction Y2.

[0057] Alternatively, the shift mechanism 105 may move the optical modulation unit 22 in the direction Y instead of moving the optical unit 106 in the direction Y. In this case as well, the image G1 projected onto the object 6 can be moved in the direction Y2.

[0058] <Information processing device 50 of the embodiment> Figure 5 shows an example of an information processing device 50 of the embodiment. The information processing device 50 of the embodiment is a tablet terminal having a touch panel 51. The touch panel 51 is a touch-operable display. For example, a user of the information processing device 50 brings the information processing device 50 into a space (e.g., a room) where a projection device 10 will be installed and projection will be performed. The information processing device 50 displays installation support images on the touch panel 51 to assist in the installation of the projection device 10 in that space.

[0059] <Hardware configuration of the information processing device 50> Figure 6 shows an example of the hardware configuration of the information processing device 50 in the embodiment. The information processing device 50 shown in Figure 5 includes, for example, a processor 61, a memory 62, a communication interface 63, a user interface 64, and a sensor 65, as shown in Figure 6. The processor 61, memory 62, communication interface 63, user interface 64, and sensor 65 are connected, for example, by a bus 69.

[0060] The processor 61 is a circuit that performs signal processing, and is, for example, a CPU that controls the entire information processing device 50. The processor 61 may also be implemented using other digital circuits such as an FPGA or a DSP (Digital Signal Processor). Furthermore, the processor 61 may be implemented using a combination of multiple digital circuits.

[0061] Memory 62 includes, for example, main memory and auxiliary memory. Main memory is, for example, RAM (Random Access Memory). Main memory is used as the work area of ​​processor 61.

[0062] Auxiliary memory is a non-volatile memory such as a magnetic disk or flash memory. Various programs for operating the information processing device 50 are stored in the auxiliary memory. The programs stored in the auxiliary memory are loaded into main memory and executed by the processor 61.

[0063] Furthermore, the auxiliary memory may include portable memory that can be removed from the information processing device 50. Portable memory may include memory cards such as USB (Universal Serial Bus) flash drives and SD (Secure Digital) memory cards, or external hard disk drives.

[0064] The communication interface 63 is a communication interface that communicates with devices outside the information processing device 50. The communication interface 63 includes at least one of a wired communication interface that communicates via wires and a wireless communication interface that communicates via wireless. The communication interface 63 is controlled by the processor 61.

[0065] The user interface 64 includes, for example, an input device that accepts user input and an output device that outputs information to the user. The input device can be implemented by, for example, a key (e.g., a keyboard) or a remote control. The output device can be implemented by, for example, a display or a speaker. In the information processing device 50 shown in Figure 5, the input device and output device are implemented by a touch panel 51. The user interface 64 is controlled by the processor 61.

[0066] The sensor 65 includes imaging devices that have an imaging optical system and an image sensor and are capable of taking images, and spatial recognition sensors that can recognize the space around the information processing device 50 in three dimensions. The imaging device includes, for example, the imaging device provided on the back surface of the information processing device 50 shown in Figure 5.

[0067] One example of a spatial recognition sensor is LIDAR (Light Detection and Ranging), which emits laser light, measures the time it takes for the laser light to hit an object and reflect back, and then determines the distance and direction to the object. However, spatial recognition sensors are not limited to this; they can also include various other sensors such as radar that emits radio waves or ultrasonic sensors that emit ultrasonic waves.

[0068] <Definition of virtual projection device 202, virtual projection plane 204, and coordinate system> Figures 7 to 9 illustrate the virtual projection device 202 and virtual projection surface 204 displayed on the touch panel 51 of the information processing device 50, corresponding to the projection device 10 and projection range 11 (Figure 1). Figures 7 to 9 also define the coordinate systems for the virtual projection device 202 and the virtual projection surface 204. However, this definition of coordinate systems is merely an example, and other coordinate systems can be adopted. Furthermore, in this example, different coordinate systems are given for the virtual projection device 202 and the virtual projection surface 204, but a common coordinate system could also be applied to both.

[0069] The virtual projection device 202 and the virtual projection surface 204 are superimposed on the spatial image 70 displayed on the touch panel 51. For example, the information processing device 50 generates correspondence information between the position coordinates of the space recognized three-dimensionally by the spatial recognition sensor 65 and the position coordinates of the spatial image 70 displayed two-dimensionally by the touch panel 51. Furthermore, the information processing device 50 generates correspondence information between the position coordinates of the virtual projection device 202 and the virtual projection surface 204, which are virtually placed in the recognized space, and the position coordinates of the spatial image 70. As a result, the information processing device 50 can superimpose the virtual projection device 202 and the virtual projection surface 204 onto the spatial image 70.

[0070] Figure 7 illustrates an example of the coordinate system of the virtual projection device 202 corresponding to the projection device 10. The virtual projection device installation surface 201, which corresponds to the floor surface in real space, is set in the spatial image 70. The virtual projection device 202 is then placed on the virtual projection device installation surface 201. That is, the virtual projection surface 204 is placed in the space indicated by the spatial image 70. The virtual projection device installation surface 201 is parallel to and overlaps with the bottom surface of the virtual projection device 202.

[0071] The projection device coordinate system CA, which is the coordinate system of the virtual projection device 202, is defined by a three-dimensional Cartesian coordinate system that includes the XA axis along the left-right direction of the virtual projection device 202, the ZA axis along the front-back direction of the virtual projection device 202, and the YA axis perpendicular to the projection device installation virtual plane 201. In this figure, the projection direction changing mechanism 104 (Figure 4) is positioned so that the second member 103 faces perpendicular to the projection device installation virtual plane 201. In this case, the projection device installation virtual plane 201 and the projection surface installation virtual plane 203 (see Figure 9) do not face each other (non-facing).

[0072] Figure 8 is another diagram illustrating an example of the coordinate system of the virtual projection device 202. In this diagram, the projection direction changing mechanism 104 positions the second member 103 so that it faces parallel to the virtual plane 201 on which the projection device is installed. In this case, the virtual plane 201 on which the projection device is installed and the virtual plane 203 on which the projection device is installed (see Figure 9) face each other. As shown in Figures 8 and 9, the projection device coordinate system CA is defined regardless of the position of the second member 103.

[0073] Figure 9 illustrates an example of a coordinate system for a virtual projection plane 204 corresponding to the projection range 11. A virtual projection plane 203 corresponding to the object to be projected 6 (Figure 1) is set in the spatial image 70, and the virtual projection plane 204 is placed on the virtual projection plane 203. That is, the virtual projection plane 204 is placed in the space indicated by the spatial image 70.

[0074] The projection plane coordinate system CB, which is the coordinate system of the virtual projection plane 204, is defined by a three-dimensional Cartesian coordinate system that includes the XB axis along the horizontal shift direction of the projection range 11 by the shift mechanism 105 (Figure 4), the ZB axis along the vertical shift direction of the projection range 11, and the YB axis perpendicular to the virtual plane 203 on which the projection plane is set.

[0075] <Overview of information processing by the information processing device 50 of the embodiment> A technology is being explored that uses the Augmented Reality (AR) function of smart devices to simulate projection from a projection device. This technology concerns the placement of virtual objects such as the virtual projection device 202 and the virtual projection device installation surface 201 described above, and the specific method for adjusting the position, size, etc., of the virtual objects after placement is important.

[0076] In augmented reality (AR), since a 3D space is captured by an imaging device and displayed on the screen as a 2D image, it is not easy to set the position of virtual objects exactly as intended at the time of placement. Therefore, users are required to fine-tune the position of virtual objects after placement, but no appropriate method has been proposed, which places a burden on users.

[0077] The information processing device 50 of this embodiment can reduce the burden on the user regarding the installation work of the virtual projection device 202 and the virtual projection surface 204.

[0078] <Processing by the information processing device 50 of the embodiment> Figure 10 is a flowchart showing an example of processing by the information processing device 50 of the embodiment. The information processing device 50 of the embodiment performs the processing shown in Figure 10, for example. The processing in Figure 10 is performed by the processor 61 shown in Figure 6, for example.

[0079] When the imaging device, sensor 65, starts imaging (step S101), the information processing device 50 recognizes the space from the image captured by the sensor 65 (step S102). Here, in recognizing the space, the information processing device 50 acquires first image data representing a first image, for example, a spatial image 70. In this embodiment, the imaging device, sensor 65, is configured integrally with the information processing device 50, but it may be an external device separate from the information processing device 50.

[0080] Next, the information processing device 50 places the virtual screen (virtual projection surface) and virtual projector (virtual projection device) at their initial positions in the space (first image data) (step S103). Here, the information processing device 50 acquires placement data regarding the placement of the virtual screen and virtual projector in the space shown by the first image. This placement data corresponds to the current placement of the virtual screen and virtual projector, and for example, it shows the initial placement of the virtual screen and virtual projector.

[0081] Next, the information processing device 50 displays an AR image on the touch panel 51, which is the output destination display device, by superimposing a virtual screen image and a virtual projector image onto the captured image (step S104).

[0082] Next, the information processing device 50 determines whether or not it has received an instruction to change the arrangement of the virtual screen image and / or virtual projector image based on the user's operation on the touch panel 51 (step S105). If the information processing device 50 has received an instruction to change the arrangement, it will acquire arrangement change data related to the arrangement change of the virtual screen and / or virtual projector in the first image.

[0083] If the information processing device 50 receives a rearrangement instruction (step S105: Yes), the information processing device 50 determines whether the rearrangement is appropriate (step S106). The determination of whether the rearrangement is appropriate is, for example, based on the spatial recognition result in step S102, whether the rearrangement is actually possible. If the rearrangement is appropriate (step S106: Yes), the information processing device 50 rearranges the virtual screen and virtual projector (step S107).

[0084] Next, the information processing device 50 updates the projection parameters based on the arrangement change (step S108). This means that the information processing device 50 generates a second image data representing the second image in which the virtual screen and / or virtual projector, whose arrangement has been changed based on the arrangement change data, is displayed in the first image.

[0085] Next, the information processing device 50 displays an AR image on the touch panel 51, which is an image of the captured image superimposed with a virtual screen image and a virtual projector image (step S109), and waits for the next arrangement change instruction. This means that the information processing device 50 outputs the second image data to the touch panel 51, which is the output destination. In this embodiment, the touch panel 51, which is the output destination, is configured integrally with the information processing device 50, but it may be an external device separate from the information processing device 50.

[0086] If the information processing device 50 has not received a rearrangement instruction (step S105: No), or if the rearrangement is not appropriate (step S106: No), the information processing device 50 waits for the next rearrangement instruction.

[0087] <User interface 64 of the information processing device 50 in this embodiment> Figures 11 to 17 illustrate a user interface 64 (Figure 6) for a user to operate the information processing device 50, particularly the virtual projection device 202 or the virtual projection surface 204. The user interface 64 in Figures 11 to 17 is displayed on a touch panel 51, which is an output device (output destination). That is, the touch panel 51 also functions as an input unit that receives input from the user for reconfiguration data related to reconfiguration of the virtual projection device 202 or the virtual projection surface 204. However, the illustrated user interface 64 is merely an example, and the user interface applicable to the information processing device 50 is not particularly limited.

[0088] User input can include, for example, pressing a physical button, using gestures such as tapping, panning, or pinching on a touchscreen, or using voice commands, camera gestures, or numerical input.

[0089] Note that Figure 11 shows only the user interface 64; however, in actual use, the touch panel 51 also displays the virtual projection device 202 and the virtual projection surface 204. The virtual projection device 202 and the virtual projection surface 204 are not shown in this diagram.

[0090] Figure 11 shows an example of a user interface 64 displayed by the information processing device 50 of the embodiment, where the touch panel 51 displays an operation image UI1 with multiple buttons. The user can operate the virtual projection device 202 or the virtual projection surface 204 by pressing various buttons, which will be described later. That is, the information processing device 50 receives input of reconfiguration data related to reconfiguration of the virtual projection device 202 or the virtual projection surface 204 from the user through the user interface 64. In this case, the information processing device 50 can control the touch panel 51 to display an image (operation image UI1) that includes an operation image for instructing a reconfiguration of the virtual projection surface 204 and an operation image for instructing a reconfiguration of the virtual projection device 202.

[0091] The operation image UI1 includes a virtual projection device operation area A1 and a virtual projection surface operation area A2. The virtual projection device operation area A1 is a user interface area for operating the virtual projection device 202. The virtual projection device operation area A1 includes an operation target switching button B11, a posture change button B12, a rotation button B13, an up / down movement button B14, and forward / backward / left / right movement buttons B15.

[0092] The target device switching button B11 is used to switch between virtual projection devices 202 when multiple virtual projection devices 202 are installed. The orientation change button B12 is used to change the orientation (direction) of the virtual projection device 202. The rotation button B13 is used to rotate the orientation (direction) of the virtual projection device 202. The up / down movement button B14 is used to move the virtual projection device 202 in the up / down direction. The forward / backward / left / right movement button B15 is used to move the virtual projection device 202 in the forward / backward / left / right directions.

[0093] The virtual projection surface operation area A2 is a user interface area for operating the virtual projection surface 204. The virtual projection surface operation area A2 includes an aspect ratio change button B21, an image setting button B22, an image rotation button B23, a projection surface rotation button B24, and up / down / left / right movement buttons B25.

[0094] The aspect ratio change button B21 is used to change the aspect ratio of the virtual projection plane 204. The image setting button B22 is used to set an image on the virtual projection plane 204. The image rotation button B23 is used to rotate the image set on the virtual projection plane 204. The projection plane rotation button B24 is used to rotate the virtual projection plane 204. The up / down / left / right movement buttons B25 are used to move the virtual projection plane 204 in the up / down / left / right directions.

[0095] Figure 12 shows another example of a user interface 64 displayed by the information processing device 50 of the embodiment, for example, an operation image in which the touch panel 51 of a smartphone, which is the information processing device 50, displays multiple buttons.

[0096] The operation image UI2 shown in Figure 12(A) displays the virtual projection device operation area A1 in Figure 12 and is an image for operating the virtual projection device 202. The operation image UI3 shown in Figure 12(B) displays the virtual projection surface operation area A2 in Figure 12 and is an image for operating the virtual projection surface 204.

[0097] Note that Figure 12 shows only the user interface 64; however, in actual use, the touch panel 51 also displays the virtual projection device 202 and the virtual projection surface 204. The virtual projection device 202 and the virtual projection surface 204 are not shown in this diagram.

[0098] In other words, the information processing device 50 receives input of reconfiguration data related to reconfiguration of the virtual projection device 202 or the virtual projection surface 204 from the user via the user interface 64. In this case, the information processing device 50 can control the system to switch between a state in which an operation image UI3 for instructing a reconfiguration of the virtual projection surface 204 is displayed on the touch panel 51, and a state in which an operation image UI2 for instructing a reconfiguration of the virtual projection device 202 is displayed on the touch panel 51. The user can switch between the images in Figure 12(A) and Figure 12(B) by performing a predetermined operation (such as tapping on the touch panel 51) and operate the virtual projection device 202 or the virtual projection surface 204.

[0099] Figure 13 shows another example of the user interface 64 displayed by the information processing device 50 of the embodiment, for example, an operation image in which the touch panel 51 of the tablet, which is the information processing device 50, displays multiple buttons. The operation image UI4 in Figure 13 is a type of input device realized by the user interface 64, and is displayed on the touch panel 51 of the information processing device 50, which is a smartphone, for example. The operation image UI4 shown in Figure 13(A) is a screen in which the user has selected the virtual projection device 202 as the target of operation by tapping the area of ​​the virtual projection device 202. The operation image UI4 shown in Figure 13(B) is a screen in which the user has selected the virtual projection surface 204 as the target of operation by tapping the area of ​​the virtual projection surface 204.

[0100] In the operation image UI4 of Figure 13(A), the virtual projection device 202 is selected as the target of operation, but in the initial state, user operation on the virtual projection device 202 is locked. Therefore, the user cannot operate the virtual projection device 202 in the initial state of Figure 13(A).

[0101] The operation image UI4 in Figure 13(A) displays the orientation change button B12 and image setting button B22 from Figure 11, as well as the size change lock release button B31, the horizontal movement lock release button B32, and the rotation lock release button B33. The size change lock release button B31 is used to release the size change of the virtual projection device 202, which is locked in the initial state. The horizontal movement lock release button B32 is used to release the horizontal movement of the virtual projection device 202, which is locked in the initial state. The rotation lock release button B33 is used to release the rotation of the virtual projection device 202, which is locked in the initial state.

[0102] On the other hand, in the operation image UI4 of Figure 13(B), the virtual projection plane 204 is selected as the target of operation, but in the initial state, user operation on the virtual projection plane 204 is locked. Therefore, the user cannot operate the virtual projection plane 204 in the initial state of Figure 13(B).

[0103] The UI4 operation image in Figure 13(B) displays the orientation change button B12A for the virtual projection plane 204 and the image setting button B22 in Figure 11, as well as the resize unlock button B31, the horizontal movement unlock button B32, and the rotation unlock button B33, similar to the screen in Figure 13(A). The resize unlock button B31 is used to release the resizing of the virtual projection plane 204, which is locked in the initial state. The horizontal movement unlock button B32 is used to release the horizontal movement of the virtual projection plane 204, which is locked in the initial state. The rotation unlock button B33 is used to release the rotation of the virtual projection plane 204, which is locked in the initial state.

[0104] Figure 14 shows the operation of moving the virtual projection device 202 horizontally in the operation image UI4 of Figure 13(A). By pressing the horizontal movement unlock button B32, the horizontal movement of the virtual projection device 202 is released. Furthermore, by the user tracing a straight line with their finger on the touch panel 51 (pan gesture), the virtual projection device 202 can be moved horizontally.

[0105] Figure 15 shows the operation of moving the virtual projection device 202 up and down in the operation image UI4 of Figure 13(A). The user can move the virtual projection device 202 up and down by tracing a straight line with their finger on the touch panel 51 (pan gesture).

[0106] Figure 16 shows the operation of rotating the virtual projection device 202 in the operation image UI4 of Figure 13(A). The rotation of the virtual projection device 202 is released when the user presses the rotation lock release button B33. Furthermore, the virtual projection device 202 can be rotated by the user tracing a circle with their finger on the touch panel 51 (rotation gesture).

[0107] Figure 17 shows the operation to change the size of the virtual projection surface 204 in the operation image UI4 of Figure 13(B). The user can release the size change lock button B31 to release the size change lock of the virtual projection surface 204. Furthermore, the user can change the size of the virtual projection surface 204 by tracing the touch panel 51 with their finger to shrink or expand the area of ​​the virtual projection surface 204 (pinch gesture).

[0108] In other words, the information processing device 50 is capable of performing at least one of the following: controlling the arrangement of the virtual projection surface 204 in response to operations performed by the user on the virtual projection surface 204 in the second image displayed on the touch panel 51; and controlling the arrangement of the virtual projection device 202 in response to operations performed by the user on the virtual projection device 202 in the second image displayed on the touch panel 51.

[0109] <Details of processing by the information processing device 50 of the embodiment> Figures 18 to 37 illustrate the details of the information processing performed by the information processing device 50 in response to changes in the arrangement of the virtual projection device 202 or the virtual projection surface 204. Figures 18 to 24 and 28 to 37 are simulation diagrams obtained by simulating operations such as changes in position, direction, and size of the virtual projection device 202 and the virtual projection surface 204 after their superimposed arrangement in the first image, the spatial image 70 (Figures 7 to 9). The virtual projection device 202 is not shown in the simulation diagrams. On the other hand, Figures 26 and 27 show the images displayed on the touch panel 51 based on this simulation. The control described below is performed, for example, by the processor 61 shown in Figure 6.

[0110] <Virtual projection device priority mode and virtual projection surface priority mode> The information processing device 50 operates under two modes: a virtual projection device priority mode and a virtual projection surface priority mode. First, the information processing device 50 can determine the position, orientation, and size of the virtual projection surface 204 according to the position and orientation of the virtual projection device 202. In this specification, this type of control is referred to as the "virtual projection device priority mode." Furthermore, the information processing device 50 can determine the installable range of the virtual projection device 202 and the position of the virtual projection device 202 according to the position, orientation, and size of the virtual projection surface 204. In this specification, this type of control is referred to as the "virtual projection surface priority mode." Below, Figures 18 to 29 show examples of control under the virtual projection device priority mode, and Figures 30 to 37 show examples of control under the virtual projection surface priority mode. However, this classification of the operating modes of the information processing device 50 is merely an example.

[0111] <Movement of virtual projector 202 in virtual projector priority mode> Figures 18 to 22 are simulation diagrams of an example in which the user instructs the virtual projector 202 to move, i.e., to change the position of the virtual projector 202, in virtual projector priority mode.

[0112] Figure 18 is a simulation diagram of the initial state in the virtual projection device priority mode. The initial state is a state in which the projection center point, without any movement of the projection range 11 by the shift mechanism of the projection device 10 (hereinafter also referred to as "lens shift"), lies on the virtual projection plane 204, and the virtual projection device 202 is directly facing the projection center point without lens shift, and the virtual projection plane 204 corresponds to the orientation of the virtual projection device 202, or the virtual projection device 202 corresponds to the orientation of the virtual projection plane 204. The initial state is realized by arrangement data corresponding to the current arrangement of the virtual projection plane 204 and the virtual projection device 202.

[0113] As a prerequisite for the initial state, the information processing device 50 acquires first image data, which is a spatial image 70 obtained by imaging by the sensor 65, an imaging device, and acquires arrangement data regarding the arrangement of the virtual projection plane 204 and the virtual projection device 202 in the space shown by the first image. This content is also common in the initial state which will be explained later.

[0114] The coordinates of the virtual projection device 202 follow the projection device coordinate system CA described in Figure 7, and the projection direction changing mechanism 104 (Figure 4) is positioned so that the second member 103 faces perpendicular to the virtual plane 201 on which the projection device is installed. As described above, the projection device coordinate system CA includes the XA axis along the left-right direction of the virtual projection device 202, the ZA axis along the front-back direction of the virtual projection device 202, and the YA axis perpendicular to the virtual plane 201 on which the projection device is installed. The ZA axis also aligns with the optical axis of the virtual projection device 202. The YA axis aligns with the normal direction of the virtual plane 201 on which the projection device is installed. Point P1 is the lens center point of the virtual projection device 202, and point P2 is the projection center point of the virtual projection plane 204 without lens shift.

[0115] The virtual projection device 202 moves in the spatial image 70 (first image) when the user operates one of the user interfaces 64 shown in Figures 11 to 17.

[0116] The user can change the position of the virtual projection device 202 by pressing, for example, the forward / backward / left / right movement button B15 (Figure 11). Figure 19 illustrates the movement of the virtual projection device 202 to the left, i.e., movement in the positive XA axis direction. The user can instruct the information processing device 50 to perform such a movement by pressing the left button of the forward / backward / left / right movement button B15. Movement of the virtual projection device 202 to the right, i.e., movement in the negative XA axis direction, can be instructed by pressing the right button of the forward / backward / left / right movement button B15.

[0117] A change in the position of the virtual projection device 202 means that the information processing device 50 acquires position change data relating to the change in the arrangement of the virtual projection surface 204 and / or the virtual projection device 202 in the first image (spatial image 70). It also means that the information processing device 50 generates second image data representing the second image displayed in the first image, based on this position change data, with the virtual projection surface 204 and / or the virtual projection device 202 having been rearranged. In this example, the information processing device 50 acquires position change data relating to the rearrangement of the virtual projection device 202 and generates second image data representing the second image of the rearranged virtual projection device 202. The acquisition of such position change data and the generation of second image data are common to all examples described later. In this specification, position change data includes data indicating a change in at least one of the following: the position of the virtual projection surface 204 and / or the virtual projection device 202, the direction (orientation) of the virtual projection surface 204 and / or the virtual projection device 202, and the size of the virtual projection surface 204, which will be explained in the examples described later.

[0118] In this case, the information processing device 50 changes the position of the virtual projection device 202 in a direction different from the lens optical axis direction of the virtual projection device 202. Then, based on the position change data described above, the information processing device 50 changes the position of the virtual projection device 202, but maintains the position of the virtual projection plane 204. That is, the projection center point P2 of the virtual projection plane 204 does not move, and the information processing device 50 changes the lens shift parameter related to the lens shift of the virtual projection device 202. The lens shift parameter is a parameter for the shift of the projection position of the virtual projection device 202. The change in the lens shift parameter corresponds to the distance D1 in Figure 19. The distance D1 corresponds to the distance between the projection center point P3 of the moved virtual projection device 202 and the projection center point P2 of the virtual projection plane 204 in the initial state, under the condition that the parameter is not changed.

[0119] The information processing device 50 outputs the second image data of the repositioned virtual projection device 202 to the touch panel 51, which is the output destination display device. The touch panel 51 displays the second image based on the second image data along with the first image (spatial image 70). This output of the second image data and display of the second image is common to all cases described later. As a result, the user can easily grasp the relationship between the virtual projection device 202 and the virtual projection surface 204 intuitively, and can easily adjust the position of the virtual projection device 202 as intended.

[0120] Figure 20 illustrates the movement of the virtual projection device 202 in the rearward direction, i.e., in the negative direction of the ZA axis. The user can instruct the information processing device 50 to perform such movement by pressing the rear button of the forward / backward / left / right movement button B15. Movement of the virtual projection device 202 in the forward direction, i.e., in the positive direction of the ZA axis, can be instructed by pressing the front button of the forward / backward / left / right movement button B15.

[0121] In this case, the information processing device 50 changes the position of the virtual projection device 202 to the lens optical axis direction of the virtual projection device 202. Then, based on the position change data described above, the information processing device 50 changes the position of the virtual projection device 202, but maintains the position of the virtual projection plane 204. That is, the projection center point P2 of the virtual projection plane 204 does not move, and the information processing device 50 changes the lens shift parameter related to the lens shift of the virtual projection device 202.

[0122] On the other hand, the information processing device 50 enlarges the size of the virtual projection surface 204. The dashed line in Figure 20 represents the virtual projection surface 204 before enlargement. When the virtual projection device 202 moves forward, the size of the virtual projection surface 204 decreases. In other words, the information processing device 50 changes the size of the virtual projection surface 204 according to the projection distance d1 from the virtual projection device 202 to the virtual projection surface 204. This makes it easier for the user to intuitively understand the relationship between the virtual projection device 202 and the virtual projection surface 204, and to easily adjust the position of the virtual projection device 202 as intended.

[0123] Figure 21 illustrates the upward movement of the virtual projection device 202, i.e., movement in the positive direction of the YA axis. The user can instruct the information processing device 50 to perform this movement by pressing the upper button of the up / down movement button B14 (Figure 12). The downward movement of the virtual projection device 202, i.e., movement in the negative direction of the YA axis, can be instructed by pressing the lower button of the up / down movement button B14.

[0124] In this case, the information processing device 50 changes the position of the virtual projection device 202 in a direction different from the lens optical axis direction of the virtual projection device 202. Then, based on the position change data described above, the information processing device 50 changes the position of the virtual projection device 202, but maintains the position of the virtual projection plane 204. That is, the projection center point P2 of the virtual projection plane 204 does not move, and the information processing device 50 changes the lens shift parameter related to the lens shift of the virtual projection device 202. The change in the lens shift parameter corresponds to the distance D3 in Figure 21. The distance D3 corresponds to the distance between the projection center point P3 of the moved virtual projection device 202 and the projection center point P2 of the virtual projection plane 204 in the initial state, under the condition that the parameter is not changed.

[0125] This makes it easier for the user to intuitively understand the relationship between the virtual projection device 202 and the virtual projection surface 204, and to easily adjust the position of the virtual projection device 202 as intended.

[0126] Figure 22, like Figure 18, is a simulation diagram of the initial state in the virtual projection device priority mode. The coordinates of the virtual projection device 202 follow the projection device coordinate system CA described in Figure 8, and the projection direction changing mechanism 104 (Figure 4) is positioned so that the second member 103 faces in a direction parallel to the virtual projection device installation surface 201. In this case as well, the information processing device 50 controls the virtual projection device 202 and the virtual projection surface 204 according to the user's instructions, just as in Figures 18 to 21.

[0127] Furthermore, the information processing device 50 can also rotate the virtual projection device 202 around the axis in the direction of the lens optical axis of the virtual projection device 202, i.e., change its direction, based on the arrangement change data. In this case, the information processing device 50 rotates the virtual projection surface 204 in accordance with the rotation of the virtual projection device 202 (change of direction).

[0128] <Movement of the virtual projection plane 204 in virtual projection device priority mode> Figures 23 and 24 are simulation diagrams of an example in which the user instructs the virtual projection plane 204 to move, i.e., to change the position of the virtual projection plane 204, in virtual projection device priority mode.

[0129] Figure 23, like Figure 18, is a simulation diagram of the initial state in the virtual projection device priority mode. The coordinates of the virtual projection plane 204 follow the projection plane coordinate system CB described in Figure 9. As described above, the projection plane coordinate system CB includes the XB axis along the horizontal shift direction of the projection range 11 by the shift mechanism 105, the ZB axis along the vertical shift direction of the projection range 11, and the YB axis perpendicular to the virtual plane 203 on which the projection plane is installed. Point P1 is the lens center point of the virtual projection device 202, and point P2 is the projection center point of the virtual projection plane 204 without shift.

[0130] When a user operates one of the user interfaces 64 shown in Figures 11 to 17, the virtual projection plane 204 moves in the spatial image 70 (first image).

[0131] The user can change the position of the virtual projection plane 204 by pressing, for example, the up / down / left / right movement buttons B25 (Figure 11). Figure 24 illustrates the movement of the virtual projection plane 204 to the right and upward, i.e., movement in the positive XB axis direction and the negative ZB axis direction. The user can instruct the information processing device 50 to perform such movements by pressing the right and up buttons of the up / down / left / right movement buttons B25. Movement of the virtual projection plane 204 to the left, i.e., movement in the negative XB axis direction, can be instructed by pressing the left button of the up / down / left / right movement buttons B25. Movement of the virtual projection plane 204 upward, i.e., movement in the negative ZB axis direction, can be instructed by pressing the up button of the up / down / left / right movement buttons B25. Movement of the virtual projection plane 204 downward, i.e., movement in the positive ZB axis direction, can be instructed by pressing the down button of the up / down / left / right movement buttons B25.

[0132] In this case, the information processing device 50 changes the position of the virtual projection plane 204 based on the arrangement change data described above, but maintains the position of the virtual projection device 202. That is, the projection center point P2 of the virtual projection plane 204 moves, and the information processing device 50 changes the lens shift parameter related to the lens shift of the virtual projection device 202. The change in the lens shift parameter corresponds to the distance D4 in Figure 24. The distance D4 is the distance between the projection center point P4 of the moved virtual projection plane and the projection center point P2 of the virtual projection plane 204 in the initial state. This makes it easier for the user to intuitively understand the relationship between the virtual projection device 202 and the virtual projection plane 204, and to easily adjust the position of the virtual projection plane 204 as intended.

[0133] Furthermore, the information processing device 50 can also rotate the virtual projection plane 204 around an axis perpendicular to the virtual projection plane 204, i.e., change its direction, based on the arrangement change data. In this case, the virtual projection device 202 is rotated in accordance with the rotation of the virtual projection plane 204 (change of direction).

[0134] <Determination of the installation orientation of the virtual projection device 202> In the examples described so far, the projection device 10 is installed on the floor, and the virtual projection device 202 is installed and used on a virtual projection device installation surface 201 that simulates the floor. However, the projection device 10 may also be used suspended from the ceiling as well as the floor. The projection device coordinate system CA in Figures 7 and 8 is intended exclusively for use in a floor-mounted configuration, and it is preferable to use a different coordinate system when using it suspended from the ceiling.

[0135] Figure 25 illustrates the normal vector and spatial coordinate system CC corresponding to the installation orientation of the virtual projection device 202. While the projection device coordinate system CA and the projection plane coordinate system CB are local coordinate systems, the spatial coordinate system CC is a world coordinate system. Figure 25(A) illustrates the normal vector corresponding to the installation orientation of the virtual projection device 202 when the projection device 10 is installed on the floor and the virtual projection device installation surface 201 is the floor. The information processing device 50 determines that the virtual projection device installation surface 201 is on the floor if the Y-axis component of the normal vector of the virtual projection device installation surface 201 is 0.9 or greater.

[0136] Figure 25(B) illustrates the normal vector corresponding to the installation orientation of the virtual projection device 202 when the projection device 10 is suspended from the ceiling and the virtual projection device installation surface 201 is the ceiling surface. The information processing device 50 determines that the virtual projection device installation surface 201 is the ceiling surface if the Y-axis component of the virtual projection device installation surface 201 is -0.9 or less.

[0137] In other words, the information processing device 50 can extract an installation orientation based on the installation position of the virtual projection device 202 in space from among the candidate installation orientations of the virtual projection device 202.

[0138] In the spatial coordinate system CC, the YA axis of the projection device coordinate system CA in Figures 7 and 8 corresponds to the direction of gravity, and the Y axis component is set to the YC axis (opposite to the direction of gravity), which is opposite to the YA axis. In other words, the information processing device 50 can reflect the selected installation orientation of the virtual projection device 202 in the second image from among the extracted installation orientations of the virtual projection device 202. As a result, the information processing device 50 can set an appropriate coordinate system according to the installation orientation of the virtual projection device 202.

[0139] The information processing device 50 may detect the installation status itself and determine whether the virtual surface 201 for the projection device is on the floor or the ceiling. Alternatively, the information processing device 50 may determine whether the virtual surface 201 for the projection device is on the floor or the ceiling by having the user operate a predetermined control unit to select whether the device is floor-mounted or ceiling-mounted.

[0140] Figure 26 is an image displayed on the touch panel 51 showing the virtual projection device 202 installed on the floor, and Figure 27 is an image displayed on the touch panel 51 showing the virtual projection device 202 suspended from the ceiling. In both screens, a list L showing the installation orientation of the virtual projection device 202 is displayed, allowing the user to select the current installation orientation.

[0141] This makes it easier for the user to intuitively understand the relationship between the installation orientation of the virtual projection device 202 and the projection range, and to select the optimal installation orientation. If the virtual projection device 202 has already had its angle set by rotation, it may or may not maintain the rotation angle.

[0142] <Limitations on lens shift of virtual projection device 202> A user may attempt to over-shift the lens of the virtual projection device 202, potentially exceeding the shift range (specification range) of the projection position of the actual projection device 10. In such cases, it is desirable for the information processing device 50 to inform the user of this fact.

[0143] Figure 28 shows the state in which the shift range within which the lens shift of the projection position of the virtual projection device 202 can be performed is displayed on the second image of the virtual projection device 202. The information processing device 50 displays the shift range F1 with a polygonal frame. This allows the user to know that there is a limit to the shift range of the virtual projection device 202. The information processing device 50 can control the display of the shift range F1 on the second image and the state in which the shift range F1 is not displayed on the second image. Furthermore, the display method of the shift range is not limited to the shift range F1 indicated by a polygonal frame, but may also be a dialog box, sound notification, etc., and is not particularly limited.

[0144] Figure 29 shows the state in which the information processing device 50 has clipped the lens shift of the projection position of the virtual projection device 202 at the edge of the shift range F1, thereby restricting further movement. The user is attempting to move the projection position outside the shift range F1 by moving the lens center point P1 of the virtual projection device 202. The information processing device 50 clips the projection position moved by the user and informs the user that such a change in position, i.e., setting the projection position outside the shift range F1, is impossible, using a symbol such as "×".

[0145] This allows the user to determine the lens shift of the projection position of the virtual projection device 202 after understanding the shift range that can be set on the actual device. In particular, when clipping the projection position at the edge of the shift range, it becomes easy to set the virtual projection device 202 or the virtual projection plane 204 at the upper limit of the lens shift. Furthermore, if movement outside the shift range is permitted, the virtual projection plane 204 can be moved to the desired location first, and then the position of the virtual projection device 202 can be adjusted later so that the virtual projection plane 204 falls within the shift range. This process enables such flexible position setting.

[0146] <Movement of virtual projection plane 204 in virtual projection plane priority mode> Figures 30 to 33 are simulation diagrams of an example in which the user instructs the movement of the virtual projection surface 204, i.e., the change of the position of the virtual projection surface 204, in virtual projection surface priority mode. The virtual projection surface 204 moves when the user operates one of the buttons shown in Figures 11 to 17. Here, assuming a scenario where the position of the virtual projection device 202 is not fixed, the user can freely move the virtual projection surface 204 without being aware of the position of the virtual projection device 202. On the other hand, assuming a scenario where the position of the virtual projection device 202 is fixed, the user can first determine the position of the virtual projection device 202 and then concentrate on adjusting the position of the virtual projection surface 204.

[0147] Figure 30 is a simulation diagram of the initial state in virtual projection plane priority mode, where the position of the virtual projection device 202 is not fixed. Although the position of the virtual projection device 202 is not fixed, the installation range of the virtual projection device 202 is limited, so the information processing device 50 displays the installation range F2 of the virtual projection device 202 with a border. That is, the installation range F2 is a type of second image, and is an image that displays the installation range in which the virtual projection device 202 can be placed.

[0148] The user can change the position of the virtual projection plane 204 by pressing, for example, the up / down / left / right movement button B25 (Figure 11). Figure 31 illustrates the movement of the virtual projection plane 204 to the left, i.e., movement in the negative direction of the XB axis. The user can instruct the information processing device 50 to perform such a movement by pressing the left button of the up / down / left / right movement button B25. The virtual projection plane 204 can be moved in other directions using a similar operation. In this case, the virtual projection plane 204, the virtual projection device 202, and the installable range F2 move as a whole.

[0149] Figure 32 is a simulation diagram of the initial state in virtual projection plane priority mode, where the position of the virtual projection device 202 is fixed. As shown in Figure 33, the user can move the virtual projection plane 204 to the left by the same operation as described in Figure 31.

[0150] In this case, the virtual projection surface 204 and the installable range F2 move together. On the other hand, the position of the virtual projection device 202 is fixed, and the virtual projection device 202 does not move. In this case, the information processing device 50 changes the position of the installable range F2 based on the change in the position of the virtual projection surface 204, that is, the arrangement change data of the virtual projection surface 204.

[0151] <Resizing virtual projection surface 204 in virtual projection surface priority mode> Figures 34 to 37 are simulation diagrams of an example where a user instructs a change in the size of the virtual projection surface 204 in virtual projection surface priority mode. The user can change the size of the virtual projection surface 204 by, for example, a pinch gesture on the touch panel 51, as shown in Figure 17. By changing the size of the virtual projection surface 204, the size of the installation range F2 of the virtual projection device 202 and the distance from the virtual projection device 202 to the virtual projection surface 204 change. This allows the user to visually understand the relationship between the virtual projection surface 204 and the installation range F2. This is useful when the position of the virtual projection device 202 is not fixed and it is desired to install it at the upper limit of the lens shift parameter. When the position of the virtual projection device 202 is fixed, this is useful when it is decided that the virtual projection device 202 will be installed on the ceiling, or when it is desired to move only in the projection direction.

[0152] Figure 34 is a simulation diagram of the initial state in virtual projection plane priority mode, where the position of the virtual projection device 202 is not fixed. The user can enlarge the virtual projection plane 204 (for example, by 20% horizontally and 10% vertically) by a pinch gesture, for example, by tracing the touch panel 51, as shown in Figure 35, and the installable range F2 will also enlarge accordingly. The same applies to shrinking the virtual projection plane 204. That is, when the information processing device 50 changes the size of the virtual projection plane 204 based on the placement change data, it changes the position and / or size of the installable range F2 in accordance with the change in the size of the virtual projection plane 204. In this case, the relative position of the virtual projection device 202 with respect to the installable range F2 does not change, and the lens shift parameter also does not change.

[0153] Figure 36 is a simulation diagram of the initial state in virtual projection plane priority mode, where the position of the virtual projection device 202 is fixed. As shown in Figure 37, the user can change the size of the virtual projection plane 204 by the same operation as described in Figure 35. In this case, the position of the virtual projection device 202 moves only in the projection direction, and the lens shift parameter changes.

[0154] Even in virtual projection plane priority mode, if the position of the virtual projection device 202 is not fixed, the information processing device 50 can change the position of the virtual projection device 202 based on the placement change data. In this case, the information processing device 50 changes the position of the virtual projection device 202 within the installable range F2.

[0155] Even in virtual projection plane priority mode, the information processing device 50 may display all installation orientations of the virtual projection device 202, or it may select and display either floor-standing or ceiling-mounted installation orientations according to user operation. When the installation orientation is changed, the information processing device 50 changes the orientation of the virtual projection plane 204 and the installable range F2 to correspond to the new orientation.

[0156] In other words, the information processing device 50 can also rotate the virtual projection plane 204 based on the arrangement change data, i.e., change its orientation. In this case, the installable range F2 is rotated (orientation changed) according to the rotation of the virtual projection plane 204.

[0157] In this case, the position of the virtual projection device 202 may be fixed or not fixed. However, if it is fixed, the position of the virtual projection device 202 may fall outside the range of the installable range F2 due to rotation of the installable range F2. In this case, the position of the virtual projection device 202 is set to the position when the lens shift parameter is clipped so that the position of the virtual projection device 202 is at the edge of the installable range F2. This makes it easier for the user to intuitively understand the relationship between the installation orientation of the virtual projection device 202 and the virtual projection plane 204, and to select the optimal installation orientation.

[0158] <Common points between virtual projection device priority mode and virtual projection surface priority mode> The following describes the common aspects between the virtual projection device priority mode and the virtual projection surface priority mode.

[0159] The user can rotate the virtual projection device 202 or the virtual projection surface 204 by pressing the rotation button B13 or the projection surface rotation button B24 in Figure 12. The virtual projection device 202 rotates around the Z axis, and the virtual projection surface 204 rotates around the Y axis. In virtual projection surface priority mode, the installation range of the virtual projection device 202 also rotates. This allows the user to set the virtual projection device 202 and the virtual projection surface 204 at a desired angle.

[0160] The user can change the aspect ratio of the virtual projection plane 204 by pressing the aspect ratio change button B21 in Figure 12. In this case, the image may be cropped, or the position (and possible installation range) of the virtual projection device 202 may be changed to maintain the diagonal length (size in inches) of the virtual projection plane 204. This allows the user to set the positions of the virtual projection device 202 and the virtual projection plane 204 to achieve the desired aspect ratio. The diagonal length of the virtual projection plane 204 can also be changed in conjunction with the change in aspect ratio. Furthermore, the information processing device 50 may change the distance between the virtual projection plane 204 and the virtual projection device 202 in conjunction with the change in aspect ratio.

[0161] By pressing the image setting button B22 in Figure 12, the user can display the selected image or video on the virtual projection surface 204. In other words, the image on the virtual projection surface 204 superimposed on the second image is the image selected by the user. This allows the user to understand the scene when the desired image or video is projected.

[0162] The user can rotate the image on the displayed virtual projection plane 204 by pressing the image rotation button B23 in Figure 12. The user can also zoom in or out of the virtual projection plane 204 using a pinch gesture, as explained in Figures 13 to 17.

[0163] The user can operate a designated control unit to display the parameters of the currently set virtual projection device 202. That is, the information processing device 50 can control the display device to show the projection parameters of the virtual projection device 202 corresponding to the arrangement of the virtual projection plane 204 and the virtual projection device 202 represented by the second image. The information processing device 50 may display the projection parameters in a separate area or on a separate device from the second image, or it may input the information into the second image. This allows the user to understand the parameters of the virtual projection device 202 numerically and use this information for more detailed design, such as in drawing studies.

[0164] Projection parameters include, for example, projection distance, lens shift value (which may be displayed as a distance), distance to each virtual placement plane, and the position and orientation of each object in the user-defined reference coordinate system.

[0165] Furthermore, if multiple combinations of the virtual projection plane 204 and the virtual projection device 202 exist, the information processing device 50 can also control the arrangement of a combination selected by the user from among the multiple combinations. This improves user convenience.

[0166] <Displaying the boundary of projected light> The information processing device 50 may, in some way, display the boundary between the space through which the projected light projected from the virtual projection device 202 is estimated to pass and the space through which the projected light is estimated not to pass, between the lens center point P1 of the virtual projection device 202 and the projection center point P2 of the virtual projection surface 204.

[0167] Figure 38 shows an example of how boundary H is represented, where boundary H is represented by lines connecting the four corners of the virtual projection plane 204 to the lens center point P1, defining the space through which the projected light is estimated to pass. Figure 39 shows another example of how boundary is represented, where the space through which the projected light is estimated to pass and the boundary are defined by a combination of triangles with one side of the virtual projection plane 204 as the base and the lens center point P1 as the vertex.

[0168] In this example, the second image represents the boundary of the projected light from the virtual projection device 202 to the virtual projection surface 204. This allows the user to understand the boundary through which the projected light passes, and to consider the placement of the virtual projection device 202, taking into account the observer's position and whether other equipment obstructs the projected light.

[0169] <Installation Assistance> In the case of a virtual projection device 202 whose position can be moved by lens shift, determining the size and position of the virtual projection surface 204 requires determining three points: the position of the virtual projection device 202, the projection center point of the virtual projection device 202 without lens shift, and the projection center point of the virtual projection device 202 with lens shift. On the other hand, the user's interest is solely focused on two points: the position of the virtual projection device 202 and the projection center point with lens shift. Therefore, if the size and position of the virtual projection surface 204 can be determined by specifying only these two points of interest, the user's effort can be reduced. Figures 40 to 42 illustrate a method for assisting the installation of the virtual projection device 202 based on this idea in virtual projection device priority mode.

[0170] Figure 40 shows the first step of the installation assistance. The information processing device 50 acquires relocation data related to the relocation of the virtual projection device 202. This relocation data includes data instructing a change in the position of the virtual projection device 202 and the first projection center PA of the virtual projection device 202 on the virtual projection plane 204 due to a shift in the projection position of the virtual projection device 202. The first projection center PA is the projection center point that the user ultimately desires.

[0171] Figure 41 shows the second step of the installation assistance. The information processing device 50 sets a second projection center PB, which is the projection center point without lens shift, at the point where a straight line parallel to the normal vector of the projection plane installation virtual plane 203 intersects the virtual projection plane installation virtual plane 203 when extended from the lens center point P1 of the virtual projection device 202.

[0172] Figure 42 shows the third step of the installation assist. The information processing device 50 changes the size of the virtual projection plane 204 based on the second projection center PB. Specifically, the information processing device 50 calculates the projection distance d, which is the distance between the lens center point P1 and the second projection center PB, and determines the size of the virtual projection plane 204. Furthermore, the information processing device 50 determines the lens shift amount from the positions of the first projection center PA and the second projection center PB and the size of the virtual projection plane 204. Then, the information processing device 50 changes the orientation of the virtual projection device 202 so that it faces the second projection center PB.

[0173] In virtual projection plane priority mode, the information processing device 50 can determine the candidate installation range for the virtual projection device 202 by determining the projection center and the size of the virtual projection plane 204. By appropriately determining the initial position of the virtual projection device 202 within the candidate installation range, the information processing device 50 can reduce the burden on the user during subsequent adjustments.

[0174] When determining the position of the virtual projection device 202 within the possible installation range F2, the information processing device 50 may set the position of the virtual projection device 202 to the position when zoom is 100% and no lens shift is performed. Alternatively, the position of the virtual projection device 202 may be set by the intersection of a line extended in the direction of the normal to the imaging surface from the point where the user taps the touch panel 51 and the candidate installation range. Furthermore, the position of the virtual projection device 202 may be set by the intersection of a line extended in the direction of the normal to the imaging surface from the camera center point when the user presses the installation button and the candidate installation range.

[0175] Furthermore, the information processing device 50 can also be configured to change the size of the virtual projection plane 204 using a general zoom function (such as optical zoom or digital zoom).

[0176] Each of the above embodiments and modifications can be implemented in combination.

[0177] This specification contains at least the following:

[0178] (1) An information processing device equipped with a processor, The above processor is First image data representing the first image obtained by imaging with the imaging device is acquired. We obtain arrangement data regarding the arrangement of the virtual projection plane and virtual projection device in the space shown in the first image above. Obtain arrangement change data relating to the arrangement change of the virtual projection plane and / or the virtual projection device in the first image above. Based on the above arrangement change data, the virtual projection plane and / or the virtual projection device whose arrangement has been changed generates a second image representing the second image displayed in the first image. Output the above second image data to the output destination. Information processing device.

[0179] (2) (1) The information processing device described above, The above arrangement change data includes data indicating a change to at least one of the following: the position of the virtual projection plane and / or the virtual projection device, the orientation of the virtual projection plane and / or the virtual projection device, and the size of the virtual projection plane. Information processing device.

[0180] (3) (1) or (2) the information processing device described above, Equipped with a display device, The output destination is the display device mentioned above. Information processing device.

[0181] (4) An information processing device described in any one of (1) to (3), The imaging device described above is provided, Information processing device.

[0182] (5) An information processing device described in any one of items (1) to (4), It includes an input unit that accepts the above-mentioned layout change data from the user. Information processing device.

[0183] (6) (5) The information processing device described above, When the above processor receives the above-mentioned reconfiguration data from the user, it performs control to display an image on a display device that includes an operation image for instructing the reconfiguration of the virtual projection surface and an operation image for instructing the reconfiguration of the virtual projection device. Information processing device.

[0184] (7) (5) The information processing device described above, The above processor is capable of switching between a state in which an operation image for instructing a change in the arrangement of the virtual projection surface is displayed on the display device when it receives the arrangement change data from the user, and a state in which an operation image for instructing a change in the arrangement of the virtual projection device is displayed on the display device. Information processing device.

[0185] (8) (5) The information processing device described above, The processor performs at least one of the following: control to change the arrangement of the virtual projection plane in the second image displayed on the display device in response to an operation performed by the user on the virtual projection plane; and control to change the arrangement of the virtual projection device in the second image displayed on the display device in response to an operation performed by the user on the virtual projection device. Information processing device.

[0186] (9) An information processing device described in any one of items (1) to (8), When the above processor changes the position of the virtual projection device based on the above repositioning data, it maintains the position of the virtual projection plane. Information processing device.

[0187] (10) (9) The information processing device described above, When the above processor changes the position of the virtual projection device in a direction different from the lens optical axis direction of the virtual projection device, it maintains the position of the virtual projection plane by changing the parameter for shifting the projection position of the virtual projection device. Information processing device.

[0188] (11) (9) or (10) information processing apparatus, The above processor changes the size of the virtual projection surface when changing the position of the virtual projection device in the lens optical axis direction of the virtual projection device. Information processing device.

[0189] (12) An information processing device described in any one of items (1) to (11), When the above processor rotates the virtual projection device about the axis in the direction of the lens optical axis of the virtual projection device based on the above arrangement change data, it rotates the virtual projection surface in accordance with the rotation of the virtual projection device. Information processing device.

[0190] (13) An information processing device described in any one of items (1) to (12), When the above processor changes the position of the virtual projection plane based on the above arrangement change data, it maintains the position of the virtual projection device. Information processing device.

[0191] (14) An information processing device described in any one of items (1) to (13), When the above processor rotates the virtual projection plane about an axis perpendicular to the virtual projection plane based on the above arrangement change data, it rotates the virtual projection device in accordance with the rotation of the virtual projection plane. Information processing device.

[0192] (15) An information processing device described in any one of items (1) to (14), The second image above shows the installation range in which the virtual projection device can be placed. Information processing device.

[0193] (16) (15) An information processing device as described above, When the above processor changes the position of the virtual projection plane based on the above arrangement change data, it changes the position of the installable range in accordance with the change in the position of the virtual projection plane. Information processing device.

[0194] (17) An information processing device as described in (15) or (16), When the above processor rotates the virtual projection plane based on the above arrangement change data, it rotates the installable range in accordance with the rotation of the virtual projection plane. Information processing device.

[0195] (18) An information processing device as described in any one of paragraphs (15) to (17), When the above processor changes the size of the virtual projection plane based on the above arrangement change data, it changes the position and / or size of the installable range in accordance with the change in the size of the virtual projection plane. Information processing device.

[0196] (19) An information processing device as described in any one of paragraphs (15) to (18), When the above processor changes the position of the virtual projection device based on the above-mentioned relocation data, it changes the position of the virtual projection device within the above-mentioned installable range. Information processing device.

[0197] (20) An information processing device described in any one of items (1) to (19), The image displayed on the virtual projection plane included in the second image above is an image selected by the user. Information processing device.

[0198] (twenty one) (20) The information processing device described above, The above processor rotates, enlarges, and reduces the image of the virtual projection surface in response to user input. Information processing device.

[0199] (twenty two) An information processing device described in any one of items (1) to (21), The above processor changes the aspect ratio of the virtual projection plane in response to user input. Information processing device.

[0200] (twenty three) (22) An information processing device as described above, The above processor changes the length of the diagonal of the virtual projection plane in response to the change in aspect ratio. Information processing device.

[0201] (twenty four) (22) An information processing device as described above, The above processor changes the distance between the virtual projection plane and the virtual projection device in accordance with the change in aspect ratio. Information processing device.

[0202] (twenty five) An information processing device described in any one of items (1) to (24), The above processor is From the candidate installation orientations of the virtual projection device described above, an installation orientation based on the installation position of the virtual projection device in the above space is extracted. The installation posture of the virtual projection device selected from the extracted installation postures is reflected in the second image. Information processing device.

[0203] (26) An information processing device described in any one of items (1) to (25), The above processor is capable of controlling the switching between a state in which the shift range for shifting the projection position of the virtual projection device is displayed in the second image, and a state in which the shift range is not displayed in the second image. Information processing device.

[0204] (27) An information processing device described in any one of items (1) to (26), The above processor controls the display device to show the projection parameters of the virtual projection device corresponding to the arrangement of the virtual projection plane and the virtual projection device represented by the second image. Information processing device.

[0205] (28) An information processing device described in any one of paragraphs (1) to (27), The above processor, when there are multiple combinations of the above virtual projection plane and the above virtual projection device, performs control to make the combination selected by user operation from among the multiple combinations subject to rearrangement. Information processing device.

[0206] (29) An information processing device described in any one of items (1) to (28), The second image above represents the boundary of the projected light from the virtual projection device to the virtual projection surface. Information processing device.

[0207] (30) An information processing device described in any one of items (1) to (29), The above arrangement change data includes data that instructs a change in the position of the virtual projection device and the first projection center of the virtual projection device on the virtual projection plane due to a shift in the projection position of the virtual projection device. The above processor is Based on the above arrangement change data, the second projection center of the virtual projection device on the virtual projection plane is set when the above projection position is not shifted. The size of the virtual projection plane is changed based on the second projection center mentioned above. Information processing device.

[0208] (31) (30) The information processing apparatus described above, The above processor changes the orientation of the virtual projection device so that it faces the second projection center. Information processing device.

[0209] (32) An information processing method using an information processing device, The processor of the above information processing device First image data representing the first image obtained by imaging with the imaging device is acquired. We obtain arrangement data regarding the arrangement of the virtual projection plane and virtual projection device in the space shown in the first image above. Obtain arrangement change data relating to the arrangement change of the virtual projection plane and / or the virtual projection device in the first image above. Based on the above arrangement change data, the virtual projection plane and / or the virtual projection device whose arrangement has been changed generates a second image representing the second image displayed in the first image. Output the above second image data to the output destination. Information processing methods.

[0210] (33) An information processing program for an information processing device, The processor of the above-mentioned information processing device: First image data representing the first image obtained by imaging with the imaging device is acquired. We obtain arrangement data regarding the arrangement of the virtual projection plane and virtual projection device in the space shown in the first image above. Obtain arrangement change data relating to the arrangement change of the virtual projection plane and / or the virtual projection device in the first image above. Based on the above arrangement change data, the virtual projection plane and / or the virtual projection device whose arrangement has been changed generates a second image representing the second image displayed in the first image. Output the second image data to the output destination. An information processing program for executing the process.

[0211] As described above, various embodiments have been explained, but it is needless to say that the present invention is not limited to such examples. It is obvious that those skilled in the art can conceive of various modification examples or correction examples within the scope described in the claims, and it is naturally understood that they also belong to the technical scope of the present invention. Further, within the scope not departing from the gist of the invention, the components in the above embodiments may be arbitrarily combined.

[0212] This application is based on a Japanese patent application (Japanese Patent Application No. 2021-214489) filed on December 28, 2021, the content of which is incorporated herein by reference.

Explanation of Reference Numerals

[0213] 1 Projection unit 2 Operation reception unit 4 Control device 4a, 62 Memory 2a, 2b, 3a, 3c, 15a Aperture 2A, 3A Hollow part 6 Object to be projected 10 Projection device 11 Projection range 12 Light modulation unit 15 Housing 20 Left - right direction 21 Light source 22 Light modulation part 23 Projection optical system 24 Control circuit 31 Second optical system 32, 122 Reflective member 33 Third optical system 34 Lens 50 Information processing device 51 Touch panel 61 Processor 63 Communication interface 64 User interface 65 Sensors 69 bus 70 Spatial Images 100 zoom 101 Main body 102 First Member 103 Second Member 104 Projection direction changing mechanism 105 Shift mechanism 106 Optical Units 121 1st optical system 201 Projection device installation virtual surface 202 Virtual Projection Device 203 Projection plane setting virtual plane 204 Virtual projection plane A1 Virtual projection device operation area A2 Virtual projection plane operation area B11 Operation Target Switching Button B12 Posture Change Button B13 Rotation Button B14 Up / Down Movement Buttons B15 Forward / Backward / Left / Right Movement Buttons B21 Aspect Ratio Change Button B22 Image settings button B23 Image Rotation Button B24 Projection plane rotation button B25 Up / Down / Left / Right Movement Buttons B31 Size Change Lock Release Button B32 Horizontal movement lock release button B33 Rotation lock release button d1 Projection distance D1, D3, D4 Distance F1 Shift Range F2 Installation range G1 Image P1 Lens center point P2~P4 Projection center point UI1~UI4 Operation Images

Claims

1. An information processing device equipped with a processor, The aforementioned processor, First image data representing the first image obtained by imaging with the imaging device is acquired. Arrangement data relating to the arrangement of the virtual projection plane and virtual projection device in the space shown in the first image is obtained. Arrangement change data is acquired, which includes data instructing a change to the position of the virtual projection device and the first projection center, which is the user-desired position on the virtual projection plane after the projection position of the virtual projection device has been shifted. Based on the aforementioned arrangement change data, a second projection center is set, which is the projection center point of the virtual projection device on the virtual projection plane before the shift of the projection position. The size of the virtual projection plane is changed based on the distance between the virtual projection device and the second projection center. Based on the arrangement change data, a second image data is generated representing a second image in which the virtual projection surface and the virtual projection device are displayed in the first image after the arrangement of at least one of the virtual projection surface and the virtual projection device has been changed. Output the second image data to the output destination. Information processing device.

2. An information processing apparatus according to claim 1, Equipped with a display device, The output destination is the display device. Information processing device.

3. An information processing apparatus according to claim 1, The imaging device is provided as described above. Information processing device.

4. An information processing apparatus according to claim 1, It includes an input unit that receives the aforementioned layout change data from the user. Information processing device.

5. An information processing apparatus according to claim 4, When the processor receives the arrangement change data from the user, it controls the display device to display an image that includes an operation image for instructing the arrangement change of the virtual projection surface and an operation image for instructing the arrangement change of the virtual projection device. Information processing device.

6. An information processing apparatus according to claim 4, The processor is capable of switching between a state in which an operation image for instructing a change in the arrangement of the virtual projection surface is displayed on the display device when it receives the arrangement change data from the user, and a state in which an operation image for instructing a change in the arrangement of the virtual projection device is displayed on the display device. Information processing device.

7. An information processing apparatus according to claim 4, The processor performs at least one of the following: control to change the arrangement of the virtual projection plane in the second image displayed on the display device in response to an operation performed by the user on the virtual projection plane; and control to change the arrangement of the virtual projection device in the second image displayed on the display device in response to an operation performed by the user on the virtual projection device. Information processing device.

8. An information processing apparatus according to claim 1, When the processor changes the position of the virtual projection device based on the repositioning data, it maintains the position of the virtual projection plane. Information processing device.

9. An information processing apparatus according to claim 8, When the processor changes the position of the virtual projection device in a direction different from the lens optical axis direction of the virtual projection device, it maintains the position of the virtual projection plane by changing the parameter for shifting the projection position of the virtual projection device. Information processing device.

10. An information processing apparatus according to claim 8, When the processor changes the position of the virtual projection device in the lens optical axis direction of the virtual projection device, it changes the size of the virtual projection surface. Information processing device.

11. An information processing apparatus according to claim 1, When the processor rotates the virtual projection device about the axis in the direction of the lens optical axis of the virtual projection device based on the arrangement change data, it rotates the virtual projection surface in accordance with the rotation of the virtual projection device. Information processing device.

12. An information processing apparatus according to claim 1, When the processor changes the position of the virtual projection plane based on the repositioning data, it maintains the position of the virtual projection device. Information processing device.

13. An information processing apparatus according to claim 1, When the processor rotates the virtual projection plane about an axis perpendicular to the virtual projection plane based on the arrangement change data, it rotates the virtual projection device in accordance with the rotation of the virtual projection plane. Information processing device.

14. An information processing apparatus according to claim 1, The second image is an image showing the installation range in which the virtual projection device can be placed. Information processing device.

15. An information processing apparatus according to claim 14, When the processor changes the position of the virtual projection plane based on the arrangement change data, it changes the position of the installable range in accordance with the change in the position of the virtual projection plane. Information processing device.

16. An information processing apparatus according to claim 14, When the processor rotates the virtual projection plane based on the arrangement change data, it rotates the installable range in accordance with the rotation of the virtual projection plane. Information processing device.

17. An information processing apparatus according to claim 14, When the processor changes the size of the virtual projection plane based on the arrangement change data, it changes the position and / or size of the installable range in accordance with the change in the size of the virtual projection plane. Information processing device.

18. An information processing apparatus according to claim 14, When the processor changes the position of the virtual projection device based on the relocation data, it changes the position of the virtual projection device within the installable range. Information processing device.

19. An information processing apparatus according to claim 1, The image displayed on the virtual projection plane included in the second image is an image selected by the user. Information processing device.

20. An information processing apparatus according to claim 19, The processor rotates, enlarges, and reduces the image of the virtual projection surface in response to user input. Information processing device.

21. An information processing apparatus according to claim 1, The processor changes the aspect ratio of the virtual projection plane in response to user input. Information processing device.

22. An information processing apparatus according to claim 21, The processor changes the length of the diagonal of the virtual projection plane in accordance with the change in aspect ratio. Information processing device.

23. An information processing apparatus according to claim 1, The aforementioned processor, From among the candidate installation orientations for the virtual projection device, an installation orientation based on the installation position of the virtual projection device in the space is extracted. The installation posture of the virtual projection device selected from the extracted installation postures is reflected in the second image. Information processing device.

24. An information processing device according to claim 1, The processor is capable of controlling the switching between a state in which a shift range in which the projection position of the virtual projection device can be shifted is displayed in the second image, and a state in which the shift range is not displayed in the second image. Information processing device.

25. An information processing device according to claim 1, The processor controls the display device to display projection parameters of the virtual projection device corresponding to the arrangement of the virtual projection plane and the virtual projection device represented by the second image. Information processing device.

26. An information processing device according to claim 1, If there are multiple combinations of the virtual projection plane and the virtual projection device, the processor performs control to make the combination selected by user operation from among the multiple combinations subject to rearrangement. Information processing device.

27. ​​An information processing apparatus according to claim 1, The second image is an image that includes an image representing the boundary of the projected light from the virtual projection device to the virtual projection surface. Information processing device.

28. An information processing device according to any one of claims 1 to 27, The processor changes the orientation of the virtual projection device so that it faces the second projection center. Information processing device.

29. An information processing method using an information processing device, The processor of the aforementioned information processing device First image data representing the first image obtained by imaging with the imaging device is acquired. Arrangement data relating to the arrangement of the virtual projection plane and virtual projection device in the space shown in the first image is obtained. Arrangement change data is acquired, which includes data instructing a change to the position of the virtual projection device and the first projection center, which is the user-desired position on the virtual projection plane after the projection position of the virtual projection device has been shifted. Based on the aforementioned arrangement change data, a second projection center is set, which is the projection center point on the virtual projection plane before the shift of the projection position. The size of the virtual projection plane is changed based on the distance between the virtual projection device and the second projection center. Based on the arrangement change data, a second image data is generated representing a second image in which the virtual projection surface and the virtual projection device are displayed in the first image after the arrangement of at least one of the virtual projection surface and the virtual projection device has been changed. Output the second image data to the output destination. Information processing methods.

30. An information processing program for an information processing device, The processor of the aforementioned information processing device, First image data representing the first image obtained by imaging with the imaging device is acquired. Arrangement data relating to the arrangement of the virtual projection plane and virtual projection device in the space shown in the first image is obtained. Arrangement change data is acquired, which includes data instructing a change to the position of the virtual projection device and the first projection center, which is the user-desired position on the virtual projection plane after the projection position of the virtual projection device has been shifted. Based on the aforementioned arrangement change data, a second projection center is set, which is the projection center point of the virtual projection device on the virtual projection plane before the shift of the projection position. The size of the virtual projection plane is changed based on the distance between the virtual projection device and the second projection center. Based on the arrangement change data, a second image data is generated representing a second image in which the virtual projection surface and the virtual projection device are displayed in the first image after the arrangement of at least one of the virtual projection surface and the virtual projection device has been changed. Output the second image data to the output destination. An information processing program used to execute a process.

Images