Image processing device, image processing method, and image processing program

Abstract

Provided are an image processing device, an image processing method, and an image processing program which make it possible to improve a user's ease-of-use relating to arrangement of a projection plane and a projection device.　An image processing device (50) acquires first image data obtained by capturing images of a physical space (70) using an image capture device. Additionally, on the basis of first position data representing a first position (81) in the physical space (70) and first normal vector data representing a first normal vector (82) corresponding to a wall (72) present at the first position (81) in the physical space (70), the image processing device (50) generates first virtual projection plane data representing a first virtual projection plane (111), and first virtual projection device data representing a first virtual projection device (112). Furthermore, on the basis of these data, the image processing device (50) generates and outputs, to a physical space image (90) represented by the first image data, the first virtual projection plane (111) and second image data representing a second image that is displayed by the first virtual projection device (112).

Description

Image processing device, image processing method, and image processing program

[0001] The present invention relates to an image processing device, an image processing method, and an image processing program.

[0002] Patent document 1 describes a projection image adjustment system that stores virtual environment installation information indicating the installation state of a projector installed so as to obtain a desired image projection state on a projection target in a virtual space, and the control setting values ​​of the projector at that time, acquires real environment installation information indicating the installation state of the projector in real space, controls the operation of the projector, corrects the control setting values ​​based on the virtual environment installation information and the real environment installation information so that there is no difference between the image projection state in real space and the desired image projection state, and controls operation based on the corrected control setting values.

[0003] Patent document 2 describes an image processing device that acquires related information relating to an object using an image of the object, generates a related image from the related information, generates a superimposed image in which the related image is superimposed on an image containing the object, and projects the generated superimposed image.

[0004] Patent document 3 describes an image processing device that acquires an input image generated by capturing an image of a real space using an imaging device, outputs an output image to a projection device for superimposing a virtual object associated with a real object shown in the input image, projects the output image onto the real object, and controls the projection of the output image by the projection device based on the position of the real object recognized using the input image.

[0005] Japanese Patent Publication No. 2018-005115 Japanese Patent Publication No. 2017-073717 Japanese Patent Publication No. 2013-235374

[0006] One embodiment of the technique of the present disclosure provides an image processing device, an image processing method, and an image processing program that can improve user convenience regarding the placement of a projection surface or a projection device.

[0007] An image processing device of one aspect of the present invention is an image processing device having a processor, wherein the processor acquires first image data obtained by capturing an image of a space with an imaging device, generates first virtual projection plane data representing a first virtual projection plane and first virtual projection device data representing a first virtual projection device based on first position data representing a first position in the space and first normal vector data representing a first normal vector of a first surface corresponding to an object present at the first position in the space, generates second image data representing a second image in which the first virtual projection plane and the first virtual projection device are displayed on a first image represented by the first image data, based on the first image data, the first virtual projection plane data, and the first virtual projection device data, and outputs the second image data to an output destination.

[0008] In one aspect of the image processing method of the present invention, a processor of an image processing device acquires first image data obtained by capturing an image of a space with an imaging device, generates first virtual projection plane data representing a first virtual projection plane and first virtual projection device data representing a first virtual projection device based on first position data representing a first position in the space and first normal vector data representing a first normal vector of a first surface corresponding to an object located at the first position in the space, generates second image data representing a second image in which the first virtual projection plane and the first virtual projection device are displayed on a first image represented by the first image data, based on the first image data, the first virtual projection plane data, and the first virtual projection device data, and outputs the second image data to an output destination.

[0009] An image processing program according to one aspect of the present invention causes a processor of an image processing device to acquire first image data obtained by capturing an image of a space with an imaging device, generate first virtual projection plane data representing a first virtual projection plane and first virtual projection device data representing a first virtual projection device based on first position data representing a first position in the space and first normal vector data representing a first normal vector of a first surface corresponding to an object located at the first position in the space, generate second image data representing a second image in which the first virtual projection plane and the first virtual projection device are displayed on a first image represented by the first image data, based on the first image data, the first virtual projection plane data, and the first virtual projection device data, and output the second image data to an output destination.

[0010] According to the present invention, it is possible to provide an image processing device, an image processing method, and an image processing program that can improve user convenience regarding the placement of a projection surface and a projection device.

[0011] 14 is a schematic diagram showing an example of a projection device 10 that is a target for installation support by the image processing device of the first embodiment. FIG. 14 is a schematic diagram showing an example of the internal configuration of the projection unit 1 shown in FIG. 1. FIG. 14 is a schematic diagram showing the external configuration of the projection device 10. FIG. 14 is a schematic cross-sectional view of the optical unit 106 of the projection device 10 shown in FIG. 3. FIG. 14 is a diagram showing an example of the external appearance of the image processing device 50. FIG. 14 is a diagram showing an example of the hardware configuration of the image processing device 50. FIG. 14 is a diagram showing an example of acquiring the attitude of the imaging device of the image processing device 50. FIG. 14 is a diagram showing an example of capturing a first image and acquiring a first position. FIG. 14 is a diagram showing an example of a physical space image represented by first image data obtained by capturing an image of a physical space 70. FIG. 14 is a flowchart showing an example of processing by the image processing device 50. FIG. 14 is an example (part 1) of an image displayed by the image processing device 50 in the processing shown in FIG. 10. FIG. 14 is an example (part 2) of an image displayed by the image processing device 50 in the processing shown in FIG. 10. FIG. 14 is a diagram showing an example of detection of an edge of a physical plane on which a first virtual projection surface 111 is placed in the physical space 70. FIG. 14 is a flowchart showing an example of a process for determining a first position 81. FIG. 14 is a diagram (part 1) showing an example of determining the first position 81 in the determination process of FIG. 17 is a flowchart showing an example of a process for determining the size of the first virtual projection plane 111. FIG. 18 is a diagram (part 1) showing an example of a process for determining the size of the first virtual projection plane 111 and the first position 81 in the determination process of FIG. 17. FIG. 19 is a diagram (part 2) showing an example of a process for determining the size of the first virtual projection plane 111 and the first position 81 in the determination process of FIG. 17. FIG. 20 is a diagram (part 1) showing an example of an operation unit for moving the first virtual projection plane 111. FIG. 21 is a diagram (part 1) showing an example of an operation unit for moving the first virtual projection plane 111. FIG. 22 is a diagram (part 2) showing an example of an operation unit for changing the angle of the first virtual projection plane 111. FIG. 23 is a diagram showing an example of an operation unit for changing the angle of the first virtual projection plane 111. FIG. 24 is a diagram showing an example of capturing a first image and acquiring the first and second positions. FIG. 25 is a diagram showing an example of a second virtual projection plane based on the second position. FIG. 26 is a diagram showing an example of coordinate axes for movement of the first virtual projection device 112. FIG. 10 is a diagram (part 1) showing an example of an operation unit for moving the first virtual projection device 112 in the x-axis direction or the z-axis direction.FIG. 2 is a diagram (part 2) showing an example of an operation unit for moving the first virtual projection device 112 in the x-axis direction or the z-axis direction. FIG. 3 is a diagram (part 1) showing an example of an operation unit for moving the first virtual projection device 112 in the y-axis direction. FIG. 4 is a diagram (part 2) showing an example of an operation unit for moving the first virtual projection device 112 in the y-axis direction. FIG. 5 is a diagram showing an example of a physical curved surface on which the projection surface 11 is placed in embodiment 2. FIG. 6 is a diagram showing an example of designating a second position group. FIG. 7 is a diagram showing an example of a first virtual curved surface virtually showing a wall 310.

[0012] Hereinafter, examples of embodiments of the present invention will be described with reference to the drawings.

[0013] First Embodiment <Example of Projection Device 10 to be Changed in Arrangement by Image Processing Device of First Embodiment> FIG. 1 is a schematic diagram showing an example of a projection device 10 to be supported in installation by the image processing device of the first embodiment.

[0014] The image processing device of the first embodiment can be used, for example, to assist in the placement of a projection device 10. The projection device 10 includes a projection unit 1, a control device 4, and an operation reception unit 2. The projection unit 1 is configured, for example, by a liquid crystal projector or a projector using LCOS (Liquid Crystal On Silicon). In the following description, the projection unit 1 is assumed to be a liquid crystal projector.

[0015] The control device 4 is a control device that controls projection by the projection device 10. The control device 4 is a device that includes a control unit configured with various processors, a communication interface (not shown) for communicating with each unit, and a memory 4a such as a hard disk, an SSD (Solid State Drive), or a ROM (Read Only Memory), and controls the projection unit 1 in an integrated manner.

[0016] The various processors in the control unit of the control device 4 include a CPU (Central Processing Unit), which is a general-purpose processor that executes programs to perform various processes, a programmable logic device (PLD), which is a processor whose circuit configuration can be changed after manufacture, such as an FPGA (Field Programmable Gate Array), or a dedicated electrical circuit, such as an ASIC (Application Specific Integrated Circuit), which is a processor having a circuit configuration designed specifically to perform specific processing.

[0017] More specifically, the structure of these various processors is an electric circuit that combines circuit elements such as semiconductor elements. The control unit of the control device 4 may be configured with one of the various processors, or may be configured with a combination of two or more processors of the same or different types (for example, a combination of multiple FPGAs or a combination of a CPU and an FPGA).

[0018] The operation reception unit 2 detects instructions from the user by receiving various operations from the user. The operation reception unit 2 may be a button, key, joystick, or the like provided on the control device 4, or may be a receiving unit or the like that receives signals from a remote controller that remotely controls the control device 4.

[0019] The projection target 6 is an object such as a screen or a wall that has a projection surface on which a projected image is displayed by the projection unit 1. In the example shown in Fig. 1, the projection surface of the projection target 6 is a rectangular plane. The top, bottom, left, and right of the projection target 6 in Fig. 1 are assumed to be the top, bottom, left, and right of the actual projection target 6.

[0020] The projection surface 11, shown by a dashed line, is an area of ​​the object 6 onto which projection light is irradiated by the projection unit 1. In the example shown in Fig. 1 , the projection surface 11 is rectangular. The projection surface 11 is a part or the entire projectable range onto which projection can be performed by the projection unit 1.

[0021] The projection unit 1, the control device 4, and the operation reception unit 2 may be realized by, for example, a single device (see, for example, FIGS. 3 and 4), or the projection unit 1, the control device 4, and the operation reception unit 2 may be separate devices that communicate with each other and cooperate with each other.

[0022] <Internal Configuration of Projection Unit 1 Shown in FIG. 1> FIG. 2 is a schematic diagram showing an example of the internal configuration of the projection unit 1 shown in FIG.

[0023] As shown in FIG. 2, the projection unit 1 includes a light source 21, a light modulation unit 22, a projection optical system 23, and a control circuit 24.

[0024] The light source 21 includes a light emitting element such as a laser or an LED (Light Emitting Diode), and emits, for example, white light.

[0025] The light modulation unit 22 is composed of three liquid crystal panels that modulate the color light emitted from the light source 21 and separated into three colors, red, blue, and green, by a color separation mechanism (not shown), based on image information, and emit each color image. These three liquid crystal panels may be equipped with red, blue, and green filters, respectively, and the white light emitted from the light source 21 may be modulated by each liquid crystal panel to emit each color image.

[0026] The projection optical system 23 receives light from the light source 21 and the light modulation unit 22, and is configured by, for example, a relay optical system including at least one lens. The light that passes through the projection optical system 23 is projected onto the object 6 to be projected.

[0027] The area of ​​the object 6 that is irradiated with light that passes through the entire range of the light modulation unit 22 is the projectable range in which projection by the projection unit 1 is possible. Within this projectable range, the area that is irradiated with light that actually passes through the light modulation unit 22 is the projection surface 11. For example, by controlling the size, position, and shape of the area of ​​the light modulation unit 22 through which light passes, the size, position, and shape of the projection surface 11 can be changed within the projectable range.

[0028] The control circuit 24 controls the light source 21, the light modulation unit 22, and the projection optical system 23 based on the display data input from the control device 4, thereby projecting an image based on this display data onto the projection target 6. The display data input to the control circuit 24 is made up of three pieces of data: red display data, blue display data, and green display data.

[0029] The control circuit 24 also enlarges or reduces the projection surface 11 (see FIG. 1 ) of the projection unit 1 by changing the projection optical system 23 based on a command input from the control device 4. The control device 4 may also move the projection surface 11 of the projection unit 1 by changing the projection optical system 23 based on an operation from the user received by the operation receiving unit 2.

[0030] The projection device 10 also includes a shift mechanism that mechanically or optically moves the projection surface 11 while maintaining the image circle of the projection optical system 23. The image circle of the projection optical system 23 is the area through which projection light incident on the projection optical system 23 passes through the projection optical system 23 appropriately in terms of light intensity loss, color separation, peripheral curvature, etc.

[0031] The shift mechanism is realized by at least one of an optical system shift mechanism that performs an optical system shift and an electronic shift mechanism that performs an electronic shift.

[0032] The optical system shift mechanism is, for example, a mechanism that moves the projection optical system 23 in a direction perpendicular to the optical axis (see, for example, FIGS. 3 and 4), or a mechanism that moves the light modulation unit 22 in a direction perpendicular to the optical axis instead of moving the projection optical system 23. The optical system shift mechanism may also be a mechanism that moves the projection optical system 23 and the light modulation unit 22 in combination.

[0033] The electronic shift mechanism is a mechanism that shifts the pseudo projection surface 11 by changing the range through which light is transmitted in the light modulation section 22 .

[0034] The projection device 10 may also include a projection direction change mechanism that moves the projection surface 11 together with the image circle of the projection optical system 23. The projection direction change mechanism is a mechanism that changes the projection direction of the projection unit 1 by changing the orientation of the projection unit 1 through mechanical rotation (see, for example, FIGS. 3 and 4).

[0035] <Mechanical Configuration of Projection Device 10> Fig. 3 is a schematic diagram showing the external configuration of the projection device 10. Fig. 4 is a schematic cross-sectional view of the optical unit 106 of the projection device 10 shown in Fig. 3. Fig. 4 shows a cross section taken along a plane along the optical path of light emitted from the main body 101 shown in Fig. 3.

[0036] 3, the projection device 10 includes a main body 101 and an optical unit 106 that protrudes from the main body 101. In the configuration shown in Fig. 3, the operation reception unit 2, the control device 4, and the light source 21, light modulation unit 22, and control circuit 24 in the projection unit 1 are provided in the main body 101. The projection optical system 23 in the projection unit 1 is provided in the optical unit 106.

[0037] The optical unit 106 includes a first member 102 supported by the main body 101 and a second member 103 supported by the first member 102 .

[0038] The first member 102 and the second member 103 may be an integrated member. The optical unit 106 may be configured to be detachable from the main body 101 (in other words, replaceable).

[0039] The main body 101 has a housing 15 (see FIG. 4) in which an opening 15a (see FIG. 4) for passing light is formed at a portion connected to the optical unit 106.

[0040] As shown in Figure 3, inside the housing 15 of the main body 101, there is provided a light source 21 and a light modulation unit 12 including a light modulation section 22 (see Figure 2) that spatially modulates the light emitted from the light source 21 based on input image data to generate an image.

[0041] The light emitted from the light source 21 is incident on the light modulation section 22 of the light modulation unit 12, and is spatially modulated by the light modulation section 22 before being emitted.

[0042] As shown in Figure 4, the image formed by the light spatially modulated by the light modulation unit 12 passes through the opening 15a of the housing 15 and enters the optical unit 106, and is projected onto the projection target 6, which is the projection object, so that the image G1 becomes visible to the observer.

[0043] As shown in Figure 4, the optical unit 106 includes a first member 102 having a hollow portion 2A connected to the inside of the main body portion 101, a second member 103 having a hollow portion 3A connected to the hollow portion 2A, a first optical system 121 and a reflecting member 122 arranged in the hollow portion 2A, a second optical system 31, a reflecting member 32, a third optical system 33, and a lens 34 arranged in the hollow portion 3A, a shift mechanism 105, and a projection direction change mechanism 104.

[0044] The first member 102 is a member having, for example, a rectangular cross-sectional outer shape, and the openings 2a and 2b are formed on planes perpendicular to each other. The first member 102 is supported by the main body 101 with the opening 2a positioned opposite the opening 15a of the main body 101. Light emitted from the light modulation section 22 of the light modulation unit 12 in the main body 101 passes through the openings 15a and 2a and enters the hollow section 2A of the first member 102.

[0045] The direction of light incident from the main body 101 into the hollow portion 2A is referred to as direction X1, the opposite direction to direction X1 is referred to as direction X2, and directions X1 and X2 are collectively referred to as direction X. In addition, in Figure 4, the direction from the front of the page toward the back and the opposite direction are referred to as direction Z. Of direction Z, the direction from the front of the page toward the back is referred to as direction Z1, and the direction from the back of the page toward the front is referred to as direction Z2.

[0046] Furthermore, a direction perpendicular to direction X and direction Z is referred to as direction Y, and within direction Y, the upward direction in Fig. 4 is referred to as direction Y1, and the downward direction in Fig. 4 is referred to as direction Y2. In the example of Fig. 4, the projection device 10 is disposed so that direction Y2 is the vertical direction.

[0047] The projection optical system 23 shown in Fig. 2 is composed of a first optical system 121, a reflecting member 122, a second optical system 31, a reflecting member 32, a third optical system 33, and a lens 34. Fig. 4 shows the optical axis K of the projection optical system 23. The first optical system 121, the reflecting member 122, the second optical system 31, the reflecting member 32, the third optical system 33, and the lens 34 are arranged along the optical axis K in this order from the light modulation unit 22 side.

[0048] The first optical system 121 includes at least one lens, and guides light traveling in the direction X1 that is incident on the first member 102 from the main body 101 to the reflecting member 122 .

[0049] The reflecting member 122 reflects the light incident from the first optical system 121 in the direction Y1. The reflecting member 122 is formed of, for example, a mirror. The first member 102 has an opening 2b formed on the optical path of the light reflected by the reflecting member 122. The reflected light passes through the opening 2b and travels to the hollow portion 3A of the second member 103.

[0050] The second member 103 is a member having a substantially T-shaped cross-sectional outer shape, and has an opening 3a formed in a position facing the opening 2b of the first member 102. Light from the main body 101 that passes through the opening 2b of the first member 102 passes through this opening 3a and enters the hollow portion 3A of the second member 103. The cross-sectional outer shapes of the first member 102 and the second member 103 are arbitrary and are not limited to those described above.

[0051] The second optical system 31 includes at least one lens, and guides the light incident from the first member 102 to the reflecting member 32 .

[0052] The reflecting member 32 reflects the light incident from the second optical system 31 in the direction X2 and guides the light to the third optical system 33. The reflecting member 32 is formed of, for example, a mirror.

[0053] The third optical system 33 includes at least one lens, and guides the light reflected by the reflecting member 32 to a lens 34 .

[0054] The lens 34 is disposed at the end of the second member 103 in the direction X2 so as to cover the opening 3c formed at this end. The lens 34 projects the light incident from the third optical system 33 onto the object 6 to be projected.

[0055] The projection direction change mechanism 104 is a rotation mechanism that rotatably connects the second member 103 to the first member 102. This projection direction change mechanism 104 allows the second member 103 to rotate freely around a rotation axis (specifically, optical axis K) that extends in direction Y. Note that the projection direction change mechanism 104 is not limited to the arrangement position shown in Fig. 4 as long as it can rotate the optical system. Furthermore, the number of rotation mechanisms is not limited to one, and multiple mechanisms may be provided.

[0056] The shift mechanism 105 is a mechanism for moving the optical axis K of the projection optical system (in other words, the optical unit 106) in a direction perpendicular to the optical axis K (direction Y in FIG. 4 ). Specifically, the shift mechanism 105 is configured to be able to change the position of the first member 102 in direction Y relative to the main body 101. The shift mechanism 105 may be one that moves the first member 102 manually, or one that moves the first member 102 electrically.

[0057] Fig. 4 shows a state in which the first member 102 has been moved to the maximum extent in the direction Y1 by the shift mechanism 105. When the first member 102 is moved in the direction Y2 by the shift mechanism 105 from the state shown in Fig. 4, the relative position between the center of the image formed by the light modulation unit 22 (in other words, the center of the display surface) and the optical axis K changes, and the image G1 projected onto the projection object 6 can be shifted (translated) in the direction Y2.

[0058] The shift mechanism 105 may be a mechanism that moves the light modulation unit 22 in the direction Y, instead of moving the optical unit 106 in the direction Y. Even in this case, the image G1 projected onto the projection target 6 can be moved in the direction Y2.

[0059] <Appearance of image processing device 50> Fig. 5 is a diagram showing an example of the appearance of the image processing device 50. The image processing device 50 is a tablet terminal having a touch panel 51. The touch panel 51 is a display that can be touched. The image processing device 50 displays, on the touch panel 51, an installation assistance image for assisting in the installation of the projection device 10 in a space.

[0060] Specifically, the image processing device 50 displays a second image as an installation support image, which is an image obtained by capturing an image of the space in which the projection device 10 is installed and projection is to be performed, and which superimposes a first virtual projection surface, which is a virtual projection surface, and a first virtual projection device, which is a virtual projection device.

[0061] <Hardware configuration of image processing device 50> Fig. 6 is a diagram showing an example of the hardware configuration of the image processing device 50. The image processing device 50 shown in Fig. 5 includes, for example, a processor 61, a memory 62, a communication interface 63, a user interface 64, and a sensor 65, as shown in Fig. 6. The processor 61, the memory 62, the communication interface 63, the user interface 64, and the sensor 65 are connected by, for example, a bus 69.

[0062] The processor 61 is a circuit that performs signal processing, and is, for example, a CPU that controls the entire image processing device 50. The processor 61 may be realized by other digital circuits such as an FPGA or a DSP (Digital Signal Processor). The processor 61 may also be realized by combining multiple digital circuits.

[0063] The memory 62 includes, for example, a main memory and an auxiliary memory. The main memory is, for example, a RAM (Random Access Memory). The main memory is used as a work area for the processor 61.

[0064] The auxiliary memory is a non-volatile memory such as a magnetic disk, a flash memory, etc. The auxiliary memory stores various programs that operate the image processing device 50. The programs stored in the auxiliary memory are loaded into the main memory and executed by the processor 61.

[0065] The auxiliary memory may also include a portable memory that is removable from the image processing device 50. Portable memories include memory cards such as a USB (Universal Serial Bus) flash drive or an SD (Secure Digital) memory card, and an external hard disk drive.

[0066] The communication interface 63 is a communication interface for communicating with devices external to the image processing device 50. The communication interface 63 includes at least one of a wired communication interface for communicating via a wire and a wireless communication interface for communicating wirelessly. The communication interface 63 is controlled by the processor 61.

[0067] The user interface 64 includes, for example, an input device that accepts operation input from the user and an output device that outputs information to the user. The input device can be realized by, for example, keys (e.g., a keyboard) or a remote control. The output device can be realized by, for example, a display or a speaker. In the image processing device 50 shown in FIG. 5 , the input device and the output device are realized by the touch panel 51. The user interface 64 is controlled by the processor 61. The image processing device 50 accepts various specifications from the user using the user interface 64.

[0068] The sensor 65 includes an imaging device that has an imaging optical system and an imaging element and is capable of capturing images, and a space recognition sensor that is capable of three-dimensionally recognizing the space around the image processing device 50. The imaging device includes, for example, the imaging device provided on the back surface of the image processing device 50 shown in FIG.

[0069] An example of the spatial recognition sensor is a LIDAR (Light Detection and Ranging) that emits laser light, measures the time it takes for the emitted laser light to hit an object and bounce back, and measures the distance and direction to the object. However, the spatial recognition sensor is not limited to this, and various sensors such as a radar that emits radio waves and an ultrasonic sensor that emits ultrasonic waves can be used.

[0070] <Acquisition of Attitude of Imaging Device of Image Processing Device 50> Fig. 7 is a diagram showing an example of acquisition of the attitude of the imaging device of the image processing device 50. As shown in Fig. 7, for example, a user of the image processing device 50 brings the image processing device 50 into a physical space 70 (e.g., a room) that is a physical space in which the projection device 10 is to be installed. In the example of Fig. 7, the physical space 70 includes at least a floor 71 and a wall 72 as physical planes.

[0071] The image processing device 50 constantly acquires the attitude (position and orientation) of the imaging device of the image processing device 50 in a three-dimensional orthogonal coordinate system with a point in the physical space 70 (for example, the position where the imaging device of the image processing device 50 is started) as the origin, the horizontal direction as the X axis, the direction of gravity as the Y axis, and the remaining axis as the Z axis. Furthermore, the image processing device 50 displays to the user on the touch panel 51 an image based on imaging data obtained by imaging with the imaging device as a through image (live view).

[0072] <Capturing First Image and Acquiring First Position> Figure 8 is a diagram showing an example of capturing a first image and acquiring a first position. Here, it is assumed that the user intends to place the projection surface 11 of the projection device 10 at a first position 81 near the center of the wall 72 (central position). In this case, as shown in Figure 8, the user holds the image processing device 50 in a position and orientation where the first position 81 is displayed on the touch panel 51.

[0073] The user then designates the first position 81 in the physical space 70 by performing a designation operation (e.g., a tap operation) on the first position 81 (position 51a on the touch panel 51) of the wall 72 displayed on the touch panel 51. This allows the image processing device 50 to acquire first position data representing the first position 81 in the three-dimensional orthogonal coordinate system shown in FIG.

[0074] The first normal vector 82 is a normal vector of a first surface corresponding to a wall 72, which is an object existing at a first position 81 in the physical space 70, in the three-dimensional orthogonal coordinate system shown in Fig. 7. The image processing device 50 acquires first normal vector data representing the first normal vector 82 based on the result of recognizing the physical space 70 with the space recognition sensor.

[0075] <Physical Space Image Represented by First Image Data Obtained by Imaging Physical Space 70> Figure 9 is a diagram showing an example of a physical space image represented by first image data obtained by imaging the physical space 70. In the state shown in Figure 8, the user instructs the image processing device 50 to capture an image with a composition in which the first position 81 is captured on the touch panel 51. This allows the image processing device 50 to acquire the first image data obtained by capturing an image of the physical space 70 including the first position 81. The first image data is data representing a physical space image 90 in which the physical space 70 is captured. The physical space image 90 is an example of a first image of the present invention.

[0076] As a result, the image processing device 50 can acquire first image data obtained by capturing an image of the physical space 70, first position data representing a first position 81 in the physical space 70, and first normal vector data representing a first normal vector 82 of a first surface corresponding to an object present at the first position 81 in the physical space 70. For example, the image processing device 50 stores the first position data and first normal vector data indicating the first position 81 and the first normal vector 82 expressed in the three-dimensional orthogonal coordinate system described in FIG. 7 . In this case, in addition to this data, the image processing device 50 also stores data indicating the position of the image processing device 50 when the physical space 70 was captured and the first image data was obtained. Alternatively, when capturing an image of the physical space 70, the image processing device 50 stores the first position data and first normal vector data indicating the first position 81 and the first normal vector 82 expressed in the three-dimensional orthogonal coordinate system centered on the image processing device 50, based on the orientation of the image processing device 50, which is constantly being acquired.

[0077] <Processing of Image Processing Device 50> Fig. 10 is a flowchart showing an example of processing of the image processing device 50. Figs. 11 and 12 show examples of images displayed by the image processing device 50 in the processing shown in Fig. 10. At the start of the processing shown in Fig. 10, the image processing device 50 is assumed to have acquired first image data obtained by capturing an image of the physical space 70, first position data representing a first position 81 in the physical space 70, and first normal vector data representing a first normal vector 82 of a first surface corresponding to a wall 72 present at the first position 81 in the physical space 70, as described with reference to Figs. 7 to 9.

[0078] First, the image processing device 50 receives a specification of the size of the first virtual projection plane from the user (step S101). The size of the first virtual projection plane is specified using an actual distance in the physical space 70, such as the length of the diagonal of the rectangular first virtual projection plane = x [inch].

[0079] Next, the image processing device 50 superimposes the first virtual projection plane on the physical space image 90 represented by the first image data based on the size of the first virtual projection plane specified in step S101, the first position data, and the first normal vector data (step S102). For example, the image processing device 50 displays an image in which the first virtual projection plane 111 is superimposed on the physical space image 90, as shown in FIG.

[0080] Specifically, the image processing device 50 generates a first virtual projection plane 111 in the physical space image 90, the first virtual projection plane 111 being centered on the first position 81 represented by the first position data, perpendicular to the first normal vector 82 represented by the first normal vector data, and having its form adjusted so as to appear as a projection plane of a specified size, and then superimposes and displays the generated first virtual projection plane 111 on the physical space image 90. Note that although the first position 81 and the first normal vector 82 are illustrated in Fig. 11 , the first position 81 and the first normal vector 82 do not necessarily need to be displayed in reality.

[0081] Next, image processing device 50 accepts from the user a designation of a model of a first virtual projection device from among a plurality of options for the model of projection device 10 (step S103). Next, image processing device 50 calculates a first projection distance, which is the distance between first virtual projection device and first virtual projection surface 111, based on the size of first virtual projection surface 111 and a projection ratio that can be set for the model designated as the model of the first virtual projection device in step S103 (step S104).

[0082] Next, based on the model of the first virtual projection device designated in step S103 and the first projection distance calculated in step S104, the image processing device 50 superimposes and displays the first virtual projection device on the physical space image 90 (step S105). For example, as shown in FIG. 12 , the image processing device 50 superimposes and displays a first virtual projection device 112, which is a three-dimensional model of the model designated as the model of the first virtual projection device, on the physical space image 90.

[0083] Specifically, image processing device 50 generates first virtual projection device 112 whose form is adjusted so that it appears to be located at a position in physical space image 90 that is a first projection distance away from the center (first position 81) of first virtual projection plane 111 in the direction of first normal vector 82, and displays generated first virtual projection device 112 in a superimposed manner on physical space image 90. Note that although first position 81 and first normal vector 82 are illustrated in Fig. 12, first position 81 and first normal vector 82 do not necessarily need to be displayed in reality.

[0084] In this way, the image processing device 50 generates second image data representing a second image in which the first virtual projection surface 111 and the first virtual projection device 112 are superimposed on the first image (physical space image 90) represented by the first image data based on the first image data, the first virtual projection surface data, and the first virtual projection device data, and displays the second image based on the second image data.

[0085] This allows the user to obtain the first position data, first normal vector data, and physical space image 90 (first image) in the physical space 70, and thereby visually grasp the size and arrangement of the projection surface 11 based on the physical plane (wall 72) of the physical space 70, as well as the positional relationship between the projection surface 11 and the projection device 10, on the physical space image 90 representing the physical space 70, even in locations other than the physical space 70 where projection is being performed onto the projection device 10.

[0086] Furthermore, since it is sufficient to acquire first position data and first normal vector data as spatial data representing the physical space 70, and it is not necessary to acquire detailed three-dimensional data of the physical space 70, the amount of data to be stored can be reduced.

[0087] Specifically, in generating the second image data, the image processing device 50 generates first virtual projection plane data based on the first position data and the first normal vector data, and generates first virtual projection device data based on the generated first virtual projection plane data.

[0088] Furthermore, the image processing device 50 determines the normal vector of the first virtual projection plane 111 in accordance with the first normal vector 82 represented by the first normal vector data. For example, the image processing device 50 generates the first virtual projection plane 111 so that the direction of the normal vector of the first virtual projection plane 111 matches the direction of the first normal vector 82. Note that in the present application, "match" is not limited to a perfect match, but also includes a general match.

[0089] Furthermore, the image processing device 50 determines the projection direction and position of the first virtual projection device 112 represented by the first virtual projection device data based on the position and size of the first virtual projection surface 111 .

[0090] In addition, the image processing device 50 determines first position data and first normal vector data based on distance data regarding the distance (first projection distance) between the object (wall 72) and the imaging device (imaging device of the image processing device 50) obtained by the spatial recognition sensor.

[0091] 8 has been described as a configuration in which the user designates the first position 81 in the physical space 70 by performing a designating operation (e.g., a tap operation) on the first position 81 of the wall 72 displayed on the touch panel 51, but the configuration is not limited to this. For example, if the image processing device 50 can detect an edge of a physical plane on which the first virtual projection surface 111 is located in the physical space 70 when capturing an image of the physical space 70, the image processing device 50 may determine the first position 81 based on the position of the detected edge.

[0092] <Detection of Edges of the Physical Plane on Which the First Virtual Projection Plane 111 is Placed in the Physical Space 70> Fig. 13 is a diagram showing an example of detection of edges of the physical plane on which the first virtual projection plane 111 is placed in the physical space 70. In the example of Fig. 13, a wall 73 exists in the physical space 70. The wall 73 is a wall that is perpendicular to the floor 71 and the wall 72. For example, if the physical plane on which the first virtual projection plane 111 is placed in the physical space 70 is the wall 72, the image processing device 50 detects edges 72a to 72d shown in Fig. 13.

[0093] End 72a is the right end of wall 72 (the boundary with wall 73). End 72b is the upper end of wall 72. End 72c is the left end of wall 72. End 72d is the lower end of wall 72.

[0094] The ends 72a to 72d can be detected, for example, by image recognition processing based on imaging data obtained by imaging using an imaging device of the image processing device 50, or based on the recognition results obtained by a space recognition sensor of the image processing device 50.

[0095] <Process for Determining First Position 81> Fig. 14 is a flowchart showing an example of the process for determining the first position 81. Figs. 15 and 16 are diagrams showing an example of determining the first position 81 in the determination process of Fig. 14. The image processing device 50 executes, for example, the process shown in Fig. 14 while displaying to the user, on the touch panel 51, a physical space image 90 represented by first image data obtained by capturing an image of the physical space 70.

[0096] First, the image processing device 50 receives from the user a designation of a physical plane (wall 72) on which the first virtual projection plane 111 is to be placed in the physical space 70, and a size of the first virtual projection plane 111 (step S141). For example, the user designates the wall 72 in the physical space 70 by performing a designation operation (e.g., a tap operation) on the wall 72 displayed on the touch panel 51.

[0097] Next, the image processing device 50 detects the edges of the physical plane (wall 72) received from the user in step S141 (step S142). For example, the image processing device 50 detects edges 72a to 72d of the wall 72 as shown in FIG.

[0098] Next, the image processing device 50 receives from the user (step S143) designation of one or more ends to be used in determining the first position 81, from among the ends of the physical plane (wall 72) detected in step S142. For example, the image processing device 50 displays the detected ends 72a to 72d of the wall 72 as candidates on the touch panel 51, and receives designation of the end by a tap operation or the like from the user.

[0099] Next, the image processing device 50 determines, based on the size of the first virtual projection surface 111 specified in step S141, whether it is possible to determine the first position 81 so that the sides of the first virtual projection surface 111 contact all of the ends specified in step S143 (step S144).

[0100] For example, suppose that the user specifies ends 72a to 72c of wall 72 in step S143. Furthermore, suppose that the size (e.g., width) of first virtual projection plane 111 specified in step S141 is different from the size (e.g., width) of wall 72. In this case, as shown in FIG. 15 , it is not possible to determine first position 81 so that the sides of first virtual projection plane 111 of the size specified in step S141 contact all of ends 72a to 72c. Note that FIG. 15 illustrates an example in which first position 81 is determined so that the sides of first virtual projection plane 111 contact only end 72b of ends 72a to 72c.

[0101] If the first position 81 cannot be determined in step S144 (step S144: No), the image processing device 50 outputs a message to the user prompting the user to exclude some of the ends designated as ends to be used in determining the first position 81, and accepts the designation of the ends to be excluded (step S145).

[0102] Then, the image processing device 50 returns to step S144, and, excluding the end portion specified in step S145, determines again whether the first position 81 can be determined so that all of the end portions are in contact with the sides of the first virtual projection plane 111. For example, suppose that the end portion 72c of the end portions 72a to 72c of the wall 72 is specified to be excluded. In this case, for example, as shown in FIG. 16 , the first position 81 can be determined so that all of the specified end portions 72a and 72b are in contact with the sides of the first virtual projection plane 111.

[0103] If the first position 81 can be determined in step S144 (step S144: Yes), the image processing device 50 determines the first position 81 so that the sides of the first virtual projection plane 111 are in contact with all of the specified ends (step S146), and ends the series of processes. For example, the image processing device 50 determines the first position 81 shown in FIG. 16 . This makes it possible to easily determine the first position 81 that allows the first virtual projection plane 111 to be closer to the end of the wall 72.

[0104] <Another Example of Obtaining the Size of First Virtual Projection Plane 111> Although the configuration in which the user specifies the size of the first virtual projection plane 111 has been described, the present invention is not limited to this configuration. For example, if the image processing device 50 can detect the edge of the physical plane on which the first virtual projection plane 111 is placed in the physical space 70 when capturing an image of the physical space 70, the image processing device 50 may determine the size of the first virtual projection plane 111 based on the position of the detected edge.

[0105] <Process for Determining Size of First Virtual Projection Plane 111> Fig. 17 is a flowchart showing an example of a process for determining the size of the first virtual projection plane 111. Figs. 18 and 19 are diagrams showing an example of determining the size of the first virtual projection plane 111 and the first position 81 in the determination process of Fig. 17. The image processing device 50 executes, for example, the process shown in Fig. 17 while displaying to the user, on the touch panel 51, a physical space image 90 represented by first image data obtained by capturing an image of the physical space 70.

[0106] First, the image processing device 50 receives from the user a designation of a physical plane (wall 72) on which the first virtual projection plane 111 is to be placed in the physical space 70 (step S171). For example, the user designates the wall 72 in the physical space 70 by performing a designation operation (e.g., a tap operation) on the wall 72 displayed on the touch panel 51.

[0107] Next, the image processing device 50 detects the edges of the physical plane (wall 72) received from the user in step S171 (step S172). For example, the image processing device 50 detects edges 72a to 72d of the wall 72 as shown in FIG.

[0108] Next, the image processing device 50 receives from the user designation of one or more of the edges of the physical plane (wall 72) detected in step S172 to be used in determining the size of the first virtual projection plane 111 and the first position 81 (step S173). For example, the image processing device 50 displays the edges 72a to 72d of the detected wall 72 as selection candidates on the touch panel 51, and receives the designation of the edge by a tap operation or the like from the user.

[0109] Next, the image processing device 50 determines whether it is possible to determine the size and first position 81 of the first virtual projection surface 111 so that the side of the first virtual projection surface 111 is in an appropriate position relative to the end portion specified in step S173 (step S174).

[0110] For example, suppose that in step S173, the user specifies ends 72a to 72d of the wall 72. Then, for example, as shown in FIG. 18 , if the size (e.g., the length of the diagonal) and first position 81 of the first virtual projection plane 111 are determined so that the right and upper edges of the first virtual projection plane 111 contact the ends 72a and 72b, the left and lower edges of the first virtual projection plane 111 do not contact the ends 72c and 72d. In this case, the image processing device 50 determines that it is not possible to determine the size and first position 81 of the first virtual projection plane 111 so that the edges of the first virtual projection plane 111 are appropriately positioned relative to the ends 72a to 72c.

[0111] In step S174, if the size and first position 81 of the first virtual projection surface 111 cannot be determined (step S174: No), the image processing device 50 outputs a message to the user prompting the user to specify the positional relationship between some of the ends whose specification was accepted in step S173 and the sides of the first virtual projection surface 111, and accepts from the user the specification of the positional relationship between some of the ends and the sides of the first virtual projection surface 111 (step S175).

[0112] Then, the image processing device 50 returns to step S174 and, based on the positional relationship specified in step S175, again determines whether it is possible to determine the size and first position 81 of the first virtual projection plane 111 so that the side of the first virtual projection plane 111 is in an appropriate position relative to the end portion specified in step S173.

[0113] For example, suppose that in step S175, the positional relationship is specified such that the ends 72a and 72c are located inside the left and right sides of the first virtual projection plane. In this case, for example, as shown in Fig. 19, by making the left and right sides of the first virtual projection plane 111 outside the ends 72a and 72c, the size and first position 81 of the first virtual projection plane 111 can be determined so that the top and bottom sides of the first virtual projection plane 111 contact the ends 72b and 72d.

[0114] If the size and first position 81 of the first virtual projection plane 111 can be determined in step S174 (step S174: Yes), the image processing device 50 determines the size and first position 81 of the first virtual projection plane 111 so that the sides of the first virtual projection plane 111 are positioned appropriately with respect to the specified end (step S176), and ends the series of processes. For example, the image processing device 50 determines the size and first position 81 of the first virtual projection plane 111 shown in FIG. 19 . This makes it possible to easily determine the first position 81 that allows the first virtual projection plane 111 to be positioned close to the end of the wall 72.

[0115] In step S175, the image processing device 50 may present to the user how the positional relationship between the end of the wall 72 and the side of the first virtual projection surface 111 needs to be set in order to determine the size and first position 81 of the first virtual projection surface 111, and may prompt the user to specify the exclusion of the end of the wall 72 or to specify the positional relationship.

[0116] This makes it possible to easily determine the size of the first virtual projection plane 111 and the size of the first virtual projection plane 111 that allows the first position 81 to be closer to the end of the wall 72.

[0117] As shown in Figures 14 to 18, the image processing device 50 may identify the position of the end of the first surface (wall 72) in the physical space image 90 (first image) based on the first image data, and determine at least one of the position and size of the first virtual projection surface 111 based on the identified position of the end.

[0118] 20 and 21 are diagrams showing an example of an operation unit for moving the first virtual projection surface 111. In a state in which the first virtual projection surface 111 and the first virtual projection device 112 are superimposed and displayed on the physical space image 90 by the touch panel 51 as shown in Fig. 12 , for example, by the process shown in Fig. 10 , the image processing device 50 may further display a first virtual projection surface operation unit 201 shown in Fig. 20 . In the examples of Figs. 20 and 21 , the first virtual projection surface operation unit 201 is an image of up, down, left, and right cursor keys, and it is possible to instruct up, down, left, and right movement of the first virtual projection surface 111 by touch operation.

[0119] For example, as shown in Figure 21, when the right cursor key of the first virtual projection surface operation unit 201 is touched, the image processing device 50 moves the superimposition position of the first virtual projection surface 111 and the first virtual projection device 112 relative to the physical space image 90 to the right.

[0120] Specifically, image processing device 50 changes first position 81 in response to the operation of first virtual projection plane operation unit 201. Then, image processing device 50 executes processes similar to steps S102 and S105 shown in Fig. 10 , for example, to display first virtual projection plane 111 and first virtual projection device 112 corresponding to changed first position 81 superimposed on physical space image 90.

[0121] 22 and 23 are diagrams showing an example of an operation unit for changing the angle of first virtual projection plane 111. In a state in which first virtual projection plane 111 and first virtual projection device 112 are superimposed and displayed on touch panel 51 on physical space image 90 as shown in Fig. 12 by the process shown in Fig. 10 , for example, image processing device 50 may further display a first virtual projection plane operation unit 221 shown in Fig. 22 .

[0122] 22 and 23 , first virtual projection plane operation unit 221 is an image of four curved cursor keys, which can be touched to instruct a change in the angle of first virtual projection plane 111. The four curved cursor keys are used to instruct rotation in a first rotation direction about the horizontal axis, rotation in a second rotation direction opposite to the first rotation direction, rotation in a third rotation direction about the vertical axis, and rotation in a fourth rotation direction opposite to the third rotation direction, respectively.

[0123] For example, as shown in Figure 22, when a touch operation is performed on any of the curved cursor keys of the first virtual projection plane operation unit 221, the image processing device 50 changes the shape of the first virtual projection plane 111 and the first virtual projection device 112 superimposed on the physical space image 90 so that the angles of the first virtual projection plane 111 and the first virtual projection device 112 appear to have changed in the physical space image 90.

[0124] Specifically, the image processing device 50 changes the first normal vector 82 in response to the operation of the first virtual projection plane operation unit 221. Then, the image processing device 50 executes processing similar to steps S102 and S105 shown in Fig. 10 , for example, to display the first virtual projection plane 111 and the first virtual projection device 112 corresponding to the changed first position 81 superimposed on the physical space image 90.

[0125] In addition, the image processing device 50 may display both the first virtual projection plane operation unit 201 shown in Figures 20 and 21 and the first virtual projection plane operation unit 221 shown in Figures 22 and 23, and make it possible to change both the position and angle of the first virtual projection plane 111.

[0126] 20 to 23 , the image processing device 50 may change the first virtual projection plane 111 superimposed on the physical space image 90 based on first input data (e.g., data based on an operation on the first virtual projection plane operation unit 201 or the first virtual projection plane operation unit 221) related to a change in at least one of the first position 81 and the first normal vector 82. Furthermore, the image processing device 50 may change the first virtual projection device 112 superimposed on the physical space image 90 in accordance with the change in the first virtual projection plane 111 superimposed on the physical space image 90.

[0127] This allows the user to adjust the position and angle of the first virtual projection surface 111, and visually grasp the size and arrangement of the projection surface 11 in the desired projection state, as well as the positional relationship between the projection surface 11 and the projection device 10.

[0128] <Accepting an instruction to change the lens shift amount> The image processing device 50 may, for example, by the process shown in FIG. 10 , superimpose the first virtual projection surface 111 and the first virtual projection device 112 on the physical space image 90 using the touch panel 51 as shown in FIG. 12 , and accept an instruction to change the lens shift amount from the user within the range that can be set for the model that has been specified as the model of the first virtual projection device 112.

[0129] When the image processing device 50 receives an instruction to change the lens shift amount, it generates a first virtual projection device 112 in the physical space image 90, which is placed at a position a first projection distance away from the center (first position 81) of the first virtual projection surface 111 in the direction of the first normal vector 82, and whose shape has been adjusted so that it appears as if the changed lens shift amount has been set, and superimposes the generated first virtual projection device 112 on the physical space image 90.

[0130] In this way, the image processing device 50 may change the first virtual projection device 112 superimposed on the physical space image 90 based on the second input data (e.g., data based on an operation on the touch panel 51) related to a change in the shift amount of the projection lens of the first virtual projection device 112. This allows the user to visually grasp the size and arrangement of the projection surface 11 and the positional relationship between the projection surface 11 and the projection device 10 when the lens shift amount is set in the projection device 10.

[0131] <Capturing a first image and obtaining first and second positions> Fig. 24 is a diagram showing an example of capturing a first image and obtaining the first and second positions. Fig. 25 is a diagram showing an example of a second virtual projection plane based on the second position. Fig. 8 describes the case where the first position 81 at which the projection plane 11 is located is obtained, but the image processing device 50 may also obtain a second position at which the projection device 10 is located.

[0132] Here, it is assumed that the user intends to place the projection screen 11 at a first position 81 on the wall 72 and the projection device 10 at a second position 241 on the floor 71. In this case, as shown in Fig. 24 , the user holds the image processing device 50 at a position and orientation such that the first position 81 and the second position 241 are displayed on the touch panel 51.

[0133] The user then designates the first position 81 in the physical space 70 by performing a designation operation (e.g., a tap operation) on a first position 81 (position 51a on the touch panel 51) on the wall 72 displayed on the touch panel 51. The user also designates the second position 241 in the physical space 70 by performing a designation operation (e.g., a tap operation) on a second position 241 (position 51b on the touch panel 51) on the floor 71 displayed on the touch panel 51. This allows the image processing device 50 to acquire first position data representing the first position 81 and second position data representing the second position 241 in the three-dimensional orthogonal coordinate system shown in FIG. 7 .

[0134] The second normal vector 242 is a normal vector of a second plane corresponding to the floor 71, which is an object existing at a second position 241 in the physical space 70, in the three-dimensional orthogonal coordinate system shown in Fig. 7. The image processing device 50 acquires second normal vector data representing the second normal vector 242 based on the result of recognizing the physical space 70 with the space recognition sensor. This allows the image processing device 50 to acquire first normal vector data representing the first normal vector 82 and second normal vector data representing the second normal vector 242.

[0135] Then, the image processing device 50 executes the processing shown in Fig. 10 . However, in step S105 shown in Fig. 10 , the image processing device 50 constructs a virtual plane 251 corresponding to the floor 71 as shown in Fig. 25 based on the second position data and the second normal vector data. Then, the image processing device 50 generates a first virtual projection device 112 whose form has been adjusted so that the first virtual projection device 112 appears to be disposed in the physical space image 90 at a first projection distance (distance D1) from the first virtual projection plane 111 in the direction of the first normal vector 82 and with its bottom surface in contact with the virtual plane 251, and superimposes and displays the generated first virtual projection device 112 on the physical space image 90.

[0136] 26 is a diagram showing an example of coordinate axes for movement of the first virtual projection device 112. Using the first virtual projection device 112 as a reference, a three-dimensional orthogonal coordinate system is defined in which the axis perpendicular to the bottom surface (virtual plane 251) of the first virtual projection device 112 is the y-axis, the left-right direction of the first virtual projection device 112 is the x-axis, and the remaining axis (the front-rear direction of the first virtual projection device 112) is the z-axis.

[0137] 27 and 28 are diagrams showing an example of an operation unit for moving the first virtual projection device 112 in the x-axis direction or the z-axis direction. In a state in which the first virtual projection surface 111 and the first virtual projection device 112 are superimposed and displayed on the touch panel 51 on the physical space image 90 as shown in Fig. 12 by the process shown in Fig. 10, for example, the image processing device 50 may further display a first virtual projection device operation unit 271 shown in Fig. 27.

[0138] In the examples of Figures 27 and 28, the first virtual projection device operation unit 271 is an image of a cursor key for indicating forward, backward, left, and right, and by touching it, it is possible to instruct the first virtual projection device 112 to move forward, backward, left, and right (z-axis and x-axis).

[0139] For example, as shown in FIG. 28, when the right cursor key of the first virtual projection device operation unit 271 is touched, the image processing device 50 moves the superimposition position of the first virtual projection device 112 on the physical space image 90 to the right.

[0140] Specifically, the image processing device 50 generates a first virtual projection device 112 whose shape has been adjusted so that the first virtual projection device 112 appears to be positioned to the right of its original position in the physical space image 90, and superimposes the generated first virtual projection device 112 on the physical space image 90.

[0141] Furthermore, moving first virtual projection device 112 in the z-axis direction changes the first projection distance, which is the distance between first virtual projection device 112 and first virtual projection plane 111. In response to this, image processing device 50 recalculates the size of first virtual projection plane 111 based on the changed first projection distance, and superimposes first virtual projection plane 111 of the recalculated size on physical space image 90.

[0142] 29 and 30 are diagrams showing an example of an operation unit for moving the first virtual projection device 112 in the y-axis direction. In a state in which the first virtual projection surface 111 and the first virtual projection device 112 are superimposed and displayed on the touch panel 51 on the physical space image 90 as shown in Fig. 12 by the process shown in Fig. 10, for example, the image processing device 50 may further display a first virtual projection device operation unit 291 shown in Fig. 29.

[0143] In the examples of Figures 29 and 30, the first virtual projection device operation unit 291 is an image of a cursor key for indicating up and down, and a touch operation can be used to indicate movement of the first virtual projection device 112 up and down (in the y-axis direction).

[0144] For example, as shown in FIG. 30, when the cursor key in front of the first virtual projection device operation unit 291 is touched, the image processing device 50 moves the position of the first virtual projection device 112 upward relative to the physical space image 90 .

[0145] Specifically, a first virtual projection device 112 is generated with its shape adjusted so that it appears to be positioned in a position moved forward from its original position in the physical space image 90, and the generated first virtual projection device 112 is superimposed on the physical space image 90.

[0146] As shown in Figures 24 to 30, the image processing device 50 may generate first virtual projection device data based on first virtual projection plane data, second position data representing a second position 241 different from the first position 81 in the physical space 70, and second normal vector data representing a second normal vector 242 of the second plane corresponding to an object (floor 71) present at the second position 241 in the physical space 70.

[0147] Second Embodiment The second embodiment will be described in terms of differences from the first embodiment.

[0148] <Physical curved surface on which projection surface 11 is arranged in embodiment 2> Fig. 31 is a diagram showing an example of a physical curved surface on which projection surface 11 is arranged in embodiment 2. An overhead view 301 and a top view 302 shown in Fig. 31 show an overhead view and a top view of wall 310, which is a physical curved surface on which projection surface 11 of projection device 10 is arranged. Here, it is assumed that the user intends to arrange projection surface 11 of projection device 10 at first position 311 near the center of wall 310 (central position). In this case, as shown in Fig. 31 , the user holds image processing device 50 at a position and orientation where first position 311 is displayed on touch panel 51.

[0149] The user then designates the first position 311 in the physical space 70 by performing a designation operation (e.g., a tap operation) on the first position 311 (position 51c on the touch panel 51) of the wall 310 displayed on the touch panel 51. This allows the image processing device 50 to acquire first position data representing the first position 311.

[0150] The first normal vector 312 is a normal vector corresponding to the first position 311 of the first surface corresponding to the wall 310, which is an object present at the first position 311 in the physical space 70. The image processing device 50 acquires first normal vector data representing the first normal vector 312 based on the result of recognizing the physical space 70 with the space recognition sensor.

[0151] <Specifying Second Position Group> Fig. 32 is a diagram showing an example of specifying the second position group. Furthermore, the image processing device 50 receives from the user an instruction for the second position group sufficient to roughly reproduce the shape of the wall 310. The instruction for the second position group is received in the same manner as the instruction for the first position 311 described in Fig. 31 .

[0152] 32, it is assumed that second positions 321a to 321d are designated as the second position group. Second normal vectors 322a to 322d are normal vectors corresponding to the second positions 321a to 321d, respectively, of the first surface corresponding to the wall 310. The image processing device 50 acquires a second normal vector data group representing the second normal vectors 322a to 322d based on the result of recognizing the physical space 70 with the space recognition sensor.

[0153] <First Virtual Curved Surface Virtually Representing Wall 310> Figure 33 is a diagram showing an example of a first virtual curved surface virtually representing wall 310. The image processing device 50 constructs, for example, a first virtual curved surface 330 based on a first position 311, a first normal vector 312, second positions 321a to 321d (second position group), and second normal vectors 322a to 322d (second normal vector group). An overhead view 341 and a top view 342 shown in Figure 33 show an overhead view and a top view of first virtual curved surface 330.

[0154] The first virtual curved surface 330 is configured as a pseudo-curved surface by arranging rectangular planes 331 to 335 adjacent to each other at different angles. The rectangular plane 331 is a plane based on the first position 311 and the first normal vector 312. The rectangular planes 332 to 335 are planes based on the second positions 321a to 321d and the second normal vectors 322a to 322d, respectively. Each of the rectangular planes 331 to 335 is configured by combining two triangular polygons, for example.

[0155] Then, the image processing device 50 executes the processing shown in Fig. 10. However, in step S102 shown in Fig. 10, the image processing device 50 generates a first virtual projection plane 111 in the physical space image 90, the first virtual projection plane 111 being centered at the first position 311, being perpendicular to the first normal vector 312, and having its form adjusted so that it appears to be projected onto the first virtual curved surface 330 at a specified size, and then superimposes and displays the generated first virtual projection plane 111 on the physical space image 90.

[0156] As a result, by acquiring the first position data, first normal vector data, second position group data, second normal vector group data, and physical space image 90 (first image) in physical space 70, the user can visually grasp the size and arrangement of the projection surface 11 based on the physical curved surface (wall 310) of physical space 70, and the positional relationship between the projection surface 11 and the projection device 10, on the physical space image 90 representing physical space 70, even in places other than physical space 70.

[0157] Furthermore, when the image processing device 50 is displaying the first virtual projection surface 111 and the first virtual projection device 112 superimposed on the physical space image 90 using the touch panel 51 based on the first virtual curved surface 330, it may accept instructions from the user to change the position or angle of the first virtual projection surface 111, as in the examples of Figures 20 and 21, and update the first virtual projection surface 111 and the first virtual projection device 112 superimposed on the physical space image 90 based on the accepted instructions.

[0158] In this way, the image processing device 50 of embodiment 2 generates first virtual projection plane data and first virtual projection device data based on first position data indicating the first position 311, first normal vector data indicating the first normal vector 312, second position group data representing the second position group (second positions 321a to 321d) on the first surface corresponding to the wall 310, and second normal vector group data representing the second normal vector group (second normal vectors 322a to 322d) corresponding to the second position group on the first surface corresponding to the wall 310.

[0159] Specifically, the image processing device 50 generates virtual surface data representing the first virtual surface 330 based on the first position data, the first normal vector data, the second position group data, and the second normal vector group data, and generates first virtual projection surface data based on the first virtual projection device data and the virtual surface data.

[0160] Furthermore, in response to an instruction from the user, the image processing device 50 may display to the user the position and angle of the first virtual projection plane 111, the position and angle of the first virtual projection device 112, the first projection distance, the projection parameters of the first virtual projection device 112, etc. At this time, the image processing device 50 may determine the origin and the directions of each axis of the above-mentioned three-dimensional orthogonal coordinate system based on a specification from the user. This allows the user to grasp, as numerical values, the positional relationship between the projection plane and the projection device that he or she visually confirmed, and the projection parameters at that time.

[0161] (Modifications) Modifications of the above embodiments will be described.

[0162] <Modification 1> Although the image processing device 50 has been described as a tablet terminal having a touch panel 51, the image processing device 50 is not limited to this configuration. For example, the image processing device 50 may be an information terminal such as a smartphone or a personal computer.

[0163] <Modification 2> Although the configuration in which the image processing device 50 displays the second image on the touch panel 51 has been described, the image processing device 50 may transmit the generated second image to another device, thereby controlling the other device to display the second image. In this case, the image processing device 50 may be a device that does not have a display device.

[0164] <Modification 3> Although the case where the physical space image 90 is an image obtained by imaging using an imaging device of the image processing device 50 has been described, the physical space image 90 may be an image obtained by imaging using a device other than the image processing device 50 and received from that device by the image processing device 50. In this case, the image processing device 50 may be a device that does not include an imaging device.

[0165] (Image Processing Program) The image processing method described in the above-described embodiment can be realized by executing a prepared image processing program on a computer. The image processing program is recorded on a computer-readable storage medium and executed by being read from the storage medium. The image processing program may be provided in a form stored on a non-transitory storage medium such as a flash memory, or may be provided via a network such as the Internet. The computer that executes the image processing program may be included in an image processing device, or may be included in an electronic device such as a smartphone, tablet terminal, or personal computer that can communicate with the image processing device, or may be included in a server device that can communicate with these image processing devices and electronic devices.

[0166] The above-described embodiments and modifications can be implemented in combination.

[0167] This specification describes at least the following:

[0168] (1) An image processing device including a processor, wherein the processor acquires first image data obtained by capturing an image of a space with an imaging device, generates first virtual projection plane data representing a first virtual projection plane and first virtual projection device data representing a first virtual projection device based on first position data representing a first position in the space and first normal vector data representing a first normal vector of a first surface corresponding to an object present at the first position in the space, generates second image data representing a second image in which the first virtual projection plane and the first virtual projection device are displayed on a first image represented by the first image data, based on the first image data, the first virtual projection plane data, and the first virtual projection device data, and outputs the second image data to an output destination.

[0169] (2) The image processing device according to (1), wherein the processor generates the first virtual projection device data based on the first virtual projection surface data.

[0170] (3) The image processing device according to (2), in which the processor determines a normal vector of the first virtual projection plane in accordance with the first normal vector.

[0171] (4) The image processing device according to (3), wherein the first virtual projection plane is a virtual projection plane having a normal vector that coincides with the first normal vector.

[0172] (5) The image processing device according to any one of (2) to (4), wherein the processor determines a projection direction and a position of the first virtual projection device based on a position and a size of the first virtual projection surface.

[0173] (6) The image processing device according to any one of (1) to (5), wherein the processor determines the first normal vector data of the first position based on distance data relating to a distance between the object and the imaging device.

[0174] (7) An image processing device according to any one of (1) to (6), wherein the processor identifies the position of an edge of the first surface in the first image based on the first image data, and determines at least one of the position and size of the first virtual projection surface based on the position of the edge.

[0175] (8) An image processing device according to any one of (1) to (7), wherein the processor changes the first virtual projection plane displayed on the second image based on first input data relating to a change in at least one of the first position and the first normal vector.

[0176] (9) An image processing device according to any one of (1) to (8), wherein the processor changes the first virtual projection device displayed in the second image based on second input data relating to a change in the shift amount of the projection lens of the first virtual projection device.

[0177] (10) An image processing device according to any one of (1) to (9), wherein the processor generates the first virtual projection device data based on the first virtual projection plane data, second position data representing a second position different from the first position in the space, and second normal vector data representing a second normal vector of a second plane corresponding to an object present at the second position in the space.

[0178] (11) The image processing device according to (1), wherein the processor generates the first virtual projection plane data and the first virtual projection device data based on the first position data, the first normal vector data, second position group data representing a second position group on the first plane, and second normal vector group data representing a second normal vector group corresponding to the second position group on the first plane.

[0179] (12) The image processing device according to (11), wherein the processor generates virtual curved surface data representing a virtual curved surface based on the first position data, the first normal vector data, the second position group data, and the second normal vector group data, and generates the first virtual projection surface data based on the first virtual projection device data and the virtual curved surface data.

[0180] (13) An image processing method, in which a processor of an image processing device acquires first image data obtained by capturing an image of a space with an imaging device, generates first virtual projection plane data representing a first virtual projection plane and first virtual projection device data representing a first virtual projection device based on first position data representing a first position in the space and first normal vector data representing a first normal vector of a first surface corresponding to an object present at the first position in the space, generates second image data representing a second image in which the first virtual projection plane and the first virtual projection device are displayed on a first image represented by the first image data, based on the first image data, the first virtual projection plane data, and the first virtual projection device data, and outputs the second image data to an output destination.

[0181] (14) An image processing program for causing a processor of an image processing device to execute the following process: acquire first image data obtained by capturing an image of a space with an imaging device; generate first virtual projection plane data representing a first virtual projection plane and first virtual projection device data representing a first virtual projection device based on first position data representing a first position in the space and first normal vector data representing a first normal vector of a first surface corresponding to an object existing at the first position in the space; generate second image data representing a second image in which the first virtual projection plane and the first virtual projection device are displayed on a first image represented by the first image data, based on the first image data, the first virtual projection plane data, and the first virtual projection device data; and output the second image data to an output destination.

[0182] Although various embodiments have been described above, it goes without saying that the present invention is not limited to such examples. It is clear that a person skilled in the art can conceive of various modifications or alterations within the scope of the claims, and it is understood that these also naturally fall within the technical scope of the present invention. Furthermore, the components of the above embodiments may be combined in any manner as long as they do not deviate from the spirit of the invention.

[0183] This application is based on a Japanese patent application (Patent Application No. 2022-057497) filed on March 30, 2022, the contents of which are incorporated herein by reference.

[0184] 1 Projection unit 2 Operation reception unit 2A, 3A Hollow portion 2a, 2b, 3a, 3c, 15a Opening 4 Control device 4a, 62 Memory 6 Projected object 10 Projection device 11 Projection surface 12 Light modulation unit 15 Housing 21 Light source 22 Light modulation unit 23 Projection optical system 24 Control circuit 31 Second optical system 32, 122 Reflecting member 33 Third optical system 34 Lens 50 Image processing device 51 Touch panel 51a, 51b, 51c Position 61 Processor 63 Communication interface 64 User interface 65 Sensor 69 Bus 70 Physical space 71 Floor 72, 73, 310 Wall 72a to 72d End 81, 311 First position 82, 312 First normal vector 90 Physical space image REFERENCE SIGNS LIST 101 Main body 102 First member 103 Second member 104 Projection direction change mechanism 105 Shift mechanism 106 Optical unit 111 First virtual projection plane 112 First virtual projection device 121 First optical system 201, 221 First virtual projection plane operation unit 241, 321a to 321d Second position 242, 322a to 322d Second normal vector 251 Virtual plane 271, 291 First virtual projection device operation unit 330 First virtual curved surface 331 to 335 Rectangular plane G1 Image D1 Distance

Claims

1. An image processing device including a processor, The processor, Acquiring first image data obtained by imaging the space with an imaging device; generating first virtual projection plane data representing a first virtual projection plane and first virtual projection device data representing a first virtual projection device based on first position data representing a first position in the space and first normal vector data representing a first normal vector of a first surface corresponding to an object existing at the first position in the space; generating second image data representing a second image in which the first virtual projection surface and the first virtual projection device are displayed on a first image represented by the first image data, based on the first image data, the first virtual projection surface data, and the first virtual projection device data; outputting the second image data to an output destination; Image processing device.

2. 2. The image processing device according to claim 1, The processor, generating the first virtual projection device data based on the first virtual projection surface data; Image processing device.

3. 3. The image processing device according to claim 2, The processor, determining a normal vector of the first virtual projection plane according to the first normal vector; Image processing device.

4. 4. The image processing device according to claim 3, The first virtual projection plane is a virtual projection plane having a normal vector that coincides with the first normal vector. Image processing device.

5. 3. The image processing device according to claim 2, The processor, determining a projection direction and a position of the first virtual projection device based on a position and a size of the first virtual projection surface; Image processing device.

6. 2. The image processing device according to claim 1, The processor, determining the first normal vector data for the first position based on distance data relating to a distance between the object and the imaging device; Image processing device.

7. 2. The image processing device according to claim 1, The processor identifies a position of an edge of the first surface in the first image based on the first image data, and determines at least one of a position and a size of the first virtual projection surface based on the position of the edge. Image processing device.

8. 2. The image processing device according to claim 1, The processor, changing the first virtual projection plane displayed on the second image based on first input data relating to a change in at least one of the first position and the first normal vector; Image processing device.

9. 2. The image processing device according to claim 1, The processor, changing the first virtual projection device displayed in the second image based on second input data related to a change in a shift amount of a projection lens of the first virtual projection device; Image processing device.

10. 10. The image processing device according to claim 1 , the processor generates the first virtual projection device data based on the first virtual projection plane data, second position data representing a second position different from the first position in the space, and second normal vector data representing a second normal vector of a second surface corresponding to an object present at the second position in the space. Image processing device.

11. 2. The image processing device according to claim 1, the processor generates the first virtual projection plane data and the first virtual projection device data based on the first position data, the first normal vector data, second position group data representing a second position group on the first plane, and second normal vector group data representing a second normal vector group corresponding to the second position group on the first plane. Image processing device.

12. The image processing device according to claim 11, The processor, generating virtual curved surface data representing a virtual curved surface based on the first position data, the first normal vector data, the second position group data, and the second normal vector group data; generating the first virtual projection surface data based on the first virtual projection device data and the virtual curved surface data; Image processing device.

13. A processor of the image processing device Acquiring first image data obtained by imaging the space with an imaging device; generating first virtual projection plane data representing a first virtual projection plane and first virtual projection device data representing a first virtual projection device based on first position data representing a first position in the space and first normal vector data representing a first normal vector of a first surface corresponding to an object existing at the first position in the space; generating second image data representing a second image in which the first virtual projection surface and the first virtual projection device are displayed on a first image represented by the first image data, based on the first image data, the first virtual projection surface data, and the first virtual projection device data; outputting the second image data to an output destination; Image processing methods.

14. The processor of the image processing device Acquiring first image data obtained by imaging the space with an imaging device; generating first virtual projection plane data representing a first virtual projection plane and first virtual projection device data representing a first virtual projection device based on first position data representing a first position in the space and first normal vector data representing a first normal vector of a first surface corresponding to an object existing at the first position in the space; generating second image data representing a second image in which the first virtual projection surface and the first virtual projection device are displayed on a first image represented by the first image data, based on the first image data, the first virtual projection surface data, and the first virtual projection device data; outputting the second image data to an output destination; An image processing program to carry out the processing.