Image processing device, image processing method and image processing program

Abstract

This invention can provide an image processing device, an image processing method and an image processing program that can improve the user's convenience related to the placement of a projection screen and of a projection device.　This image processing device (50) acquires first image data obtained by using an imaging device to image a physical space (70). The image processing device (50) also determines a first position corresponding to the position of a virtual projection screen (80) and a second position serving as the reference of the orientation of the virtual projection screen (80) in the physical space (70). The image processing device (50) also determines the orientation of the virtual projection screen (80) on the basis of the positional relationship between the first position and the second position and generates virtual projection screen data representative of the virtual projection screen (80). The image processing device (50) also generates, on the basis of the first image data and the virtual projection screen data, second image data representative of a second image in which the virtual projection screen is displayed on a first image represented by the first image data, and the image processing device (50) outputs the second image data to an output destination.

Description

Image processing device, image processing method, and image processing program

[0001] The present invention relates to an image processing device, an image processing method, and an image processing program.

[0002] Japanese Patent Laid-Open No. 2003-144992 describes an information processing device that outputs projector layout information relating to the layout of projectors based on projection conditions relating to projection by the projectors, in order to reduce the burden of designing the layout of projectors.

[0003] Patent document 2 describes an information processing device that acquires setting information regarding the projection of images by an image projection device and, based on the acquired setting information, generates a simulation image that includes multiple image projection devices and the display areas of each of multiple images projected by the multiple image projection devices.

[0004] Patent document 3 describes a projection toy in which the main body is provided with a projection unit capable of projecting images onto an object, and a first, second, and third placing unit are provided on the main body for placing the main body on a placing surface, with the first, second, and third placing units facing in different directions to facilitate projection in three directions.

[0005] International Publication No. 2019 / 012774 Pamphlet International Publication No. 2017 / 179272 Pamphlet Japanese Patent Application Laid-Open No. 2018-121964

[0006] One embodiment of the technique of the present disclosure provides an image processing device, an image processing method, and an image processing program that can improve user convenience regarding the placement of a projection surface or a projection device.

[0007] An image processing device of one aspect of the present invention is an image processing device having a processor, wherein the processor acquires first image data obtained by capturing an image of a space with an imaging device, determines a first position in the space corresponding to the position of a virtual projection surface and a second position serving as a reference for the orientation of the virtual projection surface, determines the orientation of the virtual projection surface based on the positional relationship between the first position and the second position, and generates virtual projection surface data representing the virtual projection surface, generates second image data based on the first image data and the virtual projection surface data, representing a second image in which the virtual projection surface is displayed on a first image represented by the first image data, and outputs the second image data to an output destination.

[0008] In one aspect of the image processing method of the present invention, a processor provided in an image processing device acquires first image data obtained by capturing an image of a space with an imaging device, determines a first position in the space corresponding to the position of a virtual projection surface and a second position that serves as a reference for the orientation of the virtual projection surface, determines the orientation of the virtual projection surface based on the positional relationship between the first position and the second position, and generates virtual projection surface data representing the virtual projection surface, generates second image data based on the first image data and the virtual projection surface data, representing a second image in which the virtual projection surface is displayed on a first image represented by the first image data, and outputs the second image data to an output destination.

[0009] An image processing program according to one aspect of the present invention causes a processor included in an image processing device to execute the following processing: acquire first image data obtained by capturing an image of a space with an imaging device; determine a first position in the space corresponding to the position of a virtual projection surface and a second position serving as a reference for the orientation of the virtual projection surface; determine the orientation of the virtual projection surface based on the positional relationship between the first position and the second position to generate virtual projection surface data representing the virtual projection surface; generate second image data based on the first image data and the virtual projection surface data to represent a second image in which the virtual projection surface is displayed on a first image represented by the first image data; and output the second image data to an output destination.

[0010] According to the present invention, it is possible to provide an image processing device, an image processing method, and an image processing program that can improve user convenience regarding the placement of a projection surface and a projection device.

[0011] 1 is a schematic diagram showing an example of a projection device 10 that is a target for placement support by the image processing device of the embodiment. FIG. 1 is a schematic diagram showing an example of the internal configuration of the projection unit 1 shown in FIG. 1. FIG. 2 is a schematic diagram showing the external configuration of the projection device 10. FIG. 3 is a schematic cross-sectional view of the optical unit 106 of the projection device 10 shown in FIG. 3. FIG. 4 is a diagram showing an example of the external appearance of an image processing device 50. FIG. 5 is a diagram showing an example of the hardware configuration of the image processing device 50. FIG. 6 is a diagram showing an example of a physical space in which the image processing device 50 is used. FIG. 7 is a diagram showing an example of a state in which the orientation of a virtual projection surface is undetermined. FIG. 8 is a diagram showing an example of determining the orientation of a virtual projection device by determining the orientation of a virtual projection surface 80. FIG. 9 is a diagram showing an example of a method for specifying a position in a physical space 70. FIG. 10 is a diagram showing an example of a virtual projection device installation position 91, a virtual projection surface installation position 81, and a reference point. FIG. 11 is a diagram showing a first example of the positional relationship between the virtual projection surface installation position 81 and the reference point 111. FIG. 12 is a diagram showing a second example of the positional relationship between the virtual projection surface installation position 81 and the reference point 111. FIG. 13 is a diagram showing an example of determining the orientation of the virtual projection surface 80 when a first angle Θ is less than a threshold. FIG. 14 is a diagram showing an example of a projection distance D when the first angle Θ is less than a threshold. 1 is a diagram showing an example of determining the orientation of the virtual projection plane 80 when the first angle Θ is equal to or greater than a threshold. FIG. 2 is a diagram showing an example of the projection distance D when the first angle Θ is equal to or greater than a threshold. FIG. 3 is a flowchart showing an example of processing by the image processing device 50. FIG. 4 is a diagram showing an example of recalculating the projection distance D when the orientation of the virtual projection plane 80 is changed by the user. FIG. 5 is a diagram showing an example of determining the position of the virtual projection plane 80 based on detection of a plane that serves as a reference for the position of the virtual projection plane 80. FIG. 6 is a diagram (part 1) showing an example of determining the orientation of the provisional virtual projection plane 80 based on the camera position when the first angle Θ is less than a threshold. FIG. 7 is a diagram (part 2) showing an example of determining the orientation of the provisional virtual projection plane 80 based on the camera position when the first angle Θ is less than a threshold.

[0012] Hereinafter, examples of embodiments of the present invention will be described with reference to the drawings.

[0013] (Embodiment) <Projection Device 10 to be Assisted in Placement by Image Processing Device of Embodiment> FIG. 1 is a schematic diagram showing an example of a projection device 10 to be assisted in placement by an image processing device of an embodiment.

[0014] The image processing device according to the embodiment can be used, for example, to assist in the placement of a projection device 10. The projection device 10 includes a projection unit 1, a control device 4, and an operation reception unit 2. The projection unit 1 is configured, for example, by a liquid crystal projector or a projector using LCOS (Liquid Crystal On Silicon). In the following description, the projection unit 1 is assumed to be a liquid crystal projector.

[0015] The control device 4 is a control device that controls projection by the projection device 10. The control device 4 is a device that includes a control unit configured with various processors, a communication interface (not shown) for communicating with each unit, and a memory 4a such as a hard disk, an SSD (Solid State Drive), or a ROM (Read Only Memory), and controls the projection unit 1 in an integrated manner.

[0016] The various processors in the control unit of the control device 4 include a CPU (Central Processing Unit), which is a general-purpose processor that executes programs to perform various processes, a programmable logic device (PLD), which is a processor whose circuit configuration can be changed after manufacture, such as an FPGA (Field Programmable Gate Array), or a dedicated electrical circuit, such as an ASIC (Application Specific Integrated Circuit), which is a processor having a circuit configuration designed specifically to perform specific processing.

[0017] More specifically, the structure of these various processors is an electric circuit that combines circuit elements such as semiconductor elements. The control unit of the control device 4 may be configured with one of the various processors, or may be configured with a combination of two or more processors of the same or different types (for example, a combination of multiple FPGAs or a combination of a CPU and an FPGA).

[0018] The operation reception unit 2 detects instructions from the user by receiving various operations from the user. The operation reception unit 2 may be a button, key, joystick, or the like provided on the control device 4, or may be a receiving unit or the like that receives signals from a remote controller that remotely controls the control device 4.

[0019] The projection target 6 is an object such as a screen or a wall that has a projection surface on which a projected image is displayed by the projection unit 1. In the example shown in Fig. 1, the projection surface of the projection target 6 is a rectangular plane. The top, bottom, left, and right of the projection target 6 in Fig. 1 are assumed to be the top, bottom, left, and right of the actual projection target 6.

[0020] The projection surface 11, shown by a dashed line, is an area of ​​the object 6 onto which projection light is irradiated by the projection unit 1. In the example shown in Fig. 1 , the projection surface 11 is rectangular. The projection surface 11 is a part or the entire projectable range onto which projection can be performed by the projection unit 1.

[0021] The projection unit 1, the control device 4, and the operation reception unit 2 may be realized by, for example, a single device (see, for example, FIGS. 3 and 4), or the projection unit 1, the control device 4, and the operation reception unit 2 may be separate devices that communicate with each other and cooperate with each other.

[0022] <Internal Configuration of Projection Unit 1 Shown in FIG. 1> FIG. 2 is a schematic diagram showing an example of the internal configuration of the projection unit 1 shown in FIG.

[0023] As shown in FIG. 2, the projection unit 1 includes a light source 21, a light modulation unit 22, a projection optical system 23, and a control circuit 24.

[0024] The light source 21 includes a light emitting element such as a laser or an LED (Light Emitting Diode), and emits, for example, white light.

[0025] The light modulation unit 22 is composed of three liquid crystal panels that modulate the color light emitted from the light source 21 and separated into three colors, red, blue, and green, by a color separation mechanism (not shown), based on image information, and emit each color image. These three liquid crystal panels may be equipped with red, blue, and green filters, respectively, and the white light emitted from the light source 21 may be modulated by each liquid crystal panel to emit each color image.

[0026] The projection optical system 23 receives light from the light source 21 and the light modulation unit 22, and is configured by, for example, a relay optical system including at least one lens. The light that passes through the projection optical system 23 is projected onto the object 6 to be projected.

[0027] The area of ​​the object 6 that is irradiated with light that passes through the entire range of the light modulation unit 22 is the projectable range in which projection by the projection unit 1 is possible. Within this projectable range, the area that is irradiated with light that actually passes through the light modulation unit 22 is the projection surface 11. For example, by controlling the size, position, and shape of the area of ​​the light modulation unit 22 through which light passes, the size, position, and shape of the projection surface 11 can be changed within the projectable range.

[0028] The control circuit 24 controls the light source 21, the light modulation unit 22, and the projection optical system 23 based on the display data input from the control device 4, thereby projecting an image based on this display data onto the projection target 6. The display data input to the control circuit 24 is made up of three pieces of data: red display data, blue display data, and green display data.

[0029] The control circuit 24 also enlarges or reduces the projection surface 11 (see FIG. 1 ) of the projection unit 1 by changing the projection optical system 23 based on a command input from the control device 4. The control device 4 may also move the projection surface 11 of the projection unit 1 by changing the projection optical system 23 based on an operation from the user received by the operation receiving unit 2.

[0030] The projection device 10 also includes a shift mechanism that mechanically or optically moves the projection surface 11 while maintaining the image circle of the projection optical system 23. The image circle of the projection optical system 23 is the area through which projection light incident on the projection optical system 23 passes through the projection optical system 23 appropriately in terms of light intensity loss, color separation, peripheral curvature, etc.

[0031] The shift mechanism is realized by at least one of an optical system shift mechanism that performs an optical system shift and an electronic shift mechanism that performs an electronic shift.

[0032] The optical system shift mechanism is, for example, a mechanism that moves the projection optical system 23 in a direction perpendicular to the optical axis (see, for example, FIGS. 3 and 4), or a mechanism that moves the light modulation unit 22 in a direction perpendicular to the optical axis instead of moving the projection optical system 23. The optical system shift mechanism may also be a mechanism that moves the projection optical system 23 and the light modulation unit 22 in combination.

[0033] The electronic shift mechanism is a mechanism that shifts the pseudo projection surface 11 by changing the range through which light is transmitted in the light modulation section 22 .

[0034] The projection device 10 may also include a projection direction change mechanism that moves the projection surface 11 together with the image circle of the projection optical system 23. The projection direction change mechanism is a mechanism that changes the projection direction of the projection unit 1 by changing the orientation of the projection unit 1 through mechanical rotation (see, for example, FIGS. 3 and 4).

[0035] <Mechanical Configuration of Projection Device 10> Fig. 3 is a schematic diagram showing the external configuration of the projection device 10. Fig. 4 is a schematic cross-sectional view of the optical unit 106 of the projection device 10 shown in Fig. 3. Fig. 4 shows a cross section taken along a plane along the optical path of light emitted from the main body 101 shown in Fig. 3.

[0036] 3, the projection device 10 includes a main body 101 and an optical unit 106 that protrudes from the main body 101. In the configuration shown in Fig. 3, the operation reception unit 2, the control device 4, and the light source 21, light modulation unit 22, and control circuit 24 in the projection unit 1 are provided in the main body 101. The projection optical system 23 in the projection unit 1 is provided in the optical unit 106.

[0037] The optical unit 106 includes a first member 102 supported by the main body 101 and a second member 103 supported by the first member 102 .

[0038] The first member 102 and the second member 103 may be an integrated member. The optical unit 106 may be configured to be detachable from the main body 101 (in other words, replaceable).

[0039] The main body 101 has a housing 15 (see FIG. 4) in which an opening 15a (see FIG. 4) for passing light is formed at a portion connected to the optical unit 106.

[0040] As shown in Figure 3, inside the housing 15 of the main body 101, there is provided a light source 21 and a light modulation unit 12 including a light modulation section 22 (see Figure 2) that spatially modulates the light emitted from the light source 21 based on input image data to generate an image.

[0041] The light emitted from the light source 21 is incident on the light modulation section 22 of the light modulation unit 12, and is spatially modulated by the light modulation section 22 before being emitted.

[0042] As shown in Figure 4, the image formed by the light spatially modulated by the light modulation unit 12 passes through the opening 15a of the housing 15 and enters the optical unit 106, and is projected onto the projection target 6, which is the projection object, so that the image G1 becomes visible to the observer.

[0043] As shown in Figure 4, the optical unit 106 includes a first member 102 having a hollow portion 2A connected to the inside of the main body portion 101, a second member 103 having a hollow portion 3A connected to the hollow portion 2A, a first optical system 121 and a reflecting member 122 arranged in the hollow portion 2A, a second optical system 31, a reflecting member 32, a third optical system 33, and a lens 34 arranged in the hollow portion 3A, a shift mechanism 105, and a projection direction change mechanism 104.

[0044] The first member 102 is a member having, for example, a rectangular cross-sectional outer shape, and the openings 2a and 2b are formed on planes perpendicular to each other. The first member 102 is supported by the main body 101 with the opening 2a positioned opposite the opening 15a of the main body 101. Light emitted from the light modulation section 22 of the light modulation unit 12 in the main body 101 passes through the openings 15a and 2a and enters the hollow section 2A of the first member 102.

[0045] The direction of light incident from the main body 101 into the hollow portion 2A is referred to as direction X1, the opposite direction to direction X1 is referred to as direction X2, and directions X1 and X2 are collectively referred to as direction X. In addition, in Figure 4, the direction from the front of the page toward the back and the opposite direction are referred to as direction Z. Of direction Z, the direction from the front of the page toward the back is referred to as direction Z1, and the direction from the back of the page toward the front is referred to as direction Z2.

[0046] Furthermore, a direction perpendicular to direction X and direction Z is referred to as direction Y, and within direction Y, the upward direction in Fig. 4 is referred to as direction Y1, and the downward direction in Fig. 4 is referred to as direction Y2. In the example of Fig. 4, the projection device 10 is disposed so that direction Y2 is the vertical direction.

[0047] The projection optical system 23 shown in Fig. 2 is composed of a first optical system 121, a reflecting member 122, a second optical system 31, a reflecting member 32, a third optical system 33, and a lens 34. Fig. 4 shows the optical axis K of the projection optical system 23. The first optical system 121, the reflecting member 122, the second optical system 31, the reflecting member 32, the third optical system 33, and the lens 34 are arranged along the optical axis K in this order from the light modulation unit 22 side.

[0048] The first optical system 121 includes at least one lens, and guides light traveling in the direction X1 that is incident on the first member 102 from the main body 101 to the reflecting member 122 .

[0049] The reflecting member 122 reflects the light incident from the first optical system 121 in the direction Y1. The reflecting member 122 is formed of, for example, a mirror. The first member 102 has an opening 2b formed on the optical path of the light reflected by the reflecting member 122. The reflected light passes through the opening 2b and travels to the hollow portion 3A of the second member 103.

[0050] The second member 103 is a member having a substantially T-shaped cross-sectional outer shape, and has an opening 3a formed in a position facing the opening 2b of the first member 102. Light from the main body 101 that passes through the opening 2b of the first member 102 passes through this opening 3a and enters the hollow portion 3A of the second member 103. The cross-sectional outer shapes of the first member 102 and the second member 103 are arbitrary and are not limited to those described above.

[0051] The second optical system 31 includes at least one lens, and guides the light incident from the first member 102 to the reflecting member 32 .

[0052] The reflecting member 32 reflects the light incident from the second optical system 31 in the direction X2 and guides the light to the third optical system 33. The reflecting member 32 is formed of, for example, a mirror.

[0053] The third optical system 33 includes at least one lens, and guides the light reflected by the reflecting member 32 to a lens 34 .

[0054] The lens 34 is disposed at the end of the second member 103 in the direction X2 so as to cover the opening 3c formed at this end. The lens 34 projects the light incident from the third optical system 33 onto the object 6 to be projected.

[0055] The projection direction change mechanism 104 is a rotation mechanism that rotatably connects the second member 103 to the first member 102. This projection direction change mechanism 104 allows the second member 103 to rotate freely around a rotation axis (specifically, optical axis K) that extends in direction Y. Note that the projection direction change mechanism 104 is not limited to the arrangement position shown in Fig. 4 as long as it can rotate the optical system. Furthermore, the number of rotation mechanisms is not limited to one, and multiple mechanisms may be provided.

[0056] The shift mechanism 105 is a mechanism for moving the optical axis K of the projection optical system (in other words, the optical unit 106) in a direction perpendicular to the optical axis K (direction Y in FIG. 4 ). Specifically, the shift mechanism 105 is configured to be able to change the position of the first member 102 in direction Y relative to the main body 101. The shift mechanism 105 may be one that moves the first member 102 manually, or one that moves the first member 102 electrically.

[0057] Fig. 4 shows a state in which the first member 102 has been moved to the maximum extent in the direction Y1 by the shift mechanism 105. When the first member 102 is moved in the direction Y2 by the shift mechanism 105 from the state shown in Fig. 4, the relative position between the center of the image formed by the light modulation unit 22 (in other words, the center of the display surface) and the optical axis K changes, and the image G1 projected onto the projection object 6 can be shifted (translated) in the direction Y2.

[0058] The shift mechanism 105 may be a mechanism that moves the light modulation unit 22 in the direction Y, instead of moving the optical unit 106 in the direction Y. Even in this case, the image G1 projected onto the projection target 6 can be moved in the direction Y2.

[0059] <Appearance of image processing device 50> Fig. 5 is a diagram showing an example of the appearance of the image processing device 50. The image processing device 50 is a tablet terminal having a touch panel 51. The touch panel 51 is a display that can be touched. The image processing device 50 displays, on the touch panel 51, a placement assistance image for assisting in the installation of the projection device 10 in a space.

[0060] Specifically, the image processing device 50 displays a second image as a placement support image, which is a first image obtained by capturing an image of the space where the projection device 10 is installed and projection is to be performed, and which is superimposed with an image of a virtual projection surface, which is a virtual projection surface, and an image of a virtual projection device, which is a virtual projection device.

[0061] <Hardware configuration of image processing device 50> Fig. 6 is a diagram showing an example of the hardware configuration of the image processing device 50. The image processing device 50 shown in Fig. 5 includes, for example, a processor 61, a memory 62, a communication interface 63, a user interface 64, and a sensor 65, as shown in Fig. 6. The processor 61, the memory 62, the communication interface 63, the user interface 64, and the sensor 65 are connected by, for example, a bus 69.

[0062] The processor 61 is a circuit that performs signal processing, and is, for example, a CPU that controls the entire image processing device 50. The processor 61 may be realized by other digital circuits such as an FPGA or a DSP (Digital Signal Processor). The processor 61 may also be realized by combining multiple digital circuits.

[0063] The memory 62 includes, for example, a main memory and an auxiliary memory. The main memory is, for example, a RAM (Random Access Memory). The main memory is used as a work area for the processor 61.

[0064] The auxiliary memory is a non-volatile memory such as a magnetic disk, a flash memory, etc. The auxiliary memory stores various programs that operate the image processing device 50. The programs stored in the auxiliary memory are loaded into the main memory and executed by the processor 61.

[0065] The auxiliary memory may also include a portable memory that is removable from the image processing device 50. Portable memories include memory cards such as a USB (Universal Serial Bus) flash drive or an SD (Secure Digital) memory card, and an external hard disk drive.

[0066] The communication interface 63 is a communication interface for communicating with devices external to the image processing device 50. The communication interface 63 includes at least one of a wired communication interface for communicating via a wire and a wireless communication interface for communicating wirelessly. The communication interface 63 is controlled by the processor 61.

[0067] The user interface 64 includes, for example, an input device that accepts operation input from the user and an output device that outputs information to the user. The input device can be realized by, for example, keys (e.g., a keyboard) or a remote control. The output device can be realized by, for example, a display or a speaker. In the image processing device 50 shown in FIG. 5 , the input device and the output device are realized by the touch panel 51. The user interface 64 is controlled by the processor 61. The image processing device 50 accepts various specifications from the user using the user interface 64.

[0068] The sensor 65 includes an imaging device that has an imaging optical system and an imaging element and is capable of capturing images, and a space recognition sensor that is capable of three-dimensionally recognizing the space around the image processing device 50. The imaging device includes, for example, the imaging device provided on the back surface of the image processing device 50 shown in FIG.

[0069] An example of the spatial recognition sensor is a LIDAR (Light Detection and Ranging) that emits laser light, measures the time it takes for the emitted laser light to hit an object and bounce back, and measures the distance and direction to the object. However, the spatial recognition sensor is not limited to this, and various sensors such as a radar that emits radio waves and an ultrasonic sensor that emits ultrasonic waves can be used.

[0070] <Physical Space in Which Image Processing Device 50 is Used> Fig. 7 is a diagram showing an example of a physical space in which the image processing device 50 is used. As shown in Fig. 7, for example, a user of the image processing device 50 brings the image processing device 50 into a physical space 70, which is a physical space in which the projection device 10 is to be installed.

[0071] The image processing device 50 recognizes the physical space 70 using a space recognition sensor. Specifically, the image processing device 50 recognizes the physical space 70 using a world coordinate system consisting of the X-axis, Y-axis, and Z-axis, where the X-axis represents one horizontal direction in the physical space 70, the Y-axis represents the direction of gravity in the physical space 70, and the Z-axis represents a direction perpendicular to the X-axis and Y-axis in the physical space 70. The image processing device 50 also displays a captured image based on imaging data obtained by imaging using an imaging device to the user as a through image (live view) on the touch panel 51. The imaging data is an example of first image data. The captured image is an example of a first image.

[0072] <Arrangement When a Surface That Serves as a Reference for the Position and Orientation of the Virtual Projection Plane Does Not Exist> If a surface, such as a wall or a projection screen, exists in the physical space 70 that serves as a reference for the position and orientation of the virtual projection plane, the position and orientation of the virtual projection plane can be determined relatively easily by using information about that surface. Furthermore, the position and orientation of the virtual projection device can be determined by determining and presenting the possible installation range of the virtual projection device from the virtual projection plane and having the user specify a position within the possible installation range. However, if a surface that serves as a reference for the position and orientation of the virtual projection plane does not exist, it is difficult to determine the positions and orientations of the virtual projection plane and the virtual projection device. In contrast, according to this embodiment, it is possible to efficiently determine the arrangement of the virtual projection device and the virtual projection plane, even if a surface that serves as a reference for the position and orientation of the virtual projection plane does not exist.

[0073] <State in which the orientation of the virtual projection surface has not been determined> Figure 8 is a diagram showing an example of a state in which the orientation of the virtual projection surface has not been determined. The virtual projection surface installation position 81 is the installation position in the physical space 70 of the virtual projection surface 80, which is the virtual projection surface 11. For example, the virtual projection surface installation position 81 is a point included in the virtual projection surface 80. In the example of Figure 8, the virtual projection surface installation position 81 is the center point of the rectangular virtual projection surface 80. However, the virtual projection surface installation position 81 does not have to be included in the virtual projection surface 80 as long as it is a position that defines the position of the virtual projection surface 80.

[0074] The horizontal direction of the virtual projection plane 80 is S Ｘ The vertical direction of the virtual projection plane 80 is S ＹThe direction perpendicular to the axis and the virtual projection plane 80 is S Ｚ When only the virtual projection plane installation position 81 of the virtual projection plane 80 in the physical space 70 is determined, S Ｘ axis, S Ｙ axis, and S Ｚ It is not yet decided how to set the axes relative to the X-axis, Y-axis, and Z-axis of the world coordinate system, that is, the orientation of the virtual projection plane 80 is not yet determined.

[0075] <Determining the Orientation of the Virtual Projection Device by Determining the Orientation of the Virtual Projection Surface 80> Figure 9 is a diagram showing an example of determining the orientation of the virtual projection device by determining the orientation of the virtual projection surface 80. The virtual projection device installation position 91 is the installation position in the physical space 70 of the virtual projection device, which is the virtual projection device 10. For example, the virtual projection device installation position 91 is a point included in the virtual projection device. As an example, the virtual projection device installation position 91 is a position corresponding to the projection unit 1 (e.g., the lens 34) of the projection device 10. However, the virtual projection device installation position 91 does not have to be included in the virtual projection device as long as it is a position that defines the position of the virtual projection device.

[0076] The vertical direction of the virtual projection device is P Ｙ axis, the horizontal direction of the virtual projection device is P Ｘ axis, the front-to-back direction (projection direction) of the virtual projection device is P Ｚ When the virtual projection surface installation position 81 and the direction of the virtual projection surface 80 are determined, the P Ｙ The axis is S of the virtual projection plane 80 Ｙ Set it in the same direction as the axis, and Ｚ The axis is S of the virtual projection plane 80 Ｚ By setting it in the same direction as the axis, the direction of the virtual projection device can be determined.

[0077] Once the orientation of the virtual projection device is determined, it is possible to determine the projection distance D from the virtual projection device to the virtual projection surface 80. Then, it is possible to determine the size (width and height) of the virtual projection surface 80 based on the projection distance D.

[0078] 10 is a diagram showing an example of a method for specifying a position in the physical space 70. A three-dimensional orthogonal coordinate system is defined with the position of the camera (image processing device 50) capturing and displaying the captured image as the center, with the horizontal direction of the camera as the T Ｘ , the vertical direction of the camera is T Ｙ , the depth direction of the camera is T Ｚ Define it as:

[0079] The image processing device 50 displays a position designation image on the touch panel 51. The position designation image is displayed in the physical space 70 from the camera position. Ｚ The image processing device 50 receives an instruction to change the distance d1 from the user.

[0080] As a result, for example, while looking at the position designation image displayed on the touch panel 51 of the image processing device 50, the user points the imaging device of the image processing device 50 toward a position in the physical space 70 that the user wants to designate, and adjusts the position and orientation of the image processing device 50 so that the position in the image processing device 50 that the user wants to designate is located on a line connecting the camera position and the position object P1. The user also adjusts the distance d1 by operating the image processing device 50 so that the position in the physical space 70 that the position object P1 matches. Then, the user performs a designation position determination operation on the image processing device 50 with the position in the physical space 70 that the position object P1 matches.

[0081] When the image processing device 50 receives the designated position determination operation, it determines the position of the position object P1 at that time as the position designated by the user in the physical space 70. This allows the user to designate any position in the physical space 70 to the image processing device 50 as, for example, the virtual projection surface installation position 81 or the virtual projection device installation position 91.

[0082] 11 is a diagram showing an example of the virtual projection device installation position 91, the virtual projection surface installation position 81, and the reference point. The image processing device 50 receives designation of the virtual projection device installation position 91, the virtual projection surface installation position 81, and the reference point 111 from the user using, for example, the designation method shown in FIG.

[0083] The virtual projection surface installation position 81 is a first position in the physical space 70 that corresponds to the position of the virtual projection surface 80. The reference point 111 is a second position in the physical space 70 that serves as a reference for the orientation of the virtual projection surface 80, and is a position that is not on a plane that includes the virtual projection surface 80. The reference point 111 is, for example, the virtual projection device installation position 91. In this case, the image processing device 50 only needs to accept the position specified as the virtual projection device installation position 91 as the reference point 111, and does not need to accept specification of the reference point 111 separately from the position of the virtual projection device installation position 91.

[0084] <Example of Positional Relationship Between Virtual Projection Plane Installation Position 81 and Reference Point 111> Fig. 12 is a diagram showing a first example of the positional relationship between the virtual projection plane installation position 81 and the reference point 111. Fig. 13 is a diagram showing a second example of the positional relationship between the virtual projection plane installation position 81 and the reference point 111.

[0085] The installation position plane is a plane that passes through the virtual projection surface installation position 81 and the reference point 111 specified by the user and is parallel to the direction of gravity (Y axis) of the physical space 70. The coordinate axes on the installation position plane are set to be the Y axis, which is the same as the Y axis in the physical space 70, and the X' axis, which is perpendicular to the Y axis. The Y axis on the installation position plane is vertical, and the X' axis on the installation position plane is horizontal.

[0086] The first angle Θ is the angle formed on the installation position plane between a first line segment S1 connecting the reference point 111 (second position) and the virtual projection surface installation position 81 (first position) and a second line segment S2 that passes through the reference point 111 and is parallel to the X' axis. In other words, the first angle Θ is the angle formed in the physical space 70 between the first line segment S1 connecting the reference point 111 (second position) and the virtual projection surface installation position 81 (first position) and a horizontal plane that includes the reference point 111.

[0087] 12, it is assumed that the first angle Θ is relatively small and less than the threshold value. In this case, the image processing device 50 calculates the angle between the Y axis of the physical space 70 and the S axis of the virtual projection plane 80. Ｙ The orientation of the virtual projection plane 80 is set so that the axes are parallel.

[0088] 13, it is assumed that the first angle Θ is relatively large and equal to or greater than the threshold value. In this case, the image processing device 50 calculates the angle between the Y axis of the physical space 70 and the S axis of the virtual projection plane 80. Ｚ The direction of the virtual projection plane 80 is set so that the Y axis and S axis are parallel. Ｚ The axes are opposite to each other, but the Y axis and S Ｚ The same is true when the axes are in the same direction (floor projection).

[0089] 12 and 13, the image processing device 50 determines the orientation of the virtual projection plane 80 to be a plane parallel to the direction of gravity (FIG. 12) or a plane perpendicular to the direction of gravity (FIG. 13) depending on the result of comparing the first angle Θ with the threshold. This makes it possible to determine the orientation of the virtual projection plane 80 depending on the positional relationship between the virtual projection plane installation position 81 and the reference point 111. The threshold can be set to 80 degrees, for example, but is not limited to this and can be set arbitrarily.

[0090] <Determination of the Orientation of the Virtual Projection Plane 80 When the First Angle Θ is Less than the Threshold Value> Fig. 14 is a diagram showing an example of determining the orientation of the virtual projection plane 80 when the first angle Θ is less than the threshold value. When the first angle Θ is less than the threshold value, the S of the virtual projection plane 80 as shown in Fig. 12 is Ｙ The axis is determined in the vertical direction, but the S Ｘ Axis and S Ｚ The axis is in an undetermined state.

[0091] In this case, the image processing device 50 calculates the S of the virtual projection plane 80 when viewed on a plane perpendicular to the Y axis as shown in FIG. Ｚ The S axis of the virtual projection plane 80 is directed toward the reference point 111. Ｘ Axis and S Ｚ The axis is determined, and the orientation of the virtual projection plane 80 is thereby determined.

[0092] 15 is a diagram showing an example of the projection distance D when the first angle Θ is less than the threshold. After determining the orientation of the virtual projection plane 80 as shown in FIG. 14, the image processing device 50 calculates the projection distance D of the virtual projection plane 80 by calculating the projection distance D when the first angle Θ is less than the threshold. Ｚ The intersection of a straight line parallel to the axis and a plane that passes through the virtual projection surface installation position 81 and is parallel to the virtual projection surface 80 is calculated as the projection center 151 without lens shift. Then, the image processing device 50 calculates the distance between the projection center 151 without lens shift and the virtual projection device installation position 91 as the projection distance D.

[0093] <Determination of the Orientation of the Virtual Projection Plane 80 When the First Angle Θ is Greater than or Equal to the Threshold Value> Fig. 16 is a diagram showing an example of determining the orientation of the virtual projection plane 80 when the first angle Θ is greater than or equal to the threshold value. When the first angle Θ is greater than or equal to the threshold value, the S of the virtual projection plane 80 is determined as shown in Fig. 13 . Ｚ The axis is determined in the vertical direction, but the S Ｘ Axis and S Ｙ The axis is in an undetermined state.

[0094] In this case, the image processing device 50 calculates the S of the virtual projection plane 80 when viewed on a plane perpendicular to the Y axis as shown in FIG. Ｙ The S axis of the virtual projection plane 80 is directed toward the reference point 111. Ｘ Axis and S Ｙ The axis is determined, and the orientation of the virtual projection plane 80 is thereby determined.

[0095] 17 is a diagram showing an example of the projection distance D when the first angle Θ is equal to or greater than a threshold. After determining the orientation of the virtual projection plane 80 as shown in FIG. 16, the image processing device 50 calculates the projection distance D of the virtual projection plane 80 by calculating the projection distance D when the first angle Θ is equal to or greater than a threshold. Ｚ The intersection of a straight line parallel to the axis and a plane that passes through the virtual projection surface installation position 81 and is parallel to the virtual projection surface 80 is calculated as the projection center 151 without lens shift. Then, the image processing device 50 calculates the distance between the projection center 151 without lens shift and the virtual projection device installation position 91 as the projection distance D.

[0096] <Processing by Image Processing Device 50> Fig. 18 is a flowchart showing an example of processing by the image processing device 50. The image processing device 50 executes, for example, the processing shown in Fig. 18 .

[0097] First, the image processing device 50 determines the virtual projection device installation position 91, the virtual projection surface installation position 81, and the reference point 111 (step S11). For example, the image processing device 50 receives designation of the virtual projection device installation position 91, the virtual projection surface installation position 81, and the reference point 111 as shown in Fig. 11 from the user using the designation method shown in Fig. 10 .

[0098] Next, the image processing device 50 calculates (step S12) the positional relationship between the virtual projection plane installation position 81 determined in step S11 and the reference point 111. For example, the image processing device 50 calculates the first angle Θ shown in FIGS.

[0099] Next, the image processing device 50 determines the orientation of the virtual projection surface 80 based on the positional relationship between the virtual projection surface installation position 81 calculated in step S12 and the reference point 111 (step S13). For example, as shown in FIGS. 12 and 13, the image processing device 50 determines the S of the virtual projection surface 80 based on the magnitude of the first angle Θ (the result of comparison with the threshold value). Ｙ Axis and S Ｚ The image processing device 50 determines which of the axes should be oriented in the same direction as the Y axis of the physical space 70. Furthermore, as shown in FIGS. 14 and 16, the image processing device 50 determines that the remaining axes of the virtual projection plane 80 should be oriented toward the reference point 111.

[0100] Next, the image processing device 50 determines the orientation of the virtual projection device based on the orientation of the virtual projection surface 80 determined in step S13 (step S14). For example, the image processing device 50 determines the orientation of the virtual projection device based on the orientation of the virtual projection surface 80 determined in step S13 (step S14). Ｙ The axis is S of the virtual projection plane 80 Ｙ Set it in the same direction as the axis, and Ｚ The axis is S of the virtual projection plane 80 Ｚ The orientation of the virtual projection device is determined by setting it in the same direction as the axis.

[0101] Next, the image processing device 50 calculates the projection distance D between the virtual projection device and the virtual projection surface 80 based on the orientation of the virtual projection device determined in step S14 (step S15). For example, the image processing device 50 calculates the distance between the projection center 151 without lens shift and the virtual projection device installation position 91 as the projection distance D, as shown in FIGS.

[0102] Next, the size of the virtual projection surface 80 is determined based on the projection distance D calculated in step S15 (step S16). For example, the image processing device 50 determines the width and height of the virtual projection surface 80 based on the specifications (e.g., angle of view and aspect ratio) of the projection device 10 represented by the virtual projection device and the projection distance D.

[0103] Through the processing up to this point, the position (virtual projection device installation position 91) and orientation of the virtual projection device, and the position (virtual projection surface installation position 81), orientation, and size of the virtual projection surface 80 are determined. Next, using this information, the image processing device 50 superimposes a virtual projection device image representing the virtual projection device and a virtual projection surface image representing the virtual projection surface 80 on a captured image of the physical space 70 represented by the imaging data obtained by imaging by the image processing device 50 (step S17).

[0104] Next, the image processing device 50 displays the superimposed image obtained in step S17 as a placement support image on the touch panel 51 (step S18). This allows the user to view a placement support image that virtually shows the projection device 10 and projection surface 11 placed in the physical space 70 at the position and orientation determined based on the virtual projection device installation position 91 and virtual projection surface installation position 81 specified in step S11. The placement support image is an example of a second image. Placement support image data representing the placement support image is an example of second image data.

[0105] Furthermore, the image processing device 50 may re-execute steps S17 and S18 every time the position or orientation of the image processing device 50 in the physical space 70 changes (i.e., every time the captured image changes). That is, the image processing device 50 may update the superimposed virtual projection device image and virtual projection surface image, as well as their arrangement, in the displayed placement support image in accordance with the captured image data after the change.

[0106] 19 is a diagram showing an example of recalculation of the projection distance D when the user changes the orientation of the virtual projection surface 80. When the image processing device 50 receives an instruction operation from the user to change the orientation of the virtual projection surface 80 after the processing shown in FIG. 18 , for example, the image processing device 50 changes the orientation of the virtual projection surface 80 based on the instruction operation received from the user, changes the size of the virtual projection surface 80 based on the changed orientation of the virtual projection surface 80 and the virtual projection device installation position 91, and updates the placement support image (placement support image data) to be displayed.

[0107] 19 shows a virtual projection surface 80 whose orientation has been changed. A perpendicular line 191 is drawn from the virtual projection device installation position 91 to a plane that passes through the virtual projection surface installation position 81 and is parallel to the changed virtual projection surface 80. The image processing device 50 recalculates the length of the perpendicular line 191 as a new projection distance D. The image processing device 50 also re-determines the size of the virtual projection surface 80 based on the calculated projection distance D.

[0108] Then, based on the orientation of the virtual projection surface 80 after the change and the size of the virtual projection surface 80 that has been determined again, the virtual projection surface image superimposed on the captured image is updated, and a placement support image (second image) with the updated virtual projection surface image is displayed.

[0109] The processing when the orientation of the virtual projection plane 80 is changed by the user has been described. For example, after the processing shown in FIG. 18, the image processing device 50 Ｚ When an instruction operation to change the position of the direction is received from the user, the S of the virtual projection plane 80 is Ｚ In this case, the image processing device 50 may change the position of the direction of the virtual projection plane 80 after the change.Ｚ The projection distance D is calculated based on the direction position and the virtual projection device installation position 91, the size of the virtual projection surface 80 is changed based on the calculated projection distance D, and the placement support image (placement support image data) to be displayed is updated.

[0110] 20 is a diagram showing an example of determining the position of the virtual projection plane 80 based on the detection of a plane that serves as a reference for the position of the virtual projection plane 80. When the image processing device 50 detects a line or plane that serves as a reference for the position of the virtual projection plane 80 in the physical space 70, the image processing device 50 may determine the position of the virtual projection plane 80 on a plane that includes the virtual projection plane 80 based on the position of the detected line or plane, and display a placement assistance image.

[0111] 20 , for example, a floor surface 201 exists in the physical space 70, and the image processing device 50 detects the floor surface 201 using a space recognition sensor. Since the floor surface 201 is perpendicular to the orientation of the virtual projection surface 80, the image processing device 50 recognizes that the floor surface 201 is a surface that serves as a reference for the position of the virtual projection surface 80.

[0112] In this case, between steps S16 and S17 shown in Figure 18, the image processing device 50 changes the virtual projection surface installation position 81 determined in step S11 based on the size of the virtual projection surface 80 determined in step S16 so that the end (bottom end) of the virtual projection surface 80 contacts the floor surface 201.

[0113] As a result, in step S17, the virtual projection device image and the virtual projection surface image are superimposed on the captured image in a state in which the virtual projection surface 80 has moved downward in parallel compared to when the virtual projection surface installation position 81 is not changed. This determination of the position of the virtual projection surface 80 based on the detection of the surface that serves as the reference for the position of the virtual projection surface 80 may also be performed when steps S17 and S18 are executed again due to a change in the position or orientation of the image processing device 50 in the physical space 70, as described above.

[0114] Although the case where the virtual projection surface installation position 81 is changed so that the edge of the virtual projection surface 80 contacts the detected surface (floor surface 201) has been described here, the image processing device 50 may also change the virtual projection surface installation position 81 so that the distance between the detected surface (floor surface 201) and the edge of the virtual projection surface 80 becomes a predetermined offset value. The offset value may be determined in advance or may be specified by the user.

[0115] <Determining the orientation of the provisional virtual projection surface 80 based on the camera position> Although the case where the virtual projection device installation position 91 is used as the reference point 111 that serves as the basis for the orientation of the virtual projection surface 80 has been described, the image processing device 50 may provisionally determine the orientation of the virtual projection surface 80 using the position of the image processing device 50 (camera position) before determining the final orientation of the virtual projection surface 80 based on the virtual projection device installation position 91, and display a placement support image based on the orientation of the virtual projection surface 80.

[0116] 21 and 22 are diagrams illustrating an example of determining the orientation of the provisional virtual projection plane 80 based on the camera position when the first angle Θ is less than the threshold. When the first angle Θ is less than the threshold, the S of the virtual projection plane 80 is Ｙ The axis is determined in the vertical direction, but the S Ｘ Axis and S Ｚ The axis is in an undetermined state. The camera position 211 is the position of the image processing device 50.

[0117] In the example of FIG. 21, the S Ｚ The angle between the axis and the line segment connecting the virtual projection plane installation position 81 and the camera position 211 is large, making it difficult to see the virtual projection plane 80 from the camera position 211. In such a case, the image processing device 50, for example, in response to a user operation or automatically, calculates the S Ｚ The S axis of the virtual projection plane 80 is oriented toward the camera position 211. Ｘ Axis and S Ｚ The axis is tentatively determined.

[0118] As a result, as shown in FIG. 22 , the orientation of the virtual projection surface 80 is determined based on the camera position 211, making it easier to see the virtual projection surface 80 from the camera position 211. For example, the image processing device 50 repeatedly updates the orientation of the virtual projection surface 80 shown in FIGS. 21 and 22 in accordance with the movement of the user (movement of the camera position 211). Then, upon receiving an operation from the user to stop updating the orientation of the virtual projection surface 80, the image processing device 50 sets the camera position 211 at that time or a point designated by the user as the reference point 111. In this case, the virtual projection device installation position 91 may be set before or after setting the reference point 111. As a result, the virtual projection device installation position 91, the virtual projection surface installation position 81, and the reference point 111 are determined, and processing similar to steps S12 to S18 shown in FIG. 18 is executed. In this way, by temporarily treating the camera position 211 as the reference point 111, it is possible to make the virtual projection plane 80 easily visible to the user even when the reference point 111 is not yet determined.

[0119] Although we have explained how to determine the orientation of the provisional virtual projection surface 80 based on the camera position when the first angle Θ is less than a threshold value, the image processing device 50 may also determine the orientation of the provisional virtual projection surface 80 based on the camera position when the first angle Θ is greater than or equal to a threshold value.

[0120] In this way, the image processing device 50 may determine the orientation of the virtual projection surface 80 based on the positional relationship between the virtual projection surface installation position 81 and the reference point 111, and the position of the image processing device 50 (imaging device).

[0121] As described above, the processor 61 of the image processing device 50 acquires first image data obtained by capturing an image of the physical space 70 using the imaging device of the sensor 65. The processor 61 of the image processing device 50 also determines a virtual projection surface installation position 81 (first position) corresponding to the position of the virtual projection surface 80 in the physical space 70, and a reference point 111 (second position) that is not on a plane including the virtual projection surface 80 and serves as a reference for the orientation of the virtual projection surface 80, and determines the orientation of the virtual projection surface 80 based on the positional relationship between the virtual projection surface installation position 81 and the reference point 111 to generate virtual projection surface data representing the virtual projection surface 80. The processor 61 of the image processing device 50 then generates second image data representing a second image in which the virtual projection surface 80 is displayed on the first image represented by the first image data, based on the first image data and the virtual projection surface data, and outputs the second image data to the touch panel 51 (output destination).

[0122] This makes it possible to easily determine the position and orientation of the virtual projection surface 80 even if there is no surface, such as a wall or projection screen, that serves as a reference for the position and orientation of the virtual projection surface in the physical space 70. This makes it possible to efficiently determine the placement of the virtual projection device and the virtual projection surface 80, thereby improving the convenience for the user regarding the placement of the projection surface 11 and the projection device 10.

[0123] (Modifications) Modifications of the above embodiments will be described.

[0124] <Modification 1> Although the image processing device 50 has been described as a tablet terminal having a touch panel 51, the image processing device 50 is not limited to this configuration. For example, the image processing device 50 may be an information terminal such as a smartphone or a personal computer.

[0125] <Modification 2> Although the configuration in which the image processing device 50 displays the second image on the touch panel 51 has been described, the image processing device 50 may transmit the generated second image to another device, thereby controlling the other device to display the second image. In this case, the image processing device 50 may be a device that does not have a display device.

[0126] <Modification 3> Although the case where the captured image representing the physical space 70 is an image obtained by imaging using the imaging device of the image processing device 50 has been described, the captured image may be an image obtained by imaging using a device other than the image processing device 50 and received by the image processing device 50 from that device. In this case, the image processing device 50 may be a device that does not include an imaging device.

[0127] <Modification 4> Although the case where the reference point 111 is the virtual projection device installation position 91 has been described, the reference point 111 is not limited to this and may be the position of the imaging device (image processing device 50), the position of the observer observing the virtual projection surface 80, or a combination of these positions. Note that the position of the imaging device (image processing device 50) is, for example, the origin of the world coordinate system when the image processing device 50 recognizes the physical space 70, and therefore does not need to be specified by the user. The image processing device 50 accepts a specification from the user of the position of the observer observing the virtual projection surface 80, for example, using the specification method shown in FIG. 10 .

[0128] (Image Processing Program) The image processing method described in the above-described embodiment can be realized by executing a prepared image processing program on a computer. The image processing program is recorded on a computer-readable storage medium and executed by being read from the storage medium. The image processing program may be provided in a form stored on a non-transitory storage medium such as a flash memory, or may be provided via a network such as the Internet. The computer that executes the image processing program may be included in an image processing device, or may be included in an electronic device such as a smartphone, tablet terminal, or personal computer that can communicate with the image processing device, or may be included in a server device that can communicate with these image processing devices and electronic devices.

[0129] The above-described embodiments and modifications can be implemented in combination.

[0130] This specification describes at least the following:

[0131] (1) An image processing device including a processor, wherein the processor: acquires first image data obtained by capturing an image of a space with an imaging device; determines a first position in the space corresponding to the position of a virtual projection surface and a second position serving as a reference for the orientation of the virtual projection surface; determines the orientation of the virtual projection surface based on the positional relationship between the first position and the second position, and generates virtual projection surface data representing the virtual projection surface; generates second image data based on the first image data and the virtual projection surface data, representing a second image in which the virtual projection surface is displayed on a first image represented by the first image data; and outputs the second image data to an output destination.

[0132] (2) The image processing device according to (1), wherein the second position is a position that is not on a plane that includes the virtual projection plane.

[0133] (3) The image processing device according to (1) or (2), wherein the second position is at least one of an installation position of a virtual projection device corresponding to the virtual projection surface, a position of the imaging device, and a position of an observer observing the virtual projection surface.

[0134] (4) An image processing device according to any one of (1) to (3), wherein the processor calculates a first angle, which is the angle between a first line segment connecting the second position and the first position and a plane that includes the second position and is perpendicular or horizontal to the direction of gravity, and determines the orientation of the virtual projection plane based on the magnitude of the first angle.

[0135] (5) The image processing device according to (4), wherein the processor determines the orientation of the virtual projection plane to be a plane parallel to the direction of gravity or a plane perpendicular to the direction of gravity, depending on a result of comparing the first angle with a threshold value.

[0136] (6) The image processing device according to any one of (1) to (5), wherein the processor determines the first position and the second position based on an instruction received from a user.

[0137] (7) The image processing device according to any one of (1) to (6), wherein the processor determines the orientation of the virtual projection surface based on the positional relationship and an installation position of a virtual projection device corresponding to the virtual projection surface.

[0138] (8) The image processing device according to any one of (1) to (6), wherein the processor determines the orientation of the virtual projection plane based on the positional relationship and the position of the imaging device.

[0139] (9) The image processing device according to any one of (1) to (8), wherein the processor determines the size of the virtual projection surface based on the orientation of the virtual projection surface and the installation position of the virtual projection device corresponding to the virtual projection surface, and generates the virtual projection surface data.

[0140] (10) An image processing device according to (9), wherein the processor changes the orientation of the virtual projection screen based on an instruction to change the orientation of the virtual projection screen received from a user, changes the size of the virtual projection screen based on the orientation of the virtual projection screen after the change and the installation position of the virtual projection device, and updates the virtual projection screen data and the second image data.

[0141] (11) An image processing device according to any one of (1) to (10), wherein, when the processor detects a line or a plane in the space that serves as a reference for the position of the virtual projection plane, the processor determines the position of the virtual projection plane on a plane including the virtual projection plane based on the position of the line or plane, and generates the virtual projection plane data.

[0142] (12) The image processing device according to (11), wherein the processor determines the position of the virtual projection plane so that the distance between the position of the line or plane and an end of the virtual projection plane is a predetermined offset value.

[0143] (13) An image processing device according to any one of (1) to (12), wherein the processor changes the position of the virtual projection surface based on an instruction received from a user to change the position of the virtual projection surface in a direction perpendicular to the virtual projection surface, changes the size of the virtual projection surface based on the changed position of the virtual projection surface and an installation position of a virtual projection device corresponding to the virtual projection surface, and updates the virtual projection surface data and the second image data.

[0144] (14) The image processing device according to any one of (1) to (13), wherein the image processing device is provided in an information processing terminal including the imaging device and a display device, and the output destination is the display device.

[0145] (15) An image processing method in which a processor included in an image processing device acquires first image data obtained by capturing an image of a space with an imaging device, determines a first position in the space corresponding to the position of a virtual projection surface and a second position serving as a reference for the orientation of the virtual projection surface, determines the orientation of the virtual projection surface based on the positional relationship between the first position and the second position, and generates virtual projection surface data representing the virtual projection surface, generates second image data based on the first image data and the virtual projection surface data, representing a second image in which the virtual projection surface is displayed on a first image represented by the first image data, and outputs the second image data to an output destination.

[0146] (16) An image processing program for causing a processor included in an image processing device to execute the following processes: acquire first image data obtained by capturing an image of a space with an imaging device; determine a first position in the space corresponding to the position of a virtual projection surface and a second position serving as a reference for the orientation of the virtual projection surface; determine the orientation of the virtual projection surface based on the positional relationship between the first position and the second position to generate virtual projection surface data representing the virtual projection surface; generate second image data based on the first image data and the virtual projection surface data to represent a second image in which the virtual projection surface is displayed on a first image represented by the first image data; and output the second image data to an output destination.

[0147] Although various embodiments have been described above, it goes without saying that the present invention is not limited to such examples. It is clear that a person skilled in the art can conceive of various modifications or alterations within the scope of the claims, and it is understood that these also naturally fall within the technical scope of the present invention. Furthermore, the components of the above embodiments may be combined in any manner as long as they do not deviate from the spirit of the invention.

[0148] This application is based on a Japanese patent application (Patent Application No. 2022-131119) filed on August 19, 2022, the contents of which are incorporated by reference into this application.

[0149] REFERENCE SIGNS LIST 1 Projection unit 2 Operation reception unit 2A, 3A Hollow portion 2a, 2b, 3a, 3c, 15a Opening 4 Control device 4a, 62 Memory 6 Projected object 10 Projection device 11 Projection surface 12 Light modulation unit 15 Housing 21 Light source 22 Light modulation unit 23 Projection optical system 24 Control circuit 31 Second optical system 32, 122 Reflecting member 33 Third optical system 34 Lens 50 Image processing device 51 Touch panel 61 Processor 63 Communication interface 64 User interface 65 Sensor 69 Bus 70 Physical space 80 Virtual projection surface 81 Virtual projection surface installation position 91 Virtual projection device installation position 101 Main body 102 First member 103 Second member 104 Projection direction change mechanism 105 Shift mechanism 106 Optical unit 111 Reference point 121 First optical system 151 Projection center 191 Perpendicular line 201 Floor surface 211 Camera position d1 Distance G1 Image P1 Position object S1 First line segment S2 Second line segment

Claims

1. An image processing device including a processor, The processor, Acquiring first image data obtained by imaging the space with an imaging device; determining a first position in the space corresponding to a position of a virtual projection plane and a second position serving as a reference for an orientation of the virtual projection plane in the space; determining an orientation of the virtual projection plane in the space based on a positional relationship between the first position and the second position to generate virtual projection plane data representing the virtual projection plane; generating second image data representing a second image in which the virtual projection surface is displayed on a first image represented by the first image data, based on the first image data and the virtual projection surface data; outputting the second image data to an output destination; Image processing device.

2. 2. The image processing device according to claim 1, The second position is a position that is not on a plane including the virtual projection plane. Image processing device.

3. 2. The image processing device according to claim 1, the second position is at least one of an installation position of a virtual projection device corresponding to the virtual projection surface, a position of the imaging device, and a position of a viewer viewing the virtual projection surface; Image processing device.

4. 2. The image processing device according to claim 1, The processor, Calculating a first angle between a first line segment connecting the second position and the first position and a plane that includes the second position and is perpendicular or horizontal to a direction of gravity; determining an orientation of the virtual projection plane based on the magnitude of the first angle; Image processing device.

5. 5. The image processing device according to claim 4, The processor determines the orientation of the virtual projection plane to be a plane parallel to a gravity direction or a plane perpendicular to a gravity direction according to a comparison result between the first angle and a threshold value. Image processing device.

6. 2. The image processing device according to claim 1, The processor determines the first position and the second position based on an instruction received from a user. Image processing device.

7. 2. The image processing device according to claim 1, The processor determines a direction of the virtual projection surface based on the positional relationship and an installation position of a virtual projection device corresponding to the virtual projection surface. Image processing device.

8. 2. The image processing device according to claim 1, The processor determines an orientation of the virtual projection plane based on the positional relationship and the position of the imaging device. Image processing device.

9. 2. The image processing device according to claim 1, The processor determines a size of the virtual projection screen based on a direction of the virtual projection screen and an installation position of a virtual projection device corresponding to the virtual projection screen, and generates the virtual projection screen data. Image processing device.

10. 10. The image processing device according to claim 9, The processor, changing the orientation of the virtual projection screen based on an instruction to change the orientation of the virtual projection screen received from a user; changing a size of the virtual projection plane based on the changed orientation of the virtual projection plane and an installation position of the virtual projection device; updating the virtual projection surface data and the second image data; Image processing device.

11. 2. The image processing device according to claim 1, When the processor detects a line or a plane serving as a reference for the position of the virtual projection plane in the space, the processor determines the position of the virtual projection plane on a plane including the virtual projection plane based on the position of the line or plane, and generates the virtual projection plane data. Image processing device.

12. The image processing device according to claim 11, The processor, determining a position of the virtual projection plane so that a distance between the position of the line or face and an end of the virtual projection plane is a predetermined offset value; Image processing device.

13. 2. The image processing device according to claim 1, The processor, changing a position of the virtual projection screen based on an instruction to change the position of the virtual projection screen in a direction perpendicular to the virtual projection screen received from a user; changing a size of the virtual projection plane based on the changed position of the virtual projection plane and an installation position of a virtual projection device corresponding to the virtual projection plane; updating the virtual projection surface data and the second image data; Image processing device.

14. 14. The image processing device according to claim 1, provided in an information processing terminal including the imaging device and a display device, The output destination is the display device. Image processing device.

15. A processor included in the image processing device Acquiring first image data obtained by imaging the space with an imaging device; determining a first position in the space corresponding to a position of a virtual projection plane and a second position serving as a reference for an orientation of the virtual projection plane in the space; determining an orientation of the virtual projection plane in the space based on a positional relationship between the first position and the second position to generate virtual projection plane data representing the virtual projection plane; generating second image data representing a second image in which the virtual projection surface is displayed on a first image represented by the first image data, based on the first image data and the virtual projection surface data; outputting the second image data to an output destination; Image processing methods.

16. A processor of the image processing device includes: Acquiring first image data obtained by imaging the space with an imaging device; determining a first position in the space corresponding to a position of a virtual projection plane and a second position serving as a reference for an orientation of the virtual projection plane in the space; determining an orientation of the virtual projection plane in the space based on a positional relationship between the first position and the second position to generate virtual projection plane data representing the virtual projection plane; generating second image data representing a second image in which the virtual projection surface is displayed on a first image represented by the first image data, based on the first image data and the virtual projection surface data; outputting the second image data to an output destination; An image processing program to carry out the processing.