Image processing device, image processing method, and image processing program

By generating and displaying images of the imaginary projection surface and projection device in the physical space by the image processing device, combined with touch panel operation, the problem of insufficient configuration convenience of projection device in the prior art is solved, and the precise position of the projection surface and projection device is realized. and angle adjustments improve user experience.

CN118901087APending Publication Date: 2024-11-05FUJIFILM CORP
View PDF 0 Cites 0 Cited by

Patent Information

Application Number
CN202380028591.0
Authority / Receiving Office
CN · China
Patent Type
Applications(China)
Current Assignee / Owner
Priority Date
2022-03-30
Filing Date
2023-03-03
Publication Date
2024-11-05

AI Technical Summary

Technical Problem

The prior art lacks convenience in the configuration and user interaction of the projection device, making it difficult to effectively assist users in visually grasping the positional relationship between the projection surface and the projection device in physical space, especially when the projection surface size and angle need to be adjusted. How to operate.

Method used

The image processing device acquires the image data, generates the image data of the imaginary projection surface and the imaginary projection device, and overlaps the display on the physical spatial image, providing touch panel operation so that the user can adjust the position and angle of the projection surface and the projection device, and realizes the projection surface and the projection surface and the projection surface. Precise configuration and display of projection devices.

Benefits of technology

It improves the convenience of the user in the configuration of the projection device, allows the user to intuitively adjust the position and angle of the projection surface and the projection device in the physical space, and enhances the convenience and accuracy of the use of the projection device.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN118901087A_ABST
    Figure CN118901087A_ABST
Patent Text Reader

Abstract

The invention provides an image processing device, an image processing method, and an image processing program capable of improving user convenience related to the arrangement of a projection surface or a projection device. An image processing device (50) acquires first image data obtained by imaging a physical space (70) by an imaging device. Furthermore, the image processing device (50) determines a first position (81) in the physical space (70) on the basis of first position data indicating the first position (81) in the physical space (70) and first normal vector data indicating a first normal vector (82) of a first surface corresponding to a wall (72) existing in the first position (81) in the physical space (70). First virtual projection surface data representing a first virtual projection surface (111) and first virtual projection device data representing a first virtual projection device (112) are generated. Furthermore, on the basis of these data, the image processing device (50) generates second image data indicating that a first virtual projection surface (111) and a second image of the first virtual projection device (112) are displayed on a physical space image (90) indicated by the first image data, and outputs the second image data.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present invention relates to an image processing device, an image processing method and an image processing program. Background Art

[0002] Patent document 1 describes a projection image adjustment system, which stores virtual environment setting information indicating the setting state of a projector set in a virtual space in a manner to obtain a desired image projection state for a projection object and a control setting value of the projector at this time, obtains real environment setting information indicating the setting state of the projector in a real space, controls the operation of the projector, and corrects the control setting value in a manner that the difference between the projection state of an image in the real space and the desired image projection state disappears based on the virtual environment setting information and the real environment setting information, and controls the operation based on the corrected control setting value.

[0003] Patent document 2 describes an image processing device that uses a camera image of an object to obtain relevant information related to the object, generates a related image generated from the relevant information, generates an overlapping image in which the related image is overlapped on a camera image containing the object, and projects the generated overlapping image.

[0004] Patent document 3 describes an image processing device that obtains an input image generated by photographing a real space using a camera device, outputs an output image of a virtual subject used to overlap and establish an association with a real object projected in the input image to a projection device, projects the output image onto the real object, and controls the projection of the output image based on the projection device according to the position of the real object identified using the input image.

[0005] Previous technical literature

[0006] Patent Literature

[0007] Patent Document 1: Japanese Patent Application Publication No. 2018-005115

[0008] Patent Document 2: Japanese Patent Application Publication No. 2017-073717

[0009] Patent Document 3: Japanese Patent Application Publication No. 2013-235374 Summary of the invention

[0010] One embodiment of the technology according to the present invention provides an image processing device, an image processing method, and an image processing program capable of improving user convenience related to the arrangement of a projection surface or a projection device.

[0011] Means for solving technical problems

[0012] An image processing device according to one embodiment of the present invention comprises a processor, wherein the processor performs the following processing: acquiring first image data obtained by capturing a space by a camera device; generating first virtual projection surface data representing a first virtual projection surface and first virtual projection device data representing a first virtual projection device based on first position data representing a first position in the space and first normal vector data representing a first normal vector of a first surface corresponding to an object existing at the first position in the space; generating second image data representing a second image of the first virtual projection surface and the first virtual projection device displayed on a first image represented by the first image data based on the first image data, the first virtual projection surface data and the first virtual projection device data; and outputting the second image data to an output destination.

[0013] In an image processing method of one aspect of the present invention, a processor of an image processing device performs the following processing: acquiring first image data obtained by photographing a space by a camera device; generating first virtual projection surface data representing a first virtual projection surface and first virtual projection device data representing a first virtual projection device based on first position data representing a first position in the above space and first normal vector data representing a first normal vector of a first surface corresponding to an object existing at the above first position in the above space; generating second image data representing a second image of the above first virtual projection surface and the above first virtual projection device displayed on a first image represented by the first image data based on the above first image data, the above first virtual projection surface data and the above first virtual projection device data; and outputting the above second image data to an output destination.

[0014] An image processing program according to one embodiment of the present invention is used to cause a processor of an image processing device to perform the following processing: acquiring first image data obtained by capturing a space by a camera device; generating first virtual projection surface data representing a first virtual projection surface and first virtual projection device data representing a first virtual projection device based on first position data representing a first position in the space and first normal vector data representing a first normal vector of a first surface corresponding to an object existing at the first position in the space; generating second image data representing a second image of the first virtual projection surface and the first virtual projection device displayed on a first image represented by the first image data based on the first image data, the first virtual projection surface data and the first virtual projection device data; and outputting the second image data to an output destination.

[0015] Effects of the Invention

[0016] According to the present invention, it is possible to provide an image processing device, an image processing method, and an image processing program capable of improving user convenience regarding the arrangement of a projection surface or a projection device. BRIEF DESCRIPTION OF THE DRAWINGS

[0017] Figure 1 1 is a schematic diagram showing an example of the projection device 10 of the placement assistance object based on the image processing device of Embodiment 1.

[0018] Figure 2 Yes means Figure 1 Schematic diagram of an example of the internal structure of the projection unit 1 shown.

[0019] Figure 3 1 is a schematic diagram showing the external appearance of the projection device 10 .

[0020] Figure 4 yes Figure 3 FIG. 1 is a schematic cross-sectional view of the optical unit 106 of the projection device 10 shown.

[0021] Figure 5 1 is a diagram showing an example of the appearance of the image processing device 50 .

[0022] Figure 6 It is a diagram showing an example of the hardware configuration of the image processing device 50 .

[0023] Figure 7 1 is a diagram showing an example of acquisition of the posture of the imaging device of the image processing device 50 .

[0024] Figure 8 This is a diagram showing an example of capturing the first image and acquiring the first position.

[0025] Fig. 9 This is a diagram showing an example of a physical space image represented by first image data obtained by imaging the physical space 70 .

[0026] Fig.10 This is a flowchart showing an example of processing by the image processing device 50 .

[0027] Fig.11 is Fig.10 An example (part 1) of an image displayed by the image processing device 50 during the process shown.

[0028] Fig.12 is Fig.10 An example of an image displayed by the image processing device 50 during the process shown (part 2).

[0029] Fig.13 This is a diagram showing an example of detection of the end of the physical plane where the first virtual projection plane 111 is arranged in the physical space 70 .

[0030] Fig.14 This is a flowchart showing an example of the process of determining the first position 81.

[0031] Fig.15 Yes means Fig.14 FIG. 1 is a diagram showing an example of determination of the first position 81 in the determination process.

[0032] Fig.16 Yes means Fig.14 FIG. 2 is a diagram showing an example of determination of the first position 81 in the determination process.

[0033] Fig.17 This is a flowchart showing an example of the process of determining the size of the first virtual projection plane 111 .

[0034] Fig.18 Yes means Fig.17 FIG. 1 is a diagram showing an example of determination of the size of the first virtual projection plane 111 and the first position 81 in the determination process (part 1).

[0035] Fig.19 Yes means Fig.17 FIG. 2 is a diagram showing an example of determination of the size of the first virtual projection plane 111 and the first position 81 in the determination process (part 2).

[0036] Fig. 20 This is a diagram (part 1) showing an example of an operation unit for moving the first virtual projection plane 111.

[0037] Fig.21 This is a diagram (part 2) showing an example of an operation unit for moving the first virtual projection plane 111.

[0038] Fig. 22 1 is a diagram showing an example of an operation unit for changing the angle of the first virtual projection plane 111 (part 1).

[0039] Fig.23 This is a diagram showing an example of an operation unit for changing the angle of the first virtual projection plane 111 (part 2).

[0040] Fig.24 This is a diagram showing an example of capturing a first image and acquiring a first position and a second position.

[0041] Fig.25 This is a diagram showing an example of the second virtual projection plane based on the second position.

[0042] Fig.26 1 is a diagram showing an example of coordinate axes for movement of the first virtual projection device 112 .

[0043] Fig. 271 is a diagram showing an example of an operation unit for moving the first virtual projection device 112 in the x-axis direction or the z-axis direction.

[0044] Fig.28 FIG. 2 is a diagram showing an example of an operation unit for moving the first virtual projection device 112 in the x-axis direction or the z-axis direction.

[0045] Fig.29 1 is a diagram showing an example of an operation unit for moving the first virtual projection device 112 in the y-axis direction.

[0046] Fig.30 FIG. 2 is a diagram showing an example of an operation unit for moving the first virtual projection device 112 in the y-axis direction.

[0047] Fig.31 This is a diagram showing an example of a physical curved surface on which the projection surface 11 is arranged in the second embodiment.

[0048] Fig.32 This is a diagram showing an example of specifying the second position group.

[0049] Fig.33 This is a diagram showing an example of a first imaginary curved surface that virtually shows the wall 310 . DETAILED DESCRIPTION

[0050] Hereinafter, examples of embodiments of the present invention will be described with reference to the drawings.

[0051] (Implementation Method 1)

[0052] <An example of the projection device 10 to be changed in configuration by the image processing device according to the first embodiment>

[0053] Figure 1 1 is a schematic diagram showing an example of the projection device 10 of the placement assistance object based on the image processing device of Embodiment 1.

[0054] The image processing device of the first embodiment can be used, for example, to assist in the configuration of a projection device 10. The projection device 10 includes a projection unit 1, a control device 4, and an operation receiving unit 2. The projection unit 1 is composed of, for example, a liquid crystal projector or a projector using LCOS (Liquid Crystal On Silicon). In the following, the projection unit 1 is described as a liquid crystal projector.

[0055] The control device 4 is a control device for controlling the projection of the projection device 10. The control device 4 includes a control unit composed of various processors, a communication interface (not shown) for communicating with each unit, and a memory 4a such as a hard disk, SSD (Solid State Drive) or ROM (Read Only Memory), and centrally controls the projection unit 1.

[0056] Various processors serving as the control unit of the control device 4 include a general-purpose processor that executes programs to perform various processes, namely a CPU (Central Processing Unit), a processor whose circuit structure can be changed after manufacturing, such as an FPGA (Field Programmable Gate Array), a programmable logic device (PLD), or an ASIC (Application Specific Integrated Circuit), a processor with a circuit structure specially designed to perform specific processing, such as a dedicated circuit.

[0057] More specifically, the structure of these various processors is a circuit formed by combining circuit elements such as semiconductor elements. The control unit of the control device 4 can be composed of one of the various processors, or a combination of two or more processors of the same type or different types (for example, a combination of multiple FPGAs or a combination of a CPU and an FPGA).

[0058] The operation receiving unit 2 detects instructions from the user by receiving various operations from the user. The operation receiving unit 2 may be a button, a key, a joystick, etc. provided on the control device 4, or a receiving unit for receiving signals from a remote controller for remotely operating the control device 4.

[0059] The projection object 6 is an object such as a screen or a wall having a projection surface on which the projection image is displayed by the projection unit 1. Figure 1 In the example shown, regarding the projection object 6, the projection surface of the projection object 6 is a rectangular plane. Figure 1 The up, down, left, and right directions of the projected object 6 in FIG. 6 are the actual up, down, left, and right directions of the projected object 6 .

[0060] The projection surface 11 indicated by the single-dot chain line is a region of the projection object 6 to which the projection light is irradiated by the projection unit 1. Figure 1 In the example shown, the projection plane 11 is rectangular. The projection plane 11 is a part or the whole of the projectable range that can be projected by the projection unit 1.

[0061] In addition, the projection unit 1, the control device 4 and the operation receiving unit 2 are implemented by, for example, one device (for example, refer to Figure 3 , Figure 4 ). Alternatively, the projection unit 1, the control device 4, and the operation receiving unit 2 may be different devices that cooperate by communicating with each other.

[0062] < Figure 1 The internal structure of the projection unit 1 shown>

[0063] Figure 2 Yes means Figure 1 Schematic diagram of an example of the internal structure of the projection unit 1 shown.

[0064] like Figure 2 As shown, the projection unit 1 includes a light source 21 , a light modulation unit 22 , a projection optical system 23 , and a control circuit 24 .

[0065] The light source 21 includes a light emitting element such as a laser or an LED (Light Emitting Diode), and emits, for example, white light.

[0066] The light modulator 22 is composed of three liquid crystal panels, and the liquid crystal panels modulate the light emitted from the light source 21 and separated into three colors of red, blue, and green by a color separation mechanism (not shown) according to image information to emit images of each color. Red, blue, and green filters can be mounted on the three liquid crystal panels, respectively, and each liquid crystal panel modulates the white light emitted from the light source 21 to emit images of each color.

[0067] Light from the light source 21 and the light modulator 22 enters a projection optical system 23, which is composed of, for example, a relay optical system including at least one lens. Light passing through the projection optical system 23 is projected onto the projection object 6.

[0068] In the projection object 6, the area irradiated by the light of the entire range of the light modulating unit 22 becomes the projectable range that can be projected by the projection unit 1. In the projectable range, the area irradiated by the light actually transmitted from the light modulating unit 22 becomes the projection surface 11. For example, by controlling the size, position and shape of the area through which the light is transmitted in the light modulating unit 22, the size, position and shape of the projection surface 11 are changed within the projectable range.

[0069] The control circuit 24 controls the light source 21, the light modulator 22 and the projection optical system 23 according to the display data input from the control device 4, thereby projecting an image based on the display data onto the projection object 6. The display data input to the control circuit 24 is composed of three: red display data, blue display data and green display data.

[0070] Then, the control circuit 24 changes the projection optical system 23 according to the command input from the control device 4, thereby performing projection of the projection surface 11 (reference Figure 1 Furthermore, the control device 4 can change the projection optical system 23 according to the operation from the user received through the operation receiving unit 2, thereby moving the projection surface 11 of the projection unit 1.

[0071] Furthermore, the projection device 10 has a displacement mechanism that mechanically or optically moves the projection surface 11 while maintaining the image circle of the projection optical system 23. The image circle of the projection optical system 23 is a region where the projection light incident on the projection optical system 23 appropriately passes through the projection optical system 23 from points such as light quantity reduction, color separation, and peripheral bending.

[0072] The shift mechanism is realized by at least one of an optical system shift mechanism that performs optical system shift and an electronic shift mechanism that performs electronic shift.

[0073] The optical system displacement mechanism is, for example, a mechanism for moving the projection optical system 23 in a direction perpendicular to the optical axis (for example, see Figure 3 , Figure 4 ), or a mechanism that moves the light modulator 22 in a direction perpendicular to the optical axis instead of moving the projection optical system 23. Furthermore, the optical system displacement mechanism may combine the movement of the projection optical system 23 and the movement of the light modulator 22.

[0074] The electronic displacement mechanism is a mechanism for performing a simulated displacement of the projection plane 11 by changing the light transmission range in the light modulation unit 22 .

[0075] Furthermore, the projection device 10 may include a projection direction changing mechanism for moving the projection surface 11 together with the image circle of the projection optical system 23. The projection direction changing mechanism is a mechanism for changing the projection direction of the projection unit 1 by mechanically rotating and changing the orientation of the projection unit 1 (for example, see Figure 3 , Figure 4 ).

[0076] <Mechanical Structure of Projection Device 10>

[0077] Figure 3 1 is a schematic diagram showing the external appearance of the projection device 10 . Figure 4 yes Figure 3 FIG. 1 is a schematic cross-sectional view of the optical unit 106 of the projection device 10 shown. Figure 4 Indicates that along Figure 3 The cross section along the plane of the optical path of the light emitted from the main body 101 is shown.

[0078] like Figure 3As shown in FIG. 1 , the projection device 10 includes a main body 101 and an optical unit 106 protruding from the main body 101. Figure 3 In the illustrated structure, the operation receiving unit 2, the control device 4, the light source 21, the light modulation unit 22 and the control circuit 24 of the projection unit 1 are provided in the main body 101. The projection optical system 23 of the projection unit 1 is provided in the optical unit 106.

[0079] The optical unit 106 includes a first member 102 supported by the main body 101 and a second member 103 supported by the first member 102 .

[0080] In addition, the first member 102 and the second member 103 may be an integrated member. The optical unit 106 may be configured to be detachable from the main body 101 (in other words, configured to be replaceable).

[0081] The main body 101 has an opening 15a (see FIG. 15A ) for passing light formed in a portion connected to the optical unit 106. Figure 4 ) frame 15 (reference Figure 4 ).

[0082] like Figure 3 As shown, a light source 21 and a light modulation unit 12 are provided inside the frame 15 of the main body 101. The light modulation unit 12 includes a light modulation unit 22 (see FIG. 1 ) for spatially modulating light emitted from the light source 21 according to input image data to generate an image. Figure 2 ).

[0083] The light emitted from the light source 21 enters the light modulation section 22 of the light modulation unit 12 , is spatially modulated by the light modulation section 22 , and is emitted.

[0084] like Figure 4 As shown, an image formed by the light spatially modulated by the light modulation unit 12 enters the optical unit 106 through the opening 15 a of the frame 15 and is projected onto the projection object 6 as the projection target, so that the observer visually recognizes the image G1 .

[0085] like Figure 4 As shown, the optical unit 106 comprises: a first component 102 having a hollow portion 2A connected to the interior of a main body 101; a second component 103 having a hollow portion 3A connected to the hollow portion 2A; a first optical system 121 and a reflecting component 122 arranged in the hollow portion 2A; a second optical system 31, a reflecting component 32, a third optical system 33 and a lens 34 arranged in the hollow portion 3A; a displacement mechanism 105; and a projection direction changing mechanism 104.

[0086] The first member 102 is a member having a rectangular cross-sectional shape, and the opening 2a and the opening 2b are formed on mutually perpendicular planes. The first member 102 is supported by the main body 101 in a state where the opening 2a is arranged at a position of the main body 101 opposite to the opening 15a. Light emitted from the light modulation unit 22 of the light modulation unit 12 of the main body 101 passes through the opening 15a and the opening 2a and enters the hollow portion 2A of the first member 102.

[0087] The incident direction of light entering the hollow portion 2A from the main body 101 is recorded as direction X1, the direction opposite to direction X1 is recorded as direction X2, and the directions X1 and X2 are collectively recorded as direction X. Figure 4 In the diagram, a direction from the front of the paper toward the inside and the opposite direction are described as directions Z. In the directions Z, a direction from the front of the paper toward the inside is described as direction Z1, and a direction from the inside of the paper toward the front is described as direction Z2.

[0088] Furthermore, the direction perpendicular to the direction X and the direction Z is recorded as the direction Y. In the direction Y, Figure 4 The upward direction is recorded as direction Y1, and Figure 4 The downward direction is recorded as direction Y2. Figure 4 In the example of FIG. 1 , the projection device 10 is arranged so that the direction Y2 becomes a vertical direction.

[0089] Figure 2 The projection optical system 23 shown is composed of a first optical system 121 , a reflecting member 122 , a second optical system 31 , a reflecting member 32 , a third optical system 33 , and a lens 34 . Figure 4 2 shows the optical axis K of the projection optical system 23. The first optical system 121, the reflecting member 122, the second optical system 31, the reflecting member 32, the third optical system 33 and the lens 34 are arranged along the optical axis K in this order from the light modulating unit 22 side.

[0090] The first optical system 121 includes at least one lens, and guides light that is incident from the main body 101 to the first member 102 and travels in the direction X1 to the reflective member 122 .

[0091] The reflecting member 122 reflects the light incident from the first optical system 121 in the direction Y1. The reflecting member 122 is composed of, for example, a reflecting mirror. An opening 2b is formed in the first member 102 on the optical path of the light reflected by the reflecting member 122, and the reflected light passes through the opening 2b toward the hollow portion 3A of the second member 103.

[0092] The second member 103 is a member having a substantially T-shaped cross-sectional shape, and has an opening 3a formed at a position opposite to the opening 2b of the first member 102. Light from the main body 101 that has passed through the opening 2b of the first member 102 enters the hollow portion 3A of the second member 103 through the opening 3a. The cross-sectional shape of the first member 102 or the second member 103 is arbitrary and is not limited to the cross-sectional shape.

[0093] The second optical system 31 includes at least one lens, and guides the light incident from the first member 102 to the reflecting member 32 .

[0094] The reflecting member 32 reflects the light incident from the second optical system 31 in the direction X2 and guides the light to the third optical system 33. The reflecting member 32 is constituted by, for example, a reflecting mirror or the like.

[0095] The third optical system 33 includes at least one lens, and guides the light reflected by the reflecting member 32 to the lens 34 .

[0096] The lens 34 is disposed at the end portion so as to close the opening 3c formed at the end portion on the side in the direction X2 of the second member 103. The lens 34 projects the light incident from the third optical system 33 onto the projection object 6.

[0097] The projection direction changing mechanism 104 is a rotation mechanism that rotatably connects the second member 103 to the first member 102. The projection direction changing mechanism 104 allows the second member 103 to be rotatable about a rotation axis (specifically, the optical axis K) extending along the direction Y. The projection direction changing mechanism 104 is not limited to any other mechanism as long as it can rotate the optical system. Figure 4 Furthermore, the number of the rotating mechanisms is not limited to one, and a plurality of rotating mechanisms may be provided.

[0098] The displacement mechanism 105 is used to move the optical axis K of the projection optical system (in other words, the optical unit 106) along a direction perpendicular to the optical axis K thereof ( Figure 4 Specifically, the displacement mechanism 105 is configured to be able to change the position of the first component 102 relative to the main body 101 in the direction Y. In addition to manually moving the first component 102, the displacement mechanism 105 can also electrically move the first component 102.

[0099] Figure 4 This shows the state where the first component 102 is moved to the maximum extent in the direction Y1 by the displacement mechanism 105. Figure 4In the state shown, the first component 102 is moved in the direction Y2 through the displacement mechanism 105, thereby changing the relative position of the center of the image formed by the light modulator 22 (in other words, the center of the display surface) and the optical axis K, so that the image G1 projected onto the projection object 6 can be displaced in the direction Y2 (parallel movement).

[0100] Alternatively, the displacement mechanism 105 may be a mechanism that moves the light modulator 22 in the direction Y instead of moving the optical unit 106 in the direction Y. In this case, the image G1 projected onto the object 6 can also be moved in the direction Y2.

[0101] <Appearance of Image Processing Device 50>

[0102] Figure 5 1 is a diagram showing an example of the appearance of the image processing device 50. The image processing device 50 is a tablet terminal having a touch panel 51. The touch panel 51 is a display that can be operated by touch. The image processing device 50 displays a setting auxiliary image for assisting the setting of the projection device 10 in the space through the touch panel 51.

[0103] Specifically, the image processing device 50 displays a second image as an installation auxiliary image, in which a first virtual projection plane and a first virtual projection device are superimposed on a first image obtained by photographing a space where the projection device 10 is installed and projecting.

[0104] <Hardware Configuration of Image Processing Device 50>

[0105] Figure 6 FIG. 5 is a diagram showing an example of the hardware configuration of the image processing device 50. Figure 6 As shown, Figure 5 The image processing device 50 shown includes a processor 61, a memory 62, a communication interface 63, a user interface 64, and a sensor 65. The processor 61, the memory 62, the communication interface 63, the user interface 64, and the sensor 65 are connected via a bus 69, for example.

[0106] The processor 61 is a circuit that performs signal processing, and is, for example, a CPU that manages the overall control of the image processing device 50. The processor 61 may be implemented by other digital circuits such as an FPGA or a DSP (Digital Signal Processor). Furthermore, the processor 61 may be implemented by combining a plurality of digital circuits.

[0107] The memory 62 includes, for example, a main memory and an auxiliary memory. The main memory is, for example, a RAM (Random Access Memory). The main memory is used as a work area for the processor 61.

[0108] The auxiliary memory is, for example, a nonvolatile memory such as a magnetic disk or a flash memory. Various programs for operating the image processing device 50 are stored in the auxiliary memory. The programs stored in the auxiliary memory are loaded into the main memory and executed by the processor 61.

[0109] Furthermore, the auxiliary memory may include a portable memory that is detachable from the image processing apparatus 50. The portable memory includes a memory card such as a USB (Universal Serial Bus) flash drive or an SD (Secure Digital) memory card, or an external hard disk drive.

[0110] The communication interface 63 is a communication interface for communicating with an external device of the image processing device 50 . The communication interface 63 includes at least one of a wired communication interface for communicating by wire and a wireless communication interface for communicating by wireless. The communication interface 63 is controlled by the processor 61 .

[0111] The user interface 64 includes, for example, an input device for receiving operation input from the user or an output device for outputting information to the user. The input device can be implemented, for example, by a key (such as a keyboard) or a remote controller. The output device can be implemented, for example, by a display or a speaker. Figure 5 In the image processing apparatus 50 shown, the input device and the output device are realized by the touch panel 51. The user interface 64 is controlled by the processor 61. The image processing apparatus 50 uses the user interface 64 to receive various specifications from the user.

[0112] The sensor 65 includes an imaging device that has an imaging optical system and an imaging element and can capture images, or a space recognition sensor that can three-dimensionally recognize the surrounding space of the image processing device 50. The imaging device includes, for example, a device provided at Figure 5 The image processing device 50 shown is a camera device on the back side.

[0113] An example of a space recognition sensor is LIDAR (Light Detection and Ranging), which measures the distance or direction to an object by irradiating a laser and measuring the time it takes for the irradiated laser to hit the object and bounce back. However, the space recognition sensor is not limited to this, and can be a variety of sensors such as a radar that emits radio waves or an ultrasonic sensor that emits ultrasonic waves.

[0114] <Acquisition of Posture of Imaging Device by Image Processing Device 50>

[0115] Figure 7 1 is a diagram showing an example of acquiring the posture of the imaging device of the image processing device 50. Figure 7As shown, for example, the user of the image processing device 50 brings the image processing device 50 into a physical space 70 (for example, a room) which is a physical space of the installation target of the projection device 10. Figure 7 In the example of FIG. 7 , in the physical space 70 , there are at least a bed 71 and a wall 72 as physical planes.

[0116] The image processing device 50 always obtains the posture (position and orientation) of the camera of the image processing device 50 in a three-dimensional orthogonal coordinate system with one point in the physical space 70 (for example, the position where the camera of the image processing device 50 is started) as the origin, the horizontal direction as the X axis, the gravity direction as the Y axis, and the remaining axis as the Z axis. In addition, the image processing device 50 displays a captured image based on the captured image data obtained by shooting with the camera to the user as a live view image (live preview) through the touch panel 51.

[0117] <Capturing the first image and acquiring the first position>

[0118] Figure 8 1 is a diagram showing an example of capturing the first image and acquiring the first position. Here, it is assumed that the user indicates the first position 81 near the center of the wall 72 as the position (center position) of the projection surface 11 where the projection device 10 is arranged. In this case, Figure 8 As shown, the user holds the image processing device 50 at the position and orientation where the first position 81 is displayed on the touch panel 51 .

[0119] Then, the user points to the first position 81 in the physical space 70 by performing an indication operation (e.g., a click operation) on the first position 81 of the wall 72 (the position 51a of the touch panel 51) displayed on the touch panel 51. Thus, the image processing device 50 can obtain the information indicating the first position 81 in the physical space 70. Figure 7 The first position data of the first position 81 in the three-dimensional orthogonal coordinate system shown.

[0120] The first normal vector 82 is the same as that existing in Figure 7 The normal vector of the first surface corresponding to the wall 72 at the first position 81 of the physical space 70 in the three-dimensional orthogonal coordinate system shown. The image processing device 50 obtains first normal vector data representing the first normal vector 82 based on the result of recognizing the physical space 70 by the space recognition sensor.

[0121] <Physical Space Image Represented by First Image Data Obtained by Imaging Physical Space 70>

[0122] Fig. 9 is a diagram showing an example of a physical space image represented by the first image data obtained by photographing the physical space 70. Figure 8In the state shown, the user instructs the image processing device 50 to shoot through the structural diagram of the first position 81 projected on the touch panel 51. As a result, the image processing device 50 can obtain the first image data obtained by shooting the physical space 70 including the first position 81. The first image data is data representing a physical space image 90 in which the physical space 70 is projected. The physical space image 90 is an example of the first image of the present invention.

[0123] Thus, the image processing device 50 can obtain the first image data obtained by photographing the physical space 70, the first position data indicating the first position 81 in the physical space 70, and the first normal vector data indicating the first normal vector 82 of the first surface corresponding to the object existing at the first position 81 in the physical space 70. For example, the image processing device 50 stores the first image data indicating the first position 81 in the physical space 70. Figure 7 The first position data and the first normal vector data of the first position 81 and the first normal vector 82 expressed by the three-dimensional orthogonal coordinate system described in the description. In this case, the image processing device 50 also saves, in addition to these data, data indicating the position of the image processing device 50 when the physical space 70 is photographed to obtain the first image data. Alternatively, when photographing the physical space 70, the image processing device 50 saves the first position data and the first normal vector data indicating the first position 81 and the first normal vector 82 expressed by the three-dimensional orthogonal coordinate system centered on the image processing device 50 based on the posture of the image processing device 50 always obtained.

[0124] <Processing by Image Processing Device 50>

[0125] Fig.10 This is a flowchart showing an example of processing by the image processing device 50 . Figure 11-12 is Fig.10 An example of an image displayed by the image processing device 50 during the process shown in FIG. Fig.10 When the processing shown starts, as in Figure 7 to Figure 9 As described in the figure, the image processing device 50 obtains the first image data obtained by photographing the physical space 70, the first position data representing the first position 81 in the physical space 70, and the first normal vector data representing the first normal vector 82 of the first surface corresponding to the wall 72 existing in the first position 81 in the physical space 70.

[0126] First, the image processing device 50 receives the designation of the size of the first virtual projection plane from the user (step S101). The size of the first virtual projection plane is designated using an actual distance in the physical space 70 such as the length of the diagonal of the rectangular first virtual projection plane = x [inches].

[0127] Next, the image processing device 50 displays the physical space image 90 represented by the first image data on the first virtual projection plane based on the size of the first virtual projection plane received in step S101 and the first position data and the first normal vector data (step S102). Fig.11 As shown, the image processing device 50 displays an image in which the first virtual projection plane 111 is superimposed on the physical space image 90 .

[0128] Specifically, the image processing device 50 generates a first virtual projection plane 111 in the physical space image 90, which is centered at the first position 81 represented by the first position data, is perpendicular to the first normal vector 82 represented by the first normal vector data, and is adjusted in a manner visible on a projection plane of a specified size, and displays the generated first virtual projection plane 111 superimposed on the physical space image 90. Fig.11 Although the first position 81 and the first normal vector 82 are shown in the figure, the first position 81 and the first normal vector 82 may not be actually displayed.

[0129] Next, the image processing device 50 receives the designation of the model of the first virtual projection device from the user from a plurality of options of the model of the projection device 10 (step S103). Next, the image processing device 50 calculates the first projection distance, which is the distance between the first virtual projection device and the first virtual projection plane 111, based on the size of the first virtual projection plane 111 and the projection ratio that can be set by the model designated as the model of the first virtual projection device received in step S103 (step S104).

[0130] Next, the image processing device 50 displays the first virtual projection device on the physical space image 90 in an overlapping manner according to the model of the first virtual projection device designated by step S103 and the first projection distance calculated by step S104 (step S105). Fig.12 As shown, the image processing device 50 displays the first virtual projection device 112 , which is a three-dimensional model of the model designated as the model of the first virtual projection device, superimposed on the physical space image 90 .

[0131] Specifically, the image processing device 50 generates a first virtual projection device 112 whose shape is adjusted so as to appear to be arranged at a position separated by a first projection distance from the center (first position 81) of the first virtual projection plane 111 in the direction of the first normal vector 82 in the physical space image 90, and displays the generated first virtual projection device 112 superimposed on the physical space image 90. Fig.12 Although the first position 81 and the first normal vector 82 are shown in the figure, the first position 81 and the first normal vector 82 may not be actually displayed.

[0132] In this way, the image processing device 50 generates second image data representing a second image of the first virtual projection plane 111 and the first virtual projection device 112 superimposed on the first image (physical space image 90) represented by the first image data based on the first image data, the first virtual projection plane data and the first virtual projection device data, and displays the second image based on the second image data.

[0133] Thus, by obtaining the first position data, the first normal vector data and the physical space image 90 (first image) in the physical space 70, the user can visually grasp the size and configuration of the projection surface 11 based on the physical plane (wall 72) of the physical space 70 or the positional relationship between the projection surface 11 and the projection device 10 on the physical space image 90 representing the physical space 70, even at a position other than the physical space 70 of the scene projected by the projection device 10.

[0134] Furthermore, since it is sufficient to obtain the first position data and the first normal vector data as the spatial data representing the physical space 70, it is not necessary to obtain the detailed three-dimensional data of the physical space 70, and thus it is possible to reduce the amount of data of the object to be stored.

[0135] In generating the second image data, specifically, the image processing device 50 generates first virtual projection plane data based on the first position data and the first normal vector data, and generates first virtual projection device data based on the generated first virtual projection plane data.

[0136] Furthermore, the image processing device 50 determines the normal vector of the first virtual projection plane 111 based on the first normal vector 82 represented by the first normal vector data. For example, the image processing device 50 generates the first virtual projection plane 111 in such a way that the direction of the normal vector of the first virtual projection plane 111 is consistent with the direction of the first normal vector 82. In addition, in the present application, "consistent" is not limited to completely consistent, but also includes approximately consistent.

[0137] Then, the image processing device 50 determines the projection direction and position of the first virtual projection device 112 indicated by the first virtual projection device data based on the position and size of the first virtual projection plane 111 .

[0138] Furthermore, the image processing device 50 determines the first position data and the first normal vector data based on the distance data (first projection distance) between the object (wall 72) and the camera (camera of the image processing device 50) obtained by the space recognition sensor.

[0139] <Another example of obtaining the first position>

[0140] exist Figure 8, a configuration is described in which the user indicates the first position 81 in the physical space 70 by performing an indication operation (e.g., a click operation) on the first position 81 of the wall 72 displayed on the touch panel 51, but the present invention is not limited to this configuration. For example, when the image processing device 50 is photographing the physical space 70, if the end of the physical plane in which the first virtual projection surface 111 is configured in the physical space 70 is detected, the first position 81 may be determined based on the position of the detected end.

[0141] <Detection of the End of the Physical Plane Arranging the First Virtual Projection Plane 111 in the Physical Space 70>

[0142] Fig.13 1 is a diagram showing an example of detecting the end of a physical plane in which the first virtual projection plane 111 is arranged in the physical space 70. Fig.13 In the example, there is a wall 73 in the physical space 70. The wall 73 is a wall perpendicular to the floor 71 and the wall 72. For example, when the physical plane on which the first virtual projection plane 111 is arranged in the physical space 70 is the wall 72, the image processing device 50 detects Fig.13 Ends 72a-72d are shown.

[0143] The end 72a is the right end of the wall 72 (the boundary with the wall 73). The end 72b is the upper end of the wall 72. The end 72c is the left end of the wall 72. The end 72d is the lower end of the wall 72.

[0144] The detection of the ends 72a to 72d can be performed, for example, based on image recognition processing based on image data obtained by imaging by an imaging device of the image processing device 50 or on the recognition result of a space recognition sensor of the image processing device 50.

[0145] <Determination process of the first position 81>

[0146] Fig.14 This is a flowchart showing an example of the process of determining the first position 81. Fig.15 and Fig.16 Yes means Fig.14 The image processing device 50 displays a physical space image 90 represented by the first image data obtained by photographing the physical space 70 to the user via the touch panel 51, and performs, for example, Fig.14 Processing shown.

[0147] First, the image processing device 50 receives from the user a designation of a physical plane (wall 72) on which the first virtual projection plane 111 is arranged in the physical space 70 and the size of the first virtual projection plane 111 (step S141). For example, the user designates the wall 72 in the physical space 70 by performing an indication operation (e.g., a click operation) on the wall 72 displayed on the touch panel 51.

[0148] Next, the image processing device 50 detects the end of the physical plane (wall 72) received from the user in step S141 (step S142). Fig.13 As shown, the image processing device 50 detects the ends 72 a to 72 d of the wall 72 .

[0149] Next, the image processing device 50 receives from the user a designation of one or more ends for determining the first position 81 among the ends of the physical plane (wall 72) detected in step S142 (step S143). For example, the image processing device 50 displays the detected ends 72a to 72d of the wall 72 as candidates on the touch panel 51, and receives the designation of the end from the user by a click operation or the like.

[0150] Next, the image processing device 50 determines whether the first position 81 can be determined such that the side of the first virtual projection plane 111 is in contact with all the ends received in step S143 based on the size of the first virtual projection plane 111 received in step S141 (step S144 ).

[0151] For example, it is assumed that the end portions 72a to 72c of the wall 72 are designated by the user in step S143. Also, it is assumed that the size (e.g., width) of the first virtual projection plane 111 designated in step S141 is different from the size (e.g., width) of the wall 72. In this case, Fig.15 As shown in FIG. 1 , the first position 81 cannot be determined in such a manner that the side of the first virtual projection plane 111 of the size specified in step S141 is in contact with all the end portions 72a to 72c. Fig.15 , an example is shown in which the first position 81 is determined so that the side of the first virtual projection plane 111 is in contact with only the end 72b among the end portions 72a to 72c.

[0152] In step S144, when the first position 81 cannot be determined (step S144: No), the image processing device 50 outputs information to the user urging the user to exclude a portion of the ends designated as the ends used to determine the first position 81, and receives the designation of the excluded ends (step S145).

[0153] Then, the image processing device 50 returns to step S144, excludes the end specified in step S145, and determines again whether the first position 81 can be determined in such a way that the side of the first virtual projection plane 111 is in contact with all the ends. For example, it is assumed that the end 72c of the end portions 72a to 72c of the wall 72 is excluded. In this case, for example, Fig.16 As shown, the first position 81 can be determined so that the side of the first virtual projection plane 111 is in contact with all the designated end portions 72a and 72b.

[0154] In step S144, if the first position 81 can be determined (step S144: Yes), the image processing device 50 determines the first position 81 in such a way that the side of the first virtual projection plane 111 is in contact with all the designated ends (step S146), and ends a series of processing. Fig.16 The first position 81 shown in FIG. 1 can be easily determined by this configuration so that the first virtual projection plane 111 is close to the end of the wall 72 .

[0155] <Another example of obtaining the size of the first virtual projection plane 111>

[0156] The structure for the user to indicate the size of the first virtual projection surface 111 has been described, but is not limited to this structure. For example, when the image processing device 50 is shooting the physical space 70, it can detect the end of the physical plane in which the first virtual projection surface 111 is configured in the physical space 70, and determine the size of the first virtual projection surface 111 according to the position of the detected end.

[0157] <Determination of the size of the first virtual projection plane 111>

[0158] Fig.17 This is a flowchart showing an example of the process of determining the size of the first virtual projection plane 111 . Fig.18 and Fig.19 Yes means Fig.17 The image processing device 50 displays the physical space image 90 represented by the first image data obtained by photographing the physical space 70 to the user through the touch panel 51, for example, by executing Fig.17 Processing shown.

[0159] First, the image processing device 50 receives from the user a designation of a physical plane (wall 72) on which the first virtual projection plane 111 is arranged in the physical space 70 (step S171). For example, the user designates the wall 72 in the physical space 70 by performing an indication operation (e.g., a click operation) on the wall 72 displayed on the touch panel 51.

[0160] Next, the image processing device 50 detects the end of the physical plane (wall 72) received from the user in step S171 (step S172). Fig.13 As shown, the image processing device 50 detects the ends 72 a to 72 d of the wall 72 .

[0161] Next, the image processing device 50 receives from the user a designation of one or more ends for determining the size of the first virtual projection plane 111 and the first position 81 among the ends of the physical plane (wall 72) detected in step S172 (step S173). For example, the image processing device 50 displays the detected ends 72a to 72d of the wall 72 as selection candidates on the touch panel 51, and receives the designation of the end from the user by a click operation or the like.

[0162] Next, the image processing device 50 determines whether the size of the first virtual projection plane 111 and the first position 81 can be determined so that the side of the first virtual projection plane 111 is appropriately positioned relative to the end designated in step S173 (step S174 ).

[0163] For example, it is assumed that the end portions 72a to 72d of the wall 72 are designated by the user in step S173. Then, for example, Fig.18 As shown, when the size (e.g., the length of the diagonal line) and the first position 81 of the first virtual projection plane 111 are determined so that the right and upper sides of the first virtual projection plane 111 are in contact with the ends 72a and 72b, the left and lower sides of the first virtual projection plane 111 are not in contact with the ends 72c and 72d. In this case, the image processing device 50 determines that the size and the first position 81 of the first virtual projection plane 111 cannot be determined so that the sides of the first virtual projection plane 111 are in appropriate positions relative to the ends 72a to 72c.

[0164] In step S174, when the size of the first imaginary projection plane 111 and the first position 81 cannot be determined (step S174: No), the image processing device 50 outputs a message urging the user to specify the positional relationship with the edge of the first imaginary projection plane 111 for a part of the ends designated in step S173, and receives a designation of the positional relationship with the edge of the first imaginary projection plane 111 from the user (step S175).

[0165] Then, the image processing device 50 returns to step S174, and based on the positional relationship specified by step S175, determines again whether it is possible to determine the size and the first position 81 of the first virtual projection plane 111 in such a way that the edge of the first virtual projection plane 111 is in an appropriate position relative to the end portion specified by step S173.

[0166] For example, assume that in step S175, the positional relationship of the ends 72a and 72c being located inside the left and right sides of the first virtual projection plane is specified. Fig.19 As shown, by making the left and right sides of the first imaginary projection plane 111 outside the end parts 72a and 72c, the size and the first position 81 of the first imaginary projection plane 111 can be determined in such a way that the upper and lower sides of the first imaginary projection plane 111 are connected to the end parts 72b and 72d.

[0167] In step S174, if the size of the first virtual projection plane 111 and the first position 81 can be determined (step S174: Yes), the image processing device 50 determines the size of the first virtual projection plane 111 and the first position 81 so that the side of the first virtual projection plane 111 is in an appropriate position relative to the specified end (step S176), and ends a series of processing. Fig.19 The size of the first virtual projection plane 111 and the first position 81 are shown. As a result, the first position 81 where the first virtual projection plane 111 is close to the end of the wall 72 can be easily determined.

[0168] In addition, in step S175, the image processing device 50 may prompt the user on how to set the positional relationship between the end of the wall 72 and the edge of the first imaginary projection plane 111 in determining the size of the first imaginary projection plane 111 and the first position 81, so as to urge the user to exclude the designation of the end of the wall 72 or the designation of the positional relationship.

[0169] Thus, the size of the first virtual projection plane 111 and the size of the first virtual projection plane 111 that can bring the first position 81 closer to the end of the wall 72 can be easily determined.

[0170] like Figures 14 to 18 As shown, the image processing device 50 can determine the position of the end of the first surface (wall 72) in the specific physical space image 90 (first image) based on the first image data, and determine at least one of the position and size of the first imaginary projection plane 111 based on the position of the specific end.

[0171] <Operation Section for Moving First Virtual Projection Plane 111>

[0172] Fig. 20 and Fig.21 1 is a diagram showing an example of an operation unit for moving the first virtual projection plane 111. The image processing device 50 may, for example, Fig.10 The processing shown in Fig.12In a state where the first virtual projection plane 111 and the first virtual projection device 112 are superimposed and displayed on the physical space image 90 by the touch panel 51 as shown, Fig. 20 The first virtual projection plane operation unit 201 is shown. Fig. 20 , Fig.21 In the example of FIG. 1 , the first virtual projection plane operation unit 201 is an image of up, down, left, and right cursor keys, and can instruct the up, down, left, and right movement of the first virtual projection plane 111 by a touch operation.

[0173] For example, Fig.21 As shown, when the right cursor key of the first virtual projection plane operation unit 201 is touched, the image processing device 50 moves the overlapping position of the first virtual projection plane 111 and the first virtual projection device 112 with respect to the physical space image 90 to the right.

[0174] Specifically, the image processing device 50 changes the first position 81 according to the operation of the first virtual projection plane operation unit 201. Then, the image processing device 50 performs, for example, Fig.10 The same processing as steps S102 and S105 shown above is performed to superimpose and display the first virtual projection plane 111 and the first virtual projection device 112 corresponding to the changed first position 81 on the physical space image 90 .

[0175] <Operation section for changing the angle of the first virtual projection plane 111>

[0176] Fig. 22 and Fig.23 1 is a diagram showing an example of an operation unit for changing the angle of the first virtual projection plane 111. For example, the image processing device 50 may Fig.10 The processing shown in Fig.12 In a state where the first virtual projection plane 111 and the first virtual projection device 112 are superimposed and displayed on the physical space image 90 by the touch panel 51 as shown, Fig. 22 The first virtual projection plane operating unit 221 is shown.

[0177] exist Fig. 22 , Fig.23 In the example of , the first virtual projection plane operation unit 221 is an image of four curved cursor keys, which can instruct the change of the angle of the first virtual projection plane 111 through touch operation. The four curved cursor keys are used to instruct rotation in a first rotation direction centered on the horizontal axis, rotation in a second rotation direction opposite to the first rotation direction, rotation in a third rotation direction centered on the vertical axis, and rotation in a fourth rotation direction opposite to the third rotation direction.

[0178] For example, Fig. 22As shown, if a touch operation is performed on any of the curved cursor keys of the first virtual projection plane operating unit 221, the image processing device 50 changes the form of the first virtual projection plane 111 and the first virtual projection device 112 superimposed on the physical space image 90 so that the angles of the first virtual projection plane 111 and the first virtual projection device 112 appear to have changed in the physical space image 90.

[0179] Specifically, the image processing device 50 changes the first normal vector 82 according to the operation of the first virtual projection plane operation unit 221. Then, the image processing device 50 performs, for example, Fig.10 The same processing as steps S102 and S105 shown above is performed to superimpose and display the first virtual projection plane 111 and the first virtual projection device 112 corresponding to the changed first position 81 on the physical space image 90 .

[0180] Furthermore, the image processing device 50 can display Fig. 20 and Fig.21 The first virtual projection plane operation unit 201 and Fig. 22 and Fig.23 The first virtual projection plane operating unit 221 shown in the figure is used to change both the position and the angle of the first virtual projection plane 111 .

[0181] like Figure 20 to Figure 23 As shown, the image processing device 50 can change the first virtual projection plane 111 superimposed on the physical space image 90 based on the first input data related to the change of at least one of the first position 81 and the first normal vector 82 (for example, data based on the operation of the first virtual projection plane operation unit 201 or the first virtual projection plane operation unit 221). In addition, the image processing device 50 can change the first virtual projection device 112 superimposed on the physical space image 90 based on the change of the first virtual projection plane 111 superimposed on the physical space image 90.

[0182] Thus, the user can adjust the position or angle of the first virtual projection plane 111 , and can visually grasp the size and arrangement of the projection plane 11 in a desired projection state, and the positional relationship between the projection plane 11 and the projection device 10 .

[0183] <Receiving instruction to change the lens shift amount>

[0184] For example, the image processing device 50 may Fig.10 The processing shown in Fig.12As shown, the first virtual projection plane 111 and the first virtual projection device 112 are displayed superimposed on the physical space image 90 by the touch panel 51, and an instruction to change the lens shift amount is received from the user within the range set by the model designated as the model of the first virtual projection device 112.

[0185] When the image processing device 50 receives an indication of a change in the lens shift amount, it generates, in the physical space image 90, a first virtual projection device 112 whose shape is adjusted in such a way that the device appears to be arranged at a position separated by a first projection distance from the center of the first virtual projection plane 111 (the first position 81) in the direction of the first normal vector 82 and is set with a changed lens shift amount, and displays the generated first virtual projection device 112 in an overlapping manner on the physical space image 90.

[0186] In this way, the image processing device 50 can change the first virtual projection device 112 superimposed on the physical space image 90 according to the second input data (for example, data based on the operation of the touch panel 51) related to the change of the displacement amount of the projection lens of the first virtual projection device 112. As a result, the user can visually grasp the size and arrangement of the projection surface 11 when the lens displacement amount is set for the projection device 10, and the positional relationship between the projection surface 11 and the projection device 10.

[0187] <Capturing the first image and acquiring the first and second positions>

[0188] Fig.24 This is a diagram showing an example of capturing a first image and acquiring a first position and a second position. Fig.25 : is a diagram showing an example of the second virtual projection plane based on the second position. Figure 8 In the above description, the case where the first position 81 where the projection surface 11 is arranged is acquired is described. However, the image processing device 50 may also acquire the second position where the projection device 10 is arranged.

[0189] Here, it is assumed that the user indicates the first position 81 of the wall 72 as the position where the projection surface 11 is arranged, and indicates the second position 241 of the floor 71 as the position where the projection device 10 is arranged. Fig.24 As shown, the user holds the image processing device 50 at the position and orientation where the first position 81 and the second position 241 are displayed on the touch panel 51 .

[0190] Then, the user indicates the first position 81 in the physical space 70 by performing an indication operation (e.g., a click operation) on the first position 81 of the wall 72 displayed on the touch panel 51 (the position 51a of the touch panel 51). Furthermore, the user indicates the second position 241 in the physical space 70 by performing an indication operation (e.g., a click operation) on the second position 241 of the floor 71 displayed on the touch panel 51 (the position 51b of the touch panel 51). Thus, the image processing device 50 can obtain the image representing the first position 81 in the physical space 70. Figure 7 The first position data indicating the first position 81 and the second position data indicating the second position 241 in the three-dimensional orthogonal coordinate system shown.

[0191] The second normal vector 242 is the same as that existing in Figure 7 The normal vector of the second surface corresponding to the object at the second position 241 of the physical space 70 in the three-dimensional orthogonal coordinate system shown in FIG. The image processing device 50 obtains the second normal vector data representing the second normal vector 242 based on the result of the physical space 70 being recognized by the space recognition sensor. Thus, the image processing device 50 can obtain the first normal vector data representing the first normal vector 82 and the second normal vector data representing the second normal vector 242.

[0192] Then, the image processing device 50 executes Fig.10 The image processing device 50 is Fig.10 In step S105 shown in FIG. 1 , according to the second position data and the second normal vector data, as shown in FIG. Fig.25 Then, the image processing device 50 generates, in the physical space image 90, a first virtual projection device 112 whose shape is adjusted in such a manner that the first virtual projection device 112 is spaced apart from the first virtual projection plane 111 in the direction of the first normal vector 82 by the first projection distance (distance D1) and the bottom surface is in contact with the virtual plane 251, and displays the generated first virtual projection device 112 on the physical space image 90 in an overlapping manner.

[0193] <Coordinate Axis of Movement of First Virtual Projection Device 112>

[0194] Fig.26 It is a diagram showing an example of coordinate axes for movement of the first imaginary projection device 112. With the first imaginary projection device 112 as a reference, a three-dimensional orthogonal coordinate system is defined with the axis perpendicular to the bottom surface (imaginary plane 251) of the first imaginary projection device 112 as the y-axis, the left-right direction of the first imaginary projection device 112 as the x-axis, and the remaining axis (the front-back direction of the first imaginary projection device 112) as the z-axis.

[0195] <Operation Unit for Moving the First Virtual Projection Device 112 in the X-Axis Direction or the Z-Axis Direction>

[0196] Fig. 27 and Fig.28 1 is a diagram showing an example of an operation unit for moving the first virtual projection device 112 in the x-axis direction or the z-axis direction. Fig.10 The processing shown in Fig.12 In a state where the first virtual projection plane 111 and the first virtual projection device 112 are superimposed and displayed on the physical space image 90 by the touch panel 51 as shown, Fig. 27 The first virtual projection device operating unit 271 is shown.

[0197] exist Fig. 27 , Fig.28 In the example of FIG. 1 , the first virtual projection device operation unit 271 is an image of cursor keys for indicating the front, back, left, and right directions, and can instruct the front, back, left, and right directions (z-axis and x-axis) of the first virtual projection device 112 by touch operation.

[0198] For example, Fig.28 As shown in FIG. 1 , when the right cursor key of the first virtual projection device operation unit 271 is touched, the image processing device 50 moves the superimposed position of the first virtual projection device 112 with respect to the physical space image 90 to the right.

[0199] Specifically, the image processing device 50 generates the first virtual projection device 112 whose form is adjusted so that the first virtual projection device 112 appears to be arranged at a position shifted rightward from the original position in the physical space image 90 , and displays the generated first virtual projection device 112 superimposed on the physical space image 90 .

[0200] Furthermore, if the first virtual projection device 112 is moved along the z-axis direction, the first projection distance, which is the distance between the first virtual projection device and the first virtual projection plane 111, changes. In contrast, the image processing device 50 recalculates the size of the first virtual projection plane 111 based on the changed first projection distance, and displays the first virtual projection plane 111 of the recalculated size in an overlapping manner on the physical space image 90.

[0201] <Operation Unit for Moving the First Virtual Projection Device 112 in the Y-Axis Direction>

[0202] Fig.29 and Fig.30 1 is a diagram showing an example of an operation unit for moving the first virtual projection device 112 in the y-axis direction. Fig.10 The processing shown in Fig.12In a state where the first virtual projection plane 111 and the first virtual projection device 112 are superimposed and displayed on the physical space image 90 by the touch panel 51 as shown, Fig.29 The first virtual projection device operating unit 291 is shown.

[0203] exist Fig.29 , Fig.30 In the example of FIG. 1 , the first virtual projection device operating unit 291 is an image of a cursor key for designating up and down directions, and can designate up and down (y-axis direction) movement of the first virtual projection device 112 by a touch operation.

[0204] For example, Fig.30 As shown in FIG. 1 , when the cursor key in front of the first virtual projection device operating unit 291 is touched, the image processing device 50 moves the position of the first virtual projection device 112 upward relative to the physical space image 90 .

[0205] Specifically, in the physical space image 90 , the first virtual projection device 112 whose shape is adjusted so as to appear to be arranged at a position where the first virtual projection device 112 is moved forward from its original position is generated, and the generated first virtual projection device 112 is displayed superimposed on the physical space image 90 .

[0206] like Figures 24 to 30 As shown, the image processing device 50 can generate the first virtual projection device data based on the first virtual projection surface data, the second position data representing the second position 241 in the physical space 70 that is different from the first position 81, and the second normal vector data representing the second normal vector 242 of the second surface corresponding to the object (floor 71) existing in the second position 241 in the physical space 70.

[0207] (Implementation Method 2)

[0208] Regarding the second embodiment, the parts different from the first embodiment will be described.

[0209] <Physical Curved Surface Arranged on Projection Surface 11 in Embodiment 2>

[0210] Fig.31 This is a diagram showing an example of a physical curved surface on which the projection surface 11 is arranged in the second embodiment. Fig.31 The top view 301 and the top view 302 shown in the figure show the top view and the top view of the wall 310, which is the physical curved surface on which the projection surface 11 of the projection device 10 is arranged. Here, it is assumed that the user indicates the first position 311 near the center of the wall 310 as the position (center position) of the projection surface 11 of the projection device 10. In this case, Fig.31 As shown, the user holds the image processing device 50 at the position and orientation where the first position 311 is displayed on the touch panel 51 .

[0211] Then, the user points to the first position 311 in the physical space 70 by performing a pointing operation (e.g., a tapping operation) on the first position 311 of the wall 310 (position 51c of the touch panel 51) displayed on the touch panel 51. Thus, the image processing device 50 can acquire first position data indicating the first position 311.

[0212] The first normal vector 312 is a normal vector corresponding to the first position 311 of the first surface corresponding to the wall 310 which is an object existing at the first position 311 in the physical space 70. The image processing device 50 acquires first normal vector data representing the first normal vector 312 based on the result of recognizing the physical space 70 by the space recognition sensor.

[0213] <Specification of the second position group>

[0214] Fig.32 3 is a diagram showing an example of designation of the second position group. Furthermore, the image processing device 50 receives from the user an instruction of a second position group sufficient for roughly reproducing the shape of the wall 310. The receipt of the instruction of the second position group is similar to the receipt of the instruction of the second position group in the Fig.31 The reception of the indication of the first position 311 described in is performed in the same manner.

[0215] Set to Fig.32 In the example of , the second positions 321a to 321d are specified as the second position group. The second normal vectors 322a to 322d are normal vectors of the first surface corresponding to the wall 310 that correspond to the second positions 321a to 321d, respectively. The image processing device 50 obtains the second normal vector data group representing the second normal vectors 322a to 322d based on the result of the physical space 70 being recognized by the space recognition sensor.

[0216] <Hypothetically showing the first imaginary curved surface of the wall 310>

[0217] Fig.33 3 is a diagram showing an example of a first imaginary curved surface that virtually shows the wall 310. The image processing device 50 constructs a first imaginary curved surface 330 based on, for example, a first position 311, a first normal vector 312, second positions 321a to 321d (a second position group), and second normal vectors 322a to 322d (a second normal vector group). Fig.33 The top view 341 and the top view 342 shown show the first imaginary curved surface 330 in a top view and a top view.

[0218] The first imaginary curved surface 330 is virtually formed as a curved surface by arranging rectangular planes 331 to 335 adjacent to each other at different angles. The rectangular plane 331 is a plane based on the first position 311 and the first normal vector 312. The rectangular planes 332 to 335 are planes based on the second positions 321a to 321d and the second normal vectors 322a to 322d, respectively. The rectangular planes 331 to 335 are each formed by combining two triangular polygons, for example.

[0219] Then, the image processing device 50 executes Fig.10 The image processing device 50 is Fig.10 In the step S102 shown, the image processing device 50 generates, in the physical space image 90, a first virtual projection plane 111 whose shape is adjusted in such a way that the first position 311 is centered thereon and is perpendicular to the first normal vector 312 and is projected onto the first virtual surface 330 with a specified size, and displays the generated first virtual projection plane 111 overlappingly on the physical space image 90.

[0220] Thus, by obtaining the first position data, the first normal vector data, the second position group data, the second normal vector group data and the physical space image 90 (first image) in the physical space 70, the user can visually grasp the size and configuration of the projection surface 11 based on the physical surface (wall 310) of the physical space 70 or the positional relationship between the projection surface 11 and the projection device 10 on the physical space image 90 representing the physical space 70 even at a position outside the physical space 70.

[0221] Furthermore, the image processing device 50 can overlap and display the first virtual projection plane 111 and the first virtual projection device 112 on the physical space image 90 based on the first virtual curved surface 330 through the touch panel 51. Fig. 20 , Fig.21 Similarly to the example, an instruction to change the position or angle of the first virtual projection plane 111 is received from the user, and the first virtual projection plane 111 or the first virtual projection device 112 superimposed on the physical space image 90 is updated according to the received instruction.

[0222] In this way, the image processing device 50 of embodiment 2 generates the first virtual projection surface data and the first virtual projection device data based on the first position data representing the first position 311, the first normal vector data representing the first normal vector 312, the second position group data representing the second position group (second positions 321a to 321d) on the first surface corresponding to the wall 310, and the second normal vector group data representing the second normal vector group (second normal vectors 322a to 322d) corresponding to the second position group on the first surface corresponding to the wall 310.

[0223] Specifically, the image processing device 50 generates virtual surface data representing the first virtual surface 330 based on the first position data, the first normal vector data, the second position group data and the second normal vector group data, and generates the first virtual projection surface data based on the first virtual projection device data and the virtual surface data.

[0224] Furthermore, the image processing device 50 may also display the position and angle of the first virtual projection plane 111, the position and angle of the first virtual projection device 112, the first projection distance, the projection parameters of the first virtual projection device 112, etc. to the user according to the instruction from the user. At this time, the image processing device 50 may determine the origin and the directions of the axes of the three-dimensional orthogonal coordinate system according to the designation from the user. Thus, the user can grasp the positional relationship between the projection plane and the projection device visually confirmed and the projection parameters at this time as numerical values.

[0225] (Variation Example)

[0226] Modifications of the above-described embodiments will be described.

[0227] <Variation 1>

[0228] Although the image processing device 50 is described as a tablet terminal having a touch panel 51, the image processing device 50 is not limited to such a configuration. For example, the image processing device 50 may be an information terminal such as a smartphone or a personal computer.

[0229] <Variation 2>

[0230] The structure in which the image processing device 50 displays the second image through the touch panel 51 has been described, but the image processing device 50 may also perform control such that the generated second image is sent to another device so that the second image is displayed on the other device. In this case, the image processing device 50 may be a device that does not have a display device, etc.

[0231] <Variation 3>

[0232] Although the physical space image 90 is an image obtained by photographing by the camera of the image processing device 50, the physical space image 90 may be an image obtained by photographing by a device different from the image processing device 50 and received by the image processing device 50 from the device. In this case, the image processing device 50 may be a device that does not include a camera.

[0233] (Image processing program)

[0234] In addition, the image processing method described in the above embodiment can be implemented by executing a pre-prepared image processing program by a computer. The image processing program is recorded in a computer-readable storage medium and is executed by reading it from the storage medium. Furthermore, the image processing program can be provided in the form of being stored in a non-temporary storage medium such as a flash memory, or can be provided via a network such as the Internet. The computer that executes the image processing program can be a computer included in an image processing device, or a computer included in an electronic device such as a smart phone, a tablet terminal or a personal computer that can communicate with the image processing device, or a computer included in a server device that can communicate with these image processing devices and electronic devices.

[0235] The above-described embodiments and modifications may be implemented in combination.

[0236] This manual describes at least the following matters. (1)

[0238] An image processing device comprising a processor, wherein:

[0239] The above processor performs the following processing:

[0240] Acquiring first image data obtained by photographing a space with a camera device;

[0241] Generate first virtual projection surface data representing a first virtual projection surface and first virtual projection device data representing a first virtual projection device based on first position data representing a first position in the space and first normal vector data representing a first normal vector of a first surface corresponding to an object existing at the first position in the space;

[0242] generating, based on the first image data, the first virtual projection plane data and the first virtual projection device data, second image data representing a second image of the first virtual projection plane and the first virtual projection device displayed on the first image represented by the first image data; and

[0243] The second image data is output to an output destination. (2)

[0245] The image processing device according to (1), wherein:

[0246] The above processor performs the following processing:

[0247] The first virtual projection device data is generated based on the first virtual projection plane data. (3)

[0249] The image processing device according to (2), wherein:

[0250] The processor performs processing to determine a normal vector of the first virtual projection plane based on the first normal vector. (4)

[0252] The image processing device according to (3), wherein:

[0253] The first virtual projection plane is a virtual projection plane having a normal vector that coincides with the first normal vector. (5)

[0255] The image processing device according to any one of (2) to (4), wherein:

[0256] The above processor performs the following processing:

[0257] The projection direction and position of the first virtual projection device are determined according to the position and size of the first virtual projection plane. (6)

[0259] The image processing device according to any one of (1) to (5), wherein:

[0260] The above processor performs the following processing:

[0261] The first normal vector data at the first position is determined based on distance data regarding the distance between the object and the imaging device. (7)

[0263] The image processing device according to any one of (1) to (6), wherein:

[0264] The above processor performs the following processing:

[0265] specifying a position of an end of the first surface in the first image based on the first image data;

[0266] At least one of the position and the size of the first virtual projection plane is determined based on the position of the end portion. (8)

[0268] The image processing device according to any one of (1) to (7), wherein:

[0269] The above processor performs the following processing:

[0270] The first virtual projection plane displayed on the second image is changed based on the first input data related to the change of at least one of the first position and the first normal vector. (9)

[0272] The image processing device according to any one of (1) to (8), wherein:

[0273] The above processor performs the following processing:

[0274] The first virtual projection device displayed on the second image is changed based on the second input data related to the change of the displacement amount of the projection lens of the first virtual projection device. (10)

[0276] The image processing device according to any one of (1) to (9), wherein:

[0277] The above processor performs the following processing:

[0278] The first virtual projection device data is generated based on the first virtual projection surface data, the second position data representing the second position in the space different from the first position, and the second normal vector data representing the second normal vector of the second surface corresponding to the object existing in the second position in the space. (11)

[0280] The image processing device according to (1), wherein:

[0281] The above processor performs the following processing:

[0282] The first virtual projection surface data and the first virtual projection device data are generated based on the above-mentioned first position data, the above-mentioned first normal vector data, the second position group data representing the second position group on the above-mentioned first surface, and the second normal vector group data representing the second normal vector group corresponding to the above-mentioned second position group on the above-mentioned first surface. (12)

[0284] The image processing device according to (11), wherein:

[0285] The above processor performs the following processing:

[0286] generating virtual curved surface data representing a virtual curved surface based on the first position data, the first normal vector data, the second position group data, and the second normal vector group data; and

[0287] The first virtual projection surface data is generated based on the first virtual projection device data and the virtual curved surface data. (13)

[0289] An image processing method, wherein:

[0290] The processor of the image processing device performs the following processing:

[0291] Acquiring first image data obtained by photographing a space with a camera device;

[0292] Generate first virtual projection surface data representing a first virtual projection surface and first virtual projection device data representing a first virtual projection device based on first position data representing a first position in the space and first normal vector data representing a first normal vector of a first surface corresponding to an object existing at the first position in the space;

[0293] generating, based on the first image data, the first virtual projection plane data and the first virtual projection device data, second image data representing a second image of the first virtual projection plane and the first virtual projection device displayed on the first image represented by the first image data; and

[0294] The second image data is output to an output destination. (14)

[0296] An image processing program is used to cause a processor of an image processing device to perform the following processing:

[0297] Acquiring first image data obtained by photographing a space with a camera device;

[0298] Generate first virtual projection surface data representing a first virtual projection surface and first virtual projection device data representing a first virtual projection device based on first position data representing a first position in the space and first normal vector data representing a first normal vector of a first surface corresponding to an object existing at the first position in the space;

[0299] generating, based on the first image data, the first virtual projection plane data and the first virtual projection device data, second image data representing a second image of the first virtual projection plane and the first virtual projection device displayed on the first image represented by the first image data; and

[0300] The second image data is output to an output destination.

[0301] Various embodiments have been described above, but the present invention is certainly not limited to these examples. Obviously, those skilled in the art can think of various variations or modifications within the scope described in the technical solution, and it should be understood that these variations or modifications naturally also fall within the technical scope of the present invention. Furthermore, the constituent elements in the above embodiments can be arbitrarily combined within the scope of the purpose of the invention.

[0302] In addition, this application is based on the Japanese patent application (Japanese Patent Application No. 2022-057497) filed on March 30, 2022, and the contents are incorporated in this application as a reference.

[0303] Explanation of symbols

[0304] 1-projection unit, 2-operation receiving unit, 2A, 3A-hollow part, 2a, 2b, 3a, 3c, 15a-opening, 4-control device, 4a, 62-memory, 6-projected object, 10-projection device, 11-projection surface, 12-light modulation unit, 15-frame, 21-light source, 22-light modulation unit, 23-projection optical system, 24-control circuit, 31-second optical system, 32, 122-reflection component, 33-third optical system, 34-lens, 50-image processing device, 51-touch panel, 51a, 51b, 51c-position, 61-processor, 63-communication interface, 64-user interface, 65-sensor, 69-bus, 70-physical space, 71-floor, 72, 73, 310-wall, 7 2a~72d-end part, 81, 311-first position, 82, 312-first normal vector, 90-physical space image, 101-main body, 102-first component, 103-second component, 104-projection direction changing mechanism, 105-displacement mechanism, 106-optical unit, 111-first imaginary projection plane, 112-first imaginary projection device, 121-first optical system, 201, 221-first imaginary projection plane operating part, 241, 321a~321d-second position, 242, 322a~322d-second normal vector, 251-imaginary plane, 271, 291-first imaginary projection device operating part, 330-first imaginary curved surface, 331~335-rectangular plane, G1-image, D1-distance.

Claims

1. An image processing device comprising a processor, wherein: The processor performs the following processing: Acquiring first image data obtained by photographing a space with a camera device; Generate first virtual projection surface data representing a first virtual projection surface and first virtual projection device data representing a first virtual projection device based on first position data representing a first position in the space and first normal vector data representing a first normal vector of a first surface corresponding to an object existing at the first position in the space; generating, based on the first image data, the first virtual projection surface data and the first virtual projection device data, second image data representing a second image of the first virtual projection surface and the first virtual projection device displayed on the first image represented by the first image data; and The second image data is output to an output destination.

2. The image processing device according to claim 1, wherein: The processor performs the following processing: The first virtual projection device data is generated based on the first virtual projection plane data.

3. The image processing device according to claim 2, wherein: The processor performs the following processing: The normal vector of the first virtual projection plane is determined based on the first normal vector.

4. The image processing device according to claim 3, wherein: The first virtual projection plane is a virtual projection plane having a normal vector that coincides with the first normal vector.

5. The image processing device according to any one of claims 2 to 4, wherein: The processor performs the following processing: The projection direction and position of the first virtual projection device are determined according to the position and size of the first virtual projection plane.

6. The image processing device according to any one of claims 1 to 5, wherein: The processor performs the following processing: The first normal vector data at the first position is determined based on distance data regarding the distance between the object and the imaging device.

7. The image processing device according to any one of claims 1 to 6, wherein: The processor performs the following processing: The position of the end of the first surface in the first image is identified based on the first image data, and at least one of the position and the size of the first virtual projection plane is determined based on the position of the end.

8. The image processing device according to any one of claims 1 to 7, wherein: The processor performs the following processing: The first virtual projection plane displayed on the second image is changed based on the first input data related to the change of at least one of the first position and the first normal vector.

9. The image processing device according to any one of claims 1 to 8, wherein: The processor performs the following processing: The first virtual projection device displayed on the second image is changed based on second input data related to a change in a displacement amount of a projection lens of the first virtual projection device.

10. The image processing device according to any one of claims 1 to 9, wherein: The processor performs the following processing: The first virtual projection device data is generated based on the first virtual projection surface data, the second position data representing the second position in the space different from the first position, and the second normal vector data representing the second normal vector of the second surface corresponding to the object existing at the second position in the space.

11. The image processing apparatus according to claim 1, wherein: The processor performs the following processing: The first virtual projection surface data and the first virtual projection device data are generated based on the first position data, the first normal vector data, the second position group data representing the second position group on the first surface, and the second normal vector group data representing the second normal vector group corresponding to the second position group on the first surface.

12. The image processing device according to claim 11, wherein: The processor performs the following processing: generating virtual curved surface data representing a virtual curved surface based on the first position data, the first normal vector data, the second position group data, and the second normal vector group data; and The first virtual projection surface data is generated based on the first virtual projection device data and the virtual curved surface data.

13. An image processing method, wherein: The processor of the image processing device performs the following processing: Acquiring first image data obtained by photographing a space with a camera device; Generate first virtual projection surface data representing a first virtual projection surface and first virtual projection device data representing a first virtual projection device based on first position data representing a first position in the space and first normal vector data representing a first normal vector of a first surface corresponding to an object existing at the first position in the space; generating, based on the first image data, the first virtual projection surface data and the first virtual projection device data, second image data representing a second image of the first virtual projection surface and the first virtual projection device displayed on the first image represented by the first image data; and The second image data is output to an output destination.

14. An image processing program, configured to cause a processor of an image processing device to perform the following processing: Acquiring first image data obtained by photographing a space with a camera device; Generate first virtual projection surface data representing a first virtual projection surface and first virtual projection device data representing a first virtual projection device based on first position data representing a first position in the space and first normal vector data representing a first normal vector of a first surface corresponding to an object existing at the first position in the space; generating, based on the first image data, the first virtual projection surface data and the first virtual projection device data, second image data representing a second image of the first virtual projection surface and the first virtual projection device displayed on the first image represented by the first image data; and The second image data is output to an output destination.