Image processing device, image processing method, image processing program, and system

Abstract

Provided are an image processing device, an image processing method, an image processing program, and a system which make it possible to efficiently adjust a projection state.　A processor (61) acquires virtual projection surface data relating to a virtual projection surface (11V), virtual projection device data relating to a virtual projection device (10V), and first image data obtained by an imaging device. On the basis of the first image data, the virtual projection surface data, and the virtual projection device data, the processor (61) generates second image data indicating a second image on which the virtual projection surface (11V) and the virtual projection device (10V) are displayed on the first image indicated by the first image data, and outputs the second image data to an output destination. The processor (61) generates assistance information for bringing the projection state of the projection device (10) close to a projection state indicated by the virtual projection surface data and / or the virtual projection device data, and outputs the assistance information to the output destination.

Description

Image processing device, image processing method, image processing program, and system

[0001] The present invention relates to an image processing device, an image processing method, an image processing program, and a system.

[0002] Patent document 1 describes a projection image adjustment system that, in order to facilitate the installation and adjustment of a projection display device, stores virtual environment installation information indicating the installation state of the projection display device installed so as to obtain a desired image projection state on a projection target object in a virtual space generated by a computer, and control setting values ​​of the projection display device at that time, acquires real environment installation information indicating the installation state of the projection display device in real space, and a control unit that controls the operation of the projection display device corrects the control setting values ​​based on the virtual environment installation information and the real environment installation information so that there is no difference between the image projection state in real space and the desired image projection state, and controls the operation of the projection display device based on the corrected control setting values.

[0003] Patent document 2 describes an image projection device that projects a corrected image according to a projection surface, and includes an imaging unit that captures the projected image, a correction parameter calculation unit that calculates correction parameters that correct image distortion caused by the projection surface based on the captured image, a correction unit that generates a corrected image by correcting the image using the correction parameters, a reproducibility calculation unit that calculates the reproducibility of the corrected image relative to the original image, an image generation unit that generates a guidance image related to the reproducibility, and a control unit that controls the projection of the guidance image.

[0004] Patent document 3 describes a projector that projects an image displayed on an image display unit onto a projection surface via a projection lens to facilitate installation and adjustment, and that includes a lens driving means that drives the projection lens, a receiving means that receives input of at least one projection condition, a parameter determining means that determines control parameters for the lens driving means based on the received projection condition, and a control means that controls the lens driving means based on the determined control parameters.

[0005] Japanese Patent Application Publication No. 2018-005115 International Publication No. 2007 / 072695 Pamphlet Japanese Patent Application Publication No. 2000-081601

[0006] One embodiment of the technique of the present disclosure provides an image processing device, an image processing method, an image processing program, and a system that can efficiently adjust the projection state.

[0007] (1) An image processing device including a processor, wherein the processor: acquires virtual projection surface data relating to a virtual projection surface and virtual projection device data relating to a virtual projection device; acquires first image data obtained by an imaging device; generates second image data representing a second image in which the virtual projection surface and the virtual projection device are displayed on a first image represented by the first image data based on the first image data, the virtual projection surface data, and the virtual projection device data, and outputs the second image data to an output destination; and generates assist information for bringing a projection state by a projection device closer to a projection state represented by at least one of the virtual projection surface data and the virtual projection device data, and outputs the assist information to an output destination.

[0008] (2) The image processing device according to (1), wherein the processor generates third image data representing a third image in which the assist information is displayed on the second image, and outputs the third image data to an output destination.

[0009] (3) The image processing device according to (1) or (2), wherein the processor generates audio data representing the assist information and outputs the audio data to an output destination.

[0010] (4) The image processing device according to any one of (1) to (3), wherein the projection state includes at least one of an installation state of the projection device and a state of a projection surface corresponding to the projection device.

[0011] (5) The image processing device according to (4), wherein the projection state includes an installation state of the projection device, and the processor generates the assist information representing a deviation between the installation state of the projection device based on the first image and the installation state of the virtual projection device represented by the virtual projection device data.

[0012] (6) The image processing device according to (5), wherein the installation state includes at least one of an installation form of the projection device and an installation position of the projection device.

[0013] (7) The image processing device according to (5) or (6), wherein the processor generates the assist information based on a recognition result of a worker installing the projection device, the recognition result being included in the first image.

[0014] (8) The image processing device according to any one of (4) to (7), wherein the projection state includes a state of the projection surface, and the state of the projection surface includes at least one of a position of the projection surface, a size of the projection surface, or an inclination of the projection surface.

[0015] (9) The image processing device according to (8), wherein the state of the projection surface includes at least one of a position or a size of the projection surface, and the processor generates the assist information for setting a projection condition of the projection device that changes at least one of the position or the size of the projection surface.

[0016] (10) The image processing device according to (8) or (9), wherein the state of the projection surface includes a tilt of the projection surface, and the processor generates the assist information for adjusting the tilt of the projection surface.

[0017] (11) An image processing device according to any one of (1) to (10), wherein the processor generates the assist information for bringing the installation position of the projection device closer to a position different from the installation position of the virtual projection device represented by the virtual projection device data, and bringing the state of the projection surface corresponding to the projection device closer to the state of the virtual projection surface represented by the virtual projection surface data.

[0018] (12) An image processing device according to any one of (1) to (10), wherein the processor generates the assist information for bringing the state of the projection surface corresponding to the projection device closer to the state of the virtual projection surface represented by the virtual projection surface data at the installation position of the projection device based on the first image.

[0019] (13) The image processing device according to any one of (1) to (12), wherein the output destination includes the projection device capable of projecting the assist information.

[0020] (14) The image processing device according to any one of (1) to (13), wherein the output destination includes a wearable display device that is worn by a worker who installs the projection device and is capable of displaying the assist information.

[0021] (15) The image processing device according to any one of (1) to (14), wherein the image processing device is provided in an information processing terminal having a display device capable of displaying the assist information, and the output destination includes the display device.

[0022] (16) The image processing device according to (15), wherein the information processing terminal includes the imaging device.

[0023] (17) An image processing method, in which a processor included in an image processing device acquires virtual projection surface data related to a virtual projection surface and virtual projection device data related to a virtual projection device, acquires first image data obtained by an imaging device, generates second image data representing a second image in which the virtual projection surface and the virtual projection device are displayed on a first image represented by the first image data based on the first image data, the virtual projection surface data, and the virtual projection device data, and outputs the second image data to an output destination, and generates assist information for making a projection state by a projection device closer to a projection state represented by at least one of the virtual projection surface data and the virtual projection device data, and outputs the assist information to an output destination.

[0024] (18) An image processing program for causing a processor included in an image processing device to execute the following processes: acquire virtual projection surface data relating to a virtual projection surface and virtual projection device data relating to a virtual projection device; acquire first image data obtained by an imaging device; generate second image data representing a second image in which the virtual projection surface and the virtual projection device are displayed on a first image represented by the first image data, based on the first image data, the virtual projection surface data, and the virtual projection device data, and output the second image data to an output destination; and generate assist information for making the projection state by the projection device closer to the projection state represented by at least one of the virtual projection surface data and the virtual projection device data, and output the assist information to an output destination.

[0025] (19) A system including an image processing device, an imaging device, and a projection device, the system comprising: acquiring virtual projection surface data relating to a virtual projection surface and virtual projection device data relating to a virtual projection device; acquiring first image data obtained by the imaging device; generating second image data representing a second image in which the virtual projection surface and the virtual projection device are displayed on a first image represented by the first image data based on the first image data, the virtual projection surface data, and the virtual projection device data, and outputting the second image data to an output destination; and generating assist information for making the projection state of the projection device closer to the projection state represented by at least one of the virtual projection surface data and the virtual projection device data, and outputting the assist information to the output destination.

[0026] According to the present invention, it is possible to provide an image processing device, an image processing method, an image processing program, and a system that can efficiently adjust the projection state.

[0027] FIG. 1 is a schematic diagram showing an example of a projection device 10 that is a target of installation support by the image processing device of the embodiment. FIG. 2 is a schematic diagram showing an example of the internal configuration of the projection unit 1 shown in FIG. 1. FIG. 3 is a schematic diagram showing the external configuration of the projection device 10. FIG. 4 is a schematic cross-sectional view of the optical unit 106 of the projection device 10 shown in FIG. 3. FIG. 5 is a diagram showing an example of the external appearance of an information processing terminal 50. FIG. 6 is a diagram showing an example of the hardware configuration of the information processing terminal 50. FIG. 7 is a diagram showing an example of a system of the embodiment. FIG. 8 is a diagram showing an example of a second image displayed by the information processing terminal 50. FIG. 9 is a diagram showing an example of adjustment of the projection state of the projection device 10 based on the display of the second image. FIG. 10 is a flowchart showing an example of adjustment of the projection state of the projection device 10. FIG. 11 is a diagram showing an example of a marker for adjusting the installation configuration of the projection device 10. FIG. 12 is a diagram showing an example of a display prompting a change of the mount rotation axis. FIG. 13 is a diagram showing an example of a marker for adjusting the installation position of the projection device 10. FIG. 14 is a diagram showing an example of detection of the position of the projection device 10 based on the marker. FIG. 15 is a diagram showing points recognized by the information processing terminal 50 in the camera coordinate system of FIG. 14 . FIG. 16 is a diagram showing points recognized by the information processing terminal 50 on the plane of the rear surface of the projection device 10. FIG. 17 is a diagram showing an example of a display prompting adjustment of the installation position of the projection device 10. FIG. 18 is a diagram showing another example of a marker for adjusting the installation position of the projection device 10. FIG. 19 is a diagram (part 1) showing an example of output of assist information based on the recognition result of the worker installing the projection device 10. FIG. 20 is a diagram (part 2) showing an example of output of assist information based on the recognition result of the worker installing the projection device 10. FIG. 21 is a diagram (part 3) showing an example of output of assist information based on the recognition result of the worker installing the projection device 10. FIG. 22 is a diagram (part 4) showing an example of output of assist information based on the recognition result of the worker installing the projection device 10. FIG. 23 is a diagram showing an example of the inclination of the projection surface 11. FIG. 24 is a diagram showing an example of a marker grid projected by the projection device 10. 25 is a diagram showing an example of a marker grid on the virtual projection plane 11V displayed by the information processing terminal 50. FIG. 26 is an example of a marker grid 241 of the projection device 10 on the camera plane of the imaging device 65.FIG. 27 shows an example of a marker grid 251 of a virtual projection surface 11V on the camera plane of the imaging device 65. FIG. 28 shows an example of a rectangle connecting points when the plane of the virtual projection surface 11V is used as the reference plane. FIG. 29 shows an example of a display prompting adjustment of the tilt of the projection surface 11 in the example of FIG. 28. FIG. 30 shows another example of a rectangle connecting points when the plane of the virtual projection surface 11V is used as the reference plane. FIG. 31 shows an example of a display prompting adjustment of the tilt of the projection surface 11 in the example of FIG. 30. FIG. 32 shows an example of a state in which a portion of the marker grid 241 straddles another plane (wall 6a and wall 6b). FIG. 33 shows an example of a marker grid 241 used to correct the edge of the projection surface 11. FIG. 34 shows an example of a simulation result in which the virtual projection device 10V is installed on the ceiling 6d. FIG. 35 shows an example of a simulation result in which the virtual projection device 10V is installed on the floor 6e. Fig. 36 is a diagram (part 1) showing an example of a process for aligning the center of the projection surface 11. Fig. 37 is a diagram (part 2) showing an example of a process for aligning the center of the projection surface 11. Fig. 38 is a diagram showing an example of output of assist information using the projection device 10. Fig. 39 is a schematic diagram showing another external configuration of the projection device 10. Fig. 40 is a schematic cross-sectional view of the optical unit 106 of the projection device 10 shown in Fig. 39.

[0028] Hereinafter, an example of an embodiment of the present invention will be described with reference to the drawings.

[0029] (Embodiment) <Projection Device 10 to be Assisted in Installation by Image Processing Device of Embodiment> FIG. 1 is a schematic diagram showing an example of a projection device 10 to be assisted in installation by an image processing device of an embodiment.

[0030] The image processing device according to the embodiment can be used, for example, to assist in the installation of a projection device 10. The projection device 10 includes a projection unit 1, a control device 4, and an operation reception unit 2. The projection unit 1 is configured, for example, by a liquid crystal projector or a projector using LCOS (Liquid Crystal On Silicon). In the following description, the projection unit 1 is assumed to be a liquid crystal projector.

[0031] The control device 4 is a control device that controls projection by the projection device 10. The control device 4 is a device that includes a control unit configured with various processors, a communication interface (not shown) for communicating with each unit, and a memory 4a such as a hard disk, an SSD (Solid State Drive), or a ROM (Read Only Memory), and controls the projection unit 1 in an integrated manner.

[0032] The various processors in the control unit of the control device 4 include a CPU (Central Processing Unit), which is a general-purpose processor that executes programs to perform various processes, a programmable logic device (PLD), which is a processor whose circuit configuration can be changed after manufacture, such as an FPGA (Field Programmable Gate Array), or a dedicated electrical circuit, such as an ASIC (Application Specific Integrated Circuit), which is a processor having a circuit configuration designed specifically to perform specific processing.

[0033] More specifically, the structure of these various processors is an electric circuit that combines circuit elements such as semiconductor elements. The control unit of the control device 4 may be configured with one of the various processors, or may be configured with a combination of two or more processors of the same or different types (for example, a combination of multiple FPGAs or a combination of a CPU and an FPGA).

[0034] The operation reception unit 2 detects instructions from the user by receiving various operations from the user. The operation reception unit 2 may be a button, key, joystick, or the like provided on the control device 4, or may be a receiving unit or the like that receives signals from a remote controller that remotely controls the control device 4.

[0035] The projection target 6 is an object such as a screen or a wall that has a projection surface on which a projected image is displayed by the projection unit 1. In the example shown in Fig. 1, the projection surface of the projection target 6 is a rectangular flat surface.

[0036] The projection surface 11, shown by a dashed line, is an area of ​​the object 6 onto which projection light is irradiated by the projection unit 1. In the example shown in Fig. 1 , the projection surface 11 is rectangular. The projection surface 11 is a part or the entire projectable range onto which projection can be performed by the projection unit 1.

[0037] The projection unit 1, the control device 4, and the operation reception unit 2 may be realized by, for example, a single device (see, for example, FIGS. 3 and 4), or the projection unit 1, the control device 4, and the operation reception unit 2 may be separate devices that communicate with each other and cooperate with each other.

[0038] <Internal Configuration of Projection Unit 1 Shown in FIG. 1> FIG. 2 is a schematic diagram showing an example of the internal configuration of the projection unit 1 shown in FIG.

[0039] As shown in FIG. 2, the projection unit 1 includes a light source 21, a light modulation unit 22, a projection optical system 23, and a control circuit 24.

[0040] The light source 21 includes a light emitting element such as a laser or an LED (Light Emitting Diode), and emits, for example, white light.

[0041] The light modulation unit 22 is composed of three liquid crystal panels that modulate the color light emitted from the light source 21 and separated into three colors, red, blue, and green, by a color separation mechanism (not shown), based on image information, and emit each color image. These three liquid crystal panels may be equipped with red, blue, and green filters, respectively, and the white light emitted from the light source 21 may be modulated by each liquid crystal panel to emit each color image.

[0042] The projection optical system 23 receives light from the light source 21 and the light modulation unit 22, and is configured by, for example, a relay optical system including at least one lens. The light that passes through the projection optical system 23 is projected onto the object 6 to be projected.

[0043] The area of ​​the object 6 that is irradiated with light that passes through the entire range of the light modulation unit 22 is the projectable range in which projection by the projection unit 1 is possible. Within this projectable range, the area that is irradiated with light that actually passes through the light modulation unit 22 is the projection surface 11. For example, by controlling the size, position, and shape of the area of ​​the light modulation unit 22 through which light passes, the size, position, and shape of the projection surface 11 can be changed within the projectable range.

[0044] The control circuit 24 controls the light source 21, the light modulation unit 22, and the projection optical system 23 based on the display data input from the control device 4, thereby projecting an image based on this display data onto the projection target 6. The display data input to the control circuit 24 is made up of three pieces of data: red display data, blue display data, and green display data.

[0045] The control circuit 24 also enlarges or reduces the projection surface 11 (see FIG. 1 ) of the projection unit 1 by changing the projection optical system 23 based on a command input from the control device 4. The control device 4 may also move the projection surface 11 of the projection unit 1 by changing the projection optical system 23 based on an operation from the user received by the operation receiving unit 2.

[0046] The projection device 10 also includes a shift mechanism that mechanically or optically moves the projection surface 11 while maintaining the image circle of the projection optical system 23. The image circle of the projection optical system 23 is the area through which projection light incident on the projection optical system 23 passes through the projection optical system 23 appropriately in terms of light intensity loss, color separation, peripheral curvature, etc.

[0047] The shift mechanism is realized by at least one of an optical system shift mechanism that performs an optical system shift and an electronic shift mechanism that performs an electronic shift.

[0048] The optical system shift mechanism is, for example, a mechanism that moves the projection optical system 23 in a direction perpendicular to the optical axis (see, for example, FIGS. 3 and 4), or a mechanism that moves the light modulation unit 22 in a direction perpendicular to the optical axis instead of moving the projection optical system 23. The optical system shift mechanism may also be a mechanism that moves the projection optical system 23 and the light modulation unit 22 in combination.

[0049] The electronic shift mechanism is a mechanism that shifts the pseudo projection surface 11 by changing the range through which light is transmitted in the light modulation section 22 .

[0050] The projection device 10 may also include a projection direction change mechanism that moves the projection surface 11 together with the image circle of the projection optical system 23. The projection direction change mechanism is a mechanism that changes the projection direction of the projection unit 1 by changing the orientation of the projection unit 1 through mechanical rotation (see, for example, FIGS. 3 and 4).

[0051] <Mechanical Configuration of Projection Device 10> Fig. 3 is a schematic diagram showing the external configuration of the projection device 10. Fig. 4 is a schematic cross-sectional view of the optical unit 106 of the projection device 10 shown in Fig. 3. Fig. 4 shows a cross section taken along a plane along the optical path of light emitted from the main body 101 shown in Fig. 3.

[0052] 3, the projection device 10 includes a main body 101 and an optical unit 106 that protrudes from the main body 101. In the configuration shown in Fig. 3, the operation reception unit 2, the control device 4, and the light source 21, light modulation unit 22, and control circuit 24 in the projection unit 1 are provided in the main body 101. The projection optical system 23 in the projection unit 1 is provided in the optical unit 106.

[0053] The optical unit 106 includes a first member 102 supported by the main body 101 and a second member 103 supported by the first member 102 .

[0054] The first member 102 and the second member 103 may be an integrated member. The optical unit 106 may be configured to be detachable from the main body 101 (in other words, replaceable).

[0055] The main body 101 has a housing 15 (see FIG. 4) in which an opening 15a (see FIG. 4) for passing light is formed at a portion connected to the optical unit 106.

[0056] As shown in Figure 3, inside the housing 15 of the main body 101, there is provided a light source 21 and a light modulation unit 12 including a light modulation section 22 (see Figure 2) that spatially modulates the light emitted from the light source 21 based on input image data to generate an image.

[0057] The light emitted from the light source 21 is incident on the light modulation section 22 of the light modulation unit 12, and is spatially modulated by the light modulation section 22 before being emitted.

[0058] As shown in Figure 4, the image formed by the light spatially modulated by the light modulation unit 12 passes through the opening 15a of the housing 15 and enters the optical unit 106, and is projected onto the projection target 6, which is the projection object, so that the image G1 becomes visible to the observer.

[0059] As shown in Figure 4, the optical unit 106 includes a first member 102 having a hollow portion 2A connected to the inside of the main body portion 101, a second member 103 having a hollow portion 3A connected to the hollow portion 2A, a first optical system 121 and a reflecting member 122 arranged in the hollow portion 2A, a second optical system 31, a reflecting member 32, a third optical system 33, and a lens 34 arranged in the hollow portion 3A, a shift mechanism 105, and a projection direction change mechanism 104.

[0060] The first member 102 is a member having, for example, a rectangular cross-sectional outer shape, and the openings 2a and 2b are formed on planes perpendicular to each other. The first member 102 is supported by the main body 101 with the opening 2a positioned opposite the opening 15a of the main body 101. Light emitted from the light modulation section 22 of the light modulation unit 12 in the main body 101 passes through the openings 15a and 2a and enters the hollow section 2A of the first member 102.

[0061] The direction of light incident from the main body 101 into the hollow portion 2A is referred to as direction X1, the opposite direction to direction X1 is referred to as direction X2, and directions X1 and X2 are collectively referred to as direction X. In addition, in Figure 4, the direction from the front of the page toward the back and the opposite direction are referred to as direction Z. Of direction Z, the direction from the front of the page toward the back is referred to as direction Z1, and the direction from the back of the page toward the front is referred to as direction Z2.

[0062] Furthermore, a direction perpendicular to direction X and direction Z is referred to as direction Y, and within direction Y, the upward direction in Fig. 4 is referred to as direction Y1, and the downward direction in Fig. 4 is referred to as direction Y2. In the example of Fig. 4, the projection device 10 is disposed so that direction Y2 is the vertical direction.

[0063] The projection optical system 23 shown in Fig. 2 is composed of a first optical system 121, a reflecting member 122, a second optical system 31, a reflecting member 32, a third optical system 33, and a lens 34. Fig. 4 shows the optical axis K of the projection optical system 23. The first optical system 121, the reflecting member 122, the second optical system 31, the reflecting member 32, the third optical system 33, and the lens 34 are arranged along the optical axis K in this order from the light modulation unit 22 side.

[0064] The first optical system 121 includes at least one lens, and guides light traveling in the direction X1 that is incident on the first member 102 from the main body 101 to the reflecting member 122 .

[0065] The reflecting member 122 reflects the light incident from the first optical system 121 in the direction Y1. The reflecting member 122 is formed of, for example, a mirror. The first member 102 has an opening 2b formed on the optical path of the light reflected by the reflecting member 122. The reflected light passes through the opening 2b and travels to the hollow portion 3A of the second member 103.

[0066] The second member 103 is a member having a substantially T-shaped cross-sectional outer shape, and has an opening 3a formed in a position facing the opening 2b of the first member 102. Light from the main body 101 that passes through the opening 2b of the first member 102 passes through this opening 3a and enters the hollow portion 3A of the second member 103. The cross-sectional outer shapes of the first member 102 and the second member 103 are arbitrary and are not limited to those described above.

[0067] The second optical system 31 includes at least one lens, and guides the light incident from the first member 102 to the reflecting member 32 .

[0068] The reflecting member 32 reflects the light incident from the second optical system 31 in the direction X2 and guides the light to the third optical system 33. The reflecting member 32 is formed of, for example, a mirror.

[0069] The third optical system 33 includes at least one lens, and guides the light reflected by the reflecting member 32 to a lens 34 .

[0070] The lens 34 is disposed at the end of the second member 103 in the direction X2 so as to cover the opening 3c formed at this end. The lens 34 projects the light incident from the third optical system 33 onto the object 6 to be projected.

[0071] The projection direction change mechanism 104 is a rotation mechanism that rotatably connects the second member 103 to the first member 102. This projection direction change mechanism 104 allows the second member 103 to rotate around a rotation axis (specifically, optical axis K) extending in direction Y. Note that the projection direction change mechanism 104 may be disposed in any position as shown in FIG. 4 as long as it can rotate the optical system. The number of rotation mechanisms is also not limited to one, and multiple mechanisms may be provided. For example, in the configuration of FIG. 3 , a rotation mechanism that rotatably connects the first member 102 to the main body 101 may be provided. This rotation mechanism allows the first member 102 to rotate around a rotation axis extending in direction X.

[0072] The shift mechanism 105 is a mechanism for moving the optical axis K of the projection optical system (in other words, the optical unit 106) in a direction perpendicular to the optical axis K (direction Y in FIG. 4 ). Specifically, the shift mechanism 105 is configured to be able to change the position of the first member 102 in direction Y relative to the main body 101. The shift mechanism 105 may be one that moves the first member 102 manually, or one that moves the first member 102 electrically.

[0073] Fig. 4 shows a state in which the first member 102 has been moved to the maximum extent in the direction Y1 by the shift mechanism 105. When the first member 102 is moved in the direction Y2 by the shift mechanism 105 from the state shown in Fig. 4, the relative position between the center of the image formed by the light modulation unit 22 (in other words, the center of the display surface) and the optical axis K changes, and the image G1 projected onto the projection object 6 can be shifted (translated) in the direction Y2.

[0074] The shift mechanism 105 may be a mechanism that moves the light modulation unit 22 in the direction Y, instead of moving the optical unit 106 in the direction Y. Even in this case, the image G1 projected onto the projection target 6 can be moved in the direction Y2.

[0075] <Appearance of Information Processing Terminal 50> Fig. 5 is a diagram showing an example of the appearance of the information processing terminal 50. The information processing terminal 50 is a tablet terminal having a touch panel 51. The touch panel 51 is a display that can be touched. The information processing terminal 50 displays, on the touch panel 51, an installation assistance image for assisting in the installation of the projection device 10 in a space.

[0076] Specifically, the information processing terminal 50 displays a second image as an installation support image, which is a first image obtained by capturing an image of the space where the projection device 10 is installed and projection is to be performed, and which superimposes an image of the virtual projection surface, which is the virtual projection surface 11, and an image of the virtual projection device, which is the virtual projection device 10.

[0077] <Hardware configuration of information processing terminal 50> Fig. 6 is a diagram showing an example of the hardware configuration of the information processing terminal 50. The information processing terminal 50 shown in Fig. 5 includes, for example, a processor 61, a memory 62, a communication interface 63, a user interface 64, an imaging device 65, and a space recognition sensor 66, as shown in Fig. 6. The processor 61, the memory 62, the communication interface 63, the user interface 64, the imaging device 65, and the space recognition sensor 66 are connected by, for example, a bus 69.

[0078] The processor 61 is a circuit that performs signal processing, and is, for example, a CPU that controls the entire information processing terminal 50. The processor 61 may be realized by other digital circuits such as an FPGA or a DSP (Digital Signal Processor). The processor 61 may also be realized by combining a plurality of digital circuits.

[0079] The memory 62 includes, for example, a main memory and an auxiliary memory. The main memory is, for example, a RAM (Random Access Memory). The main memory is used as a work area for the processor 61.

[0080] The auxiliary memory is, for example, a non-volatile memory such as a magnetic disk, a flash memory, etc. The auxiliary memory stores various programs that operate the information processing terminal 50. The programs stored in the auxiliary memory are loaded into the main memory and executed by the processor 61.

[0081] The auxiliary memory may also include a portable memory that is removable from the information processing terminal 50. Portable memories include memory cards such as a USB (Universal Serial Bus) flash drive and an SD (Secure Digital) memory card, and external hard disk drives.

[0082] The communication interface 63 is a communication interface for communicating with devices external to the information processing terminal 50. The communication interface 63 includes at least one of a wired communication interface for communicating via a wire and a wireless communication interface for communicating wirelessly. The communication interface 63 is controlled by the processor 61.

[0083] The user interface 64 includes, for example, an input device that accepts operation input from the user and an output device that outputs information to the user. The input device can be realized by, for example, keys (e.g., a keyboard) or a remote control. The output device can be realized by, for example, a display or a speaker. In the information processing terminal 50 shown in FIG. 5 , the input device and the output device are realized by the touch panel 51. The user interface 64 is controlled by the processor 61. The information processing terminal 50 accepts various specifications from the user using the user interface 64.

[0084] The imaging device 65 is a device that has an imaging optical system and an imaging element and is capable of capturing images. The imaging device includes, for example, an imaging device provided on the back surface (the surface opposite to the surface on which the touch panel 51 is provided) of the information processing terminal 50 shown in FIG.

[0085] The space recognition sensor 66 is a sensor that can three-dimensionally recognize the space around the information processing terminal 50. One example of the space recognition sensor 66 is a light detection and ranging (LIDAR) that emits laser light, measures the time it takes for the emitted laser light to hit an object and bounce back, and measures the distance and direction to the object. However, the space recognition sensor 66 is not limited to this, and various types of sensors such as a radar that emits radio waves or an ultrasonic sensor that emits ultrasonic waves can be used.

[0086] <System of the embodiment> Fig. 7 is a diagram showing an example of a system of the embodiment. As shown in Fig. 7, for example, a user U1 of an information processing terminal 50 brings a system including the information processing terminal 50 and the projection device 10 into a physical space 70 where the projection device 10 is to be installed. In this case, the information processing terminal 50 is an example of an image processing device in the system of the present invention.

[0087] The information processing terminal 50 recognizes the physical space 70 using the space recognition sensor 66. Specifically, the information processing terminal 50 recognizes the physical space 70 using a world coordinate system consisting of the X-axis, Y-axis, and Z-axis, where one horizontal direction in the physical space 70 is the X-axis, the direction of gravity in the physical space 70 is the Y-axis, and a direction in the physical space 70 perpendicular to the X-axis and Y-axis is the Z-axis.

[0088] Furthermore, the information processing terminal 50 displays a captured image based on the imaging data obtained by imaging using the imaging device 65 as a through image (live view) to the user on the touch panel 51. The imaging data is an example of first image data. The captured image is an example of a first image.

[0089] In the example of Fig. 7, the physical space 70 is indoors, and the wall 6a is the projection target. The top, bottom, left, and right of the wall 6a in Fig. 7 are the top, bottom, left, and right in this embodiment. The wall 6b is adjacent to the left end of the wall 6a and is perpendicular to the wall 6a. The wall 6c is adjacent to the right end of the wall 6a and is perpendicular to the wall 6a. The ceiling 6d is adjacent to the top end of the wall 6a and is perpendicular to the wall 6a. The floor 6e is adjacent to the bottom end of the wall 6a and is perpendicular to the wall 6a.

[0090] 7, the projection device 10 is installed on the floor 6e, but the projection device 10 may be installed on a pedestal or the like installed on the floor 6e, or may be installed on the wall 6b, 6c, or the ceiling 6d using a mounting fixture. The imaging range 65a is the range of imaging by the imaging device 65 of the information processing terminal 50.

[0091] While viewing the through image (second image) displayed on the touch panel 51 of the information processing terminal 50, the user U1 adjusts the position and direction of the information processing terminal 50 and the angle of view of the information processing terminal 50 so that the projection device 10 and the projection surface 11 are within the imaging range 65a (i.e., so that they are displayed on the touch panel 51).

[0092] In the example of Fig. 7, the imaging range 65a includes the wall 6a, the ceiling 6d, the floor 6e, the projection device 10, and the projection surface 11. In the example of Fig. 7, the projection device 10 is installed at an angle to the wall 6a, which is the projection target, so the projection surface 11 is trapezoidal. In the example of Fig. 7, the user U1 holds the information processing terminal 50 in his / her hand, but the information processing terminal 50 may be supported on a support member such as a tripod.

[0093] <Display of second image by information processing terminal 50> Fig. 8 is a diagram showing an example of display of a second image by information processing terminal 50. In the state shown in Fig. 7, information processing terminal 50 displays a second image in which virtual projection device 10V and virtual projection surface 11V are superimposed on a captured image (first image) obtained by capturing an image, as shown in Fig. 8.

[0094] For example, the information processing terminal 50 stores virtual projection device data related to the virtual projection device 10V and virtual projection surface data related to the virtual projection surface 11V. The virtual projection device data is data representing the position, orientation, etc. of the virtual projection device 10V in a virtual space corresponding to the physical space 70. The virtual projection surface data is data representing the position, orientation, etc. of the virtual projection surface 11V in a virtual space corresponding to the physical space 70. The virtual projection device data and the virtual projection surface data are generated, for example, by a prior simulation of the installation of the projection device 10 in the physical space 70.

[0095] The information processing terminal 50 generates and displays a second image by superimposing the virtual projection device 10V and the virtual projection surface 11V on the captured image (first image) based on the recognition result of the physical space 70 by the spatial recognition sensor 66, the virtual projection device data, and the virtual projection surface data.

[0096] <Adjusting the Projection State of the Projection Device 10 Based on the Display of the Second Image> Figure 9 is a diagram showing an example of adjusting the projection state of the projection device 10 based on the display of the second image. As shown in Figure 8, a second image is displayed in which the virtual projection device 10V and the virtual projection surface 11V are superimposed on the captured image (first image). This allows an operator (e.g., user U1) of the projection state of the projection device 10 to easily compare the current state of the projection device 10 and the projection surface 11 in the physical space 70 with the virtual projection device 10V and the virtual projection surface 11V based on a prior simulation of the installation of the projection device 10 in the physical space 70.

[0097] Based on this, the worker adjusts the position and orientation of the projection device 10 in the physical space 70 and various settings of the projection device 10 so that they approach the results of the preliminary simulation, as shown in Fig. 9. At this time, if the state of the projection device 10 and the projection surface 11 is to be reproduced exactly as the simulation results, the following problems arise.

[0098] First, simulation results contain errors and incorrect values, and even if they are applied directly to reality, the expected results may not be obtained. Furthermore, it is practically difficult to place the actual projection device 10 in a position that is exactly the same as the simulation result, and as a result, the projection surface 11 may also deviate from the simulation result, and the expected results may not be obtained. In particular, if the angle of view of the projection device 10 is wide, the deviation of the projection surface 11 becomes greater. Furthermore, when simulation results such as installation on the ceiling 6d (ceiling-suspended) or installation on a virtual pedestal that does not yet exist are used, if one wants to reproduce the actual projection before installing the projection device 10, it is physically difficult to place the projection device 10 in the position that matches the simulation, and the simulation results may not be used as is.

[0099] In contrast, the information processing terminal 50 of this embodiment generates assist information for bringing the projection state of the projection device 10 closer to the projection state represented by the simulation result and outputs the assist information to the operator, thereby enabling the operator to efficiently adjust the projection state of the projection device 10 so as to approach the simulation result. The projection state of the projection device 10 includes at least one of a state related to the projection of the projection device 10 itself and a state of the projection surface 11 of the projection device 10.

[0100] <Adjusting the Projection State of the Projection Device 10> Figure 10 is a flowchart showing an example of adjusting the projection state of the projection device 10. First, the installation configuration of the projection device 10 is adjusted (step S11). The installation configuration of the projection device 10 refers to the setting conditions of the projection device 10 itself, such as the installation style of the projection device 10 (e.g., "portrait" or "landscape"), the mounting surface (e.g., "floor" or "ceiling-mounted"), mount axis rotation (e.g., the state of the rotation mechanism that rotatably connects the first member 102 to the main body 101), and lens axis rotation (e.g., the state of the projection direction change mechanism 104). The adjustment of the installation configuration of the projection device 10 in step S11 will be described later (see, for example, Figures 11 and 12).

[0101] Next, the installation position of the projection device 10 is adjusted (step S12). The adjustment of the installation position of the projection device 10 in step S12 will be described later (see, for example, FIGS. 13 to 22). Next, the position of the projection surface 11 of the projection device 10 is adjusted (step S13). The adjustment of the position of the projection surface 11 of the projection device 10 in step S13 will be described later.

[0102] Next, the tilt of the projection surface 11 of the projection device 10 is corrected (step S14). The correction of the tilt of the projection surface 11 of the projection device 10 in step S14 will be described later (see, for example, Figures 23 to 32, etc.). Next, the edges of the projection surface 11 of the projection device 10 are corrected (step S15). The correction of the edges of the projection surface 11 of the projection device 10 in step S15 will be described later (see, for example, Figure 33, etc.).

[0103] <Adjusting the Installation Configuration of the Projection Device 10> Figure 11 is a diagram showing an example of markers for adjusting the installation configuration of the projection device 10. For example, in the projection device 10, the first member 102 is rotatable relative to the main body 101, and the second member 103 is rotatable relative to the first member 102. In this case, markers 111 to 113 are attached to the main body 101, the first member 102, and the second member 103, respectively. The markers 111 to 113 are markers with different shapes. Furthermore, markers may also be attached to portions of the first member 102 and the second member 103 that are not shown in Figure 11.

[0104] As a result, in step S11 shown in Figure 10, the information processing terminal 50 can detect which markers are reflected and which direction the reflected markers are facing based on the imaging data obtained by imaging the imaging device 65 while the projection device 10 is included in the imaging range 65a, and thereby identify the installation form of the projection device 10, such as the rotational state of the first member 102 relative to the main body 101 (mount axis rotation) and the rotational state of the second member 103 relative to the first member 102 (lens axis rotation).

[0105] Furthermore, the information processing terminal 50 can identify the installation form of the projection device 10, such as whether the projection device 10 is installed in a "portrait" or "landscape" position, or whether the projection device 10 is installed on a "floor" or "ceiling-mounted" surface, based on the imaging data obtained by imaging the imaging device 65 while the projection device 10 is included in the imaging range 65a. At this time, the information processing terminal 50 may identify the installation form of the projection device 10, such as the installation style or the surface, using the detection result of the markers on the projection device 10.

[0106] 12 is a diagram showing an example of a display that prompts a user to change the mount rotation axis. Assume that, as a result of identifying the installation form of the projection device 10, the mount rotation axis of the projection device 10 differs from the simulation result (virtual projection device data).

[0107] In this case, in step S11 shown in Fig. 10, the information processing terminal 50 displays a message 120 saying "The mount rotation axis is incorrect" on the touch panel 51. This message 120 is an example of assist information for bringing the projection state of the projection device 10 closer to the projection state represented by the simulation result.

[0108] The message 120 allows the operator to easily recognize that the mount rotation axis of the projection device 10 differs from the simulation result, and enables the operator to adjust the mount rotation axis of the projection device 10 so that it becomes approximately the same as the simulation result. In addition to the message 120, the information processing terminal 50 may also display, as assist information, guidance information that provides guidance on how to adjust the mount rotation axis of the projection device 10, etc.

[0109] Furthermore, the information processing terminal 50 may output a message or guidance information such as "The mount rotation axis is incorrect" by voice in addition to or instead of displaying it on the screen. The voice output may be performed by a speaker included in the user interface 64, for example.

[0110] In the example of Figure 12, we have explained a case where the mount rotation axis of the projection device 10, which is one of the installation forms of the projection device 10, differs from the simulation results.However, the information processing terminal 50 also generates and outputs assist information in the same way if other installation forms of the projection device 10, such as the installation style, contact surface, lens axis rotation, etc., differ from the simulation results.

[0111] Although the configuration in which the installation configuration of the projection device 10 is identified using markers (e.g., markers 111 to 113) attached to the projection device 10 has been described, the present invention is not limited to such a configuration. For example, the information processing terminal 50 may use a learning model generated by machine learning using images of various installation configurations of projection devices of the same model as the projection device 10 to identify the installation configuration of the projection device 10 based on imaging data obtained by imaging the imaging device 65 with the projection device 10 included in the imaging range 65a. In this case, it is not necessary to attach markers to the projection device 10.

[0112] <Adjusting the Installation Position of the Projection Device 10> Fig. 13 is a diagram showing an example of markers for adjusting the installation position of the projection device 10. Fig. 14 is a diagram showing an example of detection of the position of the projection device 10 based on the markers. Fig. 15 is a diagram showing each point recognized by the information processing terminal 50 in the camera coordinate system of Fig. 14. Fig. 16 is a diagram showing each point recognized by the information processing terminal 50 on the plane behind the projection device 10. It is assumed here that the projection device 10 is placed on approximately the same plane (floor 6e) as the virtual projection device 10V in the physical space 70 by step S11 shown in Fig. 10.

[0113] For example, markers 131 to 134 are attached at different positions on the back surface (the surface that is the top surface in this example) of the main body 101 of the projection device 10. As a result, in step S12 shown in Fig. 10, the information processing terminal 50 can identify the current installation position of the projection device 10 in the physical space 70 by detecting the respective positions of the markers 131 to 134 based on the imaging data obtained by imaging the imaging device 65 in a state where the projection device 10 is included in the imaging range 65a.

[0114] For example, the markers 131 to 134 are arranged on a circumference of a circle centered at a predetermined reference point 135a on the rear surface of the main body 101, and the information processing terminal 50 detects the positions of the markers 131 to 134. The points 131a, 132a, 133a, and 134a are the four corners of a rectangle inscribed with the markers 131 to 134.

[0115] Based on the detection results of the positions of markers 131 to 134, information processing terminal 50 identifies the position of reference point 135a of projection device 10 as the installation position of projection device 10. Reference point 141 is a reference point of virtual projection device 10V that corresponds to reference point 135a of projection device 10. Reference points 135a and 141 are offset from floor 6e on which projection device 10 (virtual projection device 10V) is installed by the height of projection device 10 (virtual projection device 10V).

[0116] Generally, if the positions of four points between planes are known, any point can be mapped (projectively transformed) between planes. Points 131a, 132a, 133a, and 134a and reference point 135a in Fig. 15 are determined from the detection results of markers 131 to 134 in camera coordinates. On the other hand, points 131a, 132a, 133a, and 134a and reference point 135a in Fig. 16 are known positions to which markers 131 to 134 are attached in the projection device 10.

[0117] This allows the information processing terminal 50 to determine a projective transformation matrix (homography matrix) from the camera plane in Fig. 15 to the plane of the back of the projection device 10. Then, the information processing terminal 50 maps the reference point 141 in Fig. 15 onto the plane in Fig. 16 based on this projective transformation matrix, thereby determining the center position (reference point 141) of the virtual projection device 10V on the plane in Fig. 16.

[0118] In addition, since the information processing terminal 50 knows the sizes of the markers 131 to 134 on the plane in Figure 16 and the width of the main body 101 of the projection device 10, it calculates the distance D1 between the reference point 141 and the reference point 135a in Figure 16 from the ratio of these sizes and widths.

[0119] 13 has been described as an example in which markers 131 to 134, which are different from the markers 111 to 113 shown in FIG. 11 for adjusting the installation form of the projection device 10, are attached to the projection device 10 in order to adjust the installation position of the projection device 10, but both the markers 111 to 113 for adjusting the installation form of the projection device 10 and the markers 131 to 134 for adjusting the installation position of the projection device 10 may be attached to the projection device 10. Also, a common marker attached to the projection device 10 may be used to adjust both the installation form of the projection device 10 and the installation position of the projection device 10.

[0120] Fig. 17 is a diagram showing an example of a display prompting adjustment of the installation position of the projection device 10. As in the examples of Figs. 15 and 16, it is assumed that the position of the projection device 10 (the position of the reference point 135a) differs from the simulation result (virtual projection device data).

[0121] In this case, the information processing terminal 50 displays a message 171 saying "Adjust the installation position" on the touch panel 51 in step S12 shown in Fig. 10. This message 171 is an example of assist information for bringing the projection state of the projection device 10 closer to the projection state represented by the simulation result.

[0122] Message 171 allows the worker to easily recognize that the installation position of projection device 10 is different from the simulation result, and enables the worker to adjust the installation position of projection device 10 so that it is approximately the same as the simulation result.

[0123] Furthermore, in addition to or instead of the message 171, the information processing terminal 50 may display, as assist information, guidance information that provides guidance on, for example, a method for adjusting the installation position of the projection device 10. For example, the information processing terminal 50 may display an arrow pointing from the reference point 135a to the reference point 141 as movement direction information 172 that provides guidance on the movement direction of the projection device 10. Furthermore, the information processing terminal 50 may display distance information such as "1.5 m" as movement distance information 173 that provides guidance on the movement distance of the projection device 10 (for example, the above-mentioned distance D1).

[0124] Furthermore, the information processing terminal 50 may output the message 171 "Adjust the installation position" or other guidance information by voice in addition to or instead of the screen display. The voice output can be performed by, for example, a speaker included in the user interface 64. The display image on the touch panel 51 shown in FIG. 17 is an example of a third image in which assist information is displayed on the second image.

[0125] Fig. 18 is a diagram showing another example of markers for adjusting the installation position of the projection device 10. For example, instead of the markers 131 to 134 shown in Fig. 13 etc., a marker 135 shown in Fig. 18 may be attached to the back surface (the surface that is the top surface in this example) of the main body 101 of the projection device 10. The marker 135 is attached so that, for example, the reference point 135a of the projection device 10 and the center of the marker 135 coincide with each other.

[0126] In this case, in step S12 shown in FIG. 10, the information processing terminal 50 can identify the installation position of the projection device 10 in the physical space 70 by detecting the position of the marker 135 (the position of the reference point 135a) based on the imaging data obtained by imaging the imaging device 65 with the projection device 10 included in the imaging range 65a.

[0127] 19 to 22 are diagrams showing examples of output of assist information based on the recognition result of the worker installing the projection device 10. In the examples of Fig. 19 to 22, the projection device 10 (imaging device 65) is fixed to a tripod 221 (see Fig. 22) so that the projection device 10 and the virtual projection device 10V fall within an imaging range 65a, and then imaging is performed by the imaging device 65.

[0128] As shown in Figure 19, the information processing terminal 50 detects the projection device 10 by performing object detection on the captured image 65b (video frame) represented by the imaging data obtained by imaging by the imaging device 65 based on a learning model generated by machine learning using images of projection devices of the same model as the projection device 10.

[0129] Furthermore, as shown in FIG. 20, the information processing terminal 50 detects the posture of a worker (e.g., user U1) by performing human posture detection on the captured image 65b based on a learning model generated by machine learning using images of each posture of the person.

[0130] 21 , the information processing terminal 50 calculates a movement direction 211 in which the projection device 10 should be moved so that the projection device 10 is positioned at the same position as the virtual projection device 10V. The information processing terminal 50 also calculates the direction of the calculated movement direction 211 as seen from the worker, based on the posture of the worker detected by human posture detection. In the example of FIG. 21 , the movement direction 211 is generally to the left, and the worker is also facing generally to the left, so the movement direction 211 is generally forward as seen from the worker.

[0131] 22, the information processing terminal 50 outputs a message such as "Please move forward" by voice. The voice output can be performed, for example, by a speaker included in the user interface 64. This message is an example of assist information for bringing the projection state of the projection device 10 closer to the projection state represented by the simulation results. This allows the worker to easily recognize in which direction, relative to the worker, the projection device 10 should be moved.

[0132] 10, the installation form and installation position of the projection device 10 become substantially the same as the simulation results. Therefore, in step S13 shown in Fig. 10, the position of the projection surface 11 can be adjusted by adjusting the projection conditions of the projection device 10 (screen ratio, optical zoom, optical lens shift mode, optical lens shift operation amount, etc.).

[0133] For example, the information processing terminal 50 outputs to the user U1 projection condition information indicating the projection conditions of the projection device 10, such as the screen ratio, optical zoom, optical lens shift mode, and optical lens shift operation amount, which are included in the simulation result, to the user U1, thereby prompting the user U1 to set the projection conditions of the projection device 10 to the same as the simulation result. The projection condition information in this case is an example of assist information for bringing the projection state of the projection device 10 closer to the projection state represented by the simulation result. The projection condition information can be output by screen display on the touch panel 51, audio output from a speaker included in the user interface 64, or the like.

[0134] Alternatively, the information processing terminal 50 may communicate with the projection device 10 to control the projection device 10 to set the projection conditions included in the simulation results.

[0135] Even after adjusting the position of projection surface 11 as described above, there may be cases where the plane of virtual projection surface 11V does not coincide slightly with the plane of projection surface 11. This is due to projection misalignment caused by a slight positional shift when adjusting the installation position of projection device 10 in step S12 shown in Fig. 10, or an error in surface detection in information processing terminal 50. However, here, the plane of virtual projection surface 11V and the plane of projection surface 11 are considered to be the same, that is, slight errors are allowed, and the position of projection surface 11 is adjusted.

[0136] <Correction of Tilt of Projection Surface 11> Fig. 23 is a diagram showing an example of the tilt of projection surface 11. By step S13 shown in Fig. 10, the position of projection surface 11 approximately coincides with virtual projection surface 11V, but as shown in Fig. 23, there are cases where projection surface 11 is tilted with respect to virtual projection surface 11V, causing a misalignment. This is due to, for example, a slight misalignment between the plane of virtual projection surface 11V and the plane of projection surface 11 due to the above-mentioned misalignments and errors.

[0137] Although it is possible to correct the tilt of the projection surface 11 of the projection device 10 by correcting the projected image (electronic correction), this would result in a significant deterioration in the projection image quality. Therefore, in step S14 shown in Figure 10, in order to suppress the deterioration of the projection image quality due to this correction of the projection image, the tilt is corrected as much as possible by readjusting the installation position of the projection device 10, and then the tilt is corrected by correcting the projection image.

[0138] Fig. 24 is a diagram showing an example of a marker grid projected by the projection device 10. The projection device 10 is capable of projecting, for example, a marker grid 241 for alignment onto the projection surface 11. The marker grid 241 is formed by arranging a plurality of markers at intervals. In the example of Fig. 24, the marker grid 241 is formed by arranging 30 markers in a 5 x 6 matrix.

[0139] The markers included in the marker grid 241 have different shapes, and by detecting each marker in the marker grid 241, the information processing terminal 50 can identify the position of the detected marker on the projection surface 11. Note that in the figure, each marker in the marker grid 241 is shown as a rectangle of the same shape. In the example of Fig. 24, as in the example of Fig. 23, the projection surface 11 is tilted, and therefore the marker grid 241 is also tilted.

[0140] 25 is a diagram showing an example of a marker grid on the virtual projection surface 11V displayed by the information processing terminal 50. The information processing terminal 50 may further superimpose a marker grid 251 on the second image in which the virtual projection device 10V and the virtual projection surface 11V are superimposed on the captured image (first image). The marker grid 251 virtually represents the marker grid 241. In the example of FIG. 25 , the marker grids 241 and 251 are also misaligned with each other due to the inclination of the projection surface 11 with respect to the virtual projection surface 11V.

[0141] 26 shows an example of the marker grid 241 of the projection device 10 on the camera plane of the imaging device 65. Markers 241a to 241d are markers at the four corners of the marker grid 241. The information processing terminal 50 detects the markers 241a to 241d included in the captured image 65b, and detects corner positions 261 to 264 of the marker grid 241 based on the markers 241a to 241d.

[0142] 27 shows an example of a marker grid 251 on the virtual projection plane 11V on the camera plane of the imaging device 65. Markers 251a to 251d of the marker grid 251 are markers at the four corners of the marker grid 251, corresponding to the markers 241a to 241d of the marker grid 241. Corner positions 271 to 274 are corner positions of the marker grid 251, corresponding to the corner positions 261 to 264 of the marker grid 241.

[0143] The marker grid 251 shown in Figure 27 is a marker grid when the imaging device 65 (information processing terminal 50) is completely facing the wall 6a, and the corner positions 271 to 274 are the four corners of a rectangle, but when the imaging device 65 is at an angle to the wall 6a, the corner positions 271 to 274 are the four corners of a trapezoid.

[0144] Fig. 28 shows an example of a rectangle formed by connecting points when the plane of virtual projection surface 11V is used as the reference plane. Information processing terminal 50 calculates a projection matrix that converts angular positions 271 to 274 shown in Fig. 27 into four positions on the reference plane (the plane of virtual projection surface 11V). Then, based on the calculated projection matrix, information processing terminal 50 maps angular positions 261 to 264 shown in Fig. 26 into four positions on the reference plane (the plane of virtual projection surface 11V), as shown in Fig. 28.

[0145] This makes it possible to calculate the tilt of the projection surface 11 (at angular positions 261 to 264) relative to the virtual projection surface 11V (at angular positions 271 to 274). In the example of Fig. 28, the projection surface 11 is rotated around the projection direction of the projection device 10 relative to the virtual projection surface 11V.

[0146] Fig. 29 is a diagram showing an example of a display that prompts adjustment of the tilt of projection surface 11 in the example of Fig. 28. In the example shown in Fig. 28, information processing terminal 50 displays, on touch panel 51 in step S14 shown in Fig. 10, an assistance image 290 that includes a message 291 saying "Please adjust the tilt of the main body" and a guide image 292 that guides the user to adjust the tilt so that projection device 10 rotates around the projection direction of projection device 10. This assistance image 290 is an example of assist information for bringing the projection state of projection device 10 closer to the projection state represented by the simulation result.

[0147] The support image 290 allows the operator to easily recognize that the projection device 10 is tilted in a rotational direction around the projection direction of the projection device 10 in comparison with the simulation results, and enables the operator to adjust the tilt of the projection device 10 in the rotational direction around the projection direction of the projection device 10 so that it is approximately the same as the simulation results.

[0148] The information processing terminal 50 may also display, as assist information, guidance information that provides guidance on a method for adjusting the tilt of the projection device 10 in a rotational direction around the projection direction of the projection device 10. One example of a method for adjusting the tilt of the projection device 10 is to adjust the height of an adjustable leg provided on the bottom surface of the projection device 10.

[0149] The information processing terminal 50 may output the tilt-related assist information by voice in addition to or instead of displaying it on the screen. The voice output may be performed by a speaker included in the user interface 64, for example.

[0150] 29 may be superimposed on a second image in which the virtual projection device 10V and the virtual projection surface 11V are superimposed on the captured image (first image). The image displayed by the touch panel 51 in this case is an example of a third image in which assist information is displayed on the second image.

[0151] 30 shows another example of a rectangle formed by connecting points when the plane of virtual projection surface 11V is used as the reference plane. In the example of Fig. 30, the rectangle with vertices at corner positions 261 to 264, with the plane of virtual projection surface 11V as the reference plane, has a shape in which the right side is longer than the left side. In this case, it can be determined that projection device 10 is tilted in the rotational direction about the vertical axis relative to wall 6a.

[0152] Fig. 31 is a diagram showing an example of a display that prompts adjustment of the tilt of projection surface 11 in the example of Fig. 30. In the example shown in Fig. 30, information processing terminal 50 displays, on touch panel 51 in step S14 shown in Fig. 10, support image 310 including message 311 saying "Please adjust the tilt of the main body" and guide image 312 that guides the user to adjust the tilt so that projection device 10 rotates around the vertical direction. This support image 310 is an example of assist information for bringing the projection state of projection device 10 closer to the projection state represented by the simulation results.

[0153] The support image 310 allows the operator to easily recognize that the projection device 10 is tilted in a rotational direction about the vertical direction relative to the simulation result, and enables the operator to adjust the tilt of the projection device 10 in the rotational direction about the vertical direction so that the tilt is approximately the same as the simulation result. Furthermore, the information processing terminal 50 may include, in the support image 310, guidance information that provides guidance on a method for adjusting the tilt of the projection device 10 in the rotational direction about the vertical direction.

[0154] The information processing terminal 50 may output the tilt-related assist information by voice in addition to or instead of displaying it on the screen. The voice output may be performed by a speaker included in the user interface 64, for example.

[0155] 31 may be superimposed on a second image in which the virtual projection device 10V and the virtual projection surface 11V are superimposed on the captured image (first image). The image displayed by the touch panel 51 in this case is an example of a third image in which assist information is displayed on the second image.

[0156] 23 to 31 have been described with reference to examples in which the marker grids 241 and 251 are used to specify the position of the plane. Using the marker grids 241 and 251 to specify the position of the plane has the following two advantages, for example.

[0157] To explain the first advantage, for example, due to an error or mistake in setting the virtual projection plane 11V, part of the projected marker grid 241 may straddle the walls 6a and 6b. Figure 32 is a diagram showing an example of a state in which part of the marker grid 241 straddles another plane (walls 6a and 6b). In the example of Figure 32, five markers in a row on the left side of the marker grid 241 straddle the walls 6a and 6b, and the information processing terminal 50 fails to detect these five markers.

[0158] In such a case, the information processing terminal 50 does not use markers in the marker grid 241 that cross over another plane (for example, markers that failed to be detected), but instead performs conversion to the reference plane described in Figures 28 and 30 based on markers that do not cross over another plane (for example, markers that were successfully detected) and markers in the marker grid 251 that correspond to markers that do not cross over another plane, thereby being able to detect the inclination of the projection surface 11 with respect to the virtual projection surface 11V.

[0159] The second advantage is that each marker in the marker grid 241 has a different shape and can be uniquely identified. Therefore, even if an image is captured so that only a portion of the marker grid 241 falls within the imaging range 65a, the information processing terminal 50 can detect the inclination of the projection surface 11 with respect to the virtual projection surface 11V as long as, for example, four markers in the marker grid 241 fall within the imaging range 65a.

[0160] <Correction of Edges of Projection Surface 11> By the adjustments up to step S14 shown in Fig. 10, the position and attitude of projection device 10 are adjusted to be substantially the same as the simulation results. In step S15 shown in Fig. 10, an adjustment is made to align the edges of projection surface 11 with virtual projection surface 11V.

[0161] Fig. 33 is a diagram showing an example of a marker grid 241 used to correct the edge of the projection surface 11. For example, as shown in Fig. 33, the information processing terminal 50 projects the marker grid 241 used to correct the tilt of the projection device 10 from the projection device 10 onto the wall 6a. However, in this case, it is also possible to project only the markers 241a to 241d at the four corners of the marker grid 241, and in the example of Fig. 33, only the markers 241a to 241d are projected.

[0162] 33 shows markers 241a to 241d detected by the information processing terminal 50 from imaging data obtained by the imaging device 65, and markers 251a to 251d on the virtual projection surface 11V. In the example of FIG. 33, the markers 241a to 241d are slightly misaligned with the markers 251a to 251d. In response to this, the information processing terminal 50 causes the projection device 10 to electronically shift or enlarge or reduce the projection surface 11 so that the markers 241a to 241d are aligned with the markers 251a to 251d. This allows fine adjustments to be made so that the edges of the projection surface 11 approximately coincide with the virtual projection surface 11V.

[0163] Note that before the electronic shift or enlargement / reduction, optical zoom using a zoom lens included in the projection optical system 23, optical shift using the shift mechanism 105, etc. may be performed. Furthermore, if the information processing terminal 50 cannot detect the markers 241a to 241d from the imaging data, it may determine that the positions of the markers 251a to 251d are incorrect, that is, that the markers 251a to 251d straddle a plane in the physical space 70, and control the projection device 10 to move the marker grid 241 until the markers 241a to 241d are detected.

[0164] <When the projection device 10 cannot be installed on the ground surface as per the simulation results> In the above-described embodiment, a case where the projection device 10 can be installed on the ground surface in the physical space 70 as per the simulation results was described, but the present invention is also applicable to cases where the projection device 10 cannot be installed on the ground surface in the physical space 70 as per the simulation results.

[0165] For example, when there are simulation results for installing the virtual projection device 10V on the ceiling 6d, wall 6b, or wall 6c, it may be difficult to install the projection device 10 in accordance with the simulation results before actually installing or constructing the projection device 10.

[0166] 34 is a diagram showing an example of a simulation result in which a virtual projection device 10V is installed on the ceiling 6d. In Fig. 34, a virtual space 70V is a virtual space representing the physical space 70, a virtual wall 6aV is a virtual wall representing the wall 6a, a virtual ceiling 6dV is a virtual ceiling representing the ceiling 6d, and a virtual floor 6eV is a virtual floor representing the floor 6e. In this example, it is possible to install the projection device 10 on the floor 6e, but it is assumed at this point that it is difficult to install the projection device 10 on the ceiling 6d as shown in the simulation result.

[0167] In this case, the information processing terminal 50 performs a simulation in which the projection surface 11 (virtual projection surface 11V) is maintained with the projection device 10 (virtual projection device 10V) installed on the floor 6e (virtual floor 6eV), and generates virtual projection device data and virtual projection surface data showing the results of this simulation. Fig. 35 is a diagram showing an example of the results of a simulation in which the virtual projection device 10V is installed on the floor 6e. Note that the virtual projection surface data in this case is the same as the original virtual projection surface data.

[0168] Then, the information processing terminal 50 uses the virtual projection device data and the virtual projection surface data to perform the processes described in Fig. 10. As a result, although the installation of the projection device 10 cannot be reproduced exactly as the original simulation results, it becomes possible to reproduce the projection surface 11 exactly as the simulation results.

[0169] As described in Figures 34 and 35, the information processing terminal 50 may generate and output assist information to bring the installation position of the projection device 10 (e.g., the ground surface) closer to a position different from the installation position of the virtual projection device represented by the virtual projection device data, and to bring the state of the projection surface 11 closer to the state of the virtual projection surface 11V represented by the virtual projection surface data.

[0170] <When the projection device 10 cannot be installed at a position consistent with the simulation results> In the above-described embodiment, a case has been described in which the projection device 10 can be installed at a position in the physical space 70 consistent with the simulation results, but the present invention is also applicable to cases in which the projection device 10 cannot be installed at a position in the physical space 70 consistent with the simulation results.

[0171] For example, steps S11 and S12 shown in Fig. 10 require an operator to manually adjust the projector body, which is time-consuming. Therefore, in the adjustment shown in Fig. 10, steps S11 and S12 can be omitted, and the projection device 10 can be installed in an appropriate installation form and installation position, and the projection surface 11 can be adjusted.

[0172] 10, the projection conditions (screen ratio, optical zoom, optical lens shift mode, optical lens shift operation amount, etc.) of the projection device 10 from the simulation results are used as they are. However, if steps S11 and S12 are omitted, these projection conditions cannot be used in step S13, and therefore the information processing terminal 50 performs processing to align the center of the projection surface 11, for example.

[0173] 36 and 37 are diagrams showing an example of a process for aligning the center of the projection surface 11. For example, as shown in Fig. 36, the information processing terminal 50 causes the projection device 10 to project a center marker 361 at the center position of the projection surface 11. Furthermore, the information processing terminal 50 captures a moving image of this center marker 361 using the imaging device 65, and detects the center marker 361 in each frame obtained by capturing the moving image.

[0174] 37 is the center position of the virtual projection surface 11V. The information processing terminal 50 gradually shifts the lens of the projection device 10 so that the detected center marker 361 approaches the virtual projection surface center 371. Thereafter, by executing steps S14 and S15 shown in FIG. 10, it becomes possible to reproduce the projection surface 11 as in the simulation results, although the installation of the projection device 10 cannot be reproduced as in the simulation results.

[0175] In the examples of Figures 36 and 37, we have described a process in which one central marker 361 is tracked using video processing and positioning is performed by providing feedback each time. However, as explained in step S14 of Figure 10, it is also possible to align the positions between planes using multiple markers such as marker grids 241 and 251.

[0176] As described in Figures 36 and 37, the information processing terminal 50 may generate and output assist information to bring the state of the projection surface 11 closer to the state of the virtual projection surface 11V represented by the virtual projection surface data at the installation position of the projection device 10 based on the first image (captured image).

[0177] As described above, the information processing terminal 50 generates and outputs second image data representing a second image in which the virtual projection surface and the virtual projection device are displayed on the first image represented by the first image data, based on virtual projection surface data relating to the virtual projection surface 11V, virtual projection device data relating to the virtual projection device 10V, and first image data obtained by the imaging device 65.

[0178] Furthermore, the information processing terminal 50 generates and outputs assist information for adjusting the projection state of the projection device 10 (the installation state of the projection device 10 and the state of the projection surface 11) to the projection state represented by the virtual projection surface data and the virtual projection device data. This makes it possible to efficiently adjust the projection state of the projection device 10 so as to reproduce the projection state represented by the virtual projection surface data and the virtual projection device data (e.g., a simulation result).

[0179] For example, as an example of an output form of the assist information, the processor 61 may generate and output third image data representing a third image in which the assist information is displayed on the second image. Furthermore, as an example of an output form of the assist information, the processor 61 may generate and output audio data representing the assist information. Furthermore, the processor 61 may combine these output forms of the assist information to generate and output third image data representing a third image in which the assist information is displayed on the second image and audio data representing the assist information.

[0180] The assist information is, for example, information that indicates a discrepancy between the installation state of the projection device 10 and the installation state of the virtual projection device represented by the virtual projection device data. The installation state of the projection device 10 includes at least one of the installation form of the projection device 10 (e.g., installation style, ground surface, rotation state of the mount axis or lens axis, etc.) and the installation position of the projection device 10.

[0181] Furthermore, the information processing terminal 50 may generate assist information based on the recognition result of the worker installing the projection device 10, which is included in the first image. This makes it possible to generate and output assist information that is easy for the worker installing the projection device 10 to understand.

[0182] Among the projection states of the projection device 10, the state of the projection surface 11 includes at least one of the position of the projection surface 11, the size of the projection surface 11, and the tilt of the projection surface 11. The size of the projection surface 11 is adjusted depending on the position between the projection device 10 and the projection surface 11, the focal length of the projection device 10, etc.

[0183] For example, the information processing terminal 50 generates assist information for setting the projection conditions (e.g., screen ratio, optical zoom, optical lens shift mode, optical lens shift operation amount, etc.) of the projection device 10 that change at least one of the position and size of the projection surface 11. The information processing terminal 50 may also generate assist information for adjusting the tilt of the projection surface 11.

[0184] <Modification of Output Form of Assist Information> Although the configuration has been described in which the assist information is output by screen display or audio output on the touch panel 51 provided in the information processing terminal 50, the assist information may also be output by another device that can communicate with the information processing terminal 50. For example, the information processing terminal 50 may control the projection device 10 to project the assist information onto the projection surface 11 from the projection device 10.

[0185] 38 is a diagram showing an example of output of assist information using the projection device 10. For example, the information processing terminal 50 may transmit to the projection device 10 a second image in which the virtual projection device 10V and the virtual projection surface 11V are superimposed on the captured image (first image), and assist information, thereby controlling the projection device 10 to project this information onto the projection surface 11. Although FIG. 38 describes a configuration in which the projection device 10 projects assist information related to adjustment of the installation position of the projection device 10, other assist information may also be projected by the projection device 10.

[0186] The output form of the assist information by audio output is not limited to the audio output of a message (language), but may also be non-verbal audio output such as a pulse sound whose tempo becomes faster as the simulation result approaches. The output form of the assist information may also be the length or strength of vibration by the information processing terminal 50 or a device capable of communicating with the information processing terminal 50. The output form of the assist information may also be a form in which the assist information is displayed to the worker using a wearable display device, such as AR (Augmented Reality) glasses, worn by the worker installing the projection device 10.

[0187] <Variations of the method for identifying the installation location of the projection device 10> Although the configuration using image recognition to identify the installation location of the projection device 10 has been described, the current installation location of the projection device 10 may be identified using positioning using Bluetooth (registered trademark) or the like.

[0188] <Variations of the configuration of the projection device 10> In Figures 3 and 4, the configuration of the projection device 10 has been described as being such that the optical axis K is bent twice using the reflecting member 122 and the reflecting member 32, but it is also possible to omit the reflecting member 122 and the reflecting member 32 and have a configuration in which the optical axis K is not bent, or to omit either the reflecting member 122 or the reflecting member 32 and have a configuration in which the optical axis K is bent once.

[0189] Fig. 39 is a schematic diagram showing another external configuration of the projection device 10. Fig. 40 is a schematic cross-sectional view of the optical unit 106 of the projection device 10 shown in Fig. 39. In Figs. 39 and 40, parts that are the same as those shown in Figs. 3 and 4 are given the same reference numerals and descriptions thereof will be omitted.

[0190] The optical unit 106 shown in Fig. 39 includes a first member 102 supported by a main body 101, and does not include the second member 103 shown in Fig. 3 and Fig. 4. Furthermore, the optical unit 106 shown in Fig. 39 does not include the reflecting member 122, the second optical system 31, the reflecting member 32, the third optical system 33, and the projection direction changing mechanism 104 shown in Fig. 3 and Fig. 4.

[0191] In the optical unit 106 shown in Fig. 39, the projection optical system 23 shown in Fig. 2 is composed of a first optical system 121 and a lens 34. Fig. 40 shows the optical axis K of this projection optical system 23. The first optical system 121 and the lens 34 are arranged along the optical axis K in this order from the light modulation unit 22 side.

[0192] The first optical system 121 guides light traveling in the direction X1 that is incident on the first member 102 from the main body 101 to the lens 34. The lens 34 is disposed at the end of the main body 101 on the direction X1 side so as to close the opening 3c formed at this end. The lens 34 projects the light incident from the first optical system 121 onto the projection surface 11.

[0193] Although the touch panel 51 of the information processing terminal 50 has been described as an example of a display device of the present invention, the display device of the present invention is not limited to the touch panel 51 and may be other display devices (such as other displays or the above-mentioned AR glasses) that can communicate with the information processing terminal 50.

[0194] Although the imaging device 65 of the information processing terminal 50 has been described as an example of an imaging device of the present invention, the imaging device of the present invention is not limited to the imaging device 65 and may be any other imaging device capable of communicating with the information processing terminal 50 .

[0195] <Image Processing Program> The image processing method described in the above-described embodiment can be realized by executing a prepared image processing program on a computer. The image processing program is recorded on a computer-readable storage medium and executed by being read from the storage medium. The image processing program may be provided in a form stored on a non-transitory storage medium such as a flash memory, or may be provided via a network such as the Internet. The computer that executes the image processing program may be included in the image processing device (information processing terminal 50), or may be included in an electronic device such as a smartphone, tablet terminal, or personal computer that can communicate with the image processing device, or may be included in a server device that can communicate with these image processing devices and electronic devices.

[0196] Although various embodiments have been described above, it goes without saying that the present invention is not limited to such examples. It is clear that a person skilled in the art can conceive of various modifications or alterations within the scope of the claims, and it is understood that these also naturally fall within the technical scope of the present invention. Furthermore, the components of the above embodiments may be combined in any manner as long as they do not deviate from the spirit of the invention.

[0197] This application is based on a Japanese patent application (Patent Application No. 2022-140823) filed on September 5, 2022, the contents of which are incorporated herein by reference.

[0198] DESCRIPTION OF SYMBOLS 1 Projection unit 2 Operation reception unit 2A, 3A Hollow portion 2a, 2b, 3a, 3c, 15a Opening 4 Control device 4a, 62 Memory 6 Projected object 6a, 6b, 6c Wall 6aV Virtual wall 6d Ceiling 6dV Virtual ceiling 6e Floor 6eV Virtual floor 10 Projection device 10V Virtual projection device 11 Projection surface 11V Virtual projection surface 12 Light modulation unit 15 Housing 21 Light source 22 Light modulation unit 23 Projection optical system 24 Control circuit 31 Second optical system 32, 122 Reflecting member 33 Third optical system 34 Lens 50 Information processing terminal 51 Touch panel 61 Processor 63 Communication interface 64 User interface 65 Imaging device 65a Imaging range 65b Captured image 66 Space recognition sensor 69 Bus 70 Physical space 70V Virtual space 101 Main body 102 First member 103 Second member 104 Projection direction change mechanism 105 Shift mechanism 106 Optical unit 111 to 113, 131 to 135, 241a to 241d, 251a to 251d Marker 120, 171, 291, 311 Message 121 First optical system 131a, 132a, 133a, 134a Point 135a, 141 Reference point 172 Movement direction information 173 Movement distance information 211 Movement direction 221 Tripod 241, 251 Marker grid 261 to 264, 271 to 274 Corner position 290, 310 Support image 292, 312 Guide image 361 Center marker 371 Virtual projection plane center G1 Image U1 User D1 Distance

Claims

1. An image processing device including a processor, The processor, obtaining virtual projection surface data relating to the virtual projection surface and virtual projection device data relating to the virtual projection device; Acquiring first image data obtained by an imaging device; generating second image data representing a second image in which the virtual projection surface and the virtual projection device are displayed on a first image represented by the first image data, based on the first image data, the virtual projection surface data, and the virtual projection device data, and outputting the second image data to an output destination; generating assist information representing a deviation between an installation state of the projection device based on the first image and an installation state of the virtual projection device represented by the virtual projection device data, based on a recognition result of a worker installing the projection device, the assist information being included in the first image, and outputting the assist information to an output destination; Image processing device.

2. 2. The image processing device according to claim 1, The processor, generating third image data representing a third image in which the assist information is displayed on the second image, and outputting the third image data to an output destination; Image processing device.

3. 2. The image processing device according to claim 1, The processor, generating voice data representing the assist information and outputting the voice data to an output destination; Image processing device.

4. An image processing device according to claim 1, The installation state includes at least one of an installation form of the projection device or an installation position of the projection device. Image processing device.

5. An image processing device according to claim 1, the assist information is information for making a projection state by the projection device closer to a projection state represented by at least one of the virtual projection surface data and the virtual projection device data, the projection state includes a state of a projection surface corresponding to the projection device, The state of the projection surface includes at least one of the position of the projection surface, the size of the projection surface, or the inclination of the projection surface. Image processing device.

6. An image processing device according to claim 5, The state of the projection surface includes at least one of a position or a size of the projection surface, the processor generates the assist information for setting a projection condition of the projection device that changes at least one of a position or a size of the projection surface. Image processing device.

7. An image processing device according to claim 5, the state of the projection surface includes an inclination of the projection surface, The processor generates the assist information for adjusting the inclination of the projection surface. Image processing device.

8. An image processing device according to claim 1, The processor, generating the assist information for bringing an installation position of the projection device closer to a position different from the installation position of the virtual projection device represented by the virtual projection device data, and for bringing a state of a projection surface corresponding to the projection device closer to a state of the virtual projection surface represented by the virtual projection surface data; Image processing device.

9. An image processing device according to claim 1, The processor, generating the assist information for bringing a state of the projection surface corresponding to the projection device closer to a state of the virtual projection surface represented by the virtual projection surface data, at an installation position of the projection device based on the first image; Image processing device.

10. An image processing device according to claim 1, the output destination includes the projection device capable of projecting the assist information, Image processing device.

11. The image processing device according to claim 1, the output destination includes a wearable display device that is worn by a worker who installs the projection device and is capable of displaying the assist information; Image processing device.

12. An image processing device according to any one of claims 1 to 11, an information processing terminal having a display device capable of displaying the assist information, The output destination includes the display device. Image processing device.

13. The image processing device according to claim 12, the information processing terminal is equipped with the imaging device, Image processing device.

14. A processor provided in an image processing device, obtaining virtual projection surface data relating to the virtual projection surface and virtual projection device data relating to the virtual projection device; Acquiring first image data obtained by an imaging device; generating second image data representing a second image in which the virtual projection surface and the virtual projection device are displayed on a first image represented by the first image data, based on the first image data, the virtual projection surface data, and the virtual projection device data, and outputting the second image data to an output destination; generating assist information representing a deviation between an installation state of the projection device based on the first image and an installation state of the virtual projection device represented by the virtual projection device data, based on a recognition result of a worker installing the projection device, the assist information being included in the first image, and outputting the assist information to an output destination; Image processing methods.

15. A processor provided in an image processing device, obtaining virtual projection surface data relating to the virtual projection surface and virtual projection device data relating to the virtual projection device; Acquiring first image data obtained by an imaging device; generating second image data representing a second image in which the virtual projection surface and the virtual projection device are displayed on a first image represented by the first image data, based on the first image data, the virtual projection surface data, and the virtual projection device data, and outputting the second image data to an output destination; generating assist information representing a deviation between an installation state of the projection device based on the first image and an installation state of the virtual projection device represented by the virtual projection device data, based on a recognition result of a worker installing the projection device, the assist information being included in the first image, and outputting the assist information to an output destination; An image processing program to carry out the processing.

16. An image processing device, An imaging device; A projection device; A system comprising: obtaining virtual projection surface data relating to the virtual projection surface and virtual projection device data relating to the virtual projection device; acquiring first image data obtained by the imaging device; generating second image data representing a second image in which the virtual projection surface and the virtual projection device are displayed on a first image represented by the first image data, based on the first image data, the virtual projection surface data, and the virtual projection device data, and outputting the second image data to an output destination; generating assist information representing a deviation between an installation state of the projection device based on the first image and an installation state of the virtual projection device represented by the virtual projection device data, based on a recognition result of a worker installing the projection device, the assist information being included in the first image and outputting the assist information to an output destination; system.