Information processing device, information processing method, and information processing program

Abstract

Provided is an information processing device, information processing method, and information processing program capable of more accurately simulating a projection state by a projection device.　A processor (61) is provided, wherein the processor (61): acquires spatial image data obtained by capturing an image of a space; determines the position of a virtual projection device (72) in the space; acquires a plurality of projection points (75a) to (75h), which are intersections between a plurality of projection light beams (73a) to (73h) of the virtual projection device (72) and a projection target object (74) in the space; determines the position and shape of the virtual projection surface (77) on the basis of the plurality of projection points (75a) to (75h); generates virtual projection image data representing a virtual projection image (91) in which the virtual projection surface (77) is displayed in a spatial image (71) represented by the spatial image data; and outputs the virtual projection image data to an output destination.

Description

Information processing device, information processing method, and information processing program

[0001] The present invention relates to an information processing device, an information processing method, and an information processing program.

[0002] Patent Document 1 describes an installation simulation device that acquires an installation location image of the place where a projector is installed, the installation position of the projector at the installation location, and a projection location image which is the place where the image of the projector is projected or the place where the object onto which the image is projected is installed, and displays one or both of an image in which the image of the projector is drawn at the installation position of the projector on the installation location image and an image in which a projection area image showing the area onto which the image is projected on the projection location image is drawn.

[0003] Patent document 2 describes an information processing device that acquires first information representing the three-dimensional shape of a room having a projection surface onto which a first image is projected, displays a second image showing the three-dimensional shape of the room on a display device based on the first information, acquires second information that specifies the projection range by accepting an operation that determines the projection range of the first image on the projection surface, and displays third information that indicates at least one of the number, model, and installation position in the room of one or more projectors to be installed in the room to project the first image into the projection range, superimposed on the second image based on the first information and the second information.

[0004] Patent document 3 describes a projector that includes a projection unit that projects image light onto the projection surface of a screen, an imaging unit that images an area including the projection surface, a projection control unit that causes the projection unit to project the image light onto the projection surface of the screen on which at least three markers are arranged in one direction along the curved projection surface to form a projection image, a generation unit that causes the imaging unit to image the projection image and the markers to generate the captured image, and an adjustment unit that adjusts the projection position based on the captured image so that the position of the outer edge of the projection area of ​​the projection image approximately coincides with the position of the markers.

[0005] Japanese Patent Publication No. 2023-039885 Japanese Patent Publication No. 2022-126127 Japanese Patent Publication No. 2021-158627

[0006] One embodiment of the technique of the present disclosure provides an information processing device, an information processing method, and an information processing program that are capable of more accurately simulating the projection state of a projection device.

[0007] The information processing device of the present invention is an information processing device equipped with a processor, which acquires first image data obtained by imaging a space, determines the position of a virtual projection device in the space, acquires multiple projection points which are intersections between multiple projection rays of the virtual projection device and an object in the space, determines the position and shape of a virtual projection surface based on the multiple projection points, generates second image data representing a second image in which the virtual projection surface is displayed on a first image represented by the first image data, and outputs the second image data to an output destination.

[0008] The information processing device of the present invention is an information processing device equipped with a processor, which determines the position of a virtual projection device in space, obtains multiple projection points which are intersections between multiple projection rays of the virtual projection device and an object in the space, determines the position and shape of a virtual projection surface based on the multiple projection points, generates image data representing the virtual projection surface, and outputs the image data to a display destination.

[0009] The information processing method of the present invention involves a processor of an information processing device acquiring first image data obtained by capturing an image of a space, determining the position of a virtual projection device in the space, acquiring multiple projection points which are the intersections of multiple projection rays of the virtual projection device with an object in the space, determining the position and shape of a virtual projection surface based on the multiple projection points, generating second image data representing a second image in which the virtual projection surface is displayed on a first image represented by the first image data, and outputting the second image data to an output destination.

[0010] The information processing program of the present invention causes a processor of an information processing device to execute the following process: acquire first image data obtained by capturing an image of a space; determine the position of a virtual projection device in the space; acquire multiple projection points which are the intersections of multiple projection rays of the virtual projection device and an object in the space; determine the position and shape of a virtual projection surface based on the multiple projection points; generate second image data representing a second image in which the virtual projection surface is displayed on a first image represented by the first image data; and output the second image data to an output destination.

[0011] According to the present invention, it is possible to provide an information processing device, an information processing method, and an information processing program that are capable of more accurately simulating the projection state of a projection device.

[0012] 1 is a schematic diagram showing an example of a projection device 10 that is a target for installation support by the information processing device of the embodiment. FIG. 1 is a schematic diagram showing an example of the internal configuration of the projection unit 1 shown in FIG. 1. FIG. 2 is a schematic diagram showing the external configuration of the projection device 10. FIG. 3 is a schematic cross-sectional view of an optical unit 106 of the projection device 10 shown in FIG. 3. FIG. 4 is a diagram showing an example of an information processing device 50 of the embodiment. FIG. 5 is a diagram showing an example of the hardware configuration of the information processing device 50. FIG. 6 is a diagram showing an example of generation of a virtual projection surface. FIG. 7 is a diagram showing an example of a generated virtual projection surface. FIG. 8 is a diagram showing an example of a virtual projection surface generated by increasing the number of projection points. FIG. 9 is a diagram showing an example of a virtual projection surface 77 generated with a virtual object as the projection target. FIG. 10 is a diagram showing an example of a case where a projection point 75 cannot be acquired for a projection ray 73 of a virtual projection device 72. FIG. 11 is a diagram showing an example (first generation example) of generation of a virtual projection surface when a projection point 75 of a projection ray 73 cannot be acquired. FIG. 11 is a diagram showing an example (second generation example) of generation of a virtual projection surface when a projection point 75 of a projection ray 73 cannot be acquired. FIG. 12 is a diagram showing an example (first grouping) of grouping acquired multiple projection points 75. 10 is a diagram showing an example (second grouping) of grouping a plurality of acquired projection points 75. FIG. 11 is a diagram showing a case where projection light rays 73 of a virtual projection device 72 are projected onto the virtual projection device 72 itself. FIG. 12 is a diagram showing an example of viewing an image of the virtual projection device 72 from the viewpoint of an intended viewer. FIG. 13 is a diagram showing an example of correction data 86 for making an image viewed from the viewpoint of an intended viewer have a specific shape. FIG. 14 is a diagram showing a corrected projection plane 177 corrected by correction data. FIG. 15 is a diagram showing an example of changing the position of a recognized projection point 75. FIG. 16 is a diagram showing an example of displaying a virtual projection plane using transmissive AR glasses. FIG. 17 is a diagram showing an example of displaying a lens shift range of the virtual projection device 72.

[0013] Hereinafter, an example of an embodiment of the present invention will be described with reference to the drawings.

[0014] <Projection Device 10 to be Assisted in Installation by Information Processing Apparatus of the Embodiment> FIG. 1 is a schematic diagram showing an example of a projection device 10 to be assisted in installation by an information processing apparatus of the embodiment.

[0015] The information processing apparatus according to the embodiment can be used, for example, to assist in the installation of a projection device 10. The projection device 10 includes a projection unit 1, a control device 4, and an operation reception unit 2. The projection unit 1 is configured, for example, by a liquid crystal projector or a projector using LCOS (Liquid Crystal On Silicon). In the following description, the projection unit 1 is assumed to be a liquid crystal projector.

[0016] The control device 4 is a control device that controls projection by the projection device 10. The control device 4 is a device that includes a control unit configured with various processors, a communication interface (not shown) for communicating with each unit, and a memory 4a such as a hard disk, an SSD (Solid State Drive), or a ROM (Read Only Memory), and controls the projection unit 1 in an integrated manner.

[0017] The various processors in the control unit of the control device 4 include a CPU (Central Processing Unit), which is a general-purpose processor that executes programs to perform various processes, a programmable logic device (PLD), which is a processor whose circuit configuration can be changed after manufacture, such as an FPGA (Field Programmable Gate Array), or a dedicated electrical circuit, such as an ASIC (Application Specific Integrated Circuit), which is a processor having a circuit configuration designed specifically to perform specific processing.

[0018] More specifically, the structure of these various processors is an electric circuit that combines circuit elements such as semiconductor elements. The control unit of the control device 4 may be configured with one of the various processors, or may be configured with a combination of two or more processors of the same or different types (for example, a combination of multiple FPGAs or a combination of a CPU and an FPGA).

[0019] The operation reception unit 2 detects instructions from the user by receiving various operations from the user. The operation reception unit 2 may be a button, key, joystick, or the like provided on the control device 4, or may be a receiving unit or the like that receives signals from a remote controller that remotely controls the control device 4.

[0020] The projection target 6 is an object such as a screen or a wall that has a projection surface on which a projected image is displayed by the projection unit 1. In the example shown in Fig. 1, the projection surface of the projection target 6 is a rectangular plane. The top, bottom, left, and right of the projection target 6 in Fig. 1 are assumed to be the top, bottom, left, and right of the actual projection target 6.

[0021] A projection range 11, shown by a dashed line, is an area of ​​the object 6 onto which projection light is irradiated by the projection unit 1. In the example shown in Fig. 1 , the projection range 11 is rectangular. The projection range 11 is a part or all of the projectable range onto which projection by the projection unit 1 can be performed.

[0022] The projection unit 1, the control device 4, and the operation reception unit 2 may be realized by, for example, a single device (see, for example, FIGS. 3 and 4), or the projection unit 1, the control device 4, and the operation reception unit 2 may be separate devices that communicate with each other and cooperate with each other.

[0023] <Internal Configuration of Projection Unit 1 Shown in FIG. 1> FIG. 2 is a schematic diagram showing an example of the internal configuration of the projection unit 1 shown in FIG.

[0024] As shown in FIG. 2, the projection unit 1 includes a light source 21, a light modulation unit 22, a projection optical system 23, and a control circuit 24.

[0025] The light source 21 includes a light emitting element such as a laser or an LED (Light Emitting Diode), and emits, for example, white light.

[0026] The light modulation unit 22 is composed of three liquid crystal panels that modulate the color light emitted from the light source 21 and separated into three colors, red, blue, and green, by a color separation mechanism (not shown), based on image information, and emit each color image. These three liquid crystal panels may be equipped with red, blue, and green filters, respectively, and the white light emitted from the light source 21 may be modulated by each liquid crystal panel to emit each color image.

[0027] The projection optical system 23 receives light from the light source 21 and the light modulation unit 22, and is configured by, for example, a relay optical system including at least one lens. The light that passes through the projection optical system 23 is projected onto the object 6 to be projected.

[0028] The area of ​​the object 6 that is irradiated with light that passes through the entire range of the light modulation unit 22 is the projectable range in which projection by the projection unit 1 is possible. Within this projectable range, the area that is irradiated with light that actually passes through from the light modulation unit 22 is the projection range 11. For example, by controlling the size, position, and shape of the area of ​​the light modulation unit 22 through which light passes, the size, position, and shape of the projection range 11 can be changed within the projectable range.

[0029] The control circuit 24 controls the light source 21, the light modulation unit 22, and the projection optical system 23 based on the display data input from the control device 4, thereby projecting an image based on this display data onto the projection target 6. The display data input to the control circuit 24 is made up of three pieces of data: red display data, blue display data, and green display data.

[0030] The control circuit 24 also changes the projection optical system 23 based on a command input from the control device 4, thereby enlarging or reducing the projection range 11 (see FIG. 1 ) of the projection unit 1. The control device 4 may also change the projection optical system 23 based on a user operation received by the operation receiving unit 2, thereby moving the projection range 11 of the projection unit 1.

[0031] The projection device 10 also includes a shift mechanism that mechanically or optically moves the projection range 11 while maintaining the image circle of the projection optical system 23. The image circle of the projection optical system 23 is an area through which projection light incident on the projection optical system 23 passes through the projection optical system 23 appropriately in terms of light intensity drop-off, color separation, peripheral curvature, etc.

[0032] The shift mechanism is realized by at least one of an optical system shift mechanism that performs an optical system shift and an electronic shift mechanism that performs an electronic shift.

[0033] The optical system shift mechanism is, for example, a mechanism that moves the projection optical system 23 in a direction perpendicular to the optical axis (see, for example, FIGS. 3 and 4), or a mechanism that moves the light modulation unit 22 in a direction perpendicular to the optical axis instead of moving the projection optical system 23. The optical system shift mechanism may also be a mechanism that moves the projection optical system 23 and the light modulation unit 22 in combination.

[0034] The electronic shift mechanism is a mechanism that shifts the pseudo projection range 11 by changing the range through which light is transmitted in the light modulation section 22 .

[0035] The projection device 10 may also include a projection direction change mechanism that moves the projection range 11 together with the image circle of the projection optical system 23. The projection direction change mechanism is a mechanism that changes the projection direction of the projection unit 1 by changing the orientation of the projection unit 1 through mechanical rotation (see, for example, FIGS. 3 and 4).

[0036] <Mechanical Configuration of Projection Device 10> Fig. 3 is a schematic diagram showing the external configuration of the projection device 10. Fig. 4 is a schematic cross-sectional view of the optical unit 106 of the projection device 10 shown in Fig. 3. Fig. 4 shows a cross section taken along a plane along the optical path of light emitted from the main body 101 shown in Fig. 3.

[0037] 3, the projection device 10 includes a main body 101 and an optical unit 106 that protrudes from the main body 101. In the configuration shown in Fig. 3, the operation reception unit 2, the control device 4, and the light source 21, light modulation unit 22, and control circuit 24 in the projection unit 1 are provided in the main body 101. The projection optical system 23 in the projection unit 1 is provided in the optical unit 106.

[0038] The optical unit 106 includes a first member 102 supported by the main body 101 and a second member 103 supported by the first member 102 .

[0039] The first member 102 and the second member 103 may be an integrated member. The optical unit 106 may be configured to be detachable from the main body 101 (in other words, replaceable).

[0040] The main body 101 has a housing 15 (see FIG. 4) in which an opening 15a (see FIG. 4) for passing light is formed at a portion connected to the optical unit 106.

[0041] As shown in Figure 3, inside the housing 15 of the main body 101, there is provided a light source 21 and a light modulation unit 12 including a light modulation section 22 (see Figure 2) that spatially modulates the light emitted from the light source 21 based on input image data to generate an image.

[0042] The light emitted from the light source 21 is incident on the light modulation section 22 of the light modulation unit 12, and is spatially modulated by the light modulation section 22 before being emitted.

[0043] As shown in Figure 4, the image formed by the light spatially modulated by the light modulation unit 12 passes through the opening 15a of the housing 15 and enters the optical unit 106, and is projected onto the projection target 6, which is the projection object, so that the image G1 becomes visible to the observer.

[0044] As shown in Figure 4, the optical unit 106 comprises a first member 102 having a hollow portion 2A connected to the inside of the main body portion 101, a second member 103 having a hollow portion 3A connected to the hollow portion 2A, a first optical system 121 and a reflecting member 122 arranged in the hollow portion 2A, a second optical system 31, a reflecting member 32, a third optical system 33, and a lens 34 arranged in the hollow portion 3A, a shift mechanism 105, and a projection direction change mechanism 104.

[0045] The first member 102 is a member having, for example, a rectangular cross-sectional outer shape, and the openings 2a and 2b are formed on planes perpendicular to each other. The first member 102 is supported by the main body 101 with the opening 2a positioned opposite the opening 15a of the main body 101. Light emitted from the light modulation section 22 of the light modulation unit 12 of the main body 101 passes through the openings 15a and 2a and enters the hollow section 2A of the first member 102.

[0046] The direction of light incident from the main body 101 into the hollow portion 2A is referred to as direction X1, the opposite direction to direction X1 is referred to as direction X2, and directions X1 and X2 are collectively referred to as direction X. In addition, in Figure 4, the direction from the front of the page toward the back and the opposite direction are referred to as direction Z. Of direction Z, the direction from the front of the page toward the back is referred to as direction Z1, and the direction from the back of the page toward the front is referred to as direction Z2.

[0047] Furthermore, a direction perpendicular to direction X and direction Z is referred to as direction Y, and within direction Y, the upward direction in Fig. 4 is referred to as direction Y1, and the downward direction in Fig. 4 is referred to as direction Y2. In the example of Fig. 4, the projection device 10 is disposed so that direction Y2 is the vertical direction.

[0048] The projection optical system 23 shown in Fig. 2 is composed of a first optical system 121, a reflecting member 122, a second optical system 31, a reflecting member 32, a third optical system 33, and a lens 34. Fig. 4 shows the optical axis K of the projection optical system 23. The first optical system 121, the reflecting member 122, the second optical system 31, the reflecting member 32, the third optical system 33, and the lens 34 are arranged along the optical axis K in this order from the light modulation unit 22 side.

[0049] The first optical system 121 includes at least one lens, and guides light traveling in the direction X1 that is incident on the first member 102 from the main body 101 to the reflecting member 122 .

[0050] The reflecting member 122 reflects the light incident from the first optical system 121 in the direction Y1. The reflecting member 122 is formed of, for example, a mirror. The first member 102 has an opening 2b formed on the optical path of the light reflected by the reflecting member 122. The reflected light passes through the opening 2b and travels to the hollow portion 3A of the second member 103.

[0051] The second member 103 is a member having a substantially T-shaped cross-sectional outer shape, and has an opening 3a formed in a position facing the opening 2b of the first member 102. Light from the main body 101 that passes through the opening 2b of the first member 102 passes through this opening 3a and enters the hollow portion 3A of the second member 103. The cross-sectional outer shapes of the first member 102 and the second member 103 are arbitrary and are not limited to those described above.

[0052] The second optical system 31 includes at least one lens, and guides the light incident from the first member 102 to the reflecting member 32 .

[0053] The reflecting member 32 reflects the light incident from the second optical system 31 in the direction X2 and guides the light to the third optical system 33. The reflecting member 32 is formed of, for example, a mirror.

[0054] The third optical system 33 includes at least one lens, and guides the light reflected by the reflecting member 32 to a lens 34 .

[0055] The lens 34 is disposed at the end of the second member 103 in the direction X2 so as to cover the opening 3c formed at this end. The lens 34 projects the light incident from the third optical system 33 onto the object 6 to be projected.

[0056] The projection direction change mechanism 104 is a rotation mechanism that rotatably connects the second member 103 to the first member 102. This projection direction change mechanism 104 allows the second member 103 to rotate freely around a rotation axis (specifically, optical axis K) that extends in direction Y. Note that the projection direction change mechanism 104 is not limited to the arrangement position shown in Fig. 4 as long as it can rotate the optical system. Furthermore, the number of rotation mechanisms is not limited to one, and multiple mechanisms may be provided.

[0057] The shift mechanism 105 is a mechanism for moving the optical axis K of the projection optical system (in other words, the optical unit 106) in a direction perpendicular to the optical axis K (direction Y in FIG. 4 ). Specifically, the shift mechanism 105 is configured to be able to change the position of the first member 102 in direction Y relative to the main body 101. The shift mechanism 105 may be one that moves the first member 102 manually, or one that moves the first member 102 electrically.

[0058] Fig. 4 shows a state in which the first member 102 has been moved to the maximum extent in the direction Y1 by the shift mechanism 105. When the first member 102 is moved in the direction Y2 by the shift mechanism 105 from the state shown in Fig. 4, the relative position between the center of the image formed by the light modulation unit 22 (in other words, the center of the display surface) and the optical axis K changes, and the image G1 projected onto the projection object 6 can be shifted (translated) in the direction Y2.

[0059] The shift mechanism 105 may be a mechanism that moves the light modulation unit 22 in the direction Y, instead of moving the optical unit 106 in the direction Y. Even in this case, the image G1 projected onto the projection target 6 can be moved in the direction Y2.

[0060] <Information Processing Device 50 of the Embodiment> FIG. 5 is a diagram illustrating an example of an information processing device 50 of the embodiment. The information processing device 50 of the embodiment is a tablet terminal or the like having a touch panel 51. The touch panel 51 is a display that can be touched. For example, when performing projection using the projection device 10 in a space such as a room, the information processing device 50 is used to find an appropriate installation position for the projection device 10 and the projection target 6 onto which light from the projection device 10 is to be projected. A user of the information processing device 50 brings the information processing device 50 into a space (room) where projection using the projection device 10 will be performed. The information processing device 50 displays an installation assistance image on the touch panel 51 to assist in installing the projection device 10 and the projection target 6 in the space.

[0061] For example, the information processing device 50 displays, as an installation assistance image, a second image in which an image of a virtual projection surface, which is a virtual projection surface, and an image of a virtual projection device, which is a virtual projection device, are superimposed on a first image obtained by capturing an image of a space in which the projection device 10 is installed and projection is to be performed. The user of the information processing device 50 can obtain information regarding the installation of the projection device 10 and the projection target 6 while referring to the installation assistance image.

[0062] <Hardware configuration of information processing device 50> Fig. 6 is a diagram showing an example of the hardware configuration of the information processing device 50. The information processing device 50 shown in Fig. 5 includes, for example, a processor 61, a memory 62, a communication interface 63, a user interface 64, and a sensor 65, as shown in Fig. 6. The processor 61, the memory 62, the communication interface 63, the user interface 64, and the sensor 65 are connected by, for example, a bus 69.

[0063] The processor 61 is a circuit that performs signal processing, and is, for example, a CPU that controls the entire information processing device 50. The processor 61 may be realized by other digital circuits such as an FPGA or a DSP (Digital Signal Processor). The processor 61 may also be realized by combining multiple digital circuits.

[0064] The memory 62 includes, for example, a main memory and an auxiliary memory. The main memory is, for example, a RAM (Random Access Memory). The main memory is used as a work area for the processor 61.

[0065] The auxiliary memory is a non-volatile memory such as a magnetic disk, a flash memory, etc. The auxiliary memory stores various programs that operate the information processing device 50. The programs stored in the auxiliary memory are loaded into the main memory and executed by the processor 61.

[0066] The auxiliary memory may also include a portable memory that is removable from the information processing device 50. Portable memories include memory cards such as a USB (Universal Serial Bus) flash drive or an SD (Secure Digital) memory card, and an external hard disk drive.

[0067] The communication interface 63 is a communication interface for communicating with devices external to the information processing device 50. The communication interface 63 includes at least one of a wired communication interface for communicating via a wire and a wireless communication interface for communicating wirelessly. The communication interface 63 is controlled by the processor 61.

[0068] The user interface 64 includes, for example, an input device that accepts operation input from the user and an output device that outputs information to the user. The input device can be realized by, for example, keys (e.g., a keyboard) or a remote control. The output device can be realized by, for example, a display or a speaker. In the information processing device 50 shown in FIG. 5 , the input device and the output device are realized by the touch panel 51. The user interface 64 is controlled by the processor 61. The information processing device 50 accepts various specifications from the user using the user interface 64.

[0069] The sensor 65 includes an imaging device that has an imaging optical system and an imaging element and is capable of capturing images, and a space recognition sensor that is capable of three-dimensionally recognizing the space around the information processing device 50. The imaging device includes, for example, the imaging device provided on the back surface of the information processing device 50 shown in FIG.

[0070] An example of the spatial recognition sensor is a LIDAR (Light Detection and Ranging) that emits laser light, measures the time it takes for the emitted laser light to hit an object and bounce back, and measures the distance and direction to the object. However, the spatial recognition sensor is not limited to this, and various sensors such as a radar that emits radio waves and an ultrasonic sensor that emits ultrasonic waves can be used.

[0071] <First Mode of Processing by Information Processing Device 50> A first mode of processing by the information processing device 50 will be described with reference to Fig. 7 and Fig. 8. Fig. 7 is a diagram showing an example of generation of a virtual projection surface. Fig. 8 is a diagram showing an example of the generated virtual projection surface.

[0072] In generating the virtual projection surface, the processor 61 of the information processing device 50 acquires spatial image data representing a spatial image 71 of the physical space in which the projection device 10 is installed and projection is performed by the projection device 10. The "spatial image 71" may be, for example, a captured image of the space captured by an imaging device, or an image generated from a 3D (dimensional) model of the space.

[0073] The imaging device that captures the space may be an imaging device that is integrated with the information processing device 50, or may be an external imaging device. The spatial image data is, for example, imaging data of the space obtained by imaging with the imaging device, or image data generated from a 3D model. The spatial image 71 is an example of a "first image" in the present invention. The spatial image data is an example of "first image data" in the present invention.

[0074] 7 , the processor 61 determines the position of a virtual projection device 72, which is a virtual projection device, in the space. The processor 61 also acquires a projection point 75, which is the intersection of a projection ray 73 projected from the virtual projection device 72 and a projection target object 74 in the space. The position of the virtual projection device 72 may be determined based on an instruction from the user of the information processing device 50, or may be determined based on an analysis process of the space image 71.

[0075] The projection light ray 73 projected from the virtual projection device 72 is, for example, a linear light beam that passes through any one point on a rectangular virtual reference projection plane 76 generated as seen from the virtual projection device 72. The projection point 75 shown in FIG. 7 is the point where the projection light ray 73 from the virtual projection device 72 passes through a point 76a on the virtual reference projection plane 76 and intersects with the projection object 74. The projection object 74 may be an object present in space detected by spatial recognition, or it may be a virtual object. In this example, the projection object 74 is the wall surface of a room formed in a curved shape. The projection object 74 is an example of the "object" of the present invention.

[0076] As shown in FIG. 8 , the processor 61 acquires a plurality of projection points 75 (in this example, projection points 75a to 75h) which are the intersections between a plurality of projection rays 73 (in this example, projection rays 73a to 73h) projected from a virtual projection device 72 and a projection object 74. The processor 61 determines the position and shape of a virtual projection plane 77 based on the acquired plurality of projection points 75a to 75h. The "plurality of projection points" refers to the number of projection points necessary to determine the virtual projection plane 77, and is, for example, four or more projection points. Determining the position and shape of the virtual projection plane 77 means determining the plurality of projection points 75a to 75h by, for example, polygon processing or by approximation processing using a polynomial surface.

[0077] The processor 61 is capable of executing a plane determination process that determines the position and shape of a virtual projection plane assuming that the projection surface of the projection object 74 is planar, and a non-planar determination process that determines the position and shape of a virtual projection plane assuming that the projection surface of the projection object 74 is non-planar. In the non-planar determination process, the processor 61 determines the position and shape of the virtual projection plane 77 based on a plurality of projection points 75. The plane determination process is an example of a "first process" in the present invention. The non-planar determination process is an example of a "second process" in the present invention.

[0078] The processor 61 generates a virtual projection image 91 based on the virtual projection device 72 and the virtual projection surface 77. The virtual projection image 91 is an image in which the virtual projection device 72 and the virtual projection surface 77 are displayed on the spatial image 71 represented by the spatial image data. For example, the virtual projection image 91 is an image in which the virtual projection device 72 and the virtual projection surface 77 are superimposed on the spatial image 71 represented by the spatial image data. The virtual projection image 91 is an example of a "second image" in the present invention.

[0079] The processor 61 acquires virtual projection image data representing the virtual projection image 91. The processor 61 outputs the acquired virtual projection image data to a predetermined output destination. The output destination of the virtual projection image data may be, for example, a display unit (touch panel 51) provided in the information processing device 50, or an external display device. This makes it possible to display the virtual projection image 91 represented by the virtual projection image data. The virtual projection image data is an example of the "second image data" in the present invention.

[0080] As described above, the information processing device 50 of the first embodiment displays a virtual projection image 91 in which a virtual projection surface 77 determined by a plurality of projection points 75 is displayed on the spatial image 71 onto which projection is performed by the projection device 10, and outputs the virtual projection image 91 to the touch panel 51 of the information processing device 50. According to this configuration, when a user who performs projection using the virtual projection device 72 installs the virtual projection device 72 at a certain position in the installation space, it is possible to easily and more accurately simulate the shape of the virtual projection surface 77 of the virtual projection device 72 projected onto the projection target object 74.

[0081] The processor 61 determines the position and shape of the virtual projection surface 77 based on, for example, spatial image data representing the spatial image 71 on which the virtual projection device 72 and the virtual projection surface 77 are superimposed, but the processor 61 may also determine the position and shape of the virtual projection surface 77 based on other spatial image data (for example, spatial image data obtained by capturing an image at a different time or spatial image data obtained by capturing an image with a different imaging device).

[0082] <Second Mode of Processing by Information Processing Device 50> A second mode of processing by the information processing device 50 will be described with reference to Fig. 9. Fig. 9 is a diagram showing an example of a virtual projection plane generated by increasing the number of projection points.

[0083] When determining the virtual projection surface 77, the processor 61 determines the number of projection points 75 to be acquired based on the distance L between the virtual projection device 72 and the projection object 74. The distance between the virtual projection device 72 and the projection object 74 may be, for example, the distance on the optical axis of the virtual projection device 72, or a representative value of the distances on multiple projection light rays 73 (e.g., average value, mode, minimum value, maximum value, etc.). Determining the number of projection points 75 based on the distance L means that the longer the distance L, the larger the size of the determined virtual projection surface 77, so the number of projection points 75 is increased to ensure a virtual projection surface 77 with an accurate shape. In this example, the virtual projection surface 77 is determined using 21 acquired projection points 75, three vertically and seven horizontally.

[0084] Furthermore, when determining the virtual projection plane 77, the processor 61 may determine the number of projection points 75 to be acquired based on the results of space recognition. Determining the number of projection points 75 based on the results of space recognition means, for example, increasing the number of projection points 75 to be acquired the more undulating the recognized space (the more complex the space). The number may be determined by increasing the number of projection points 75 the more undulating the space is, or by comprehensively considering the number and size of the undulations. Alternatively, the number may be determined by first determining the virtual projection plane 77 with an appropriate number of projection points 75, and then determining an appropriate number of projection points 75 based on the degree of correspondence between the determined virtual projection plane 77 and the recognized space. The recognized space may be limited, for example, to the area in the projection direction in which the virtual projection device 72 projects the projection light beam 73.

[0085] The actual projection device 10 emits countless projection light rays 73. Increasing the number of projection points 75 used in the simulation makes the shape of the virtual projection surface 77 closer to the projection surface of the actual projection device 10, but increases the processing load of the processor 61, leading to a decrease in the display frame rate and response to operations, and increased battery consumption. In contrast, the information processing device 50 of the second embodiment makes it possible to change the number of projection points 75 that determine the virtual projection surface 77 depending on the distance L between the virtual projection device 72 and the projection target object 74 or the complexity of the recognized space. This makes it possible to improve the reproducibility of the shape of the projection surface while maintaining the responsiveness of the processor 61.

[0086] Furthermore, when determining the virtual projection surface 77, the projection device 10 may determine the number of projection points 75 to be acquired in response to a user operation. That is, when the user performs an operation that requires recalculation of the shape of the virtual projection surface 77 as a result of a movement-related operation, such as changing the position of the virtual projection device 72 or the virtual projection surface 77 or changing the aspect ratio of the virtual projection surface 77, the projection device 10 may temporarily change the number of projection points 75 to be acquired. For example, the projection device 10 may leave only the portion corresponding to the outer frame of the projection surface and delete the other projection points to determine the number of projection points 75 that allow the general shape of the virtual projection surface 77 to be determined. When it is determined that the user has finished the movement-related operation, for example, when a specified time has elapsed since the last change in the position of the virtual projection device 72 or the virtual projection surface 77, the projection device 10 recalculates the shape of the virtual projection surface 77 using the original number of projection points 75.

[0087] When the spatial structure is complex, it is necessary to increase the number of projection points 75 in order to reproduce the shape of the projection surface. On the other hand, if all projection points 75 are re-determined every time the user slightly changes the position of the virtual projection device 72 or the virtual projection surface 77, the delay until the results of the user operation are reflected increases. In contrast, as described above, when the user is performing continuous operations, by temporarily reducing the number of projection points 75 to an extent that allows the general shape of the virtual projection surface 77 to be seen, the user can roughly grasp the changes in the virtual projection surface 77 caused by the operations and perform the operations comfortably.

[0088] Furthermore, when determining the virtual projection plane 77, the projection device 10 may determine the number of projection points 75 to be acquired in accordance with a user operation to specify the number of projection points 75. The user operation to specify the number of projection points 75 may be an operation to directly specify the number of projection points 75, or an operation to increase or decrease the current number of projection points 75.

[0089] <Third Mode of Processing by Information Processing Device 50> A third mode of processing by the information processing device 50 will be described with reference to Fig. 10. Fig. 10 is a diagram showing an example of a virtual projection surface 77 on which a virtual object is generated as a projection object.

[0090] As shown in FIG. 10 , the processor 61 acquires multiple (eight in this example) projection points 75, which are the intersections of multiple (eight in this example) projection rays 73 projected from the virtual projection device 72 and a virtual projection object 81, and determines a virtual projection plane 77 based on the acquired projection points 75. The virtual projection object 81 is a virtual object arranged as if it exists in a virtual space representing the projection space of the projection device 10. The virtual projection object 81 is an object that can be arranged based on, for example, an instruction (operation) from a user of the information processing device 50. The virtual projection object 81 in this example is, for example, a movable installed object having a cylindrical shape. The virtual projection object 81 is an example of an "object" in the present invention.

[0091] According to the third form of information processing device 50, the virtual projection object 81 can be considered as a spatial structure, so that the projection object before construction can be virtually placed in the space and a projection simulation can be performed assuming a space that matches the user's intentions.

[0092] <Fourth Mode of Processing by Information Processing Device 50> A fourth mode of processing by the information processing device 50 will be described with reference to Fig. 11. Fig. 11 is a diagram showing an example of a case where a projection point 75 cannot be acquired on a projection ray 73 of a virtual projection device 72.

[0093] The information processing device 50 can recognize structures in space, for example, by using an imaging device mounted on the information processing device 50, but there is a limit to the range of recognition. For this reason, even if a structure actually exists that intersects with the projection ray 73 from the virtual projection device 72, there may be cases where the projection point 75 cannot be obtained because it is outside the recognition range of the imaging device.

[0094] 11 , when attempting to acquire the intersections of a plurality of projection light rays 73a to 73h projected from a virtual projection device 72 and a projection object 74, the imaging device of the information processing device 50 is unable to recognize a partial region 74p of the projection object 74. In this case, it is possible to acquire projection points 75a to 75c and 75e to 75h, which are the intersections of the projection light rays 73a to 73c and 73e to 73h and the projection object 74, but it is not possible to acquire the projection point of the projection light ray 73d that intersects with the region 74p of the projection object 74.

[0095] Therefore, when the processor 61 of the information processing device 50 recognizes that there is a projection ray 73 for which a projection point 75 cannot be obtained among the multiple projection rays 73, the processor 61 notifies the user of the recognition result. For example, as shown in Fig. 11, the processor 61 displays the path of the projection ray 73d for which a projection point cannot be obtained with a thicker line than the other projection rays 73a to 73c and 73e to 73h.

[0096] According to the information processing device 50 of the fourth embodiment, by notifying the user that the projection points 75 cannot be acquired, the user can be prompted to take measures such as changing the installation position of the virtual projection device 72, moving the position of the information processing device 50 so that the spatial structure can be recognized, or increasing the number of projection points 75 that determine the virtual projection surface 77. This makes it possible to acquire the projection points 75 of the area that could not be acquired.

[0097] <Fifth Form of Processing by Information Processing Device 50> A fifth form of processing by the information processing device 50 will be described with reference to Fig. 12 and Fig. 13. Fig. 12 is a diagram showing an example (first generation example) of generating a virtual projection plane when the projection point 75 of the projection ray 73 cannot be obtained. Fig. 13 is a diagram showing an example (second generation example) of generating a virtual projection plane when the projection point 75 of the projection ray 73 cannot be obtained.

[0098] 12, when attempting to acquire the intersections of projection rays 73a to 73h of a virtual projection device 72 and a projection object 74, the information processing device 50 is unable to recognize a partial region 74p of the projection object 74. In this case, it is possible to acquire projection points 75a to 75c and 75e to 75h, which are the intersections of projection rays 73a to 73c and 73e to 73h and the projection object 74, but it is not possible to acquire the projection point of projection ray 73d that intersects with region 74p of the projection object 74. This state is similar to the state described in FIG.

[0099] Therefore, when there is a projection ray 73 for which a projection point 75 cannot be obtained among the multiple projection rays 73, the processor 61 of the information processing device 50 excludes the projection point 75 for the projection ray 73 for which a projection point 75 cannot be obtained and determines the position and shape of the virtual projection plane 77 based on the obtained projection points 75. Specifically, as shown in Fig. 12, the processor 61 determines the position and shape of the virtual projection plane 77 based on the projection points 75a to 75c and 75e to 75h of the projection rays 73a to 73c and 73e to 73h for which a projection point can be obtained, without considering the projection point of the projection ray 73d for which a projection point cannot be obtained.

[0100] If there are projection points 75 that cannot be obtained, the virtual projection surface 77 can be created by excluding the projection points 75 that could not be obtained, and it is possible to display a virtual projection surface 77 with a shape close to the actual projection even in a space where it is difficult to recognize the structure.

[0101] 13 , when attempting to acquire the intersections of projection light rays 73a to 73h of the virtual projection device 72 and the projection object 74, it is assumed that the range of space recognition possible by the information processing device 50 is insufficient and it is not possible to recognize a space portion 82 including a right-half region 74q of the projection object 74. In this case, it is possible to acquire projection points 75a, 75b, 75g, and 75h which are the intersections of the projection light rays 73a, 73b, 73g, and 73h and the projection object 74, but it is not possible to acquire the projection points of the projection light rays 73c, 73d, 73e, and 73f which intersect with the region 74q of the projection object 74.

[0102] Therefore, when there is a projection ray 73 for which a projection point 75 cannot be obtained among the multiple projection rays 73, the processor 61 determines the position and shape of the virtual projection plane 77 based on the projection point 75, which is the intersection of the virtual projection object based on the spatial recognition result and the projection ray 73 for which a projection point 75 cannot be obtained, and the obtained projection point 75. The virtual projection object based on the spatial recognition result is, for example, a virtual projection object generated based on a portion of the projection object 74 that can be recognized, and corresponds to the unrecognized portion.

[0103] 13 , the processor 61 generates an area 74q of the right half of the projection object 74 as a virtual projection object 84 based on the position and shape of the left half of the projection object 74 that has been spatially recognized. The processor 61 acquires projection points 85c, 85d, 85e, and 85f, which are the intersections of the generated virtual projection object 84 and projection light rays 73c, 73d, 73e, and 73f. The processor 61 determines the position and shape of the virtual projection plane 77 based on the projection points 85c, 85d, 85e, and 85f, which are the intersections with the virtual projection object 84, and the projection points 75a, 75b, 75g, and 75h, which are the intersections with the projection object 74.

[0104] When there is a projection point 75 where a part of the projection object 74 cannot be recognized and cannot be acquired, a virtual projection object 84 is generated based on the projection object 74 that can be recognized, so that a virtual projection surface 77 with a shape close to the actual projection can be displayed even for a spatial structure that is difficult to recognize.

[0105] (Third generation example) Although not shown in the figure, if there is a projection ray 73 among multiple projection rays 73 for which a projection point 75 cannot be obtained, the processor 61 may determine the position and shape of the virtual projection surface 77, for example, as follows.

[0106] The processor 61 may determine the position and shape of the virtual projection plane 77 based on a projection point 75 that is the intersection of a plane based on a projection point 75 that can be acquired among the multiple projection rays 73 and a projection ray 73 for which a projection point 75 cannot be acquired, and the acquired projection point 75. The plane based on the acquired projection point 75 is a plane with the same orientation as the plane created using the acquired projection point 75, i.e., a tangent plane of the projection object 74 at the position of the acquired projection point 75.

[0107] For example, the processor 61 generates a tangent plane of the projection object 74 at the position of the acquired projection point 75. The processor 61 acquires a projection point 75, which is the intersection of the generated tangent plane and a projection ray 73 for which a projection point 75 cannot be acquired. The processor 61 determines the position and shape of the virtual projection plane 77 based on the projection point 75, which is the intersection with the tangent plane, and the projection point 75, which is the intersection with the projection object 74. Even when the tangent plane of the projection object 74 is generated at the position of the acquired projection point 75, as in this generation example, it is possible to display a virtual projection plane 77 with a shape close to the actual projection.

[0108] <Sixth Form of Processing by Information Processing Device 50> A sixth form of processing by the information processing device 50 will be described with reference to Fig. 14 and Fig. 15. Fig. 14 is a diagram showing an example (first grouping) of grouping the acquired multiple projection points 75. Fig. 15 is a diagram showing an example (second grouping) of grouping the acquired multiple projection points 75.

[0109] The processor 61 of the information processing device 50 groups the acquired multiple projection points 75 based on a predetermined criterion, and determines the position and shape of the virtual projection surface 77 corresponding to the group. "Corresponding to the group" means, for example, for each group. The processor 61 generates multiple virtual projection surfaces 77 of different types for each group.

[0110] (First Grouping) When generating the virtual projection surface 77, for example, as shown in FIG. 14, projection points 75a to 75h are acquired, which are the intersections of projection rays 73a to 73h of the virtual projection device 72 and the projection object 74 (including projection object portions 74A and 74B).

[0111] The processor 61 groups the acquired projection points 75a to 75h based on the spatial recognition result. "Based on the spatial recognition result" means, for example, when two types of projection target portions 74A and 74B having a step in the front-to-back direction as shown in Figure 14 can be recognized. The processor 61 groups the acquired projection points 75a to 75h into projection points 75a, 75b, 75g, and 75h acquired on the projection target portion 74A and projection points 75c, 75d, 75e, and 75f acquired on the projection target portion 74B.

[0112] For example, the processor 61 classifies projection points 75a, 75b, 75g, and 75h, which are the intersections of the projection target portion 74A and the projection rays 73a, 73b, 73g, and 73h, as the first group 75A, and classifies projection points 75c, 75d, 75e, and 75f, which are the intersections of the projection target portion 74B and the projection rays 73c, 73d, 73e, and 73f, as the second group 75B.

[0113] (Second Grouping) When generating the virtual projection plane 77, for example, it is assumed that projection points 75i to 75o are acquired as intersections with the projection ray, as shown in Fig. 15. Note that Fig. 15 is a top view of the projection points 75i to 75o as seen from above (a direction perpendicular to the projection direction).

[0114] In this case, the processor 61 groups the acquired multiple projection points 75i to 75o based on the normal vectors based on the projection points 75i to 75o and the projection object 74. For example, the processor 61 obtains a normal vector 131 at the projection point 75l, which is one point on the projection object 74, and creates a surface 132 represented by this normal vector 131. The processor 61 also detects the distance between the surface 132 represented by the normal vector 131 and each projection point, and classifies the projection points 75k, 75l, and 75m that are close to the surface 132 into a third group 75C.

[0115] Furthermore, the processor 61 determines a normal vector 133 at a projection point 75j, which is one point on the projection object 74, and creates a surface 134 represented by this normal vector 133. Similarly, the processor 61 detects the distance between the surface 134 represented by the normal vector 133 and each projection point, and classifies the projection points 75i, 75j, 75n, and 75o that are close to the surface 134 into a fourth group 75D.

[0116] 14, for example, when the processor 61 cannot recognize two types of projection target portions 74A, 74B having a step in the front-to-rear direction, the processor 61 may group the acquired projection points 75a to 75h based on the distance between the acquired projection points 75a to 75h. The distance between the projection points 75a to 75h is, for example, the distance between adjacent projection points. The processor 61 groups the projection points 75a to 75h according to the distance between adjacent projection points.

[0117] Specifically, the processor 61 detects the distance between adjacent projection points 75a and 75b, and the distance between adjacent projection points 75c and 75d. The processor 61 groups the projection points 75a and 75b, which are close to each other, in the same group, and groups the projection points 75c and 75d, which are far from each other. Similarly, the processor 61 detects the distance between adjacent projection points 75g and 75h, and the distance between adjacent projection points 75e and 75f. The processor 61 groups the projection points 75g and 75h, which are close to each other, in the same group as the projection points 75a and 75b, and groups the projection points 75e and 75f, which are far from each other, in the same group as the projection points 75c and 75d. The processor 61 groups the projection points 75a, 75b, 75g, and 75h into a first group 75A, and groups the projection points 75c, 75d, 75e, and 75f into a second group 75B.

[0118] (Fourth Grouping) The processor 61 may group the acquired multiple projection points 75 into projection points 75 corresponding to projection objects 74 that are objects existing in the space, and projection points 75 corresponding to virtual projection objects 81 that are placed in a virtual space that represents the space. For example, when the projection object 74 and the virtual projection object 81 are set as objects in the space, the processor 61 may group the acquired multiple projection points 75 into a group including projection points 75 that are intersections with the projection object 74 shown in Fig. 8 and a group including projection points 75 that are intersections with the virtual projection object 81 shown in Fig. 10.

[0119] (Fifth Grouping) Based on a user instruction, the processor 61 may group the acquired multiple projection points 75. For example, based on a user instruction, the processor 61 may treat the projection points 75 arbitrarily selected by the user as the same group.

[0120] The projection object 74 onto which the projection light ray 73 is projected is not necessarily a continuous surface (e.g., a wall surface), but may have a protruding object such as a pillar along the surface. In such cases, it is desirable to minimize projection onto the pillar portion and to project the important portion of the image onto the wall. Therefore, as in this embodiment, by grouping the projection points 75 that determine the virtual projection surfaces 77 in different states, it is possible to accurately determine the installation position and orientation of the virtual projection device 72, thereby enabling the projection simulation to more closely match the user's intentions.

[0121] <Seventh Mode of Processing by Information Processing Device 50> A seventh mode of processing by the information processing device 50 will be described with reference to Fig. 16. Fig. 16 is a diagram showing a case where a projection ray 73 of a virtual projection device 72 is projected onto the virtual projection device 72 itself.

[0122] When at least one of the plurality of projection light rays 73 intersects with a plurality of projection objects 74 in space, the processor 61 of the information processing device 50 groups the plurality of acquired projection points 75 according to the corresponding projection objects 74. The plurality of projection objects 74 include, for example, a wall surface that is a target object for projection, and the body of the virtual projection device 72 itself that projects the projection light rays 73.

[0123] Among the projection light rays 73 projected from the virtual projection device 72 toward the projection object 74, some projection light rays 73 reach the projection object 74 directly as they are, and some projection light rays 73 intersect with an object that exists between the virtual projection device 72 and the projection object 74. For example, as shown in Fig. 16, among the projection light rays 73p to 73x projected from the virtual projection device 72, there are projection light rays 73p to 73u that reach the projection object 74 directly as they are, and projection light rays 73v to 73x that are projected (eclipsed) by a part of the main body 72a of the virtual projection device 72 before reaching the projection object 74.

[0124] In this case, the processor 61 classifies projection points 75p to 75u, which are the intersections of projection light rays 73p to 73u that directly reach the target projection object 74 and the projection object 74, as a fifth group 75E. The processor 61 also classifies projection points 175v to 175x, which are the intersections of projection light rays 73v to 73x that are projected onto a part of the virtual projection device 72 and the virtual projection device 72, as a sixth group 175F. The processor 61 also classifies projection points 75v to 75x, which are the intersections of projection light rays 73v to 73x that would have been projected onto the target projection object 74 if they had not been hit by the virtual projection device 72, as a seventh group 75F. The processor 61 determines multiple intersections (intersections with the virtual projection device 72 and the projection object 74) for one projection light (projection ray 73v to 73x), obtains multiple corresponding projection points (projection points 175v to 175x, projection points 75v to 75x), and groups them into separate groups.

[0125] The processor 61 generates virtual projection image data representing a virtual projection image 91 in which the virtual projection surface 77 is displayed in a manner according to the group. "In a manner according to the group" means, for example, differentiating the groups in different ways. For example, different colors are used according to the groups, or different display shapes of the projection points 75 are used according to the groups.

[0126] According to the information processing device 50 of the seventh aspect, the projection points 75 of the projection light rays 73 that are projected onto another object before reaching the target projection object 74 can be grouped into a different group from the projection points 75 of the projection light rays 73 that are projected directly onto the target projection object 74, and can be displayed in a manner corresponding to the group. Therefore, it is possible to perform a simulation while grasping, for example, how the virtual projection plane 77 determined on the projection object 74 is chipped.

[0127] <Eighth Form of Processing by Information Processing Device 50> The eighth form of processing by the information processing device 50 will be described with reference to Figs. 17 to 19. Fig. 17 is a diagram showing an example of viewing an image from the virtual projection device 72 from the viewpoint of an intended viewer. Fig. 18 is a diagram showing an example of correction data 86 for making the image viewed from the viewpoint of the intended viewer have a specific shape. Fig. 19 is a diagram showing a corrected projection plane 177 corrected by the correction data.

[0128] 17 , it is assumed that projection rays 73 (e.g., projection rays 73a, 73d, 73e, and 73h) are projected from a virtual projection device 72 toward a projection object 74, and projection points 75 (e.g., projection points 75a, 75d, 75e, and 75h) are obtained as intersections of the projection rays 73 and the projection object 74. It is also assumed that an assumed viewer M, who is located to the left of the virtual projection device 72, is viewing the image projected from the virtual projection device 72.

[0129] If the shape of the projection object 74 is non-planar, the image will appear distorted when viewed by a viewer positioned to the left or right of the front of the projection object 74 (for example, the position of the intended viewer M). Therefore, the information processing device 50 performs processing to correct the image projected from the virtual projection device 72 onto the projection object 74 so that an undistorted image is viewed from the position of the intended viewer M.

[0130] Specifically, the processor 61 of the information processing device 50 generates correction data 86 (see FIG. 18 ) for correcting the image to be projected from the virtual projection device 72 onto the virtual projection surface 77, based on the position and shape of the virtual projection surface 77 determined by the multiple projection points 75 a, 75 d, 75 e, 75 h, etc. The correction data 86 is correction data for correcting the image projected onto the virtual projection surface 77 so that it appears in a specific shape from a certain viewpoint in space. The viewpoint is, for example, the viewpoint of an assumed viewer M assumed in space. The specific shape is the shape (e.g., rectangular) of the image that the virtual projection device 72 is attempting to project onto the projection target object 74.

[0131] When generating the correction data 86, the processor 61 generates virtual projection image data of a virtual projection image 91 in which a rectangular apparent projection surface 176 that appears at a certain distance from the viewpoint of the intended viewer M is displayed. The certain distance can be set by the user. Therefore, the size of the apparent projection surface 176 can be changed according to the certain distance set by the user. The virtual projection image 91 may display not only the apparent projection surface 176 but also the virtual projection surface 77. The apparent projection surface 176 is an example of the "second virtual projection surface" of the present invention.

[0132] The processor 61 determines points 175a to 175i at which lines 173a to 173i, which extend from the viewpoint of the intended viewer M and pass through points on the apparent projection surface 176, intersect with the projection object 74. The points 175a to 175i are points that determine a corrected projection surface 177 (see FIG. 19) on the projection object 74 that corresponds to the rectangular apparent projection surface 176 from the viewpoint of the intended viewer M.

[0133] The correction data 86 is correction data that corrects the position of the virtual projection plane 77 projected from the virtual projection device 72 and generated on the projection object 74 so that it coincides with the position of the corrected projection plane 177 on the projection object 74 as seen by the intended viewer M. Here, among the points 175a to 175i, the correction data is considered for points 175a, 175c, 175i, and 175g that correspond to the projection points 75a, 75d, 75e, and 75h that are the intersections of the above-mentioned projection light rays 73a, 73d, 73e, and 73h with the projection object 74.

[0134] The correction data 86 is created based on the correspondence between the pixels of the image and the coordinates on the virtual projection plane 77. If the coordinate values ​​of the projection points 75a, 75d, 75e, and 75h are (0,0), (1,0), (1,1), and (0,1), the coordinate values ​​of the corresponding points 175a, 175c, 175i, and 175g are (0.18, 0.08), (0.82, 0.08), (0.82, 0.93), and (0.18, 0.93). Therefore, the correction data values ​​for matching the position of the virtual projection plane 77 to the position of the corrected projection plane 177 are (0.18, 0.08) for the projection point 75a on the virtual projection plane 77, (0.82, 0.08) for the projection point 75d, (0.82, 0.93) for the projection point 75e, and (0.18, 0.93) for the projection point 75h. These correction data values ​​are generated as data 86a, 86d, 86e, and 86h in the correction data 86 shown in Fig. 18. The data size of the correction data 86 may be set automatically or by the user depending on the number of pixels of the image, for example, so that the larger the number of pixels, the larger the data size.

[0135] The processor 61 generates virtual projection image data of a virtual projection image 91 that displays an image of the corrected projection plane 177 corrected based on the generated correction data 86 as projected onto the projection target 74. The processor 61 may display the corrected projection plane 177 together with the apparent projection plane 176 in the virtual projection image 91, as shown in FIG.

[0136] Viewers viewing an image projected from the projection device 10 do not necessarily view it from near the location where the projection device 10 is installed. For example, they may view it from a distance from the projection device 10, at an angle to the projection surface. In this case, if the shape of the projection target is not flat, the image on the projection surface will appear distorted to viewers viewing it from an angle. In contrast, by generating correction data 86 as in this embodiment, even if the viewer views the image from a certain viewpoint in space, a distortion-free image can be created for the viewer, and the created distortion-free image can be displayed on the projection target. Furthermore, if viewers are expected to view the image from multiple directions, it is possible to verify in advance how the image will appear when viewed from a viewing position other than the ideal viewing position (e.g., near the virtual projection device 72).

[0137] <Ninth mode of processing by information processing device 50> A ninth mode of processing by the information processing device 50 will be described with reference to Fig. 20. Fig. 20 is a diagram showing an example of changing the position of a recognized projection point 75.

[0138] As shown in FIG. 20 , the processor 61 of the information processing device 50 acquires, for example, projection points 75c, 75d1, 75e, and 75f as intersections between projection light rays 73c, 73d, 73e, and 73f projected from a virtual projection device 72 and a projection target 74. In this case, if the user determines that the position of projection point 75d1 is inappropriate after comparing it with the positions of other projection points 75c, 75e, and 75f, the user can change the position of projection point 75d1 by operating the information processing device 50. In response to a user operation, the processor 61 performs processing to, for example, move the position of projection point 75d1 to the position of projection point 75d2. When the position of a projection point is changed by a user operation, the processor 61 may display the projection point or the thickness of the projection light rays corresponding to the change target by changing the color or thickness of the projection light rays during the change operation. Furthermore, the processor 61 may, for example, add or delete projection points based on a user operation.

[0139] The projection point 75 obtained from the intersection of the projection ray 73 and the projection target object 74 in space may be acquired at a position different from the position that should be acquired due to inaccurate recognition of the spatial structure, etc. As in this embodiment, by allowing the user to move the position of the projection point 75d1 to a position (projection point 75d2) that the user deems appropriate, a projection simulation closer to the user's intention is possible. Furthermore, even if a virtual projection point is acquired by estimating the spatial structure, there may be cases where the projection point cannot be created as intended by the user. Therefore, by allowing the user to delete or create a projection point, a projection simulation closer to the user's intention is possible.

[0140] <Tenth mode of processing by information processing device 50> A tenth mode of processing by the information processing device 50 will be described with reference to Fig. 21. Fig. 21 is a diagram showing an example of a display on a virtual projection surface using transmissive AR glasses.

[0141] The processor 61 of the information processing device 50 determines the position of a virtual projection device 72 in a space where projection is performed by the projection device 10. The processor 61 also acquires a plurality of projection points 75, which are points of intersection between a projection ray 73 of the virtual projection device 72 and a projection target object 74 in the space. The processor 61 then determines the position and shape of a virtual projection plane 77 based on the acquired plurality of projection points 75. These processes are similar to those in the above-described embodiment.

[0142] The processor 61 generates virtual projection surface image data representing the determined virtual projection surface 77. The processor 61 outputs the generated virtual projection surface image data to a display destination. In this embodiment, the output destination is, for example, a pair of see-through AR (Augmented Reality) glasses. For example, as shown in FIG. 21 , the processor 61 optically synthesizes and displays a virtual projection surface image 277 representing the virtual projection surface 77 in the user's field of view 270, i.e., on the see-through AR glasses worn by the user.

[0143] According to the tenth form of the information processing device 50, when a user who projects using the virtual projection device 72 installs the virtual projection device 72 at a certain position in the installation space, it is possible to easily and more accurately simulate the shape in which the virtual projection surface image 277 of the virtual projection device 72 will be projected onto the projection target object.

[0144] <Eleventh form of processing by the information processing device 50> The tenth form of processing by the information processing device 50 will be described with reference to Fig. 22. Fig. 22 is a diagram showing a display example of the lens shift range of the virtual projection device 72.

[0145] The processor 61 of the information processing device 50 acquires spatial image data representing a spatial image 71 of the space where projection is performed by the projection device 10. The processor 61 also determines the position of the virtual projection device 72 and the position of the virtual projection object 374, which is the object of projection by the virtual projection device 72, in the space.

[0146] The processor 61 also acquires a plurality of projection points 75, which are the intersections of the projection light rays 73 of the virtual projection device 72 and the projection object 74, and determines the position and shape of the virtual projection plane 77 based on the acquired plurality of projection points 75. The processor 61 also sets a standard projection object 381 between the virtual projection device 72 and the virtual projection object 374.

[0147] In addition, the processor 61 sets a rectangular projection surface that appears at a certain distance from the virtual projection device 72 as a standard projection surface 377 among the projection surfaces of the projection light ray 73 projected from the virtual projection device 72 toward the virtual projection object 374.

[0148] The processor 61 also determines the range in which the lens of the virtual projection device 72 can be shifted as a lens shift range 378 in which a standard projection surface 377 on the standard projection object 381 can be shifted. The lens shift range 378 is the range in which the center point of the standard projection surface 377 can be shifted. The position of the standard projection object 381 may be determined automatically or by the user. The standard projection object 381 may also be hidden.

[0149] The processor 61 generates a virtual projection image 91 in which the virtual projection device 72, the virtual projection surface 77, and the lens shift range 378 arranged between the virtual projection device 72 and the virtual projection surface 77 are displayed on the spatial image 71 represented by the spatial image data. The processor 61 acquires virtual projection image data representing the virtual projection image 91 and outputs the acquired virtual projection image data to, for example, a display unit (touch panel 51) included in the information processing device 50. By displaying the lens shift range 378 arranged between the virtual projection device 72 and the virtual projection surface 77, the user can easily recognize the range in which lens shift is possible in the virtual projection device 72.

[0150] Note that this embodiment is not limited to a configuration in which a plurality of projection points 75, which are intersections between a plurality of projection rays 73 of the virtual projection device 72 and an object (e.g., a virtual projection object 374) in the spatial image 71, are acquired, and the position and shape of the virtual projection surface 77 are determined based on the plurality of projection points 75. For example, the processor 61 may determine the position and shape of the virtual projection surface 77 as a planar shape based on the recognition result of the virtual projection object 374.

[0151] The control method described in the above-described embodiment can be realized by executing a prepared control program on a computer. The control program is recorded on a computer-readable storage medium and executed by being read from the storage medium. The control program may be provided in a form stored on a non-transitory storage medium such as a flash memory, or may be provided via a network such as the Internet. The computer that executes the control program may be included in the control device, or may be included in an electronic device such as a smartphone, tablet, or personal computer that can communicate with the control device, or may be included in a server device that can communicate with these control devices and electronic devices.

[0152] As explained above, this specification describes at least the following:

[0153] (1) An information processing device including a processor, wherein the processor: acquires first image data obtained by capturing an image of a space; determines a position of a virtual projection device in the space; acquires multiple projection points that are intersections of multiple projection rays of the virtual projection device and an object in the space; determines the position and shape of a virtual projection surface based on the multiple projection points; generates second image data representing a second image in which the virtual projection surface is displayed on a first image represented by the first image data; and outputs the second image data to an output destination.

[0154] (2) The information processing device according to (1), wherein the object includes a physical object that exists in the space.

[0155] (3) The information processing device according to (1) or (2), wherein the object includes a virtual object arranged in a virtual space representing the space.

[0156] (4) The information processing device according to any one of (1) to (3), wherein the processor determines the number of the projection points to be acquired based on a distance between the virtual projection device and the object.

[0157] (5) The information processing device according to any one of (1) to (4), wherein the processor determines the number of the projection points to be acquired based on a result of the space recognition.

[0158] (6) The information processing device according to any one of (1) to (5), wherein the processor determines the number of the projection points to be acquired based on a user instruction.

[0159] (7) The information processing device according to any one of (1) to (6), wherein the processor notifies a user when there is a projection ray for which the projection point cannot be obtained among the plurality of projection rays.

[0160] (8) An information processing device according to any one of (1) to (7), wherein, when there is a projection ray for which the projection point cannot be acquired among the plurality of projection rays, the processor determines the position and shape of the virtual projection plane based on the projection point that can be acquired.

[0161] (9) An information processing device according to any one of (1) to (8), wherein, when there is a projection ray for which the projection point cannot be obtained among the plurality of projection rays, the processor determines the position and shape of the virtual projection plane based on the projection point that can be obtained and a projection point that is the intersection of the plane based on the projection point that can be obtained and the projection ray for which the projection point cannot be obtained, and the projection point that can be obtained.

[0162] (10) An information processing device according to any one of (1) to (9), wherein, when there is a projection ray for which the projection point cannot be obtained among the plurality of projection rays, the processor determines the position and shape of the virtual projection plane based on the projection point that is the intersection of the virtual object based on the spatial recognition result and the projection ray for which the projection point cannot be obtained, and the projection point that can be obtained.

[0163] (11) An information processing device according to any one of (1) to (10), wherein the processor is capable of executing a first process of determining the position and shape of the virtual projection plane assuming that the projection surface of the object is flat, and a second process of determining the position and shape of the virtual projection plane assuming that the projection surface of the object is non-flat, and in the second process, the position and shape of the virtual projection plane are determined based on the multiple projection points.

[0164] (12) The information processing device according to any one of (1) to (11), wherein the processor divides the acquired projection points into groups and determines a position and a shape of the virtual projection plane in accordance with the groups.

[0165] (13) The information processing device according to (12), in which the processor groups the acquired plurality of projection points based on a result of the space recognition.

[0166] (14) The information processing device according to (12), in which the processor groups the acquired plurality of projection points based on distances between the acquired plurality of projection points.

[0167] (15) The information processing device according to (12), in which the processor groups the acquired plurality of projection points based on the projection points and normal vectors based on the object.

[0168] (16) The information processing device according to (12), wherein the objects include physical objects present in the space and virtual objects arranged in a virtual space representing the space, and the processor groups the acquired projection points into projection points corresponding to the physical objects and projection points corresponding to the virtual objects.

[0169] (17) The information processing device according to (12), in which the processor groups the acquired projection points based on a user instruction.

[0170] (18) The information processing device according to (12), wherein, when at least one of the plurality of projection rays intersects with a plurality of objects in the space, the processor groups the plurality of acquired projection points according to the corresponding objects.

[0171] (19) The information processing device according to any one of (12) to (18), wherein the processor generates the second image data representing the second image in which the virtual projection surface is displayed in a manner according to the group.

[0172] (20) The information processing device according to any one of (1) to (19), wherein the processor generates correction data for correcting an image to be projected onto the virtual projection surface from the virtual projection device based on the determined position and shape of the virtual projection surface.

[0173] (21) The information processing device according to (20), wherein the correction data is correction data for correcting the image projected onto the virtual projection surface so that the image appears in a specific shape from a viewpoint within the space.

[0174] (22) The information processing device according to (21), wherein the processor generates the second image data representing the second image on which the second virtual projection surface of the specific shape is displayed.

[0175] (23) The information processing device according to any one of (20) to (22), wherein the processor generates the second image data representing the second image displayed when the image corrected based on the correction data is projected onto the virtual projection surface.

[0176] (24) An information processing device including a processor, wherein the processor acquires first image data obtained by capturing an image of a space, determines a position of a virtual projection device in the space, determines a position and a shape of a virtual projection surface, determines a lens shift range in which the lens of the virtual projection device can be shifted, generates second image data representing a second image in which the virtual projection device, the virtual projection surface, and the lens shift range positioned between the virtual projection device and the virtual projection surface are displayed on a first image represented by the first image data, and outputs the second image data to an output destination.

[0177] (25) An information processing device including a processor, wherein the processor determines a position of a virtual projection device in space, acquires multiple projection points that are intersections of multiple projection rays of the virtual projection device with an object in the space, determines the position and shape of a virtual projection surface based on the multiple projection points, generates image data representing the virtual projection surface, and outputs the image data to a display destination.

[0178] (26) An information processing method, in which a processor of an information processing device acquires first image data obtained by capturing an image of a space, determines a position of a virtual projection device in the space, acquires multiple projection points which are intersections of multiple projection rays of the virtual projection device and an object in the space, determines the position and shape of a virtual projection surface based on the multiple projection points, generates second image data representing a second image in which the virtual projection surface is displayed on a first image represented by the first image data, and outputs the second image data to an output destination.

[0179] (27) An information processing program for causing a processor of an information processing device to execute the following processes: acquire first image data obtained by capturing an image of a space; determine the position of a virtual projection device in the space; acquire multiple projection points which are intersections of multiple projection rays of the virtual projection device and an object in the space; determine the position and shape of a virtual projection plane based on the multiple projection points; generate second image data representing a second image in which the virtual projection plane is displayed on a first image represented by the first image data; and output the second image data to an output destination.

[0180] Although various embodiments have been described above, it goes without saying that the present invention is not limited to such examples. It is clear that a person skilled in the art can conceive of various modifications or alterations within the scope of the claims, and it is understood that these also naturally fall within the technical scope of the present invention. Furthermore, the components of the above embodiments may be combined in any manner as long as they do not deviate from the spirit of the invention.

[0181] This application is based on a Japanese patent application (Patent Application No. 2023-140988) filed on August 31, 2023, the contents of which are incorporated herein by reference.

[0182] 1 Projection unit 2 Operation reception unit 2A, 3A Hollow portion 2a, 2b, 3a, 3c, 15a Opening 4 Control device 4a, 62 Memory 6 Projected object 10 Projection device 11 Projection range 12 Light modulation unit 15 Housing 21 Light source 22 Light modulation unit 23 Projection optical system 24 Control circuit 31 Second optical system 32, 122 Reflecting member 33 Third optical system 34 Lens 50 Information processing device 51 Touch panel 61 Processor 63 Communication interface 64 User interface 65 Sensor 69 Bus 71 Spatial image 72 Virtual projection device 72a Main body 73, 73a to 73h, 73p to 73x Projected light ray 75, 75a to 75x Projection point 74 Projection object 74A, 74B Projection object portion 74p, 74q Area 75A First group 75B Second group 75C Third group 75D Fourth group 75E Fifth group 75F Seventh group 76a One point 77 Virtual projection surface 81, 84, 374 Virtual projection object 82 Space portion 86 Correction data 86a, 86d, 86e, 86h Data 91 Virtual projection image 101 Main body portion 102 First member 103 Second member 104 Projection direction change mechanism 105 Shift mechanism 106 Optical unit 121 First optical system 131, 133 Normal vector 132, 134 Surface 173a to 173i Straight line 175a to 175i Point 175F Sixth group 176 Apparent projection surface 177 Corrected projection surface 277 Virtual projection surface image 377 Standard projection surface 378 Lens shift range 381 Standard projection object G1 Image

Claims

1. An information processing device having a processor, wherein the processor: acquires first image data obtained by capturing an image of a space; determines a position of a virtual projection device in the space; acquires a plurality of projection points which are intersections of a plurality of projection rays of the virtual projection device and an object in the space; determines a position and shape of a virtual projection surface based on the plurality of projection points; generates second image data representing a second image in which the virtual projection surface is displayed on a first image represented by the first image data; and outputs the second image data to an output destination.

2. An information processing device according to claim 1, wherein the object includes an object existing within the space.

3. An information processing device according to claim 1, wherein the object includes a virtual object arranged in a virtual space representing the space.

4. An information processing device according to claim 1, wherein the processor determines the number of projection points to be acquired based on a distance between the virtual projection device and the object.

5. An information processing device according to claim 1, wherein the processor determines the number of projection points to be obtained based on a result of the space recognition.

6. An information processing device according to claim 1, wherein the processor determines the number of projection points to be acquired in response to a user operation.

7. An information processing device according to claim 1, wherein the processor notifies a user when there is a projection ray for which the projection point cannot be obtained among the plurality of projection rays.

8. An information processing device according to claim 1, wherein, when there is a projection ray for which the projection point cannot be obtained among the plurality of projection rays, the processor determines the position and shape of the virtual projection plane based on the projection point that has been obtained.

9. An information processing device according to claim 1, wherein, when there is a projection ray for which the projection point cannot be obtained among the multiple projection rays, the processor determines the position and shape of the virtual projection plane based on the projection point that was obtained and a projection point that is the intersection between the surface based on the projection point that was obtained and the projection ray for which the projection point cannot be obtained, and the projection point that was obtained.

10. An information processing device according to claim 1, wherein, when there is a projection ray for which the projection point cannot be obtained among the plurality of projection rays, the processor determines the position and shape of the virtual projection plane based on a projection point that is the intersection between a virtual object based on the spatial recognition result and the projection ray for which the projection point cannot be obtained, and the projection point that was obtained.

11. An information processing device as described in claim 1, wherein the processor is capable of executing a first process of determining the position and shape of the virtual projection plane assuming that the projection surface of the object is flat, and a second process of determining the position and shape of the virtual projection plane assuming that the projection surface of the object is non-flat, and in the second process, the position and shape of the virtual projection plane are determined based on the multiple projection points.

12. An information processing device according to claim 1, wherein the processor divides the acquired plurality of projection points into groups, and determines the position and shape of the virtual projection plane in correspondence with each group.

13. An information processing device according to claim 12, wherein the processor groups the acquired multiple projection points based on the result of the space recognition.

14. An information processing device according to claim 12, wherein the processor groups the acquired multiple projection points based on distances between the acquired multiple projection points.

15. An information processing device according to claim 12, wherein the processor divides the acquired multiple projection points into groups based on the projection points and normal vectors based on the object.

16. An information processing device according to claim 12, wherein the objects include physical objects existing in the space and virtual objects arranged in a virtual space representing the space, and the processor groups the acquired multiple projection points into projection points corresponding to the physical objects and projection points corresponding to the virtual objects.

17. An information processing device according to claim 12, wherein the processor divides the acquired plurality of projection points into groups based on a user instruction.

18. An information processing device according to claim 12, wherein the processor, when at least any one of the plurality of projection rays intersects with a plurality of objects in the space, groups the acquired plurality of projection points according to the corresponding objects.

19. An information processing device according to claim 12, wherein the processor generates the second image data representing the second image in which the virtual projection surface is displayed in a manner corresponding to the group.

20. An information processing device according to claim 1, wherein the processor generates correction data for correcting an image to be projected from the virtual projection device onto the virtual projection surface based on the determined position and shape of the virtual projection surface.

21. An information processing device according to claim 20, wherein the correction data is correction data for correcting the image projected onto the virtual projection surface so that the image appears in a specific shape from a viewpoint within the space.

22. An information processing apparatus according to claim 21, wherein said processor generates said second image data representing said second image in which said second virtual projection surface of said particular shape is to be displayed.

23. An information processing device according to any one of claims 20 to 22, wherein the processor generates the second image data representing the second image displayed when the image corrected based on the correction data is projected onto the virtual projection surface.

24. An information processing device having a processor, wherein the processor determines a position of a virtual projection device in space, obtains a plurality of projection points which are intersections between a plurality of projection rays of the virtual projection device and an object in the space, determines a position and shape of a virtual projection plane based on the plurality of projection points, generates image data representing the virtual projection plane, and outputs the image data to a display destination.

25. An information processing method, in which a processor of an information processing device acquires first image data obtained by capturing an image of a space, determines a position of a virtual projection device in the space, acquires a plurality of projection points which are intersections of a plurality of projection rays of the virtual projection device and an object in the space, and determines the position and shape of a virtual projection surface based on the plurality of projection points, generates second image data representing a second image in which the virtual projection surface is displayed on a first image represented by the first image data, and outputs the second image data to an output destination.

26. An information processing program for causing a processor of an information processing device to execute the following processes: acquire first image data obtained by capturing an image of a space; determine a position of a virtual projection device in the space; acquire multiple projection points which are the intersections of multiple projection rays of the virtual projection device and an object in the space; determine the position and shape of a virtual projection surface based on the multiple projection points; generate second image data representing a second image in which the virtual projection surface is displayed on a first image represented by the first image data; and output the second image data to an output destination.

Images