Projection system and projection control method

Abstract

Provided is a projection system employing a projection device that projects simultaneously onto a plurality of projection surfaces.　This projection system is equipped with: a user recognizing unit for recognizing a user present in a space; a projection environment recognizing unit for recognizing a projection surface onto which a video can be projected in the space; and a control unit for controlling the projection device such that the video is projected onto the projection surface recognized by the projection environment recognizing unit, for the user recognized by the user recognizing unit. The projection device can project the video simultaneously onto a plurality of surfaces. The control unit controls the projection device such that the video is projected simultaneously onto two or more projection surfaces recognized by the projection environment recognizing unit.

Description

[[Technical Field]]

[0001] The technology disclosed in the present specification (hereinafter referred to as "the present disclosure") relates to a projection system and a projection control method for projecting an image onto one or more projection surfaces. [[Background Art]]

[0002] A projection apparatus, also called a "projector", has long been used for applications such as presentations because it can project a large image onto a screen and present it to a plurality of people at the same time. Recently, with the emergence of projection mapping technology that pastes projected images onto three-dimensional objects, the applications of projection apparatuses have been further expanded. For example, an image projection apparatus has been proposed that identifies a plurality of projection surfaces within a projection range based on an image pickup signal obtained by capturing the projection range of a projection unit with an image pickup unit, and allocates a seed image and a UI image to each projection surface for projection (see Patent Document 1). This image projection apparatus can correct the size, brightness, and chromaticity of an image in consideration of the identified state of each projection surface. [[Prior Art Document]] [[Patent Document]]

[0003] [[Patent Document 1]] Japanese Unexamined Patent Application Publication No. 2011-244044 [[Patent Document 2]] Japanese Unexamined Patent Application Publication No. 2020-112674 [[Patent Document 3]] Japanese Unexamined Patent Application Publication No. 2019-139308 [[Summary of the Invention]] [[Problem to be Solved by the Invention]]

[0004] An object of the present disclosure is to provide a projection system and a projection control method using a projection apparatus capable of simultaneous projection onto a plurality of projection surfaces. [[Means for Solving the Problem]]

[0005] This disclosure has been made in consideration of the above issues, and its first aspect is: A user recognition unit that recognizes users present in space, A projection environment recognition unit recognizes a projection surface capable of projecting an image into the aforementioned space, A control unit controls the projection device to project an image onto the projection surface recognized by the projection environment recognition unit for the user recognized by the user recognition unit, The projection system is equipped with the following features. The projection device is capable of simultaneously projecting images onto multiple surfaces. The control unit controls the projection device to simultaneously project images onto two or more projection surfaces recognized by the projection environment recognition unit.

[0006] However, the term "system" as used here refers to a logical collection of multiple devices (or functional modules that perform specific functions), without regard to whether each device or functional module resides within a single enclosure. In other words, both a single device consisting of multiple components or functional modules, and a collection of multiple devices, qualify as a "system."

[0007] At least one of the user recognition unit and the projection environment recognition unit performs recognition based on sensor information detected by sensors installed in the space.

[0008] The projection device is equipped with a phase-modulated spatial light modulator and can simultaneously project images onto multiple different surfaces in the up / down, left / right, and depth directions. Therefore, the projection environment recognition unit recognizes multiple different projection surfaces in the up / down, left / right, and depth directions.

[0009] The user recognition unit defines the user's characteristics and state. The projection system relating to the first aspect further includes a content selection unit that selects content to be displayed to the user based on the defined user information, and a projection surface determination unit that determines a projection surface on which the selected content is projected.

[0010] The projection environment recognition unit detects information about the recognized projection surface, such as its attributes, shape, area, and characteristics (reflectance, brightness, chromaticity).

[0011] Furthermore, the projection system relating to the first side further includes a projection parameter correction unit that corrects the projection parameters for the projection surface determined by the projection surface determination unit. The projection parameter correction unit limits at least one of the following based on the design values ​​of the projection device: the distance between different projection surfaces in the depth direction, the number of projection surfaces, or the projection size; determines the priority of the multiple projection surfaces determined by the projection surface determination unit; and corrects the brightness, chromaticity, and size of the projected image on the projection surface.

[0012] Furthermore, the second aspect of this disclosure is, A user recognition step that recognizes users present in a space, A projection environment recognition step involves recognizing a projection surface capable of projecting an image into the aforementioned space, A control step in which the projection device controls the projection device to project an image onto the projection surface recognized in the projection environment recognition step for the user recognized in the user recognition step, This is a projection control method that has [a certain characteristic]. [Effects of the Invention]

[0013] According to this disclosure, it is possible to provide a projection system and projection control method that use a projection device capable of simultaneously projecting onto multiple projection surfaces to project content onto a projection surface recognized in space for a user recognized in space.

[0014] The effects described herein are merely illustrative, and the effects brought about by this disclosure are not limited to those described herein. Furthermore, this disclosure may produce additional effects beyond those described above.

[0015] Further purposes, features, and advantages of this disclosure will become apparent through a more detailed description based on the embodiments and accompanying drawings described below. [Brief explanation of the drawing]

[0016] [Figure 1]FIG. 1 is a diagram showing the configuration of a projection system 100. [Figure 2] FIG. 2 is a diagram showing the configuration of a projection system 200. [Figure 3] FIG. 3 is a diagram showing an example of sensor information input by an input unit 201. [Figure 4] FIG. 4 is a diagram showing another example of sensor information input by the input unit 201. [Figure 5] FIG. 5 is a diagram showing a projection surface detected from the sensor information shown in FIG. 3. [Figure 6] FIG. 6 is a diagram showing a projection surface detected from the sensor information shown in FIG. 4. [Figure 7] FIG. 7 is a diagram for explaining the operation of a projection device 110. [Figure 8] FIG. 8 is a diagram showing the configuration of a projection system 800. [Figure 9] FIG. 9 is a diagram showing an example of a recognition result (a recognition result inside a vehicle) by a user recognition unit 802-1. [Figure 10] FIG. 10 is a diagram showing another example of a recognition result (a recognition result outside a vehicle) by the user recognition unit 802-1. [Figure 11] FIG. 11 is a diagram showing an example of a user characteristic database. [Figure 12] FIG. 12 is a diagram showing the configuration of a projection system 1200. [Figure 13] FIG. 13 is a diagram showing an example of a detection result (a detection result inside a vehicle) by a projection surface detection unit 1203-1. [Figure 14] FIG. 14 is a diagram showing an example of a detection result (a detection result outside a vehicle) by the projection surface detection unit 1203-1. [Figure 15] FIG. 15 is a diagram showing an example of a user characteristic database storing content and projection surfaces. [Figure 16] FIG. 16 is a diagram showing the configuration of a projection system 1600. [Figure 17] FIG. 17 is a diagram showing an example of a projection surface database. [Figure 18] Figure 18 shows the projection surface database defined during the daytime. [Figure 19] Figure 19 shows the projection surface database defined for nighttime. [Figure 20] Figure 20 shows an example where the distance between the two projection surfaces is smaller than the design value. [Figure 21] Figure 21 is a top view showing a vehicle parked near a wall. [Figure 22] Figure 22 is a side view showing a vehicle parked near a wall. [Figure 23] Figure 23 shows an example of correcting the amount of light projected onto each projection surface according to a prioritization rule. [Figure 24] Figure 24 shows how a vehicle equipped with a projection system projects corrected light onto the window surface and the road surface while driving through a city. [Figure 25] Figure 25 shows a method for correcting projection parameters according to the distance from the projection device 1610. [Figure 26] Figure 26 shows a specific example of performing projection parameter correction according to the distance from the projection device 1610. [Figure 27] Figure 27 shows a specific example of performing projection parameter correction when projecting an exterior advertisement onto a sidewalk from a projection system 1600 mounted on a vehicle. [Figure 28] Figure 28 shows a specific example of performing projection parameter correction when projecting an exterior advertisement onto a sidewalk from a projection system 1600 mounted on a vehicle. [Figure 29] Figure 29 shows the configuration of the projection system 2900. [Figure 30] Figure 30 shows an example of the configuration of an application information database. [Figure 31] Figure 31 shows the projection device and sensors installed inside the vehicle. [Figure 32]Figure 32 shows an example of a user characteristics database (Example 1) constructed in the application "Multiple Surface Projection in a Vehicle". [Figure 33A] Figure 33A shows an example of food projection mapping (Example 2). [Figure 33B] Figure 33B shows an example of food projection mapping (Example 2). [Figure 33C] Figure 33C shows an example of food projection mapping (Example 2). [Figure 34] Figure 34 shows an example of a provisional user characteristics database (Example 2). [Figure 35] Figure 35 shows an example of the final user characteristics database (Example 2). [Figure 36] Figure 36 shows the projection surface (Example 3) detected on the lane and on the surface of the rolling ball. [Figure 37] Figure 37 shows an example of the configuration of a user characteristics database (Example 3). [Figure 38] Figure 38 shows the projection surfaces (Example 3) assigned to the content on the lane and the surface of the rolling ball. [Figure 39] Figure 39 shows the projection surfaces (Example 3) assigned to the content on the lane and the surface of the rolling ball. [Figure 40] Figure 40 shows the CAVE system 4000. [Figure 41] Figure 41 shows an example (Example 4) of a user characteristics database constructed in the application "Viewing of Gaze Area in CAVE System (Multi-Person Support)". [Figure 42] Figure 42 shows an example of an aerial display. [Figure 43] Figure 43 shows an example of the display of a touch indicator on an aerial display. [Figure 44]Figure 44 shows an example of the display of a touch indicator on an aerial display. [Figure 45] Figure 45 is a diagram illustrating the operating principle of a phase-modulation projection device. [Figure 46] Figure 46 is a diagram illustrating the operating principle of a phase-modulation projection device. [Modes for carrying out the invention]

[0017] The present disclosure will be described below in the following order, with reference to the drawings.

[0018] A. Overview B. Projection principle C. System Configuration C-1. System Configuration Example (1) C-2. System Configuration Example (2) C-3. Projection device capable of simultaneously projecting onto multiple projection surfaces with different depth directions. C-4. System Configuration Example (3) C-5. System Configuration Example (4) C-6. System Configuration Example (5) C-7. Correction of projection parameters C-7-1. Restrictions on the projection surface C-7-2. Determining the Priority of the Projection Surface C-7-3. Correction of the projection surface C-7-3-1.Geometric correction C-7-3-2. Brightness Correction C-7-3-3. Chromaticity correction C-7-3-4. Specific Examples C-8. Modality of Input Information C-9. Metadata detection from user-owned devices D. Applications D-1. Example 1 D-2. Example 2 D-3. Example 3 D-4. Example 4 D-5. Example 5 D-6. Example 6

[0019] A. Overview Projection devices have already been proposed that simultaneously project onto multiple projection surfaces and correct the size, brightness, and chromaticity of the image according to the condition of the projection surfaces (see Patent Document 1). In contrast, this disclosure further proposes a projection system that includes a user recognition function that recognizes users present in space and a projection environment recognition function that recognizes the projection environment of the space, and projects content onto the projection surface recognized in the space for the user recognized in the space.

[0020] When the projection system is mounted on a vehicle, the space includes both the interior and exterior of the vehicle. According to this disclosure, content can be projected onto recognized projection surfaces both inside and outside the vehicle, for example, directed at passengers or pedestrians near the vehicle. The space may also be an indoor space where an immersive VR (Virtual Reality) system such as CAVE (Cave Automatic Virtual Environment) or Warp is constructed. According to this disclosure, content can be projected onto one or more recognized projection surfaces within the immersive space, directed at each user within that immersive space.

[0021] According to the projection system to which this disclosure applies, the user recognition function not only recognizes users present in space, but also further recognizes the characteristics and state of each user, and can select appropriate content considering the characteristics and state of the users.

[0022] Furthermore, a projection system to which this disclosure applies can recognize projection surfaces in space using a projection environment recognition function and assign projection surfaces to each user. In this case, projection surfaces can be assigned according to the content selected by each user. In addition, projection parameters can be corrected considering the projection environment, the characteristics of the projection surface, and the image quality when projecting content.

[0023] The projection system described herein uses a projection device capable of simultaneously projecting onto multiple projection surfaces. These multiple projection surfaces refer to multiple projection surfaces that differ not only in the vertical and horizontal directions (in other words, projection directions) but also in the depth direction. For reference, the image projection device described in Patent Document 1 is capable of simultaneously projecting onto two projection surfaces that differ in the vertical and horizontal directions, but it cannot simultaneously project onto multiple projection surfaces that differ in the depth direction.

[0024] According to this disclosure, by using a projection device capable of simultaneously projecting onto multiple projection surfaces in the vertical, horizontal, and depth directions, adaptive content projection control that takes into account both the user's state and the projection environment can be realized, thereby improving usability. In particular, since a single projection device can simultaneously project onto multiple projection surfaces, including the depth direction, it saves space and is advantageous, for example, in automotive systems. Furthermore, by concentrating the projection light of the projection device only where needed, energy efficiency and cost efficiency can be improved.

[0025] B. Projection principle As described above, the projection system according to this disclosure uses a projection device capable of simultaneously projecting onto multiple projection surfaces that differ in the depth direction in addition to the up, down, left, and right directions. In this embodiment, an optical modulation element is used to realize simultaneous projection of images onto multiple projection surfaces that differ in the depth direction.

[0026] Generally, spatial light modulators (SLMs) only exist that can independently modulate amplitude and phase. In the former amplitude modulation method, the interference fringe intensity distribution of object light and reference light is displayed on an amplitude-modulated SLM, for example, using a CGH (Computer-Generated Hologram), and the reconstructed light of the object is generated by illuminating this amplitude-modulated SLM with reference light. Although a detailed explanation is omitted, this amplitude modulation method can reproduce the object light directly, but it has the problem of generating a lot of unwanted light.

[0027] On the other hand, phase modulation, including holography, allows for the creation of arbitrary light intensity at arbitrary positions by controlling the wavefront of light. In the phase modulation method, the phase distribution of object light is displayed on a phase-modulated SLM, and when this phase-modulated SLM is illuminated with reference light, the wavefront of the transmitted light is controlled to generate reconstructed light of the object. The phase modulation method is advantageous over the amplitude modulation method because, in addition to being able to correctly reproduce the phase of object light, it can reconstruct object light efficiently using light without generating unwanted components. Phase-modulated LCOS (Liquid crystal on silicon) and phase-modulated MEMS (Microelectromechanical Systems) are used as phase-modulated SLMs. For example, Patent Document 2 mentions a projector using an SLM. Below, a method of projecting onto different projection surfaces in the depth direction using the phase modulation method will be described.

[0028] The object light O(x,y) on the xy-plane can be represented by an amplitude component A0(x,y) and a phase component exp(iφ0(x,y)), as shown on the right-hand side of equation (1) below. Here, the xy-plane is defined as the 0 position in the depth direction. The 0 position in the depth direction corresponds to the "SLM plane" or "hologram plane" where the phase-modulated SLM is installed. If this object light O(x,y) can be reproduced at the 0 position in the depth direction, it becomes possible to freely record and reproduce light.

[0029]

number

[0030] The phase distribution P(x,y) of object light O(x,y) is as shown in equation (2) below, as can be seen from equation (1) above. In the phase modulation method, by displaying this phase distribution P(x,y) on a phase-modulated SLM (such as a phase-modulated LCOS) placed at the 0 position in the depth direction and multiplying it by a reference light R(x,y), the regenerated object light O'(x,y) can be generated as shown in equation (3) below. The right-hand side of equation (3) below consists only of the regenerated light obtained by phase-modulating the reference light R(x,y). Therefore, it can be understood that the phase modulation method allows for the efficient regeneration of object light without generating unwanted components.

[0031]

number

[0032]

number

[0033] Next, we will explain the case of reconstructing the wavefront information of two object beams O1(x1,y1) and O2(x2,y2) positioned at different positions r1 and r2 in the depth direction, with reference to Figures 45 and 46. Each object beam O1(x1,y1) and O2(x2,y2) is expressed by the following equations (4) and (5), respectively. Here, (x1,y1) is the xy coordinate system on position r1 in the depth direction, and (x2,y2) is the xy coordinate system on position r2 in the depth direction.

[0034]

number

[0035]

number

[0036] As shown in Figure 45, each object light O1(x1,y1) and O2(x2,y2) are positioned at different positions r1 and r2 in the depth direction from the 0 position. First, the wavefront information of each object light O1(x1,y1) and O2(x2,y2) is combined in the virtual space to calculate the object light O(x,y) at the 0 position in the depth direction. Equation (6) below shows the calculation formula for combining each object light O1(x1,y1) and O2(x2,y2). However, since it cannot be solved analytically, equation (6) below shows an approximate formula. Various methods for calculating the approximate formula have been proposed, but a detailed explanation is omitted here.

[0037]

number

[0038] Then, when regenerating the object light O1(x1,y1) and O2(x2,y2), the phase information exp(iφ0(x,y)) included in the composite wavefront information calculated using the right-hand side of equation (6) above is displayed on the SLM placed at the 0 position in the depth direction, and the collimated reference light (or parallel light) A is generated as shown in Figure 46. R When (x,y) is incident on the SLM, the wavefront of the transmitted light is controlled by the SLM. As a result, the regenerated light O1'(x1,y1) and O2'(x2,y2) for each object shown in equations (7) and (8) below can be regenerated at the same position as in the calculation shown in Figure 45.

[0039]

number

[0040]

number

[0041] Each regenerated light O1'(x1,y1) and O2'(x2,y2) is equivalent to arbitrary two-dimensional information. Therefore, it can be said that it is possible to simultaneously project images V1 and V2 onto two surfaces at different positions r1 and r2 in the up / down / left / right and depth directions, respectively.

[0042] As described above, the phase modulation method allows for the efficient reproduction of object light without generating unwanted components. Therefore, the best embodiment of this disclosure uses a phase modulation projection device. By enabling the reproduction of the light distribution at any position in each of the x, y, and z directions, a single projection device can simultaneously project onto multiple projection surfaces that differ in depth as well as in the up, down, left, and right directions. However, the amplitude modulation method can also realize display on multiple surfaces with different depths based on the principle of holography, and if the generation of unwanted components and the resulting decrease in light utilization efficiency are not a concern, an amplitude modulation projection device may be used instead.

[0043] Since a single projection unit can simultaneously project onto multiple projection surfaces, including those in the depth direction, it saves space. Of course, if there are no constraints on space efficiency, energy consumption, or cost, a projection system employing a multi-projector system that integrates multiple projection units is also acceptable. In the following description, unless otherwise specified, the projection system will be assumed to use only one phase-modulation projection unit.

[0044] While there are several methods for generating the phase distribution to be displayed on a phase-modulated SLM, such as the GS method or methods for calculating freeform phase, these are not the only options available.

[0045] C. System Configuration Section C describes the configuration of a projection system to which this disclosure applies.

[0046] C-1. System Configuration Example (1) Figure 1 schematically shows the configuration of a projection system 100 to which the present disclosure is applied. The illustrated projection system 100 includes a user recognition unit 101, a projection environment recognition unit 102, and an output control unit 103.

[0047] The user recognition unit 101 recognizes users present within the projection range of the projection system 100. The user recognition unit 101 primarily recognizes users based on sensor information acquired by sensors installed in the same space. When the projection system 100 is mounted on a vehicle, the user recognition unit 101 recognizes users inside the vehicle (such as passengers) and users outside the vehicle (such as pedestrians around the vehicle). The user recognition unit 101 also recognizes the characteristics and state of the users, but details of this will be described later.

[0048] The projection environment recognition unit 102 recognizes the areas within the space that can be projected by the projection system 100 that can actually be projected onto by the projection device 110 as projection surfaces. The projection environment recognition unit 102 basically recognizes the projection surfaces based on sensor information acquired by sensors installed in the same space. The projection environment recognition unit 102 also recognizes the characteristics and state of the projection surfaces, but the details of this will be described later.

[0049] The output control unit 103 controls the output of the projection device 110 to project an image onto the projection surface and display information to the user, based on the recognition results of the user recognition unit 101 and the projection environment recognition unit 102. The projection device 110 is a phase-modulation type projection device as described in section B above, and is capable of simultaneously projecting onto multiple different projection surfaces in the up / down / left / right and depth directions.

[0050] The output control unit 103 basically controls the projection operation of the projection device 110 by assigning one or more appropriate projection surfaces from the one or more projection surfaces recognized by the projection environment recognition unit 102 to one or more users recognized by the user recognition unit 101. Figure 1 shows an example in which the projection device 110 simultaneously projects onto two projection surfaces: a first projection surface (x1, y1, z1) and a second projection surface (x2, y2, z2). Furthermore, the output control unit 103 can select appropriate content considering the characteristics and state of the user recognized by the user recognition unit 101 and assign a projection surface suitable for projecting the selected content, but the details of this will be described later.

[0051] C-2. System Configuration Example (2) Figure 2 schematically shows the configuration of a projection system 200 to which this disclosure is applied. The illustrated projection system 200 comprises an input unit 201, a user recognition unit 202, a projection environment recognition unit 203, and an output control unit 204. However, components that are the same as those included in the projection system 100 shown in Figure 1 are shown with the same names.

[0052] The input unit 201 receives sensor information acquired by sensors installed within the space that can be projected by the projection system 200. Alternatively, the input unit 201 may be the sensor itself installed in the space. The sensors consist of image sensors, distance sensors, etc. The sensors may also include position sensors such as thermal cameras, ultrasonic sensors, touch sensors, and GPS (Global Positional System) sensors, as well as various other sensors capable of sensing information about the spatial environment. If the space within the projection range is the interior of a vehicle, the input unit 201 receives sensor information from, for example, in-vehicle sensors installed inside the vehicle. If the user recognition unit 202 also recognizes users outside the vehicle, the input unit 201 shall also receive sensor information from outside the vehicle.

[0053] The user recognition unit 202 recognizes the user based on sensor information supplied from the input unit 201, and further recognizes the user's characteristics and state. When the projection system 200 is mounted on a vehicle, the user recognition unit 202 recognizes users inside the vehicle (such as passengers) and users outside the vehicle (such as pedestrians around the vehicle).

[0054] The projection environment recognition unit 203 includes a projection surface detection unit 203-1. Based on sensor information supplied from the input unit 201, the projection surface detection unit 203-1 detects as projection surfaces the areas that can actually be projected by the projection device 210 within the space that can be projected by the projection system 100.

[0055] The output control unit 204 controls the output of the projection device 210 to project an image onto the projection surface and display information to the user, based on the recognition results of the user recognition unit 202 and the projection surface detection unit 203-1. The projection device 210 is a phase-modulation type projection device as described in section B above, and is capable of simultaneously projecting onto multiple different projection surfaces in the up / down / left / right and depth directions.

[0056] The output control unit 204 basically controls the projection operation of the projection device 210 by assigning one or more appropriate projection surfaces from the one or more projection surfaces recognized by the projection surface detection unit 203-1 to one or more users recognized by the user recognition unit 204, and projecting images onto the projection surfaces. Furthermore, the output control unit 204 selects appropriate content considering the characteristics and status of the user recognized by the user recognition unit 202, and assigns a projection surface suitable for projecting the selected content.

[0057] The projection surface detection unit 203-1 will be described in more detail. The projection surface detection unit 203-1 detects an area within the projection range of the projection device 210 that satisfies one or more thresholds, such as an area greater than or equal to a predetermined threshold, a curvature less than or equal to a predetermined threshold, or a gradient greater than or equal to a predetermined threshold (angle between the projected light and the projection surface), as a projection surface. The thresholds for detecting the projection surface may be defined for each user, for each piece of content, or for a combination of user and content. Furthermore, the thresholds for detecting the projection surface may be defined for each application to which the projection system 200 is applied.

[0058] Figures 3 and 4 illustrate sensor information acquired by sensors installed within the projection range of the projection system 200. Here, it is assumed that the projection system 200 is mounted on a vehicle and that sensor information from inside the vehicle and around the vehicle is acquired as sensor information. The sensors consist of image sensors, distance sensors, etc., and the input unit 201 receives image data from inside the vehicle as shown in Figure 3 and image data from around the vehicle as shown in Figure 4. The input unit 201 may also simultaneously receive sensor information acquired by a thermal camera, ultrasonic sensor, touch sensor, and various other sensors capable of sensing information about the spatial environment in the same environment.

[0059] The projection surface detection unit 203-1 detects multiple projection surfaces suitable for the user inside the vehicle recognized by the user recognition unit 202, based on the in-vehicle image data shown in Figure 3. In the example shown in Figure 5, the projection surface detection unit 203-1 detects multiple projection surfaces suitable for a user (not shown) sitting in the rearmost seat, such as the headrests of the front seats, the ceiling, and the pillars. These surfaces have an area greater than or equal to a predetermined threshold, a curvature less than or equal to a predetermined threshold, and a gradient (angle between the projected light and the projection surface) greater than or equal to a predetermined threshold. In Figure 5, each detected projection surface is shown in light gray. The projection surface detection unit 203-1 also does not detect areas that have an area greater than or equal to a predetermined threshold and a curvature less than or equal to a predetermined threshold, but whose gradient is less than the threshold.

[0060] Furthermore, the projection surface detection unit 203-1 detects a projection surface suitable for the user around the vehicle recognized by the user recognition unit 202, based on the image data of the vehicle's surroundings shown in Figure 4. In the example shown in Figure 6, when a woman waiting at a traffic light at an intersection near the vehicle is recognized as an external user, the projection surface detection unit 203-1 detects a projection surface, such as the road surface in front of the woman, that has an area greater than or equal to a predetermined threshold, a curvature less than or equal to a predetermined threshold, and a gradient (angle between the projected light and the projection surface) greater than or equal to a predetermined threshold. In Figure 6, the projection surface detected on the road surface in front of the woman is shown in light gray.

[0061] C-3. Projection device capable of simultaneously projecting onto multiple projection surfaces with different depth directions. As explained in Section B above, this disclosure uses a phase-modulation projection device to achieve simultaneous projection onto multiple projection surfaces that differ in the depth direction in addition to the up, down, left, and right directions.

[0062] Figure 7 shows an example of the operation of projection device 110 in Figure 1. The operation of projection device 210 in Figure 2 is similar.

[0063] The projection device 110 has a phase-modulated SLM (such as a phase-modulated LCOS) 701. Simultaneous projection onto multiple projection surfaces that differ in depth in addition to the up, down, left, and right directions The phase information contained in the composite wavefront information of the image to be projected onto each of the two projection surfaces (x1, y1, z1) and (x2, y2, z2) that differ in depth direction is displayed on the phase-modulated SLM 701, and when regenerated light (approximately parallel light) obtained by collimating the illumination light from a light source (not shown in Figure 7) is incident from behind the phase-modulated SLM 701, the desired image is simultaneously projected onto each of the projection surfaces (x1, y1, z1) and (x2, y2, z2) in front of it.

[0064] While there are several methods for generating the phase distribution displayed on the phase-modulated SLM701, such as the GS method or methods for calculating freeform phase (as mentioned above), the algorithm is not limited to these.

[0065] Furthermore, a luminance modulation panel (not shown) may be placed after the phase-modulated SLM701. By using a luminance modulation panel, the luminance dynamic range of the projected image can be expanded and the resolution can be improved. In this case, a process is performed to determine the transmittance or reflectance of the luminance modulation panel. However, it should be noted that expanding the luminance dynamic range will decrease the overall brightness, and in addition to this, the resolution of the projected image can be improved.

[0066] By using a phase-modulation projection device 110, it is possible to simultaneously project onto multiple projection surfaces that differ in the depth direction. When the projection system 100 is mounted on a vehicle, as shown in Figure 5, multi-projection onto multiple projection surfaces inside the vehicle, such as headrests, pillars, and the ceiling, can be achieved.

[0067] Furthermore, the projection device 110 allows for focusing on moving objects by changing the phase distribution displayed on the phase-modulated SLM 701. For example, in projection mapping interaction with food, focusing can be maintained even when the plate is lifted. Other examples of focusing include the ability to create moving effects in various projection mapping events (described later), such as fashion shows, bowling and other sports competitions.

[0068] The projection device 110 can also be used as a light source for structured light. Structured light is a method of three-dimensional measurement in which structured light, which is patterned in a dot-like manner, is shone onto an object, and depth information can be obtained from the distortion of the pattern. By using the phase-modulated projection device 110, projection can be performed without changing the density according to the depth.

[0069] Furthermore, the projection device 110 can be applied to VR systems such as CAVE and Warp, enabling wide-area image display with a single unit.

[0070] Furthermore, the projection device 110 can be used as an indicator for a touch sensor in an aerial display that projects images into the air (described later). By using the projection device 110, a dark, small point can be displayed when the distance is far, and a large, bright point can be displayed when the distance is close.

[0071] C-4. System Configuration Example (3) Figure 8 schematically shows the configuration of a projection system 800 to which the present disclosure is applied. The illustrated projection system 800 comprises an input unit 801, a user information detection unit 802, a projection environment recognition unit 803, and an output control unit 804. However, components that are the same as those included in the projection system 100 shown in Figure 1 or the projection system 200 shown in Figure 2 are shown with the same names. The projection system 800 is primarily characterized by its method of detecting user information.

[0072] The input unit 801 receives sensor information acquired by sensors installed within the space that can be projected by the projection system 800. Alternatively, the input unit 801 may be the sensor itself installed in the space. The sensors consist of image sensors, distance sensors, etc. The sensors may also include thermal cameras, ultrasonic sensors, touch sensors, and various other sensors capable of sensing information about the spatial environment. For example, the input unit 801 receives sensor information from in-vehicle sensors installed inside the vehicle. If the user recognition unit 802 also recognizes users outside the vehicle, the input unit 801 shall also receive sensor information from outside the vehicle.

[0073] The user information detection unit 802 includes a user recognition unit 802-1 and a user definition unit 802-2.

[0074] The user recognition unit 802-1 recognizes users based on sensor information supplied from the input unit 801. The user recognition unit 802-1 uses image information from an RGB camera and a distance sensor as sensor information to detect the number of users, the users' positions, and the orientation and gaze of the users' faces. The user recognition unit 802-1 can achieve such posture recognition using posture estimation models such as OpenPose, developed at Carnegie Mellon University. When the projection system 800 is mounted on a vehicle, the user recognition unit 802-1 recognizes users inside the vehicle (such as passengers) and users outside the vehicle (such as pedestrians around the vehicle).

[0075] The user definition unit 802-2 defines the characteristics and state of the user recognized by the user recognition unit 802-1. When the user recognition unit 802-1 recognizes multiple users, it defines the characteristics and state for each user. The user definition unit 802-2 defines the characteristics data of the user recognized from the image by comparing it with, for example, a database describing user stereotype information. The user definition unit 802-2 also defines the user's state, such as wakefulness or sleep, based on the recognition results by the user recognition unit 802-1. The user definition unit 802-2 can estimate the user's state using parameters such as the number of blinks and eye movements. The user definition unit 802-2 may also estimate the user's attributes using a trained machine learning model. The user definition unit 802-2 stores the characteristics and state defined for each user recognized by the user recognition unit 802-1 in the user characteristics database.

[0076] The projection environment recognition unit 803 includes a projection surface detection unit 803-1. Based on sensor information supplied from the input unit 801, the projection surface detection unit 803-1 detects as projection surfaces areas that can actually be projected by the projection device 810 within the space that can be projected by the projection system 800. As already explained in section C-2 above, the projection surface detection unit 803-1 detects as projection surfaces areas that satisfy conditions defined by one or more thresholds, such as an area greater than or equal to a predetermined threshold, a curvature less than or equal to a predetermined threshold, or a gradient greater than or equal to a predetermined threshold (angle between the projected light and the projection surface).

[0077] The output control unit 804 controls the output of the projection device 810 to project an image onto the projection surface and display information to the user, based on the recognition results of the user recognition unit 802-1 and the projection surface detection unit 803-1. The projection device 810 is a phase-modulation type projection device as described in section B above, and is capable of simultaneously projecting onto multiple different projection surfaces in the up / down / left / right and depth directions.

[0078] Next, using the case where the projection system 800 is mounted on a vehicle as an example, we will specifically and in detail explain the operation of the user recognition unit 802-1 and the user definition unit 802 within the user information detection unit 802.

[0079] The user recognition unit 802-1 recognizes three users inside the vehicle (user1, user2, user3) based on image data from inside the vehicle, for example as shown in Figure 9. The user recognition unit 802-1 also recognizes a user (user4) walking around the vehicle based on image data from the area surrounding the vehicle, for example as shown in Figure 10. An entry for each user (user1, user2, user3, user4) is then created in the user characteristics database.

[0080] Next, the user definition unit 802-2 associates stereotype information with each recognized user and defines characteristic data for each user. For example, it defines characteristic data for user1 as "male, 30s, office worker," user2 as "female, 30s, housewife," user3 as "infant," and user4 as "female, 20s, university student."

[0081] Furthermore, the user definition unit 802-2 defines the user's state, such as whether they are awake or asleep, for each recognized user. The user definition unit 802-2 can estimate the user's state using parameters such as the number of blinks and eye movements. For example, it defines characteristic data such as "driving" for user1, "awake" for user2, "awake" for user3, and "awake, waiting at a traffic light" for user4.

[0082] Furthermore, the user definition unit 802-2 defines whether it is possible (Yes) or impossible (No) to display information to the user, based on the user's characteristics and state. Displaying information here means projecting an image onto the projection surface assigned to the user by the projection device 810. For example, the ability to display information to each user is defined as "No" for user1, "Yes" for user2, "No" for user3, and "Yes" for user4. For example, a user who is awake is defined as "Yes" for information display, but a user who is driving, sleeping, awake but operating a smartphone or listening to music, or a baby is defined as "No" for information display.

[0083] The characteristics, status, and information displays defined for each user by the user-defined unit 802-2 are then stored in each user's entry in the user characteristics database. Figure 11 shows an example of the user characteristics database, which stores the information for each user defined by the user-defined unit 802-1.

[0084] It is expected that the user characteristics and states defined for the recognition results by the user recognition unit 802-1 will differ depending on the definition rules and machine learning model used by the user definition unit 802-2. Figure 11 is an example of a user characteristics database, and it is expected that different characteristics and states will be defined depending on the definition rules and machine learning model used.

[0085] The output control unit 804 controls the display of information to each user based on the user characteristics database shown in Figure 11. Specifically, for user2 and user4, who are defined as "Yes" for information display in the user characteristics database, the output control unit 804 assigns one of the projection surfaces detected by the projection surface detection unit 803-1 to each user and controls the output of the projection device 810 to project the content image onto each projection surface. On the other hand, for user1 and user3, who are defined as "No" for information display in the user characteristics database, the output control unit 804 does not assign a projection surface and therefore does not project images to these users.

[0086] C-5. System Configuration Example (4) Figure 12 schematically shows the configuration of a projection system 1200 to which this disclosure is applied. The illustrated projection system 1200 comprises an input unit 1201, a user information detection unit 1202, a projection environment recognition unit 1203, and an output control unit 1204. However, components that are the same as those included in the projection system 800 shown in Figure 8 are shown with the same names. The main feature of the projection system 1200 is that it selects content to be displayed to the user and determines the projection surface based on user information.

[0087] The input unit 1201 receives sensor information acquired by sensors installed within the space that can be projected by the projection system 1200. Alternatively, the input unit 1201 may be the sensor itself installed in the space. For example, the input unit 1201 receives sensor information from an in-vehicle sensor installed inside the vehicle. If the user recognition unit 1202 also recognizes users outside the vehicle, the input unit 1201 shall also receive sensor information from outside the vehicle.

[0088] The user information detection unit 1202 includes a user recognition unit 1202-1 and a user definition unit 1202-2. The user recognition unit 1202-1 uses a pose estimation model such as OpenPose to detect the number of users, the location of the users, and the orientation and gaze of the users from the sensor information supplied from the input unit 1201. The user definition unit 1202-2 then defines the characteristics and state of the users recognized by the user recognition unit 1202-1 and stores the characteristics and state defined for each user in the corresponding user entry in the user characteristics database.

[0089] The projection environment recognition unit 1203 includes a projection surface detection unit 1203-1. Based on sensor information supplied from the input unit 1201, the projection surface detection unit 1203-1 detects as projection surfaces areas that can actually be projected by the projection device 1210 within the space that can be projected by the projection system 1200. As already explained in section C-2 above, the projection surface detection unit 1203-1 detects as projection surfaces areas that satisfy conditions defined by one or more thresholds, such as an area greater than or equal to a predetermined threshold, a curvature less than or equal to a predetermined threshold, or a gradient greater than or equal to a predetermined threshold (angle between the projected light and the projection surface).

[0090] The output control unit 1204 includes a content selection unit 1204-1 and a projection surface determination unit 1204-2. The content selection unit 1204-1 selects content to be displayed to the user based on user information recognized by the user recognition unit 1202. The projection surface determination unit 1204-2 determines the projection surface from among the projection surfaces detected by the projection surface detection unit 1203-1 to project the content image.

[0091] Furthermore, the output control unit 1204 performs projection size determination processing on the projection surface assigned to each content by the projection surface determination unit 1204-2, and projection brightness determination processing to determine the brightness and chromaticity of the image to be projected. In the projection size determination processing, the projectionable size on the projection surface is calculated based on the distance between the projection surface and the projection device 1210, and the projectionable size is compared with the recommended size of the content, and the content is reduced in size as necessary so that it fits within the projectionable size. In the projection brightness determination processing, the output and correction amount of the projection device 1210 are calculated from the characteristics of the projection surface (chromaticity, brightness, reflectance). Regarding chromaticity, a chromaticity correction value for the color space is calculated based on the chromaticity information of the projection surface. If the chromaticity of a pixel in the content exceeds the displayable chromaticity of the projection device 1210, the chromaticity of that pixel is determined to match the displayable chromaticity of the projection device 1210. Regarding luminance, the possible output of the projection device 1210 is calculated from the total number of projection surfaces onto which the content is projected and the content to be projected. Furthermore, the achievable luminance is calculated considering the reflectance of the projection surfaces, and this is compared with the luminance of the original signal of the content. If the output luminance is less than that of the original signal, the output luminance is reduced to the achievable luminance, or the setting values ​​(current value, duty cycle value, etc.) of the projection device 1210 at that time are calculated. When determining with chromaticity priority, the determination is made in the order of chromaticity → luminance, but when determining with luminance priority, the determination is not limited to this order, and the determination method is not limited to the above. The determined display information such as projection size, luminance, and chromaticity is stored in the corresponding content entry in the user characteristics database.

[0092] The output control unit 1204 controls the output of the projection device 1210 to project an image onto the projection surface and display information to the user. The projection device 1210 is a phase-modulation type projection device as described in section B above, and is capable of simultaneously projecting onto multiple projection surfaces that differ in the up / down / left / right and depth directions. The output control unit 1204 performs display image generation processing based on the projection size, projection brightness, and chromaticity determined for each projection surface onto which content is projected. The display image generation processing includes display target generation processing, display phase distribution generation processing, and drive parameter setting processing. In the display target generation processing, brightness distribution targets to be displayed for each monochrome or color channel are generated. In the display phase distribution generation processing, a phase distribution is generated for each calculated brightness distribution target. If brightness correction is performed, optical correction information corresponding to the amount of brightness correction is added to the generated phase distribution. Examples of phase distribution generation algorithms include the GS method and the free-form method, but are not limited to these. In the drive parameter setting process, the drive parameters of the light source of the projection device 1210 (current value in the case of CW, duty cycle in the case of pulse) are set so that the determined luminance and chromaticity are displayed. Also, if a luminance modulation panel is arranged after the phase modulation type SLM, the transmittance or reflectance of the luminance modulation panel is determined. The output control unit 1204 outputs the determined information to the projection device 1210 to project the image.

[0093] The following describes the content selection unit 1204-1 and the projection surface determination unit 1204-2 in detail.

[0094] The content selection unit 1204-1 selects content to display to the user based on user information defined in the user definition unit 1202-1 (i.e., the user characteristics database). Specifically, the content selection unit 1204-1 compares the user's characteristics and status defined in the user definition unit 1202-2 with the content database (such as attribute information for each piece of content) to match the content to be displayed to the user. The matching can be performed using collaborative filtering (CF), other recommendation technologies, or machine learning models, but is not limited to any particular method. The content selection unit 1204-1 then stores the information of the selected content for the user (such as the content name or URL (Uniform Resource Locator) and other content access information) in the corresponding user entry in the user characteristics database.

[0095] The content selection unit 1204-1 may select multiple pieces of content for a single user. If there are multiple pieces of content to display to a single user, the content may be stored in the user's entry in the user characteristics database in order of priority based on rules (1) to (3) below, for example. Alternatively, the priority order of multiple pieces of content may be determined based on a trained machine learning model rather than a rule-based system like the one below.

[0096] (1) Prioritization based on predetermined user preferences (2) Prioritization based on the user's current location (e.g., prioritizing information on nearby facilities and events, and local weather forecasts) (3) Prioritization based on the current time (e.g., prioritizing news at 7am, 12pm, and 6pm)

[0097] Furthermore, the content selection unit 1204-1 may select multiple pieces of content to be displayed to a single user. In this case, entries may be added to the user characteristics database to store two or more pieces of content for the relevant user.

[0098] Furthermore, the content selection unit 1204-1 only needs to select content for users whose information display is defined as "Yes," and does not select content for users whose information display is defined as "No." Alternatively, entries for users whose information display is defined as "No" may be deleted from the user characteristics database.

[0099] The projection surface determination unit 1204-2 determines a projection surface from among the projection surfaces detected by the projection surface detection unit 1203-1 to project the content selected by the content selection unit 1204-1. The projection surface determination unit 1204-2 performs projection surface determination processing for each piece of content selected by the user by the content selection unit 1204-1. First, the projection surface determination unit 1204-2 determines whether a projection surface detected by the projection surface detection unit 1203-1 exists within the field of view of the user for whom the content has been selected. If a projection surface detected by the projection surface detection unit 1203-1 exists within the field of view of the target user, the projection surface determination unit 1204-2 associates the projection surface with the user (or the content selected by the user) and stores it in the corresponding user entry in the user characteristics database. On the other hand, if no projection surface exists within the field of view of the target user, the projection surface determination unit 1204-2 does not associate a projection surface with that user.

[0100] The projection surface determination unit 1204-2 may assign a projection surface to the content based on any of the following priority orders (1) to (6).

[0101] (1) In descending order of projection surface area (absolute value) (2) In order of largest field of view relative to the user (3) In order of proximity between the user and the projection surface (4) In order of closest to the recommended screen size or recommended aspect ratio of the content (5) In order of highest contrast between the background and content of the projection surface (6) In order of lowest background brightness of the projection surface

[0102] Next, using the example of a system mounted in a vehicle, we will specifically explain the operation of the projection system 1200 in which it selects content to be displayed to the user and determines the projection surface based on user information.

[0103] The user recognition unit 1202-1 recognizes three users inside the vehicle (user1, user2, user3) based on in-vehicle image data, for example, as shown in Figure 9. The user recognition unit 1202-1 also recognizes a user (user4) walking around the vehicle based on in-vehicle image data, for example, as shown in Figure 10. Next, the user definition unit 1202-2 associates stereotype information with each recognized user (user1, user2, user3, user4) to define user-specific characteristic data. A user characteristic database, as shown in Figure 11, is then constructed.

[0104] The projection surface detection unit 1203-1 of the projection environment recognition unit 1203 detects as projection surfaces locations where the projection device 1210 can actually project, both inside and outside the vehicle, based on sensor information supplied from the input unit 1201. Here, the projection surface detection unit 1203-1 detects a total of nine projection surfaces, #001 to #009, inside the vehicle as shown in Figure 13, and one projection surface, #101, on the road surface near user 4 outside the vehicle as shown in Figure 14.

[0105] The content selection unit 1204-1 selects content to display for each of the two users, user2 and user4, who are defined as "Yes" for information display in the user characteristics database shown in Figure 11. Here, the content selection unit 1204-1 selects three content items for user2: "Okinawa resort advertisement," "news," and "map," and one content item for user4: "shopping mall advertisement." These selected content items are then stored in the corresponding user entries in the user characteristics database.

[0106] Next, the projection surface determination unit 1204-2 determines which projection surfaces will project each of the content selected by the content selection unit 1204-1, "Okinawa Resort Advertisement," "News," "Map," and "Shopping Mall Advertisement," from among the projection surfaces detected by the projection surface detection unit 1203-1. As described above, the projection surface determination unit 1204-2 determines the projection surfaces that exist within the field of view of the user for which the content has been selected, and determines which projection surfaces to display the content selected by the user from among the projection surfaces within the user's field of view, in order of priority (as described above). Here, projection surface #009 detected within user2's field of view is determined to be the projection surface for displaying the content "Okinawa Resort Advertisement" and "News," another projection surface #003 detected within user2's field of view is determined to be the projection surface for displaying the content "Map," and projection surface #101 detected within user4's field of view is determined to be the projection surface for displaying the content "Shopping Mall Advertisement."

[0107] The projection plane determination unit 1204-2 then stores the projection plane assigned to each content in the corresponding user entry in the user characteristics database. Figure 15 illustrates a user characteristics database that stores information on the content selected by the content selection unit 1204-1 for the user, and the projection plane assigned to each content by the projection plane determination unit 1204-2.

[0108] C-6. System Configuration Example (5) Figure 16 schematically shows the configuration of a projection system 1600 to which this disclosure is applied. The illustrated projection system 1600 includes an input unit 1601, a user information detection unit 1602, a projection environment recognition unit 1603, an output control unit 1604, and a projection parameter correction unit 1605. However, components that are the same as those included in the projection system 1200 shown in Figure 12 are shown with the same names. The main feature of the projection system 1600 is that it defines the characteristics of the projection surface based on sensor information and determines a projection surface with characteristics suitable for projecting content.

[0109] The input unit 1601 receives sensor information acquired by sensors installed within the space that can be projected by the projection system 1600. Alternatively, the input unit 1601 may be the sensor itself installed in the space. For example, the input unit 1601 receives sensor information from an in-vehicle sensor installed inside the vehicle. If the user recognition unit 1602 also recognizes users outside the vehicle, the input unit 1601 shall also receive sensor information from outside the vehicle.

[0110] The user information detection unit 1602 includes a user recognition unit 1602-1 and a user definition unit 1602-2. The user recognition unit 1602-1 uses a posture estimation model such as OpenPose to detect the number of users, the location of the users, and the orientation and gaze of the users from the sensor information supplied from the input unit 1601. The user definition unit 1602-2 then defines the characteristics and state of the users recognized by the user recognition unit 1602-1 and stores the characteristics and state defined for each user in the corresponding user entry in the user characteristics database.

[0111] The projection environment recognition unit 1603 includes a projection surface detection unit 1603-1 and a projection surface definition unit 1603-2. Based on sensor information supplied from the input unit 1601, the projection surface detection unit 1603-1 detects as projection surfaces the areas that can actually be projected by the projection device 1610 within the space that can be projected by the projection system 1600 (same as above). The projection system 1600 uses a projection surface database to manage the characteristics of each projection surface detected by the projection surface detection unit 1603-1. An entry for each projection surface detected by the projection surface detection unit 1603-1 is provided in the projection surface database.

[0112] The projection surface definition unit 1603-2 recognizes the characteristics of the projection surface detected by the projection surface detection unit 1603-1 and stores that information in the corresponding entry in the projection surface database. Specifically, the projection surface definition unit 1603-2 assigns characteristic information such as attributes, shape, area, reflectance, chromaticity, and brightness to the projection surface and stores it in the projection surface database. Regarding the attributes of the projection surface, the projection surface definition unit 1603-2 may cluster them by comparing them with a predetermined database, or it may cluster them using a trained machine learning model. Regarding the shape, area, reflectance, chromaticity, and brightness of the projection surface, the projection surface definition unit 1603-2 calculates them based on the sensor information acquired by the input unit 1601. Figure 17 shows an example of a projection surface database that stores the characteristic information defined for each projection surface detected inside and outside a vehicle, as shown in Figures 13 and 14. However, chromaticity is represented by the coordinate position on the chromaticity map.

[0113] The characteristic data of the projection surface defined by the projection surface definition unit 1603-2 can be used in the subsequent projection surface determination unit 1604-2 to determine a projection surface suitable for the content, and further, in the projection parameter correction unit 1605 to correct the projection parameters.

[0114] The output control unit 1604 includes a content selection unit 1604-1 and a projection surface determination unit 1604-2. The content selection unit 1604-1 selects content to be displayed to the user based on user information recognized by the user recognition unit 1602. The projection surface determination unit 1604-2 determines, from among the projection surfaces detected by the projection surface detection unit 1603-1, the projection surface to be assigned to the user, or a projection surface that has the characteristics to project the image of the selected content to the user. Information regarding the content selected by the content selection unit 1604-1 and the projection surface determined by the projection surface determination unit 1604-2 is stored in the user characteristics database (as described above).

[0115] The projection parameter correction unit 1605 has the function of correcting projection parameters such as the content selected by the content selection unit 1604-1 and the projection surface determined by the projection surface determination unit 1604-2 in order to maintain projection quality. For example, if the area of ​​the projection surface is smaller than the recommended screen size of the content, the projection parameter correction unit 1605 performs processing to reduce the screen size of the original content. Also, if the brightness and chromaticity of the projected image differ significantly from the original content due to the characteristics of the background projection surface, the projection parameter correction unit 1605 performs signal processing to bring them closer to the original brightness and chromaticity. Display information such as projection size, brightness, and chromaticity corrected by the projection parameter correction unit 1605 is stored in the corresponding content entry in the user characteristics database. Detailed functions of the projection parameter correction unit 1605 are described in the following section C-7. The correction processing by the projection parameter correction unit 1605 may be performed in real time.

[0116] The projection device 1610 is a phase-modulation type projection device as described in Section B above, and is capable of simultaneously projecting onto multiple projection surfaces that differ in the vertical, horizontal, and depth directions. Based on the projection size, projection brightness, and chromaticity corrected by the projection parameter correction unit 1605, a display image generation process is performed to generate an image to be projected by the projection device 1610. This display image generation process may be performed by the output control unit 1604 or by the projection parameter correction unit 1605. The display image generation process is as described in Section C-5 above, so a detailed explanation is omitted here. The projection device 1610 projects the image based on the information determined by the display image generation process.

[0117] Next, we will specifically explain the effect of the projection system 1600 being equipped with a projection surface definition function, using the example of installation in a vehicle. However, it is assumed that within the projection environment recognition unit 1603, the projection surface database is updated in real time (or at short control cycles) based on sensor information acquired moment by moment by the input unit 1601. Figure 18 shows the projection surfaces detected inside the vehicle during the day and the projection surface database storing the characteristic data of each projection surface defined during the day. Figure 19 shows the projection surfaces detected inside the vehicle at night and the projection surface database storing the characteristic data of each projection surface defined at night.

[0118] Here, the content selection unit 1603-1 recommends a screen size of 0.45m. 2 This section considers an example of determining the optimal projection surface from among the in-vehicle projection surfaces #001 to #004 shown in Figure 13 when selecting content. However, the brightness of the projection device 1610 is set to 500 ml, and the recommended screen size of the content is set to 0.45 m. 2 The maximum area occupancy rate (the ratio of the effective projection area that can actually be projected out of the total projection range that the projection device 1610 can project from) shall be set to 10%. The projection surface determination unit 1604-2 shall then prioritize the projection surfaces based on the following conditions (1) and (2).

[0119] (1) The closest size to the recommended screen size for the content, either the same as or larger than the recommended screen size. (2) Content with a contrast ratio of 2:1 or higher on the projection surface.

[0120] First, let's explain how to determine the projection surface for this content based on the projection surface database defined for daytime. During the daytime, based on the projection surface database in Figure 18, projection surfaces #003 and #001 are selected in order of proximity to the recommended screen size for the content, according to condition (1) above. Next, let's calculate the content brightness when projected onto each projection surface #001 and #003. The content brightness when projected onto projection surface #001 is 1768 nits, as shown in equation (9) below. Referring to Figure 18, the brightness of projection surface #001 during the daytime is 52 nits. Therefore, the contrast between the content and projection surface #001 is 1768:52 = 34:1, and it can be seen that projection surface #001 during the daytime also satisfies condition (2) above.

[0121]

number

[0122] On the other hand, the content brightness when projected onto projection surface #003 is 2475 nits, as shown in equation (10) below. Referring to Figure 18, the brightness of projection surface #003 during the daytime is 10000 nits. Therefore, the contrast between the content and projection surface #003 is 0.24:1, and it can be seen that projection surface #003 during the daytime does not satisfy the above condition (2). Therefore, during the daytime, the projection surface determination unit 1604-2 determines projection surface #001, which satisfies both the above conditions (1) and (2), as the projection surface suitable for projecting content.

[0123]

number

[0124] Next, we will explain how to determine the projection surface for this content based on the projection surface database defined for nighttime. At night, based on the projection surface database in Figure 19, projection surfaces #003 and #001 are selected in order of proximity to the recommended screen size for the content, according to condition (1) above. Next, let's calculate the content brightness when projected onto each projection surface #001 and #003. The content brightness when projected onto projection surface #001 is 1768 nits, as shown in equation (9) above, but referring to Figure 19, the brightness of projection surface #001 at night is 10 nits. Therefore, the contrast between the content and projection surface #001 is 177:1, and it can be seen that projection surface #001 also satisfies condition (2) above, even at night.

[0125] Furthermore, the content brightness when projected onto projection surface #003 is 2475 nits, as shown in equation (10) above. However, referring to Figure 19, the brightness of the projection surface at night is 20 nits. Therefore, the contrast between the content and projection surface #003 is 123:1, and at night, projection surface #003 also satisfies the above condition (2). Consequently, at night, the projection surface determination unit 1604-2 determines projection surface #003, which is closest to the recommended content area, as the projection surface suitable for projecting the content.

[0126] Thus, in the projection system 1600, the projection surface determined for the same content changes based on the projection environment sensed in real time, such as daytime and nighttime. In other words, by combining the projection surface definition unit 1603-2 and the projection surface determination unit 1604-2, the projection system 1600 can dynamically determine a projection surface more suitable for the content selected by the user, thereby enabling more flexible projection with improved usability.

[0127] C-7. Correction of the projection surface The projection system 1600 shown in Figure 16 has a function to correct projection parameters such as the content selected by the content selection unit 1604-1 and the projection surface determined by the projection surface determination unit 1604-2 in order to maintain projection quality. Furthermore, if the brightness and chromaticity of the projected image differ significantly from the original content due to the characteristics of the background projection surface, the projection parameter correction unit 1605 performs signal processing to bring them closer to the original brightness and chromaticity.

[0128] Section C-7 describes the projection parameter correction function performed by the projection parameter correction unit 1605 on the content selected by the content selection unit 1604-1 and the projection surface determined by the projection surface determination unit 1604-2.

[0129] C-7-1. Restrictions on the projection surface First, Section C-7-1 describes the function of the projection parameter correction unit 1605 in which it imposes restrictions on the projection surface determined by the projection surface determination unit 1604.

[0130] The projection device 1610 is capable of simultaneously projecting images onto multiple projection surfaces with different depth directions using a phase modulation method. However, as shown in Figure 20, if the distance between two different projection surfaces 2001 and 2002 is smaller than the design value, the projection device 1610 may not be able to switch the image completely because the far projection surface is too close. Specifically, as shown in Figures 21 and 22, this occurs when a vehicle is parked in a parking space near a wall such as a fence, and the projection surface determination unit 1604-2 determines the window surface of the vehicle window and the wall surface near the window as the projection surface (Figure 21 shows a view of a vehicle parked near a wall from above, and Figure 22 shows a view of the vehicle from the side).

[0131] Furthermore, while the examples shown in Figures 20 to 22 show two projection surfaces with short distances, it is conceivable that the distances between three or more projection surfaces may be smaller than the design value. In such cases, the projection parameter correction unit 1605 may pick two projection surfaces at a time from multiple projection surfaces (such as a car window and the adjacent wall) and compare them in pairs to determine which projection surface to prioritize. The projection parameter correction unit 1605 may, for example, prioritize the projection surface closer to the user. When the projection system 1600 is mounted on a vehicle, as shown in Figures 21 to 22, the projection parameter correction unit 1605 may prioritize projection surfaces inside the vehicle. This is because projecting the wall outside the vehicle would cause the image to mix with the image of the projection surface closer to the user, i.e., inside the vehicle, making it difficult to see. In the example shown in Figure 20, the projection parameter correction unit 1605 may correct the ratio of brightness allocated to the projection surface 2001 on the near side (the window surface of the car window) and the projection surface 2002 on the far side (the wall surface outside the car) to 100%:0%.

[0132] Furthermore, the projection parameter correction unit 1605 may impose the following restrictions on the projection surface: (1) to (3).

[0133] (1) If the number of projection surfaces is close to the design value of the projection device 1610, the projection parameter correction unit 1605 narrows down the number of projection surfaces to a predetermined number or less according to a predetermined prioritization rule. For example, the projection parameter correction unit 1605 may prioritize selecting projection surfaces closer to the user. If the projection system 1600 is mounted on a vehicle, the projection parameter correction unit 1605 may prioritize selecting projection surfaces inside the vehicle.

[0134] (2) If the amount of light projected onto the projection surface exceeds the design value of the projection device 1610, the projection parameter correction unit 1605 may correct the projection area of ​​the projection surface to be smaller according to a predetermined priority rule. For example, the projection size of each projection surface may be reduced uniformly, or the projection size may be reduced starting from the projection surface closest to the user.

[0135] (3) The projection parameter correction unit 1605 performs a process to reduce the screen size of the original content when the area of ​​the projection surface is smaller than the recommended screen size of the content.

[0136] C-7-2. Determining the Priority of the Projection Surface Section C-7-2 describes the function of the projection parameter correction unit 1605 in determining the priority of multiple projection surfaces determined by the projection surface determination unit 1604-2.

[0137] If the amount of light projected onto each projection surface exceeds the design value of the projection device 1610, the projection parameter correction unit 1605 adjusts the distribution of brightness to each projection surface according to a predetermined prioritization rule. Examples of prioritization rules include (1) and (2) below. However, the following rules are merely examples, and other rules can also be determined.

[0138] (1) Based on location information, ads will be prioritized in locations that the user has set as high priority. (2) Prioritize brightness at locations close to the user.

[0139] Figure 23 shows an example of the operation of the projection parameter correction unit 1605 when the content selection unit 1604-1 selects the content for the in-vehicle news "ABC" and the exterior advertisement "DEF", and the projection surface determination unit 1604-2 determines the window surface as the projection surface for the in-vehicle news "ABC" and the road surface as the projection surface for the exterior advertisement "DEF". For example, in the case where a vehicle is traveling through a busy downtown area where a lot of advertising revenue can be expected by displaying it outside the vehicle during the daytime when brightness is needed, the projection of the exterior advertisement "DEF" is prioritized over the in-vehicle news "ABC", and the allocation of projection light amount to each projection surface (or each content) is corrected. In the example shown in Figure 23, the projection parameter correction unit 1605 allocates 20% of the projection light amount to the in-vehicle news "ABC" and 80% of the projection light amount to the exterior advertisement "DEF". Figure 24 also shows a vehicle equipped with projection system 1600 driving through the city, projecting the in-car news "ABC" onto the window surface and the exterior advertisement "DEF" onto the road surface, each with corrected projection light intensity.

[0140] C-7-3. Correction of the projection surface Section C-7-3 describes the function of the projection parameter correction unit 1605, which corrects projection parameters based on information obtained from the projection surface determination unit 1604-2 and the projection surface definition unit 1603-2 so that the actual image projected onto the projection surface is appropriate. Examples of correction functions include geometric correction, brightness correction, and chromaticity correction. Each correction function will be described below.

[0141] C-7-3-1.Geometric correction The projection parameter correction unit 1605 calculates the size and shape of the image when the content selected by the content selection unit 1604-1 is actually projected onto the projection surface, based on the relative positional relationship between the projection surface and the projection device 1610 determined by the projection surface determination unit 1604-1, and the field of view information of the projection device 1610. When the projection system 1600 is mounted on a vehicle, the relative position of the projection surface and the projection device 1610 can be calculated based on the vehicle's current position information measured by a GPS sensor or the like, or the predicted vehicle route information. The projection parameter correction unit 1605 then corrects the projection parameters to minimize the difference between the recommended size and shape of the content and the calculated size and shape of the projected image. The projection parameter correction unit 1605 also performs so-called trapezoidal correction due to the inclination of the projection surface.

[0142] C-7-3-2. Brightness Correction If there is a difference between the luminance value of the original content signal and the luminance value projected onto the projection surface, the projection parameter correction unit 1605 corrects the video signal so that the projected luminance value approaches the luminance value of the original signal. Of course, the projection parameter correction unit 1605 may also use means other than signal processing to correct the projected luminance value so that it approaches the luminance value of the original signal.

[0143] The projection parameter correction unit 1605 may pre-calculate the luminance value projected onto the projection surface based on information from the projection surface database (attributes, shape, reflectance, luminance, chromaticity, etc. of the projection surface) and the design values ​​of the projection device 1610. Alternatively, the projection parameter correction unit 1605 may receive sensor information from the input unit 1601 that senses the luminance value of the projection surface that has actually been projected, and use it for luminance correction.

[0144] Furthermore, the projection parameter correction unit 1605 calculates the possible output of the projection device 1610 from the total number of projection surfaces onto which the content is projected and the content to be projected, and further calculates the achievable brightness considering the reflectivity of the projection surfaces. Then, it compares the achievable brightness with the brightness of the original signal of the content, and if it is less than the brightness of the original signal, it reduces the output brightness to the achievable brightness, or calculates the setting value (current value, duty cycle value, etc.) of the projection device 1610 at that time.

[0145] C-7-3-3. Chromaticity correction If there is a difference between the chromaticity of the original content signal and the chromaticity projected onto the projection surface, the projection parameter correction unit 1605 performs chromaticity correction in the color space of the video signal so that the projected chromaticity approaches the chromaticity of the original signal. Of course, the projection parameter correction unit 1605 may also perform chromaticity correction to bring the projected chromaticity closer to the chromaticity of the original signal using means other than signal processing. Furthermore, if the chromaticity of a pixel in the content exceeds the displayable chromaticity of the projection device 1610, the projection parameter correction unit 1605 corrects the chromaticity of that pixel to match the displayable chromaticity of the projection device 1610.

[0146] The projection parameter correction unit 1605 may pre-calculate the chromaticity projected onto the projection surface based on information from the projection surface database (such as the attributes, shape, reflectance, brightness, and chromaticity of the projection surface) and the design values ​​of the projection device 1610. Alternatively, the projection parameter correction unit 1605 may receive sensor information from the input unit 1601 that senses the chromaticity of the projection surface that has actually been projected, and use it for chromaticity correction.

[0147] White balance correction is performed with priority given to the light source control of the projection device 1610. For this reason, the projection parameter correction unit 1605 may calculate the light source control parameters and pass this information to the projection device 1610.

[0148] Furthermore, the projection parameter correction unit 1605 may perform correction of projection parameters using either a chromaticity-prioritizing or luminance-prioritizing method. In the case of chromaticity-prioritizing, the correction process is performed in the order of chromaticity correction → luminance correction, but in the case of luminance-prioritizing, the order of processing is not limited to this.

[0149] C-7-3-4. Specific Examples Section C-7-3-4 describes specific examples of projection parameter correction performed by the projection parameter correction unit 1605.

[0150] Figure 25 shows how the projection parameter correction unit 1605 performs corrections according to the distance from the projection device 1610. When projecting images to three locations: near, medium, and far, the projection parameter correction unit 1605 performs brightness correction so that the image becomes brighter at far distances and darker at near distances. In addition, if the projection surface is bright, the brightness of the image projected onto that projection surface may be reduced. Furthermore, as geometric correction, the projection parameter correction unit 1605 performs scaling processing on the image so that the size of the image projected onto each projection surface becomes constant.

[0151] Figure 25 shows an example of projecting arrows for indicating the direction of travel (or navigation) onto three projection surfaces: near, medium, and far. The projection parameter correction unit 1605 first assigns projection light amounts of 20%, 40%, and 100% to the near, medium, and far projection surfaces, respectively. Here, if the projection surface is bright, the contrast between the background (projection surface) and the projected content will be small, so a correction is performed to adjust the contrast by increasing the brightness. In the example shown in Figure 25, the medium-range projection surface is bright, so a correction is performed to increase the assigned projection light amount from 40% to 60%. By performing such brightness correction, the projection light amount can be effectively distributed according to the distance and the brightness of the projection surface, thereby enabling high brightness of the projection device 1610. In addition, the projection parameter correction unit 1605 also performs trapezoidal correction as a geometric correction, which involves enlarging the image projected onto the near projection surface and shrinking the image projected onto the far projection surface. This ensures that the size of the image projected onto the short-range, medium-range, and long-range projection surfaces remains constant.

[0152] Figure 26 shows a specific example of projection parameter correction when the projection system 1600 is mounted on a vehicle (not shown in Figure 26). This figure shows an example where arrows 2601 to 2604 indicating the direction of travel are projected onto multiple locations on the road surface when the vehicle approaches a Y-junction where the main road branches off to the left onto a side road. The projection parameter correction unit 1605 first assigns projection light intensity to each projection surface according to the distance, so that distant areas are brighter and closer areas are darker. Therefore, projection light intensity is assigned in the order of arrow 2601, arrow 2602, arrow 2603, and arrow 2604. Furthermore, since the projection surface on the gutter is brighter than the projection surface on the asphalt, the projection light intensity assigned to arrow 2602 projected onto the gutter is increased to adjust the contrast. By performing such brightness correction, the projection light intensity can be effectively distributed according to the distance and the brightness of the projection surface, thereby achieving high brightness of the projection device 1610. Furthermore, the projection parameter correction unit 1605 performs geometric correction, such as trapezoidal correction, according to the inclination and shape of the projection surface. In addition, the projection parameter correction unit 1605 enlarges the image projected onto nearby projection surfaces and reduces the image projected onto distant projection surfaces so that the size of the image projected onto each projection surface is constant (the reduction and enlargement processing is performed in the order of arrows 2601, 2602, 2603, and 2604). Through these projection parameter corrections, arrows 2601 to 2604, which appear the same to the user (for example, a driver sitting in the driver's seat), can be projected onto multiple projection surfaces on the road surface that differ in distance and attributes.

[0153] Figures 27 and 28 show how an out-of-vehicle advertisement is projected onto a sidewalk towards pedestrians near the vehicle using a projection system 1600 mounted on the vehicle. However, Figure 27 shows the projection of an out-of-vehicle advertisement from the vehicle onto the sidewalk at a certain time T, while Figure 28 shows the projection of an out-of-vehicle advertisement from the vehicle onto the sidewalk at time T+ΔT, which is ΔT later. In the example shown in Figure 27, at time T, the content selection unit 1604-1 projects the selected content (out-of-vehicle advertisement) for the pedestrian onto the projection surface determined by the projection surface determination unit 1604-2 for the pedestrian. During ΔT, the vehicle moves in the direction of travel and is approaching an intersection and about to turn left. Therefore, the distance from the projection device 1610 to the projection surface on the sidewalk increases, and the inclination of the projection surface with respect to the optical axis also changes. Therefore, the projection parameter correction unit 1605 corrects (brightens) the amount of projected light allocated to the projection surface on the sidewalk according to the change in distance to the projection surface, and also performs geometric correction according to the inclination of the projection surface with respect to the optical axis. The relative position of the projection surface and the projection device 1610 can be calculated based on the current location information of the vehicle measured by a GPS sensor or the like, and the predicted route information of the vehicle. As a result, as shown in Figure 28, the coordinates of the projection surface on the sidewalk and the size of the projected image remain constant with respect to pedestrians, so that pedestrians can continue to see the same out-of-vehicle advertisement from time T to time T+ΔT.

[0154] When content is presented continuously, the projection surface (sidewalk) assigned to the user (pedestrian) is fixed, and the projection parameter correction unit 1605 updates the projection parameters (geometric correction, brightness correction, chromaticity correction) frame by frame. This ensures usability while the vehicle equipped with the projection system 1600 is in motion.

[0155] C-8. Modality of Input Information Up until now, the input unit 1601 has been described as primarily receiving image information from image sensors, distance sensors, etc., and performing user recognition by the user recognition unit 1602-1 and user information definition by the user definition unit 1602-2.

[0156] The input unit 1601 may also acquire audio data using a microphone or the like. In this case, conversations, sounds from the user's daily life, and other environmental sounds can be acquired from the audio data, making it possible to more accurately understand the user's attributes and state.

[0157] C-9. Metadata detection from user-owned devices It has become common for users to carry multi-functional information devices such as smartphones, tablets, and smartwatches. These types of devices often store various user metadata, such as schedule information, e-ticket information, and information about services and facilities to which they have registered accounts.

[0158] Therefore, the input unit 1601 may acquire user metadata from the user's device. In this case, the user definition unit 1602-2 can define user information more accurately, and the content selection unit 1604-1 can select content that is more suitable for the user's attributes and state.

[0159] D. Applications Figure 29 schematically shows the configuration of a projection system 2900 to which this disclosure is applied. The illustrated projection system 2900 comprises an input unit 2901, a user information detection unit 2902, a projection environment recognition unit 2903, an output control unit 2904, a projection parameter correction unit 2905, and an application information storage unit 2906. However, components that are the same as those included in the projection system 1600 shown in Figure 16 are shown with the same names. The main feature of the projection system 2900 is that it can be deployed to a wide range of applications (used in various scenes) by managing application information consisting of parameters related to the assignment of projection surfaces and content selection for each application (or system use).

[0160] The input unit 2901 receives sensor information acquired by sensors installed within the space that can be projected by the projection system 2900. Alternatively, the input unit 2901 may be the sensor itself installed in the space. For example, the input unit 2901 receives sensor information from an in-vehicle sensor installed inside the vehicle. If the user recognition unit 2902 also recognizes users outside the vehicle, the input unit 2901 shall also receive sensor information from outside the vehicle.

[0161] The user information detection unit 2902 includes a user recognition unit 2902-1 and a user definition unit 2902-2. The user recognition unit 2902-1 uses a posture estimation model, such as OpenPose, to detect the number of users, their positions, and the orientation and gaze of their faces from sensor information supplied from the input unit 2901. The user definition unit 2902-2 then defines the characteristics and state of the users recognized by the user recognition unit 2902-1 and stores the characteristics and state defined for each user in the corresponding user entry in the user characteristics database.

[0162] The projection environment recognition unit 2903 includes a projection surface detection unit 2903-1 and a projection surface definition unit 2903. Based on sensor information supplied from the input unit 2901, the projection surface detection unit 2903-1 detects as projection surfaces areas that can actually be projected by the projection device 2910 within the space that can be projected by the projection system 2900 (same as above). The projection system 2900 uses a projection surface database to manage the characteristics of each projection surface detected by the projection surface detection unit 2903-1. An entry for each projection surface detected by the projection surface detection unit 2903-1 is provided in the projection surface database. The projection surface definition unit 2903-2 recognizes the characteristics of the projection surface detected by the projection surface detection unit 2903-1, such as attributes, shape, area, reflectance, chromaticity, and brightness, and stores this information in the corresponding entry in the projection surface database.

[0163] The output control unit 2904 includes a content selection unit 2904-1 and a projection surface determination unit 2904-2. The content selection unit 2904-1 selects content to be displayed to the user based on user information recognized by the user recognition unit 2902. The projection surface determination unit 2904-2 determines the projection surface from among the projection surfaces detected by the projection surface detection unit 2903-1 to project the content image. Information regarding the content selected by the content selection unit 2904-1 and the projection surface determined by the projection surface determination unit 2904-2 is stored in the user characteristics database (as described above). However, the content selection unit 2904-1 selects content according to the provisions regarding content selection based on the relevant application information, which are managed by the application information storage unit 2906. The projection surface determination unit 2904-2 determines the projection surface according to the provisions regarding projection surface allocation based on the relevant application information.

[0164] The projection parameter correction unit 2905 corrects projection parameters such as limiting and prioritizing the projection surface when projecting images from the projection device 2910, geometric correction of the projected image, and brightness and chromaticity correction in order to maintain projection quality (see Section C-7 above for details).

[0165] The projection device 2910 is a phase-modulation type projection device as described in Section B above, and is capable of simultaneously projecting onto multiple projection surfaces that differ in the vertical, horizontal, and depth directions. Based on the projection size, projection brightness, and chromaticity corrected by the projection parameter correction unit 2905, a display image generation process is performed to generate an image to be projected by the projection device 2910. This display image generation process may be performed by the output control unit 2904 or by the projection parameter correction unit 2905. The display image generation process is as described in Section C-5 above, so a detailed explanation is omitted here. The projection device 2910 projects the image based on the information determined by the display image generation process.

[0166] The application information storage unit 2906 manages the application information database by creating a database of application information consisting of parameters related to the assignment of projection surfaces and content selection for each application (or system use). The content selection unit 2904-1 then selects content according to the provisions for content selection based on the relevant application information. The projection surface determination unit 2904-2 then determines the projection surface according to the provisions for projection surface assignment based on the relevant application information.

[0167] Figure 30 shows an example of the configuration of the application information database managed by the application information storage unit 2906. The application information database shown in the figure stores application information for seven applications: "Advertising projection inside and outside the vehicle," "Multiple surface projection inside the vehicle (headrests, ceiling, pillars, etc.)," ​​"Food projection mapping," "Projection mapping events (fashion shows, bowling)," "Viewing area in the CAVE system (supports multiple people)," "Viewing area in the CAVE system (energy saving support for a single surface)," "Touch indicator for aerial displays," and "Structured light source system."

[0168] In the illustrated application information database, the multiplane display flag indicates whether to assign multiple projection planes to a single user. An application with this flag set to TRUE will assign multiple projection planes to a single user, while an application with this flag set to FALSE will assign only one projection plane to a single user.

[0169] Furthermore, the recommended number of planes is the number of projection surfaces allocated per user. For applications where the multiplane display flag is TRUE, the number of projection surfaces used by each user can be predetermined. On the other hand, for applications where the multiplane display flag is FALSE, the recommended number of planes is necessarily 1.

[0170] The multi-content flag indicates whether the information projected onto all projection planes is a single piece of content, provided that the multi-plane display flag is TRUE, when information is presented to multiple projection planes. An application with the multi-content flag set to FALSE presents information relevant to all projection planes (i.e., it is single content). An application with the multi-content flag set to TRUE presents information that is not relevant to each projection plane (i.e., it is multi-content).

[0171] Note that the assignment of the multi-plane display flag, recommended number of planes, and multi-content flag to each application in the application information database shown in Figure 30 is just an example and is not necessarily limited to the example shown in Figure 30.

[0172] D-1. Example 1 Section D-1 describes an example in which the projection system 2900 is applied to the application "Multiple surface projection inside a vehicle".

[0173] In this embodiment, as shown in Figure 31, the projection device 2910 and sensor 3101 are installed in the rear of the vehicle interior, which is equipped with three rows of seats. Sensor 3101 includes an image sensor such as an RGB camera and a distance sensor, and the sensor information acquired by sensor 3101 is input to the projection system 2900 from the input unit 2901. Furthermore, the projection device 2910 can simultaneously project onto multiple projection surfaces that differ in the up / down / left / right and depth directions using a phase modulation method. As shown in Figure 30, in the application "Multiple Surface Projection in Vehicle Interior," the multiplane display flag is set to TRUE, the recommended number of planes is set to 4, and the multicontent flag is set to TRUE.

[0174] As shown in Figure 5, the projection surface detection unit 2903-1 detects multiple projection surfaces on the headrest, ceiling, pillars, etc. The projection surface definition unit 2903-2 then recognizes the characteristics of each detected projection surface and stores them in the projection surface database.

[0175] The content selection unit 2904-1 selects content to be displayed for each target user, corresponding to the number of recommended planes. Users who have indicated "Yes" for information presentation in the user characteristics database are the ones to be presented with content. Since the multi-content flag is set to TRUE in this application, the content selection unit 2904-1 selects only the number of recommended planes (4) of unrelated content for each target user.

[0176] The projection plane determination unit 2904-2 performs the process of assigning projection planes to each target user in the user characteristics database. Since the multiplane display flag is TRUE in this application, the projection plane determination unit 2904-2 scans all projection planes for each target user and determines projection planes for each user up to the number of recommended planes specified in the user characteristics database. However, it determines the number of projection planes to actually assign to each user so as not to exceed the maximum number of projection planes of the projection device 2910. If more projection planes than the number of recommended planes are available for a given user, the projection plane determination unit 2904-2 determines projection planes up to the number of recommended planes based on a predetermined prioritization rule (described above).

[0177] If the same projection plane is a candidate for multiple target users, the display information will be compared among the users. If the display information is the same, the same projection plane will be assigned to all target users. If the display information differs among the users, the projection plane will be assigned to one user as is, and a different projection plane will be assigned to the other users. If no other projection planes are available, the assignment of a projection plane to that user will be abandoned.

[0178] Figure 32 shows an example of a user characteristics database built for the application "Multi-plane projection inside a vehicle." However, the multi-plane display flag is set to TRUE, the recommended number of planes is 4, and the maximum number of projection surfaces for projection device 2910 is 10.

[0179] D-2. Example 2 Section D-2 describes an example in which the projection system 2900 is applied to the application "Food Projection Mapping." "Food Projection Mapping" is an interaction that projects images of appetizing characters or other visuals onto a plate of food, as shown in Figure 33, for example. By applying the projection system 2900 to this application, the projected image on the plate can always remain in focus, even when the user lifts the plate.

[0180] In this embodiment, a projection device 2910 and sensors (such as image sensors like RGB cameras and distance sensors) are installed on the ceiling of a room such as a kitchen (not shown), and sensor information acquired by the sensors is input to the projection system 2900 from the input unit 2901. As shown in Figure 30, in the application "Cooking Projection Mapping", the multiplane display flag is set to TRUE, the recommended number of planes is 3, and the multicontent flag is set to FALSE.

[0181] As shown in Figure 33A, the projection surface detection unit 2903-1 detects three projection surfaces 3301 to 3303 for projecting content related to food (cake) in a projection environment where a plate of food (cake in the illustrated example) is placed on a table. The projection surface definition unit 2903-2 then recognizes the characteristics of each detected projection surface 3301 to 3303 and stores them in the projection surface database. In a projection environment where a plate is placed on a table as shown in the figure, projection surfaces are detected on both the table and the plate.

[0182] The content selection unit 2904-1 selects only the recommended number of content items (3) for the target user and stores them in the user characteristics database. Since the multi-content flag is set to FALSE in this application, the content selection unit 2904-1 selects a series of 3 content items related to cooking (cake) for the target user.

[0183] Figure 34 shows an example of the user characteristics database configuration when the content selection unit 2904-1 selects a series of content, "Rabbit Cafe," for a single target user (user ID #Uaaaa) detected by the user information detection unit 2902. "Rabbit Cafe" consists of a total of six related content items, a-1 to a-3 and b-1 to b-3. However, the user characteristics database shown in Figure 34 is a provisional one where content candidates have been selected for the target user, and the linking of projection planes to content has not yet been determined. Also, since "Rabbit Cafe" consists of six related content items, exceeding the recommended number of planes (3), it is necessary to narrow down the selected content to the recommended number of planes.

[0184] The projection plane determination unit 2904-2 performs the process of assigning projection planes to target users in the user characteristics database. Since the multi-content flag is FALSE in this application, the projection plane determination unit 2904-2 scans the target user for the number of content items selected by the content selection unit 2904-1 and assigns the target user the number of recommended planes that match the recommended values ​​(such as recommended screen size) associated with the content. Then, the projection plane determination unit 2904-2 stores the projection plane assigned to each of the content items in the corresponding entry in the user characteristics database.

[0185] If the same projection plane is a candidate for multiple target users, the display information will be compared among the users. If the display information is the same, the same projection plane will be assigned to all target users. If the display information differs among the users, the projection plane will be assigned to one user as is, and a different projection plane will be assigned to the other users. If no other projection planes are available, the assignment of a projection plane to that user will be abandoned.

[0186] In the provisional user characteristics database after content selection shown in Figure 34, six related content items, exceeding the recommended number of three planes, have been selected for the target user (User ID: #Uaaaa). In such cases, the projection plane determination unit 2904-2 narrows down the six related content items to three related content items (a-1, a-3, a-2), which is the recommended number of planes, based on a predetermined prioritization rule (described above), and assigns projection planes (#0001, #0005, #0010) that match the recommended values ​​associated with each content item to each item. Figure 35 shows the final user characteristics database constructed based on the user characteristics database shown in Figure 34. Figure 33B shows an example of food projection mapping where three related content items are projected onto projection planes 3301-3303 detected on both the table and the plate. In a projection environment where a plate is placed on a table as shown in Figure 33A, two contents, "Rabbit Cafe a-1" and "Rabbit Cafe a-2," are projected onto projection surfaces 3301 and 3302 detected on the table, and "Rabbit Cafe a-3" is projected onto projection surface 3303 detected on the plate. Furthermore, Figure 33C shows an example of food projection mapping when the plate is lifted from the table. In this disclosure, when the plate is lifted, the two contents, "Rabbit Cafe a-1" and "Rabbit Cafe a-2," projected onto the table remain unchanged. On the other hand, the content "Rabbit Cafe a-3," projected onto the plate, is projected onto the lifted plate as an image with the same size, brightness, and focus as when the plate was placed on the table. Incidentally, if you were to perform similar projection mapping using a typical projection device, when you lift the plate, only the content on the plate would shrink, resulting in an image that is bright and out of focus, which is a problem.

[0187] D-3. Example 3 Section D-3 describes an example of applying the projection system 2900 to the application "Projection Mapping Event" to perform moving effects. In conventional projection mapping, the projection object that serves as the screen is fixed and does not move dynamically. In contrast, by applying the projection system 2900, projection mapping with a high degree of freedom in the depth direction can be continued even when the projection object moves dynamically, improving usability.

[0188] When applying the projection system 2900 to the application "Projection Mapping Event" to perform moving effects, set the multiplane display flag to TRUE, the recommended number of planes to 7, and the multicontent flag to FALSE, as shown in Figure 30.

[0189] Image sensors such as RGB cameras and distance sensors are installed in various locations within the bowling alley, such as on the ceiling, and sensor information is input from the input unit 2901. The user recognition unit 2902-1 recognizes users within the bowling alley from the sensor information supplied from the input unit 2901, and the user definition unit 2902-2 defines the characteristics and status of the recognized users and stores them in the user characteristics database. The user definition unit 2902-2 defines "Yes" for target users who are the target of information presentation.

[0190] The projection surface detection unit 2903-1 detects areas within the bowling alley that can be projected onto as projection surfaces based on sensor information supplied from the input unit 2901. In this application, not only stationary objects such as the lane floor but also the surfaces of dynamically moving objects such as rolling balls are detected as projection surfaces. This makes it possible to project images that follow the rolling ball, enabling moving effects. The projection surface definition unit 2903-2 recognizes the characteristics of projection surfaces such as lanes and rolling balls and stores them in the projection surface database. Figure 36 shows examples of multiple projection surfaces detected on the lane and the surface of the rolling ball. In the figure, each projection surface is shown in gray.

[0191] The content selection unit 2904-1 selects only the recommended number of planes (7 items) of content for the target user. Since the multi-content flag is set to FALSE in this application, the content selection unit 2904-1 selects a series of content for the target user. The content selection unit 2904-1 then stores the series of content and the recommended projection plane information (recommended size, recommended brightness, etc.) for each in the user characteristics database.

[0192] The projection plane determination unit 2904-2 performs the process of assigning projection planes to the target user in the user characteristics database. Since the multi-content flag is set to FALSE in this application, the projection plane determination unit 2904-2 scans the target user for the number of content items selected by the content selection unit 2904-1 and assigns the target user the number of recommended planes that match the recommended values ​​(such as recommended screen size) associated with the content. Then, the projection plane determination unit 2904-2 stores the projection plane assigned to each of the content items in the corresponding entry in the user characteristics database.

[0193] Figure 37 shows the final user characteristics database in the application, which stores a series of content selected for the target user and information on the projection surface assigned to each piece of content.

[0194] As shown in Figure 36, this application can use the lane floor, the surface of the rolling ball, or the front wall (masking) as projection surfaces to display information to target users, such as bowlers who throw the ball. By using the projection system 2900 according to this disclosure in this application, the display can be maintained at the same brightness while the ball is rolling, thus improving usability.

[0195] However, projecting an image onto the lane before the throw makes it difficult for the bowler to read the lane conditions, and projecting an image in front of the rolling ball makes it difficult to observe the ball's trajectory. Therefore, the projection surface determination unit 2904-2 may assign the projection surface on the lane after the ball has passed to the content, as shown in Figures 38 and 39. As shown in Figure 38, immediately after the throw, only one projection surface 3801 in front of the lane is assigned to the content. In this way, no image is projected onto the lane before the throw or before the ball has passed, so visibility is not obstructed, and usability is further improved. Subsequently, when the ball rolls down the lane and reaches the vicinity of the pin deck, four projection surfaces 3901 to 3904 on the lane are assigned to the content. Therefore, as the ball moves, the area on the lane that can be projected expands, and more projection surfaces can be effectively utilized.

[0196] While Section D-3 describes a specific example of applying the "Projection Mapping Event" application to bowling, it can be similarly applied to other sports competitions and non-sports events to create moving effects. For example, in a fashion show, the runway floor or the costumes and bodies of the models walking on the runway can be used as projection surfaces to create moving effects.

[0197] D-4. Example 4 Section D-4 describes an example in which the projection system 2900 is applied to the application "Viewing of the gaze area in a CAVE system (supports multiple users)".

[0198] Figure 40 shows an example of the CAVE system 4000. The illustrated CAVE system 4000 consists of a space (such as a room) enclosed on all four sides by walls, and each wall can be used as a projection surface. Users can enter the space and enjoy the images projected onto the walls. By applying the projection system 2900 to the CAVE system 4000, content selected by each user can be presented to the area that each user is focusing on. Furthermore, since the projection system 2900 can simultaneously project onto multiple different projection surfaces in the up / down, left / right, and depth directions using the projection device 2910, it is more space-efficient, cost-efficient, and energy-efficient compared to using a number of projectors corresponding to the number of projection surfaces.

[0199] When applying projection system 2900 to the application "Viewing of the gaze area in the CAVE system (multi-user support)", set the multiplane display flag to TRUE, the recommended number of planes to 5, and the multicontent flag to TRUE, as shown in Figure 30.

[0200] Image sensors 4020, such as RGB cameras and distance sensors, are installed above or on the ceiling of the space where the CAVE system 4000 is constructed, to capture users and projection environments present in the space. The user recognition unit 2902-1 recognizes users in the space from sensor information supplied from the input unit 2901, and the user definition unit 2902-2 defines the characteristics and state of the recognized users and stores them in the user characteristics database. Since users who enter the space basically intend to view the video, the user definition unit 2902-2 defines the display of information for all users in the space as Yes (i.e., target users). In addition, the projection surface detection unit 2903-1 detects the four walls surrounding the space as projection surfaces based on the sensor information supplied from the input unit 2901. Note that one wall surface may be detected as one projection surface, or one wall surface may be divided into multiple regions and each region may be detected as a projection surface.

[0201] The content selection unit 2904-1 selects only the number of recommended planes (5) of content for each target user. Since the multi-content flag is set to TRUE in this application, the content selection unit 2904-1 selects content that is not relevant to the target user. The content selection unit 2904-1 then stores the content selected for each target user and the recommended projection plane information (such as recommended size and recommended brightness) for that content in the user characteristics database.

[0202] The projection plane determination unit 2904-2 performs the process of assigning projection planes to target users in the user characteristics database. Since the multi-content flag is set to TRUE in this application, the projection plane determination unit 2904-2 scans all projection planes for each target user and determines projection planes for each user up to the number of recommended planes specified in the user characteristics database. However, it determines the number of projection planes to actually assign to each user so as not to exceed the maximum number of projection planes of the projection device 2910. If more projection planes than the number of recommended planes are available for a given user, the projection plane determination unit 2904-2 determines projection planes up to the number of recommended planes based on a predetermined prioritization rule (described above).

[0203] If the same projection plane is a candidate for multiple target users, the display information will be compared among the users. If the display information is the same, the same projection plane will be assigned to all target users. If the display information differs among the users, the projection plane will be assigned to one user as is, and a different projection plane will be assigned to the other users. If no other projection planes are available, the assignment of a projection plane to that user will be abandoned.

[0204] Figure 41 shows an example of a user characteristics database constructed in the application "Viewing of the gaze area in the CAVE system (multi-user support)". However, the multiplane display flag is set to TRUE, the recommended number of planes is 5, and the maximum number of projection surfaces for projection device 2910 is 10.

[0205] The projection system 2900 controls content output in the CAVE system 4000 based on a user characteristics database as shown in Figure 41, so that, as shown in Figure 40, each user 4001 and 4002 can simultaneously project content determined based on the attributes of users 4001 and 4002 onto their respective projection surfaces 4011 and 4012, which are determined to be on nearby walls. For the sake of simplicity in the diagram, the CAVE system 400 shown in Figure 40 only accommodates two users in the space, but even when accommodating three or more users, it is possible to similarly assign a projection surface to each user and present individual content.

[0206] D-5. Example 5 Section D-4 above described an example of a CAVE system for viewing projected images of an arm on another person's arm. Section D-5 describes an example of applying the projection system 2900 to the application "Viewing of a gaze area in a CAVE system (energy saving on a single surface)". Here again, we assume the CAVE system 4000 with the configuration shown in Figure 40.

[0207] When applying projection system 4000 to the application "Viewing the gaze area in the CAVE system (energy saving on a single plane)", set the multiplane display flag to FALSE, the recommended number of planes to 1, and the multicontent flag to FALSE, as shown in Figure 30.

[0208] The operation of the user information detection unit 2902 and the projection environment recognition unit 2903 is the same as in section D-4 above, so a detailed explanation is omitted here.

[0209] Since the multi-content flag is set to FALSE, the content selection unit 2904-1 selects one piece of content for each target user. Also, since the multi-plane display flag is set to FALSE, the projection plane determination unit 2904-2 determines one projection plane for each target user that matches the recommended screen information for the content assigned to that user. The information selected and determined by the content selection unit 2904-1 and the projection plane determination unit 2904-2, respectively, is then stored in the user characteristics database.

[0210] As a result, since each user accommodated in the space is assigned a single projection surface for viewing the video, compared to Embodiment 4 above, where multiple projection surfaces are assigned to each user, energy savings can be achieved in the CAVE system 4000 by suppressing the output of the projection device 2910.

[0211] D-6. Example 6 Section D-6 describes an example in which the projection system 2900 is applied to the application "Touch indicator for aerial display".

[0212] Here, an aerial display refers to a display device that can display images three-dimensionally in empty real space (air). For example, it uses one or more stationary projectors and combines lenses, half-mirrors, etc., to focus light in the air and display an image in the air (see, for example, Patent Document 3).

[0213] As one application of aerial displays, a UI (User Interface) screen can be displayed in the empty air where no physical object exists, for example, for a user to operate a device. Using an aerial display, a UI screen can be placed at any arbitrary location without the need to install a physical real-world display device. Fig. 42 shows an example in which a UI screen 4200 including menu buttons for file operations such as open, close, save, and print is displayed by an aerial display. However, even when a user attempts to touch a menu button on the UI screen displayed by the aerial display, there is no physical entity that can be actually touched and operated, so the user cannot obtain a tactile sensation at the fingertip that the user has touched the menu button, which makes the operation difficult.

[0214] Accordingly, using the projection system 2900, an indicator that indicates the distance between the aerial display UI screen and the user's fingertip attempting to touch the screen is displayed, and the tactile sensation at the fingertip is compensated for by the visual information provided by the indicator. Specifically, when the distance between the user's fingertip and the UI screen is large, a dark and small indicator is displayed, and when the user's fingertip approaches the UI screen, a bright and large indicator is displayed.

[0215] When applying the projection system 2900 to the application "Aerial Display Touch Indicator", as shown in Fig. 30, the multi-plane display flag is specified as TRUE, the recommended number of planes is specified as 4, and the multi-content flag is specified as FALSE.

[0216] Since the operations of the user information detection unit 2902 and the projection environment recognition unit 2903 are the same as those in Section D-4 above, detailed description thereof is omitted herein. Since the multi-content flag is specified as FALSE, the content selection unit 2904-1 selects a series of contents to be allocated to a target user based on the user information defined by the user definition unit 2902-1. Further, the projection surface determination unit 2904-2 determines projection surfaces matching the recommended screen information of the content allocated to the target user one by one. Then, the information respectively selected and determined by the content selection unit 2904-1 and the projection surface determination unit 2904-2 is stored in the user characteristic database.

[0217] FIGS. 43 and 44 show an example in which the projection system 2900 is applied to the application "touch indicator for aerial display". FIG. 43 shows a state where a dark and small indicator 4301 is displayed when the distance of the user's fingertip from the UI screen 4200 shown in FIG. 42 is large. Further, FIG. 44 shows a state where a bright and large indicator 4401 is displayed when the user's fingertip approaches the UI screen 4200. The user can grasp the sense of distance to the UI screen 4200 based on the visual information obtained from the indicators 4301 and 4401 shown in FIGS. 43 and 44 even if the tactile sensation of the UI screen 4200 cannot be obtained. [Industrial Applicability]

[0218] The present disclosure has been described in detail above with reference to specific embodiments. However, it is obvious that those skilled in the art can make modifications and substitutions to the embodiments without departing from the scope of the present disclosure.

[0219] Although the present specification has been described focusing on embodiments in which the projection system according to the present disclosure is applied to vehicles and the like, the gist of the present disclosure is not limited thereto.

[0220] Furthermore, the projection system described herein basically uses a phase-modulation projection device to simultaneously project onto multiple projection surfaces that differ in the up / down, left / right, and depth directions. However, if there are no constraints such as space efficiency or energy efficiency, other types of projection devices such as amplitude-modulation projection devices (even with amplitude-modulation, it is possible to realize display on multiple surfaces with different depths using the principle of holography) or multi-projectors can also be used.

[0221] In short, this disclosure has been explained in the form of examples, and the contents of this specification should not be interpreted restrictively. The claims should be considered in order to determine the gist of this disclosure.

[0222] Furthermore, this disclosure may also take the following form.

[0223] (1) A user recognition unit that recognizes users present in space, A projection environment recognition unit recognizes a projection surface capable of projecting an image into the aforementioned space, A control unit controls the projection device to project an image onto the projection surface recognized by the projection environment recognition unit for the user recognized by the user recognition unit, A projection system equipped with the following features.

[0224] (1-1) The space is a vehicle or other mobile device, The user recognition unit recognizes users inside and outside the mobile device. The projection environment recognition unit recognizes the projection surfaces inside and outside the mobile device. The projection system described in (1) above.

[0225] (1-2) The space is surrounded by walls, The projection environment recognition unit recognizes the projection surface from the wall surface surrounding the space. The projection system described in (1) above.

[0226] (2) The projection device is capable of projecting images onto multiple surfaces simultaneously, The control unit controls the projection device to simultaneously project images onto two or more projection surfaces recognized by the projection environment recognition unit, The projection system according to claim 1.

[0227] (3) At least one of the user recognition unit and the projection environment recognition unit performs recognition based on sensor information detected by a sensor installed in the space, The projection system according to any one of the above (1) or (2).

[0228] (3-1) The sensor includes at least one of an image sensor, a depth sensor, a thermographic camera, an ultrasonic sensor, and a touch sensor, The projection system according to the above (3).

[0229] (4) The projection environment recognition unit detects, within the projectable range of the projection device, a region that satisfies a condition defined by one or more threshold values, such as an area equal to or larger than a predetermined threshold, a curvature equal to or smaller than a predetermined threshold, or a gradient equal to or larger than a predetermined threshold, as a projection surface, The projection system according to any one of the above (1) to (3).

[0230] (4-1) The threshold value is defined for each user, each content to be projected, or each application to which the projection system is applied, The projection system according to the above (4).

[0231] (5) The projection device is capable of simultaneously projecting images onto a plurality of different surfaces in vertical, horizontal, and depth directions, The projection environment recognition unit recognizes a plurality of projection surfaces different in vertical, horizontal, and depth directions, The projection system according to any one of the above (1) to (4).

[0232] (6) The projection device is a projection device comprising a phase modulation type spatial light modulator, The projection system according to the above (5).

[0233] (7) The user recognition unit defines the user's characteristics and state. A projection system as described in any of (1) through (6) above.

[0234] (7-1) The user recognition unit detects at least one of the following: the number of users, the location of the users, the orientation of the users' faces, and the gaze of the users, and defines the characteristics and state of the users based on the detection results. The projection system described in (7) above.

[0235] (7-2) The user recognition unit recognizes the user's posture using a posture estimation model (Openpose) from image information captured from the space by an RGB camera or depth sensor. The projection system described in (7-1) above.

[0236] (7-3) The user recognition unit defines user attributes (gender, age, occupation, etc.) and user state (awake or asleep) as user characteristic data. The projection system described in (7) above.

[0237] (7-4) The user recognition unit defines user attributes by associating stereotype information. The projection system described in (7-3) above.

[0238] (7-5) The user recognition unit defines the user's state based on the number of blinks or the movement of the eyes. The projection system described in (7-3) above.

[0239] (7-6) The user recognition unit defines the user's characteristics or state based on metadata held by the device the user possesses. The projection system described in (7) above.

[0240] (8) The system further includes a content selection unit that selects content to be displayed to the user based on defined user information. The projection system described in (7) above.

[0241] (8-1) The content selection unit selects content based on a predetermined priority. The projection system described in (8) above.

[0242] (8-2) Prioritization is based on at least one of the following: user preferences, user location, and current time. The projection system described in (8-1) above.

[0243] (8-3) Link the selected content to the user. The projection system described in (8) above.

[0244] (8-4) The selected content and its display information, including its recommended size and recommended brightness, are associated with the user. The projection system described in (8) above.

[0245] (9) Further comprising a projection surface determination unit that determines the projection surface on which the selected content is projected, The projection system described in (8) above.

[0246] (9-1) The projection surface determination unit determines whether the projection surface recognized by the projection environment recognition unit is within the user's field of view, and assigns the user a projection surface based on the determination result. The projection system described in (9) above.

[0247] (9-2) The projection surface determination unit determines the projection surface based on a predetermined priority. The projection system described in (9) above.

[0248] (9-3) The priority includes at least one of the following: the area of ​​the projection surface, the size of the field of view relative to the user, the distance between the user and the projection surface, the degree of matching between the content selected by the user and the recommended screen size or recommended field of view, the degree of contrast between the background of the projection surface and the content, and the low brightness of the background of the projection surface. The projection system described in (9-2) above.

[0249] (9-4) The determined projection surface is linked to the user or content. The projection system described in (9) above.

[0250] (10) The projection environment recognition unit further detects information about the recognized projection surface. A projection system as described in any of (1) through (9) above.

[0251] (11) The projection environment recognition unit defines the attributes, shape, area, and characteristics (reflectance, brightness, chromaticity) of the projection surface. The projection system described in (10) above.

[0252] (11-1) The projection environment recognition unit calculates the shape, area, and characteristics of the projection surface based on sensor information detected by sensors installed in the space. The projection system described in (11) above.

[0253] (11-2) The projection environment recognition unit performs clustering of projection surfaces based on the data of each projection surface. The projection system described in (11) above.

[0254] (12) The projection parameter correction unit further comprises a projection parameter correction unit that corrects the projection parameters for the projection surface determined by the projection surface determination unit. The projection system described in (9) above.

[0255] (13) The projection parameter correction unit limits at least one of the following based on the design values ​​of the projection device: the distance between different projection surfaces in the depth direction, the number of projection surfaces, or the projection size. The projection system described in (12) above.

[0256] (14) The projection parameter correction unit determines the priority of the multiple projection surfaces determined by the projection surface determination unit. The projection system described in either (12) or (13) above.

[0257] (15) The projection parameter correction unit performs correction on at least one of the brightness, chromaticity, and size of the projected image on the projection surface. A projection system as described in any of (12) to (14) above.

[0258] (16) The sensor includes a microphone, The user recognition unit recognizes the user's state based on voice data, conversation, or background noise input from the microphone. The projection system described in (3) above.

[0259] (17) The content selection unit selects content to be displayed to the user based on user information recognized from metadata of the device the user possesses. The projection system described in either (8) or (9) above.

[0260] (18) Further comprising an application information storage unit that stores application information defining the display method for each application, Based on application information, perform at least one of the following: select content or determine the projection surface. A projection system as described in any of (1) through (17) above.

[0261] (19) Application information includes information indicating whether it is possible to display content using multiple projection surfaces, the recommended number of projection surfaces, and whether it is possible to display content unrelated to each projection surface when multiple projection surfaces are used. The projection system described in (18) above.

[0262] (19-1) In the case of an application that allows the use of multiple projection surfaces, the number of projection surfaces to be used by the target user shall be assigned to the target user based on the recommended number of projection surfaces, so as not to exceed the maximum number of displayable projection surfaces of the projection device, from among the projection surfaces determined for the target user. The projection system described in (19) above.

[0263] (19-2) If the same projection plane is assigned to multiple users, the duplicate assignment of the projection plane will be maintained if there is common display information for each user. The projection system described in (19) above.

[0264] (20) A user recognition step that recognizes a user present in space, A projection environment recognition step involves recognizing a projection surface capable of projecting an image into the aforementioned space, A control step in which the projection device controls the projection device to project an image onto the projection surface recognized in the projection environment recognition step for the user recognized in the user recognition step, A projection control method having the following characteristics. [Explanation of Symbols]

[0265] 100... Projection system, 101... User recognition unit 102...Projection environment recognition unit, 103...Output control unit, 110...Projection device 200...Projection system, 201...Input unit, 202...User recognition unit 203...Projection environment recognition unit, 203-1...Projection surface detection unit 204…Output control unit, 210…Projection device 701... Phase-modulated SLM 800...Projection system, 801...Input unit, 802...User information detection unit 802-1...User recognition unit, 802-2...User definition unit 803...Projection environment recognition unit, 803-1...Projection surface detection unit 804…Output control unit, 810…Projection device 1200...Projection system, 1201...Input unit 1202...User information detection unit, 1202-1...User recognition unit 1202-2…User-defined section, 1203…Projection environment recognition section 1203-1…Projection surface detection unit, 1204…Output control unit 1204-1…Content Selection Unit, 1204-2…Projection Surface Determination Unit 1210…Projection device 1600... Projection system, 1601... Input unit 1602...User information detection unit, 1602-1...User recognition unit 1602-2...User-defined section, 1603...Projection environment recognition section 1603-1…Projection surface detection unit, 1603-2…Projection surface definition unit 1604...Output control unit, 1604-1...Content selection unit 1604-2…Projection surface determination unit, 1605…Projection parameter correction unit 1610…Projection device 2900...Projection system, 2901...Input unit 2902...User information detection unit, 2902-1...User recognition unit 2902-2…User-defined section, 2903…Projection environment recognition section 2903-1…Projection surface detection unit, 2903-2…Projection surface definition unit 2904...Output control unit, 2904-1...Content selection unit 2904-2…Projection surface determination unit, 2905…Projection parameter correction unit 2906...Application information storage unit, 2910...Projection device 3101...Sensor, 4000...CAVE system

Claims

1. A user recognition unit that recognizes users present in space, A projection environment recognition unit recognizes a projection surface capable of projecting an image into the aforementioned space, An application information storage unit stores application information that defines the allocation of projection surfaces for each application, including information indicating whether it is possible to display content using multiple projection surfaces, the recommended number of projection surfaces, and whether it is possible to display content unrelated to each projection surface when multiple projection surfaces are used. A control unit controls the projection device to project an image onto a projection surface determined by the projection environment recognition unit according to the application information, for a user recognized by the user recognition unit, and A projection system equipped with the following features.

2. The projection device is capable of simultaneously projecting images onto multiple surfaces. The control unit controls the projection device to simultaneously project images onto two or more projection surfaces recognized by the projection environment recognition unit. The projection system according to claim 1.

3. At least one of the user recognition unit and the projection environment recognition unit performs recognition based on sensor information detected by sensors installed in the space. The projection system according to claim 1.

4. The projection environment recognition unit detects, from within the projection range of the projection device, an area that satisfies one or more thresholds, such as an area greater than or equal to a predetermined threshold, a curvature less than or equal to a predetermined threshold, or a gradient greater than or equal to a predetermined threshold, as a projection surface. The projection system according to claim 1.

5. The projection device is capable of simultaneously projecting images onto multiple different surfaces in the vertical, horizontal, and depth directions. The projection environment recognition unit recognizes multiple projection surfaces that are different in the vertical, horizontal, and depth directions. The projection system according to claim 1.

6. The projection device is a projection device equipped with a phase-modulated spatial light modulator. The projection system according to claim 5.

7. The user recognition unit defines the user's characteristics and state. The projection system according to claim 1.

8. The system further includes a content selection unit that selects content to display to the user based on defined user information. The projection system according to claim 7.

9. It further includes a projection surface determination unit that determines the projection surface on which the selected content will be projected. The projection system according to claim 8.

10. The projection environment recognition unit further detects information about the recognized projection surface. The projection system according to claim 1.

11. The projection environment recognition unit defines the attributes, shape, area, and characteristics (reflectance, brightness, chromaticity) of the projection surface. The projection system according to claim 10.

12. The system further includes a projection parameter correction unit that corrects the projection parameters for the projection surface determined by the projection surface determination unit. The projection system according to claim 9.

13. The projection parameter correction unit limits at least one of the following based on the design values ​​of the projection device: the distance between different projection surfaces in the depth direction, the number of projection surfaces, or the projection size. The projection system according to claim 12.

14. The projection parameter correction unit determines the priority of the multiple projection surfaces determined by the projection surface determination unit. The projection system according to claim 12.

15. The projection parameter correction unit performs correction on at least one of the following parameters of the projected image on the projection surface: brightness, chromaticity, and size. The projection system according to claim 12.

16. The aforementioned sensor includes a microphone. The user recognition unit recognizes the user's state based on voice data, conversation, or background noise input from the microphone. The projection system according to claim 3.

17. The content selection unit selects content to be displayed to the user based on user information recognized from metadata held by the user's device. The projection system according to claim 8 or 9.

18. Furthermore, content selection is performed based on application information. The projection system according to claim 1.

19. A user recognition step of recognizing a user present in space, A projection environment recognition step involves recognizing a projection surface capable of projecting an image into the aforementioned space, A control step in which the projection device controls the projection device to project an image onto a projection surface determined according to application information that defines the allocation of projection surfaces for each application, including information indicating whether it is possible to display content using multiple projection surfaces from among the projection surfaces recognized in the projection environment recognition step, the recommended number of projection surfaces, and whether it is possible to display content unrelated to each projection surface when multiple projection surfaces are used, for the user recognized in the user recognition step, A projection control method having the following characteristics.

Images