Display method, display system and program

Abstract

To experience a service provided by a projection image with a sense of presence being with a character.SOLUTION: A display method includes: acquiring a first image including an image of a user; acquiring the direction of the user's face on the basis of the first image; and projecting a first projection image including a character facing a first direction corresponding to the direction of the user's face onto a projection surface.SELECTED DRAWING: Figure 3

Description

Technical Field

[0001] The present invention relates to a display method, a display system, and a program.

Background Art

[0002] Patent Document 1 discloses an image display device that displays a production image together with a character based on a transmission signal for transmitting a television. After receiving the transmission signal for transmitting the television, for example, this image display device displays a production image including a character's impression string for the content of the television program on the condition that a predetermined time has elapsed.

Prior Art Documents

Patent Documents

[0003]

Patent Document 1

Summary of the Invention

Problems to be Solved by the Invention

[0004] According to the image display device of Patent Document 1, although a user can communicate with a character about the content of a television program, the user cannot obtain a sense of presence of watching the television program together with the character.

Means for Solving the Problems

[0005] A display method according to one aspect of the present invention includes obtaining a first image including a user's image, obtaining a direction of the user's face based on the first image, and projecting a first projection image including a character facing a first direction corresponding to the direction of the user's face onto a projection surface.

[0006] A display system according to one aspect of the present invention includes a camera that images a user, obtains a first image including an image of the user imaged by the camera, obtains the orientation of the user's face based on the first image, and projects a first projection image including a character facing a first direction corresponding to the orientation of the user's face onto a projection surface, and a projector that executes the projection.

[0007] A program according to one aspect of the present invention causes a computer to obtain a first image including an image of a user, obtain the orientation of the user's face based on the first image, and project a first projection image including a character facing a first direction corresponding to the orientation of the user's face onto a projection surface.

Brief Description of the Drawings

[0008]

Figure 1

Figure 2

Figure 3

Figure 4

Figure 5

Figure 6

Figure 7

Figure 8

Figure 9

Figure 10

Figure 11

Figure 12

Figure 13

Figure 14

Figure 15

Figure 16

Mode for Carrying Out the Invention

[0009] Hereinafter, an embodiment of the present disclosure will be described with reference to the drawings. In each of the following drawings, for the sake of easy viewing of each component, the scale of the dimensions may be shown differently depending on the component.

[0010] FIG. 1 is a diagram schematically showing the configuration of the display system 1 in the present embodiment. As shown in FIG. 1, the display system 1 includes a projector 2 and a camera 3. The camera 3 is connected to the projector 2 via a communication cable (not shown).

[0011] The projector 2 generates image light L based on image data and projects the image light L onto the projection screen SC. When the image light L is projected onto the projection screen SC, an image based on the image data is displayed on the projection screen SC. In the following description, the image displayed on the projection screen SC may be referred to as a "projected image". Also, in the following description, projecting the image light L may be rephrased as projecting a projected image. The projection screen SC is an example of a projection surface.

[0012] For example, when the projector 2 is arranged at a position farther from the projection screen SC than the user 100 who views the projected image, due to the image light L being projected onto the projection screen SC from behind the user 100, there is a possibility that the shadow of the user 100 overlaps the projected image. In the present embodiment, in order to avoid the shadow of the user 100 overlapping the projected image, the projector 2 is arranged at a position close to the projection screen SC. In this case, the projector 2 is preferably a short-focus projector.

[0013] The camera 3 is arranged such that at least the user 100 and the projector 2 are within the angle of view of the camera 3. In FIG. 1, the camera 3 is arranged above the projection screen SC. The camera 3 images a region within the angle of view of the camera 3 in accordance with an imaging command signal transmitted from the projector 2 to the camera 3. The camera 3 transmits imaging image data indicating the captured image obtained by the imaging operation to the projector 2. The captured image is an image of a region within the angle of view of the camera 3. Therefore, the captured image includes at least the images of the user 100 and the projector 2. The captured image is an example of a first image.

[0014] As will be described in detail later, based on the captured image obtained from the camera 3, the projector 2 acquires the orientation of the face of the user 100, and projects onto the projection screen SC a projection image including a character facing a first direction corresponding to the orientation of the face of the user 100. In the present embodiment, the character is a 3D (three-dimensions) character. The character may be an avatar of the user 100, or may be a 3D model of a real or fictional person, animal, robot, or the like.

[0015] FIG. 2 is a block diagram schematically showing the configuration of the projector 2. As shown in FIG. 2, the projector 2 includes an optical device 10, an input device 20, a communication device 30, a speaker 40, a storage device 50, and a processor 60.

[0016] The optical device 10 generates image light L under the control of the processor 60, and projects the generated image light L onto the screen SC. The optical device 10 includes a first image generation panel 11, a second image generation panel 12, a third image generation panel 13, a dichroic prism 14, and a projection optical system 15.

[0017] The first image generation panel 11 generates red image light LR representing a red image and emits it to the dichroic prism 14. The first image generation panel 11 has a plurality of pixels arranged in a matrix, and each of the plurality of pixels emits red light. The red image light LR is emitted from the first image generation panel 11 by controlling the emission light amount of the red light for each pixel by the processor 60.

[0018] The second image generation panel 12 generates green image light LG representing a green image and emits it to the dichroic prism 14. The second image generation panel 12 has a plurality of pixels arranged in a matrix, and each of the plurality of pixels emits green light. The green image light LG is emitted from the second image generation panel 12 by controlling the emission light amount of the green light for each pixel by the processor 60.

[0019] The third image generation panel 13 generates blue image light LB representing a blue image and emits it to the dichroic prism 14. The third image generation panel 13 has a plurality of pixels arranged in a matrix, and each of the plurality of pixels emits blue light. By controlling the emission light amount of the blue light for each pixel by the processor 60, the blue image light LB is emitted from the third image generation panel 13.

[0020] For example, each of the image generation panels 11, 12, and 13 is a self-emitting electro-optical device such as an OLED (Organic Light Emitting Diode) panel or a μLED (Micro Light Emitting Diode) panel. Note that each of the image generation panels 11, 12, and 13 may be a non-self-emitting electro-optical device such as a liquid crystal panel or a DMD (Digital Micromirror Device). When each of the image generation panels 11, 12, and 13 is a non-self-emitting electro-optical device, light from a light source (not shown) such as an LED is separated into red light, green light, and blue light, respectively. The red light is incident on the first image generation panel 11. The green light is incident on the second image generation panel 12. The blue light is incident on the third image generation panel 13. Also, a single image generation panel may be used to emit light of each color in a time-division manner.

[0021] The dichroic prism 14 generates image light L representing a color image by combining the red image light LR, the green image light LG, and the blue image light LB and emits it to the projection optical system 15. The projection optical system 15 is composed of a plurality of optical elements such as lenses, and enlarges and projects the image light L emitted from the dichroic prism 14 toward the screen SC. Although not shown in the figure, the projection optical system 15 is provided with a mechanism capable of adjusting optical parameters such as the lens shift amount, the lens focus amount, and the lens zoom amount. By controlling these mechanisms by the processor 60, the optical parameters of the projection optical system 15 are adjusted.

[0022] The input device 20 is a device that receives the input operations of the user 100 for the projector 2. As an example, the input device 20 includes an operation unit 21 and a light receiving unit 22. The operation unit 21 is composed of a plurality of operation keys provided on the projector 2. For example, the operation keys include a power key, a menu call key, a direction key, a determination key, a volume adjustment key, and the like. The operation keys may be hardware keys or software keys displayed on a touch panel provided on the projector 2. The operation unit 21 outputs, as an operation signal, an electrical signal generated when each operation key is operated by the user 100 to the processor 60.

[0023] The light receiving unit 22 includes a photoelectric conversion circuit that receives infrared light transmitted from a remote controller (not shown) of the projector 2 and converts it into an electrical signal. The light receiving unit 22 outputs, as a remote operation signal, the electrical signal obtained by the photoelectric conversion of the infrared light to the processor 60. A plurality of operation keys are provided on the remote controller in the same manner as the operation unit 21. The remote controller converts an electrical signal generated when each operation key provided on the remote controller is operated by the user 100 into infrared light and transmits it to the projector 2. That is, the remote operation signal output from the light receiving unit 22 is substantially the same as the electrical signal generated when each operation key of the remote controller is operated by the user 100. When the remote controller transmits a radio wave signal in accordance with a short-range wireless communication standard such as Bluetooth (registered trademark), a receiving device that receives the radio wave signal may be provided instead of the light receiving unit 22.

[0024] The communication device 30 is a communication interface that relays communication between the processor 60 and an external device. The processor 60 communicates with the camera 3 via the communication device 30. For example, the processor 60 transmits an imaging command signal to the camera 3 via the communication device 30. Further, the processor 60 receives imaging image data from the camera 3 via the communication device 30. The speaker 40 outputs sound according to the sound signal output from the processor 60.

[0025] The storage device 50 includes a non-volatile memory that stores programs and various setting data necessary for causing the processor 60 to execute various processes, and a volatile memory that is used as a temporary storage destination for data when the processor 60 executes various processes. For example, the non-volatile memory is an EEPROM (Electrically Erasable Programmable Read-Only Memory) or a flash memory, etc. The volatile memory is, for example, a RAM (Random Access Memory), etc.

[0026] The processor 60 is an arithmetic processing unit that controls the overall operation of the projector 2 according to a program stored in advance in the storage device 50. For example, the processor 60 is composed of one or more CPUs (Central Processing Unit). A part or all of the functions of the processor 60 may be composed of circuits such as a DSP (Digital Signal Processor), an ASIC (Application Specific Integrated Circuit), a PLD (Programmable Logic Device), and an FPGA (Field Programmable Gate Array). The processor 60 executes various processes in parallel or sequentially. The processor 60 is an example of a computer.

[0027] Hereinafter, the processes executed by the processor 60 according to a program stored in advance in the storage device 50 will be described. FIG. 3 is a flowchart showing the image display process executed by the processor 60. The processor 60 executes the image display process shown in FIG. 3 by reading and executing a program from the storage device 50. The processor 60 executes the image display process at a predetermined time interval.

[0028] As shown in FIG. 3, when the processor 60 starts image display processing, first, the processor 60 acquires a captured image from the camera 3 (step S1). Specifically, in step S1, the processor 60 transmits an imaging command signal to the camera 3 via the communication device 30. When the camera 3 receives the imaging command signal, the camera 3 captures an area within the angle of view of the camera 3. The camera 3 transmits imaging image data indicating the captured image obtained by the imaging operation to the processor 60. The processor 60 receives the imaging image data from the camera 3 via the communication device 30. Through the above operations, the processor 60 acquires a captured image from the camera 3.

[0029] Subsequently, the processor 60 determines whether the image of the projector 2 is included in the captured image (step S2). As already described, when the projector 2 is arranged at a position close to the projection screen SC, the captured image includes the image of the projector 2. By executing step S2, the processor 60 determines whether the projector 2 is arranged at a position close to the projection screen SC.

[0030] For example, in step S2, the processor 60 determines whether the image of the projector 2 is included in the captured image by using a known image identification technique such as pattern matching or a learning model obtained by machine learning such as deep learning.

[0031] If the image of the projector 2 is not included in the captured image (step S2: No), the processor 60 skips the remaining steps S3 to S11 and ends the image display processing. That is, when the projector 2 is not arranged at a position close to the projection screen SC, the processor 60 ends the image display processing.

[0032] On the other hand, when the image of the projector 2 is included in the captured image (step S2: Yes), the processor 60 proceeds to the next step S3. That is, when the projector 2 is placed at a position close to the projection screen SC, the processor 60 proceeds to step S3.

[0033] When the processor 60 proceeds to step S3, it determines whether the image of the user 100 is included in the captured image (step S3). As already described, when there is a user 100 who views the projection image displayed on the projection screen SC, the captured image includes the image of the user 100. By executing step S3, the processor 60 determines whether the user 100 exists.

[0034] For example, in step S3, the processor 60 determines whether the image of the user 100 is included in the captured image by using a known image recognition technique such as pattern matching or a learning model obtained by machine learning such as deep learning.

[0035] When the image of the user 100 is not included in the captured image (step S3: No), the processor 60 skips the remaining steps S4 to S11 and ends the image display process. That is, when the user 100 does not exist, the processor 60 ends the image display process.

[0036] On the other hand, when the image of the user 100 is included in the captured image (step S3: Yes), the processor 60 proceeds to the next step S4. That is, when the user 100 exists, the processor 60 proceeds to step S4.

[0037] When the processor 60 proceeds to step S4, it obtains information regarding the position of the projector 2 by calculating the position of the projector 2 based on the captured image (step S4). In the following description, the information regarding the position of the projector 2 is referred to as the position information of the projector 2. The processor 60 stores the position information of the projector 2 in the storage device 50.

[0038] Note that if the processor 60 confirms that the position information of the projector 2 is stored in the storage device 50 before starting the process of step S4, it may skip step S4 and proceed to step S5.

[0039] Subsequently, the processor 60 obtains information regarding the orientation, position, and size of the face of the user 100 by calculating the orientation, position, and size of the face of the user 100 based on the captured image (step S5). In order to obtain the information regarding the orientation of the face of the user 100, the processor 60 executes the mesh analysis process described below.

[0040] FIG. 4 is a flowchart showing the mesh analysis process executed by the processor 60. As shown in FIG. 4, when starting the mesh analysis process, the processor 60 first assigns feature points to the face of the user 100 based on the captured image (step S21). Subsequently, the processor 60 constructs a mesh of the face of the user 100 based on the feature points assigned to the face of the user 100 (step S22).

[0041] Finally, the processor 60 obtains information regarding the orientation of the face of the user 100 by calculating the orientation of the face of the user 100 based on the constructed mesh (step S23). For example, in step S23, the processor 60 calculates the direction in which the feature points are concentrated on the mesh as the orientation of the face of the user 100.

[0042] The processor 60 acquires information regarding the orientation of the face of the user 100 by executing the mesh analysis process as described above. As described above, since the technique of measuring the face orientation based on the mesh of a person's face is a known technique, the description of the specific processes from steps S21 to S23 is omitted. In the following description, the orientation of the face of the user 100 may be referred to as the user direction.

[0043] Also, in the following description, the information regarding the orientation of the face of the user 100 is referred to as the orientation information of the user 100, the information regarding the position of the user 100 is referred to as the position information of the user 100, and the information regarding the size of the user 100 is referred to as the size information of the user 100. The processor 60 stores the orientation information, position information, and size information of the user 100 in the storage device 50.

[0044] Next, returning to FIG. 3, the description will be continued. After the processor 60 executes the process of step S5 including the above-described mesh analysis process, based on the position information of the projector 2, the processor 60 calculates the position and size of the projection area projected on the projection screen SC, thereby acquiring information regarding the position and size of the projection image (step S6).

[0045] In the following description, the information regarding the position and size of the projection image is referred to as the position information and size information of the projection image. The processor 60 stores the position information and size information of the projection image in the storage device 50. Note that if the processor 60 confirms that the position information and size information of the projection image are stored in the storage device 50 before starting the process of step S6, the processor 60 may skip step S6 and shift to step S7.

[0046] Subsequently, the processor 60 determines whether the face of the user 100 is facing the projection image based on the orientation information and position information of the user 100 and the position information and size information of the projection image (step S7).

[0047] For example, in step S7, the processor 60 calculates a straight line extending from the position of the user 100 in the user direction as the line of sight of the user 100. Then, when the line of sight of the user 100 passes inside the projected image, the processor 60 determines that the face of the user 100 is not facing the projected image. On the other hand, when the line of sight of the user 100 passes outside the projected image, the processor 60 determines that the face of the user 100 is not facing the projected image.

[0048] When the face of the user 100 is facing the projected image (step S7: Yes), the processor 60 determines, based on the 3D image data, the first direction corresponding to the user direction as the direction in which the character faces (step S8). The 3D image data includes position information (coordinate information), color information, brightness information, etc. of each point constituting the background object and the character arranged in a virtual space defined in a three-dimensional world coordinate system. Hereinafter, the process of step S8 will be described with reference to FIG. 5.

[0049] FIG. 5 is an explanatory diagram regarding a specific method for determining the first direction corresponding to the user direction as the direction of the character. In FIG. 5, the solid arrow indicated by the reference D1 indicates the line of sight of the user 100. First, the processor 60 calculates a straight line extending from the position PA1 of the user 100 in the user direction as the line of sight D1 of the user 100.

[0050] Subsequently, the processor 60 calculates the coordinates of the point PB1 through which the line of sight D1 of the user 100 passes among the points included in the projected image projected onto the projection screen SC. The coordinates of the point PB1 included in the projected image are the coordinates in a two-dimensional screen coordinate system. For example, the origin of the screen coordinate system coincides with the point located at the upper left corner of the projected image.

[0051] Subsequently, the processor 60 calculates the coordinates of the point PC1 on the background object 110 that is located at the tip of the user 100's line of sight D1 among the background objects arranged in the virtual space. For example, the processor 60 calculates the coordinates of the point PC1 on the background object 110 by converting the coordinates of the point PB1 in the screen coordinate system into the coordinates of the world coordinate system. In the following description, the point PC1 on the background object 110 is referred to as the fixation point.

[0052] Finally, the processor 60 determines the direction in which the character faces based on the position (coordinates) of the character and the coordinates of the fixation point PC1 in the virtual space. For example, as shown in FIG. 5, when the character 120 is arranged at the position PC2 in the virtual space, the processor 60 determines the direction D2 from the position PC2 of the character 120 to the fixation point PC1 as the direction in which the character 120 faces.

[0053] For example, when the character 130 is arranged at the position PC3 in the virtual space, the processor 60 determines the direction D3 from the position PC3 of the character 130 to the fixation point PC1 as the direction in which the character 130 faces. For example, when the character 140 is arranged at the position PC4 in the virtual space, the processor 60 determines the direction D4 from the position PC4 of the character 140 to the fixation point PC1 as the direction in which the character 140 faces.

[0054] The direction D2 in which the character 120 faces, the direction D3 in which the character 130 faces, and the direction D4 in which the character 140 faces are each the first direction corresponding to the line of sight D1 of the user 100, that is, the user direction. As can be understood from the above description, the direction in which the character faces, that is, the first direction corresponding to the user direction, changes according to the position of the character arranged in the virtual space.

[0055] Next, returning to FIG. 3, the description will continue. After the processor 60 executes the process of step S8 as described above, it proceeds to step S10. On the other hand, when the face of the user 100 is not facing the projected image (step S7: No), the processor 60 determines the direction in which the character faces as a predetermined second direction (step S9).

[0056] For example, the predetermined second direction may be the most recent first direction among the first directions determined in the past by the process of step S8. Also, for example, the predetermined second direction may be a direction determined as a default value for the user 100. Note that the direction determined as the default value is also a direction defined in the world coordinate system. After the processor 60 executes the process of step S9, it proceeds to step S10.

[0057] When the processor 60 proceeds to step S10 after the process of step S8 or step S9, it determines the size of the character based on the position information and size information of the user 100 (step S10). The process of this step S10 is performed on the avatar of the user 100 among the characters arranged in the virtual space. Therefore, when the avatar of the user 100 does not exist among the characters arranged in the virtual space, the processor 60 may skip step S10 and proceed to step S11.

[0058] Finally, the processor 60 controls the optical device 10 to project a first projected image including a character facing the direction determined in step S8 or a second projected image including a character facing the direction determined in step S9 onto the projection screen SC (step S11).

[0059] For example, in step S11, the processor 60 rewrites the position information of the points constituting each character so that each character arranged in the virtual space faces the direction determined in step S8 or step S9. Further, when there is an avatar of the user 100 among the characters arranged in the virtual space, the processor 60 rewrites the position information of the points constituting the avatar so that the size of the avatar of the user 100 becomes the size determined in step S10.

[0060] The processor 60 converts the 3D image data in which the position information of the points constituting the character is rewritten as described above into 2D image data for generating a projection image, and based on this 2D image data, controls the emission light amount of each pixel included in each of the image generation panels 11, 12, and 13. As a result, a first projection image including a character facing the direction determined in step S8 or a second projection image including a character facing the direction determined in step S9 is projected onto the projection screen SC. Further, when there is an avatar of the user 100 among the characters included in the projection image, the size of the avatar of the user 100 becomes the size determined in step S10.

[0061] The above is the description of the image display process executed by the processor 60. FIG. 6 is a diagram showing a first example of a projection image displayed on the projection screen SC by executing the above image display process. The projection image 200 in the first example is an image that provides the user 100 with an experience of watching a movie together with the characters.

[0062] The projection image 200 includes a screen 210 on which a movie is shown as a background object. Further, the projection image 200 includes two characters 220 and 230 arranged as movie viewers. One of the characters 220 and 230 may be an avatar of the user 100.

[0063] For example, when the point PC5 on the screen 210 is the fixation point located at the tip of the line of sight of the user 100, the direction D5 from the position of the character 220 to the fixation point PC5 is determined as the direction in which the character 220 faces. Also, in this case, the direction D6 from the position of the character 230 to the fixation point PC5 is determined as the direction in which the character 230 faces.

[0064] As a result, as shown in FIG. 6, each of the characters 220 and 230 included in the projected image 200 faces the fixation point PC5 located at the tip of the line of sight of the user 100. Therefore, the user 100 can obtain a sense of presence as if watching a movie together with the characters 220 and 230 arranged as the audience.

[0065] In the first example of the projected image, the projected image 200 that provides the user 100 with the experience of watching a movie together with the characters has been given. However, in addition to this, the projected image may be, for example, an image that provides the user 100 with the experience of watching a TV program, a play, a painting, or music together with the characters.

[0066] FIG. 7 is a diagram showing a second example of the projected image displayed on the projection screen SC by executing the above-described image display process. The projected image 300 in the second example is an image that provides the user 100 with the experience of playing tennis together with the characters.

[0067] The projected image 300 includes the tennis court 310 as a background object. Also, the projected image 300 includes the avatar 320 of the user 100, the partner 330 of the user 100, and the opponent players 340 and 350 as characters. For example, the avatar 320 of the user 100 is arranged on the tennis court 310 as a backcourt player, and the partner 330 of the user 100 is arranged on the tennis court 310 as a forecourt player.

[0068] For example, when the point PC6 on the opponent player 340 is the fixation point located at the tip of the line of sight of the user 100, the direction D7 from the position of the avatar 320 to the fixation point PC6 is determined as the direction in which the avatar 320 faces. Also, in this case, the direction D8 from the position of the partner 330 to the fixation point PC6 is determined as the direction in which the partner 330 faces.

[0069] As a result, as shown in FIG. 7, the avatar 320 and the partner 330 included in the projected image 300 face the fixation point PC6 located at the tip of the line of sight of the user 100, so that the user 100 can obtain a sense of presence of playing tennis together with the partner 330.

[0070] In addition, in the second example of the projected image, an example in which the avatar 320 of the user 100 is displayed in the projected image 300 has been given. However, alternatively, the projected image may not display the avatar 320 of the user 100, for example.

[0071] Also, in the second example of the projected image, the projected image 300 that provides the user 100 with an experience of playing tennis together with a character has been given. However, alternatively, the projected image may be an image that provides the user 100 with an experience of playing badminton, table tennis, or beach volleyball together with a character, for example.

[0072] (Effect of this Embodiment) By the processor 60 executing the above-described image display process, the display method of this embodiment described below is realized.

[0073] That is, the display method of this embodiment includes acquiring a captured image including an image of the user 100 from the camera 3 (first step), acquiring the orientation of the face of the user 100 based on the captured image (second step), and projecting a first projected image including a character facing a first direction corresponding to the orientation of the face of the user 100 onto the projection screen SC (third step).

[0074] The first step is realized by the processor 60 executing step S1 of the image display process. The second step is realized by the processor 60 executing step S5 of the image display process. The third step is realized by the processor 60 executing steps S8, S10, and S11 of the image display process.

[0075] According to the above display method, since the first projected image including the character facing the first direction corresponding to the orientation of the face of the user 100 is projected onto the projection screen SC, the user 100 can obtain a sense of presence that he / she is experiencing the service provided by the first projected image together with the character.

[0076] The display method of the present embodiment further includes determining whether the face of the user 100 faces the projected image based on the orientation of the face of the user 100, and when it is determined that the face of the user faces the projected image, projecting the first projected image onto the projection screen SC. Such a display method is realized by the processor 60 executing steps S7, S8, S10, and S11 of the image display process.

[0077] According to the above display method, since the first projected image is projected onto the projection screen SC when the face of the user 100 faces the projected image, when the line of sight of the user 100 calculated based on the orientation of the face of the user 100 is not located on the projected image, it is possible to avoid the character facing an unnatural direction corresponding to the orientation of the face of the user 100.

[0078] The display method of the present embodiment further includes determining whether the face of the user 100 faces the projected image based on the orientation of the face of the user 100, and when it is determined that the face of the user 100 does not face the projected image, projecting a second projected image including a character facing a predetermined second direction onto the projection screen SC. Such a display method is realized by the processor 60 executing steps S7, S9, S10, and S11 of the image display process.

[0079] According to the above display method, when the face of user 100 is not facing the projected image, a second projected image including a character facing a predetermined second direction is projected onto the projection screen SC. Therefore, even when user 100 is not looking at the projected image, the direction in which the character faces can be determined.

[0080] In the above display method, the predetermined second direction may be the newest first direction among the first directions determined in the past. According to this display method, even when the state changes from the state where the face of user 100 is facing the projected image to the state where the face of user 100 is not facing the projected image, the direction in which the character faces can be maintained in the first direction determined when the face of user 100 was facing the projected image. Thereby, continuity can be given to the direction in which the character faces regardless of the orientation of the face of user 100.

[0081] Also, in the above display method, the predetermined second direction may be a direction determined as a default value by user 100. According to this display method, when the face of user 100 is not facing the projected image, the direction in which the character faces can be determined by user 100 in an arbitrary direction in advance.

[0082] In the display method of the present embodiment, obtaining the orientation of the face of user 100 based on the captured image includes assigning feature points to the face of user 100 based on the captured image, constructing a mesh of the face of user 100 based on the feature points assigned to the face of user 100, and obtaining information regarding the orientation of the face of user 100 based on the mesh of the face of user 100. The above display method is realized by the processor 60 executing steps S21, S22, and S23 of the mesh analysis process included in the image display process.

[0083] According to the above display method, the orientation of the face of user 100 can be obtained from the captured image through simple arithmetic processing.

[0084] The display system 1 of this embodiment includes a camera 3 that captures user 100, and a projector 2. The projector 2 acquires a captured image including the image of user 100 captured by the camera 3, obtains the orientation of the face of user 100 based on the captured image, and projects a first projection image including a character facing a first direction corresponding to the orientation of the face of user 100 onto the projection screen SC.

[0085] According to the above display system 1, since the first projection image including the character facing the first direction corresponding to the orientation of the face of user 100 is projected onto the projection screen SC, user 100 can obtain a sense of presence that he / she is experiencing the service provided by the first projection image together with the character.

[0086] The program of this embodiment causes a computer (processor 60) to acquire a captured image including the image of user 100, obtain the orientation of the face of user 100 based on the captured image, and project a first projection image including a character facing a first direction corresponding to the orientation of the face of user 100 onto the projection screen SC.

[0087] According to the above program, since the first projection image including the character facing the first direction corresponding to the orientation of the face of user 100 is projected onto the projection screen SC, user 100 can obtain a sense of presence that he / she is experiencing the service provided by the first projection image together with the character.

[0088] 〔Modification Example〕 Although the embodiments of the present disclosure have been described above, the technical scope of the present disclosure is not limited to the above embodiments, and various modifications can be made without departing from the spirit of the present disclosure. For example, the following modification examples are given as modification examples of the present disclosure.

[0089] (1) In the above embodiment, the form of projecting the first projection image onto the projection screen SC when the face of the user 100 faces the projection image is exemplified, but the present disclosure is not limited thereto. For example, the display method may include determining whether the face of the user 100 faces the first area included in the projection image based on the orientation of the face of the user 100, and when it is determined that the face of the user 100 faces the first area included in the projection image, projecting the first projection image onto the projection screen SC.

[0090] Specifically, for example, when the face of the user 100 faces the screen 210 included in the projection image 200 shown in FIG. 6, it may be determined that the face of the user 100 faces the first area included in the projection image, and the first projection image may be projected onto the projection screen SC. Thereby, when the face of the user 100 faces a direction deviating from the screen 210, it is possible to avoid the characters 220 and 230 facing unnatural directions corresponding to the orientation of the face of the user 100.

[0091] (2) In the above embodiment, the form of projecting the second projection image including the character facing the predetermined second direction onto the projection screen SC when the face of the user 100 does not face the projection image is exemplified, but the present disclosure is not limited thereto. For example, the display method may include determining whether the face of the user 100 faces the second area included in the projection image based on the orientation of the face of the user 100, and when it is determined that the face of the user 100 faces the second area included in the projection image, projecting the second projection image including the character facing the predetermined second direction onto the projection screen SC.

[0092] Specifically, for example, when the face of user 100 is facing an area other than screen 210 included in the projected image 200 shown in FIG. 6, it is determined that the face of user 100 is facing a second area included in the projected image, and a second projected image including a character facing a predetermined second direction may be projected onto projection screen SC. Thereby, when the face of user 100 is not facing screen 210, the second projected image 200 including characters 220 and 230 facing a predetermined second direction is projected onto projection screen SC, so that even when user 100 is not looking at screen 210, the directions in which characters 220 and 230 are facing can be determined.

[0093] (3) FIG. 8 is an image diagram showing a case where the size of character 400 included in the projected image exceeds the size of projection screen SC. As shown in FIG. 8, for example, when the size of character 400 included in the projected image exceeds the size of projection screen SC, processor 60 may reduce the size of character 400 to a size that fits within projection screen SC. In this case, processor 60 may reduce the size of character 400 to 0.9 times the size of projection screen SC. FIG. 9 is an image diagram showing a case where the size of character 400 included in the projected image is reduced to 0.9 times the size of projection screen SC.

[0094] (4) FIG. 10 is an image diagram showing a case where user 100 is at a position where the projected image does not fit within their field of view. As shown in FIG. 10, for example, when user 100 is at a position where the projected image does not fit within their field of view, processor 60 may cause only the upper body of character 400 to be shown in the projected image. FIG. 11 is an image diagram showing a state where only the upper body of character 400 is shown in projected image 500.

[0095] (5) Figure 12 is an image diagram showing the case where the user 100 is sitting. As shown in Figure 12, for example, when the user 100 is sitting, the processor 60 may make the posture of the character 410 included in the projection image the same as the posture of the user 100, that is, a sitting posture. Figure 13 is an image diagram showing the case where the character 410 in the sitting posture is included in the projection image 500.

[0096] (6) In the above embodiment, the form of imaging the user 100 with one camera 3 is exemplified, but the present disclosure is not limited to this, and the user 100 may be imaged with a plurality of cameras. Figure 14 is an image diagram showing the case where, in addition to the camera 3, a camera 4 for imaging the user 100 from behind the user 100 is installed.

[0097] As shown in Figure 14, when, in addition to the camera 3, a camera 4 for imaging the user 100 from behind the user 100 is installed, the processor 60 may project, onto the projection screen SC, a projection image including the image of the back of the user 100 included in the captured image obtained from the camera 4 together with the character 420. Figure 15 is an image diagram showing the case where the image 430 of the back of the user 100 and the character 420 are included in the projection image 500.

[0098] (7) Figure 16 is an image diagram showing the case where the projection screen SC is curved. As shown in Figure 16, for example, a projection image may be projected from the projector 2 onto the curved projection screen SC. Thereby, the user 100 can feel the character located on the edge side of the projection image closer. For example, in Figure 16, the user 100 can see the profile of the character 120 located on the left side as viewed from the user 100.

[0099] 〔Summary of the Present Disclosure〕 Hereinafter, a summary of the present disclosure is appended.

[0100] (Appendix 1) Obtaining a first image including a user's image, obtaining the orientation of the user's face based on the first image, and projecting a first projected image including a character facing a first direction corresponding to the orientation of the user's face onto a projection surface. A display method including the above steps.

[0101] According to the display method described in Appendix 1, since a first projected image including a character facing a first direction corresponding to the orientation of the user's face is projected onto the projection surface, the user can obtain a sense of presence that he or she is experiencing the service provided by the first projected image together with the character.

[0102] (Appendix 2) Further including: determining whether the user's face is facing the projected image based on the orientation of the user's face; and when it is determined that the user's face is facing the projected image, projecting the first projected image onto the projection surface. The display method described in Appendix 1.

[0103] According to the display method described in Appendix 2, since the first projected image is projected onto the projection surface when the user's face is facing the projected image, it is possible to avoid the character facing an unnatural direction corresponding to the orientation of the user's face when the user's face is facing away from the projected image.

[0104] (Appendix 3) Further including: determining whether the user's face is facing the projected image based on the orientation of the user's face; and when it is determined that the user's face is not facing the projected image, projecting a second projected image including a character facing a predetermined second direction onto the projection surface. The display method described in Appendix 1.

[0105] According to the display method described in Appendix 3, since a second projected image including a character facing a predetermined second direction is projected onto the projection surface when the user's face is not facing the projected image, it is possible to determine the direction in which the character faces even when the user is not looking at the projected image.

[0106] (Appendix 4) The method of display according to Appendix 3, wherein the second direction is the most recent first direction among the first directions determined in the past.

[0107] According to the method of display described in Appendix 4, even when the user's face changes from a state facing the projected image to a state not facing the projected image, the direction in which the character faces can be maintained in the first direction determined in the state where the user's face faces the projected image. Thereby, regardless of the orientation of the user's face, continuity can be given to the direction in which the character faces.

[0108] (Appendix 5) The method of display according to Appendix 3, wherein the second direction is the direction determined as the default value by the user.

[0109] According to the method of display described in Appendix 5, when the user's face is not facing the projected image, the user can determine in advance in an arbitrary direction the direction in which the character faces.

[0110] (Appendix 6) Further including: determining whether the user's face faces a first area included in the projected image based on the orientation of the user's face; and when it is determined that the user's face faces the first area included in the projected image, projecting the first projected image onto the projection surface. The method of display according to Appendix 1.

[0111] According to the method of display described in Appendix 6, when the user's face faces a direction outside the first area, it is possible to avoid the character facing an unnatural direction corresponding to the orientation of the user's face.

[0112] (Appendix 7) Further including: determining whether the user's face faces a second area included in the projected image based on the orientation of the user's face; and when it is determined that the user's face faces the second area included in the projected image, projecting onto the projection surface a second projected image including a character facing a predetermined second direction. The method of display according to Appendix 1.

[0113] According to the display method described in Supplementary Note 7, when the user's face is facing the second area, a second projection image including a character facing a predetermined second direction is projected onto the projection surface. Therefore, even when the user is not looking at the first area, the direction in which the character faces can be determined.

[0114] (Supplementary Note 8) Obtaining the orientation of the user's face based on the first image includes assigning feature points to the user's face based on the first image, constructing a mesh of the user's face based on the feature points assigned to the user's face, and obtaining information regarding the orientation of the user's face based on the mesh, and is the display method according to any one of Supplementary Notes 1 to 7.

[0115] According to the display method described in Supplementary Note 8, the orientation of the user's face can be obtained from the first image through simple arithmetic processing.

[0116] (Supplementary Note 9) A display system including a camera that images a user, obtaining a first image including an image of the user imaged by the camera, obtaining the orientation of the user's face based on the first image, and projecting, onto a projection surface, a first projection image including a character facing a first direction corresponding to the orientation of the user's face, and a projector that executes the above.

[0117] According to the display system described in Supplementary Note 9, since a first projection image including a character facing a first direction corresponding to the orientation of the user's face is projected onto the projection surface, the user can obtain a sense of presence as if experiencing the service provided by the first projection image together with the character.

[0118] (Appendix 10) Obtaining a first image including a user's image, obtaining the orientation of the user's face based on the first image, and projecting a first projected image including a character facing a first direction corresponding to the orientation of the user's face onto a projection surface, a program for causing a computer to execute.

[0119] According to the program described in Appendix 10, since the first projected image including the character facing the first direction corresponding to the orientation of the user's face is projected onto the projection surface, the user can obtain a sense of presence that he / she is experiencing the service provided by the first projected image together with the character.

Explanation of Signs

[0120] 1… Display system, 2… Projector, 3… Camera, 10… Optical device, 11… First image generation panel, 12… Second image generation panel, 13… Third image generation panel, 14… Dichroic prism, 15… Projection optical system, 20… Input device, 21… Operation unit, 22… Light receiving unit, 30… Communication device, 40… Speaker, 50… Storage device, 60… Processor, 100… User, L… Image light, SC… Projection screen

Claims

1. Obtaining a first image including an image of a user; Obtaining the orientation of the face of the user based on the first image; Projecting a first projected image including a character facing a first direction corresponding to the orientation of the face of the user onto a projection surface; A display method comprising the above.

2. Judging whether the face of the user faces the projected image based on the orientation of the face of the user; When it is judged that the face of the user faces the projected image, projecting the first projected image onto the projection surface; The display method according to claim 1, further comprising the above.

3. Judging whether the face of the user faces the projected image based on the orientation of the face of the user; When it is judged that the face of the user does not face the projected image, projecting a second projected image including a character facing a predetermined second direction onto the projection surface; The display method according to claim 1, further comprising the above.

4. The display method according to claim 3, wherein the second direction is the newest first direction among the first directions determined in the past.

5. The display method according to claim 3, wherein the second direction is a direction determined as a default value by the user.

6. Judging whether the face of the user faces a first area included in the projected image based on the orientation of the face of the user; When it is judged that the face of the user faces the first area included in the projected image, projecting the first projected image onto the projection surface; The display method according to claim 1, further comprising the above.

7. Judging whether the face of the user faces a second area included in the projected image based on the orientation of the face of the user; When it is judged that the face of the user faces the second area included in the projected image, projecting a second projected image including a character facing a predetermined second direction onto the projection surface; The display method according to claim 1, further comprising the above.

8. Obtaining the orientation of the face of the user based on the first image includes: Assigning feature points to the face of the user based on the first image; Constructing a mesh of the face of the user based on the feature points assigned to the face of the user; Obtaining information regarding the orientation of the face of the user based on the mesh; The display method according to any one of claims 1 to 7, including

9. A camera for imaging a user, Obtaining a first image including an image of the user captured by the camera, Obtaining the orientation of the user's face based on the first image, Projecting a first projection image including a character facing a first direction corresponding to the orientation of the user's face onto a projection surface, A projector that executes A display system comprising

10. Obtaining a first image including an image of a user, Obtaining the orientation of the user's face based on the first image, Projecting a first projection image including a character facing a first direction corresponding to the orientation of the user's face onto a projection surface, A program that causes a computer to execute

Images