Entertainment Systems and Programs

Abstract

To provide an interactive entertainment system using a display device and a terminal device.SOLUTION: The entertainment system includes: a recognition processing unit which performs recognition processing of a display object for recognition; a content processing unit which performs processing of causing a user to visually recognize a content resulting from adding a second content displayed on a display unit of a terminal device to a first content displayed on a display device; and an accepting unit. The content processing unit performs processing of displaying a display object for input on the display unit of the terminal device, and the accepting unit accepts response input of a user to the display object for input. The content processing unit modifies the first content on the display device or the second content on the terminal device in accordance with the response input of the user to the display object for input.SELECTED DRAWING: Figure 7

Description

[Technical field]

[0001] The present invention relates to an entertainment system, a program, and the like. [Background technology]

[0002] Conventionally, entertainment systems using AR (Augmented Reality), MR (Mixed Reality), etc. have been known. A conventional technology for such entertainment systems is the system disclosed in Patent Document 1. In the system of Patent Document 1, when a content such as a performance image and a marker displayed on a display device is captured by a terminal device, a predetermined display object is synthesized with the content on the display device and displayed on the display unit of the terminal device. [Prior art documents] [Patent documents]

[0003] [Patent Document 1] JP 2011-258120 A Summary of the Invention [Problem to be solved by the invention]

[0004] However, the system of Patent Document 1 only uses an AR technique to overlay and display objects that appear to pop out onto the content of the display device, and does not propose a user input to the AR displayed objects. For this reason, it was not possible to realize an interactive entertainment system using a display device and a terminal device.

[0005] According to the present disclosure, it is possible to provide an interactive entertainment system and a program using a display device and a terminal device. [Means for solving the problem]

[0006] One aspect of the present disclosure relates to an entertainment system using a display device and a user's terminal device, the entertainment system including: a recognition processing unit that performs a recognition process of a recognition display object displayed by the display device; a content processing unit that performs a process of making the user visually recognize a content obtained by adding a second content displayed on a display unit of the terminal device to a first content displayed on the display device based on a result of the recognition process; and a reception unit that receives an input from the user, the content processing unit performs a process of displaying an input display object on the display unit of the terminal device for the user to input a response, the reception unit receives the response input of the user to the input display object, and the content processing unit changes the first content on the display device or the second content on the terminal device according to the response input of the user to the input display object. Another aspect of the present disclosure relates to a program that causes a computer to function as each of the above units, or a computer-readable information storage medium that stores the program.

[0007] According to one aspect of the present disclosure, an input display object for a user to input a response is displayed, and a first content on the display device changes or a second content on the terminal device changes in response to the user's response input to the input display object. In this way, while the first content or the second content is changed in response to the user's response input to the input display object, the user can visually recognize the content consisting of the first content and the second content. Therefore, the first content or the second content changes interactively in response to the user's response input to the input display object, and an interactive entertainment system or the like can be realized using the display device and the user's terminal device.

[0008] In another aspect of the present disclosure, the content processing unit may change at least one of the display mode of the input display object, the position of the input display object, and the input conditions of the response input to the input display object, depending on the display status on the display device.

[0009] In this way, it is possible to display the input display object in an appropriate display mode according to the display status on the display device, to arrange the input display object in an appropriate arrangement position according to the display status on the display device, or to input a response under appropriate input conditions according to the display status on the display device.

[0010] In one aspect of the present disclosure, the display device may include a positional relationship detection unit that detects a positional relationship between the display device and the terminal device, and the content processing unit may change at least one of the display mode of the input display object, the position of the input display object, and the input conditions for the response input to the input display object according to the positional relationship.

[0011] In this way, it is possible to display the input display object in an appropriate display mode according to the positional relationship between the display device and the terminal device, to arrange the input display object in an appropriate arrangement position according to the positional relationship, or to input a response under appropriate input conditions according to the positional relationship.

[0012] In another aspect of the present disclosure, when a display object of the display device is selected, the content processing unit may change at least one of the display mode of the input display object, the position of the input display object, and the input conditions of the response input to the input display object, depending on information of the selected display object.

[0013] In this way, when a display object on the display device is selected, the input display object can be displayed in an appropriate display mode according to the information of the selected display object, the input display object can be positioned in an appropriate position according to the information of the display object, or a response can be input under appropriate input conditions according to the information of the display object.

[0014] In another aspect of the present disclosure, the content processing unit may perform a process of changing the recognition display object displayed on the display device.

[0015] In this way, various processes can be executed based on the results of the recognition process of the recognition display object, so that a variety of contents in which the second content is added to the first content can be provided to the user.

[0016] In another aspect of the present disclosure, the content processing unit may change the recognition display object displayed on the display device according to a display status on the display device.

[0017] In this way, it becomes possible to display the recognition display object in a display mode or the like that corresponds to the display situation on the display device, and it becomes possible to realize appropriate recognition processing or the like of the recognition display object.

[0018] In another aspect of the present disclosure, the content processing unit may provide feedback corresponding to an input by the user on the terminal device to the first content on the display device or the second content on the terminal device.

[0019] In this way, the user's input on the terminal device can be reflected in the first content or the second content, and the user can visually recognize the content in which the second content is added to the first content.

[0020] In another aspect of the present disclosure, the content processing unit may provide the feedback corresponding to the user's input to the input display object displayed on the display unit of the terminal device.

[0021] In this way, it becomes possible to change the display mode, etc. of the input display object in response to user input on the terminal device, making it possible to display input display objects in a variety of display modes in response to user input.

[0022] In another aspect of the present disclosure, the content processing unit may provide the feedback indicating that the response input of the user to the input display object has been accepted to the first content or the second content.

[0023] In this way, the fact that the user has input a response to the input display object can be reflected in the first content or the second content, making it possible for the user to recognize it.

[0024] In one aspect of the present disclosure, the system further includes an aggregation processing unit that performs an aggregation process of the response inputs in the multiple terminal devices, and the content processing unit may change the first content or the second content based on a result of the aggregation process of the response inputs.

[0025] In this way, it becomes possible to display the first content or the second content according to the result of the tallying process of the response inputs from the multiple terminal devices.

[0026] In another aspect of the present disclosure, the content processing unit may perform processing to display, as the second content, an object to be placed in a virtual space set corresponding to the position of the terminal device on the display unit of the terminal device.

[0027] In this way, a virtual space can be set to correspond to the position of the terminal device, and objects placed in the virtual space can be added to the first content as objects of the second content, making them visible to the user.

[0028] In another aspect of the present disclosure, the content processing unit may change the display object on the terminal device in conjunction with the first content on the display device.

[0029] In this way, the second content on the terminal device changes in conjunction with the first content on the display device, and it becomes possible for the user to visually recognize the content consisting of the first and second contents that are linked together. [Brief description of the drawings]

[0030] [Figure 1] 2 shows an example of a system configuration of the entertainment system according to the present embodiment. [Diagram 2]FIG. 13 is a detailed explanatory diagram of a method for adding a second content of a terminal device to a first content of a display device. [Diagram 3] FIG. 13 is an explanatory diagram of a method for adding a second content of a terminal device to a first content of a display device. [Figure 4] FIG. 1 is a diagram showing an example of a see-through HMD that is a terminal device. [Diagram 5] 1 is a block diagram showing an example of the configuration of an entertainment system according to an embodiment of the present invention. [Figure 6] 6(A) and 6(B) are diagrams showing an example of the hardware configuration of the entertainment system of this embodiment. [Figure 7] 4 is a flowchart illustrating a processing example of the present embodiment. [Figure 8] 10 is a flowchart for explaining a process of changing the display mode, etc. of an input object and a recognition object according to the display status, etc. of the display device. [Figure 9] 9(A) and 9(B) are explanatory diagrams regarding detection of the positional relationship between a display device and a terminal device and selection of an object to be displayed on the display device. [Figure 10] 10(A), 10(B), and 10(C) are explanatory diagrams of a method for changing the display mode, arrangement position, and input conditions of an input display object. [Figure 11] 10 is a flowchart for explaining a process of providing feedback in response to a user's input on the terminal device to a first content on the display device or a second content on the terminal device. [Figure 12] 12(A) and 12(B) are explanatory diagrams of a method for providing feedback to a second content in a terminal device in response to a user's input. [Figure 13] FIG. 13 is an explanatory diagram of a technique for providing feedback in response to a user's input to a first content on a display device. [Figure 14] 11 is a flowchart for explaining a process of providing feedback to an input display object in response to a response input by a user. [Figure 15] FIG. 13 is an explanatory diagram of a method for changing the first content etc. on the display device based on the result of a compilation process of response inputs from a user. [Figure 16] 10 is a flowchart illustrating a process for displaying an object to be placed in a virtual space set in the terminal device, based on a result of the recognition process. [Figure 17] 11 is an explanatory diagram of a method for displaying an object to be placed in a virtual space set in a terminal device based on the result of a recognition process. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0031] The present embodiment will be described below. Note that the present embodiment described below does not unduly limit the contents of the claims. Furthermore, not all of the configurations described in the present embodiment are necessarily essential configurations.

[0032] 1. Entertainment System FIG. 1 shows an example of a system configuration of the entertainment of this embodiment. The entertainment of this embodiment is realized using a display device 10 and a user's terminal device 20. The display device 10 is, for example, a main display device. Specifically, the display device 10 is a monitor (display) of an information processing device such as a PC (personal computer), a television monitor, or a large monitor installed at a venue for a live show or club. The terminal device 20 is a portable terminal device carried by a user, or a wearable terminal device worn by a user on a part of the body such as the head.

[0033] As shown in FIG. 1, the display device 10 displays the first content CT1. The first content CT1 is, for example, visual content such as a broadcast video, a game video, an environmental video, a sightseeing video, or a simulation video that can be visually recognized by the user. The first content CT1 may include auditory content such as music, game sounds, environmental sounds, or voices. In FIG. 1, a video (broadcast video, game video) showing a character CH performing a DJ performance is displayed on the display device 10 as the first content CT1. The display device 10 also displays a display object MK for recognition. The display object MK for recognition is an object to be recognized in the recognition process, and can be realized by, for example, a marker for recognition. The display object MK for recognition may be realized by an image of the first content CT1 displayed on the display device 10. For example, the display object MK for recognition may be realized by using a background image of the first content CT1, an image of a display object such as the character CH, or a display pattern of the first content CT1.

[0034] The second content CT2 is displayed on the display unit 30 of the terminal device 20. For example, in FIG. 1, a display object OB (object) for effects (for dramatic effects) is displayed as the second content CT2. That is, a display object OB for effects whose position and direction change in conjunction with the video playback of the first content CT1 is displayed. For example, the display object OB, which is the second content CT2, is displayed so as to be added (superimposed) to the first content CT1. That is, in this embodiment, for example, a recognition process of a display object MK for recognition is performed. Then, based on the result of the recognition process, a process is performed to allow the user to visually recognize the content in which the second content displayed on the display unit 30 of the terminal device 20 is added to the first content CT1 displayed on the display device 10. For example, when the display object MK for recognition is recognized by the recognition process, an image of the second content such as the display object OB is synthesized with the image of the first content CT1 by AR (Augmented Reality) or MR (Mixed Reality) processing, and is displayed on the display unit 30 of the terminal device 20. Specifically, the terminal device 20 has, for example, an imaging unit 162 (camera) in Fig. 5 described later, and the imaging unit 162 provided on the rear side of the terminal device 20 (behind the display unit 30) captures an image of the first content CT1 of the display device 10. Then, an image in which the image of the captured first content CT1 is combined with the image of the display object OB is displayed on the display unit 30 of the terminal device 20. In this way, a content image in which an image of the second content CT2 such as an effect (performance effect) is added to the image of the first content CT1 is displayed, so that it is possible to apply a performance effect by AR or MR to the first content CT1, and an entertainment system with a high performance effect can be realized.

[0035] In FIG. 1, when the recognition display object MK is recognized, the second content CT2 such as a display object OB is added to the first content CT1, but this embodiment is not limited to this. For example, a motion sensor such as a gyro sensor or an acceleration sensor is provided in the terminal device 20. Then, the motion sensor or the like is used to detect position information and attitude information of the terminal device 20, and the second content CT2 may be displayed even when the imaging unit 162 is not facing the display device 10 using the detected position information and attitude information. For example, in FIG. 1, even when the imaging unit 162 of the terminal device 20 faces the ceiling or the like instead of facing the display device 10, the display object OB for effect may be displayed in the direction of the ceiling or the like.

[0036] 2 and 3 are explanatory diagrams of a method of adding a second content CT2 of a terminal device 20 to a first content CT1 of a display device 10. For example, in FIG. 2 and FIG. 3, a composite image (mixed image, superimposed image, added image) of the first content CT1 and the second content CT2 is displayed on the display unit 30 of the terminal device 20. Specifically, an AR or MR image is displayed in which a display object OB representing an effect or the like moves so as to spread in all directions from the first content CT1. This makes it possible to give the user a feeling as if an effect is occurring from the first content CT1, and improves the presentation effect of the content image.

[0037] In this embodiment, in addition to such effect display object OB, a process is performed in which input display objects DBA, DBB, and DBC for the user to input a response are displayed on the display unit 30 of the terminal device 20. For example, a process is performed in which the input display objects DBA, DBB, and DBC are added to the first content CT1 as, for example, the second content CT2. Then, a user's response input to the input display objects DBA, DBB, and DBC is accepted, and the first content CT1 on the display device 10 or the second content CT2 on the terminal device 20 is changed according to the response input.

[0038] For example, in FIG. 2 and FIG. 3, the display object DBS prompts the user to decide the next song to be played by the DJ character CH. Then, the input display objects DBA and DBB show candidates for the next song to be played, and the user performs a response input (operation) to select the input display object corresponding to the song he / she desires from among the input display objects DBA and DBB. For example, if the display unit 30 of the terminal device 20 is a touch panel type display, the user performs a response input to select the input display objects DBA and DBB by performing a touch operation on the input display objects DBA and DBB. Then, the first content CT1 on the display device 10 changes in response to this response input. For example, if the user selects the input display object DBA, the first song corresponding to the input display object DBA is selected as the next song to be played, and the first content CT1 in which the character CH performs a DJ performance corresponding to the first song is displayed on the display device 10. Also, when the user selects the input display object DBB, a second song corresponding to the input display object DBB is selected as the next song to be played, and the first content CT1 in which the character CH performs a DJ performance corresponding to the second song is displayed on the display device 10. As shown in FIG. 15 described later, a counting process of the response inputs of the users at the multiple terminal devices 20-1 to 20-n may be performed, and the next song to be played may be determined based on the result of the counting process, and the first content CT1 may be changed. That is, the next song to be played is determined based on the votes of the users of the multiple terminal devices 20-1 to 20-n, and the first content CT1 in which the character CH performs a DJ performance corresponding to the determined song to be played is displayed on the display device 10.

[0039] 2 and 3, an input display object DBC for changing the second content CT2 is displayed. When the user makes a response input (operation) to select the input display object DBC by touch operation or the like, the second content CT2 changes. For example, the second content CT2 changes to a content whose content corresponds to the user's response input. Specifically, in FIG. 2 and FIG. 3, when the user makes a response input to select the input display object DBC, the display mode of the effect display object OB, which is the second content CT2, such as the type, shape, color, or animation pattern, changes. For example, the display mode of the effect display object OB may be changed to a first display mode, a second display mode, a third display mode, and so on, each time the user makes a response input to select the input display object DBC. This allows the user to change the content of the second content CT2 added to the first content CT1, thereby improving the presentation effect.

[0040] The input display objects DBA, DBB, DBC may be three-dimensional objects having three-dimensional coordinates, or may be two-dimensional objects. For example, an effect display object OB may also be used as an input display object. For example, when selecting a first piece of music as the next piece to be played, the user makes a response input to select one or more display objects OB set to a first color (e.g., red). On the other hand, when selecting a second piece of music as the next piece to be played, the user makes a response input to select one or more display objects OB set to a second color (e.g., white).

[0041] As described above, in this embodiment, the input display objects DBA and DBB for the user to input a response are displayed, and the first content CT1 displayed on the display device 10 is changed according to the user's response input to the input display objects DBA and DBB. For example, the first content CT1 is changed to a content whose content corresponds to the user's response input. Alternatively, the input display object DBC for the user to input a response is displayed, and the second content CT2 displayed by the terminal device 20 is changed according to the user's response input to the input display object DBC. For example, the second content CT2 is changed to a content whose content corresponds to the user's response input. In this way, for example, the input display object is displayed as an AR or MR display object, and the first content CT1 on the display device 10 or the second content CT2 on the terminal device 20 can be changed according to the user's response input to the input display object. Then, it becomes possible to display to the user a content in which the first content CT1 and the second content CT2 that change according to the user's response input to the input display object are mixed. For example, conventional AR and MR display objects are displayed only to improve the performance effect, and do not improve the interactivity of the user. In contrast, in this embodiment, an input display object for the user to input a response is displayed as the second content CT2 that is added to the first content CT1 and displayed to the user. Then, the first content CT1 and the second content CT2 change interactively according to the user's response input to this input display object, thereby improving the interactivity in the entertainment system. Therefore, it is possible to realize an interactive entertainment system using the display device 10 and the user's terminal device 20.

[0042] For example, the display device 10 of the present embodiment is a large monitor (screen) at a live venue, or a monitor of a PC or a television on which video distribution is performed. When a user looks into the monitor through a camera (imaging unit) of a smartphone or the like, which is the terminal device 20, a three-dimensional virtual space is set extending from the space within the screen to the outside of the screen, with a code (QR code (registered trademark), etc.), picture, or pattern displayed on the monitor screen as a display object for recognition (marker). Then, effects linked to the content within the monitor screen are projected and displayed into this virtual space outside the screen. Also, by sending information linked to the content within the monitor screen, such as excitement information and tempo information of a song, to the application of the terminal device 20 via a network, it becomes possible to display effects linked to the performance within the screen. For example, an effect in which the size of particles changes in synchronization with the rhythm can be displayed. Therefore, by linking information via a network, rather than simply using a display application on the edge side, it becomes possible to perform a performance that matches the content. Also, a button serving as an input display object can be installed on the screen viewed from the smartphone, which is the terminal device 20, and information can be sent to the performance within the screen. For example, a voting button for the song that the DJ character will play next can be installed. And by sending information in both directions via an Internet connection, for example, it will be possible to not only stream content, but also to create interaction between performers and audiences through the production at live venues.

[0043] In the above, the terminal device 20 has been described as a user-portable terminal device as shown in Fig. 1, but the terminal device 20 of this embodiment is not limited to this. For example, the terminal device 20 may be a wearable terminal device that the user wears on a part of the body such as the head. For example, Fig. 4 shows an example of a see-through HMD 200 (head-mounted display device), which is one type of wearable terminal device. Specifically, the HMD 200 in Fig. 4 is an example of an optically see-through HMD.

[0044] The HMD 200 in FIG. 4 has a temple section 240 and a goggle section 242. The temple section 240 has a built-in speaker. The goggle section 242 is provided with a display device 243 and a holographic optical element 244. For example, the display section 30 of the terminal device 20 can be realized by the display device 243, the holographic optical element 244, etc. The display device 243 is provided with a display unit configured with a microdisplay, a mirror, a prism, etc. The display device 243 is provided with a display unit for the left eye and a display unit for the right eye, thereby realizing stereoscopic vision. In addition, by providing a pair of microdisplays in each of the display units for the left eye and the right eye, the position of the virtual image can be changed.

[0045] Display light from the display device 243 is guided to the front of the user's eyes while being refracted inside the light guide plate of the holographic optical element 244. The display light is then refracted toward the eyeball by the holographic optical element 244, and sent to the eyeball. This makes it appear as if there is a virtual image (a reconstructed image of a hologram) in front of the eye. The light guide plate of the holographic optical element 244 is also called a combiner, and the combiner, which is a half mirror, makes it appear as if the real image of the outside world and the virtual image are overlapped, thereby realizing AR and MR.

[0046] The goggle unit 242 is provided with an RGB camera 246, a depth camera 247, and environment recognition cameras 248 and 249. The RGB camera 246 can be used to capture an image of the user in the front direction. The depth camera 247 can be used to obtain depth information in the front direction. The goggle unit 242 is provided with an emitter (not shown) for the depth camera. The environment recognition cameras 248 and 249 can be used to recognize the environment around the user. The goggle unit 242 is also equipped with an inertial measurement unit (IMU), which is a motion sensor composed of an acceleration sensor, a gyro sensor, and the like. The position and direction of the user's head are detected based on an image captured by a camera provided in the goggle unit 242 and measurement information from the inertial measurement unit, thereby realizing head tracking. It is also possible to obtain position information and direction information of the user. In addition, by processing using the RGB camera 246, the depth camera 247, etc., it is possible to obtain position information (relative position information), direction information (relative direction information), or posture information of other users around the user. For example, this information is obtained by processing called Kinect (registered trademark). For example, by obtaining posture information, it becomes possible to detect what action the other users have performed.

[0047] According to the HMD 200 of FIG. 4, the second content CT2 can be displayed to the user by the display unit 30 realized by, for example, the display device 243, the holographic optical element 244, etc. That is, the second content CT2 can be displayed to the user as a virtual image. In addition, the first content CT1 displayed on the display device 10 can be visually recognized by the user as a real image of the outside world by a combiner using the holographic optical element 244. This makes it possible to realize a process of making the user visually recognize the content in which the second content is added to the first content displayed on the display device 10. Note that the wearable terminal device 20 is not limited to the optically transparent HMD 200 shown in FIG. 4, and may be, for example, a video transparent HMD 200. For example, the first content CT1 of the display device 10 is photographed by the imaging unit 162 provided in the HMD 200. Then, by combining an image of the second content CT2 with the photographed image, it is possible to realize a process of making the user visually recognize the content in which the second content is added to the first content.

[0048] Furthermore, when the HMD 200 is used, the user's response input to the input display object may be realized, for example, by the user operating a controller (not shown), or may be realized by detecting that the user's line of sight is directed toward the input display object by eye tracking, etc. Alternatively, the response input to the input display object may be detected by detecting the movement of the user's head, etc., by a motion sensor provided in the HMD 200.

[0049] Fig. 5 is a block diagram showing a configuration example of the entertainment system (content providing system, processing system, game system) of this embodiment. Note that the amusement system of this embodiment is not limited to the configuration of Fig. 5, and various modifications are possible, such as omitting some of the components or adding other components.

[0050] The operation unit 160 allows the user (player) to perform various inputs (operations). The operation unit 160 can be realized by, for example, operation buttons, directional keys, a keyboard, a joystick, a lever, or a touch panel display. For example, in the case of a touch panel display, the display unit 30 also functions as the operation unit 160.

[0051] The imaging unit 162 (camera) captures an image of a subject (an object to be imaged). The imaging unit 162 can be realized by an image sensor such as a CCD or CMOS sensor, an optical system including a focus lens, and the like.

[0052] The storage unit 170 stores various types of information. The storage unit 170 functions as a work area for the processing unit 100, the communication unit 196, and the like. Programs and data required for executing the programs are held in the storage unit 170. The functions of the storage unit 170 can be realized by a semiconductor memory (DRAM, VRAM), a hard disk drive (HDD), a solid state drive (SSD), an optical disk device, or the like. The storage unit 170 stores information on content, such as a first content and a second content. The content information is information on images and sounds constituting the content, and the like. The image information on the content is information on displayed objects constituting the content, such as information on objects that are displayed objects, and the like.

[0053] The information storage medium 180 is a computer-readable medium for storing programs, data, and the like. The information storage medium 180 can be realized by an optical disk (DVD, BD, CD), a HDD, or a semiconductor memory (ROM). The processing unit 100 performs various processes of this embodiment based on the programs (data) stored in the information storage medium 180. That is, the information storage medium 180 stores programs (programs for causing the computer to execute the processes of each unit) for causing a computer (a device equipped with an input device, a processing unit, a storage unit, and an output unit) to function as each unit of this embodiment.

[0054] The display unit 30 outputs the image generated by this embodiment, and the function thereof can be realized by an LCD, an organic EL display, a touch panel display, an HMD (Head Mounted Display), etc. The sound output unit 192 outputs the sound generated by this embodiment, and the function thereof can be realized by a speaker, headphones, etc.

[0055] The communication unit 196 communicates with the outside (other devices) via a wired or wireless network, and the function can be realized by hardware such as a communication ASIC or a communication processor, or by communication firmware.

[0056] The programs (data) for causing a computer to function as each unit of this embodiment may be distributed from an information storage medium held by a server (host device) to the information storage medium 180 (or the storage unit 170) via the network and the communication unit 196. Such use of the information storage medium by the server can also be included within the scope of this embodiment.

[0057] The processing unit 100 (processor) performs reception processing, recognition processing, detection processing, counting processing, content processing, and the like, based on programs and data stored in the storage unit 170, input information from the operation unit 160, and the like.

[0058] Each process of the present embodiment performed by each unit of the processing unit 100 can be realized by a processor (a processor including hardware). For example, each process of the present embodiment can be realized by a processor that operates based on information such as a program, and a memory that stores information such as a program. The functions of each unit of the processor may be realized by individual hardware, or the functions of each unit may be realized by integrated hardware. For example, the processor includes hardware, and the hardware may include at least one of a circuit that processes a digital signal and a circuit that processes an analog signal. For example, the processor may be configured with one or more circuit devices (e.g., ICs, etc.) mounted on a circuit board, or one or more circuit elements (e.g., resistors, capacitors, etc.). The processor may be, for example, a CPU (Central Processing Unit). However, the processor is not limited to a CPU, and various processors such as a GPU (Graphics Processing Unit) or a DSP (Digital Signal Processor) can be used. The processor may also be a hardware circuit using an ASIC. The processor may also include an amplifier circuit, a filter circuit, etc. that process an analog signal. The memory (storage unit 170) may be a semiconductor memory such as an SRAM or a DRAM, or a register. Alternatively, the memory may be a magnetic storage device such as a hard disk drive (HDD), or an optical storage device such as an optical disk drive. For example, the memory stores computer-readable instructions, and the instructions are executed by the processor to realize the processing of each part of the processing unit 100. The instructions here may be a set of instructions constituting a program, or may be instructions that instruct the hardware circuitry of the processor to operate.

[0059] The processing unit 100 includes a receiving unit 102, a recognition processing unit 104, a positional relationship detection unit 106, a counting processing unit 108, and a content processing unit 110. As described above, each process of the present embodiment executed by each of these units can be realized by a processor (or a processor and a memory). Note that various modifications are possible, such as omitting some of these components (each unit) or adding other components.

[0060] The reception unit 102 performs a process of receiving an input from a user. For example, when a user performs an input such as a response input using the operation unit 160, the reception unit 102 receives the input from the user. For example, the reception unit 102 monitors the user's input using the operation unit 160, and when the user performs an input using the operation unit 160, the reception unit 102 acquires the input information.

[0061] The recognition processing unit 104 performs a recognition process on the display object for recognition. For example, as described with reference to FIG. 1 to FIG. 3, the recognition processing unit 104 performs a recognition process on the display object for recognition displayed by the display device 10. The recognition process is an image recognition process that analyzes and recognizes an image of the display object for recognition. The display object for recognition is an object to be recognized by a marker such as an AR marker. The marker may be a QR Code (registered trademark) marker. The display object for recognition is not limited to a marker such as an AR marker, and may be an object (object) recognized by a matching process of feature points or a contour shape, such as a markerless AR. That is, the display object for recognition is not limited to an AR marker such as a square marker or a special shape marker, and may be an object recognized by a process of natural feature point matching or contour shape matching. The display object for recognition can be used as information that triggers the execution of a process of adding a second content to a first content and having the user visually recognize the second content. The display object for recognition may be information that designates an object such as an input display object displayed as the second content.

[0062] The positional relationship detection unit 106 detects the positional relationship between the display device 10 and the terminal device 20. For example, the positional relationship detection unit 106 detects the relative positional relationship of the terminal device 20 with respect to the display device 10. Here, the positional relationship may be the relationship between the position of the display device 10 and the position of the terminal device 20, or the relationship between the direction (direction in which the display device 10 faces) and the direction (direction in which the terminal device 20 faces). For example, the positional relationship may be the relative directional relationship of the terminal device 20 with respect to the display device 10. The positional relationship detection unit 106 detects the positional relationship between the display device 10 and the terminal device 20, for example, by using a display object for recognition. For example, the position, size, or direction of the display object for recognition in a captured image (display screen of the display unit 30) of the display device 10 can be detected to detect the positional relationship between the display device 10 and the terminal device 20. For example, by detecting whether the display object for recognition is displayed at a predetermined planned position, it can be detected whether the terminal device 20 is appropriately positioned directly opposite the display device 10. In addition, the positional relationship regarding the distance between the display device 10 and the terminal device 20 can be detected based on the size of the display object for recognition. In addition, based on the direction of the recognition display object, the directional relationship between the display device 10 and the terminal device 20 can be detected as a positional relationship. In addition, for example, by detecting a change from the initial positional relationship between the display device 10 and the terminal device 20 using a motion sensor or the like of the terminal device 20, the positional relationship between the display device 10 and the terminal device 20 at each timing can be detected.

[0063] The counting processing unit 108 performs various counting processes. For example, the counting processing unit 108 performs counting of response inputs in a plurality of terminal devices. For example, the counting processing unit 108 performs a process of counting what kind of response inputs have been made in a plurality of terminal devices. For example, the counting processing unit 108 performs a process of counting the number of users who have made a first response input and the number of users who have made a second response input in a plurality of terminal devices. The counting process may be a counting process that counts the number of users who have made each response input or the number of terminal devices, or may be a counting process that uses various statistical processes.

[0064] The content processing unit 110 performs various content processing. The content processing is, for example, various processing related to the content, such as display processing of the image of the content, such as the first content and the second content, and output processing of the sound of the content. The display processing of the image of the content is not limited to the processing of generating and displaying the image of the content, but may be a processing of instructing the display of the image of the content, or instructing the content of the image of the content to be displayed. The content processing unit 110 may include, for example, a first display processing unit that performs display processing of the first content on the display device 10, and a second display processing unit that performs display processing of the second content on the display unit 30 of the terminal device 20.

[0065] The entertainment system of the present embodiment using the display device 10 and the user's terminal device 20 includes a reception unit 102, a recognition processing unit 104, and a content processing unit 110, as shown in FIG. 5. The recognition processing unit 104 performs a recognition process of a display object for recognition displayed by the display device 10. For example, it performs an image recognition process of the display object for recognition. The content processing unit 110 performs a content process based on the result of the recognition process. For example, the content processing unit 110 performs a process of making the user visually recognize a content in which a second content displayed on the display unit 30 of the terminal device 20 is added to a first content displayed on the display device 10 based on the result of the recognition process. For example, the content processing unit 110 performs a process of synthesizing (mixing) the second content with the first content. For example, it performs a process for displaying to the user a content image in which an image of the first content and an image of the second content are synthesized. The reception unit 102 also performs a process of receiving a user's input. For example, it performs a process of receiving a user's input using the operation unit 160.

[0066] Then, as described with reference to FIG. 2, FIG. 3, etc., the content processing unit 110 performs a process of displaying an input display object on the display unit 30 of the terminal device 20 for the user to input a response. For example, the content processing unit 110 performs a process of displaying an input display object at a given position on the display screen of the display unit 30 of the terminal device 20. Then, the reception unit 102 performs a process of receiving a response input from the user to the input display object. For example, when an input display object is displayed on the display unit 30 and the user performs a response input such as selecting the input display object, the reception unit 102 receives the response input from the user. Then, the content processing unit 110 changes the first content on the display device 10 or the second content on the terminal device 20 in response to the user's response input to the input display object. For example, the content processing unit 110 performs a process of changing the display contents or display mode, etc. of the first content, or changing the display contents or display mode, etc. of the second content. For example, the content processing unit 110 performs a process of displaying the first content or the second content with the display contents or display mode, etc., according to the response input from the user. For example, the content processing unit 110 performs processing to change the display contents or display mode of the first content or the second content depending on whether the user's response input is the first response input or the second response input. Note that both the first content and the second content may be changed depending on the user's response input to the input display object.

[0067] Furthermore, the content processing unit 110 performs processing to change at least one of the display mode of the input object, the arrangement position of the input object, and the input condition for a response input to the input object, in accordance with the display status of the display device 10. For example, when the display device 10 is in a first display status, the content processing unit 110 displays the input object in a first display mode, arranges the input object at a first arrangement position, or sets the input condition for a response input to the input object to a first input condition. When the display device 10 is in a second display status, the content processing unit 110 displays the input object in a second display mode, arranges the input object at a second arrangement position, or sets the input condition for a response input to the input object to a second input condition.

[0068] The display status of the display device 10 includes the status of the display object of the display device 10 (type, arrangement position, display mode), the status of the background image, the status of the display screen (brightness, color of the screen), the status of the display scene, the status of the recognition display object such as a marker, the display time status, or the display timing status. The display mode of the input display object includes the color, shape, size, brightness, transparency (opacity), texture, or display time of the input display object. The arrangement position of the input display object includes the position or direction of the input display object on the display screen of the display unit 30. The input conditions for the response input to the input display object include the input time, input method, input timing, or conditions (settings) regarding the pressing force or touch range on a touch panel display for the response input.

[0069] The entertainment system also includes a positional relationship detection unit 106 that detects a positional relationship (including a directional relationship) between the display device 10 and the terminal device 20. The content processing unit 110 performs processing to change at least one of the display mode of the input display object, the arrangement position of the input display object, and the input condition of the response input to the input display object, according to the detected positional relationship. For example, when the display device 10 and the terminal device 20 are in a first positional relationship, the content processing unit 110 displays the input display object in a first display mode, arranges the input display object at a first arrangement position, or sets the input condition of the response input to the input display object to a first input condition. When the display device 10 and the terminal device 20 are in a second positional relationship, the content processing unit 110 displays the input display object in a second display mode, arranges the input display object at a second arrangement position, or sets the input condition of the response input to the input display object to a second input condition.

[0070] Furthermore, when an object of the display device 10 is selected, the content processing unit 110 performs a process of changing at least one of the display mode of the input object, the arrangement position of the input object, and the input condition of the response input to the input object, according to the information of the selected object. The information of the object is information such as the type, shape, color, size, brightness, transparency (opacity), texture, or display time of the object. For example, it is assumed that an object (object) such as a character of the display device 10 is selected by a user's input or a user's gaze position. In this case, the content processing unit 110 changes the display mode, arrangement position, or input condition of the response input of the input object, according to the information of the selected object. For example, when the selected object is an object of a first type, a first shape, a first color, a first size, a first brightness, a first transparency, or a first texture, the content processing unit 110 displays the input object in a first display mode, arranges the input object at a first arrangement position, or sets the input condition of the response input to the input object to a first input condition. Furthermore, if the selected display object is of a second type, a second shape, a second color, a second size, a second brightness, a second transparency, or a second texture, the content processing unit 110 displays the input display object in a second display mode, places the input display object at a second placement position, or sets the input conditions for the response input to the input display object to second input conditions.

[0071] The content processing unit 110 also performs a process of changing the recognition display object displayed on the display device 10. For example, instead of displaying only one recognition display object on the display device 10 on which the first content is displayed, the recognition display object is updated and changed as needed. For example, the content processing unit 110 changes the recognition display object according to elapsed time, changes the recognition display object at a given timing such as the timing of an event occurrence or the timing of the progress of the content, or updates and changes the recognition display object when a given update condition is satisfied.

[0072] Furthermore, the content processing unit 110 changes the recognition display object displayed on the display device 10 according to the display status of the display device 10. The display status of the display device 10 is the status of the display object of the display device 10, the status of the background image, the status of the display screen, the status of the display scene, the status of the display time, or the status of the display timing. For example, when the display device 10 is in a first display status, the content processing unit 110 displays a recognition display object of a first type, a first shape, a first color, a first size, a first brightness, a first transparency, or a first texture on the display device 10. Furthermore, when the display device 10 is in a second display status, the content processing unit 110 displays a recognition display object of a second type, a second shape, a second color, a second size, a second brightness, a second transparency, or a second texture on the display device 10.

[0073] The content processing unit 110 also provides feedback to the first content on the display device 10 or the second content on the terminal device 20 in response to a user input on the terminal device 20. The user input on the terminal device 20 may be a user response input corresponding to the input display object, or may be an input other than the response input. The feedback to the first content in response to the user input is feedback to the display object of the first content, feedback to the display contents or display mode of the first content, etc. The feedback to the display object of the first content is feedback that changes the color, shape, size, brightness, transparency, texture, display time, type, etc. of the display object of the first content. The feedback to the display contents or display mode of the first content is feedback that changes the display contents or display mode of the first content. Similarly, the feedback to the second content in response to the user input is feedback to the display object of the second content, feedback to the display content or display mode of the second content, etc. The feedback to the display object of the second content is feedback that changes the color, shape, size, brightness, transparency, texture, display time, type, etc. of the display object of the second content. The feedback on the display content or the display mode of the second content is feedback that changes the display content or the display mode of the second content.

[0074] For example, the content processing unit 110 performs feedback according to the user's input on the input display object displayed on the display unit 30 of the terminal device 20. For example, the feedback is performed to change the color, shape, size, brightness, transparency, texture, display time, or type of the input display object according to the user's input. For example, when the user performs a first input, the content processing unit 110 performs feedback to set the input display object to a first color, a first shape, a first size, a first brightness, a first transparency, a first texture, a first display time, or a first type. When the user performs a second input, the content processing unit 110 performs feedback to set the input display object to a second color, a second shape, a second size, a second brightness, a second transparency, a second texture, a second display time, or a second type.

[0075] The content processing unit 110 also provides feedback to the first content or the second content indicating that the user's response input to the input display object has been accepted. For example, feedback is provided to the first content or the second content to inform the user that the response input has been properly accepted. For example, the first content displayed on the display device 10 notifies the user that the user's response input to the input display object has been accepted, or the user is made to visually recognize this. For example, the voting result based on the response inputs of a plurality of users may be displayed to the user by the first content on the display device 10. Alternatively, the second content displayed on the terminal device 20 notifies the user that the user's response input to the input display object has been accepted, or the user is made to visually recognize this. For example, when the user's response input is accepted, the color, shape, size, brightness, transparency, texture, or display time of the display object such as the input display object in the second content is controlled to notify the user that the user's response input has been accepted.

[0076] The entertainment system also includes a tallying process unit 108 that performs tallying process of the response inputs in the multiple terminal devices. The content processing unit 110 changes the first content or the second content based on the result of the tallying process of the response inputs. For example, the content processing unit 110 performs a process of displaying a first content having a display content or a display mode corresponding to the result of the tallying process of the response inputs on the display device 10, or displaying a second content having a display content or a display mode corresponding to the result of the tallying process of the response inputs on the display unit 30 of the terminal device 20. For example, the content processing unit 110 performs a process of displaying an indication indicating the result of the tallying process as the first content on the display device 10, or as the second content on the terminal device 20.

[0077] The content processing unit 110 also performs a process of displaying, as the second content, a display object arranged in a virtual space set corresponding to the position of the terminal device 20 on the display unit 30 of the terminal device 20. For example, the content processing unit 110 sets the virtual space so as to correspond to the position of the terminal device 20. For example, a virtual camera (viewpoint) in the virtual space is set at the position of the terminal device 20. This makes it possible to generate a virtual space image in which the display device 10 is viewed through a window at the position of the virtual camera. Then, the content processing unit 110 performs a process of displaying, on the display unit 30 of the terminal device 20, a display object arranged in the virtual space thus set. For example, the content processing unit 110 performs a process of generating a virtual space image in which the display object is arranged in the virtual space as a virtual space image seen from a virtual camera set at the position of the terminal device 20. Then, the content processing unit 110 performs a process of displaying, on the display unit 30 of the terminal device 20, an image in which an image of the first content and a virtual space image, which is an image of the second content, are mixed together.

[0078] Furthermore, the content processing unit 110 performs processing to change a display object on the terminal device 20 in conjunction with the first content on the display device 10. For example, when an image or sound of the first content on the display device 10 changes, processing to change the display content or display mode of a display object displayed on the display unit 30 of the terminal device 20 in conjunction with the change is performed. For example, the processing to change a display object on the terminal device 20 in conjunction with the first content can be realized by controlling the display of the display object on the terminal device 20 using parameters (parameters representing tempo, excitement, etc.) used in the presentation processing of the first content.

[0079] 6(A) and 6(B) show examples of the hardware configuration of the entertainment system of this embodiment. In FIG. 6(A), a terminal device 20, a server 50, and an information processing device 40 are communicatively connected via a network 60 realized by the Internet or the like. In FIG. 6(A), the entertainment system of this embodiment can be realized by the processing of the terminal device 20, or by distributed processing of the terminal device 20, the information processing device 40, and the server 50. The information processing device 40 is, for example, a desktop PC or a notebook PC. The server 50 is a server operated by an operator of the entertainment system, and performs, for example, various management processes of the entertainment system and the aforementioned tallying process of the response input. In addition, the display process of the first content on the display device 10 can be realized, for example, by the information processing device 40 connected to the display device 10, and the display process of the second content can be realized by the terminal device 20. For example, in the configuration of FIG. 6(A), when a display object for recognition displayed on the display device 10 is recognized by the recognition process of the terminal device 20, a program corresponding to the display object for recognition is provided to the terminal device 20 by the server 50. The terminal device 20 operates based on the program to generate the second content and to add the second content to the first content and display the second content. Meanwhile, in FIG. 6(B), the game device 70 performs a process of displaying the first content on the display device 10, and the game device 70 is connected to the terminal device 20 by wireless or wired communication. The game device 70 is, for example, a stationary game device for home or commercial use. The terminal device 20 is, for example, a portable game device or a portable communication terminal device such as a smartphone. The process of generating the second content and the process of adding the second content to the first content and displaying the second content can be realized by, for example, the process of the terminal device 20 or by distributed processing of the terminal device 20 and the game device 70.

[0080] 2. Processing of this embodiment Next, various processes in this embodiment will be described in detail.

[0081] 2.1 Processing according to response input to input display object FIG. 7 is a flowchart for explaining a processing example of the entertainment system of this embodiment. First, a recognition process of the display object for recognition displayed by the display device 10 is performed (step S1). For example, as shown in FIG. 1, the display device 10 displays the display object for recognition MK, and the terminal device 20 photographs the display object for recognition MK by the imaging unit 162, and performs a recognition process of the image of the display object for recognition MK shown in the photographed image. Then, based on the result of the recognition process, a process is performed in which the user visually recognizes the content in which the second content displayed by the terminal device 20 is added to the first content displayed by the display device 10 (step S2). For example, as shown in FIG. 2 and FIG. 3, the terminal device 20 performs a process of synthesizing (mixing, superimposing) the image of the second content CT2, such as the display object for effect OB and the display objects for input DBA, DBB, DBC, on the image of the first content CT1 shown in the photographed image of the imaging unit 162. Then, a process is performed in which the image of the content in which the image of the second content CT2 is added to the image of the first content CT1 is displayed on the display unit 30.

[0082] Next, it is determined whether the user has made a response input to the input display object displayed on the display unit 30 of the terminal device 20, and if the user has made a response input, the response input is accepted (steps S3 and S4). For example, the terminal device 20 determines whether the user has made a response input to select the input display objects DBA, DBB, and DBC shown in Figs. 2 and 3, and if the response input is made, the response input is accepted. Then, a process is performed to change the first content on the display device 10 or the second content on the terminal device 20 according to the user's response input to the input display object (step S5). For example, when the user makes a response input to the input display objects DBA and DBB in Figs. 2 and 3, the first content CT1 displayed on the display device 10 changes. Specifically, the user selects either the input display object DBA or DBB to determine the next song to be played by the DJ character CH. For example, when the user selects the input display object DBA, the first content CT1 in which the character CH performs a DJ performance of the song corresponding to the input display object DBA is displayed on the display device 10. Furthermore, when the user selects the input display object DBB, the first content CT1 in which the character CH performs a DJ performance of a song corresponding to the input display object DBB is displayed on the display device 10. Furthermore, when the user makes a response input to the input display object DBC in Figures 2 and 3, the second content CT2 displayed by the terminal device 20 changes. For example, when the user makes a response input to select the input display object DBC, the display mode of the type, shape, color, animation pattern, etc. of the effect display object OB, which is the second content CT2, changes.

[0083] As described above, according to the present embodiment, an input object is displayed as an AR or MR object, and the first content CT1 on the display device 10 changes, or the second content CT2 on the terminal device 20 changes, depending on the user's response input to the input object. In this way, the first content CT1 or the second content CT2 (at least one of CT1 and CT2) changes depending on the user's response input to the input object, and the user can visually recognize a content in which the first content CT1 and the second content CT2 are mixed. Therefore, the first content CT1 or the second content CT2 changes interactively depending on the user's response input to the input object, and an interactive entertainment system can be realized using the display device 10 and the user's terminal device 20.

[0084] 2.2 Processing according to display situation, positional relationship, and displayed object information In this embodiment, the display mode of the input object, the arrangement position of the input object, or the input conditions for a response input to the input object are changed according to the display status on the display device 10, the positional relationship between the display device 10 and the terminal device 20, or information on the object selected on the display device 10. Also, in this embodiment, a process is performed to change the recognition object displayed on the display device 10. For example, the recognition object is changed according to the display status on the display device 10. FIG. 8 is a flowchart illustrating the process of changing the display mode, etc. of the input object and the recognition object according to the display status, etc. of the display device 10.

[0085] First, the entertainment system of this embodiment acquires the display status on the display device 10 (step S11). For example, the status of the display object of the display device 10, the status of the background image, the status of the display screen, the status of the display scene, the status of the display object for recognition, the status of the display time, or the status of the display timing, etc. are acquired. In addition, the positional relationship between the display device 10 and the terminal device 20 is detected (step S12). For example, as shown in FIG. 9(A), the relative positional relationship of the terminal device 20 with respect to the display device 10 is acquired. This positional relationship may include the relative directional relationship of the terminal device 20 with respect to the direction of the display device 10. For example, the positional relationship between the display device 10 and the terminal device 20 can be detected by detecting the position, size, or direction (perspective) of the display object for recognition MK shown on the display unit 30 (captured image) of the terminal device 20 in FIG. 1. Alternatively, the positional relationship may be detected by a motion sensor provided in the terminal device 20, the positional relationship may be detected by using a sensor device such as an imaging sensor provided in the display device 10, or the positional relationship may be detected by using a measuring device provided separately from the terminal device 20 and the display device 10. Also, information on the display object selected on the display device 10 is acquired (step S13). For example, in Fig. 10(B), a character CHE is selected on the display device 10. For example, when a user performs an input operation to select the character CHE, or when the user directs his or her gaze toward the character CH as if gazing at the character CHE, the selection of the character CHE can be detected by detecting the input operation or the direction of the user's gaze. Then, as information on the display object, which is information on the character CHE, information such as the type, shape, color, size, brightness, transparency, texture, or display time of the display object is acquired.

[0086] Next, the display mode, arrangement position, or input condition of the input display object, or the response input is changed according to the display situation, the positional relationship, or the information of the selected display object (step S14). That is, the display mode, arrangement position, or input condition of the input display object, or the response input is changed according to the display situation of the display device 10 acquired in step S11, the positional relationship between the display device 10 and the terminal device 20 detected in step S12, or the information of the display object acquired in step S13. Specifically, the display mode, such as the color, shape, size, brightness, transparency, texture, or display time, of the input display object, is changed. Alternatively, the position or direction of the input display object on the display screen of the display unit 30, as the arrangement position of the input display object, is changed. Alternatively, the input time, input method, input timing, or input conditions such as the pressing force or touch range on the touch panel display for the response input to the input display object, for example, is changed. Also, the recognition display object displayed on the display device 10 is changed according to the display situation on the display device 10 (step S15).

[0087] For example, in Fig. 10(A), input display objects DBY and DBN are displayed to allow the user to select "YES" or "NO." In Fig. 10(A), the display aspects of the input display objects DBY and DBN, such as color, shape, size, brightness, transparency, texture, or display time, are changed according to the display status of the display device 10, the positional relationship between the display device 10 and the terminal device 20, or information on the selected display object.

[0088] For example, the display mode such as the color, shape, size, brightness, or transparency of the input display objects DBY and DBN is changed according to the display conditions such as the conditions of the display objects of the display device 10, the conditions of the background image, or the conditions of the display screen such as the brightness and color of the screen. For example, the color, shape, size, brightness, or transparency of the input display objects DBY and DBN is adjusted according to the color, shape, size, or type of the display objects such as the characters of the display device 10. For example, the color, shape, size, brightness, or transparency of the input display objects DBY and DBN is adjusted for the color, shape, size, or type of the display objects so that the input display objects DBY and DBN are easily recognized by the user. Also, the color, shape, size, brightness, or transparency of the input display objects DBY and DBN is adjusted according to the conditions of the background image on the display device 10 and the conditions of the display screen such as the brightness and color of the display screen so that the input display objects DBY and DBN are easily recognized by the user. Similarly, the color, shape, size, brightness, or transparency of the input display objects DBY and DBN is adjusted according to the positional relationship between the display device 10 and the terminal device 20. For example, when the display device 10 and the terminal device 20 are located at a distance from each other, the input display objects DBY and DBN are made more conspicuous in color and shape, or the input display objects DBY and DBN are made larger, brighter, or more opaque, so that the user can easily recognize the input display objects DBY and DBN. Also, as shown in FIG. 9B, the color, shape, size, brightness, transparency, etc. of the input display objects DBY and DBN are adjusted according to the information of the display object such as the character CHE selected by the user. For example, the color, shape, size, brightness, transparency, etc. of the input display objects DBY and DBN are adjusted according to the color, shape, size, transparency, etc. of the display object selected by the user. For example, the input display objects DBY and DBN are set to a color, shape, size, or transparency corresponding to the color, shape, size, or transparency of the selected display object.

[0089] Also, as shown in FIG. 10B, the positions of the input display objects DBY and DBN are changed according to the display status of the display device 10, the positional relationship between the display device 10 and the terminal device 20, or information on the selected display object. For example, as shown in FIG. 10B, the input display objects DBY and DBN are arranged (displayed) at a position that avoids the display objects of the first content CT1 displayed on the display device 10. In this way, it is possible to prevent a situation in which the input display objects DBY and DBN make it difficult to see the display objects of the first content CT1. Alternatively, the input display objects DBY and DBN are arranged in the area of ​​the background image displayed on the display device 10. In this way, it is possible to effectively use the area of ​​the background image to display the input display objects DBY and DBN. Also, when the display device 10 and the terminal device 20 are brought closer to each other in a positional relationship, the input display objects DBY and DBN are arranged at a position according to the area of ​​the first content CT1 that is expanded on the display unit 30 of the terminal device 20. For example, the input objects DBY and DBN are arranged at positions that avoid the main display area of ​​the first content CT1 (area where the main objects of the first content CT1 are displayed). In this way, the input objects DBY and DBN can be displayed at appropriate positions according to the positional relationship between the display device 10 and the terminal device 20. Also, as shown in FIG. 9(B), when an object such as the character CHE is selected by the user, the input objects DBY and DBN are arranged at positions that avoid the selected object. In this way, it is possible to prevent a situation in which the input objects DBY and DBN make it difficult to see the object selected by the user.

[0090] Also, as shown in FIG. 10C, the input conditions for the response input to the input display objects DBY and DBN are changed according to the display situation of the display device 10, the positional relationship between the display device 10 and the terminal device 20, or the information of the selected display object. For example, the input conditions such as the input time, input method, input timing, or pressing force and touch range for the response input to the input display objects DBY and DBN are changed. For example, assume that the display area of ​​the first content CT1 on the display device 10 is large, or the distance between the display device 10 and the terminal device 20 is close to each other, so that the main display area of ​​the first content CT1 is expanded. In this case, for example, the input display objects DBY and DBN are displayed in a small size, for example, in the edge area of ​​the display screen of the display unit 30 so as to avoid the main display area of ​​the first content CT1. If the input display objects DBY and DBN are displayed in a small size in the edge area of ​​the display screen, it becomes difficult to perform the response input operation to select the input display objects DBY and DBN. In such a case, the input time, which is an input condition for the response input, is shortened so that the input display objects DBY and DBN can be selected in a short input time. Alternatively, the pressure on the touch panel display, which is an input condition for the response input, is reduced so that the input display objects DBY and DBN can be selected by a touch operation with a small pressure. Alternatively, the touch range (detection range of the touch operation) on the touch panel display is widened so that the selection of the input display objects DBY and DBN by the touch operation is made easier. Alternatively, the input method of the input display objects DBY and DBN is changed to an input method that makes the selection easier, or the input timing of the input display objects DBY and DBN is changed to an input timing that makes the selection easier.

[0091] In this embodiment, the recognition display object displayed on the display device 10 is changed according to the display situation on the display device 10. For example, the type, color, shape, size, brightness, etc. of the recognition display object are changed according to the display situation such as the display time of the display device 10, the situation of the display object, the situation of the background image, or the situation of the display screen. For example, the recognition display object is sequentially updated and displayed according to the display time, which is the elapsed time since the first content CT1 was displayed on the display device 10. For example, when the first content CT1 is the first scene, the first type of recognition display object is displayed, and when the first content CT1 is the second scene, the second type of recognition display object is displayed. Also, when the first content CT1 is the third scene, the third type of recognition display object is displayed. This makes it possible to specify the second content CT2 appropriate for the first scene, second scene, and third scene of the first content CT1 by the first, second, and third types of recognition display object, and to add the specified second content CT2 to the first content CT1 and display it. Alternatively, the type of the recognition display object may be switched depending on the type of the first content CT1 displayed on the display device 10. For example, the type of the recognition display object may be switched depending on whether the first content CT1 displayed on the display device 10 is a video, a still image, or a game screen. Alternatively, the color, shape, size, brightness, etc. of the recognition display object may be changed depending on the status of the display object of the display device 10, the status of the background image, or the status of the display screen such as the brightness and color of the screen. For example, even when the display status of the display device 10 changes, the color, shape, size, brightness, etc. of the recognition display object are changed so that the recognition display object is appropriately recognized.

[0092] The process of changing the recognition display object is not limited to the process of changing the recognition display object based on the display situation, and various modifications are possible. For example, the recognition display object may be changed according to the elapsed time. For example, the recognition display object is changed according to the elapsed time since the first content CT1 was displayed. Alternatively, the recognition display object may be changed at a given timing. For example, the recognition display object is changed at the timing when a given event occurs, or at a given timing in the progress of the first content CT1. Alternatively, it is determined whether or not an update condition for the recognition display object is satisfied, and if the update condition is satisfied, the recognition display object is updated and changed. The update condition is set based on various situations such as the content situation and the user situation. In this way, various processes can be executed based on the result of the recognition process of the recognition display object that is updated and changed, so that it is possible to provide the user with various contents in which the second content is added to the first content.

[0093] 2.3 Feedback of user input to primary and secondary content In this embodiment, feedback according to a user's input on the terminal device 20 is provided to a first content CT1 on the display device 10 or a second content CT2 on the terminal device 20. For example, feedback according to the user's input is provided to an input display object displayed on the display unit 30 of the terminal device 20. Alternatively, feedback indicating that a user's response input to an input display object has been accepted is provided to the first content CT1 or the second content CT2. FIG. 11 is a flowchart illustrating a process of providing feedback according to a user's input on the terminal device 20 to the first content CT1 or the second content CT2.

[0094] First, it is determined whether or not the user has made an input on the terminal device 20 (step S21). This user input may be a response input to an input display object, or may be an input other than the response input. Then, if the user has made an input, the input is accepted (step S22). Then, feedback according to the user's input on the terminal device 20 is provided to the first content CT1 on the display device 10 or the second content CT2 on the terminal device 20 (step S23). In this way, the user's input on the terminal device 20 can be reflected in the first content CT1 or the second content CT2, and the content in which the second content CT2 is added to the first content CT1 can be displayed to the user.

[0095] For example, in this embodiment, feedback according to a user input is provided to an input display object displayed on the display unit 30 of the terminal device 20. For example, an input display object that changes according to a user input on the terminal device 20 is displayed on the display unit 30 of the terminal device 20. In this way, it becomes possible to change, for example, the color, shape, size, brightness, transparency, texture, display time, or type of the input display object according to the user input on the terminal device 20, and it becomes possible to display an input display object in a variety of display modes according to the user input.

[0096] For example, in FIG. 12(A), the user can select the shape of the input display object by inputting on the terminal device 20, and feedback according to the user's input is provided to the input display object. For example, the shape of the input display object can be a square, a star, a diamond, or the like, according to the user's desire. This allows the user to customize the input display object by changing the shape, etc. of the input display object. Note that in addition to the shape of the input display object, the user may be able to select the color, size, brightness, texture, etc. by inputting on the terminal device 20.

[0097] Alternatively, the input display object displayed on the display unit 30 of the terminal device 20 may be changed by changing the recognition display object displayed on the display device 10 in response to the user's input on the terminal device 20. For example, in FIG. 12(B), each user of the terminal device 20 can decorate the input display object to create an original input display object. Then, the recognition display object corresponding to the created original input display object is displayed on the display device 10. In this way, users other than the user who created the input display object can use the input display object by performing a recognition process on the recognition display object using the terminal device of the other user. Therefore, for example, a plurality of users who are friends with each other can share and use the originally created input display object.

[0098] In this embodiment, feedback indicating that the user's response input to the input display object has been accepted is provided to the first content CT1. For example, feedback to inform the user that the response input has been properly accepted is provided to the first content CT1. In this way, the fact that the user's response input to the input display object has been made can be reflected in the first content CT1 and made visible to the user. For example, in this embodiment, as shown in Figs. 2 and 3, input display objects DBA and DBB for selecting the next music piece to be played are displayed, and the user makes a response input to select one of the input display objects DBA and DBB. Then, for example, as will be described later in Fig. 15, the voting results for the next music piece to be played based on the response inputs of multiple users are displayed to the user by the first content CT1 of the display device 10. For example, in Fig. 13, a display object DBF indicating that the music piece MA is more prevalent than the music pieces MA and MB, which are candidates for the next music piece to be played, based on the voting results of multiple users is displayed on the display device 10. This allows the user to recognize the voting result based on his / her response input to the input display object, from the display object DBF of the first content CT1 displayed on the display device 10.

[0099] In this embodiment, feedback indicating that the user's response input to the input display object has been accepted is provided to the second content CT2. For example, the second content CT2 displayed on the terminal device 20 notifies the user or allows the user to visually recognize that the user's response input to the input display object has been accepted. In this way, the fact that the user's response input to the input display object has been made can be reflected in the second content CT2 and made visible to the user. For example, FIG. 14 is a flowchart illustrating a process of changing the input display object, which is the second content CT2, in response to the user's response input to the input display object.

[0100] First, it is determined whether or not the user has made a response input to the input display object displayed on the display unit 30 of the terminal device 20, and if the user has made a response input, the response input is accepted (steps S31 and S32). Then, a process is performed to change the shape, color, or text of the input display object, or to erase the display of the input display object (step S33). For example, when the user makes a response input to the input display object, the shape or color of the input display object is changed to a shape or color that allows the user to recognize that a response input has been made. Alternatively, text that allows the user to recognize that a response input has been made is added to the input display object. In this way, it is possible to allow the user to recognize that the user's response input has been made appropriately, and the user interface is improved. Alternatively, a process is performed to erase the display of the input display object after the user has made a response input to the input display object. In this way, when the response input to the input display object is completed and the input display object is no longer necessary, the display of the input display object is erased, thereby preventing the input display object from interfering with the display of other content.

[0101] In this embodiment, as shown in FIG. 15, the response inputs in the terminal devices 20-1 to 20-n are counted. That is, the counting processing unit 108 collects the response inputs to the input display objects made in each of the terminal devices 20-1 to 20-n via the network 60, and counts these response inputs. Then, the content processing unit 110 changes the first content CT1 displayed on the display device 10. For example, as shown in FIG. 13, a display object DBF showing a voting result, which is a counting result of the response inputs, is displayed on the display device 10. Then, the first content CT1 in which the song MA that is determined to be dominant based on the voting result is played as the next performance song is displayed on the display device 10. For example, the content of a DJ character showing the performance of the song MA is displayed on the display device 10 as the first content CT1. In this way, the first content CT1 according to the result of the counting process of the response inputs in the terminal devices 20-1 to 20-n can be displayed on the display device 10. Alternatively, the second content CT2, the display contents or display form of which correspond to the result of the tallying process of the response inputs to the input display objects, may be displayed on the display unit 30 of the terminal device 20. For example, the display object DBF showing the voting result in FIG. 13 may be displayed as the second content CT2 on the display unit 30 of the terminal device 20. Also, the performance effect of the second content CT2 added to the first content CT1 may be changed according to the result of the tallying process. For example, the type of display object for effects or the animation pattern may be changed according to the result of the tallying process. For example, if the song selected by the user is selected by the voting result, a display object for effects with a stronger performance effect is displayed as the second content CT2 added to the first content CT1.

[0102] In this embodiment, a process is performed to display, as the second content CT2, a display object arranged in a virtual space set in correspondence with the position of the terminal device 20 on the display unit 30 of the terminal device 20. In this way, a virtual space is set to correspond to the position of the terminal device 20, and the display object arranged in the virtual space can be added to the first content CT1 and displayed as a display object of the second content CT2. In this case, in this embodiment, the display object on the terminal device 20 is changed in conjunction with the first content CT1 on the display device 10. In this way, the second content CT2 on the terminal device 20 also changes in conjunction with the first content CT1 on the display device 10, and the first content CT1 and the second content CT2 that are linked to each other can be mixed and displayed to the user. FIG. 16 is a flowchart explaining a process of displaying a display object arranged in a virtual space on the terminal device based on the result of the recognition process.

[0103] First, a recognition process is performed on the display device 10 to recognize the display object for recognition (step S41). Then, based on the result of the recognition process, the display object arranged in the virtual space set in correspondence with the position of the display device 10 is displayed on the display unit 30 of the terminal device 20 (step S42). Then, in conjunction with the first content CT1 on the display device 10, the display object on the terminal device 20 is changed (step S43).

[0104] For example, as shown in FIG. 17, a virtual camera VC is set at a position corresponding to the position of the terminal device 20. Then, a virtual space corresponding to this virtual camera VC is set, and a process of generating a virtual space image in which an effect display object OB or the like is arranged in the virtual space as a virtual space image seen from the virtual camera VC in the virtual space is performed. For example, a process of generating a virtual space image is performed by a three-dimensional image drawing process such as a known hidden surface removal process or perspective transformation process. Then, a process of displaying an image in which the image of the first content and the virtual space image, which is an image of the second content, are synthesized on the display unit 30 of the terminal device 20 is performed. This makes it possible to realize a process of making the user visually recognize an image of a content in which the image of the second content in the terminal device 20 is added to the image of the first content in the display device 10. In this case, the setting of the virtual camera VC can be performed using the display object MK for recognition described in FIG. 1 to FIG. 3. For example, the direction of the display object MK for recognition as seen from the terminal device 20 is set to, for example, the line of sight direction that is the reference of the virtual camera VC. For example, even if the direction of the terminal device 20 changes and it is no longer facing the display device 10, the direction in which the terminal device 20 faces is detected using, for example, a motion sensor, and display objects OB for effect purposes are displayed in the line of sight of the virtual camera VC. In this way, it appears as if display objects OB exist not only around the display device 10 but also around the user himself, thereby further improving the presentation effect of AR and MR.

[0105] In this embodiment, the display object OB in the terminal device 20 is changed in conjunction with the first content CT1 in the display device 10. For example, as shown in Figs. 1 to 3, when the first content CT1 is a content in which the character CH performs a DJ performance to the music being played, the position, direction, or display pattern of the display object OB in the terminal device 20 is changed so as to be linked to the tempo (rhythm) of the music being played. For example, the position, direction, or display pattern of the display object OB is changed so as to be synchronized with the tempo of the music being played. Alternatively, the position, direction, or display pattern of the display object OB is changed according to the excitement level of the first content CT1. In this way, the display object OB of the first content CT1 and the second content CT2 added to the first content CT1 are moved or changed in conjunction (synchronization), so that the performance effect can be further improved.

[0106] In this embodiment, as shown in Fig. 17, the display object OB arranged in the virtual space is changed in response to the user's input on the terminal device 20. For example, the display pattern of the display object OB arranged in the virtual space is changed or the type of the display object OB is changed in response to the user's response input to the input display object DBC of Figs. 2 and 3. For example, every time the user performs an input operation on the input display object DBC, the display pattern of the display object OB is changed from a first display pattern to a second display pattern to a third display pattern. In this way, the second content CT2 on the terminal device 20 changes in response to the user's input, and the interactivity of the entertainment system can be further improved.

[0107] Although the present embodiment has been described in detail as above, it will be easily understood by those skilled in the art that many modifications are possible without substantially departing from the novel matters and effects of the present disclosure. Therefore, all such modifications are intended to be included in the scope of the present disclosure. For example, a term described at least once in the specification or drawings together with a different term having a broader meaning or synonymy may be replaced with that different term anywhere in the specification or drawings. In addition, the recognition process of the display object for recognition, the content process, the display process, the positional relationship detection process, the counting process, etc. are not limited to those described in the present embodiment, and methods, processes, and configurations equivalent to these are also included in the scope of the present disclosure. [Explanation of symbols]

[0108] 10: display device, 20: terminal device, 20-1 to 20-n: terminal devices, 30: display unit, 40: information processing device, 50: server, 60: network, 70: game device, 100: processing unit, 102: receiving unit, 104: recognition processing unit, 106: positional relationship detection unit, 108: tallying unit, 110: content processing unit, 160: operation unit, 162: imaging unit, 170...Storage unit, 180...Information storage medium, 192...Sound output unit, 196...Communication unit, 200...HMD, 240...temple part, 242...goggle part, 243...display device, 244...holographic optical element, 246...RGB camera, 247...depth camera, 248...Environmental Recognition Camera, 249...Environmental Recognition Camera, CH, CHE...character, CT1...first content, CT2...second content, DBA, DBB, DBC, DBY, DBN...input display items, DBS, DBF...display items, MA, MB...track, MK...recognition display, OB...display, VC...virtual camera

Claims

1. An entertainment system using a display device and a user terminal device, a recognition processing unit that performs a recognition process on a display object for recognition displayed by the display device; a content processing unit that performs processing for causing the user to visually recognize a content obtained by adding a second content displayed on a display unit of the terminal device to a first content displayed on the display device based on a result of the recognition processing; A reception unit that receives an input from the user; Including, The content processing unit includes: performing a process of displaying an input display object on the display unit of the terminal device, the input display object being used by the user to input a response; The reception unit is receiving the response input by the user to the input display object; The content processing unit includes: changing the first content on the display device or the second content on the terminal device in response to the response input by the user to the input display object; The content processing unit includes: An entertainment system characterized in that, when an object displayed on the display device is selected by the user, at least one of the display mode of the input object, the position of the input object, and the input conditions for the response input to the input object is changed according to information about the selected object.

2. In claim 1, The content processing unit includes: An entertainment system comprising: a display object for inputting a color, a shape, a size, or a transparency of the display object selected by the user;

3. In claim 1 or 2, The content processing unit includes: An entertainment system comprising: a display device for inputting said input object displayed at a position avoiding the display object selected by said user.

4. An entertainment system using a display device and a user terminal device, a recognition processing unit that performs a recognition process on a display object for recognition displayed by the display device; a positional relationship detection unit that detects a positional relationship between the display device and the terminal device; a content processing unit that performs processing for causing the user to visually recognize a content obtained by adding a second content displayed on a display unit of the terminal device to a first content displayed on the display device based on a result of the recognition processing; A reception unit that receives an input from the user; Including, The content processing unit includes: performing a process of displaying an input display object on the display unit of the terminal device, the input display object being used by the user to input a response; The reception unit is receiving the response input by the user to the input display object; The content processing unit includes: changing the first content on the display device or the second content on the terminal device in response to the response input by the user to the input display object; The content processing unit includes: An entertainment system characterized in that at least one of the display mode of the input display object, the placement position of the input display object, and the input conditions of the response input to the input display object is changed depending on the positional relationship.

5. In claim 4, The content processing unit includes: An entertainment system characterized in that when the display device and the terminal device are brought closer to each other in a positional relationship, causing the display area of ​​the first content on the display unit of the terminal device to be expanded, the input display object is displayed at a position corresponding to the expanded display area of ​​the first content.

6. In claim 4 or 5, The content processing unit includes: An entertainment system characterized in that when the display device and the terminal device are brought closer to each other in the positional relationship, causing the display area of ​​the first content on the display unit of the terminal device to expand, the input display object is displayed in a smaller size and the input conditions for the response input to the input display object are set to easier input conditions.

7. In any one of claims 1 to 6, The content processing unit includes: An entertainment system characterized by changing at least one of the display mode of the input display object, the position of the input display object, and the input conditions of the response input to the input display object depending on the display status on the display device.

8. In any one of claims 1 to 7, The content processing unit includes: An entertainment system characterized by performing processing to change the recognition display object displayed on the display device.

9. In claim 8, The content processing unit includes: An entertainment system characterized in that the recognition display object displayed on the display device is changed according to a display state on the display device.

10. In any one of claims 1 to 9, The content processing unit includes: An entertainment system, characterized in that feedback according to an input from the user at the terminal device is provided to the first content at the display device or the second content at the terminal device.

11. In claim 10, The content processing unit includes: An entertainment system, characterized in that the feedback according to the user's input is provided to the input display object displayed on the display unit of the terminal device.

12. In claim 10 or 11, The content processing unit includes:

13. An entertainment system comprising: a display object for inputting input from a user; a display object for inputting input from the user;

13. In any one of claims 1 to 12, a counting processing unit that performs counting processing of the response inputs in the plurality of terminal devices, The content processing unit includes: An entertainment system comprising: a display unit that displays a display content including a display area that displays a display image of the display area and a display region that displays the display image;

14. In any one of claims 1 to 13, The content processing unit includes: An entertainment system characterized by performing processing to display, as the second content, an object to be placed in a virtual space set corresponding to the position of the terminal device on the display unit of the terminal device.

15. In claim 14, The content processing unit includes: An entertainment system, characterized in that a displayed object on said terminal device is changed in conjunction with said first content on said display device.

16. A program for use in an entertainment system using a display device and a user's terminal device, a recognition processing unit that performs a recognition process on a display object for recognition displayed by the display device; a content processing unit that performs processing for causing the user to visually recognize a content obtained by adding a second content displayed on a display unit of the terminal device to a first content displayed on the display device based on a result of the recognition processing; As a reception unit for receiving an input from the user, Make your computer work The content processing unit includes: performing a process of displaying an input display object on the display unit of the terminal device, the input display object being used by the user to input a response; The reception unit is receiving the response input by the user to the input display object; The content processing unit includes: changing the first content on the display device or the second content on the terminal device in response to the response input by the user to the input display object; The content processing unit includes: When an object on the display device is selected by the user, at least one of the display mode of the input object, the position of the input object, and the input conditions for the response input to the input object is changed according to information about the selected object.

17. A program for use in an entertainment system using a display device and a user's terminal device, a recognition processing unit that performs a recognition process on a display object for recognition displayed by the display device; a positional relationship detection unit that detects a positional relationship between the display device and the terminal device; a content processing unit that performs processing for causing the user to visually recognize a content obtained by adding a second content displayed on a display unit of the terminal device to a first content displayed on the display device based on a result of the recognition processing; As a reception unit for receiving an input from the user, Make your computer work The content processing unit includes: performing a process of displaying an input display object on the display unit of the terminal device, the input display object being used by the user to input a response; The reception unit is receiving the response input by the user to the input display object; The content processing unit includes: changing the first content on the display device or the second content on the terminal device in response to the response input by the user to the input display object; The content processing unit includes: a display mode of the input display object, a position of the input display object, and an input condition of the response input to the input display object, the display mode of the input display object, and the position of the input display object are changed according to the positional relationship.

Images