Programs and Systems

Abstract

To improve interest. [Solution] The program according to the present invention causes the computer (10,20) to control the direction (gaze direction) of the virtual camera in accordance with the viewer's movements, and if the direction (gaze direction) of the virtual camera is not directed toward a specific object (target performer character), it executes control (gaze guidance control) to guide the direction (gaze direction) of the virtual camera toward the specific object (target performer character). This makes it possible to guide the viewer's gaze (gaze direction) toward a specific object (target performer character), thereby improving the level of interest.

Description

Technical Field

[0001] The present invention relates to a program and a system.

Background Art

[0002] Conventionally, a program for controlling a virtual camera according to the movement of a viewer has been known (see Patent Document 1).

Prior Art Document

Patent Document

[0003]

Patent Document 1

Summary of the Invention

Problems to be Solved by the Invention

[0004] An object of the present invention is to improve the interestingness.

Means for Solving the Problems

[0005] To achieve the above object, in a first invention, the program causes a computer to control the direction of a virtual camera according to the movement of a user, and when the direction is not directed to a specific object, executes control for guiding the direction to the specific object. In the program according to the first invention, it becomes possible to guide the user's line of sight to face a specific object, and it becomes possible to improve the interestingness.

[0006] The program according to a second invention is the program according to the first invention, and as the control, at least one of the direction, movement, and arrangement of the specific object is changed. In the program relating to the second invention, by changing at least one of the orientation, movement, and arrangement of a specific object to a special manner, it becomes possible to guide the user's gaze toward a specific object, thereby improving its appeal.

[0007] The program according to the third invention, in the program according to the first or second invention, changes at least one of the arrangement and field of view of the virtual camera as control. In the program relating to the third invention, by controlling at least one of the virtual camera's placement and field of view so that a specific object is included in the virtual camera's image, it becomes possible to guide the user's gaze toward a specific object, thereby improving the level of interest.

[0008] The program according to the fourth invention performs processing on the image of the virtual camera as control in the program according to the first or second invention. In the program relating to the fourth invention, by processing the image of the virtual camera to make it easier to focus on a specific object, it becomes possible to guide the user's gaze toward a specific object, thereby improving its appeal.

[0009] The program according to the fifth invention is such that, in the program according to the first or second invention, the specific object can be set by the user. The program relating to the fifth invention allows the user (viewer or broadcaster) to set specific objects that guide the user's gaze, thereby improving the level of interest.

[0010] The program according to the sixth invention is a program according to the first or second invention in which the specific object is a group consisting of multiple objects. The program relating to the sixth invention makes it possible to guide the user's gaze towards a group of multiple objects, thereby improving its appeal.

[0011] The program according to the seventh invention provides, in the program according to the first or second invention, at least one of the information, items, and content related to the specific object when the direction is directed toward the specific object during the execution of the control. In the program relating to the seventh invention, interest can be enhanced by providing at least one of information, items, and content with respect to a specific object that the user has shown interest in.

[0012] The program according to the eighth invention, in the program according to the first or second invention, executes control to guide the direction to another object if, during the execution of the control, the direction is not facing the specific object. In the program relating to the eighth invention, if the user does not show interest in a specific object that is subject to special control, it becomes possible to guide the user's gaze to other objects, thereby improving interest.

[0013] The program according to the ninth invention terminates the control if, during the execution of the control, the direction is not facing the specific object, in the program according to the first or second invention. In the program relating to the ninth invention, if the user does not show interest in a specific object that is subject to special control, it becomes possible to revert the image of the virtual camera to an image in a state where no control is being performed, thereby improving the level of interest.

[0014] The program according to the tenth invention is such that, in the program according to the first or second invention, the control targeting the same specific object is not repeatedly executed for a single user. In the program according to the tenth invention, excessive control is suppressed, and it becomes possible to improve the interestingness.

[0015] The program according to the eleventh invention is provided with a period during which the execution of the control is prohibited in the program according to the first or second invention. In the program according to the eleventh invention, it becomes possible to suppress a situation that obstructs the progress of a performance, and it becomes possible to improve the interestingness.

[0016] The program according to the twelfth invention notifies a distributor of the execution of the control in the program according to the first or second invention. In the program according to the twelfth invention, it becomes possible to improve the convenience.

[0017] The program according to the thirteenth invention enables a distributor to determine whether or not the direction is directed to the specific object in the program according to the first or second invention. In the program according to the thirteenth invention, it becomes possible to improve the convenience.

[0018] The system according to the fourteenth invention includes a computer that controls the direction of a virtual camera according to the movement of a user, and executes control for guiding the direction to the specific object when the direction is not directed to the specific object. In the system according to the fourteenth invention, it becomes possible to guide the user's line of sight to face a specific object, and it becomes possible to improve the interestingness.

Effect of the Invention

[0019] According to the present invention, it becomes possible to improve the interestingness.

Brief Description of the Drawings

[0020] [Figure 1] It is a diagram showing a schematic configuration of the VR experience system 1. [Figure 2]It is a diagram showing the functional configuration of the information processing apparatus 12. [Figure 3] It is a diagram showing the functional configuration of the server apparatus 20. [Figure 4] It is a diagram showing an example of the UI image g1 for distributor settings. [Figure 5] It is a diagram showing a first example of gaze guidance control. [Figure 6] It is a diagram showing a second example of gaze guidance control. [Figure 7] It is a diagram showing an example of the UI image g2 for viewer settings. [Figure 8] It is a diagram showing an example of the field of view of the distributor.

Mode for Carrying Out the Invention

[0021] Hereinafter, embodiments of the present invention will be described with reference to the drawings. In the present embodiment, an example in which the present invention is applied to the VR experience system 1 will be described. The VR experience system 1 is a system that provides a VR (Virtual Reality) experience to a user. As the VR experience provided by the VR experience system 1, various events (live, concert, musical, drama, game, etc.) implemented in a virtual space (metaverse, game space, live space, etc.) can be used. At this time, the VR experience may be either participatory or viewing. Also, as the VR experience, it may be single play that can be experienced by one user, or multiplayer that can be shared by multiple users. Also, the virtual space may be either a three-dimensional space or a two-dimensional space. This embodiment describes an example in which a VR live performance is provided by the VR experience system 1. In the following description, "field of view" includes concepts such as field of vision, and "gaze" includes concepts such as gaze, focus, and viewpoint. Furthermore, the user who is provided with the VR experience (the viewer) is referred to as the "viewer," while the user who provides the VR experience (the broadcaster) is referred to as the "broadcaster."

[0022] (Outline configuration of VR experience system 1) First, let me explain the general configuration of VR Experience System 1. Figure 1 shows a schematic configuration of VR experience system 1. As shown in Figure 1, the VR experience system 1 is configured to include a plurality of terminal devices 10 connected to a network 2, and one or more server devices 20 connected to the network 2. Each terminal device 10 and server device 20 is connected to each other via the network 2 so that they can communicate with one another.

[0023] (Network 2 configuration) Network 2 consists of a mobile communication system that includes, for example, the Internet and wireless base stations. The mobile communication system consists of, for example, 3G, 4G, 5G mobile communication systems, LTE (Long Term Evolution), and wireless networks that can connect to the Internet via access points.

[0024] (Hardware configuration of terminal device 10) Next, the hardware configuration of terminal device 10 will be described. Terminal device 10 is a device used by users (viewers / broadcasters). Terminal device 10 consists of a wearable device 11 and an information processing device 12. Here, the wearable device 11 and the information processing device 12 may be configured as a single unit or as separate components. The wearable device 11 is worn on the user's head. The wearable device 11 comprises a display 11a, a motion sensor 11b, a speaker 11c, and a controller 11d. The wearable device 11 can be comprised of, for example, a head-mounted display (HMD), VR goggles, VR glasses, smart glasses, AR (Augmented Reality) glasses, a glasses display, a smartphone, a feature phone, a PDA (Personal Digital Assistant), a tablet computer, etc. The display 11a displays a VR image. For example, the display 11a displays a VR image for the right eye and a VR image for the left eye. This allows the user to view the image for the right eye with their right eye and the image for the left eye with their left eye, and based on the parallax between their eyes, they can recognize the VR image as a three-dimensional image. The display 11a can be either a transparent or opaque display device. The motion sensor 11b tracks (detects) the posture information (rotation angle, tilt, etc.) of the wearable device 11 (i.e., the user's head). The motion sensor 11b is configured by appropriately combining a gyroscope, accelerometer, angular acceleration sensor, geomagnetic sensor, etc. The motion sensor 11b may also be configured to be installed outside the wearable device 11. For example, the motion sensor 11b may be configured as an infrared sensor that detects an infrared reflective marker provided on the wearable device 11. Speaker 11c outputs audio associated with the provision of VR images. Speaker 11c is composed of earphones, headphones, etc. Controller 11d accepts user input.

[0025] The information processing device 12 is a computer (information processing device) used by the user (viewer / broadcaster). The information processing device 12 consists of a PC (personal computer), smartphone, feature phone, PDA, tablet computer, etc. The information processing device 12 comprises a processor 12a, memory 12b, storage 12c, communication interface 12d, and input / output interface 12e. These components are connected to each other via a communication bus 12f. The processor 12a controls various operations of the information processing device 12. The processor 12a reads a program from the storage 12c, expands the read program into memory 12b, and executes the expanded program. The processor 12a is composed of, for example, a CPU (Central Processing Unit), an MPU (Micro Processing Unit), a GPU (Graphics Processing Unit), etc. Memory 12b temporarily stores programs and data. By temporarily storing programs and various data read by the processor 12a from storage 12c, memory 12b provides a workspace for the processor 12a. Memory 12b also temporarily stores various data generated by the processor 12a while it is operating according to the program. Memory 12b is composed of volatile memory such as RAM (Random Access Memory). Storage 12c stores various programs and data. Storage 12c is composed of non-volatile memory such as ROM (Read Only Memory), flash memory, and HDD (Hard Disk Drive). In particular, storage 12c stores programs and data for experiencing VR live using the wearable device 11. The communication IF12d controls the transmission and reception of various types of data between the information processing device 12 and external communication devices. The communication IF12d controls communication using, for example, a wireless LAN (Local Area Network), a wired LAN, or short-range wireless communication. The input / output IF12e is an interface for inputting data from an external source to the information processing device 12 and outputting data from the information processing device 12 to an external source. The information processing device 12 is connected to the wearable device 11 (specifically, the display 11a, motion sensor 11b, speaker 11c, and controller 11d) via the input / output IF12e. The information processing device 12 outputs a signal to the display 11a for displaying a VR image, inputs a signal from the motion sensor 11b, outputs a signal to the speaker 11c for outputting sound, and inputs a signal from the controller 11d via the input / output IF12e. The input / output IF12e is configured as, for example, USB (Universal Serial Bus), DVI (Digital Visual Interface), etc.

[0026] (Hardware configuration of server device 20) Next, we will describe the hardware configuration of the server device 20. The server device 20 is a computer (information processing device). The server device 20 is composed of a general-purpose computer such as a workstation or PC. The server device 20 consists of a processor 21, memory 22, storage 23, communication interface 24, and input / output interface 25. These components are connected to each other via a communication bus 26. The processor 21 controls various operations of the server device 20. The processor 21 reads programs from the storage 23, expands the read programs into memory 22, and executes the expanded programs. The processor 21 is composed of, for example, a CPU, MPU, GPU, etc. Memory 22 temporarily stores programs and data. Memory 22 provides the processor 21 with a workspace by temporarily storing programs and various data read by the processor 21 from storage 23. Memory 22 also temporarily stores various data generated by the processor 21 while it is operating according to the program. Memory 22 is composed of volatile memory such as RAM. Storage 23 stores various programs and data. Storage 23 is composed of non-volatile memory such as ROM, flash memory, and HDD. In particular, storage 23 stores programs and data for providing VR live events to the terminal device 10. The communication IF24 controls the transmission and reception of various types of data between the server device 20 and external communication equipment. The communication IF24 controls communication using, for example, wireless LAN, wired LAN, short-range wireless communication, etc. The input / output interface 25 is an interface for inputting data from an external source to the server device 20 and outputting data from the server device 20 to an external source. The input / output interface 25 is composed of, for example, USB, buttons, a mouse, a keyboard, etc.

[0027] (Functional configuration of the information processing device 12) Next, the functional configuration of the information processing device 12 will be described. Figure 2 shows the functional configuration of the information processing device 12. The information processing device 12 functions as an input device that accepts user input operations and as an output device that outputs VR images and sound. As shown in Figure 2, the information processing device 12 functions as a control unit 100 and a storage unit 110 through the cooperation of the processor 12a, memory 12b, storage 12c, communication IF 12d, input / output IF 12e, etc. The memory unit 110 stores programs and various data for providing VR live performances. The control unit 100 comprehensively controls the operation of the information processing terminal 12 by executing a program stored in the memory unit 110. During the provision of VR live streaming, the control unit 100 communicates with the server device 20 as needed to send and receive information. For example, the control unit 100 transmits the posture information of the wearable device 11 (i.e., the user's head) received from the motion sensor 11b to the server device 20. The control unit 100 also transmits the VR images and audio received from the server device 20 to the wearable device 11. The control unit 100 includes an operation reception unit 101, a posture information acquisition unit 102, a display control unit 103, and an audio control unit 104. The control unit 100 can also function as other functional blocks (not shown) depending on the nature of the VR experience provided. The operation reception unit 101 detects and accepts user input operations to the controller 11d. The operation reception unit 101 determines the type of user input operation to the controller 11d and transmits the result to the server device 20. The posture information acquisition unit 102 acquires posture information and position information of the wearable device 11 (i.e., the user's head). Specifically, the posture information acquisition unit 102 acquires posture information and position information of the user's head wearing the wearable device 11 at a predetermined rate based on input values ​​from the motion sensor 11b. The posture information acquisition unit 102 then transmits the acquired posture information and position information to the server device 20. Here, "position information" can be, for example, coordinates indicating the position of the wearable device 11 in the three-dimensional space of the real world (real space). Also, "posture information" can be, for example, the tilt of the wearable device 11 in the three axes of vertical, horizontal, and height. The display control unit 103 transmits the VR image (VR image related to the VR live event) received from the server device 20 to the wearable device 11. The wearable device 11 then displays the VR image received from the display control unit 103 on the display 11a. The audio control unit 104 transmits the audio (audio related to the VR live performance) received from the server device 20 to the wearable device 11. The wearable device 11 then outputs the audio received from the audio control unit 104 through the speaker 11c.

[0028] (Functional configuration of server device 20) Next, the functional configuration of the server device 20 will be described. Figure 3 shows the functional configuration of the server device 20. The server device 20 communicates with the information processing device 12 to implement functions that support the provision of VR live performances by the terminal device 10. For example, when the information processing device 12 downloads an application related to VR live performances for the first time, the server device 20 provides the information processing device 12 with the data necessary for running the VR live performance. As shown in Figure 3, the server device 20 functions as a control unit 200 and a storage unit 210 through the cooperation of the processor 21, memory 22, storage 23, communication IF 24, input / output IF 25, etc.

[0029] The memory unit 210 stores programs and various data for providing VR live performances. The memory unit 210 also stores user information. In the memory unit 210, user information is stored for each terminal device 10 (user account). The control unit 200 comprehensively controls the operation of the server device 20 by executing programs stored in the memory unit 210. For example, the control unit 200 transmits various data and programs to the terminal device 10. The control unit 200 includes a VR control unit 201. The control unit 200 can also function as other functional blocks (not shown) to support the provision of the VR experience by the terminal device 10, depending on the nature of the VR experience being provided. The VR control unit 201 communicates with the information processing device 12 to support the provision of VR live events by the terminal device 10. The VR control unit 201 is composed of a virtual space setting unit 202, a VR live progress unit 203, an avatar setting unit 204, a VR image control unit 205, and a gaze guidance control execution unit 206.

[0030] The virtual space setting unit 202 sets up a virtual space for running a VR live performance. The virtual space can be, for example, a live space that includes a live stage and seating areas. The VR live performance unit 203 conducts the VR live performance in a virtual space (for example, on a live stage). The VR live performance is a VR experience in which one or more performer characters (for example, avatars) perform (singing, playing an instrument, dancing, etc.). The performer characters appearing in the VR live performance can be either NPCs (Non-Player Characters) or avatars that move in response to the streamer's movements. The VR live performance proceeds according to the normal program, except when the gaze guidance control described later is performed. The VR live performance unit 203 also transmits audio related to the VR live performance (singing, playing an instrument, etc.) to the information processing device 12. Before the VR live performance begins, the avatar setting unit 204 places an avatar corresponding to the viewer (hereinafter referred to as the "viewer avatar") at a location selected in response to the viewer's input within the virtual space. For example, the viewer avatar is placed at a seat selected in response to the viewer's input from among several viewing seats provided in the virtual space. The VR image control unit 205 positions virtual cameras at the head position of the viewer avatar. For example, it positions a virtual camera for the right eye at the position of the viewer avatar's right eye, and a virtual camera for the left eye at the position of the viewer avatar's left eye. Then, it controls the direction of the virtual cameras (virtual camera for the right eye and virtual camera for the left eye) (hereinafter referred to as "gaze direction") according to the viewer's movements (including the movements of the user's head, eyes, pupils, eyeballs, facial muscles around the eyes, eyelids, etc.). Specifically, the VR image control unit 205 controls the position of the virtual cameras in the virtual space based on the position information received from the posture information acquisition unit 102. The VR image control unit 205 also controls the gaze direction of the virtual cameras in the virtual space based on the posture information received from the posture information acquisition unit 102. The VR image control unit 204 then controls the range of the image captured by the virtual cameras (hereinafter referred to as "field of view image") based on the position of the virtual cameras and the gaze direction. As a result, the viewing direction of the virtual camera placed in the virtual space changes in accordance with (synchronizes with) the orientation of the viewer's face (i.e., the direction of their gaze) while wearing the wearable device 11. The VR image control unit 205 then generates a VR image at a predetermined rate based on the field of view image captured by the virtual camera and transmits the generated VR image to the information processing device 12. As a result, the display 11a of the wearable device 11 displays the view of the virtual space from the viewer avatar's perspective in real time. The viewer can then view the VR live performance taking place in the virtual space (for example, on a live stage) via the virtual camera. In this process, the viewer can change the viewing direction of the virtual camera by changing the orientation of their face (i.e., the direction of their gaze), thereby changing the view of the scenery they see. In this embodiment, the virtual camera's gaze direction is controlled by inferring the gaze direction from posture information. However, the terminal device 10 may also be configured to include information acquisition means for acquiring information about the user's (viewer / broadcaster's) eyes (e.g., eye movements, pupil state, eyeball movements, changes in facial muscles around the eyes, blinking, etc.), and the virtual camera's gaze direction may be controlled by inferring the gaze direction from the user's eye information acquired by the information acquisition means. In this case, the information acquisition means can be a camera, an infrared sensor, a visible light sensor, etc. For example, by using a camera that images the user as the information acquisition means, the virtual camera's gaze direction can be controlled based on the user's eye information contained in the image captured by the camera. Furthermore, the means for detecting the direction of gaze to control the virtual camera (hereinafter referred to as "gaze detection means") can be automatically switched according to the specifications of the equipment used by the user, the environment in which the user is using the equipment (size of the space the user is in, brightness of the space the user is in, etc.), the application software being run, etc., or the user can manually set (select) which gaze detection technology to use to control the virtual camera, or select and change it as needed. In addition, multiple gaze detection means can be used simultaneously to detect the direction of gaze with higher accuracy. Furthermore, various gaze detection means not listed here may be adopted as appropriate to improve convenience.

[0031] The gaze guidance control execution unit 206 executes gaze guidance control for a target performer character when the gaze guidance conditions are met during the VR live performance. The "Target Performer Character" is selected from among the performer characters appearing in the VR live performance. For example, the Target Performer Character can be a performer character selected by the streamer from among the performer characters appearing in the VR live performance before the start of the VR live performance. Alternatively, the Target Performer Character can be a performer character selected by the viewer from among the performer characters appearing in the VR live performance before the start of the VR live performance. Alternatively, the Target Performer Character can be a performer character from among the performer characters appearing in the VR live performance that meets certain conditions. In this case, "a performer character that meets certain conditions" could be, for example, the performer character that came last in a popularity poll conducted in advance (or in real time during the VR live performance), or the performer character that has the fewest viewers looking in the direction of the virtual camera at a certain point during the VR live performance. Note that the Target Performer Character may be a group consisting of multiple performer characters. For example, in a VR live performance where multiple groups appear, it is acceptable for any one of those groups to be designated as the target performer character.

[0032] The "gaze guidance condition" is met when the viewer's virtual camera's gaze direction (viewer's gaze direction) is not directed towards the target performer character. For example, the gaze guidance condition can be configured to be met when the viewer's virtual camera's gaze direction is not directed towards the target performer character (not directed towards the target performer character's eyes, head, or body). Specifically, the gaze guidance condition can be configured to be met when, within a predetermined time elapsed from the start of the VR live performance, the viewer's virtual camera's gaze direction is not directed towards the target performer character, at a predetermined timing during the VR live performance, or when, during the VR live performance, the viewer's virtual camera's gaze direction is not directed towards the target performer character for a predetermined period of time. Alternatively, the gaze guidance condition can be configured to be met when the target performer character is not within the viewer's virtual camera's field of view (i.e., the field of view image captured by the virtual camera / viewer's field of view). Specifically, the gaze guidance conditions can be configured to be met when: the target performer character is not in the viewer's virtual camera's field of view until a predetermined time has elapsed from the start of the VR live performance; the target performer character is not in the viewer's virtual camera's field of view at a predetermined timing during the VR live performance; or the duration of the state in which the target performer character is not in the viewer's virtual camera's field of view reaches a predetermined time during the VR live performance.

[0033] "Eye-gaze guidance control" is a special control (a special control not executed in the normal program described above) that guides the virtual camera's gaze direction (the viewer's gaze direction) towards the target performer character. As eye-tracking control, it is possible to implement control that changes at least one of the following: the direction of the target performer character's gaze, its movements, and its positioning. For example, as eye-tracking control, it is possible to implement control that directs the target performer character's gaze towards the viewer's virtual camera (directing the target performer character's gaze towards the viewer avatar's head). Furthermore, as eye-tracking control, it is possible to implement control that increases the frequency of the target performer character directing its gaze towards the viewer's virtual camera (directing the target performer character's gaze towards the viewer avatar's head). Additionally, as eye-tracking control, it is possible to implement control that causes the target performer character to perform special actions (for example, calling out to the viewer avatar (viewer), winking at the viewer's virtual camera, waving at the viewer's virtual camera, etc.). Furthermore, as a gaze guidance control, it is possible to change the position of the target performer character (for example, moving closer to the viewer avatar (viewer's virtual camera), moving in front of other performer characters (towards the viewer avatar), moving to the very front (towards the very front on the viewer avatar side), moving from a position outside the viewer's virtual camera's field of view to a position within the field of view, etc.). In this case, objects related to the target performer character (for example, musical instruments played by the target performer character, sets surrounding the target performer character, etc.) may also be configured to change their position along with the target performer character. Furthermore, as a gaze guidance control, it is possible to perform control that changes at least one of the following: the position of the viewer's virtual camera and the field of view. For example, as a gaze guidance control, it is possible to perform control that changes the position of the viewer's virtual camera to an overhead position, a position on the back (away from the stage), etc., so that the target performer character comes into the view of the viewer's virtual camera. Also, as a gaze guidance control, it is possible to perform control that widens the field of view of the viewer's virtual camera so that the target performer character comes into the view of the viewer's virtual camera. Furthermore, as eye-tracking control, it is possible to perform processing on the viewer's field of view image (VR image) captured by the viewer's virtual camera. For example, as eye-tracking control, it is possible to perform processing on the target performer character in the field of view image by applying a predetermined effect (e.g., a sparkling effect). Also, as eye-tracking control, it is possible to perform processing on the background (surroundings) of the target performer character in the field of view image by applying a predetermined effect (e.g., blurring, reducing the number of displayed objects). Also, as eye-tracking control, it is possible to perform processing on other performer characters in the field of view image excluding the target performer character by applying a predetermined effect (e.g., blurring, not displaying, etc.). Here, the eye-tracking control execution unit 206, while eye-tracking control is being performed, sends a notification (hereinafter referred to as "eye-tracking notification") to the broadcaster terminal (for example, the server device 20, the broadcaster's terminal device 10, etc.) indicating that eye-tracking control is being performed, and displays it on the broadcaster terminal's display. The eye-tracking notification may include information specifying the performer character (target performer character) that is the target of eye-tracking control, information specifying the viewer (user name) performing the eye-tracking control, and so on.

[0034] Furthermore, the gaze guidance control execution unit 206 executes interest stimulation control when the conditions for successful gaze guidance are met during the execution of gaze guidance control. The "successful gaze guidance condition" is met when the viewer's virtual camera's gaze direction is directed towards the target performer character. For example, the successful gaze guidance condition can be configured to be met when the viewer's virtual camera's gaze direction is directed towards the target performer character (directed towards the target performer character's eyes, head, or body). Specifically, the successful gaze guidance condition can be configured to be met when, during the execution of gaze guidance control, the duration of the viewer's virtual camera's gaze direction being directed towards the target performer character reaches a predetermined time, or when, during the execution of gaze guidance control, the frequency of the viewer's virtual camera's gaze direction being directed towards the target character reaches a predetermined standard. Alternatively, the successful gaze guidance condition can be configured to be met when the target performer character enters the viewer's virtual camera's field of view (i.e., the field of view image captured by the virtual camera - the viewer's field of view). Specifically, the conditions for successful gaze guidance can be configured to be met when, during the execution of gaze guidance control, the duration of time during which the target performer character is within the viewer's virtual camera's field of view reaches a predetermined time, or when the frequency with which the target performer character enters the viewer's virtual camera's field of view reaches a predetermined standard. "Interest-generating control" is a special control (a special control not executed in the normal program described above) designed to arouse interest in the target performer character. As an interest-generating control, the system can implement controls to provide viewers with at least one of the following: information about the target performer character, items related to the target performer character, and content related to the target performer character. For example, as an interest-generating control, the system can implement controls to send the target performer character's profile information to the viewer's information processing device 12 and display it on the display 11a. Also, as an interest-generating control, the system can implement controls to assign (associate with the viewer's account) an item related to the target performer character that the viewer's avatar can wear (for example, a T-shirt or fan printed with the target performer character's name). In this case, the viewer's avatar may automatically wear the assigned item. Furthermore, as an interest-generating control, the system can implement controls to display on the display 11a UI (User Interface) images for sending objects (for example, tips or gifts) to the target performer character, UI images for contacting the target performer character (for example, chatting), etc. Furthermore, if it is determined that the conditions for successful gaze guidance have been met, the system may be configured to execute a control that brings a performer character different from the target performer character (hereinafter referred to as the "additional performer character") into the viewer's virtual camera field of view. The additional performer character can be a performer character related to the target performer character (for example, a performer character that performs choreography in conjunction with the target performer character). In this case, the system may be configured to display the additional performer character in addition to the target performer character, or to display the additional performer character in place of the target performer character.

[0035] Here, the gaze guidance control execution unit 206 may be configured to execute a second gaze guidance control targeting a different performer character (a new target performer character) than the performer character that was the target of the first gaze guidance control (target performer character) if the conditions for successful gaze guidance are not met during the execution of the first gaze guidance control. In this case, the performer character that is the target of the second gaze guidance control may be configured to be a performer character with a lower priority (next highest priority) than the performer character that was the target of the first gaze guidance control. Alternatively, the configuration can be set so that the performer character targeted for the second gaze guidance control is a performer character that is more popular than the performer character targeted for the first gaze guidance control (for example, a performer character that ranked higher (e.g., 1st place) in a popularity poll conducted in advance (or in real time) than the performer character targeted for the first gaze guidance control, or a performer character that has a larger number of viewers pointing the virtual camera in the direction of their gaze at a predetermined timing during the VR live performance than the performer character targeted for the first gaze guidance control (e.g., the most)). Furthermore, the gaze guidance control execution unit 206 can be configured to terminate the second gaze guidance control and return to the normal program if the gaze guidance success condition is not met during the execution of the second gaze guidance control. In this case, internally, the normal program continues to run even during the execution of each gaze guidance control, and if the gaze guidance success condition is not met during the execution of the second gaze guidance control, the program can be configured to return to the current elapsed time in the normal program (elapsed time since the start of the normal program). Furthermore, if the conditions for successful gaze guidance are not met during the execution of the first gaze guidance control, the system may be configured to terminate the first gaze guidance control and return to normal program execution. Furthermore, the gaze guidance control execution unit 206 may be configured so that gaze guidance control targeting the same performer character is not repeatedly executed for a single viewer. For example, during the execution of a VR live performance, if gaze guidance control targeting performer character A is executed for a single viewer, the system may be configured so that, regardless of whether the gaze guidance success condition is met during the execution of the gaze guidance control, gaze guidance control targeting performer character A is not executed again after the completion of the gaze guidance control. Alternatively, during the execution of a VR live performance, if gaze guidance control targeting performer character A is executed for a single viewer, the system may be configured so that, only if the gaze guidance success condition is not met during the execution of the gaze guidance control, gaze guidance control targeting performer character A is not executed again after the completion of the gaze guidance control. In other words, during the execution of a VR live performance, if gaze guidance control targeting performer character A is performed for a single viewer, and the conditions for successful gaze guidance are met during the execution of said gaze guidance control, the system may be configured to allow gaze guidance control targeting performer character A to be performed again after the completion of said gaze guidance control. Furthermore, the system may be configured to include a prohibited period during the execution of the VR live performance in which the execution of gaze guidance control is prohibited. In other words, the gaze guidance control execution unit 206 may be configured not to execute gaze guidance control during the VR live performance if the gaze guidance conditions are met for the target performer character, but it is within a predetermined prohibited period. The prohibited period may be, for example, a period during which attention should be focused on a performer character other than the target performer character (for example, a period during which a solo part of a performer character other than the target performer character is performed, or a period during which a scene in which a performer character other than the target performer character is the main character is performed). Furthermore, the broadcaster's terminal (e.g., server device 20, broadcaster's terminal device 10, etc.) may be configured to determine whether the viewer's virtual camera gaze direction is directed towards the target performer character during the VR live performance. For example, the gaze guidance control execution unit 206 may be configured to display information on the broadcaster's terminal's display indicating whether the gaze direction of each viewer's virtual camera is directed towards the target performer character for multiple viewers simultaneously watching the VR live performance during the performance. In this case, an icon indicating whether the viewer's virtual camera gaze direction is directed towards the target performer character may be added to the viewer avatar corresponding to each viewer.

[0036] (Overview of VR Live) Next, we will explain the overview of the VR live performance provided by VR Experience System 1. Before the VR live performance begins, viewers select a viewing seat from among several seats provided on the live stage (virtual space) to watch (view / experience) the VR live performance. In this case, a price (e.g., items, points, etc.) is set according to the location of the viewing seat, and viewers can purchase the viewing seat they wish to buy by paying (consuming) the price corresponding to the seat they wish to purchase. In this embodiment, multiple viewers can simultaneously watch the VR live performance by each selecting a viewing seat to watch the VR live performance. For each viewer, once a viewing seat for the VR live performance is selected, a viewer avatar corresponding to that viewer is placed in that seat, and a virtual camera corresponding to that viewer is placed at the head position of the viewer avatar. Each viewer can then view the VR live performance taking place on the stage using the view captured by the virtual camera corresponding to them as their field of view. At this time, each viewer (viewer wearing the wearable device 11) can change the direction of the virtual camera corresponding to them by changing the orientation of their face (i.e., the direction of their gaze). In a VR live performance, one of the performer characters appearing in the VR live performance is designated as the target performer character. During the VR live performance, the gaze direction of the virtual camera corresponding to each viewer is monitored. During the VR live performance, it is also determined whether the gaze guidance condition has been met for each viewer. In this case, the gaze guidance condition is determined to have been met if the gaze direction of the virtual camera corresponding to each viewer is not directed towards the target performer character. If the gaze guidance condition is determined to have been met for a viewer, gaze guidance control is executed for that viewer. In this case, gaze guidance control is executed only for viewers for whom the gaze guidance condition has been met, and for other viewers for whom the gaze guidance condition has not been met, gaze guidance control is not executed, and the program proceeds as normal. However, it is also possible to configure the system so that gaze guidance control is executed not only for viewers for whom the gaze guidance condition has been met, but also for other viewers for whom the gaze guidance condition has not been met. In this embodiment, special control is performed as gaze guidance control to guide the gaze direction of the virtual camera toward the target performer character. As a result, for viewers on whom gaze guidance control is performed, the gaze direction of the virtual camera corresponding to that viewer is guided toward the target performer character. Therefore, it is possible to give viewers on whom gaze guidance control is performed an opportunity to look toward the target performer character. Furthermore, while eye-tracking control is being executed, it is determined whether the eye-tracking success condition has been met for the viewer on whom the eye-tracking control is being executed. In this case, the eye-tracking success condition is determined to have been met if the gaze direction of the virtual camera corresponding to that viewer is directed toward the target performer character. If the eye-tracking success condition is determined to have been met for each viewer, interest-generating control is executed for that viewer in addition to eye-tracking control. In this case, interest-generating control is executed only for viewers on whom the eye-tracking success condition has been determined to have been met, and interest-generating control is not executed for other viewers on whom the eye-tracking success condition has not been met. In this embodiment, special control is performed as interest-stimulating control to stimulate interest in the target performer character. This makes it possible to give viewers who have undergone interest-stimulating control a reason to become interested in the target performer character. Furthermore, for viewers who are determined to have met the conditions for successful gaze guidance, the gaze guidance control and interest stimulation control may be configured to continue until the end of the VR live performance. Alternatively, for viewers who are determined to have met the conditions for successful gaze guidance, the gaze guidance control and interest stimulation control may be configured to continue for a predetermined period of time only. The VR live performance provided by VR Experience System 1 makes it possible to give each viewer an opportunity to become aware of a performer character they were not previously aware of, thereby improving the level of interest. For example, a streamer can effectively promote a performer character they want to showcase by setting that character as their target performer character. Alternatively, each viewer can increase the opportunities for their gaze to align with that character's by setting a performer character they support as their target performer character.

[0037] (Examples) Next, embodiments of the present invention will be described. Figure 4 shows an example of UI image g1 for streamer settings. Figure 5 shows a first example of eye-tracking control. Figure 6 shows a second example of eye-tracking control. Figure 7 shows an example of UI image g2 for viewer settings. Figure 8 shows an example of the streamer's field of view. In this embodiment, a VR live performance consists of two acts: the first and the second. The first act is a VR experience in which five performer characters (performer character A, performer character B, performer character C, performer character D, and performer character E) sing and dance. The second act is a VR experience in which two groups (group A and group B) sing and dance. Group A consists of three performer characters. Group B consists of two performer characters. In this embodiment, each performer character is an NPC (non-player character).

[0038] Before the start of the VR live stream, the streamer can configure various settings related to eye-tracking control on their streaming terminal (e.g., server device 20, terminal device 10, etc.). As shown in Figure 4, in this embodiment, a UI image g1 for streaming settings is displayed on the streamer terminal's display. The streamer can then configure various settings related to eye-tracking control using the UI image g1 for streaming settings. The UI image g1 for streamer settings allows for one or more control settings. Each control setting allows for the execution of gaze guidance control targeting one performer character (or group). Specifically, each control setting includes the following configuration items: "Target," "Condition," "Control Content," "Control Time," and "Priority." In the "Target" section, it is possible to select (set) the performer character group to be targeted for gaze guidance control (i.e., the performer character group to be designated as the target performer character). In this embodiment, in the "Target" section, it is possible to select any of performer character A, performer character B, performer character C, performer character D, performer character E, group A, and group B.

[0039] In the "Conditions" section, it is possible to select (set) the content of the gaze guidance conditions. In this embodiment, the content of the gaze guidance conditions can be selected from the following: "Within XX minutes from the start of the performance, the performer character selected in "Target" is not in the viewer's field of vision", "For XX minutes or more continuously, the performer character selected in "Target" is not in the viewer's field of vision", "Within XX minutes from the start of the performance, the gaze of the performer character selected in "Target" does not meet the viewer's gaze", "For XX minutes or more continuously, the gaze of the performer character selected in "Target" does not meet the viewer's gaze", "Within XX minutes from the start of the performance, the viewer's gaze is not directed towards the performer character selected in "Target", "For XX minutes or more continuously, the viewer's gaze is not directed towards the performer character selected in "Target", etc. Here, "the performer character selected as 'Target' is not in the viewer's field of view" can mean that the performer character selected as 'Target' is not included in the field of view image captured by the virtual camera corresponding to the viewer (for example, the entire body or head of the performer character selected as 'Target' is not included). Also, "the gaze of the performer character selected as 'Target' does not match the viewer's gaze" can mean that the direction of the gaze of the virtual camera corresponding to the viewer and the direction of the gaze of the performer character selected as 'Target' are not facing each other (for example, their eyes or faces are not looking at each other). Furthermore, "the viewer's gaze is not directed towards the performer character selected as 'Target'" can mean that the direction of the gaze of the virtual camera corresponding to the viewer is not directed towards the performer character selected as 'Target' (for example, the body or head of the performer character selected as 'Target'). In the "Control Content" section, it is possible to select (set) the content of the eye-tracking control. In this embodiment, it is possible to select one of the following as the content of the eye-tracking control: "Overhead View," "Layout Change," etc. Here, "overhead view" is a control that changes the position and field of view of the virtual camera to an overhead position and field of view so that the performer character selected in "Target" is included in the view image of the viewer's virtual camera. "Change position" is a control that changes the position of the performer character selected in "Target" so that the performer character selected in "Target" is included in the view image of the viewer's virtual camera. The "Control Time" setting allows you to select (set) the maximum time for which eye-tracking control will continue. The "Priority" setting allows you to select (set) the priority of other control settings when multiple control settings are applied.

[0040] When a VR live streamer makes one or more control settings in the streamer settings UI image g1, eye-tracking control is executed in accordance with those control settings during the execution of the VR live stream. In other words, during the execution of a VR live performance, for each control setting, it is determined whether the gaze guidance condition selected in "Condition" is met for the performer character selected in "Target". If the gaze guidance condition selected in "Condition" is met for the performer character selected in "Target", the gaze guidance control selected in "Control Content" is executed. In this case, if the gaze guidance success condition is met between the start of the gaze guidance control and the elapsed control time selected in "Control Time", interest stimulation control is executed in addition to the gaze guidance control. On the other hand, if the gaze guidance success condition is not met between the start of the gaze guidance control and the elapsed control time selected in "Control Time", the gaze guidance control is terminated according to the elapsed control time selected in "Control Time", and the program returns to normal. In particular, if the gaze guidance conditions are met for multiple control settings and the timing of executing gaze guidance control overlaps, the gaze guidance control will be executed for the control setting with the higher priority selected in "Priority", and the gaze guidance control will not be executed for the control setting with the lower priority selected in "Priority". With this configuration, by designating the performer character that the streamer wants to promote as the target performer character among the performer characters appearing in the VR live performance, it becomes possible to effectively promote that performer character.

[0041] For example, in the example shown in Figure 4, for control setting 1, performer character E is selected under "Target," "Not in sight for 3 minutes or more" is selected under "Condition," "Overhead view" is selected under "Control content," and "5 seconds" is selected under "Control time." This allows monitoring to be performed during the first performance to check whether the performer character E is within the viewer's virtual camera's field of view (s) for each viewer. If the duration of time during which the performer character E is not within the viewer's virtual camera's field of view (s) reaches 3 minutes, gaze guidance control is executed. At this time, as part of the gaze guidance control, the position and angle of the virtual camera are changed to an overhead position and angle so that the performer character E enters the viewer's virtual camera's field of view (s) (see Figure 5(b)). In particular, the position and angle of the virtual camera are changed to an overhead position and angle so that the performer character E's face and eyes enter the viewer's virtual camera's field of view (s). At this time, the direction of the performer character E's gaze may also be controlled to face the viewer's virtual camera (the streamer's avatar's eyes). Furthermore, for the viewer in question, if the conditions for successful gaze guidance are met within 5 seconds of the start of gaze guidance control, interest stimulation control will be executed in addition to gaze guidance control. If the conditions for successful gaze guidance are met, gaze guidance control and interest stimulation control will continue until the end of the first performance. On the other hand, if the conditions for successful gaze guidance are not met within 5 seconds of the start of gaze guidance control, gaze guidance control will end in accordance with the 5-second elapsed time from the start of gaze guidance control, and the program will return to normal. Furthermore, if the conditions for successful gaze guidance are met within 5 seconds from the start of gaze guidance control for the viewer, the system may be configured to allow the viewer to move to a special virtual space (e.g., a VIP room) with performer character E (target performer character) at a predetermined timing (e.g., during a break in the VR live performance, or after the VR live performance ends), and to communicate with performer character E (target performer character) in that special virtual space (e.g., to have a conversation). In this case, the actions of performer character E (target performer character) may be controlled by AI (Artificial Intelligence). Alternatively, if performer character E (target performer character) is an avatar that moves in response to the streamer's movements, the system may be configured to allow direct communication with the avatar (performer character E). In this case, if the amount of money spent by the viewer during the VR live performance (the amount spent on items, points, etc.) reaches a predetermined amount, it may be configured so that the viewer can communicate directly with the avatar, and if the predetermined amount has not been reached, it may be configured so that the viewer can communicate with the AI-controlled performer character E. Here, if the conditions for successful gaze guidance are not met within 5 seconds from the start of the first gaze guidance control, a second gaze guidance control with different content from the first gaze guidance control may be executed. For example, the second gaze guidance control may be a special control that is more effective in guiding the gaze direction of the virtual camera toward the target performer character compared to the first gaze guidance control. Specifically, the first gaze guidance control may be a control that causes performer character E to enter the viewer's virtual camera's field of view s with their gaze direction toward the viewer's virtual camera, and the second gaze guidance control may be a control that causes performer character E to speak to the viewer (for example, call the viewer by name), performer character E to approach the viewer's viewer avatar, or performer character E to be enlarged.

[0042] On the other hand, in the example shown in Figure 4, for control setting 3, Group B is selected for "Target," "Not looking towards within 3 minutes of the start of the performance" is selected for "Condition," "Change placement" is selected for "Control content," and "3 seconds" is selected for "Control time." This allows monitoring during the execution of the second performance to check whether the gaze direction of each viewer's virtual camera is directed towards a performer character belonging to Group B. If, within 3 minutes from the start of the second performance, the gaze direction of the viewer's virtual camera is not directed towards any of the performer characters belonging to Group B (see Figure 6(a)), gaze guidance control is executed. In this case, as part of the gaze guidance control, the arrangement of Group B is changed so that Group B enters the viewer's virtual camera's field of view s (especially the gaze direction) (in particular, in this embodiment, so that Group B is positioned in front of Group A (towards the viewer avatar)) (see Figure 6(b)). In this case, the gaze direction of each performer character belonging to Group B may be controlled to be directed towards the viewer's virtual camera (the eyes of the broadcaster avatar). Furthermore, for the viewer in question, if the conditions for successful gaze guidance are met within 3 seconds of the start of gaze guidance control, interest stimulation control will be executed in addition to gaze guidance control. If the conditions for successful gaze guidance are met, gaze guidance control and interest stimulation control will continue until the end of the second performance. On the other hand, if the conditions for successful gaze guidance are not met within 3 seconds of the start of gaze guidance control, gaze guidance control will end in accordance with the 3-second elapsed time from the start of gaze guidance control, and the program will return to normal.

[0043] In this embodiment, the VR live streamer can configure various settings related to eye-tracking control in the streamer settings UI image g1. However, it is also acceptable to configure the system so that each viewer can configure various settings related to eye-tracking control in the viewer settings UI image g2 shown in Figure 7. In other words, the viewer setting UI image g2 is displayed on the display of each viewer's information processing device 12. Each viewer can then make various settings related to eye-tracking control using the viewer setting UI image g2. The viewer settings UI image g2 allows for the configuration of one or more control settings. Each control setting allows for the configuration of the execution of gaze guidance control targeting one performer character (or group). Specifically, each control setting includes the following configuration items: "Target," "Condition," "Control Content," and "Priority." In the "Target" section, it is possible to select (set) the performer character group (i.e., the performer character group to be used as the target performer character) for eye-tracking control. For example, in the "Target" section, it is possible to select any of the following: performer character A, performer character B, performer character C, performer character D, performer character E, group A, and group B. Under "Conditions," you can select (set) the content of the gaze guidance conditions. For example, the content of the gaze guidance conditions can be any of the following: "Within XX minutes of the start of the performance, the performer character selected in "Target" is not in the viewer's field of vision," "For XX minutes or more continuously, the performer character selected in "Target" is not in the viewer's field of vision," "Within XX minutes of the start of the performance, the gaze of the performer character selected in "Target" does not meet the viewer's gaze," "For XX minutes or more continuously, the gaze of the performer character selected in "Target" does not meet the viewer's gaze," "Within XX minutes of the start of the performance, the viewer's gaze is not directed towards the performer character selected in "Target," or "For XX minutes or more continuously, the viewer's gaze is not directed towards the performer character selected in "Target." Under "Control Content," you can select (set) the content of the eye-tracking control. For example, you can select one of the above options, such as "Overview" or "Layout Change," as the content of the eye-tracking control. The "Priority" setting allows you to select (set) the priority of other control settings when multiple control settings are applied. When one or more control settings are made by each viewer in the viewer settings UI image g2, gaze guidance control is executed during the VR live performance according to those control settings. In other words, during the execution of a VR live performance, for each control setting, it is determined whether the gaze guidance condition selected in "Condition" is met for the performer character selected in "Target". If the gaze guidance condition selected in "Condition" is met for the performer character selected in "Target", the gaze guidance control selected in "Control Content" is executed. In particular, if the gaze guidance conditions are met for multiple control settings and the timing of executing gaze guidance control overlaps, the gaze guidance control will be executed for the control setting with the higher priority selected in "Priority", and the gaze guidance control will not be executed for the control setting with the lower priority selected in "Priority". This configuration allows each viewer to set the performer character they support as their target performer character, thereby increasing the opportunities for their gaze to align with that character.

[0044] Furthermore, in this embodiment, each performer character is an NPC (non-player character). However, each performer character may also be an avatar that moves in response to the streamer's movements. In other words, the broadcaster's terminal (for example, server device 20, terminal device 10, etc.) is configured to include a motion capture function. Based on the broadcaster's motion data acquired by the motion capture function, the movements of the avatar (performer character) corresponding to the broadcaster are controlled. This makes it possible to make the avatar (performer character) corresponding to the broadcaster move in conjunction with the broadcaster's movements. In this case, each streamer may configure their system to be able to determine whether or not each viewer's gaze is directed towards the avatar (performer character) corresponding to that streamer during the VR live performance. For example, as shown in Figure 8, during a VR live performance, the VR image displayed on the display 11a of the wearable device 11 worn by each broadcaster displays information for each viewer watching the VR live performance indicating whether the viewer's virtual camera is directed towards the performer character (avatar) P corresponding to the broadcaster. In this case, for each viewer, if the viewer's virtual camera is directed towards the performer character P, a "○" mark is added to the viewer avatar corresponding to that viewer. On the other hand, for each viewer, if the viewer's virtual camera is not directed towards the performer character P, a "×" mark is added to the viewer avatar corresponding to that viewer. This means that, for example, if a streamer is attracting the attention of viewers in the left-hand seating area from the perspective of the live stage, but not the attention of viewers in the right-hand seating area, the streamer can appeal to the viewers in the right-hand seating area to attract their attention. Here, for a streamer who has spent a specific reward (e.g., an item, points, etc.), the display 11a of the wearable device 11 worn by the streamer may be configured to display the username of each viewer whose gaze is not directed towards the performer character P corresponding to the streamer. This allows the streamer to address each viewer whose gaze is not directed towards their performer character P by their username (e.g., by voice or chat), thereby guiding them to direct their gaze towards the character P. Figure 8 shows an example where each streamer is provided with VR images based on a third-person perspective, but a configuration where VR images based on a first-person perspective are provided is also acceptable.

[0045] Furthermore, in this embodiment, the system may be configured so that eye-tracking control is not performed for viewers who have paid a specific fee, or for viewers who have entered into a specific contract (e.g., a subscription contract).

[0046] Furthermore, in this embodiment, it is determined whether or not the gaze guidance condition is met based on the gaze direction of each viewer (the gaze direction of the virtual camera), and if the gaze guidance condition is met, gaze guidance control is executed for that viewer. However, it is also acceptable to configure the system to set up a group of multiple viewers, determine whether the gaze guidance condition is met based on the gaze direction (gaze direction of the virtual camera) of the multiple viewers in the group, and if the gaze guidance condition is met, perform gaze guidance control on the multiple viewers in the group. For example, it is acceptable to set up a group of three viewers, and if the gaze direction of any of the three viewers in the group is not directed towards the target performer character, the gaze guidance condition is met and perform gaze guidance control on the three viewers in the group. In this case, each of the three viewers in the group may wear a wearable device 11, and the wearable device 11 may individually collect posture information (or eye information) to detect the gaze direction. Alternatively, the three viewers in the group may share a single large display device (display), and a device capable of detecting gaze direction, installed around the display device, may collectively collect posture information (or eye information) for the three viewers in the group to detect their gaze direction. Furthermore, in this embodiment, the target performer character is not limited to a person, but can be other objects such as animals, machines, plants, or installed objects.

[0047] (modified version) Although embodiments and examples of the present invention have been described above, various modifications can be made to the above embodiments and examples. For example, the above embodiments and examples show an example in which the present invention is applied to a VR experience system 1 that provides a VR experience. However, the present invention may also be applied to systems that provide MR (Mixed Reality) experiences, AR (Augmented Reality) experiences, etc. When the present invention is applied to MR and AR, if a specific person is not included in an image taken by a user using a camera of a live event in the real world (real space), the system may perform eye-tracking control to include the specific person in the image being taken by the user and direct the user's gaze toward that person, as described in the above embodiment. In this case, the eye-tracking control may be configured to use sound, light, images, etc., to guide the user to photograph the specific person. For example, the eye-tracking control may be configured to use an arrow image or sound to indicate the direction the camera should be pointed so that the camera is pointed toward the direction where the specific person is located. In this case, the direction in which the specific person is located may be guided in real time by acquiring location information transmitted from a device owned by the specific person (a device that transmits and receives location information). Of course, in the case of live events, information regarding the progress of the performance, such as the position of the specific person, is predetermined for each performance, so the direction in which the specific person is located may be indicated by acquiring the position of the specific person at the current time based on that information. Alternatively, location information obtained from images taken by other users currently participating in the same live event, or from images taken by cameras installed at the live venue, may be shared to indicate the direction in which the specific person is located. Furthermore, when a user is using a camera to photograph a live event in the real world (real space), if the image being photographed by the user does not include a specific person, the system may be configured to replace the image with an image of that specific person taken by another camera as a form of eye-tracking control. In this case, the determination of whether or not the image being photographed by the user includes the specific person can be made by determining the degree of match through image recognition with a person image (image of the specific person) that has been set in advance as the subject to be photographed. In this case, the image recorded on the recording medium (recording device) of the user's camera may be either an image of the real world actually taken by the user with the camera (hereinafter referred to as "real image"), or an image in which all or part of the real image has been forcibly replaced with an image from a camera (an image of the specific person taken by another camera) that has been forcibly distributed by the operator. In this modified example, the target of gaze guidance control is explained as a person, but the target of gaze guidance control is not limited to people; it can be other objects such as animals, equipment, plants, or installed objects. Furthermore, the examples described above are merely illustrations, and the present invention may be realized by adding, combining, partially deleting, or replacing other components. Furthermore, in the examples described above, known configurations not explicitly stated in the specification (configurations known at the time of filing) may be applied as appropriate.

[0048] (Note) Some of the features of the present invention are described below.

[0049] (First invention) "assignment" To improve interest. "Solution" The first invention involves a program that causes the computer (10,20) to control the direction (gaze direction) of the virtual camera in accordance with the user's (viewer's) movements, and if the direction (gaze direction) of the virtual camera is not facing a specific object (target performer character), it executes control (gaze guidance control) to guide the direction (gaze direction) of the virtual camera toward the specific object (target performer character). "effect" It becomes possible to guide the user's (viewer's) gaze (direction of gaze) toward a specific object (target performer character), thereby improving the level of interest.

[0050] (Second invention) "assignment" To improve interest. "Solution" The program according to the second invention, in the program according to the first invention, changes at least one of the following as control (gaze guidance): direction (gaze direction), movement, and position of a specific object (target performer character). "effect" By changing at least one of the following aspects of a specific object (the character being performed): its direction (direction of gaze), movement, and positioning, it becomes possible to guide the user's (viewer's) gaze (direction of gaze) toward that specific object (the character being performed), thereby improving the level of interest.

[0051] (Third invention) "assignment" To improve interest. "Solution" The program according to the third invention changes at least one of the following as control (eye-tracking control) in the program according to the first or second invention: the arrangement of the virtual camera and the field of view. "effect" By controlling at least one of the virtual camera's placement and field of view so that a specific object (the target performer character) is included in the virtual camera's image (viewpoint image), it becomes possible to guide the user's (viewer's) gaze (direction of gaze) toward the specific object (the target performer character), thereby improving the level of engagement.

[0052] (Fourth invention) "assignment" To improve interest. "Solution" The program according to the fourth invention applies processing to the image of a virtual camera (field of view image) as a control (eye-tracking control) in the program according to the first or second invention. "effect" By processing the virtual camera image (field of view image) to make it easier to focus on a specific object (target performer character), it becomes possible to guide the user's (viewer's) gaze (direction of gaze) toward that specific object (target performer character), thereby improving the level of interest.

[0053] (Fifth invention) "assignment" To improve interest. "Solution" The program relating to the fifth invention is such that, in the program relating to the first or second invention, a specific object (target performer character) can be set by the user (viewer or broadcaster). "effect" It will become possible for users (viewers or broadcasters) to set specific objects (target performer characters) that guide the user's (viewer's) gaze (direction of gaze), thereby improving the level of interest.

[0054] (The sixth invention) "assignment" To improve interest. "Solution" The program according to the sixth invention is one in which, in the program according to the first or second invention, a specific object (target performer character) is a group consisting of multiple objects (performer characters). "effect" It becomes possible to guide the user's (viewer's) gaze (direction of gaze) toward a group consisting of multiple objects, thereby improving interest.

[0055] (The seventh invention) "assignment" To improve interest. "Solution" The program according to the seventh invention provides, in the program according to the first or second invention, at least one of the following information, items, and content related to a specific object (target performer character) when the direction of the virtual camera (direction of gaze) is directed toward a specific object (target performer character) during the execution of control (gaze guidance control). "effect" By providing at least one of the following—information, items, or content—related to a specific object (target performer character) that the user (viewer) has shown interest in, it becomes possible to enhance their interest.

[0056] (The eighth invention) "assignment" To improve interest. "Solution" The program according to the eighth invention, in the program according to the first or second invention, executes control to guide the direction of the virtual camera (direction of gaze) to another object (performer character) if, during the execution of control (gaze guidance control), the direction of the virtual camera (direction of gaze) is not directed toward a specific object (target performer character). "effect" If a user (viewer) shows no interest in a specific object (target performer character) that is subject to special control (gaze guidance control), it becomes possible to guide the user's (viewer's) gaze (direction of gaze) to another object (performer character), thereby improving the level of interest.

[0057] (The Ninth Invention) "assignment" To improve interest. "Solution" The program according to the ninth invention terminates the control (gaze guidance control) in the program according to the first or second invention if the direction of the virtual camera (gaze direction) is not facing a specific object (target performer character) while the control (gaze guidance control) is being executed. "effect" If the user (viewer) does not show interest in a specific object (target performer character) that is subject to special control (gaze guidance), it is possible to revert the virtual camera image (field of view image) to an image where the control (gaze guidance control) is not being performed, thereby improving the level of interest.

[0058] (The tenth invention) "assignment" To improve interest. "Solution" The program according to the tenth invention is such that, in the program according to the first or second invention, control (gaze guidance control) targeting the same specific object (performer character) is not repeatedly executed for a single user (viewer). "effect" This prevents excessive control and improves the level of enjoyment.

[0059] (The Eleventh Invention) "assignment" To improve interest. "Solution" The program according to the eleventh invention includes a period (prohibition period) during which the execution of control (gaze guidance control) is prohibited in the program according to the first or second invention. "effect" This makes it possible to prevent situations that disrupt the progress of the performance, thereby improving its entertainment value.

[0060] (The twelfth invention) "assignment" To improve convenience. "Solution" The program according to the twelfth invention notifies the distributor of the execution of control (gaze guidance control) in the program according to the first or second invention. "effect" This will improve convenience.

[0061] (The thirteenth invention) "assignment" To improve convenience. "Solution" The program relating to the thirteenth invention enables the broadcaster to determine whether the direction of the virtual camera (direction of gaze) is directed toward a specific object (target performer character) in the program relating to the first or second invention. "effect" This will improve convenience.

[0062] (The Fourteenth Invention) "assignment" To improve interest. "Solution" The system (1) according to the fourteenth invention includes a computer (10, 20) that controls the direction (direction of gaze) of a virtual camera in accordance with the movement of the user (viewer), and if the direction (direction of gaze) of the virtual camera is not directed toward a specific object (target performer character), it performs control (gaze guidance control) to guide the direction (direction of gaze) of the virtual camera toward the specific object (target performer character). "effect" It becomes possible to guide the user's (viewer's) gaze (direction of gaze) toward a specific object (target performer character), thereby improving the level of interest.

[0063] Furthermore, the first to fourteenth inventions described above can be adapted as appropriate to inventions of apparatus, systems, media, and methods. [Explanation of symbols]

[0064] 1. Game System 2 Network 10 Terminal devices 11 Wearable devices 12 Information Processing Devices 20 Server Devices 201 VR Control Unit 202 Virtual Space Configuration Section 203 VR Live Production Team 204 Avatar Settings Section 205 VR Image Control Unit 206 Eye-line guidance control execution unit

Claims

1. On the computer, The direction of the virtual camera is controlled according to the user's movements. A program that, if the aforementioned direction is not directed toward a specific object, causes the program to execute control to guide the aforementioned direction toward the specific object.

2. The program according to claim 1, wherein the control involves changing at least one of the orientation, movement, and position of the specific object.

3. The program according to claim 1 or 2, wherein the control involves changing at least one of the arrangement and field of view of the virtual camera.

4. The program according to claim 1 or 2, wherein the control involves processing the image of the virtual camera.

5. The program according to claim 1 or 2, wherein the aforementioned specific object can be set by the user.

6. The program according to claim 1 or 2, wherein the specific object is a group consisting of multiple objects.

7. The program according to claim 1 or 2, which, when the direction is directed toward the specific object during the execution of the control, provides at least one of the information, items, and content related to the specific object.

8. The program according to claim 1 or 2, wherein, during the execution of the control, if the direction is not facing the specific object, the program executes control to guide the direction toward another object.

9. The program according to claim 1 or 2, wherein if the direction is not facing the specific object during the execution of the control, the control is terminated.

10. The program according to claim 1 or 2, wherein the control targeting the same specific object is not repeatedly executed for a single user.

11. The program according to claim 1 or 2, wherein a period is provided during which the execution of the control is prohibited.

12. The program according to claim 1 or 2, which notifies the distributor of the execution of the control.

13. The program according to claim 1 or 2, which enables the distributor to determine whether the aforementioned direction is directed toward the specific object.

14. The direction of the virtual camera is controlled according to the user's movements. A system comprising a computer that performs control to guide the direction toward a specific object if the direction is not directed toward a specific object.

Images