Programs and Systems

Abstract

The present invention provides a program, etc., that can suitably display objects on a screen. [Solution] The program causes the computer to select a movable object displayed on the display screen, and displays the selected movable object on the display screen while maintaining predetermined display settings. Preferably, the program allows the computer to receive input from the user to change the display settings, and, according to the changed display settings, to perform a charge to the user or a process related to the consumption of a consumption medium owned by the user.

Description

Technical Field

[0001] The present invention relates to a program and a system.

Background Art

[0002] Patent Document 1 discloses an example of a technique for allowing a user to view content in a virtual space.

Prior Art Documents

Patent Documents

[0003]

Patent Document 1

Summary of the Invention

Problems to be Solved by the Invention

[0004] In one aspect, an object is to provide a program or the like that can suitably present an object on a display screen.

Means for Solving the Problems

[0005] The program causes a computer to select a movable object displayed on a display screen and display the selected movable object on the display screen while maintaining a predetermined display setting.

Effects of the Invention

[0006] In one aspect, an object on a display screen can be suitably presented.

Brief Description of the Drawings

[0007] [Figure 1] It is an explanatory diagram showing a configuration example of an HMD system. [Figure 2] It is a block diagram showing a configuration example of a PC. [Figure 3] It is a block diagram showing a configuration example of an HMD. [Figure 4] This is a block diagram showing an example server configuration. [Figure 5] This is an explanatory diagram regarding the display control process for the virtual space. [Figure 6] This is an explanatory diagram regarding the avatar selection process. [Figure 7] This figure shows an example of the display settings screen. [Figure 8] This is an explanatory diagram regarding the avatar tracking display process. [Figure 9] This flowchart shows an example of the processing steps performed by a PC. [Modes for carrying out the invention]

[0008] The present invention will be described in detail below with reference to the drawings illustrating its embodiments. (Embodiment) Figure 1 is an explanatory diagram showing an example configuration of an HMD (Head Mounted Display) system. This embodiment describes an HMD system that tracks and displays objects in the user's line of sight in a virtual space. The HMD system comprises an information processing device 1, an HMD 2, a controller 3, and a server 4. The information processing device 1 and the server 4 are connected via a network N such as the Internet.

[0009] Information processing device 1 is an information processing device capable of various information processing and information transmission / reception, such as a personal computer, smartphone, or tablet terminal. In this embodiment, information processing device 1 is assumed to be a personal computer, and for simplicity, it will be referred to as PC1 below. PC1 receives content related to the virtual space from server 4 and controls the display of the content on HMD2.

[0010] HMD2 is a display device worn on the user's head that receives content output from PC1 and displays that content. In this embodiment, an HMD2 that displays images in synchronization with PC1 is described as an example, but HMD2 may be a smartphone-mounted HMD, or a standalone HMD that does not require external devices such as PC1.

[0011] Controller 3 is a controller equipped with a stick, buttons, etc., and accepts user input. The input is sent to PC1 and reflected in the display on HMD2.

[0012] Server 4 is a server computer that distributes content related to the virtual space to PC 1. In this embodiment, as an example of content, live streaming content in which performers' avatars perform on a stage will be described. Server 4 distributes data related to the virtual space to each user's PC 1 while synchronizing the position and movement of the performers' and each user's (live viewer's) avatars.

[0013] In this embodiment, when displaying the live streaming content described above, PC1 performs tracking display, which keeps the performer's avatar (an object that can move on the stage) that is in the user's line of sight displayed on the HMD2's display screen. Specifically, as described below, PC1 controls a virtual camera placed in the virtual space to track the performer's avatar so that the avatar is kept displayed.

[0014] Figure 2 is a block diagram showing an example configuration of PC1. PC1 comprises a control unit 11, a main memory unit 12, a communication unit 13, a display unit 14, an input unit 15, and an auxiliary memory unit 16. The control unit 11 is a processor such as one or more CPUs (Central Processing Units), MPUs (Micro-Processing Units), GPUs (Graphics Processing Units), etc. By reading and executing the program P1 stored in the auxiliary storage unit 16, it performs various information processes. The main storage unit 12 is a temporary storage area such as SRAM (Static Random Access Memory), DRAM (Dynamic Random Access Memory), etc., and temporarily stores the data necessary for the control unit 11 to execute arithmetic processing. The communication unit 13 is a communication module for performing communication-related processes, and transmits and receives information with the outside. The display unit 14 is a display screen such as a liquid crystal display, and displays an image. The input unit 15 is an operation interface such as a keyboard, mouse, etc., and accepts operation inputs. The auxiliary storage unit 16 is a non-volatile storage area such as a hard disk, and stores the program P1 (program product) and other data necessary for the control unit 11 to execute processing.

[0015] Note that the PC1 may be provided with a reading unit for reading a portable storage medium 1a such as a CD (Compact Disk)-ROM, DVD (Digital Versatile Disc)-ROM, etc., and read and execute the program P1 from the portable storage medium 1a.

[0016] FIG. 3 is a block diagram showing a configuration example of the HMD2. The HMD2 includes a display unit 21, an audio output unit 22, a communication unit 23, a position / tilt detection unit 24, and a line-of-sight detection unit 25. The display unit 21 is a display screen for displaying images, and it displays the images output from the PC 1. The audio output unit 22 is a speaker for outputting audio, and it plays back the audio output from the PC 1. The communication unit 23 is a communication module for performing communication-related processing, and it transmits and receives information with the PC 1 and the like. The position and inclination detection unit 24 includes, for example, an acceleration sensor, a gyro sensor, a geomagnetic sensor, etc., and detects the position and inclination of the HMD 2 (the user's face). The gaze detection unit 25 is a camera or the like for performing so-called eye tracking, and it recognizes the movement of the user's eyes (pupils) and detects the direction of the gaze.

[0017] Note that the HMD 2 may further include an audio input unit (microphone) or the like for receiving audio input.

[0018] FIG. 4 is a block diagram showing a configuration example of the server 4. The server 4 includes a control unit 41, a main memory unit 42, a communication unit 43, a display unit 44, an input unit 45, and an auxiliary storage unit 46. The control unit 11 is a processor such as one or more CPUs, and it performs various information processes by reading and executing the program P4 stored in the auxiliary storage unit 46. The main memory unit 42 is a temporary storage area such as a RAM, and it temporarily stores the data necessary for the control unit 41 to execute arithmetic processing. The communication unit 43 is a communication module for performing communication-related processing, and it transmits and receives information with the external. The display unit 44 is a display screen such as a liquid crystal display, and it displays images. The input unit 45 is an operation interface such as a keyboard and a mouse, and it receives operation inputs. The auxiliary storage unit 46 is a non-volatile storage area such as a hard disk and a large-capacity memory, and it stores the program P4 (program product) and other data necessary for the control unit 41 to execute processing.

[0019] Note that the PC 1 may include a reading unit for reading a portable storage medium 4a such as a CD-ROM, and it may read and execute the program P4 from the portable storage medium 4a.

[0020] Figure 5 is an explanatory diagram regarding the display control process of the virtual space V. The outline of this embodiment will be described below.

[0021] Figure 5A illustrates how a virtual camera 51, corresponding to the user's viewpoint, is placed in the virtual space V and captures images of the field of view 52. In this embodiment, the virtual space V is a virtual live venue where live streaming takes place. A stage S is placed in the virtual space V, and the performer's avatar A (an example of an object that can move in the virtual space V) performs on the stage S.

[0022] When a user begins viewing content, PC1 first places a virtual camera 51 at the initial position of the virtual space V (for example, the center of the virtual space V). PC1 identifies the field of view 52 of the virtual space V visible to the user based on the position and orientation of the virtual camera 51, and displays the image of that field of view 52 on the HMD2. Figure 5B illustrates an example of the HMD2 display screen. In the example in Figure 5A, avatar A and stage S are contained within the field of view 52, ​​so avatar A and stage S are displayed on the display screen in Figure 5B.

[0023] PC1 detects the position and tilt of HMD2 using the position and tilt detection unit 24 mounted on HMD2. In conjunction with the detected position and tilt of HMD2, PC1 controls the position and orientation of the virtual camera 51 placed in the virtual space V. PC1 identifies the field of view 52 according to the change in the position and orientation of the virtual camera 51 and changes the displayed image of HMD2. In this way, PC1 changes the displayed image (field of view 52) in accordance with the movement of HMD2.

[0024] Now, let's consider the case where Avatar A moves on Stage S. In this case, if the viewer user does not change the orientation of their face (HMD2) or change the display magnification, Avatar A may go out of frame or its display size may become smaller. Also, even if the user operates the virtual camera 51 to follow Avatar A, it may not be able to keep up depending on Avatar A's movements.

[0025] Therefore, in this embodiment, the user selects avatar A that they want to display preferentially on the screen, sets the display magnification and parts to be displayed, and performs tracking display to keep avatar A displayed. This allows avatar A to be displayed on the screen with more preferred display settings without having to operate the field of view (virtual camera 51).

[0026] Figure 6 is an explanatory diagram regarding the selection process for avatar A. Figure 6A shows an overhead view of the virtual space V. Figure 6B shows a display image related to the field of view 52. Note that Figures 6A and 6B are illustrated assuming that there are multiple avatars A1 to A5 on the stage S.

[0027] For example, PC1 detects the user's gaze using the gaze detection unit 25 mounted on the HMD2. PC1 then determines that the avatar A in the line of sight is the avatar A to be followed. Figure 6A illustrates the case where the gaze is focused on avatar A3 located in front of the virtual camera 51. For example, if PC1's gaze is focused on a specific avatar A for a predetermined period of time, it overlays an icon 61 indicating that the gaze is focused, as shown in Figure 6B. If a predetermined operation input (e.g., button press) is received from the controller 3 while the icon 61 is displayed, PC1 overlays the display settings screen shown in Figure 7 onto the display screen.

[0028] Figure 7 shows an example of the display settings screen. The display settings screen includes a default button 71 and a customizable button 72. The default button 71 is a button for displaying avatar A in a tracking manner with the current display settings. When input is received for the default button 71, PC1 controls the virtual camera 51, etc., to perform tracking display while maintaining the current display magnification of the virtual camera 51.

[0029] The customization button 72 is a button for changing the display settings when the avatar is displayed in a follow-up view. When input is received to the customization button 72, PC1 overlays the customization screen shown in Figure 7B. The customization screen displays the amount of money required to change the display settings, corresponding to the type of display setting (e.g., enlarged display of avatar A, color enhancement, display parts, display angle), and the selection button 73.

[0030] PC1 changes various display settings by accepting input from the selection button 73 on the customization screen. In this case, PC1 performs billing processing for the user according to the content of the changed display settings. Specifically, PC1 communicates with server 4 and processes the payment of the displayed charge amount corresponding to the type of display setting that has been changed. Once the payment is complete, PC1 controls the virtual camera 51, etc., to maintain the changed display settings and continue the display accordingly.

[0031] In this embodiment, a charge is applied when the display settings are changed. However, the system may also accept changes to the display settings by consuming some form of consumable medium (e.g., points, items, in-game currency, etc.) that the user possesses within the game.

[0032] Let's return to Figure 6 and continue the explanation. Figure 6C illustrates an example of the screen during tracking display. PC1 controls the virtual camera 51, etc., to display Avatar A while maintaining the display settings set above. For example, consider the case where the display settings are customized and an enlarged display of Avatar A and a display part (e.g., the head) are selected. In this case, PC1 controls the position and orientation of the virtual camera 51 so that the selected display part is located in the center of the display screen (field of view 52), and increases the display magnification of the virtual camera 51 to a predetermined magnification. Note that the display magnification may be changed arbitrarily by the user. As a result, an image of the head of Avatar A is displayed, as shown in Figure 6C.

[0033] Figure 8 is an explanatory diagram regarding the tracking display process of avatar A. Figures 8A and 8B illustrate how the virtual camera 51 tracks avatar A.

[0034] In the example shown in Figure 8A, the orientation of the virtual camera 51 is fixed, and the position of the virtual camera 51 is moved in accordance with the movement of avatar A. For example, as shown in Figure 8A, PC1 controls the position of the virtual camera 51 so that the relative positional relationship (distance) between avatar A and the virtual camera 51 is fixed. This ensures that avatar A is always displayed on the screen.

[0035] In the example shown in Figure 8B, the orientation of the virtual camera 51 is changed in accordance with the movement of avatar A, while the position of the virtual camera 51 remains fixed. For example, as shown in Figure 8B, PC1 controls the orientation of the virtual camera 51 so that avatar A fits within the field of view 52 of the virtual camera 51. Even in this case, the system is controlled so that avatar A is always displayed on the screen.

[0036] PC1 may perform the tracking display using either of the processes illustrated in Figures 8A and 8B. Furthermore, the user may be allowed to choose which of the processes in Figures 8A and 8B is used for the tracking display.

[0037] Furthermore, the tracking display processing illustrated in Figures 8A and 8B is merely an example, and this embodiment is not limited thereto. For example, PC1 may control the virtual camera 51 to move around avatar A in accordance with avatar A's rotational movement (turning motion) so that the virtual camera 51 always keeps the front (face) of avatar A in its field of view. In this way, PC1 only needs to be able to perform tracking display so that avatar A is at least within the field of view 52, ​​and the specific control processing content is not particularly limited.

[0038] As described above, PC1 causes HMD2 to display the selected avatar A in a tracking manner. Here, if certain conditions are met, PC1 may change the target of tracking from the avatar A that is currently being tracked to another avatar A (hereinafter referred to as "second avatar A").

[0039] For example, if Avatar A, which PC1 is following, leaves a designated area in virtual space V, PC1 will change its target to a second Avatar A. Specifically, if Avatar A, which PC1 is following, leaves stage S, PC1 will change its target to one of the remaining Avatar A in stage S (for example, the Avatar A that is closest to the Avatar A that was being followed at the time of its departure).

[0040] Furthermore, for example, PC1 changes the target of its tracking to a second avatar A in response to the user's gaze. Specifically, if the user's gaze is fixed on a second avatar A other than the target avatar A for a predetermined period of time, PC1 changes the target of its tracking to that second avatar A.

[0041] Thus, if the tracking target avatar A goes out of frame or the user's gaze is focused on a different avatar A, the tracking target may be changed to a second avatar A. When switching the tracking target, PC1 may display the display settings screen exemplified in Figure 7 to accept display settings, or it may simply carry over the current display settings (display magnification, etc.) and continue to track the second avatar A.

[0042] Furthermore, if the avatar A being followed is changed to a second avatar A, and the original avatar A re-enters Stage S, the avatar A displayed as the follower may be changed back to the original avatar A.

[0043] As described above, PC1 causes HMD2 to track avatar A. For example, if PC1 receives a predetermined operation input from controller 3 to cancel tracking display, it terminates tracking control of virtual camera 51, places virtual camera 51 back in its original position and orientation, and displays the image of the field of view 52 before tracking display on HMD2. PC1 may also cancel tracking display in conjunction with avatar A's departure, for example.

[0044] In the above example, eye tracking was used to select avatar A to be followed, but this embodiment is not limited to this. For example, PC1 may assume that the user's gaze is in the center of the field of view 52 (i.e., in front of the virtual camera 51) and select avatar A located in the center of the field of view 52 as the target to be followed. Alternatively, PC1 may superimpose icons 61 on all avatars A1 to A5 located within the field of view 52 and accept selection of the target to be followed from avatars A1 to A5 on which the icon 61 is displayed, via manual input through the controller 3. Furthermore, PC1 may accept selection input for avatar A via voice input. In other words, PC1 only needs to be able to appropriately select avatar A displayed on the display screen, and a configuration that detects the user's gaze is not essential.

[0045] Furthermore, although the above describes a configuration in which Avatar A is selected according to the user's gaze, etc., after the content has started to be displayed (while Avatar A is being displayed), this embodiment is not limited to this. For example, PC1 may accept input from the user to set the Avatar A to be followed on a menu screen (not shown) before the content is displayed, and then display the set Avatar A to follow.

[0046] As described above, according to this embodiment, a movable avatar A (object) displayed on the display screen is selected, and the selected avatar A is displayed on the HMD2 while maintaining predetermined display settings. This allows the user to continue viewing avatar A without operating the field of view (virtual camera 51).

[0047] Figure 9 is a flowchart showing an example of the processing procedure performed by PC1. Based on Figure 9, the processing performed by PC1 will be explained. The control unit 11 of PC1 places a virtual camera 51 at the initial position of the virtual space V and displays an image of the field of view 52 of the virtual camera 51 on the HMD2 (step S11). The control unit 11 detects the position and tilt of the HMD2 using the position / tilt detection unit 24 and changes the displayed image (field of view 52) by controlling the position and tilt of the virtual camera 51 in the virtual space V according to the position and tilt.

[0048] The control unit 11 detects the user's gaze using the gaze detection unit 25 of the HMD2 (step S12). The control unit 11 selects the avatar A (object) that is in the user's line of sight on the display screen as the avatar A to be followed (step S13). For example, if the control unit 11 receives a predetermined operation input while the user's gaze is focused on avatar A, it selects the avatar A in the line of sight as the avatar A to be followed.

[0049] The control unit 11 displays the display settings screen illustrated in Figure 7A on the HMD2 and accepts input from the default button 71 or the customize button 72 to determine whether or not to customize the display settings of Avatar A (step S14). If it is determined that customization is not to be performed (S14: NO), the control unit 11 starts a follow display that displays Avatar A while maintaining the current display settings (step S15). The control unit 11 then proceeds to step S19.

[0050] If it is determined that the display settings should be customized (S14: YES), the control unit 11 displays the customization screen illustrated in Figure 7B on the HMD2 and accepts input to change the display settings of Avatar A (step S16). Upon receiving input to change the display settings, the control unit 11 executes processing related to billing the user according to the changed display settings (step S17). The control unit 11 starts a follow-up display that displays Avatar A while maintaining the changed display settings (step S18).

[0051] The control unit 11 determines whether or not to change the avatar A being followed to the second avatar A by determining whether or not predetermined conditions are met (step S19). For example, the control unit 11 determines whether or not the avatar A being followed has left a predetermined area (stage S) in the virtual space V. Also, for example, the control unit 11 determines whether or not the user's gaze has been focused on the second avatar A for a predetermined period of time. If it is determined to change to the second avatar A (S19: YES), the control unit 11 changes the avatar A being followed to the second avatar A (step S20).

[0052] After executing the process in step S20, or if the result in step S19 is NO, the control unit 11 determines whether or not to cancel the tracking display by determining whether or not it has received a predetermined operation input from the user (step S21). If it determines that the tracking display should be canceled (S21: YES), the control unit 11 cancels the tracking display and returns the process to step S11.

[0053] If it is determined that the tracking display will not be canceled (S21: NO), the control unit 11 determines whether or not to terminate the image display on the HMD2 by determining whether or not it has received a predetermined operation input from the user (step S22). If it is determined that it will not terminate (S22: NO), the control unit 11 returns to step S19. If it is determined that it will terminate (S22: YES), the control unit 11 terminates the series of processes.

[0054] Although the above description uses the display of content related to the virtual space V as an example, this embodiment is not limited to this. For example, this system may be applied to free-viewpoint video generated by placing multiple cameras in real space. That is, when displaying free-viewpoint video, the system may select a target object to follow based on the user's gaze, and control the reference viewpoint (change the viewpoint camera) so that the selected object is displayed with the same display settings.

[0055] Alternatively, this system may be applied to video viewing. That is, when displaying a video from a video streaming service, the system may select an object to be tracked based on the user's gaze, and then extract (enlarge) the display area of ​​the selected object from the displayed image so that the selected object is displayed with the same display settings.

[0056] Thus, the content to be displayed in this embodiment is not limited to content related to the virtual space V.

[0057] Based on the above, according to this embodiment, avatar A (object) on the display screen can be presented in a suitable manner.

[0058] (modified version) The above-described embodiment explains a configuration in which the display settings (customization) are changed on the condition of charging the user. Server 4 may also aggregate the changes made to the display settings by each user and present the aggregated results.

[0059] For example, Server 4 aggregates the number of times each user changed their display settings and the amount charged for each type of display setting, such as enlarged display, color enhancement, and display parts for Avatar A. Alternatively, Server 4 may aggregate the display settings for each Avatar A. Server 4 displays the aggregated display settings on the display unit 44. This allows the operator of the system to understand what display settings each user used when browsing.

[0060] Server 4 may also present the aggregated results to the user (content viewer). For example, Server 4 may present display settings that have been set (changed) more than a threshold as popular display settings to the user. This provides the user with reference information on what display settings they should use.

[0061] The embodiments disclosed herein should be considered in all respects to be illustrative and not restrictive. The scope of the present invention is indicated by the claims, not in the sense described above, and all modifications within the sense and scope equivalent to the claims are intended.

[0062] The matters described in each embodiment can be combined with each other. Furthermore, the independent and dependent claims described in the claims can be combined with each other in any combination, regardless of the form of reference. In addition, the claims use a form in which claims referencing two or more other claims (multi-claim form), but are not limited to this. A form in which multi-claims referencing at least one multi-claim (multi-multi-claim) may also be used.

[0063] The contents of this disclosure are listed below.

[0064] (Note 1) On the computer, Select a movable object displayed on the screen. The selected movable object is displayed on the display screen while maintaining the predetermined display settings. program. (Note 2) To the aforementioned computer, The system accepts input from the user to change the aforementioned display settings. Depending on the changed display settings, the system will perform a charge to the user or a process related to the consumption of the user's owned consumption media. The program described in Appendix 1. (Note 3) To the aforementioned computer, Before displaying the object, the system accepts input to set the object to be displayed on the display screen. The user selects one or more of the objects displayed on the aforementioned display screen. The program described in Appendix 1 or 2. (Note 4) To the aforementioned computer, The display settings for displaying the object are determined according to the display settings of the display screen when the object is selected. The program described in any of the appendices 1 to 3. (Note 5) To the aforementioned computer, The user is prompted to select the object displayed in their line of sight. The program described in any of the appendices 1 to 4. (Note 6) To the aforementioned computer, The system selects the object by receiving a predetermined operation input from the user while their gaze is focused on the object. The program described in Appendix 5. (Note 7) To the aforementioned computer, If the object is no longer displayed on the aforementioned display screen, the object to be displayed on the aforementioned display screen is changed to a second object. The program described in any of the appendices 1 to 6. (Note 8) To the aforementioned computer, If the user's gaze is fixed for a predetermined period of time on a second object other than the object displayed on the aforementioned display screen, the object displayed on the aforementioned display screen is changed to the second object. The program described in any of the appendices 1 to 7. (Note 9) A system comprising a control unit, The control unit, Select a movable object displayed on the screen, The selected movable object is displayed on the display screen while maintaining the predetermined display settings. system. (Note 10) On the computer, Select a movable object displayed on the screen. The selected movable object is displayed on the display screen while maintaining the predetermined display settings. Information processing methods. (Note 11) An information processing device comprising a control unit, The control unit, Select a movable object displayed on the screen, The selected movable object is displayed on the display screen while maintaining the predetermined display settings. Information processing device. [Explanation of Symbols]

[0065] 1 PC (Information Processing Unit) 11 Control Unit 12 Main memory 13 Communications Department 14 Display section 15 Input section 16 Auxiliary storage P1 Program 2 HMD 21 Display section 22 Audio output section 23 Communications Department 24 Position and tilt detection unit 25 Eye-line detection unit 3 Controllers 4 servers 41 Control Unit 42 Main memory 43 Communications Department 44 Display section 45 Input section 46 Auxiliary storage P4 Program

Claims

1. On the computer, Select a movable object displayed on the screen. The selected movable object is displayed on the display screen while maintaining the predetermined display settings. program.

2. To the aforementioned computer, The system accepts input from the user to change the aforementioned display settings. Depending on the changed display settings, the system will perform a charge to the user or a process related to the consumption of the user's owned consumption media. The program according to claim 1.

3. To the aforementioned computer, Before displaying the object, the system accepts input to set the object to be displayed on the display screen. The user selects one or more of the objects displayed on the aforementioned display screen. The program according to claim 1.

4. To the aforementioned computer, The display settings for displaying the object are determined according to the display settings of the display screen when the object is selected. The program according to claim 1.

5. To the aforementioned computer, The user is prompted to select the object displayed in their line of sight. The program according to claim 1.

6. To the aforementioned computer, The system selects the object by receiving a predetermined operation input from the user while their gaze is focused on the object. The program according to claim 5.

7. To the aforementioned computer, If the object is no longer displayed on the aforementioned display screen, the object to be displayed on the aforementioned display screen is changed to a second object. The program according to claim 1.

8. To the aforementioned computer, If the user's gaze is fixed for a predetermined period of time on a second object other than the object displayed on the aforementioned display screen, the object displayed on the aforementioned display screen is changed to the second object. The program according to claim 1.

9. A system comprising a control unit, The control unit, Select a movable object displayed on the screen, The selected movable object is displayed on the display screen while maintaining the predetermined display settings. system.

Images