Information Processing Apparatus, Information Processing Method, and Non-Transitory Computer-Readable Medium

Abstract

The present feature relates to an information processing apparatus, an information processing method, and a non-transitory computer-readable medium, and by using the information processing apparatus, the information processing method, and the non-transitory computer-readable medium, it is possible to improve the communication aspect regarding multiple users watching the same content. Content data is acquired, a first field-of-view image corresponding to the field of view of a first user is cut out from a content image based on the content data, and field-of-view information indicating the field of view of a second user watching the content image is also acquired. On a display device, the first field-of-view image is displayed and the field of view of the second user is displayed based on the field-of-view information of the second user. The present feature can be applied, for example, to a case where multiple users watch content such as an omnidirectional image or a free viewpoint image.

Description

[0001] This application is a divisional application of an application with national application number 201880045231.0, international filing date of June 29, 2018, entry date into the country of January 6, 2020, and invention title of "Information Processing Apparatus, Information Processing Method, and Program". Technical Field

[0002] The present technology relates to an information processing apparatus, an information processing method, and a program, and in particular, the present technology relates to an information processing apparatus, an information processing method, and a program that can improve communication regarding, for example, a plurality of users who view the same content. Background Art

[0003] For example, assume that a head-mounted display (hereinafter, simply referred to as an HMD) is used to view content such as a spherical image including an image in a case where the entire circumference is viewed from a viewpoint. In this case, a part of the range of the entire image of the content is displayed on the HMD. A user wearing the HMD can move the user's head to move the displayed range on the HMD, so that the range of the image (scene in the image) that can be viewed by the user in the spherical image, that is, the user's field of view, moves (for example, see Patent Document 1).

[0004] For example, when a user wearing an HMD faces the right side, the displayed range on the HMD is moved to the right side of the currently displayed range in the spherical image. Therefore, the user can see the right side of the current field of view. In addition, when, for example, a user wearing an HMD faces upward, the displayed range on the HMD is moved above the currently displayed range in the spherical image. Therefore, the user can see above the current field of view.

[0005] Citation List

[0006] Patent Document

[0007] Patent Document 1: JP 2005-354644A Summary of the Invention

[0008] Technical Problem

[0009] Incidentally, in the case where a plurality of users wearing HMDs view the same content, each user can view an arbitrary range in the image of the entire content. However, the range viewed by the user (the user's field of view) and the range viewed by other users (the other users' fields of view) may not coincide. Therefore, there is room for improvement in communication regarding, for example, a plurality of users who view the same content.

[0010] In view of such a situation, the present technology is made, and the present technology can improve communication regarding a plurality of users who view the same content.

[0011] Solution to the Problem

[0012] The information processing apparatus or program of the present technology is an information processing apparatus or program for causing a computer to function as an information processing apparatus. The information processing apparatus includes: a content acquisition unit configured to acquire data of content; an image cutting unit configured to cut out a first field-of-view image corresponding to the field of view of a first user from a content image based on the data of the content; a field-of-view information acquisition unit configured to acquire field-of-view information indicating the field of view of a second user viewing the content image; and a display control unit configured to control a display device to display the first field-of-view image, and configured to control the display device to display the field of view of the second user based on the field-of-view information of the second user.

[0013] The information processing method of the present technology is an information processing method, which includes: acquiring data of content; cutting out a first field-of-view image corresponding to the field of view of a first user from a content image based on the data of the content; acquiring field-of-view information indicating the field of view of a second user viewing the content image; and controlling a display device to display the first field-of-view image, and controlling the display device to display the field of view of the second user based on the field-of-view information of the second user.

[0014] In the information processing apparatus, information processing method, and program of the present technology, data of content is acquired, and a first field-of-view image corresponding to the field of view of a first user is cut out from a content image based on the data of the content. In addition, field-of-view information indicating the field of view of a second user viewing the content image is acquired. Further, the first field-of-view image is displayed on a display device, and the field of view of the second user is displayed on the display device based on the field-of-view information of the second user.

[0015] Advantageous Effects of the Invention

[0016] According to an aspect of the present technology, communication among multiple users viewing the same content can be improved.

[0017] Note that the advantageous effects described here may not be limited, and the advantageous effects may be any of the advantageous effects described in the present disclosure. BRIEF DESCRIPTION OF THE DRAWINGS

[0018] Figure 1 is a block diagram showing a configuration example of a content viewing system according to the present technology.

[0019] Figure 2 is a block diagram showing a configuration example of an information processing apparatus.

[0020] Figure 3 is a block diagram showing a configuration example of a display device.

[0021] Figure 4 is a diagram showing an example of an entire image.

[0022] Figure 5 It is a diagram showing a display example of a field-of-view image cut out from a content image.

[0023] Figure 6 It is an enlarged view of a wide-area image superimposed on the field-of-view image.

[0024] Figure 7 It is a diagram showing a display example when a field-of-view direction indication mark is superimposed and displayed on the field-of-view image.

[0025] Figure 8 It is a diagram showing a display example of a field-of-view image when the field of view of a user is narrower than that of other users.

[0026] Figure 9 It is a diagram showing a display example of a field-of-view image when the field of view of a user is wider than that of other users.

[0027] Figure 10 It is a diagram showing a display example when a field-of-view direction indication line is superimposed and displayed on the field-of-view image.

[0028] Figure 11 It is a diagram showing a display example of a tracking image for tracking the field of view of other users.

[0029] Figure 12 It is a diagram showing a display example of a tracking image for tracking the field of view of other users.

[0030] Figure 13 It is a diagram showing a display example when a field-of-view direction indication mark corresponding to a user's speech is highlighted.

[0031] Figure 14 It is a flowchart describing the field-of-view information presentation process.

[0032] Figure 15 It is a diagram showing a display example of a wide-area image switched from the field-of-view image.

[0033] Figure 16 It is a flowchart describing the image switching process.

[0034] Figure 17 It is a diagram showing an example of the rotation amount that moves the field-of-view image beyond the user's head.

[0035] Figure 18 It is a diagram showing the relationship between the movement of the user's line of sight and the movement of the field-of-view image.

[0036] Figure 19 It is a diagram showing the relationship between the movement of the user's line of sight and the movement of the field-of-view image.

[0037] Figure 20 It is a diagram for describing the angle of eye gaze movement.

[0038] Figure 21 It is a diagram for describing the angle of head rotation.

[0039] Figure 22 It is a flowchart for describing the processing of moving the field-of-view image.

[0040] Figure 23 It is a diagram for describing the method of selecting an avatar to be displayed.

[0041] Figure 24 It is a diagram showing an example of the display of an avatar.

[0042] Figure 25 It is a flowchart for describing the avatar display processing.

[0043] Figure 26 It is a block diagram showing an example of the configuration of a general-purpose computer. Detailed Description of the Invention

[0044] Hereinafter, the best mode for implementing the present technology (hereinafter referred to as the embodiment) will be described in detail with reference to the accompanying drawings. Note that the embodiments will be described in the following order.

[0045] 1. View the configuration example of the system according to the content of the present technology

[0046] 2. Presentation of the field of view of each user

[0047] 3. Tracking image for tracking the field of view of other users

[0048] 4. Highlight user speech

[0049] 5. Field-of-view information presentation processing of the user device 20

[0050] 6. Switching between the field-of-view image and the wide-area image

[0051] 7. Movement of the field-of-view image

[0052] 8. Selection of the avatar to be displayed

[0053] 9. A series of processes executed by software

[0054] Note that in this specification, it is assumed that a system represents a set of multiple constituent elements (devices, modules (components), etc.), and not all constituent elements must be located in the same housing. Therefore, both multiple devices housed in separate housings and connected by a network and one device including multiple modules contained in one housing can be assumed as the system in this specification.

[0055] <1. View a configuration example of the system according to the content of the present technology>

[0056] Figure 1 is a block diagram showing a configuration example of a viewing system according to the content of the present technology.

[0057] The content viewing system 10 enables the viewing fields of other users to be understood when multiple users view the same content. Note that the content includes images. Obviously, there can be sounds corresponding to the images. However, only the images of the content will be mentioned below, and the sounds of the content will not be mentioned.

[0058] The times when multiple users view the same content do not have to be the same time. For example, when a user (referred to as user A) views the content, the viewing field information indicating the viewing field of user A can be saved. At a later time, the viewing field of user A can be presented to other users (referred to as user B) who view the same content based on the saved viewing field information of user A.

[0059] The content provided by the content viewing system 10 can be a spherical image including an image in the case of viewing the entire circumference from a viewpoint, or a free viewpoint image including an image in the case of viewing the entire circumference when the viewpoint is moved.

[0060] The spherical image and the free viewpoint image can be any of the following: an image obtained by photographing a real space (a photographed image), an image of a real space being photographed in real time (an image being photographed), an image obtained by using computer graphics to generate a VR (virtual reality) space of a game, etc., an image including a virtual object superimposed on a real space, etc.

[0061] The content viewing system 10 includes a server device 40 and user devices 20-1 to 20-N (N is a natural number) used by users. Hereinafter, the user devices 20-1 to 20-N will be simply referred to as user devices 20 without having to separately distinguish the user devices 20-1 to 20-N.

[0062] The user device 20 includes an information processing device 21 and a display device 22. The information processing device 21 mainly performs processing of cutting out a part of the image of the content to generate a viewing field image to be displayed on the display device 22. The display device 22 mainly displays the viewing field image. In addition, the display device 22 uses other user devices 20 to display the viewing fields of other users to present the viewing fields of other users to the users of the user device 20.

[0063] The field-of-view image displayed on the display device 22 in the image of the content being viewed by the user of the user device 20. Thus, the range of the scene in the field-of-view image is the field of view of the user viewing the field-of-view image. The viewing angle of the field-of-view image, i.e., the field-of-view size of the field of view of the user viewing the field-of-view image (the field of view provided by the display device 22 to the user viewing the field-of-view image (field of view)) varies depending on the display software that is the displayed image or the display device 22. For example, there are display devices with a viewing angle of 90 degrees for the field-of-view image, display devices with a viewing angle of 210 degrees for the field-of-view image, and so on.

[0064] Examples of the display device 22 include display devices worn and used on the user's head, such as AR (augmented reality) glasses and other HMDS. However, the display device 22 can be a flat display device such as a television receiver, or can be a display device for projecting an image such as a projector.

[0065] Note that the information processing device 21 and the display device 22 included in the user device 20 can be integrated or can be placed in different housings and arranged separately. The connection between the information processing device 21 and the display device 22 can be a wired connection or can be a wireless connection.

[0066] The server device 40 is connected to the user device 20 via the Internet 31. The server device 40 includes a content distribution unit 41, a field-of-view information management unit 42, and a communication management unit 43.

[0067] The content distribution unit 41 distributes the data of the content via the Internet 31 according to a request from the user device 20. The data of the content can be distributed to the user device 20 at the same time or at different times.

[0068] The field-of-view information management unit 42 acquires the field-of-view information representing the field of view of the user when the content is viewed in each user device 20 from each user device 20 and manages the field-of-view information.

[0069] The field-of-view information includes at least one of content identification information, elapsed time information, viewpoint information, field-of-view center information, or field-of-view size information.

[0070] The content identification information is information for identifying the content. The elapsed time information exists in the case where the image of the content changes over time, and the elapsed time information is information representing the time elapsed from the top of the content (the time position where the content is reproduced). The viewpoint information is information representing the position of the viewpoint in the content space, which is the space of the content (scene) in the free-viewpoint image, in the case where the content is a free-viewpoint image.

[0071] The center-of-field information is information indicating the center of the field of view (coordinates of the center of the field of view), which is the center of the field-of-view image (the range of the user's field of view) cut out from the image of the content in the user device 20, i.e., the content image based on content data. The field-of-view size information is information indicating the size of the field of view, which is the size of the field-of-view image cut out from the content image in the user device 20, that is, the size of the range of the user's field of view for viewing the field-of-view image (the size of the field of view provided by the display device 22 to the user viewing the field-of-view image). Note that the user can arbitrarily set the field-of-view size within the range permitted by the display device 22. Additionally, the center of the field of view can be regarded as the center of the user's field of view for viewing the field-of-view image, and the field-of-view size can be regarded as the size of the user's field of view for viewing the field-of-view image.

[0072] The content being viewed by the user can be specified with reference to the content identification information. In the case where the content specified by the content identification information is content that changes over time, the time position (timing) (search position) of the content being viewed by the user can be specified with reference to the elapsed time information. The field of view of the user viewing the content specified by the content identification information, that is, the field-of-view image (range of the field-of-view image) in the content image that the user views, can be specified with reference to the center-of-field information, the field-of-view size information, and the viewpoint information (if necessary). Note that, in order to roughly understand the user's field of view (visual field), field-of-view information that does not include the field-of-view size information can be sent and received. Additionally, the field-of-view size of each user can be assumed to be the same fixed value, and the field of view of each user can be specified from the center of the field of view.

[0073] The communication management unit 43 manages communication such as message exchange using voice or characters between users viewing the same content.

[0074] Note that the functions of the server device 40 can be provided to at least one of the user devices 20, and multiple user devices 20 including the functions of the server device 40 can be interconnected via an intranet or the like.

[0075] Figure 2 is a block diagram showing a configuration example of the information processing device 21 included in the user device 20.

[0076] The information processing device 21 includes a communication unit 51, a content holding unit 52, a field-of-view image determination unit 53, an image cutting unit 54, a field-of-view information holding unit 55, an input unit 58, a trigger detection unit 59, and a display control unit 60.

[0077] The communication unit 51 is connected to the server device 40 via the Internet 31 and serves as a content acquisition unit that acquires data of content. In addition, the communication unit 51 acquires image data of a user image (a user image representing the user), such as an icon and an avatar of the user. The data of the content acquired by the communication unit 51 from the server device 40 is recorded in the content holding unit 52. In addition, the communication unit 51 notifies the server device 40 of the field of view information of the user (the first user) sequentially generated by the field of view information holding unit 55 using the user device 20. In addition, the communication unit 51 serves as a field of view information acquisition unit that acquires the field of view information of other users from the server device 40 using other user devices 20 to view the content. The communication unit 51 outputs the field of view information of other users acquired from the server device 40 to the field of view information holding unit 55.

[0078] The content holding unit 52 holds the data of the content acquired from the server device 40. In addition, the content holding unit 52 holds image data of the user image to be superimposed and displayed on an image (field of view image) cut out from the content image based on the content data, such as an icon and an avatar for representing each user.

[0079] The field of view image determination unit 53 determines the center of the field of view based on at least one of the amount of movement of the line of sight notified by the line of sight detection unit 71 ( Figure 3 ) of the display device 22 or the amount of movement of the head notified by the head movement detection unit 72 ( Figure 3 ). In addition, the field of view image determination unit 53 determines the field of view image (the range of the field of view image) of the user to be cut out from the content image based on the center of the field of view and the field of view size (the field of view size of the field of view provided (restricted) to the user viewing the field of view image by the display device 22). For example, in the case where the display device 22 is an HMD, the field of view image determination unit 53 determines the center of the field of view based on at least one of the movement of the head of the user associated with the HMD as the display device 22 or the movement of the line of sight of the user, and determines the field of view image of the user based on the center of the field of view and the field of view size.

[0080] In addition, the field of view image determination unit 53 moves the field of view image of the user based on the line of sight of the user in response to the line of sight of the user approaching the edge of the field of view image of the user.

[0081] Note that the field of view image determination unit 53 may make the rotation angle, which is the amount of movement of the field of view image of the user, greater than the rotation angle of the head of the user based on the rotation angle of the line of sight of the user and the rotation angle of the head.

[0082] In addition, the field-of-view image determination unit 53 may determine the initial position of the field-of-view image corresponding to the time when the user has basically started watching the content (the time when the user has started using the user device 20), that is, the initial position of the center of the field of view of the field-of-view image, based on the field-of-view information of other users. For example, the field-of-view image determination unit 53 may specify, based on the field-of-view information of other users, the area in the content image where the fields of view of other users equal to or greater than a predetermined number are concentrated, and may determine the position in this area as the initial position of the center of the field of view. In addition, the field-of-view image determination unit 53 may determine the field-of-view image to be cut out from the content image based on the initial position of the center of the field of view, that is, determine the image of the range centered on the initial position of the center of the field of view in the content image as the field-of-view image. In this case, the user can easily communicate with other users immediately after starting to watch the content.

[0083] The image cutting unit 54 cuts out (and thus generates) a field-of-view image corresponding to the user's field of view from the content image based on the content data, that is, the field-of-view image determined by the field-of-view image determination unit 53. In addition, the image cutting unit 54 may obtain the field-of-view information of other users from the field-of-view information holding unit 55, and cut out, as the field-of-view image, the user's field-of-view size (the field-of-view size indicated in the field-of-view size information included in the user's field-of-view information) or an image of the field-of-view size of other users from the content image according to the field-of-view information of other users. The field-of-view image may be displayed on the display device 22 as, for example, a tracking image for tracking the field of view of other users. According to the tracking image, the user can set the field of view of other users as the user's field of view and watch the same image as the field-of-view image watched by other users.

[0084] The field-of-view information holding unit 55 sequentially updates and holds the field-of-view information in the user device 20, and outputs the field-of-view information to the communication unit 51 so that the communication unit 51 notifies the server device 40 of the field-of-view information. In addition, the field-of-view information holding unit 55 holds the field-of-view information of other users obtained by the communication unit 51 from the server device 40.

[0085] The input unit 58 includes an operation device such as a remote controller, a voice input device such as a microphone, and an imaging device such as a camera. The input unit 58 uses the operation device to input the user's key operations, uses the voice input device to input the user's voice, or uses the imaging device to input an image obtained by photographing the user.

[0086] The trigger detection unit 59 detects key operations, voice commands, postures, etc. as triggers for predetermined actions based on the user's operations, voices, images, etc. input by the input unit 58.

[0087] The display control unit 60 controls the display to cause the display device 22 to display a field-of-view image and a user image representing the user, such as an icon and an avatar corresponding to each user. In addition, the display control unit 60 controls the display based on the field-of-view information of each user to cause the display device 22 to display the field of view (information representing the field of view) of the user who needs to be displayed.

[0088] For example, the display control unit 60 causes the display device 22 to display a field-of-view position indication image indicating the position of the field of view of another user based on the field-of-view information of the other user. The field-of-view position indication image includes a wide-area image 112 ( Figure 5 ), and the wide-area image 112 includes a field-of-view image (first field-of-view image) of the user of the user device 20 and a field-of-view image (second field-of-view image) corresponding to the field of view of another user. In addition, the field-of-view position indication image includes a symbol image that indicates the position of the field of view of another user and is superimposed on the field-of-view image of the user of the user device 20. Examples of the symbol image include a field-of-view direction indication mark, a field-of-view direction indication line, etc., which will be described later.

[0089] Note that the display control unit 60 can control the display device 22 to switch the field-of-view image of the user of the user device 20 and the wide-area image in response to a predetermined trigger. The predetermined trigger includes at least one of a key operation, a voice command, a movement of the head, or a gesture operation of the user of the user device 20.

[0090] In addition, the display control unit 60 superimposes a user image of another user, such as an avatar, on the field-of-view image of the user and causes the display device 22 to display the image. In the case where there are multiple other users, the display control unit 60 can superimpose at least one of the multiple user images corresponding to the multiple other users on the field-of-view image of the user and cause the display device 22 to display the image.

[0091] For example, the display control unit 60 can set a priority for a user image, such as an avatar, of another user to be superimposed and displayed on the field-of-view image, etc. in the display device 22, and can cause the display device 22 to display some or all of the user images of the multiple other users according to the priority.

[0092] For example, the display control unit 60 can control (determine the priority) whether to superimpose each user image of the multiple user images of the multiple other users on the field-of-view image of the user according to the positional relationship between the field-of-view image of the user and each field of view of the multiple other users.

[0093] In addition, for example, the display control unit 60 can preferentially superimpose the user image of another user among the multiple other users who has a field of view (field of view provided by the field-of-view image) relatively close to the field-of-view image of the user on the field-of-view image of the user and cause the display device 22 to display the image.

[0094] In addition, for example, the display control unit 60 may preferentially superimpose a part of the plurality of user images of a plurality of other users on the field-of-view image of the user according to the communication history between the user and the plurality of other users, and may cause the display device 22 to display the image.

[0095] Figure 3 is a block diagram showing a configuration example of the display device 22 included in the user device 20. In particular, Figure 3 shows a configuration example suitable for the case where the display device 22 is an HMD mounted on the user's head.

[0096] The display device 22 includes a line-of-sight detection unit 71, a head movement detection unit 72, a display unit 73, and a voice input / output unit 74.

[0097] The line-of-sight detection unit 71 detects the user's line of sight. For example, corneal reflection or any other method and technique can be used to detect the line of sight. In addition, the line-of-sight detection unit 71 detects the angle from the middle of the field-of-view image (the middle of the field of view) displayed on the display unit 73 to the user's line of sight (the line-of-sight movement angle) as the movement amount of the line of sight, and sends the line-of-sight movement angle to the field-of-view image determination unit 53 of the information processing device 21.

[0098] The head movement detection unit 72 detects the head rotation angle of the user wearing the HMD as the display device 22 as the movement amount of the head, and sends the head rotation angle to the field-of-view image determination unit 53 of the information processing device 21. The head rotation angle of the user wearing the HMD as the display device 22 is also the rotation angle of the HMD.

[0099] The display unit 73 displays a field-of-view image or the like based on an image signal such as a field-of-view image provided from the display control unit 60.

[0100] The voice input / output unit 74 includes, for example, a microphone and a speaker, and is configured to output the voice of the content (the content voice of the data based on the content) and input the user's voice. The input voice of the user is used for, for example, communication between users watching the same content. That is, the input voice of the user is sent to other user devices 20, and is output from the speaker of the voice input / output unit 74 of the other user devices 20.

[0101] As described above, the HMD as the display device 22 includes a line-of-sight detection unit 71 and a head movement detection unit 72, and detects the movement of the user's line of sight and the movement of the head in the HMD (associated with the HMD).

[0102] <2. Presentation of the Field of View of Each User>

[0103] Figure 4 This is a diagram showing an example of the entire image that includes the entire content image unfolded on a plane.

[0104] The entire image 100 is an image obtained by unfolding the entire content image, which is a spherical image, on a plane using an equidistant cylindrical projection.

[0105] Note that the projection used to unfold the entire content image, which is a spherical image, on a plane can be a projection other than the equidistant cylindrical projection. For example, the Mercator projection method can be used instead of the equidistant cylindrical projection. Hereinafter, the image unfolded on a plane using the equidistant cylindrical projection will be referred to as the equidistant cylindrical projection. Similarly, the image unfolded on a plane using the Mercator projection method will be referred to as the Mercator projection.

[0106] In this case, it is assumed that the content is shared (viewed) by three users A to C. However, the times when users A to C view the content may not be the same time.

[0107] The display device 22 used by user A displays a field-of-view image generated by cutting out a field-of-view range 101A from the content image according to the field-of-view center information and the field-of-view size information (the range of the field-of-view size indicated in the field-of-view size information, centered on the field-of-view center indicated in the field-of-view center information). Similarly, the display device 22 used by user B displays a field-of-view image generated by cutting out a field-of-view range 101B from the content image according to the field-of-view center information and the field-of-view size information of user B. The display device 22 used by user C displays a field-of-view image generated by cutting out a field-of-view range 101C from the content image according to the field-of-view center information and the field-of-view size information of user C.

[0108] The field-of-view range 101A represents the field of view (the range of the field of view) of user A. Similarly, the field-of-view range 101B represents the field of view of user B, and the field-of-view range 101C represents the field of view of user C.

[0109] <Method of Using a Wide-Area Image>

[0110] Figure 5 This is a diagram showing a display example of the field-of-view image 111A that is displayed on the display device 22 used by user A and is generated by cutting out the field-of-view range 101A from the content image.

[0111] However, in Figure 5 this case, referring to Figure 5 , the wide-area image 112 is superimposed and displayed at a predetermined position in the field-of-view image 111A (at the upper left in Figure 5 this case) according to a predetermined operation of user A.

[0112] The wide-area image 112 is a field-of-view position indication image that indicates the positions of the fields of view of other users (in this case, users B and C) who share (view) the same content to present the fields of view of other users to a user (in this case, user A).

[0113] The wide-area image 112 is generated by using all or part of the entire image 100. That is, the wide-area image 112 is generated by cutting out a range from the entire image 100 that includes the field-of-view image (first field-of-view image) of the user and the field-of-view images (second field-of-view images) of other users. Therefore, in the case where the fields of view of multiple users sharing the content are dispersed, the wide-area image 112 is generated based on an image obtained by cutting out a large range of the entire image 100 or based on the entire entire image 100. In the case where the fields of view of multiple users sharing the content are concentrated, the wide-area image 112 is generated based on an image obtained by cutting out a small range from the entire image 100. Note that the wide-area image 112 can always be generated based on the entire entire image 100.

[0114] Figure 6 is an enlarged view of the wide-area image 112 superimposed on the field-of-view image 111A ( Figure 5 ) used by user A.

[0115] The wide-area image 112 displayed on the display device 22 used by user A (the display device 22 used by user A) includes the field-of-view image of user A and the field-of-view images of users B and C, who are other users sharing the same content with user A. Therefore, the wide-area image 112 includes the field-of-view range of user A and the field-of-view ranges of users B and C. Note that although the wide-area image 112 is Figure 6 an equidistant cylindrical projection in

[0116] In Figure 6 , the field-of-view range display 113A indicating the field-of-view range of user A, the field-of-view range display 113B indicating the field-of-view range of user B, the icon 114B as the user image of user B, the field-of-view range display 113C indicating the field-of-view range of user C, and the icon 114C as the user image of user C are superimposed and displayed on the wide-area image 112.

[0117] Based on the field-of-view range display 113B and the icon 114B, the field-of-view range 101B ( Figure 4 ) of user B is presented to user A. Similarly, based on the field-of-view range display 113C and the icon 114C, the field-of-view range 101C ( Figure 4 ) of user C is presented to user A.

[0118] Therefore, the wide-area image 112 is superimposed and displayed on the field-of-view image 111A ( Figure 5 ) displayed on the display device 22 used by user A, and user A can understand the fields of view of other users (in this case, users B and C) of the shared content.

[0119] This can improve communication among multiple users viewing the same content. That is, in a case where users perform communication such as message exchange using voice or characters regarding something being viewed by the users, situations such as miscommunication can be suppressed, and smooth communication between users can be assisted.

[0120] Note that instead of superimposing and displaying the wide-area image 112 on the field-of-view image 111A ( Figure 5 ), the wide-area image 112 can be displayed on a display device other than the display device 22, such as a television receiver.

[0121] Next, another method of presenting the field-of-view range of other users of the shared content to the user will be described.

[0122] <Method Using a Field-of-View Direction Indication Mark>

[0123] Figure 7 FIG. is a display example showing a case where a field-of-view direction indication mark 121 indicating the direction of the position of the field of view of other users of the shared content is superimposed and displayed as a symbol image indicating the position of the field of view of other users on the field-of-view image 111A displayed on the display device 22 used by the user.

[0124] In the Figure 7 display example, the field-of-view direction indication mark 121B corresponding to user B and the field-of-view direction indication mark 121C corresponding to user C are superimposed and displayed on the field-of-view image 111A (the field-of-view image of user A) corresponding to user A.

[0125] The field-of-view direction indication mark 121B is obtained by surrounding the icon 114B ( Figure 6 ) corresponding to user B with a graphic including a sharp protrusion, and the sharp protrusion indicates the direction of the field of view of user B.

[0126] Similarly, the field-of-view direction indication mark 121C is obtained by surrounding the icon 114C ( Figure 6 ) corresponding to user C with a graphic (a graphic showing a so-called speech bubble), and the sharp protrusion indicates the direction of the field of view of user C.

[0127] On the field-of-view image 111A displayed on the display device 22 used by user A, the field-of-view direction indicator marks 121B and 121C superimposed thereon allow user A to understand the direction of the position of the fields of view of other users (in this case, users B and C) of the shared content. User A can, for example, rotate their head in the upper left direction indicated by the sharp protrusion of the field-of-view direction indicator mark 121B to move the field of view of the user (user A) towards the field of view of user B.

[0128] Similarly, user A can, for example, rotate their head in the right direction indicated by the sharp protrusion of the field-of-view direction indicator mark 121C to move the field of view of the user (user A) towards the field of view of user C.

[0129] Figure 8 is a diagram showing a display example of a field-of-view image in the case where the field of view of a user is narrower than that of other users.

[0130] As described above Figure 7 shown, when user A, who is viewing the field-of-view image 111A including the superimposed field-of-view direction indicator mark 121C, rotates their head in the right direction indicated by the sharp protrusion of the field-of-view direction indicator mark 121C, the field-of-view image 111A (as the range cut out from the field-of-view image 111A) displayed on the display device 22 moves to the right side of the content image. Thus, the field of view of the user (user A) gradually approaches the field of view of user C. Therefore, the sharp protrusion of the field-of-view direction indicator mark 121C gradually decreases. In addition, when the field of view of the user (user A) is included in the field of view of user C that is wider than the field of view of the user (user A) (when the field-of-view image of user A (displayed on the display device 22) is included in the field-of-view image of user C), as Figure 8 shown, the sharp protrusion of the field-of-view direction indicator mark 121C disappears.

[0131] Therefore, the size of the protrusion of the field-of-view direction indicator mark 121C superimposed on the field-of-view image 111A displayed on the display device 22 of user A allows user A to understand the proximity and degree of coincidence between the field of view of the user (user A) and the field of view of other users (in this case, user C).

[0132] In addition, when the protrusion of the field-of-view direction indicator mark 121C disappears, it can be determined that the user (user A) and other users (in this case, user C) are viewing the same thing. Therefore, in this state, user A and user C can perform communication such as message exchange, and situations such as miscommunication can be suppressed.

[0133] Figure 9 is a diagram showing a display example of a field-of-view image in the case where the field of view of a user is wider than that of other users.

[0134] As described above Figure 7As described in [above], when user A, who is viewing the field-of-view image 111A including the superimposed field-of-view direction indication marker 121C, rotates their head in the rightward direction indicated by the sharp protrusion of the field-of-view direction indication marker 121C, the field of view of the user (user A) gradually approaches the field of view of user C. Therefore, the sharp protrusion of the field-of-view direction indication marker 121C gradually decreases. In addition, when the field of view of user C, which is narrower than the field of view of the user (user A), is included in the field of view of the user (user A), as Figure 9 shown, the sharp protrusion of the field-of-view direction indication marker 121C disappears, and the field-of-view range display 141C representing the field of view of user C is superimposed and displayed on the field-of-view image 111A.

[0135] Therefore, the size of the protrusion of the field-of-view direction indication marker 121C superimposed on the field-of-view image 111A displayed on the display device 22 used by user A and the field-of-view range display 141C allow user A to understand the proximity and degree of coincidence (overlap degree) between the field of view of the user (user A) and the field of view of other users (in this case, user C).

[0136] In addition, when the protrusion of the field-of-view direction indication marker 121C disappears, it can be determined that the user (user A) and other users (in this case, user C) are viewing the same thing. Therefore, in this state, user A and user C can perform communication such as message exchange, and situations such as miscommunication can be suppressed. Note that the field-of-view direction indication marker 121C and the field-of-view range display 141C are field-of-view position indication images indicating the type of the field-of-view position of user C.

[0137] <Method of Using Field-of-View Direction Indication Line>

[0138] Figure 10 is a diagram showing a display example in the case where the field-of-view direction indication line 131 indicating the direction of the position of the field of view of other users representing shared content is superimposed and displayed as a symbol image indicating the position of the field of view of other users on the field-of-view image 111A displayed on the display device 22 used by the user.

[0139] In Figure 10 the display example, the field-of-view direction indication line 131B is superimposed and displayed on the field-of-view image 111A corresponding to the field of view of user A.

[0140] In the field-of-view direction indication line 131B, one end of the straight line represents the direction of the position of the field of view of user B (in this case, the upper left direction). In addition, the icon 132B corresponding to user B is superimposed and displayed on the field-of-view image 111A near the field-of-view direction indication line 131B to indicate that the field-of-view direction indication line 131B corresponds to user B.

[0141] On the field-of-view image 111A displayed on the display device 22 used by user A, the field-of-view direction indicator line 131B superimposed thereon allows user A to understand the direction of the position of the field of view of another user (in this case, user B) of the shared content. User A can, for example, rotate their head in the upper left direction along the field-of-view direction indicator line 131B to move the field of view of the user (user A) towards the field of view of user B.

[0142] This can suppress situations such as miscommunication in cases where users perform communication such as message exchange, and can assist in smooth communication between users.

[0143] <3. Tracking Image of the Field of View of Another User>

[0144] The user device 20 can track the field of view of another user of another user device 20 based on the field-of-view information of the other user device 20 obtained from the server device 40.

[0145] Figure 11 It is a diagram showing an example of the display of the tracking image AC displayed on the display unit 73 of the display device 22 when the user device 20 used by user A tracks the field of view of user C in a case where the field-of-view image of user A (the range cut out from the content image as the field-of-view image) is larger than the field-of-view image of user C such that the field of view of user A is wider than the field of view of user C.

[0146] The user device 20 used by user A can generate the tracking image AC by cutting out an image of the field of view size of user A or the field of view size of user C as the field-of-view image from the content image centered at the center of the field of view indicated in the field-of-view information of user C based on the field-of-view information of user C.

[0147] In Figure 11 In the tracking image 151AC shown in A of , the field-of-view range display 152C representing the field of view of user C and the icon 153C corresponding to user C are superimposed and displayed on the field-of-view image of the field of view size of user A.

[0148] In Figure 11 In the tracking image 151AC shown in B of , the masking display 154 is superimposed on the range outside the field of view of user C in the field-of-view image of the field of view size of user A, and the icon 153C corresponding to user C is superimposed and displayed on the range of the field of view of user C.

[0149] In Figure 11 In the tracking image 151AC shown in C of , the field-of-view image of the field of view size of user C is expanded to correspond to the field of view size of user A, and the icon 153C corresponding to user C is further superimposed and displayed.

[0150] Figure 12This is a diagram showing an example of the tracking image AC displayed on the display unit 73 of the display device 22 when the user device 20 used by user A tracks the field of view of user C in a case where the field of view image of user A is smaller than that of user C, resulting in a narrower field of view for user A than for user C.

[0151] In Figure 12 the tracking image 161AC shown in A of

[0152] In Figure 12 the tracking image 161AC shown in B of

[0153] The user device 20 can switch between a normal mode for determining the field of view image of the user displayed on the display device 22 and a tracking mode for displaying Figures 11 to 12 the tracking image shown in

[0154] <4. Highlight user's speech>

[0155] When the field of view (including the direction of the field of view) of other users is always displayed on the field of view image 111A corresponding to the field of view of user A, it becomes difficult for user A to see the field of view image 111A as the number of other users increases. Therefore, when displaying the field of view of other users, for example, only the field of view of the user who sends a message during communication between users can be displayed, or the field of view of the user who sends a message can be highlighted.

[0156] Figure 13 This is a diagram showing an example of the display when the user device 20 used by user A highlights the field of view direction indicator mark 171C corresponding to user C in response to the sending of a message by user C.

[0157] In Figure 13 this case, the field of view direction indicator mark 171C highlighted by thickening the contour line is superimposed on the field of view image 111A corresponding to the field of view of user A. In addition, the text display 172 corresponding to the voice of user C ( Figure 13 "Look here" in

[0158] Note that, in order to allow user A to visually and more easily distinguish another user who sends a message from yet another user who does not send a message, the field-of-view direction indication marker 171B corresponding to yet another user (in this case, user B) who does not send a message can be displayed by changing the outline to a dotted line or by reducing the brightness to make the field-of-view direction indication marker 171B inconspicuous.

[0159] <5. Visual field information presentation process of user device 20>

[0160] Next, a visual field information presentation process for presenting the position of the visual fields of other users to the user while displaying a visual field image corresponding to the user's visual field on the display device 22 of the user device 20 will be described.

[0161] Figure 14 is a flowchart describing the visual field information presentation process.

[0162] In the information processing device 21 ( Figure 2 ) of the user device 20 used by user A, in step S1, the communication unit 51 is connected to the server device 40 via the Internet 31 and acquires the data of the content. The data of the content acquired from the server device 40 is held in the content holding unit 52.

[0163] In addition, in step S1, the communication unit 51 starts acquiring the visual field information of other users (for example, users B and C) who are viewing the same content from the visual field information management unit 42 of the server device 40.

[0164] In step S2, the visual field image determination unit 53 sets the position in the range viewed by more other users as the initial position of the visual field center based on the visual field information of other users. In addition, the visual field image determination unit 53 determines the user's (corresponding to the visual field) visual field image in the content image based on the initial position of the visual field center and the visual field size. By setting the initial position of the visual field center at the position in the range viewed by more other users, the user (in this case, user A) can immediately view the same image as other users as the visual field image, and the user can communicate easily.

[0165] In step S3, the image cutting unit 54 cuts out the visual field image 111A ( Figure 5 ) determined by the visual field image determination unit 53 from the content image to generate the visual field image 111A.

[0166] In step S4, the display control unit 60 superimposes a field-of-view position indication image on the field-of-view image 111A as a display representing the field of view of other users. Subsequently, the display control unit 60 supplies an image signal of the field-of-view image 111A including the superimposed field-of-view position indication image to the display device 22, and causes the display unit 73 to display the field-of-view image 111A.

[0167] Here, the field-of-view position indication image includes a wide-area image 112( Figure 6 ), a symbol image, etc. The symbol image includes field-of-view direction indication marks 121B and 121C( Figure 7 ), a field-of-view direction indication line 131B( Figure 10 ), etc.

[0168] In step S5, the field-of-view image determination unit 53 determines whether to move the field-of-view image displayed on the display unit 73 based on the amount of movement (rotation angle) of the line of sight and the amount of movement (rotation angle) of the head. Here, when the field-of-view image determination unit 53 determines to move the field-of-view image, the process proceeds to step S6.

[0169] In step S6, the field-of-view information holding unit 55 updates and holds the user's field-of-view information based on the amount of movement of the line of sight and the amount of movement of the head, and outputs the field-of-view information to the communication unit 51. The communication unit 51 notifies the server device 40 of the updated field-of-view information of the user via the Internet 31. Note that the field-of-view information notified to the server device 40 will be provided to other user devices 20.

[0170] The field-of-view image determination unit 53 determines a field-of-view image corresponding to the user's field of view in the content image based on the updated field-of-view information. The process returns to step S3, and the subsequent processing is repeated until the reproduction of the content ends. In this way, a field-of-view image corresponding to the user's field of view in the content image is determined based on the updated field-of-view information, and the field-of-view image displayed on the display unit 73 moves based on the amount of movement of the user's line of sight and the amount of movement of the head (as the range cut out from the content image for the field-of-view image moves).

[0171] In addition, when the field-of-view image determination unit 53 determines not to move the field-of-view image in step S5, the field-of-view image determination unit 53 sets the previous field-of-view image as the field-of-view image corresponding to the user's field of view again, and skips step S6. Then, the process returns to step S3, and the subsequent processing is repeated until the reproduction of the content ends.

[0172] Based on the above field of view information presentation process, the field of view of other users (including the direction of the position of the field of view) is displayed to present the field of view of other users to the user. The user can move the line of sight and head based on this presentation to move the user's field of view, and the user can immediately see the same image as that of other users. This can suppress situations such as miscommunication in the communication between users.

[0173] <6. Switching between the field of view image and the wide - area image>

[0174] In the description so far, the wide - area image ( Figure 6 ) is superimposed and displayed on the field of view image ( Figure 5 ) or is displayed on other display devices. However, the display device 22 can switch and display the field of view image and the wide - area image.

[0175] When switching the display of the display device 22 from the field of view image to the wide - area image, the field of view image can be instantaneously switched to the wide - area image. Additionally, the field of view image can be reduced, or the viewpoint can be moved, so as to gradually change the field of view image into the wide - area image.

[0176] When, for example, a user (e.g., user A) wants to know the field of view (including the direction of the field of view) of another user (e.g., user B), the field of view image can be switched to the wide - area image. Additionally, even in the absence of other users, the field of view image can be switched to the wide - area image when user A wants to know the position of the user's field of view in the entire content image or the user's viewpoint (in the case where the content is a free - viewpoint image).

[0177] Examples of triggers for switching the field of view image to the wide - area image are as follows.

[0178] For example, the operation of a key on the remote controller included in the input unit 58 ( Figure 2 ) operated by the user can be a trigger, and the field of view image can be switched to display the wide - area image only during the period when the user is pressing the key on the remote controller.

[0179] Additionally, for example, a voice command spoken by the user can be a trigger. The field of view image can be switched to the wide - area image in response to a predetermined voice command “zoom out” spoken by the user, and the wide - area image can be returned to the field of view image in response to the voice command “return” spoken by the user.

[0180] Additionally, for example, the movement of the user's head can be a trigger. When the user faces directly downward, the field of view image can be switched to an aerial view or the like as the wide - area image. When the user faces directly upward, the field of view image can be switched to an image of a 2D map as the wide - area image corresponding to the content space.

[0181] In addition, for example, communication between users can be a trigger. When a user starts communicating with another user, or when the distance to another user increases during communication such that the other user is out of the field of view, the image can be switched to a wide-area image. Thereafter, the field-of-view image and the wide-area image can be switched in response to a predetermined pose operation (e.g., a nod of the user's head or a movement of the hand).

[0182] In addition, for example, an event in the content can be a trigger. In response to the event, the field-of-view image can be switched to a wide-area image including the location of the event.

[0183] Furthermore, in a case where the content is, for example, a real-time image in the real space, the generation of an alarm in the real space (e.g., discovery of a suspicious person, fire, operation of an emergency button, etc.) can be a trigger. In response to the alarm, the field-of-view image can be switched to a wide-area image including the location where the alarm is generated.

[0184] In addition, the content of a conversation that is communication between users can be analyzed, and when the analysis result indicates predetermined content, the switching between the field-of-view image and the wide-area image can be triggered.

[0185] In addition, the field-of-view image and the wide-area image can be switched in response to a user's gesture (action) of looking left or right or a pulling gesture.

[0186] Next, Figure 15 is a diagram showing a display example of a wide-area image switched from a field-of-view image. Figure 15 A in is a display example of a field-of-view image 181 displayed on a display device 22 worn by user A.

[0187] Figure 15 B in is a display example of an aerial view wide-area image 182 switched from the field-of-view image 181 ( Figure 15 A in) in response to the detection of a trigger. A field-of-view marker 183 indicating the position of the user (i.e., the viewpoint) and the direction of the field of view is superimposed on the wide-area image 182. The aerial view is displayed at an angle such that the field-of-view marker 183 is not within the shadow of an object such as a main building on the wide-area image 182, while giving priority to making the current field of view indicated by the field-of-view marker 183 easily understandable.

[0188] Figure 15 C in is a display example of an isometric cylindrical projection wide-area image 184 switched from the field-of-view image 181 ( Figure 15 A in) in response to the detection of a trigger. A field-of-view range display 185A representing the field of view of user A is superimposed on the wide-area image 184.

[0189] Figure 15 D in is triggered when, for example, user A starts communicating with user B and is switched from the field-of-view image 181 (Figure 15 A) Display example of the wide-area image 186 of the switched equidistant cylindrical projection. The field-of-view range display 187A representing the field of view of user A and the field-of-view range display 187B representing the field of view of user B are superimposed on the wide-area image 186.

[0190] Figure 15 E in is when, for example, user A starts communicating with user B and is triggered from the field-of-view image 181 ( Figure 15 A) Display example of the wide-area image 188 of the switched equidistant cylindrical projection. The wide-area image 188 is an extended image including the fields of view of user A and user B. The field-of-view range display 189A representing the field of view of user A and the field-of-view range display 189B representing the field of view of user B are superimposed on the wide-area image 188.

[0191] Regarding the wide-area image, when the fields of view of multiple users change significantly while the multiple users are communicating while watching content, an overhead view etc. that simultaneously includes the fields of view of multiple users (field-of-view images of multiple users) can be used as the wide-area image and automatically displayed. Regarding AR content, for example, an independent camera can be used to take pictures of the real world, and the images can be used to generate the wide-area image.

[0192] In addition, in the case where there is a user wearing AR glasses and another user remotely viewing the image of a live camera arranged in the user's space, an overhead image including the fields of view of the user and the other user can simultaneously be used as the wide-area image and automatically displayed during the communication between the user and the other user.

[0193] In addition, when the fields of view of multiple users change significantly, the display of the wide-area image can take precedence over the display of the field-of-view image.

[0194] In addition, in the case where it is estimated that multiple users are focusing on different objects in the content image, an image including the different objects (where the images of the different objects can be viewed) can be used as the wide-area image.

[0195] In addition, the wide-area image can be an image in which the image is shrunk to super-wide-angle to display a predetermined range around the field-of-view image of the user in the content image and display the positions of other users.

[0196] In addition, regarding the overhead view as the wide-area image, the viewpoint of the overhead view and the viewing angle of the overhead view can be selected based on the direction of the HMD as the display device 22.

[0197] When the user recognizes another user and starts a conversation, that is, for example, when the user selects another user from a menu displaying an icon including the other user and starts a voice call or a text chat, the display of the wide-area image can be triggered.

[0198] In addition, when another user leaves the user during a conversation such as a voice call between the user and the other user, the display of the wide-area image can be triggered.

[0199] Note that in the case where the content is a free-viewpoint image, an image of a scene of the content space included in the free-viewpoint image viewed from an arbitrary direction from an arbitrary viewpoint can be a field-of-view image, and the field-of-view image can be displayed on the HMD that is the display device 22. However, the movable range of the user wearing the HMD is restricted. Therefore, regarding the movement of the field-of-view image, the user can be allowed to move the image in the rotational direction, and the user device 20 can automatically perform a parallel movement. In the case of viewing such a free-viewpoint image, the wide-area image can also be displayed as described above.

[0200] In addition, the wide-area image can be displayed to cover the entire field of view of the user. Further, the wide-area image can be displayed below the field of view of the user so as to be attached to the ground when the user faces downward. In addition, a window can be displayed in a part of the field-of-view image, and the wide-area image can be displayed in the window.

[0201] Furthermore, the wide-area image can be displayed in a blurred manner using a brightness lower than that of the field-of-view image.

[0202] Based on the wide-area image, in the case where the field of view of the user is different from that of another user and the communication is not smooth, a wide-area image including the fields of view of the user and the other user can be displayed, and the communication obstacle (defect) can be improved.

[0203] Moreover, based on the wide-area image, a user who cannot recognize the position in the free-viewpoint image can easily return to the desired position or can move a long distance in a short time.

[0204] <Image switching process for switching between the field-of-view image and the wide-area image>

[0205] Next, Figure 16 is a flowchart for describing the image switching process for switching between the field-of-view image and the wide-area image.

[0206] However, the movement of the field of view of the user will not be mentioned in the description of the image switching process.

[0207] In step S11, the information processing device 21 generates a field-of-view image corresponding to the field of view of the user, and causes the display unit 73 to display the generated field-of-view image.

[0208] Specifically, the image cutting unit 54 of the information processing device 21 cuts out the field-of-view image from the content image held in the content holding unit 52 to generate a field-of-view image, and the display control unit 60 supplies the image signal of the generated field-of-view image to the display device 22 to cause the display unit 73 to display the field-of-view image.

[0209] In step S12, the information processing device 21 determines whether there is a trigger for indicating a switch between the field-of-view image and the wide-area image. Specifically, the trigger detection unit 59 of the information processing device 21 determines whether a wide-area trigger for indicating a switch to the wide-area image is detected based on the user's key operation, the user's voice, the user's image, etc. input by the input unit 58. Here, when the trigger detection unit 59 determines that the wide-area trigger is not detected, the process returns to step S11. Therefore, the field-of-view image is still displayed on the display unit 73.

[0210] On the other hand, when the trigger detection unit 59 determines in step S12 that the wide-area trigger is detected, the process proceeds to step S13.

[0211] In step S13, the information processing device 21 generates a wide-area image, and the display device 22 switches the display of the display unit 73 from the field-of-view image to the wide-area image.

[0212] Specifically, for example, the display control unit 60 superimposes the field-of-view range display (e.g., the field-of-view range displays 187A and 187B of D in Figure 15 ) representing the field of view of each user on the entire image 100 of the content image held in the content holding unit 52 ( Figure 4 ) to generate a wide-area image (e.g., the wide-area image 186 of D in Figure 15 ), and provides the image signal of the generated wide-area image to the display device 22 to cause the display unit 73 to display the wide-area image.

[0213] Note that the display control unit 60 can determine the display position of the field-of-view range display representing the field of view of each user based on the field-of-view information of each user held in the field-of-view information holding unit 55.

[0214] After step S13, the process proceeds to step S14, and the trigger detection unit 59 determines whether a field-of-view trigger for indicating a switch to the field-of-view image is detected based on the user's key operation, the user's voice, the user's image, etc. input by the input unit 58. Here, when the trigger detection unit 59 determines that the field-of-view trigger is not detected, the process returns to step S13. Therefore, the wide-area image is still displayed on the display unit 73.

[0215] When the trigger detection unit 59 determines in step S14 that the field-of-view trigger is detected, the process returns to step S11, and the display of the display unit 73 is switched to the field-of-view image.

[0216] According to the above image switching process, the user can perform operations, voice, actions, postures, etc. as triggers for switching between the field-of-view image and the wide-area image.

[0217] A user who views a wide - area image can understand the field of view of other users and can suppress situations such as miscommunication in communication between users.

[0218] In addition, a user who views a wide - area image can understand the field of view and the viewing point of the user (the position of the user in the content space in the case where the content is a free - viewpoint image). Therefore, for example, in a case where a user cannot recognize the position of the user in the content space, the user can quickly return to the desired position.

[0219] <7. Movement of the field - of - view image>

[0220] Next, the movement of the field - of - view image will be described.

[0221] In an HMD, a user rotates the head to move the field - of - view image (the range of the field - of - view image) to be cut out from the content image, and thus moves the user's field of view. However, for example, in a case where the field - of - view image is moved to a position directly behind the field - of - view image currently displayed on the display device 22 in the content image, the body also needs to be rotated together with the head because a person generally cannot rotate only the head by 180 degrees. Therefore, if a user often looks left or right or backward while wearing the HMD, the user will feel physical fatigue.

[0222] Therefore, the user device 20 in the present embodiment moves the field - of - view image (the field - of - view center indicated in the field - of - view center information included in the field - of - view information for determining the field - of - view image) by more than the amount of rotation of the head to reduce the user's physical fatigue. That is, the rotation angle of the field - of - view image is made larger than the rotation angle of the user's head. For example, when the user rotates the head by 90 degrees in the horizontal direction, the field - of - view image rotates and moves by 180 degrees in the horizontal direction.

[0223] Figure 17 is a diagram showing an example in which the field - of - view image is rotated and moved by more than the amount of rotation of the user's head.

[0224] Figure 17 A in shows a situation of viewing from above a state where a user faces forward in a predetermined direction and views a spherical image as the content image 200. In the content image 200, there is an object 201 in front of the user and an object 202 behind the user. In this case, when the user faces forward, the object 201 exists in the field - of - view image (the range of the field - of - view image) 203 to be cut out from the content image 200, and the user can visually recognize the object 201. On the other hand, the object 202 exists outside the user's field - of - view image 203, and the user cannot visually recognize the object 202.

[0225] As Figure 17As shown in B of FIG. 0, when the user rotates the head 90 degrees in this state, the field-of-view image 203 is rotated in the same direction as the head rotation in the content image 200 by, for example, 180 degrees which is greater than the head rotation angle. In this case, the object 201 is outside the field-of-view image 203 after rotation, and the user cannot visually recognize the object 201. However, the object 202 is within the field-of-view image 203 after rotation, and the user can visually recognize the object 202.

[0226] However, as Figure 17 shown, in the case where the field-of-view image only moves by an amount equal to or exceeding the rotation amount of the head, it may make the user feel, for example, discomfort or uneasiness. Therefore, in the user device 20, the field-of-view image can be rotated and moved by more than the rotation amount of the head only when the user takes an action that can be considered as a strong willingness of the user to greatly move the field-of-view or under conditions where the user is less likely to notice a large movement of the field-of-view.

[0227] <Movement of the Field-of-View Image According to the Movement of the Line of Sight>

[0228] When only the line of sight is moved without rotating the head, the user device 20 can move the field-of-view image, thereby moving the user's field of view.

[0229] Figure 18 and Figure 19 are diagrams showing the relationship between the movement of the user's line of sight and the movement of the field-of-view image.

[0230] Figure 18 A of FIG. 21 is an example of the entire image 190 obtained by unfolding the content image on a plane, and the range 191 on the entire image 190 indicates the range cut out as the field-of-view image 192 ( Figure 18 B of FIG. 22) to be displayed on the display unit 73 of the display device 22. Figure 18 B of FIG. 25 is an example of the display of the field-of-view image 192, and the X mark indicates the line-of-sight position 195 of the user on the field-of-view image 192. In Figure 18 the case of B of FIG. 27, the line-of-sight position 195 is located at the lower center of the field-of-view image 192.

[0231] Figure 19 A of FIG. 31 indicates the movement of the user's line of sight. In Figure 19 A of FIG. 33, the line-of-sight position 195 of the user is moved to the lower side of the left edge of the field-of-view image 192. Figure 19 B of FIG. 35 shows the range cut out as the field-of-view image after the movement of the field-of-view image according to the movement of the line of sight, that is, after the movement of the line of sight shown in Figure 19 A of FIG. 37.

[0232] As Figure 19As shown in A of FIG. 0, when the user moves the line of sight so that the line-of-sight position 195 approaches the edge of the field-of-view image, it can be considered that the user is willing to view outside the range 191 of the current field-of-view image. Therefore, the field-of-view image determination unit 53 moves the field-of-view image based on the line-of-sight position 195 in response to the line-of-sight position 195 approaching the edge of the field-of-view image. That is, the range cut out as the field-of-view image is moved from the range 191 ( Figure 17 of A) to the range 196 ( Figure 18 of B) in the direction same as the moving direction of the line of sight. The movement of the field-of-view image based on the line-of-sight position 195 (of the line of sight after movement) can be performed, for example, by setting the line-of-sight position 195 as the center of the field of view to determine the field-of-view image to be cut out from the content image.

[0233] <Movement of the field-of-view image according to the movement amount of the line of sight and the rotation amount of the head>

[0234] In the case where the user quickly moves the line of sight and rotates the head, it may be difficult to make the user notice a large movement of the field of view. Therefore, when the rotation angle and the rotation amount (rotation angle) of the head as the movement amount of the user's line of sight are equal to or greater than a threshold value within a certain time, the user device 20 moves the field-of-view image by more than the rotation amount of the head.

[0235] However, in the case where the rotation direction of the head is different from the moving direction of the line of sight, the user may be viewing an object currently displayed in the field-of-view image, and if the movement of the field-of-view image is equal to or greater than the actual rotation of the head in such a state, it may make the user feel unnatural or uncomfortable. Therefore, in the case where the rotation direction of the head and the moving direction of the line of sight are in opposite directions, the movement of the field-of-view image is made equal to or greater than the actual rotation of the head only when the difference between the rotation of the head and the movement of the line of sight (the line-of-sight movement angle described later) changes by more than a threshold value within a certain time.

[0236] <Definition of the line-of-sight movement angle and the head rotation angle>

[0237] Figure 20 is a diagram for describing the definition of the line-of-sight movement angle representing the movement amount of the user's line of sight, and Figure 20 shows the state of viewing the user's field of view provided by the display device 22 from above. As Figure 20 shown, the angle from the field-of-view middle representing the middle of the field of view to the line-of-sight direction after the line of sight moves is defined as the line-of-sight movement angle representing the rotation angle as the movement amount of the user's line of sight. In addition, for example, the angle from the field-of-view middle to the left edge of the field of view is defined as the positive line-of-sight movement angle, and the angle from the field-of-view middle to the right edge of the field of view is defined as the negative line-of-sight movement angle.

[0238] Figure 21This is a diagram for defining the head rotation angle that represents the amount of rotation of the user's head, and Figure 21 shows a situation where the user is viewing a spherical image as the content image 200 from above. The head rotation angle is defined such that the predetermined direction indicated by Figure 21 A of is 0 degrees with respect to the current facing direction of the user's head. Additionally, for example, the angle from the predetermined direction to the left is defined as the positive head rotation angle, and the angle from the predetermined direction to the right is defined as the negative head rotation angle.

[0239] Note that in Figure 21 A, the head rotation angle is 0 degrees, and the line-of-sight movement angle is a positive value. In Figure 21 B, the head rotation angle is a positive value, and the line-of-sight movement angle is a positive value greater than the Figure 21 line-of-sight movement angle in A. In Figure 21 C, the head rotation angle is a positive value, and the line-of-sight movement angle is a negative value.

[0240] <Field-of-View Image Movement Processing Based on the Amount of Movement of the Line of Sight and the Amount of Rotation of the Head>

[0241] Next, field-of-view image movement processing based on the amount of movement of the line of sight and the amount of rotation of the head will be described.

[0242] Figure 22 This is a flowchart of the field-of-view image movement processing for moving the field-of-view image based on the amount of movement of the line of sight and the amount of rotation of the head. The field-of-view image movement processing is executed while viewing the content.

[0243] In step S21, the head movement detection unit 72 of the display device 22 ( Figure 3 ) starts acquiring the head rotation angle of the user. The acquired head rotation angle of the user is notified to the field-of-view image determination unit 53 of the information processing device 21 ( Figure 2 ). In step S22, the line-of-sight detection unit 71 of the display device 22 detects the line of sight of the user and starts acquiring the line-of-sight position 195 ( Figure 18 ) and the line-of-sight movement angle of the user. The line-of-sight position 195 and the line-of-sight movement angle are notified to the field-of-view image determination unit 53 of the information processing device 21.

[0244] In step S23, the field-of-view image determination unit 53 determines whether the line of sight of the user has approached the edge of the field-of-view image based on the notification from the line-of-sight detection unit 71, and when it is determined that the line of sight of the user has approached the edge of the field-of-view image, the process proceeds to step S24. In step S24, the field-of-view image determination unit 53 moves the field-of-view image from the current position in the content image in the same direction as the movement direction of the line of sight. Accordingly, the field-of-view image displayed on the display unit 73 moves in the line-of-sight direction.

[0245] Note that, when the field-of-view image determination unit 53 in step S23 does not determine that the user's line of sight has approached the edge of the field-of-view image, step S24 is skipped, and the process proceeds to step S25.

[0246] In step S25, the field-of-view image determination unit 53 determines whether the combined angle of the user's head rotation angle and line-of-sight movement angle within a certain time is equal to or greater than a threshold value. When the field-of-view image determination unit 53 determines that the combined angle of the user's head rotation angle and line-of-sight movement angle within a certain time is equal to or greater than the threshold value, that is, for example, when the user's head and line of sight move in the same direction at a speed equal to or greater than a certain speed, or when although the user's head and line of sight move in opposite directions, one of the movement (rotation) of the user's head and the movement (rotation) of the line of sight is sufficiently greater than the other, the process proceeds to step S26. In step S26, the field-of-view image determination unit 53 moves (rotates) the field-of-view image from the current position in the content image by more than the head rotation angle. Note that any method can be used to calculate the movement amount (rotation angle) of the field-of-view image when moving the field-of-view image by more than the head rotation angle. Then, the process returns to step S23, and the subsequent processing is repeated.

[0247] On the other hand, when the field-of-view image determination unit 53 determines in step S25 that the combined angle of the user's head rotation angle and line-of-sight movement angle within a certain time is less than the threshold value, the process proceeds to step S27. In step S27, the field-of-view image determination unit 53 moves the field-of-view image from the current position in the content image according to the head rotation angle. Then, the process returns to step S23, and the subsequent processing is repeated.

[0248] According to the above-described field-of-view image movement processing, when the user moves the line of sight to the edge of the field-of-view image, it can be assumed that the user is willing to view the outside (outside the scene) of the current field-of-view image (the scene in the field-of-view image). Therefore, the field-of-view image is moved in the direction in which the user moves the line of sight.

[0249] In addition, when the combined angle of the user's head rotation angle and line-of-sight movement angle within a certain time is equal to or greater than the threshold value, it may be difficult to make the user notice a large movement of the field of view. Therefore, the field-of-view image is moved by more than the head rotation angle. Thus, the user can, for example, rotate the head to a certain extent without rotating the body to move the field of view to the directly rear. Therefore, the field of view can be easily moved when viewing the content without increasing the physical load.

[0250] <8. Selection of the avatar to be displayed>

[0251] When the content is a free viewpoint image while multiple users are viewing the same content, each user can arbitrarily change the user's viewpoint. Therefore, another user (the viewpoint of another user) can exist in the user's field of view (the field of view image viewed by the user). In such a case, a character such as an avatar that is a model of another user can be displayed (superimposed) at the position (the viewpoint of another user) of the other user in the field of view image (the content space in the field of view image). The avatar can be a drawing or a photograph. In addition, the avatar can be displayed two-dimensionally or three-dimensionally.

[0252] Displaying the avatar of another user on the field of view image can be expected to have the beneficial effect that users can communicate more smoothly.

[0253] However, if there are multiple other users in the field of view image and all the avatars corresponding to the multiple other users are displayed, there may be troubles, for example, the field of view image is occupied by the avatars of other users (the avatars corresponding to other users), or it becomes difficult to find a communication partner (avatar).

[0254] Therefore, in the case of displaying an avatar, a mechanism for selecting the avatar to be displayed on the field of view image can be implemented.

[0255] In the user device 20 of the present embodiment, the display control unit 60 of the information processing device 21 ( Figure 2 ) selects the avatar to be displayed (superimposed) on the field of view image.

[0256] Figure 23 It is a plan view for describing the method of selecting the avatar to be displayed on the field of view image in the display control unit 60. Figure 23 It shows the situation of viewing the content space 301 from above in the case where multiple users are viewing content with the same free viewpoint image.

[0257] The content space 301 includes an avatar 311A corresponding to user A, an avatar 311B corresponding to user B, an avatar 311C corresponding to user C, and an avatar 311D corresponding to user D. Figure 23 It shows a range 312A in the field of view image of user A, a range 312B in the field of view image of user B, a range 312C in the field of view image of user C, and a range 312D in the field of view image of user D.

[0258] The display control unit 60 of the information processing device 21 of the user device 20 can be in descending order of the possibility of users viewing the same thing (that is, for example, according to the overlapping area of the ranges in the field of view image (here, it is Figure 23Set the priority of other users (avatars of other users) existing among other users (avatars) in the field-of-view image of the user in descending order) of the viewing area), and display the avatars of other users according to the priority.

[0259] For example, the display control unit 60 of the information processing device 21 of the user device 20 used by user A designates other users (in the case of Figure 23 users B, C, and D) existing in the range 312A in the field-of-view image of user A, and detects the overlapping area between the range 312A in the field-of-view image of user A and the ranges in the field-of-view images of other users. Specifically, the display control unit 60 detects the area of each of the following: the overlapping range 315AB between the range 312A and the range 312B in the field-of-view image of user B, the overlapping range 315AC between the range 312A and the range 312C in the field-of-view image of user C, and the overlapping range 315AD between the range 312A and the range 312D in the field-of-view image of user D.

[0260] In Figure 23 the case where, the area of the overlapping range 315AD is the largest, followed by the overlapping ranges 315AB and 315AC. Therefore, the priority for displaying the avatars is determined in the order of user D, user B, and user C. In addition, the avatars are displayed based on the priority according to the number of avatars of other users displayed in the field-of-view image of user A. Note that the number of avatars of other users displayed in the field-of-view image is predetermined, and the user can change the number of avatars.

[0261] Figure 24 is a diagram corresponding to the state shown in Figure 23 , which shows the field-of-view image of user A displayed on the display device 22 of the user device 20 used by user A and a display example of the avatars corresponding to other users displayed in this field-of-view image. Here, Figure 24 corresponds to the case where the number of avatars of other users displayed in the field-of-view image is 1.

[0262] As Figure 24 shown, the avatar 311D corresponding to user D indicated by the solid line is superimposed and displayed on the field-of-view image 321A of user A. Note that the avatar 311B corresponding to user B and the avatar 311 corresponding to user C indicated by the dashed line are not displayed. However, in the case where the number of avatars of other users displayed in the field-of-view image is 2, the avatars 311D and 311B are displayed. In the case where the number of avatars of other users displayed in the field-of-view image is 3 or more, the avatars 311D, 311B, and 311C are displayed.

[0263] Note that here, the priority of the avatar to be displayed in the field-of-view image of user A is set according to the overlapping area between the range in the field-of-view image of user A and the range in the field-of-view image of another user. Therefore, the avatar of another user with a field of view close to the field-of-view image (range in the field-of-view image) of user A tends to be preferentially displayed in the field-of-view image of user A.

[0264] In addition to setting the priority of displaying the avatar in the field-of-view image of user A according to the proximity of the field of view of another user to the field-of-view image of user A as described above, the priority can also be set according to the positional relationship between the field-of-view image (range in the field-of-view image) of user A and the field of view of another user. In this way, whether to display (overlay) the avatar of another user on the field-of-view image of user A can be controlled according to the positional relationship between the field-of-view image of user A and the field of view of another user.

[0265] <Avatar display process>

[0266] Next, Figure 25 is a flowchart of the avatar display process of setting the priority of the avatar as described above to display the avatar.

[0267] In step S31, the display control unit 60 reads the field-of-view information of the user and the field-of-view information of other users held in the field-of-view information holding unit 55. In step S32, the display control unit 60 refers to the field-of-view information of the user and other users to determine whether the field-of-view image of the user (that is, the field-of-view image of the user determined according to the field-of-view information of the user (range in the field-of-view image)) includes more than a predetermined number of other users (viewpoints of other users). The predetermined number corresponds to the preset number of avatars of other users to be displayed in the field-of-view image of the user.

[0268] In step S32, when the display control unit 60 determines that the field-of-view image of the user includes more than a predetermined number of other users (viewpoints of other users), the process proceeds to step S33. In step S33, the display control unit 60 refers to the field-of-view information of the user and other users, and sets the priority of the avatars of other users in descending order of the size of the overlapping range between the field of view (range in the field-of-view image) and the field-of-view image (range in the field-of-view image) of the user.

[0269] In step S34, the display control unit 60 determines the predetermined number of avatars of other users (avatars corresponding to other users) to be displayed in the field-of-view image of the user according to the set priority, and superimposes the determined avatars of other users on the field-of-view image. In this way, a predetermined number of avatars are superimposed and displayed on the field-of-view image displayed on the display unit 73.

[0270] On the other hand, in the case where the display control unit 60 determines in step S32 that the user's field-of-view image does not include more than a predetermined number of other users (viewpoints of other users), the avatars of all users present in the user's field-of-view image will be displayed, and the process proceeds to step S35.

[0271] In step S35, the display control unit 60 determines that the avatars of all other users present in the user's field-of-view image are the avatars to be displayed in the field-of-view image, and superimposes the determined avatars of other users on the field-of-view image. In this way, equal to or less than the predetermined number of avatars are superimposed and displayed on the field-of-view image displayed on the display unit 73.

[0272] According to the above avatar display process, the user can only see other avatars that are considered to be viewing the same or similar field-of-view images, and the user cannot see other avatars. This can suppress problems in communication such as miscommunication with other users corresponding to the avatars that can be viewed. This can also suppress the situation where the user's field of view is occupied by avatars.

[0273] In addition, the user can arbitrarily set the priority regarding the display of avatars corresponding to other users. In addition, the priority can be set according to the communication history. For example, the more the number of message exchanges, the higher the priority. In addition, the priority set based on these various criteria can be switched.

[0274] In addition, in the case where the content is time-varying content, the priority can be set not only by considering the area of the overlapping range of the field-of-view image (the range in the field-of-view image), but also by considering the overlapping time of the field-of-view images.

[0275] Note that the avatar display process can also be applied to the case where an icon of the user (an icon corresponding to the user), a real scene image of the user, a character (string) representing the user (an image of the character), or other user images representing the user are displayed instead of the avatar.

[0276] In addition, the icon can be displayed first, and the display can be changed to display only the outline of the avatar, monochromatic display of the avatar, full-color display of the avatar, etc. according to the closeness of the relationship between users or the degree of communication.

[0277] In addition, mutual authentication can also be performed between users before displaying the icon or avatar.

[0278] Note that for the content of either the free viewpoint image or the spherical image, various methods can also be used to control whether to display (superimpose) user images such as avatars of other users on the user's field-of-view image.

[0279] For example, in the user's field-of-view image, avatars of other users with a field of view close to that of the user, i.e., other users with similar field-of-view images, can be displayed at a position close to the position of the user in the user's field-of-view image.

[0280] In addition, based on the proximity of the search positions (temporal positions of the reproduced content) of the same content viewed by the user and other users, the proximity of the viewpoints of the user and other users in the case where the content is a free-viewpoint image such as a 3D game, the history of past conversations between the user and other users, the proximity of relationships in the SNS, etc., avatars of other users who are likely to interact with the user can be preferentially displayed on the user's field-of-view image.

[0281] Furthermore, for each user's avatar, user information about the user, such as the content of the user's conversation (text chat) and the user's nickname, can be displayed together with the avatar. In the case where avatars of other users can be displayed on the user's field-of-view image, the display of avatars of other users to be displayed in the user's field-of-view image can be restricted according to the relationship between the user and other users, etc.

[0282] For example, when the avatar of another user (the position of the avatar) is in the user's field of view, but the user's avatar is not in the field of view of the other user, details of the display related to the avatar of the other user (i.e., for example, the angle of the neck and the expression of the avatar of the other user, the text of the conversation, etc.) may not be displayed in the user's field-of-view image, and a simple avatar (i.e., for example, only the silhouette (external form) of the avatar) can be displayed. In addition, a dialog box for requesting communication permission can be displayed in the field-of-view image of each of the user and the other user, and when communication is permitted for each of the user and the other user, a display related to the avatar of the partner can be displayed in the field-of-view image. This can prevent the user from peeping at the communication of other users. Note that, in addition, only the icon of the partner can be displayed before communication is permitted for each of the user and the other user, and when communication is permitted for each of the user and the other user, the avatar of the partner or a live silhouette can be displayed.

[0283] Furthermore, when avatars of other users are displayed in the user's field-of-view image, 3D images, etc. can be used as avatars of other users. The field-of-view image can be displayed such that the field-of-view image moves around the avatar of the other user according to the movement of the user's neck or body, and this allows the user to feel the real presence of the other user.

[0284] In addition, when the avatars of other users with a field-of-view image close to that of the user, i.e., other users with similar field-of-view images, are displayed in the user's field-of-view image, when the other users turn their necks to move away in the user's field-of-view image, the avatars of the other users can be moved to a position farther from the user in the user's field-of-view image, or the avatar of another user (a user different from the other users) with a field of view close to that of the user can be displayed in the user's field-of-view image instead of the avatar of the other users.

[0285] In addition, the display related to the avatars of other users to be displayed in the user's field-of-view image can be changed according to the relationship between the user and the other users. For example, partial information (less information) related to the avatars of other users can be displayed in the field-of-view image of a user who has ever talked with the other users and registered the other users as friends. All displays (more information) related to the avatars of other users can be displayed in the field-of-view image of a user who is talking with the other users.

[0286] In addition, in a case where the user can perform a zoom-in or zoom-out operation to adjust the viewing angle (the range of the scene in the image) of the image displayed on the display device 22, the priority of displaying the avatars of other users in the user's field-of-view image can also be set in consideration of whether the viewing angle of the image displayed on the user's display device 22 is similar to the viewing angle of the image displayed on the display device 22 of the other users.

[0287] In addition, in a case where the content includes a plurality of spherical images captured by a plurality of spherical cameras, the priority of displaying the avatars of other users in the user's field-of-view image can also be set in consideration of whether the user and the other users are viewing the spherical images captured by the same spherical camera, or whether the user and the other users are viewing the spherical images captured by different spherical cameras.

[0288] In addition, in a case where the content is a free viewpoint image, the proximity of the fields of view of the user and the other users can be determined based on, for example, the angles of the necks of the user and the other users, whether the same substance (object) exists in the field-of-view images of the user and the other users, etc. Whether the same substance exists in the field-of-view images of the user and the other users can be determined by, for example, the combination of the positions (viewpoints) of the user and the other users and the viewing angles of the field-of-view images of the user and the other users.

[0289] In addition, regarding the display of avatars, a mode of displaying the avatars of other specific users and a mode of preferentially displaying the avatars of other users who have talked with the user in the past can be provided in the user's field-of-view image, and the mode of displaying avatars can be switched to this mode according to the user's operation.

[0290] In this way, an avatar can be displayed to allow a large number of unspecified users to view the same or similar scenes in the same content to connect with each other, and the conversation can proceed smoothly. In addition, priorities can be set for the avatar (other users) according to the proximity of the field of view, permission for communication, etc., and the avatar can be displayed according to the priorities. This can prevent a large number of avatars that obstruct the viewing of the content from covering the entire field of view image.

[0291] <9. A series of processes executed by software>

[0292] A series of processes can be executed by hardware or can be executed by software. In the case where a series of processes are executed by software, the program included in the software is installed on a computer. Here, examples of the computer include a computer incorporated in dedicated hardware that can execute various functions by installing various programs, and a general-purpose personal computer, etc.

[0293] Figure 26 is a block diagram showing a configuration example of the hardware of a computer that executes a series of processes using a program.

[0294] In a computer 500, a CPU (Central Processing Unit) 501, a ROM (Read Only Memory) 502, and a RAM (Random Access Memory) 503 are connected to each other via a bus 504.

[0295] An input / output interface 505 is also connected to the bus 504. An input unit 506, an output unit 507, a storage unit 508, a communication unit 509, and a drive 510 are connected to the input / output interface 505.

[0296] The input unit 506 includes a keyboard, a mouse, a microphone, etc. The output unit 507 includes a display, a speaker, etc. The storage unit 508 includes a hard disk, a non-volatile memory, etc. The communication unit 509 includes a network interface, etc. The drive 510 drives a removable medium 511, such as a magnetic disk, an optical disk, a magneto-optical disk, and a semiconductor memory.

[0297] In a computer configured in this way, the CPU 501 loads, for example, a program stored in the storage unit 208 onto the RAM 203 via the input / output interface 205 and the bus 204, and executes the program to execute a series of processes.

[0298] The program executed by the computer (CPU 501) can be provided by, for example, recording the program on a removable medium 511 such as a packaged medium. In addition, the program can be provided via a wired or wireless transmission medium such as a local area network, the Internet, and digital satellite broadcasting.

[0299] In the computer 500, a removable medium 511 can be installed on the drive 510 to install a program on the storage unit 508 through the input / output interface 505. Additionally, the communication unit 509 can receive a program through a wired or wireless transmission medium to install the program on the storage unit 508. Further, a program can be pre-installed on the ROM 502 or the storage unit 508.

[0300] Note that the program executed by the computer 500 can be a program that executes processing in the chronological order described in this specification, or the program can be a program for executing processing in parallel or for executing processing at a necessary time, such as when a call processing is made.

[0301] Note that the beneficial effects described in this specification are merely illustrative, and the beneficial effects are not limited thereto. There can also be other advantageous effects.

[0302] Embodiments of the present technology are not limited to the above-described embodiments, and various changes can be made without departing from the scope of the present technology.

[0303] <1>

[0304] An information processing apparatus, comprising:

[0305] A content acquisition unit configured to acquire data of content;

[0306] An image cutting unit configured to cut out a first field-of-view image corresponding to the field of view of a first user from a content image based on the data of the content;

[0307] A field-of-view information acquisition unit configured to acquire field-of-view information indicating the field of view of a second user who views the content image; and

[0308] A display control unit configured to control a display device to display the first field-of-view image, and configured to control the display device to display the field of view of the second user based on the field-of-view information of the second user.

[0309] <2>

[0310] The information processing apparatus according to <1>, wherein

[0311] The display control unit is configured to control the display device to display a field-of-view position indication image indicating the position of the field of view of the second user based on the field-of-view information of the second user.

[0312] <3>

[0313] The information processing apparatus according to <2>, wherein

[0314] The field of view position indication image includes a wide-area image, and the wide-area image includes the first field of view image and a second field of view image corresponding to the field of view of the second user.

[0315] <4>

[0316] The information processing apparatus according to <3>, wherein

[0317] The display control unit is configured to control the display device to switch between the first field of view image and the wide-area image in response to a predetermined trigger.

[0318] <5>

[0319] The information processing apparatus according to <4>, wherein

[0320] The predetermined trigger includes at least one of a key operation, a voice command, a movement of the head, or a gesture operation performed by the first user.

[0321] <6>

[0322] The information processing apparatus according to any one of <3> to <5>, wherein

[0323] The wide-area image includes at least one of an equidistant cylindrical projection, a Mercator projection, an aerial view, an overhead view, or a two-dimensional map.

[0324] <7>

[0325] The information processing apparatus according to any one of <2> to <6>, wherein

[0326] The field of view position indication image includes a symbol image superimposed on the first field of view image, and the symbol image indicates the position of the field of view of the second user.

[0327] <8>

[0328] The information processing apparatus according to any one of <1> to <7>, further comprising:

[0329] A field of view image determination unit configured to determine the first field of view image to be cut out from the content image based on at least one of the movement of the line of sight of the first user or the movement of the head of the first user.

[0330] <9>

[0331] The information processing apparatus according to <8>, wherein

[0332] The field of view image determination unit is configured to move the first field of view image based on the line of sight of the first user in response to the line of sight of the first user approaching the edge of the first field of view image.

[0333] <10>

[0334] The information processing apparatus according to <8> or <9>, wherein

[0335] The field-of-view image determination unit is configured to make the rotation angle of the first field-of-view image greater than the rotation angle of the head of the first user based on the rotation angle of the line of sight of the first user and the rotation angle of the head of the first user.

[0336] <11>

[0337] The information processing apparatus according to any one of <8> to <10>, wherein

[0338] The field-of-view image determination unit is configured to determine an initial position of the first field-of-view image of the first user corresponding to a timing when the first user substantially starts viewing the content based on the field-of-view information of the second user.

[0339] <12>

[0340] The information processing apparatus according to any one of <8> to <11>, wherein

[0341] The display device includes a head-mounted display, and

[0342] The field-of-view image determination unit is configured to determine the first field-of-view image based on at least one of the movement of the head of the first user associated with the head-mounted display or the movement of the line of sight of the first user.

[0343] <13>

[0344] The information processing apparatus according to any one of <1> to <12>, wherein

[0345] The second user includes a plurality of users,

[0346] The display control unit is configured to control the display device to superimpose at least one of a plurality of user images corresponding to the plurality of users on the first field-of-view image, and

[0347] The display control unit is configured to control whether to superimpose each user image among the plurality of user images on the first field-of-view image according to a positional relationship between the first field-of-view image and the field of view of each user among the plurality of users.

[0348] <14>

[0349] The information processing apparatus according to <13>, wherein

[0350] The display control unit is configured to control the display device to preferentially superimpose, on the first field-of-view image, a user image corresponding to a user among the plurality of users who has a field of view relatively close to the field of view of the first field-of-view image.

[0351] <15>

[0352] The information processing apparatus according to <13> or <14>, wherein

[0353] The display control unit is configured to control the display device to preferentially superimpose, on the first field-of-view image, a part of the plurality of user images according to a communication history between the first user and the plurality of users.

[0354] <16>

[0355] The information processing apparatus according to any one of <1> to <15>, wherein

[0356] The content image includes a spherical image or a free viewpoint image.

[0357] <17>

[0358] The information processing apparatus according to any one of <1> to <16>, further comprising:

[0359] The display device.

[0360] <18>

[0361] An information processing method, comprising:

[0362] Obtaining data of content;

[0363] Cutting out a first field-of-view image corresponding to the field of view of a first user from a content image based on the data of the content;

[0364] Obtaining field-of-view information indicating the field of view of a second user who views the content image; and

[0365] Controlling a display device to display the first field-of-view image, and controlling the display device to display the field of view of the second user based on the field-of-view information of the second user.

[0366] <19>

[0367] A program for causing a computer to function as the following units:

[0368] A content acquisition unit configured to acquire data of content;

[0369] An image cutting unit configured to cut out a first field-of-view image corresponding to the field of view of a first user from a content image of content-based data;

[0370] A field-of-view information acquisition unit configured to acquire field-of-view information representing the field of view of a second user viewing the content image; and

[0371] A display control unit configured to control a display device to display the first field-of-view image and configured to control the display device to display the field of view of the second user based on the field-of-view information of the second user.

[0372] List of reference numerals

[0373] 10 Content viewing system, 20 User device, 21 Information processing device, 22 Display device, 31 Internet, 40 Server device, 41 Content distribution unit, 42 Field-of-view information management unit, 43 Communication management unit, 51 Communication unit, 52 Content holding unit, 53 Field-of-view image determination unit, 54 Image cutting unit, 55 Field-of-view information holding unit, 58 Input unit, 59 Trigger detection unit, 60 Display control unit, 71 Line-of-sight detection unit, 72 Head movement detection unit, 73 Display unit, 74 Voice input / output unit, 100 Entire image, 101 Field-of-view range, 111 Field-of-view image, 112 Wide-area image, 113 Field-of-view range display, 121 Field-of-view direction indication marker, 131 Field-of-view direction indication line, 132 Icon, 141 Field-of-view range display, 151 Tracking image, 162 Icon, 163 Arrow, 200 Content image, 301 Content space, 311 Avatar, 500 Computer, 501 CPU.

Claims

1. An information processing apparatus, comprising: a circuit system configured to: acquire first field-of-view information representing a first field of view of a first user in a wide-area image; acquire second field-of-view information representing a second field of view of a second user in the wide-area image; and control a display device to: display a first field-of-view image for the first user based on the first field-of-view information; display the wide-area image for the first user based on detection of a specific trigger; display a field-of-view indication image based on display of the first field-of-view image, wherein the field-of-view indication image indicates a position of the second field of view on the first field-of-view image; and display a field-of-view range based on display of the wide-area image, wherein the field-of-view range indicates a range of the second field of view on the wide-area image.

2. The information processing apparatus according to claim 1, wherein the field-of-view indication image includes a symbol image superimposed on the first field-of-view image, and the symbol image indicates the position of the second field of view of the second user.

3. The information processing apparatus according to claim 1, wherein the circuit system is further configured to control the display device to simultaneously display the first field-of-view image and the wide-area image for the first user.

4. The information processing apparatus according to claim 1, wherein the specific trigger includes at least one of a key operation, a voice command, a movement of the head, or a gesture operation performed by the first user.

5. The information processing apparatus according to claim 1, wherein the wide-area image includes at least one of an equidistant cylindrical projection, a Mercator projection, an aerial view, an overhead view, or a two-dimensional map.

6. The information processing apparatus according to claim 1, wherein the circuit system is further configured to determine the first field-of-view image to be cut out from a content image, and cut out the first field-of-view image from the content image based on at least one of a movement of the first user's line of sight or a movement of the first user's head.

7. The information processing apparatus according to claim 6, wherein movement of the first field-of-view image is based on the first user's line of sight and the first user's line of sight approaching an edge of the first field-of-view image.

8. The information processing apparatus according to claim 6, wherein based on a rotation angle of the first user's line of sight and a rotation angle of the first user's head, a rotation angle of the first field-of-view image is greater than the rotation angle of the first user's head.

9. The information processing apparatus according to claim 6, wherein the display device includes a head-mounted display, and the circuit system is further configured to determine the first field-of-view image based on at least one of a movement of the first user's head or a movement of the first user's line of sight associated with the head-mounted display.

10. The information processing apparatus according to claim 6, wherein the content image includes one of a spherical image or a free viewpoint image.

11. The information processing apparatus according to claim 1, wherein the second user corresponds to a plurality of users, and the circuit system is further configured to: Control the display device to superimpose at least one of a plurality of user images corresponding to the plurality of users on the first field of view image; and Based on the positional relationship between the first field of view image and the field of view of each user among the plurality of users, control the superimposition of each user image among the plurality of user images on the first field of view image.

12. The information processing apparatus according to claim 11, wherein, The circuit system is further configured to control the display device to superimpose a specific user image among the plurality of user images on the first field of view image.

13. The information processing apparatus according to claim 12, wherein, Among the plurality of user images, the field of view of the specific user image is relatively close to the first field of view image.

14. The information processing apparatus according to claim 11, wherein, The circuit system is further configured to control the display device to superimpose a part of the plurality of user images on the first field of view image based on the communication history between the first user and the plurality of users.

15. An information processing method, comprising: Obtain first field of view information representing the first field of view of a first user in a wide area image; Obtain second field of view information representing the second field of view of a second user in the wide area image; and Control a display device to: Display a first field of view image to the first user based on the first field of view information; Display the wide area image to the first user based on the detection of a specific trigger; Display a field of view indication image based on the display of the first field of view image, wherein the field of view indication image indicates the position of the second field of view on the first field of view image; and Display a field of view range based on the display of the wide area image, wherein the field of view range indicates the range of the second field of view on the wide area image.

16. A non-transitory computer-readable medium storing computer-executable instructions that, when executed by a computer, cause the computer to perform operations, the operations comprising: Obtain first field of view information representing the first field of view of a first user in a wide area image; Obtain second field of view information representing the second field of view of a second user in the wide area image; and Control a display device to: Display a first field of view image to the first user based on the first field of view information; Display the wide area image to the first user based on the detection of a specific trigger; Display a field of view indication image based on the display of the first field of view image, wherein the field of view indication image indicates the position of the second field of view on the first field of view image; and Display a field of view range based on the display of the wide area image, wherein the field of view range indicates the range of the second field of view on the wide area image.

Images