Information processing device provided with 360-degree image viewpoint position recognition unit

Abstract

An information processing device is provided with: a 360-degree image processing unit that generates a 360-degree image on the basis of a plurality of images captured by a plurality of imaging devices worn by a first user; and a 360-degree image viewpoint position recognition unit that recognizes viewpoint position coordinates of the first user in the 360-degree image.

Description

Technical Field

[0001] The present technology relates to an information processing device, a display device, and an image sharing system. Background Art

[0002] Recently, an image sharing system has been proposed that enables sharing of a 360-degree image covering the entire surrounding environment captured by a wearable camera device under a first-person viewpoint to allow other people to virtually experience the same things that the person has experienced. In such an image sharing system, a 360-degree image covering the surrounding environment of a user wearing the camera device is transmitted in real time to a device such as a head-tracking head-mounted display (HMD) or a screen used by other people to allow other people to freely explore and observe the 360-degree image and communicate with the user wearing the camera device.

[0003] Such an image sharing system has a feature of allowing others to share 360-degree images captured from the first-person viewpoint of the user wearing the camera device, but there is a problem that by simply sharing the 360-degree images, others cannot know the direction the user wearing the camera device is facing or what the user is looking at.

[0004] Therefore, a technology has been proposed in which a mark or the like indicating the field of view of the user wearing the imaging device is superimposed on a 360-degree image to convey the direction in which the user wearing the imaging device is looking.

[0005] Citation List

[0006] Patent Literature

[0007] Patent Document 1: Japanese Patent Application Publication No. 2021-170341 Summary of the invention

[0008] Problems to be solved by the present invention

[0009] However, there is an unresolved problem that the mark indicating the field of view disclosed in Patent Document 1 can convey the direction in which the user wearing the camera device is looking, but cannot convey to other people where the user is focusing. In particular, for capturing an image of a scene that conveys the skill of the user wearing the camera device, it is important to convey to other people where the user wearing the camera device is focusing.

[0010] The present technology was completed in view of such a situation, and therefore an object of the present technology is to provide an information processing device, a display device, and an image sharing system that enable sharing of the viewpoint position in a 360-degree image of a user wearing a camera device with other users.

[0011] Solution to the problem

[0012] In order to solve the problem described above, the first technology is an information processing device, which includes: a 360-degree image processing unit, which generates a 360-degree image based on multiple captured images captured by multiple camera devices worn by a first user; and a 360-degree image viewpoint position identification unit, which identifies the viewpoint position coordinates of the first user in the 360-degree image.

[0013] In addition, the second technology is a display device that displays a 360-degree image generated based on multiple captured images captured by multiple camera devices worn by a first user and the viewpoint position coordinates of the first user identified in the 360-degree image, so as to present the 360-degree image and the viewpoint position coordinates to a second user different from the first user.

[0014] In addition, the third technology is an image sharing system, which includes: an information processing device, the information processing device includes a 360-degree image processing unit and a 360-degree image viewpoint position identification unit, the 360-degree image processing unit generates a 360-degree image based on multiple captured images captured by multiple camera devices worn by a first user, and the 360-degree image viewpoint position identification unit identifies the viewpoint position coordinates of the first user in the 360-degree image; and a display device, the display device displays the 360-degree image and the viewpoint position coordinates of the first user identified in the 360-degree image to present the 360-degree image and the viewpoint position coordinates to a second user different from the first user. BRIEF DESCRIPTION OF THE DRAWINGS

[0015] Figure 1 is a block diagram showing the configuration of the image sharing system 10 .

[0016] Figure 2 is a block diagram showing the configuration of the main body side device 100 according to the first embodiment.

[0017] Figure 3 1 is a diagram showing how the viewpoint detection camera 101 , the front view camera 102 , and the 360-degree camera 103 are installed according to the first embodiment.

[0018] Figure 4 2 is a block diagram showing the configuration of the image-side device 200 .

[0019] Figure 5 is a flowchart showing processing executed by the information processing device 150 according to the first embodiment.

[0020] Figure 6 A is a diagram showing viewpoint position coordinates in the left-eye image and the right-eye image, Figure 6 B is a diagram showing the viewpoint position coordinates in the front view image, and Figure 6 C is a diagram showing viewpoint position coordinates in a 360-degree image.

[0021] Figure 7 is a diagram for describing coordinate transformation of viewpoint position coordinates according to the first embodiment.

[0022] Figure 8 is a graph used to describe calibration.

[0023] Fig. 9 is a block diagram showing the configuration of a main body-side device 100 for calibration.

[0024] Fig.10 is a graph used to describe calibration.

[0025] Fig.11 is a diagram for describing a method for determining whether the number of viewpoint positions having a correspondence relationship established by calibration is sufficient.

[0026] Fig.12 is a block diagram showing the configuration of the subject-side device 100 for sharing viewpoint position coordinates.

[0027] Fig.13 2 is a block diagram showing the configuration of the imaging side device 200 for sharing viewpoint position coordinates.

[0028] Fig.14 is a flow chart illustrating a process for determining whether to synchronize fields of view.

[0029] Fig.15 is a diagram for describing a first method for guiding a field of view for synchronization.

[0030] Fig.16 is a diagram for describing a second method for guiding a field of view for synchronization.

[0031] Fig.17 is a diagram for describing a third method for guiding a field of view for synchronization.

[0032] Fig.18 : is a flowchart showing a process for generating viewpoint position coordinates as subject-side meta information based on the viewpoint stay time.

[0033] Fig.19 2 is a diagram showing an example of how to display an icon indicating the viewpoint position coordinates of the video-side device 200.

[0034] Fig. 20 is a diagram for describing an example of how to change an icon indicating viewpoint position coordinates based on the viewpoint stay time.

[0035] Fig.21 : is a flowchart showing a process for generating viewpoint position coordinates as subject-side meta information based on the viewpoint stay time and utterance content.

[0036] Fig. 22 2 is a diagram showing an example of how to display an icon indicating the viewpoint position coordinates of the video-side device 200 and the utterance content.

[0037] Fig.23 : is a diagram showing an example of how to display the viewpoint position coordinates of a plurality of image-side users.

[0038] Fig.24 2 is a block diagram showing the configuration of a main body-side device 100 according to the second embodiment.

[0039] Fig.25 1 is a diagram showing how the viewpoint detection camera 101 , the front view camera 102 , and the 360-degree camera 103 are installed according to the second embodiment.

[0040] Fig.26 is a flowchart showing processing executed by the information processing device 150 according to the second embodiment.

[0041] Fig. 27 is a diagram for describing coordinate transformation of viewpoint position coordinates according to the second embodiment.

[0042] Fig.28 is a diagram showing another example of the 360-degree imaging device 103 according to the second embodiment.

[0043] Fig.29 is a block diagram showing the configuration of a main body-side device 100 according to the third embodiment.

[0044] Fig.30 1 is a diagram showing how the viewpoint detection camera 101 , the front view camera 102 , and the 360-degree camera 103 are installed according to the third embodiment.

[0045] Fig.31 is a diagram for describing coordinate transformation of viewpoint position coordinates according to the third embodiment.

[0046] Fig.32 is a diagram showing another example of the 360-degree imaging device 103 according to the third embodiment.

[0047] Fig.33 is a block diagram showing a modification of the image sharing system 10 .

[0048] Fig.34 is a diagram showing a modification in the case where the forward-looking camera 102 is not provided.

[0049] Fig.35 1 is a block diagram showing the configuration of the main body side device 100 without the forward-view camera device 102. DETAILED DESCRIPTION

[0050] Hereinafter, embodiments of the present technology will be described with reference to the drawings. Note that the description will be given in the following order.

[0051] <1. First Embodiment>

[0052] [1-1. Configuration of Image Sharing System 10]

[0053] [1-2. Assigning viewpoint position coordinates to 360-degree images]

[0054] [1-2-1. Configuration of the main body side device 100 and the image side device 200]

[0055] [1-2-2. Transformation of viewpoint position coordinates]

[0056] [1-2-3. Calibration]

[0057] [1-3. Sharing of viewpoint position coordinates]

[0058] [1-3-1. Configuration of the main body side device 100 and the image side device 200]

[0059] [1-3-2. Processing to realize sharing of viewpoint position coordinates: field of view synchronization]

[0060] [1-3-3. Processing for Sharing Viewpoint Position Coordinates: Display of Viewpoint Position Coordinates]

[0061] <2. Second Embodiment>

[0062] [2-1. Assigning viewpoint position coordinates to 360-degree images]

[0063] [2-1-1. Configuration of main body side device 100]

[0064] [2-1-2. Transformation of viewpoint position coordinates]

[0065] <3. Third Embodiment>

[0066] [3-1. Assigning viewpoint position coordinates to a 360-degree image]

[0067] [3-1-1. Configuration of main body side device 100]

[0068] [3-1-2. Transformation of viewpoint position coordinates]

[0069] <4. Modification>

[0070] <1. First Embodiment>

[0071] [1-1. Configuration of Image Sharing System 10]

[0072] Will refer to Figure 1 The configuration of the image sharing system 10 is described. The image sharing system 10 includes a subject that provides a 360-degree image captured by a camera device and an image that receives the 360-degree image. The subject-side device is referred to as a subject-side device 100, and a user who uses the subject-side device 100 is referred to as a subject-side user. The subject-side user corresponds to the first user in the Summary of the Invention section. In addition, the image-side device is referred to as an image-side device 200, and a user who uses the image-side device 200 is referred to as an image-side user. The image-side user corresponds to the second user in the Summary of the Invention section, and the image-side device corresponds to the display device in the Summary of the Invention section.

[0073] The subject-side device 100 and the image-side device 200 are connected via a network. The number of image-side devices 200 connected to the subject-side device 100 may be one or more. There is no limitation on the number of image-side devices 200.

[0074] In the image sharing system 10, a 360-degree image generated by capturing an image of a real space using a camera included in the subject-side device 100 is transmitted from the subject-side device 100 to the image-side device 200. Then, the image-side device 200 receives and displays the 360-degree image, thereby allowing the image-side user to view the 360-degree image. In addition, in the image sharing system 10, the subject-side device 100 and the image-side device 200 transmit and receive audio, thereby also allowing the subject-side user and the image-side user to have a conversation. Note that conversation is not an essential requirement in the present technology.

[0075] [1-2. Assigning viewpoint position coordinates to 360-degree images]

[0076] [1-2-1. Configuration of the main body side device 100 and the image side device 200]

[0077] Next, we will refer to Figures 2 to 4 The configurations of the main body-side device 100 and the image-side device 200 according to the first embodiment are described.

[0078] The main body side device 100 includes a viewpoint detection camera device 101, a front view camera device 102, a 360-degree camera device 103, a position and orientation detection unit 104, a viewpoint position detection unit 105, a front view image viewpoint position recognition unit 106, a 360-degree image viewpoint position recognition unit 107, a 360-degree image processing unit 108, a rotation compensation processing unit 109, an audio input unit 110, an audio output unit 111 and a communication unit 112.

[0079] The viewpoint detection camera device 101, the front view camera device 102 and the 360-degree camera device 103 are all wearable camera devices including a lens, an imaging element and an image signal processing circuit. The viewpoint detection camera device 101, the front view camera device 102 and the 360-degree camera device 103 are all worn by the subject side user. Figure 3 As shown in , a viewpoint detection camera 101, a front view camera 102, and a 360-degree camera 103 are mounted on a glasses-like frame and a band, and a subject-side user wears the frame and the band.

[0080] The viewpoint detection camera 101 is a camera that captures images of the subject side user's eyes to detect the viewpoint position coordinates of the subject side user. The viewpoint detection camera 101 includes a right viewpoint detection camera 101R that captures an image of the subject side user's right eye and a left viewpoint detection camera 101L that captures an image of the left eye. In the following description, the right viewpoint detection camera 101R and the left viewpoint detection camera 101L are simply referred to as the viewpoint detection camera 101 unless otherwise distinguished.

[0081] The front view camera 102 is a camera that captures an image of the real space in front of the subject-side user, and is fixedly installed at approximately the center in the width direction of the subject-side user's face to face forward.

[0082] The 360-degree camera 103 is a camera that captures a wide range of images in all directions (i.e., up, down, left, and right) around the subject-side user. The 360-degree camera 103 may also be referred to as an omnidirectional camera or a spherical camera. The 360-degree camera 103 includes an ultra-wide-angle lens capable of capturing a field of view exceeding 180 degrees, including a front camera 103F that captures images in front of the subject-side user and a rear camera 103R that captures images behind the subject-side user, and acquires the front image and the rear image in one shot. The front image and the rear image are output to the 360-degree image viewpoint position recognition unit 107. Figure 3 As shown in FIG. 1A , in the first embodiment, the front camera 103F is installed at approximately the center in the width direction of the face. Figure 3 As shown in FIG. 1B , the rear camera device 103R is installed at approximately the center in the width direction behind the head.

[0083] Then, the 360-degree image processing unit 108 combines the front image and the rear image to form one 360-degree image. Note that the number of cameras constituting the 360-degree camera 103 is not limited to two, and may be any number. In addition, the arrangement of the cameras constituting the 360-degree camera 103 is not limited to a specific arrangement, and the number and arrangement of the cameras may be appropriately set according to the desired coverage area of ​​the image to be captured. Note that in order to suppress distortion of viewpoint detection information, the 360-degree camera 103 is desirably arranged at the same height as the eyes of the subject side user or as close to the height of the eyes as possible.

[0084] The forward-looking camera 102 and the 360-degree camera 103 configured as described above can capture an image of the real space from a position close to the viewpoint of the subject-side user. Note that the viewpoint detection camera 101 and the forward-looking camera 102 do not necessarily need to be mounted together on the frame, and can be configured as separate camera devices, and the subject-side user can wear all the camera devices.

[0085] The position and orientation detection unit 104 includes various sensors that detect the position and orientation of the forward-view camera 102 and the 360-degree camera 103 mounted on the head of the subject-side user. Examples of the sensor include an inertial measurement unit (IMU), an inertial sensor (accelerometer, angular velocity sensor, gyroscope for two-axis or three-axis directions), light detection and ranging, laser imaging detection and ranging (LiDAR), time of flight (ToF) sensor, global navigation satellite system (GNSS), global positioning system (GPS), etc. The position and orientation detection unit 104 outputs the position and orientation information to the 360-degree image viewpoint position recognition unit 107 and the rotation compensation processing unit 109. Note that in the processing performed by the 360-degree image viewpoint position recognition unit 107, the position and orientation information is not necessarily required.

[0086] Note that the position and orientation detection unit 104 can replace the various sensors mentioned above or be combined with the various sensors mentioned above to extract feature points or features from the forward-looking image and the 360-degree image, and detect the position and orientation of the forward-looking camera device 102 and the 360-degree camera device 103 through angle estimation based on the displacement of the feature points or features.

[0087] The viewpoint detection camera 101, the forward-looking camera 102, the 360-degree camera 103, and the position and orientation detection unit 104 are controlled according to a predetermined synchronization signal. As long as the subject-side device 100 transmits a 360-degree image to the image-side device 200, imaging and sensing are continuously performed at a predetermined frequency, and eye images, forward-looking images, front images, rear images, and position and orientation information are output.

[0088] The viewpoint position detection unit 105 detects the viewpoint position coordinates of the right eye of the subject side user in the right eye image and the viewpoint position coordinates of the left eye of the subject side user in the left eye image, both of which are captured by the viewpoint detection camera 101. The viewpoint position detection unit 105 outputs the viewpoint position detection result to the front view image viewpoint position recognition unit 106. For example, the viewpoint position detection unit 105 can detect the viewpoint position coordinates by detecting the pupil from the eye image. In addition, the viewpoint position detection unit 105 can estimate the viewpoint dwell time based on the viewpoint position, pupil movement, etc.

[0089] The front view image viewpoint position recognition unit 106 recognizes viewpoint position coordinates in the front view image based on the viewpoint position coordinates in the eye image detected by the viewpoint position detection unit 105 and the front view image. The front view image viewpoint position recognition unit 106 outputs the viewpoint position recognition result to the 360-degree image viewpoint position recognition unit 107. In the following description, the viewpoint position recognition result from the front view image viewpoint position recognition unit 106 may be referred to as a first viewpoint position recognition result.

[0090] The 360-degree image viewpoint position recognition unit 107 recognizes the viewpoint position coordinates of the subject side user in the front image based on the front image as a part of the 360-degree image, the position and orientation information, and the first viewpoint position recognition result. The 360-degree image viewpoint position recognition unit 107 outputs the front image, the rear image, and the second viewpoint position recognition result to the 360-degree image processing unit 108. In the following description, the viewpoint position recognition result from the 360-degree image viewpoint position recognition unit 107 may be referred to as the second viewpoint position recognition result.

[0091] The 360-degree image processing unit 108 combines the front image captured by the front camera 103F and the rear image captured by the rear camera 103R, and further performs predetermined image processing (such as stitching and color adjustment) to generate a 360-degree image. The front camera 103F and the rear camera 103R constitute the 360-degree camera 103. The 360-degree image processing unit 108 outputs the 360-degree image to the rotation compensation processing unit 109.

[0092] The rotation compensation processing unit 109 compensates for the rotation or shaking of the head of the subject side user in the 360-degree image. Therefore, the 360-degree image can be fixed in space regardless of the rotation or shaking of the head of the subject side user, and the user viewing the 360-degree image can check the movement of the viewpoint and other operations without losing the overall image of the space.

[0093] The audio input unit 110 includes a microphone, an audio processing circuit, etc. to collect audio uttered by a main body side user.

[0094] The audio output unit 111 includes a speaker, an audio processing circuit, and the like to output the audio of the image-side user transmitted from the image-side device 200. Note that the audio input unit 110 and the audio output unit 111 are not essential components.

[0095] The communication unit 112 is a communication module that transmits image data, audio data, etc. to the imaging side device 200 through a network and receives image data, audio data, etc. from the imaging side device 200. Specific examples of communication methods may include (whether wired or wireless) cellular communication, Wi-Fi, Bluetooth (registered trademark), near field communication (NFC), Ethernet (registered trademark), High Definition Multimedia Interface (HDMI (registered trademark)), Universal Serial Bus (USB), etc.

[0096] The information processing device 150 includes a viewpoint position detection unit 105, a front view image viewpoint position recognition unit 106, a 360-degree image viewpoint position recognition unit 107, and a 360-degree image processing unit 108. The information processing device 150 according to the present embodiment operates on a device such as a personal computer, a tablet terminal, or a smartphone, but such a device may have a function as the information processing device 150 in advance, or a device having a function as a computer may execute a program to implement the information processing device 150 and the information processing method. In addition, the control unit may execute the program to function as the information processing device 150. The program may be pre-installed in the main body side device 100, or may be distributed via download, storage medium, etc. and installed by a user, etc. In addition, the information processing device 150 may be configured as a single device.

[0097] Furthermore, although not shown, the main body-side device 100 may include a control unit, a storage unit, and an input unit.

[0098] The control unit includes a central processing unit (CPU), a random access memory (RAM), a read only memory (ROM), etc. The CPU performs various types of processing according to a program stored in the ROM, and issues commands to control the entire main body side device 100 and each unit thereof.

[0099] The storage unit is, for example, a large-capacity storage medium such as a hard disk or a flash memory. The storage unit stores data, a database, an application, and the like used by the main body-side device 100 .

[0100] The input unit is used by a main body side user to input an operation instruction or the like to the main body side device 100. When the user provides input to the input unit, a control signal corresponding to the input is generated, and each unit performs various processes according to the control signal.

[0101] like Figure 4As shown in , the image side device 200 includes a communication unit 201, a position and orientation detection unit 202, an image processing unit 203, a display unit 204, an audio input unit 205, and an audio output unit 206.

[0102] The communication unit 201 , the audio input unit 205 , and the audio output unit 206 are similar to those included in the main body-side device 100 .

[0103] The position and orientation detection unit 202 includes various sensors that detect the position and orientation of the imaging side device 200. Examples of the sensor include IMU, inertial sensor (accelerometer, angular velocity sensor, gyroscope for two-axis or three-axis directions), LiDAR, ToF sensor, GNSS, GPS, etc. The position and orientation detection unit 202 outputs the position and orientation information to the image processing unit 203.

[0104] The image processing unit 203 identifies and cuts out a display area to be displayed on the display unit 204 from the 360-degree image transmitted by the main body side device 100 based on the position and orientation information, and outputs the display area to the display unit 204. The display area in the 360-degree image displayed on the display unit 204 becomes the field of view of the image side user. The display area changes in a manner that depends on the position and orientation of the image side device 200, so that when the image side user turns right, the display area is converted to the right in the 360-degree image, and when the image side user turns left, the display area is converted to the left in the 360-degree image. Therefore, the image side user can freely change the viewpoint in the 360-degree image.

[0105] The display unit 204 is a display device such as a liquid crystal display or an organic electroluminescent (EL) display, which displays a display area in the 360-degree image output from the image processing unit 203 and is used by the user to use the user interface (UI) of the imaging side device 200.

[0106] The image side device 200 is configured as described above. Examples of the image side device 200 include a head mounted display, a smartphone, a tablet terminal, a smart watch, a personal computer, a wearable device, a television, a projector, a portable game console, a portable music player, etc. The control program for executing the processing according to the present technology may be pre-installed in the image side device 200, or may be distributed via download, a storage medium, etc. and installed by the user himself.

[0107] [1-2-2. Transformation of viewpoint position coordinates]

[0108] Next, we will refer to Figure 5Describe the processing performed by the information processing device 150. With the sharing of 360-degree images achieved by the image sharing system 10, the information processing device 150 needs to generate a 360-degree image, transform the viewpoint position coordinates indicating the position where the subject-side user is looking into coordinates in the 360-degree image, and identify which part of the 360-degree image the subject-side user is looking at.

[0109] Note that as Figure 5 As a prerequisite for the processing in, it is assumed that the right eye image and the left eye image have been captured by the viewpoint detection camera 101, the front view image has been captured by the front view camera 102, and the front image and the rear image have been captured by the front camera 103F and the rear camera 103R constituting the 360-degree camera 103.

[0110] In addition, the position and orientation information has been detected by the position and orientation detection unit 104. Using the rotation angle of the head movement of the subject side user, the position and orientation estimation result is expressed as (Δθ, Δψ).

[0111] First, in step S101, the viewpoint position detection unit 105 detects viewpoint position coordinates from the right eye image and the left eye image captured by the viewpoint detection camera 101. The viewpoint position coordinates are expressed in xy coordinates and are expressed as Figure 6 As shown in A, (xg1, yg1) is detected for the left eye image, and (xg2, yg2) is detected for the right eye image.

[0112] Next, in step S102, the forward-view image viewpoint position recognition unit 106 recognizes the viewpoint position coordinates in the forward-view image based on the viewpoint position coordinates in the forward-view image and the right-eye image and the left-eye image captured by the forward-view camera 102. Figure 6 As shown in B of FIG. 1 , the viewpoint position coordinates in the front view image become (x′, y′) and are expressed by the following Equation 1.

[0113] [Mathematical formula 1]

[0114] (x', y')=f'(xg1, yg1, xg2, yg2)

[0115] f' represents a coordinate transformation function determined by a general eye tracking calibration. To obtain f', positional relationship information about the positional relationship between the left viewpoint detection camera 101L, the right viewpoint detection camera 101R and the front view camera 102 is required, so the positional relationship information is pre-stored in the information processing device 150.

[0116] Next, in step S103, the 360-degree image viewpoint position identification unit 107 identifies the viewpoint position coordinates in the front image as a part of the 360-degree image based on the viewpoint position coordinates and the position and orientation information in the front view image. Figure 6 As shown in C, the viewpoint position coordinates in the front image become (x, y) and are expressed by the following Equation 2.

[0117] [Mathematical formula 2]

[0118]

[0119] f represents a coordinate transformation function determined by general eye tracking calibration. In order to obtain f, positional relationship information about the positional relationship between the 360-degree camera 103 and the front view camera 102 is required, and thus the positional relationship information is pre-stored in the information processing device 150 .

[0120] Next, in step S104, the 360-degree image processing unit 108 combines the front image and the rear image, and also performs predetermined image processing to generate a 360-degree image. As a result of generating a 360-degree image using the front image to which the viewpoint position coordinates are assigned as described above, the viewpoint position coordinates are assigned to the 360-degree image.

[0121] Next, in step S105 , the rotation compensation processing unit 109 performs rotation compensation processing on the 360-degree image to compensate for the rotation or shaking of the head of the subject-side user.

[0122] Then, in step S106, the information processing device 150 outputs the 360-degree image to which the viewpoint position coordinates are assigned. The output 360-degree image is transmitted to the image-side device 200 via the communication unit 112 and the network.

[0123] Next, we will refer to Figure 7 Coordinate transformation for assigning viewpoint position coordinates in a front-view image to a 360-degree image, which is performed by the 360-degree image viewpoint position identification unit 107, is described.

[0124] As described above, in the main body side device 100, the front camera device 103F as a part of the 360-degree camera device 103 is firmly installed at the front center of the main body side user's head, and the rear camera device 103R is firmly installed at the rear center of the main body side user's head. In addition, the front view camera device 102 is firmly installed at the front center of the main body side user's head.

[0125] Therefore, the relationship between the front image and the rear image that constitute the 360-degree image and the forward-looking image is as follows: Figure 7As shown. The front image is placed at the center, and the rear image is placed at the left and right ends of the front image, thereby forming a 360-degree image. The imaging area of ​​the front-view camera 102 is set at the center of the 360-degree image, that is, the center of the front image.

[0126] First, in the coordinate transformation used to assign the viewpoint position coordinates to the 360-degree image, as Figure 7 As shown in , a grid with a predetermined size is set on the front view image. In addition, a grid with a predetermined size and the same number of intersections as the grid set on the front view image is also set on the front image. Then, calibration is performed in advance to generate a correspondence between the intersections of the grid on the front image and the intersections of the grid on the front view image as a lookup table (LUT). The LUT is pre-generated for all intersections of the front view image and the grid set on the front image. The details of the calibration will be described below.

[0127] Typically, the field of view of the front camera 103F is wider than that of the front camera 102, and in the present embodiment, the front camera 103F and the front camera 102 have a fixed positional relationship and both face forward. Therefore, by performing a coordinate transformation based on a pre-generated LUT, it is possible to identify that the intersection of the grid on the front image corresponds to the position on the front image, and the viewpoint position coordinates identified in the front image can be rendered onto the front image. Therefore, the viewpoint position coordinates can be assigned to the 360-degree image. In the first embodiment, the coordinate transformation is performed before the 360-degree image is generated by combining the front image and the rear image.

[0128] Note that increasing the grid granularity allows the accuracy of coordinate transformation to be improved, but results in an increase in the processing load of the coordinate transformation process. On the other hand, decreasing the grid granularity results in a decrease in the accuracy of coordinate transformation, but allows the processing load of the coordinate transformation process to be reduced. Therefore, the user or operator of the image sharing system 10 can set the grid granularity according to the accuracy of coordinate transformation, the computing power of the subject-side device 100, etc.

[0129] For transmission of 360-degree images to the imaging side device 200, a frame rate of at least 30 frames per second (fps) is desired, so the number of data entries of the LUT is pre-increased by linear or nonlinear interpolation in order to reduce the calculation load during viewpoint coordinate transformation.

[0130] Note that in addition to the coordinate transformation based on the LUT, a method for identifying the viewpoint position in the front image based on features near the viewpoint position in the front view image may be employed.

[0131] [1-2-3. Calibration]

[0132] Next, we will refer to Figures 8 to 11 Describes calibration for creating a LUT used in coordinate transformation for assigning viewpoint position coordinates to a 360-degree image.

[0133] Calibration may be performed before the communication between the subject-side device 100 and the image-side device 200 is started, or may be performed during the communication between the subject-side device 100 and the image-side device 200. For example, in a case where the installation position of the camera of the subject-side device 100 changes every day, calibration needs to be performed before the communication starts. In addition, in a case where the subject-side user changes the installation position of the camera or removes and reinstalls the camera, for example, during the communication between the subject-side device 100 and the image-side device 200, calibration needs to be performed during the communication. Note that in a case where the position and orientation information of both the front-view camera 102 and the 360-degree camera 103 can be acquired, the position and orientation information can be used to correct the correspondence between the front-view image and the front image (360-degree image), which eliminates the need for calibration.

[0134] First, refer to Figure 8 The following describes calibration performed before the start of communication between the subject-side device 100 and the imaging-side device 200. This calibration is performed by the subject-side user (or a staff member related to the subject-side user).

[0135] The main body side device 100 is connected to the external device 300 (eg, PC), and the front view image and the 360-degree image are output to the external device 300. Then, the external device 300 displays a calibration UI that displays the front view image and the 360-degree image on one screen on the display 310. Figure 8 As shown in , the front view image and the 360-degree image are preferably displayed side by side on the display 310. On the calibration UI, a grid with a predetermined granularity is superimposed on the front view image. The program for generating the calibration UI needs to be pre-installed in the external device 300. Note that the external device 300 and the display 310 may be integrated into a single device, or may be independent devices connected in a wired or wireless manner.

[0136] Then, the subject-side user views the calibration UI displayed on the display 310 to visually check the position of each intersection of the grid on the front view image, and provides an input for specifying the same position as the intersection of the grid on the front view image to the 360-degree image. By repeating the above operation for multiple intersections of the grid on the front view image, a correspondence relationship between the coordinates of the front view image and the coordinates of the front image can be established to create a LUT. Note that in order to improve the accuracy of the LUT, it is desirable to establish a correspondence relationship with respect to more (or all) grid intersections.

[0137] Note that in the case where calibration is performed during communication between the subject-side device 100 and the imaging-side device 200, similar to Figure 8 The calibration UI in the calibration UI can be displayed on a wearable device (e.g., a glasses-type display or an HMD) worn by the subject-side user to allow the subject-side user to perform calibration.

[0138] Note that although the configuration in which the main body side device 100 is connected to the external device 300 has been described above, Fig. 9 As shown in , the main body-side device 100 itself may include a calibration display unit 113 for generating and displaying a calibration UI.

[0139] Next, a description will be given of a first example of calibration performed during communication between the main body side device 100 and the imaging side device 200. This calibration is performed by the imaging side user (or a staff member related to the imaging side user).

[0140] The main device 100 is connected to an external device 300 (such as a PC) and communicates with the external device 300. Figure 8 In a similar manner, the front view image and the 360-degree image are output to the external device 300. Then, the external device 300 displays a calibration UI that displays the front view image and the 360-degree image on one screen on the display 310. On the calibration UI, a grid with a predetermined granularity is superimposed on the front view image.

[0141] Then, the image side user views the calibration UI displayed on the display 310 to visually check the position of each intersection of the grid on the front view image, and provides an input for specifying the same position as the intersection of the grid on the front view image to the 360-degree image. By repeating the above operation for multiple intersections of the grid on the front view image, a correspondence between the coordinates of the front view image and the coordinates of the front image can be established to create a LUT. Note that in order to improve the accuracy of the LUT, it is desirable to establish a correspondence with respect to more (or all) grid intersections. This method is useful in a case where the subject side user cannot view the calibration UI.

[0142] Note that in the method for superimposing a grid on a front-view image, when the grid size is refined and the correspondence between the front image and the front-view image is established at more grid intersections, the accuracy of the LUT is improved, but the calibration work becomes complicated. On the other hand, when the grid size is increased and the correspondence between the front image and the front-view image is established at fewer grid intersections, the accuracy of the LUT is reduced, but the calibration work becomes simple.

[0143] Next, we will refer to Fig.10A second example of calibration performed during communication between the subject-side device 100 and the imaging-side device 200 is described. In the second example, the imaging-side user performs calibration while the subject-side user and the imaging-side user interact with each other.

[0144] like Fig.10 As shown in , the main body side device 100 is connected to the external device 300 (such as a PC), and the front view image and the 360-degree image are output to the external device. Then, the external device 300 displays a calibration UI that displays the front view image and the 360-degree image on one screen on the display 310. The calibration UI does not necessarily have to superimpose a grid on the front view image. The calibration UI is viewed by the imaging side user.

[0145] Then, the image side user instructs the subject side user to look at a specific position by voice, etc., and when the subject side user looks at the specific position, an icon indicating the viewpoint position is superimposed on the front view image on the calibration UI. The icon indicating the viewpoint position can be displayed by outputting the first viewpoint position recognition result from the front view image viewpoint position recognition unit 106 to the external device 300 and rendering the icon on the calibration UI based on the first viewpoint position recognition result.

[0146] The image-side user visually checks the same position as the icon indicating the viewpoint position superimposed on the front view image in the 360-degree image, and provides an input for specifying the position in the 360-degree image.

[0147] exist Fig.10 In the example in , the image side user finds the position that the subject side user is looking at from the 360-degree image, which is indicated by the icon (1) on the front view image, and provides an input for specifying the position. Similarly, the image side user finds the position that the subject side user is looking at from the 360-degree image, which is indicated by the icon (2) on the front view image, and provides an input for specifying the position. By repeating the above operation at different positions, the correspondence between the coordinates of the front view image and the coordinates of the front image can be established to create a LUT. Fig.10 The positions of icons (1) and (2) in FIG. 1 are examples only.

[0148] Note that the LUT may also be generated by a method in which the subject side user fixes the viewpoint position even without an instruction from the image side user, and a signal indicating that the viewpoint position has been fixed is sent to the image side user, and the image side user confirms the viewpoint position and establishes a correspondence. As described above, the image side user may also perform calibration in response to a signal from the subject side user. In addition, in the event that there is a deviation in the multiple viewpoint positions of the subject side user, the image side user may instruct the subject side user, for example, to "align the viewpoint in this direction" to eliminate the deviation in the viewpoint position, thereby allowing an improvement in the accuracy of the LUT.

[0149] like Fig.11 As shown in , whether the number of viewpoint positions with established correspondences for calibration is sufficient can be determined by, for example, comparing a predetermined threshold with the distance between the viewpoint position with established correspondences and other viewpoint positions with established correspondences and located near the viewpoint position. Fig.11 As shown in A of FIG. 1 , in the case where the distance between the viewpoint positions is greater than or equal to the threshold, it is determined that the number of viewpoint positions having established correspondence is insufficient. On the other hand, as Fig.11 As shown in B, when the distance between the viewpoint positions is less than or equal to the threshold, it is determined that the number of viewpoint positions with established correspondence is sufficient. Note that the threshold is preset based on the number of viewpoint positions to establish correspondence, the accuracy of the LUT, etc.

[0150] [1-3. Sharing of viewpoint position coordinates]

[0151] [1-3-1. Configuration of the main body side device 100 and the image side device 200]

[0152] Next, the sharing of viewpoint position coordinates when a 360-degree image is shared between the subject-side device 100 and the image-side device 200 will be described. Fig.12 and Fig.13 The configuration of the subject-side device 100 and the image-side device 200 for sharing the viewpoint position coordinates will be described. Figure 2 and Figure 4 Description A description of the configuration.

[0153] like Fig.12 As shown in , the main body side device 100 includes a sharing processing unit 114 , a meta information processing unit 115 , and an output unit 116 .

[0154] The sharing processing unit 114 determines whether to perform a process of sharing viewpoint position coordinates between the subject side device 100 and the imaging side device 200, and generates subject side meta information according to the determination result. Then, the subject side meta information is transmitted to the imaging side device 200 via the communication unit 112. The process of sharing viewpoint position coordinates includes "field of view synchronization" and "display of viewpoint position coordinates". The details of such a process will be described later.

[0155] The subject side meta-information includes viewpoint position coordinates, information indicating the range of the forward-view image in the 360-degree image, information indicating the field of view of the subject side user, angle information indicating the position of the forward-view image in the 360-degree image, information about field of view synchronization and asynchrony, viewpoint dwell time, etc.

[0156] The meta information processing unit 115 generates an output for the output unit 116 based on the image-side meta information transmitted from the image-side device 200 .

[0157] The output unit 116 includes a display that displays information for guiding the subject side user's field of view in a specific direction to synchronize the fields of view, an actuator for guiding the subject side user's field of view in a specific direction to synchronize the fields of view, and the like.

[0158] like Fig.13 As shown in , the image side device 200 includes an information input unit 207 , an input information processing unit 208 , a meta information processing unit 209 , and an output unit 210 .

[0159] The information input unit 207 is used by the user on the image side to input information. Examples of the information input unit 207 include a touch panel, a mouse, a VR controller, a viewpoint tracking device, etc., but any device can be used as long as information can be input.

[0160] The input information processing unit 208 generates image-side meta information based on the information input by the information input unit 207 and the position and orientation information acquired from the position and orientation detection unit 202. The image-side meta information includes information on synchronization and asynchrony of the visual field, viewpoint position coordinates of the user on the image side, etc. The image-side meta information is transmitted to the subject-side device 100 via the communication unit 201 and the network.

[0161] The meta information processing unit 209 determines output for the output unit 210 based on the subject-side meta information transmitted from the subject-side device 100 .

[0162] The output unit 210 is used to guide the visual field of the user on the imaging side to synchronize the visual fields. Although described in detail later, the output unit 210 includes an actuator that guides the visual field of the user on the imaging side by vibration or the like.

[0163] [1-3-2. Processing to realize sharing of viewpoint position coordinates: field of view synchronization]

[0164] Next, the field of view synchronization as a process for achieving the sharing of viewpoint position coordinates will be described. Field of view synchronization means that the subject side user and the image side user see the same area (field of view) in the 360-degree image shared between the subject side device 100 and the image side device 200. The image side user can display and see various areas in the 360-degree image by operating the image side device 200 or changing the position and orientation of the image side device 200. Therefore, the field of view (front view image) of the subject side user and the field of view (display area for the image side device 200) of the image side user do not necessarily coincide with each other. Therefore, the image side user does not necessarily see the area that the subject side user wants the image side user to see. Therefore, by synchronizing the field of view of the subject side user and the field of view of the image side user, the subject side user and the image side user can see the same area in the 360-degree image. By synchronizing the field of view of the subject side user and the field of view of the image side user, the viewpoint position coordinates in the field of view can be shared.

[0165] First, refer to Fig.14 , the determination as to whether to perform field synchronization in the shared processing unit 114 of the main body side device 100 will be described.

[0166] First, in step S201, the audio input unit 110 obtains the audio emitted by the subject side user and converts the audio into text. The conversion of audio to text can be achieved by, for example, machine learning, deep learning, etc. Note that it is also possible to obtain the audio emitted by the image side user and convert it into text. In this case, the image side device 200 transmits the audio data to the subject side device 100.

[0167] In addition, in step S202, the viewpoint information of the subject side user is obtained through the viewpoint detection camera device 101, the infrared sensor, etc., and the viewpoint dwell time is estimated. The viewpoint dwell time can be estimated based on the position of the viewpoint, pupil movement, etc. Note that the viewpoint information of the image side user can also be obtained. In this case, the image side device 200 transmits the viewpoint information of the image side user to the subject side device 100.

[0168] In addition, in step S203, the field of view synchronization and asynchronous information is obtained. In the case where the image side user wishes to synchronize the field of view, the image side user turns on the field of view synchronization switch by inputting to the information input unit 207. The field of view synchronization switch corresponds to an instruction issued by the second user (image side user) in the content of the invention to perform processing related to sharing of viewpoint position coordinates. When the field of view synchronization switch is turned on, processing for synchronizing the field of view is performed. The input information processing unit 208 of the image side device 200 generates synchronization and asynchronous information as image side meta information based on the information input for the field of view synchronization switch. Then, the image side meta information is transmitted to the subject side device 100, and the sharing processing unit 114 obtains the image side meta information.

[0169] Note that steps S201 to S203 are not necessarily performed in this order, and there is no limitation on the order as long as the information in each step can be acquired before step S204.

[0170] Next, in step S204, it is determined whether any of the following three conditions are satisfied. The first of the three conditions is whether a specific indicator word is included in the speech content converted into text. This is because, in the case where a specific indicator word exists in the speech content of the subject side user and the image side user, it is considered that the subject side user and the image side user are trying to see a specific object or position in the 360-degree image, and the field of view should be synchronized.

[0171] The specific indicator words are pre-stored in the indicator word DB. Examples of the specific indicator words include "this", "there", "that", "over there", "right", "left", "up", "down", etc. Note that the indicator words are not limited to the above, and the subject side user and the image side user, the operator of the image sharing system 10, etc. can add any desired indicator words. The indicator word DB can be stored in a storage unit included in the subject side device 100, or can be maintained by the sharing processing unit 114 itself.

[0172] The second condition among the three conditions is whether the viewpoint stay time is greater than or equal to a predetermined threshold. This is because, when the viewpoint stay time is greater than or equal to the predetermined threshold, it is considered that the subject side user is paying attention to the position where the viewpoint stays, and the subject side user wants the image side user to see the position where the viewpoint stays.

[0173] The third condition among the three conditions is whether the visual field synchronization switch is turned on with reference to the visual field synchronization and asynchronous information. This is because when the synchronization switch is turned on, it is considered that the user on the video side wants to synchronize the visual field.

[0174] When any one of the three conditions is satisfied, the process proceeds to step S205 (YES in step S204 ).

[0175] Then, in step S205 , a process of synchronizing the fields of view is performed as a process of sharing the viewpoint position coordinates of the subject-side user with the image-side user.

[0176] Note that this determination does not necessarily need to be made based on the above three conditions, but may be made based on any one condition or two conditions.

[0177] Note that in the case where there are multiple image-side devices 200 (i.e., there are multiple image-side users), it is possible to determine which image-side user has priority in the field of view synchronization based on a flag called field of view priority. The field of view priority can be set based on, for example, the viewpoint dwell time of each image-side user, the amount of money charged to each image-side user for using the service of the image sharing system 10, the density of the fields of view of multiple image-side users, the intensity of the indicator words in the speech content of each image-side user such as the quantity or volume, etc.

[0178] Field of view synchronization can be achieved through a number of methods for directing the field of view.

[0179] Will refer to Fig.15 A first method for guiding the field of view is described. The first method is a method for forcibly switching the image display on the image-side device 200.

[0180] The image processing unit 203 converts the display area in the 360-degree image displayed on the display unit 204 based on the angle information indicating the field of view of the subject side user as the subject side meta-information so that the display area coincides with the field of view of the subject side user. The angle information indicates the position of the front view image in the 360-degree image.

[0181] For example, Fig.15 As shown in A of FIG. 1 , the field of view of the subject side user is in a specific direction, and the field of view of the image side user is in a direction different from the field of view of the subject side user. In this case, when the subject side user wants the image side user to see the same field of view as the subject side user, the display area in the 360-degree image on the image side device 200 is as follows: Fig.15 Thus, the front view image corresponding to the field of view of the subject side user is displayed on the image side device 200, that is, the field of view of the image side user can be forcibly guided so that the field of view of the image side user overlaps with the field of view of the subject side user.

[0182] Next, we will refer to Fig.16 A second method for guiding the field of view is described. The second method is a method for displaying an icon for guiding the field of view of the imaging side user on the display unit 204 of the imaging side device 200. Examples of the icon include an arrow icon.

[0183] The image processing unit 203 renders an arrow icon indicating the direction of a straight line extending from the center coordinates of the field of view of the subject side user (the front view image in the 360-degree image) to the center coordinates of the current field of view of the imaging side user (the display area of ​​the imaging side device 200) onto the display area in the 360-degree image, and outputs the resulting image to the display unit 204.

[0184] When the image-side user switches the field of view in the direction indicated by the arrow icon displayed on the display unit 204, the field of view of the image-side user can be made to coincide with the field of view of the subject-side user. In the case where the image-side device 200 is a portable device such as a smartphone, in order to switch the field of view, the image-side user only needs to move the image-side device 200 in the direction indicated by the arrow icon. In addition, in the case where the image-side device 200 is an HMD, in order to switch the field of view, the image-side user only needs to turn their face in the direction indicated by the arrow icon.

[0185] The direction indicated by the arrow icon is the direction of the straight line connecting the center coordinates of the field of view of the imaging side user and the center coordinates of the field of view of the subject side user as the sharing side, so that the imaging side user can be intuitively indicated in which direction to move the field of view.

[0186] In addition, the length of the arrow icon can be set to be proportional to the length of the straight line connecting the center coordinates of the field of view of the image side user and the center coordinates of the field of view of the subject side user. Therefore, it is possible to intuitively indicate to the image side user how much to move the field of view. Fig.16 In the example of A of FIG. 1 , the straight line connecting the center coordinates of the field of view of the image side user and the center coordinates of the field of view of the subject side user is longer, so the arrow icon becomes longer. Fig.16 In the example of B, the straight line connecting the center coordinates of the field of view of the image-side user and the center coordinates of the field of view of the subject-side user is shorter, so the arrow icon becomes shorter.

[0187] Will refer to Fig.17 A third method for guiding the visual field is described. The third method is a guiding method using a phenomenon called a hanger reflex. The hanger reflex is a reflex movement in which a sensation generated by applying pressure to the temporal part (temporalis muscle) of the head with a hanger or the like is transmitted to the cerebral cortex, and the head rotates due to the relaxation of the sternocleidomastoid muscle on the side where the pressure is applied.

[0188] In order to utilize the hanger reflection, an actuator for applying pressure to the temporal region (temporalis muscle) needs to be brought into contact with the temporal region of the image-side user. In addition, when the image-side user rotates their head, the image-side user's field of view (display area of ​​the display unit 204) needs to change, so that the image-side device 200 needs to be an HMD.

[0189] In order to make the image-side user turn right, pressure needs to be applied to the left temporal part, and in order to make the image-side user turn left, pressure needs to be applied to the right temporal part, so that the actuator AC of the HMD needs to be set at a position that contacts the left and right temporal parts of the image-side user. The actuator AC corresponds to the output unit 210.

[0190] The meta-information processing unit 209 of the image side device 200 generates a control signal for activating one of the left actuator AC and the right actuator AC based on the angle information which is the subject side meta-information transmitted from the subject side device 100, and outputs the control signal to the actuator AC. The angle information indicates the position of the front view image in the 360-degree image.

[0191] For example, in Fig.17 The actuator AC shown in A is in an inactive state. When the left actuator ACL is activated, as shown in Fig.17 As shown in B, the image side user rotates his head to the right, and the image side user turns right accordingly. Fig.17 The actuator AC shown in A is in an inactive state. When the right actuator ACR is activated, as shown in Fig.17 As shown in C, the image side user rotates his head to the left, and the image side user turns left accordingly. In response to the rotation of the image side user's head, the image displayed on the display unit 204 of the image side device 200 as an HMD is converted in the direction of the head rotation. Therefore, the field of view of the image side user can be overlapped with the field of view of the subject side user.

[0192] Note that electrical stimulation of the semicircular canals may be used instead of, or in conjunction with, the use of the coat-hanger reflex.

[0193] For example, a boundary line or the like indicating the field of view of the subject side user is displayed on the display unit 204 of the imaging side device 200, allowing the imaging side user to check whether the field of view of the subject side user and the field of view of the imaging side user overlap with reference to the boundary line or the like. In addition, using a picture-in-picture mechanism, an image indicating the field of view of the subject side user is continuously displayed on the display unit 204 of the imaging side device 200, and the imaging side user can check with reference to the image.

[0194] Note that the field of view of the subject-side user may also be guided to coincide with the field of view of the image-side user by using the guidance of the hanger reflection. In this case, it is necessary to set the actuators at the positions that contact the left and right temporal parts of the subject-side user. The meta-information processing unit 115 of the subject-side device 100 generates a control signal for activating one of the left actuator and the right actuator based on the position and orientation information as the image-side meta-information transmitted from the image-side device 200, and outputs the control signal to the actuator as the output unit 116. Then, when the actuator is activated, the head of the subject-side user can be moved by a motion similar to that of the reference image. Fig.17 Thus, the field of view of the subject side user can be made to coincide with the field of view of the image side user, for example, and in the case where the image side user wishes to look to the left, the field of view of the subject side user can be guided to the left.

[0195] [1-3-3. Processing for Sharing Viewpoint Position Coordinates: Display of Viewpoint Position Coordinates]

[0196] Next, "display of viewpoint position coordinates" as a process for implementing shared viewpoint position coordinates will be described. Display of viewpoint position coordinates means that the viewpoint position coordinates assigned to the 360-degree image by the process in the information processing device 150 described above are displayed together with the 360-degree image on the display unit 204 of the imaging side device 200 and presented to the imaging side user. Displaying the viewpoint position coordinates on the imaging side device 200 allows the imaging side user to grasp where the subject side user is looking at and where the subject side user is focusing.

[0197] First, refer to Fig.18 A process of generating viewpoint position coordinates as subject-side meta information in the shared processing unit 114 of the subject-side device 100 is described.

[0198] First, in step S301, the viewpoint information of the subject side user is obtained through the viewpoint detection camera 101, infrared sensor, etc., and the viewpoint dwell time is estimated. The viewpoint dwell time can be estimated based on the viewpoint position, viewpoint direction, pupil movement and dilation, etc.

[0199] Next, in step S302, it is determined whether the viewpoint stay time is greater than or equal to a predetermined threshold value. In the case where the viewpoint stay time is greater than or equal to the predetermined threshold value, the process proceeds to step S303 (Yes in step S302).

[0200] Then, in step S303, viewpoint position coordinates are generated as subject side meta information. The viewpoint position coordinates are viewpoint position coordinates recognized in the 360-degree image by the 360-degree image viewpoint position recognition unit 107. The subject side meta information is transmitted to the imaging side device 200 together with the 360-degree image via the communication unit 112 and the network.

[0201] Next, how to display the viewpoint position coordinates as the subject-side meta information on the image-side device 200 will be described.

[0202] When the imaging side device 200 receives the 360-degree image and the viewpoint position coordinates as the subject-side meta information, the image processing unit 203 renders an icon indicating the viewpoint position coordinates onto the 360-degree image and outputs the resulting image to the display unit 204 .

[0203] Then, if Fig.19 As shown in , the display area and viewpoint position coordinates in the 360-degree image for the imaging side device 200 are displayed on the display unit 204 and presented to the imaging side user. Fig.19In the 360-degree image, the viewpoint position coordinates are rendered and displayed as dot icons. Therefore, in the 360-degree image, the image side user can understand where the subject side user is looking at and where the subject side user is paying attention.

[0204] Note that the icon indicating the viewpoint position coordinates can be changed based on the viewpoint stay time of the subject side user. Fig. 20 As shown in FIG. 1A , when the viewpoint stay time is less than a predetermined threshold, the size of the icon is reduced in proportion to the viewpoint stay time. Fig. 20 As shown in FIG. 2B , when the viewpoint dwell time is greater than a predetermined threshold, the size of the icon increases in proportion to the viewpoint dwell time. Therefore, the image side user can be visually notified of the degree to which the subject side user pays attention to the viewpoint position. In addition, instead of or in addition to the size of the icon, the saturation, color, etc. of the icon can be changed based on the viewpoint dwell time. The image processing unit 203 performs icon deformation processing based on the viewpoint dwell time as the subject side meta information.

[0205] Next, we will refer to Fig.21 A process of generating viewpoint position coordinates as subject-side meta information based on the viewpoint stay time and the utterance content of the subject-side user in the shared processing unit 114 of the subject-side device 100 is described.

[0206] First, in step S401, the viewpoint information of the subject side user is obtained through the viewpoint detection camera 101, infrared sensor, etc., and the viewpoint dwell time is estimated. The viewpoint dwell time can be estimated based on the viewpoint position, viewpoint direction, pupil movement and dilation, etc.

[0207] Next, in step S402, the audio input unit 110 obtains the audio emitted by the subject-side user and converts the audio into text. The conversion of audio into text can be achieved by, for example, machine learning, deep learning, etc.

[0208] Next, in step S403, it is determined whether the viewpoint dwell time is greater than or equal to a predetermined threshold, and whether a specific indicator word is contained in the utterance content converted into text. Examples of specific indicator words include "this", "there", "that", "over there", "right", "left", "up", "down", etc., similar to those described above. Specific indicator words are pre-stored in the indicator word DB.

[0209] In a case where the viewpoint stay time is greater than or equal to the predetermined threshold and the specific indicator word is contained in the utterance content, the process proceeds to step S404 (Yes in step S403 ).

[0210] Then, in step S404, viewpoint position coordinates are generated as subject side meta information. The viewpoint position coordinates are viewpoint position coordinates recognized in the 360-degree image by the 360-degree image viewpoint position recognition unit 107. The subject side meta information is transmitted to the imaging side device 200 together with the 360-degree image via the communication unit 112 and the network.

[0211] The display of the viewpoint position coordinates as the subject-side meta information on the image-side device 200 is similar to that of the reference Fig.19 Note that in the case where subject-side meta-information is generated based on the utterance content of the subject-side user, such as Fig. 22 As shown in , the utterance content can be included in the subject side meta information and displayed together with the icon indicating the viewpoint position coordinates on the display unit of the image side device 200. Therefore, the image side user can more accurately grasp the object or position indicated by the utterance content of the subject side user.

[0212] Conversely, the viewpoint position coordinates of the image side user may also be presented to the subject side user. In this case, the image side device 200 or an external device connected to the image side device 200 needs to include a viewpoint detection camera 101, a viewpoint position detection unit 105, and a viewpoint position recognition unit similar to those of the subject side device 100. Then, the viewpoint position coordinates are transmitted to the subject side device 100 as image side meta information.

[0213] The meta information processing unit 115 of the subject side device 100 renders an icon indicating the viewpoint position coordinates onto the front view image corresponding to the field of view of the subject side user displayed on the output unit 116 based on the viewpoint position coordinates as the image side meta information.

[0214] Note that, in the case where a plurality of image-side devices 200 are connected to one main-side device 100, Fig.23 As shown in , the viewpoint position coordinates of multiple image side users can be displayed as icons on the output unit 116 of the main body side device 100. At this time, the viewpoint position coordinates of multiple image side users can be indicated by multiple icons with different colors or shapes, or the user name can be displayed to make each image side user distinguishable. In addition, in the case where the viewpoint position coordinates of the image side user are outside the boundary of the front view image corresponding to the field of view of the main body side user, the direction in which the viewpoint position coordinates exist can be indicated by an arrow icon. Therefore, the viewpoint position coordinates inside the front view image and the viewpoint position coordinates outside the front view image can be distinguished.

[0215] The processing according to the first embodiment is performed as described above. According to the first embodiment, the viewpoint position coordinates of the subject side user can be assigned to the 360-degree image transmitted from the subject side device 100 to the image side device 200. In addition, by transmitting the viewpoint position coordinates together with the 360-degree image to the image side device 200 and displaying the viewpoint position coordinates on the image side device 200, the image side user can be notified of the direction the subject side user is facing and where the subject side user is looking. Therefore, the image side user can grasp the position, place, etc. that the subject side user is paying attention to, and can easily grasp the movement of the subject side user's hands or body. The real-time sharing of the 360-degree image with the viewpoint position coordinates added as described above can promote the interaction between the subject side user and the image side user.

[0216] In addition, the subject side user and the image side user can mutually understand the object or area they are paying attention to, the movement of their corresponding viewpoints, etc. by presenting their corresponding viewpoint position coordinates to each other. Therefore, compared with the case of using gestures or voice commands, the subject side user and the image side user can continuously, immediately and specifically issue action instructions related to the viewpoint.

[0217] <2. Second Embodiment>

[0218] [2-1. Assigning viewpoint position coordinates to 360-degree images]

[0219] [2-1-1. Configuration of main body side device 100]

[0220] Next, a second embodiment of the present technology will be described. Fig.24 and Fig.25 The configuration of the subject-side device 100 is described. Note that the configuration of the image sharing system 10 and the configuration of the image-side device 200 are similar to those in the first embodiment.

[0221] like Fig.24 As shown, the main body side device 100 includes a viewpoint detection camera device 101, a front view camera device 102, a 360-degree camera device 103, a position and orientation detection unit 104, a viewpoint position detection unit 105, a front view image viewpoint position recognition unit 106, a 360-degree image viewpoint position recognition unit 107, a 360-degree image processing unit 108, a rotation compensation processing unit 109, an audio input unit 110, an audio output unit 111 and a communication unit 112.

[0222] The configuration, arrangement, and installation method of the viewpoint detection camera 101 and the front view camera 102 are similar to those in the first embodiment.

[0223] The 360-degree camera 103 includes a wide-angle lens capable of capturing a 180-degree field of view, and includes a front camera 103F that captures an image obliquely in front of a subject-side user and a rear camera 103R that captures an image obliquely behind the subject-side user. The configuration of acquiring the front image and the rear image in one shot is similar to that in the first embodiment. Fig.25 As shown in , in the second embodiment, the front camera 103F is installed diagonally in front of the face. In addition, the rear camera 103R is installed diagonally behind the head. As described above, in the second embodiment, the installation positions of the front camera 103F and the rear camera 103R are different from the installation positions in the first embodiment.

[0224] According to the second embodiment, the front image captured by the front camera 103F and the rear image captured by the rear camera 103R, the front camera 103F and the rear camera 103R constitute a 360-degree camera 103, and are first output to the 360-degree image processing unit 108, rather than being output to the 360-degree image viewpoint position recognition unit 107.

[0225] The 360-degree image processing unit 108 combines the front image and the rear image to generate a 360-degree image. The 360-degree image processing unit 108 outputs the generated 360-degree image to the 360-degree image viewpoint position recognition unit 107.

[0226] The 360-degree image viewpoint position recognition unit 107 recognizes the viewpoint position coordinates in the 360-degree image based on the 360-degree image, the position and orientation information, and the first viewpoint position recognition result. The 360-degree image viewpoint position recognition unit 107 outputs the 360-degree image and the second viewpoint position recognition result to the rotation compensation processing unit 109.

[0227] The second embodiment differs from the first embodiment in that a 360-degree image is generated by the 360-degree image processing unit 108 before the 360-degree image viewpoint position recognition unit 107 recognizes the viewpoint position coordinates, and the viewpoint position coordinates are recognized in the generated 360-degree image rather than in the front image.

[0228] The other configuration is similar to that in the first embodiment.

[0229] [2-1-2. Transformation of viewpoint position coordinates]

[0230] Next, we will refer to Fig.26The processing performed by the information processing device 150 according to the second embodiment is described. With the sharing of the 360-degree image achieved by the image sharing system 10, the information processing device 150 needs to generate a 360-degree image, transform the viewpoint position coordinates indicating the position where the subject-side user is looking into coordinates in the 360-degree image, and identify which part of the 360-degree image the subject-side user is looking at.

[0231] Note that as Fig.26 As a prerequisite for the processing in, it is assumed that the right eye image and the left eye image have been captured by the viewpoint detection camera 101, the front view image has been captured by the front view camera 102, and the front image and the rear image have been captured by the front camera 103F and the rear camera 103R constituting the 360-degree camera 103.

[0232] In addition, the position and orientation information has been detected by the position and orientation detection unit 104. Using the rotation angle of the head movement of the subject side user, the position and orientation estimation result is expressed as (Δθ, Δψ).

[0233] Steps S101 and S102 are similar to those in the first embodiment. In a similar manner to the first embodiment, the viewpoint position coordinates (x', y') in the front view image are expressed by the above-mentioned equation 1.

[0234] Next, in step S501 , the 360-degree image processing unit 108 combines the front image and the rear image, and also performs predetermined image processing to generate a 360-degree image.

[0235] Next, in step S502, the 360-degree image viewpoint position identification unit 107 identifies the viewpoint position coordinates in the 360-degree image based on the viewpoint position coordinates in the front view image. Similar to the first embodiment, the viewpoint position coordinates (x, y) in the 360-degree image are represented by the above-mentioned equation 2.

[0236] Steps S105 and S106 are similar to those in the first embodiment.

[0237] Next, we will refer to Fig. 27 Coordinate transformation for assigning viewpoint position coordinates in a front-view image to a 360-degree image, which is performed by the 360-degree image viewpoint position identification unit 107, is described.

[0238] As described above, in the second embodiment, the front camera 103F as a part of the 360-degree camera 103 is installed diagonally in front of the face of the subject side user, and the rear camera 103R is installed diagonally behind the head of the subject side user. In addition, the front view camera 102 is firmly installed at the front center of the head of the subject side user.

[0239] Therefore, the relationship between the front image and the rear image that constitute the 360-degree image and the forward-looking image is as follows: Fig. 27 As shown in . The front image and the rear image are arranged side by side to form a 360-degree image. The imaging area of ​​the front-view camera 102 is set at the center of the 360-degree image, spanning the boundary between the front image and the rear image.

[0240] In the coordinate transformation for assigning the viewpoint position coordinates to the 360-degree image, first, a grid is set on the front view image. In addition, a grid having the same number of intersections as the grid set on the front view image is also set on the 360-degree image. As described above, in the second embodiment, since the imaging area of ​​the front view camera 102 is set at the center of the 360-degree image, straddling the boundary between the front image and the rear image, it is necessary to generate a 360-degree image and set a grid on the 360-degree image before the viewpoint coordinate transformation.

[0241] Then, based on the LUT generated in advance by calibration, the intersection of the grid on the front view image corresponds to the position on the 360-degree image through coordinate transformation, and the viewpoint position coordinates identified in the front view image are rendered on the 360-degree image. In the second embodiment, the coordinate transformation is performed after the 360-degree image is generated by combining the front image and the rear image.

[0242] For transmission of 360-degree images to the imaging side device 200, a frame rate of at least 30 fps is desired, so the number of data entries of the LUT is increased in advance by linear or nonlinear interpolation in order to reduce the calculation load during viewpoint coordinate transformation.

[0243] The calibration method is similar to that in the first embodiment. Note that in addition to the coordinate transformation based on the LUT, there may be other methods for recognizing the viewpoint position in the 360-degree image based on features near the viewpoint position in the front view image.

[0244] Note that Fig.28 As shown in , even in the case where the 360-degree camera 103 includes a left camera 103Le that captures an image of the left side of the subject side user and a right camera 103Ri that captures an image of the right side of the subject side user, the viewpoint position coordinates can be assigned to the 360-degree image in a manner similar to the second embodiment.

[0245] The processing according to the second embodiment is performed as described above. According to the second embodiment, even when the arrangement of the 360-degree camera device 103 is different from the arrangement in the first embodiment, the viewpoint position coordinates of the subject side user can be assigned to the 360-degree image shared between the subject side device 100 and the image side device 200. In addition, the viewpoint position coordinates can also be assigned to the 360-degree image obtained by combining the front image and the rear image. Note that the processing for achieving the sharing of the viewpoint position coordinates between the subject side device 100 and the image side device 200 is similar to the processing in the first embodiment.

[0246] <3. Third Embodiment>

[0247] [3-1. Assigning viewpoint position coordinates to a 360-degree image]

[0248] [3-1-1. Configuration of main body side device 100]

[0249] Next, a third embodiment of the present technology will be described. Fig.29 and Fig.30 The configuration of the subject-side device 100 is described. Note that the configuration of the image sharing system 10 and the configuration of the image-side device 200 are similar to those in the first embodiment.

[0250] like Fig.29 As shown in the figure, the main body side device 100 includes a viewpoint detection camera device 101, a front view camera device 102, a 360-degree camera device 103, a first position and orientation detection unit 104a, a second position and orientation detection unit 104b, a viewpoint position detection unit 105, a front view image viewpoint position recognition unit 106, a 360-degree image viewpoint position recognition unit 107, a 360-degree image processing unit 108, a rotation compensation processing unit 109, an audio input unit 110, an audio output unit 111 and a communication unit 112.

[0251] The configuration, arrangement, and installation method of the viewpoint detection camera 101 and the front view camera 102 are similar to those in the first embodiment.

[0252] The 360-degree camera 103 includes a wide-angle lens capable of capturing a 180-degree field of view, and includes a front camera 103F that captures an image in front of a subject-side user and a rear camera 103R that captures an image behind the subject-side user. The configuration of acquiring the front image and the rear image in one shot is similar to that in the first embodiment. Fig.30As shown in , in the third embodiment, the front camera 103F and the rear camera 103R are mounted on the shoulders of the user on the subject side. As described above, in the third embodiment, the mounting positions of the front camera 103F and the rear camera 103R are different from those in the first embodiment.

[0253] In the third embodiment, since the 360-degree camera 103 is mounted on the shoulder (body) instead of the face or head of the subject side user, the position and orientation of the 360-degree camera 103 change in response to the movement of the subject side user's body. On the other hand, the position and orientation of the front view camera 102 change in response to the movement of the subject side user's face. Therefore, the front view camera 102 and the 360-degree camera 103 operate independently, and the positional relationship is not fixed, but continuously changes in a manner that depends on the position and orientation of the subject side user's face.

[0254] In the third embodiment, the front image captured by the front camera 103F and the rear image captured by the rear camera 103R, the front camera 103F and the rear camera 103R constituting the 360-degree camera 103, are first output to the 360-degree image processing unit 108, rather than being output to the 360-degree image viewpoint position recognition unit 107.

[0255] The 360-degree image processing unit 108 combines the front image and the rear image to generate a 360-degree image. The 360-degree image processing unit 108 outputs the generated 360-degree image to the 360-degree image viewpoint position recognition unit 107. The third embodiment is different from the first embodiment in that the 360-degree image is generated by the 360-degree image processing unit 108 before the 360-degree image viewpoint position recognition unit 107 recognizes the viewpoint position coordinates, and the viewpoint position coordinates are recognized in the 360-degree image instead of the front image.

[0256] The first position and orientation detection unit 104a detects the position and orientation of the forward-looking camera 102 installed on the face of the subject side user. The second position and orientation detection unit 104b detects the position and orientation of the 360-degree camera 103 installed on the shoulder (body) of the subject side user. The sensors used as the first position and orientation detection unit 104a and the second position and orientation detection unit 104b are similar to the sensors described in the first embodiment. The first position and orientation detection unit 104a outputs the first position and orientation information to the 360-degree image viewpoint position recognition unit 107 and the rotation compensation processing unit 109. In addition, the second position and orientation detection unit 104b outputs the second position and orientation information to the 360-degree image viewpoint position recognition unit 107 and the rotation compensation processing unit 109.

[0257] As described above, in the third embodiment, the main body side device 100 includes a first position and orientation detection unit 104a and a second position and orientation detection unit 104b. This is because the front view camera 102 is mounted on the head of the main body side user, and the 360-degree camera 103 is mounted on the body, and the positional relationship between the front view camera 102 and the 360-degree camera 103 is not fixed, so it is necessary to detect the positions and orientations of the front view camera 102 and the 360-degree camera 103 respectively.

[0258] The 360-degree image viewpoint position recognition unit 107 recognizes the viewpoint position coordinates in the 360-degree image based on the 360-degree image, the first position and orientation information, the second position and orientation information, and the first viewpoint position recognition result. The 360-degree image viewpoint position recognition unit 107 outputs the 360-degree image and the second viewpoint position recognition result to the rotation compensation processing unit 109.

[0259] The other configuration is similar to that in the first embodiment.

[0260] [3-1-2. Transformation of viewpoint position coordinates]

[0261] Next, a description will be given of the processing performed by the information processing device 150. For the sharing of a 360-degree image implemented by the image sharing system 10, the information processing device 150 needs to generate a 360-degree image, transform the viewpoint position coordinates indicating the position at which the subject-side user is looking into coordinates in the 360-degree image, and identify which part of the 360-degree image the subject-side user is looking at.

[0262] The flowchart showing the process for transformation of viewpoint position coordinates performed by the information processing device 150 is similar to Fig.26 A flowchart of the second embodiment is shown in FIG.

[0263] Note that as Fig.26 As a prerequisite for the processing in, it is assumed that the right eye image and the left eye image have been captured by the viewpoint detection camera 101, the front view image has been captured by the front view camera 102, and the front image and the rear image have been captured by the front camera 103F and the rear camera 103R constituting the 360-degree camera 103.

[0264] In addition, the position and orientation information has been detected by the position and orientation detection unit 104. Using the rotation angle of the head movement of the subject side user, the position and orientation estimation result is expressed as (Δθ, Δψ).

[0265] Next, we will refer to Fig.31Coordinate transformation for assigning viewpoint position coordinates in a front-view image to a 360-degree image, which is performed by the 360-degree image viewpoint position identification unit 107, is described.

[0266] As described above, in the third embodiment, the front camera 102 is firmly mounted at the front center of the head of the subject side user. In addition, the front camera 103F as a part of the 360-degree camera 103 is mounted on the shoulder of the subject side user facing forward, and the rear camera 103R is mounted on the shoulder of the subject side user facing backward.

[0267] Therefore, the relationship between the front view image, the rear image and the front image, and the rear image and the front image constituting the 360-degree image is as follows: Fig.31 . The 360-degree image includes a front image and a rear image arranged side by side. In addition, as described above, since the positional relationship between the front view camera 102 and the 360-degree camera 103 changes in a manner that depends on the position and orientation of the face and body of the subject-side user, the imaging area of ​​the front view camera 102 is not fixed at a specific position in the 360-degree image, but continuously changes.

[0268] In the coordinate transformation for assigning the viewpoint position coordinates to the 360-degree image, first, as Fig.31 As shown, a grid is set on the front view image. In addition, a grid is set on the 360-degree image. Note that in the third embodiment, the positional relationship between the front view image and the 360-degree image continuously changes. Therefore, the relative positional relationship between the front view camera 102 and the 360-degree camera 103 is identified based on the first position and orientation information and the second position and orientation information, and the position of the front view image in the 360-degree image is identified based on the positional relationship. Then, a grid having the same number of intersections as the grid set on the front view image is also set in the position of the front view image on the 360-degree image. Therefore, the viewpoint position coordinates can be transformed using the grid set on the 360-degree image and the grid set on the front view image.

[0269] Therefore, based on the LUT generated in advance by calibration, the intersection of the grid on the front view image is identified by coordinate transformation to correspond to the position on the 360-degree image, and the viewpoint position coordinates identified in the front view image are rendered on the 360-degree image. In the third embodiment, the coordinate transformation is performed after the 360-degree image is generated by combining the front image and the rear image.

[0270] Note that the forward-looking image needs to be distorted in a manner that depends on the position of the forward-looking image in the 360-degree image. The curvature may be calculated based on the optical parameters of the 360-degree camera 103 .

[0271] For transmission of 360-degree images to the imaging side device 200, a frame rate of at least 30 fps is desired, so the number of data entries of the LUT is increased in advance by linear or nonlinear interpolation in order to reduce the calculation load during viewpoint coordinate transformation.

[0272] The calibration method is similar to that in the first embodiment. Note that in addition to the coordinate transformation based on the LUT, there may be other methods for recognizing the viewpoint position in the 360-degree image based on features near the viewpoint position in the front view image.

[0273] Note that Fig.32 As shown in , even in a case where the 360-degree camera 103 includes a left camera 103Le that captures an image of the left side of a subject side user and a right camera 103Ri that captures an image of the right side of the subject side user and is mounted on the body of the subject side user, the viewpoint position coordinates can be assigned to the 360-degree image in a manner similar to the third embodiment.

[0274] Even in the case where the 360-degree camera 103 is not arranged at the first-person viewpoint position of the subject-side user as in the third embodiment, the viewpoint position coordinates can be assigned to the 360-degree image.

[0275] The processing according to the third embodiment is performed as described above. According to the third embodiment, even if the arrangement of the 360-degree camera 103 is different from the arrangement in the first embodiment, and the positional relationship between the front view camera 102 and the 360-degree camera 103 changes in a manner that depends on the position and orientation of the face of the subject side user, the viewpoint position coordinates of the subject side user can be assigned to the 360-degree image.

[0276] <4. Modification>

[0277] Although the embodiments of the present technology have been specifically described above, the present technology is not limited to the above-described embodiments, and various modifications based on the technical idea of ​​the present technology are possible.

[0278] The image types supported by the image sharing system 10 are not particularly limited, and examples of the image types may include still images, moving images, and frame images constituting the same image.

[0279] In the first to third embodiments, the processing is performed by the main body side device 100 and the image side device 200, but Fig.33 As shown in , the processing performed by the information processing device 150 can also be performed by the server 400.

[0280] The server 400 includes at least a first communication unit 401, a second communication unit 402, and a control unit and a storage unit (not shown). The server 400 is, for example, a cloud server.

[0281] Therefore, even in the case where the subject-side device 100 has low computing power, processing can be performed on the server 400 with higher throughput. In addition, the processing load on the subject-side device 100 can be reduced, power consumption can be reduced, etc. In addition, by processing performed by the server 400 having high computing power, a 360-degree image with a high frame rate can be transmitted to the imaging-side device 200. In addition, the subject-side device 100 can be reduced in size and reduced in cost.

[0282] Note that part of the processing performed by the information processing device 150 may be performed by the main body side device 100 , and the rest may be performed by the server 400 .

[0283] In the first to third embodiments, the subject-side device 100 includes the front-view camera 102, but even in the case where the subject-side device 100 does not include the front-view camera 102, the viewpoint position coordinates can be assigned to the 360-degree image. Since there is no front-view image without the front-view camera 102, the viewpoint position coordinates of the left-eye image and the right-eye image cannot be recognized as the viewpoint position coordinates in the front-view image. Therefore, rendering is performed as follows.

[0284] Fig.34 The first example shown in A is a case where there is no front-view camera 102 under the same conditions as the first embodiment described above. In this case, a predetermined area in the front image is virtually defined as an imaging area of ​​the front-view camera 102, and a conversion file between the viewpoint detection camera 101 and the front camera 103F as a part of the 360-degree camera 103 is generated, so that the viewpoint position coordinates can be rendered on the 360-degree image.

[0285] Fig.34 The second example shown in B is a case where there is no front-view camera 102 under the same conditions as the second embodiment described above. In this case, a predetermined area in the 360-degree image is virtually defined as an imaging area of ​​the front-view camera 102, and a conversion file between the viewpoint detection camera 101 and the front camera 103F as a part of the 360-degree camera 103 is generated, so that the viewpoint position coordinates can be rendered on the 360-degree image.

[0286] Fig.34 The third example shown in C is a case where there is no forward-looking camera device 102 under the same conditions as the third embodiment described above. In this case, Fig.34 A or Fig.34 In B, calibration is performed based on the positional relationship between the viewpoint detection camera 101 and the 360-degree camera 103, and then transformation of the viewpoint position coordinates is performed similarly to the third embodiment, so that the viewpoint position coordinates can be rendered on a 360-degree image.

[0287] As described above, in the absence of the forward-view camera 102 and the forward-view image, the viewpoint position coordinates in the 360-degree image are expressed by the following Equation 3.

[0288] [Math.3]

[0289]

[0290] The configuration of the main body side device 100 without the front view camera device 102 is as follows Fig.35 as shown in . Fig.35 The configuration is shown in the case where the forward-view camera 102 is not present in the main body side device 100 according to the first embodiment. Since the forward-view camera 102 is not present, the forward-view image viewpoint position recognition unit 106 becomes unnecessary.

[0291] The present technology may also have the following configurations. (1)

[0293] An information processing device, comprising:

[0294] a 360-degree image processing unit that generates a 360-degree image based on a plurality of captured images captured by a plurality of camera devices worn by the first user; and

[0295] A 360-degree image viewpoint position recognition unit is configured to recognize the viewpoint position coordinates of the first user in the 360-degree image. (2)

[0297] The information processing device according to (1) further includes a forward-view image viewpoint position identification unit, which identifies the viewpoint position coordinates of the first user in the forward-view image based on the forward-view image captured by the forward-view camera device that captures the image in front of the first user and the viewpoint position coordinates of the first user. (3)

[0299] The information processing device according to (2), further including a viewpoint position detection unit that detects viewpoint position coordinates of the first user from an eye image obtained by capturing an image of the eyes of the first user. (4)

[0301] The information processing device according to (3), wherein the front view image viewpoint position recognition unit transforms viewpoint position coordinates in the eye image into coordinates in the front view image. (5)

[0303] The information processing device according to (4), wherein the 360-degree image viewpoint position recognition unit transforms viewpoint position coordinates in the front view image into coordinates in the 360-degree image. (6)

[0305] The information processing device according to any one of (1) to (5), wherein the 360-degree image viewpoint position recognition unit transforms viewpoint position coordinates in an eye image obtained by capturing an image of the first user's eyes into coordinates in the 360-degree image. (7)

[0307] The information processing device according to any one of (1) to (6), wherein the 360-degree image viewpoint position identification unit identifies viewpoint position coordinates of the first user in the captured image constituting the 360-degree image. (8)

[0309] The information processing device according to any one of (1) to (7), wherein the 360-degree image viewpoint position recognition unit recognizes viewpoint position coordinates of the first user in the 360-degree image generated by the 360-degree image processing unit. (9)

[0311] The information processing device according to any one of (1) to (8), wherein the 360-degree image is transmitted to a display device of a second user different from the first user. (10)

[0313] The information processing device according to any one of (1) to (9), further including a sharing processing unit that determines whether to perform a process related to sharing of viewpoint position coordinates between the first user and the second user. (11)

[0315] The information processing device according to (10), wherein the sharing processing unit determines to perform processing related to sharing of viewpoint position coordinates when the viewpoint stay time of the first user is greater than or equal to a predetermined threshold. (12)

[0317] The information processing device according to (10) or (11), wherein, when a predetermined indicator word is included in the utterance content of the first user, the sharing processing unit determines to execute processing related to sharing of the viewpoint position coordinates. (13)

[0319] The information processing device according to any one of (10) to (12), wherein when the second user issues an instruction to execute processing related to sharing of viewpoint position coordinates, the sharing processing unit determines to execute processing related to sharing of viewpoint position coordinates. (14)

[0321] The information processing device according to (8), wherein the viewpoint position coordinates are displayed on a display device of the second user through a process related to sharing of the viewpoint position coordinates. (15)

[0323] The information processing device according to (8), wherein, in order to synchronize the field of view of the first user with the field of view of the second user, the field of view of the second user is guided by processing related to sharing of viewpoint position coordinates. (16)

[0325] The information processing device according to (15), wherein the field of view of the second user is guided by applying pressure to the temple of the second user. (17)

[0327] The information processing device according to (15) or (16), wherein the field of view of the second user is guided by switching the display on the display device of the second user. (18)

[0329] The information processing device according to any one of (15) to (17), wherein the field of view of the second user is guided by displaying an icon indicating a direction toward the display on a display device of the second user. (19)

[0331] A display device, configured to:

[0332] A 360-degree image generated based on a plurality of captured images captured by a plurality of camera devices worn by a first user and viewpoint position coordinates of the first user identified in the 360-degree image are displayed, and the 360-degree image and viewpoint position coordinates are presented to a second user different from the first user. (20)

[0334] An image sharing system, comprising:

[0335] An information processing device, the information processing device comprising:

[0336] a 360-degree image processing unit that generates a 360-degree image based on a plurality of captured images captured by a plurality of camera devices worn by the first user; and

[0337] a 360-degree image viewpoint position recognition unit, the 360-degree image viewpoint position recognition unit identifying the viewpoint position coordinates of the first user in the 360-degree image; and

[0338] A display device displays a 360-degree image and viewpoint position coordinates of a first user identified in the 360-degree image, and presents the 360-degree image and the viewpoint position coordinates to a second user different from the first user.

[0339] Reference Symbols List

[0340] 10Image Sharing System

[0341] 102 Forward-looking camera device

[0342] 105 Viewpoint position detection unit

[0343] 106 Forward view image viewpoint position recognition unit

[0344] 107 360-degree image viewpoint position recognition unit

[0345] 108 360-degree image processing unit

[0346] 114 shared processing units

[0347] 150 Information processing device

[0348] 200 Image side device (display device)

Claims

1. An information processing device, comprising: a 360-degree image processing unit that generates a 360-degree image based on a plurality of captured images captured by a plurality of camera devices worn by the first user; as well as A 360-degree image viewpoint position recognition unit is configured to recognize the viewpoint position coordinates of the first user in the 360-degree image.

2. The information processing device according to claim 1, further comprising: A forward-view image viewpoint position recognition unit that recognizes the viewpoint position coordinates of the first user in the forward-view image based on the forward-view image captured by a forward-view camera that captures images in front of the first user and the viewpoint position coordinates of the user.

3. The information processing device according to claim 2, further comprising: A viewpoint position detection unit detects viewpoint position coordinates of the first user from an eye image obtained by capturing an image of the eyes of the first user.

4. The information processing device according to claim 3, in, The front view image viewpoint position recognition unit transforms the viewpoint position coordinates in the eye image into coordinates in the front view image.

5. The information processing device according to claim 4, in, The 360-degree image viewpoint position recognition unit transforms the viewpoint position coordinates in the front view image into coordinates in the 360-degree image.

6. The information processing device according to claim 1, in, The 360-degree image viewpoint position recognition unit transforms the viewpoint position coordinates in an eye image obtained by capturing an image of the first user's eyes into coordinates in the 360-degree image.

7. The information processing device according to claim 1, in, The 360-degree image viewpoint position recognition unit recognizes viewpoint position coordinates of the first user in the captured image constituting the 360-degree image.

8. The information processing device according to claim 1, in, The 360-degree image viewpoint position recognition unit recognizes viewpoint position coordinates of the first user in the 360-degree image generated by the 360-degree image processing unit.

9. The information processing device according to claim 1, in, The 360-degree image is transmitted to a display device of a second user different from the first user.

10. The information processing device according to claim 1, further comprising: A sharing processing unit determines whether to perform a process related to sharing of the viewpoint position coordinates between the first user and the second user.

11. The information processing device according to claim 10, in, In a case where the viewpoint stay time of the first user is greater than or equal to a predetermined threshold, the sharing processing unit determines to perform processing related to sharing of the viewpoint position coordinates.

12. The information processing device according to claim 10, in, The sharing processing unit determines to execute processing related to sharing of the viewpoint position coordinates when a predetermined indicator word is included in the utterance content of the first user.

13. The information processing device according to claim 10, in, In a case where the second user issues an instruction to execute a process related to sharing of the viewpoint position coordinates, the sharing processing unit determines to execute the process related to sharing of the viewpoint position coordinates.

14. The information processing device according to claim 8, in, The viewpoint position coordinates are displayed on the display device of the second user through the processing related to the sharing of the viewpoint position coordinates.

15. The information processing device according to claim 8, in, In order to synchronize the first user's field of view with the second user's field of view, the second user's field of view is guided through a process related to sharing of the viewpoint position coordinates.

16. The information processing device according to claim 15, in, The field of view is directed by applying pressure to the temples of the second user.

17. The information processing device according to claim 15, in, The field of view is directed by switching a display on a display device of the second user.

18. The information processing device according to claim 15, in, The field of view is guided by displaying an icon indicating a direction to be guided to a display on a display device of the second user.

19. A display device, configured to: Display a 360-degree image generated based on multiple captured images captured by multiple camera devices worn by a first user and the viewpoint position coordinates of the first user identified in the 360-degree image, and present the 360-degree image and the viewpoint position coordinates to a second user different from the first user.

20. An image sharing system, comprising: An information processing device, the information processing device comprising: a 360-degree image processing unit that generates a 360-degree image based on a plurality of captured images captured by a plurality of camera devices worn by the first user; and a 360-degree image viewpoint position recognition unit, the 360-degree image viewpoint position recognition unit identifying the viewpoint position coordinates of the first user in the 360-degree image; and A display device displays the 360-degree image and the viewpoint position coordinates of the first user identified in the 360-degree image, and presents the 360-degree image and the viewpoint position coordinates to a second user different from the first user.

Images