Information processing device, information processing method, and information processing program

Abstract

To meet a viewer demand such as wanting to continue viewing a video in a specific direction.SOLUTION: An information processing device includes an acquisition unit and a determination unit. The acquisition unit acquires change information related to the change of the imaging direction of a spherical camera provided on a moving body. The determination unit determines a region corresponding to a direction specified by a viewer out of a first spherical image imaged by the spherical camera in a first imaging direction, as a first display region. When the spherical camera images in a second imaging direction which is different from the first imaging direction, the determination unit determines a region corresponding to the first display region of the first spherical image out of a second spherical image corresponding to the second imaging direction, as a second display region, based on the change information related to the change from the first imaging direction to the second imaging direction.SELECTED DRAWING: Figure 7

Description

[Technical Field]

[0001] The present invention relates to an information processing apparatus, an information processing method, and an information processing program. [Background Art]

[0002] Conventionally, there has been known a technique that allows a viewer to view a video (moving image) captured by a movable imaging apparatus in real time. For example, there is known a technique that provides a viewer staying indoors with a sense of realism as if the viewer were moving outdoors along with the photographer, by allowing the viewer to view a video captured by an imaging apparatus that is moving outdoors. [Prior Art Documents] [Patent Documents]

[0003] [Non-Patent Document 1] Takeo Yazaki, 10 others, "Study on Navigation Verification Results and Applicability in Urban Areas Using Wearable Telepresence System "T-Leap"", Proceedings of the 25th Annual Conference of the Virtual Reality Society of Japan, September 2020, [online], [retrieved September 28, Reiwa 3], Internet <http: / / conference.vrsj.org / ac2020 / program / doc / 1A1-4_PR0158.pdf> [Summary of the Invention] [Problem to be Solved by the Invention]

[0004] However, with conventional techniques, if the imaging apparatus moves while the viewer is viewing the video, the orientation of the displayed video changes. Accordingly, the displayed video also changes in conjunction with the imaging direction of the imaging apparatus, which makes it difficult to meet the viewer's desire to continue viewing a video of a specific direction.

[0005] The present application has been made in view of the above circumstances, and an object thereof is to meet the viewer's desire to continue viewing a video of a specific direction. [Means for Solving the Problem]

[0006] The information processing device according to the present invention includes an acquisition unit that acquires change information relating to a change in the imaging direction of a 360-degree camera provided on a moving object, and a determination unit that determines a region of a first 360-degree image captured by the 360-degree camera in a first imaging direction to be a first display region corresponding to a direction specified by the viewer, and the determination unit is characterized in that, when the 360-degree camera captures an image in a second imaging direction different from the first imaging direction, it determines a region of a second 360-degree image corresponding to the second imaging direction to be a second display region, based on change information relating to the change from the first imaging direction to the second imaging direction, and the region of the second 360-degree image corresponding to the second imaging direction to be a second display region. [Effects of the Invention]

[0007] According to one embodiment of the system, it is possible to satisfy the viewer's desire to continue watching images in a specific direction. [Brief explanation of the drawing]

[0008] [Figure 1] Figure 1 shows an example of the configuration of an information processing system according to an embodiment. [Figure 2] Figure 2 shows an example of a broadcaster terminal configuration according to an embodiment. [Figure 3] Figure 3 shows an example of how the broadcaster terminal according to the embodiment is attached. [Figure 4] Figure 4 shows an example of a 360-degree image according to the embodiment. [Figure 5] Figure 5 is a diagram illustrating the relationship between the coordinate systems (physical space, broadcaster side, and viewer side) according to this embodiment. [Figure 6] Figure 6 shows an example of the configuration of a viewer terminal according to the embodiment. [Figure 7] Figure 7 shows an example of the configuration of an information processing device according to the embodiment. [Figure 8] Figure 8 shows an example of how the displayed image on the viewer's side changes in response to a change in the imaging direction. [Figure 9] Figure 9 shows the relationship between the terminal coordinate system before the change in imaging direction and the surrounding objects. [Figure 10] Figure 10 shows the relationship between the viewer coordinate system and the image displayed on the viewer's side. [Figure 11] Figure 11 shows the relationship between the terminal coordinate system after a change in imaging direction and the viewer's displayed image when there is no field of view fixation. [Figure 12] Figure 12 is a diagram illustrating the processing performed by the information processing device according to the embodiment. [Figure 13] Figure 13 is a diagram illustrating the processing performed by the information processing device according to this embodiment. [Figure 14] Figure 14 is a flowchart showing an example of information processing according to the embodiment. [Figure 15] Figure 15 is a hardware configuration diagram showing an example of a computer that implements the functions of an information processing device. [Modes for carrying out the invention]

[0009] The following describes in detail, with reference to the drawings, the embodiments for implementing the information processing device, information processing method, and information processing program according to the present application (hereinafter referred to as "embodiments"). Note that these embodiments do not limit the information processing device, information processing method, and information processing program according to the present application. Furthermore, the same parts are denoted by the same reference numerals in each of the following embodiments, and redundant descriptions are omitted.

[0010] (Embodiment) [1. Configuration of the Information Processing System] The information processing system 1 shown in FIG. 1 will be described. As shown in FIG. 1, the information processing system 1 includes a distributor terminal 10, a viewer terminal 20, and an information processing apparatus 100. The distributor terminal 10, the viewer terminal 20, and the information processing apparatus 100 are communicably connected by wire or wirelessly via a predetermined communication network (Network N). FIG. 1 is a diagram showing a configuration example of the information processing system 1 according to the embodiment. Note that the information processing system 1 shown in FIG. 1 may include a plurality of viewer terminals 20.

[0011] The distributor terminal 10 is an information processing apparatus used by a distributor who distributes video to viewers (for example, a distributor who performs distribution while moving outdoors). The distributor terminal 10 may be any apparatus as long as it has an omnidirectional camera capable of capturing images 360 degrees around the distributor and can implement the processing in the embodiment. For example, the distributor terminal 10 is a mobile wearable-type apparatus that is worn by the distributor.

[0012] Further, the distributor terminal 10 includes a gyro sensor, and detects changes in the imaging direction of the omnidirectional camera provided in the distributor terminal 10 (including, for example, rotation of the imaging direction of the camera). For example, when the imaging direction of the omnidirectional camera rotates 30 degrees to the right, the gyro sensor outputs a gyro value that indicates, by a feature amount (for example, a vector amount), that the imaging direction of the omnidirectional camera has rotated 30 degrees to the right. Then, the distributor terminal 10 detects a change in the imaging direction of the omnidirectional camera based on the output gyro value. Note that the greater the change in the imaging direction, the greater the gyro value of the feature amount that is output. For example, when rotating 90 degrees, the change in the imaging direction is larger than when rotating 30 degrees, so a gyro value with a larger feature amount is output.

[0013] Hereinafter, it is assumed that the distributor terminal 10 is used by a distributor T11.

[0014] The viewer terminal 20 is an information processing device used by a viewer who views an image distributed via the distributor terminal 10. The viewer terminal 20 may be any device such as a smartphone, a tablet terminal, a notebook PC, a desktop PC, a mobile phone, a PDA, or the like, as long as the processing according to the embodiment can be implemented.

[0015] The viewer terminal 20 is, for example, a smart device such as a smartphone or a tablet, and is a mobile terminal device capable of communicating with any server device via a wireless communication network such as 3G to 5G (Generation) or LTE (Long Term Evolution). Further, the viewer terminal 20 may include a screen such as a liquid crystal display having a touch panel function, and may accept various operations on display data such as content, such as a tap operation, a slide operation, a scroll operation, or the like, using a finger, a stylus, or the like.

[0016] Hereinafter, the viewer terminal 201 is used by a viewer U11, and the viewer terminal 202 is used by a viewer U12. When it is not necessary to distinguish between the viewer terminals 201 and 202, they are referred to as "the viewer terminal 20". Further, when it is not necessary to distinguish between the viewers U11 and U12, they are simply referred to as "viewers". In some cases, the viewer terminal 20 may also be referred to as a viewer. That is, in the following description, a viewer may also be read as the viewer terminal 20.

[0017] The information processing apparatus 100 is an information processing apparatus intended to meet a viewer's request to continue viewing a video in a specific direction, and may be any apparatus as long as the processing according to the embodiment can be implemented. The information processing apparatus 100 is implemented by, for example, a server apparatus, a cloud system, or the like.

[0018] Note that Figure 1 shows the case where the information processing device 100 and the broadcaster terminal 10 are separate devices, but the information processing device 100 and the broadcaster terminal 10 may be integrated. Similarly, Figure 1 shows the case where the information processing device 100 and the viewer terminal 20 are separate devices, but the information processing device 100 and the viewer terminal 20 may be integrated. Furthermore, the information processing device 100 and a server device (not shown) that relays between the broadcaster terminal 10 and the viewer terminal 20 may be integrated.

[0019] [2. Configuration of the broadcaster's device] Next, the configuration of the broadcaster terminal 10 according to the embodiment will be described using Figures 2 to 4. Figure 2 is a diagram showing an example of the configuration of the broadcaster terminal 10 according to the embodiment. As shown in Figure 2, the broadcaster terminal 10 has a communication unit 11, an input unit 12, an output unit 13, a 360-degree camera 14, a gyro sensor 15, and a control unit 16.

[0020] (Communications Section 11) The communication unit 11 is implemented, for example, by a NIC (Network Interface Card). The communication unit 11 is connected to a predetermined network N by wire or wireless connection and transmits and receives information with the information processing device 100, etc., via the predetermined network N.

[0021] (Input section 12) The input unit 12 accepts various operations from the broadcaster. For example, the input unit 12 may accept various operations from the broadcaster via a touch panel display. Alternatively, the input unit 12 may accept various operations from buttons on the broadcaster terminal 10, or from a keyboard or mouse connected to the broadcaster terminal 10.

[0022] (Output section 13) The output unit 13 is a display screen for a tablet terminal or the like, which is realized by, for example, a liquid crystal display or an organic EL (Electro-Luminescence) display, and is a display device for displaying various information.

[0023] For example, when a tour guide acts as the broadcaster and supports the shopping of multiple tourists who are remotely viewing the product, the output unit 13 displays information sent from multiple tourists who are remotely viewing the product. For example, the output unit 13 displays information related to inquiries from tourists, such as "Please pick up the product and take a closer look."

[0024] (360° camera 14) The 360-degree spherical camera 14 captures images of the 360-degree surroundings of the broadcaster T11. Figure 3 shows an example of the broadcaster terminal 10 being mounted according to this embodiment. The 360-degree spherical camera 14 captures images of the 360-degree surroundings with the vertical direction LA1 of the 360-degree spherical camera 14 as the reference point. Specifically, the 360-degree spherical camera 14 captures images of the 360-degree surroundings with the vertical direction LA1 of the 360-degree spherical camera 14 as the Z-axis of the terminal coordinate system described later. Hereafter, the vertical direction LA1 of the 360-degree spherical camera 14 will be referred to as the Z-axis of the terminal coordinate system described later.

[0025] Figure 4 shows an example of a 360-degree spherical image according to the embodiment. The 360-degree spherical image IM1 is a 360-degree spherical image of a specific moment in time captured by the 360-degree camera 14 (for example, a 360-degree spherical image captured in a frame of the video), and each viewer views the video of a specific area of ​​the 360-degree spherical image IM1. For example, viewer U11 views the video of the display area IM11 that viewer U11 wants to view, and viewer U12 views the video of the display area IM12 that viewer U12 wants to view. Note that since the 360-degree spherical image IM1 is captured in 360 degrees, the image is actually distorted, but when displayed on the viewer terminal 20, the image of the distorted image is displayed. Note that the 360-degree camera 14 is installed on the broadcaster terminal 10 and is therefore an imaging device that can move together with the broadcaster terminal 10.

[0026] (Gyro sensor 15) The gyro sensor 15 outputs a gyro value that represents the change in the imaging direction of the 360-degree camera 14 as a feature when the imaging direction of the 360-degree camera 14 changes (for example, including rotation). The imaging direction of the 360-degree camera 14 changes, for example, according to the movement of the broadcaster terminal 10. For example, when the broadcaster bows, the broadcaster terminal 10 rotates forward in accordance with the broadcaster's movement, so the imaging direction of the 360-degree camera 14 changes downward. The gyro value is represented by feature quantities such as vectors, but feature quantities such as direction are output based on the terminal coordinate system described later.

[0027] (Control Unit 16) The control unit 16 is, for example, a controller, and is implemented by a CPU (Central Processing Unit) or MPU (Micro Processing Unit) executing various programs stored in the internal storage device of the broadcaster terminal 10 using RAM (Random Access Memory) as the working area. For example, these various programs include application programs installed on the broadcaster terminal 10. For example, these various programs include application programs that display information transmitted from multiple remote viewers. The control unit 16 is also implemented by an integrated circuit such as an ASIC (Application Specific Integrated Circuit) or an FPGA (Field Programmable Gate Array).

[0028] As shown in Figure 2, the control unit 16 has a transmission unit 161 and realizes or executes the information processing operations described below.

[0029] (Transmitter 161) The transmitting unit 161 transmits the 360-degree images captured by the 360-degree camera 14, for example, frame by frame (or segment by segment) of the video, to the information processing device 100 in real time. The transmitting unit 162 also transmits the gyro value output by the gyro sensor 15 when transmitting the 360-degree images. The transmitting unit 162 also transmits the coordinate information of the 360-degree images captured by the 360-degree camera 14. This coordinate information is based on the coordinate system of the 360-degree camera 14 (hereinafter referred to as the "terminal coordinate system" as appropriate). This coordinate information makes it possible to identify which image of the 360-degree images corresponds to which coordinate in the terminal coordinate system. The terminal coordinate system is the coordinate system of the broadcaster, while the coordinate system of the viewer will be referred to as the "viewer coordinate system" as appropriate.

[0030] Figure 5 is a diagram illustrating the relationship between the coordinate systems (physical space, broadcaster side, and viewer side) according to the embodiment. The left side of Figure 5 (FG1) shows the physical space coordinate system (FG11). The physical space coordinate system corresponds to a global coordinate system based on the center of the Earth FG12, that is, a fixed coordinate system that ignores planetary-scale movements such as the rotation of the Earth's axis.

[0031] The center of Figure 5 (FG2) shows the terminal coordinate system (FG13). The terminal coordinate system (FG13) is a local coordinate system relative to the physical space coordinate system (FG11). The terminal coordinate system (FG13) is a local coordinate system relative to the physical space coordinate system (FG11) based on the orientation of the 360-degree camera 14, and rotates according to the movement of the moving object (in this case, the broadcaster T11) on which the 360-degree camera 14 is mounted.

[0032] The right side of Figure 5 (FG3) shows the viewer coordinate system (FG21), which is the local coordinate system relative to the terminal coordinate system (FG13). The viewer coordinate system (FG21) is a virtual coordinate system used to render the viewer's screen from the 360-degree image. The direction of the virtual camera (viewer's field of view), which is the viewer's viewpoint, rotates based on the viewer coordinate system FG21 through operations such as those performed in a browser. In other words, the viewer coordinate system (FG21) is the coordinate system of the virtual camera, which is the viewer's viewpoint.

[0033] Conventional technologies have made it difficult to meet viewers' demands to fix their field of view relative to the physical space coordinate system (FG11), regardless of the rotation of the terminal coordinate system (FG13).

[0034] [3. Viewer terminal configuration] Next, the configuration of the viewer terminal 20 according to the embodiment will be described using Figure 6. Figure 6 is a diagram showing an example of the configuration of the viewer terminal 20 according to the embodiment. As shown in Figure 6, the viewer terminal 20 has a communication unit 21, an input unit 22, an output unit 23, and a control unit 24.

[0035] (Communications Department 21) The communication unit 21 is implemented, for example, by a NIC (Network Interface Card). The communication unit 21 is connected to a predetermined network N by wire or wireless connection and transmits and receives information with the information processing device 100, etc., via the predetermined network N.

[0036] (Input section 22) The input unit 22 accepts various operations from the viewer. For example, the input unit 22 may accept various operations from the viewer via a touch panel function on the display surface. Alternatively, the input unit 22 may accept various operations from buttons provided on the viewer terminal 20, or from a keyboard or mouse connected to the viewer terminal 20.

[0037] (Output section 23) The output unit 23 is a display screen for a tablet terminal or the like, which is implemented by, for example, a liquid crystal display or an organic EL display, and is a display device for displaying various information.

[0038] For example, the output unit 23 displays the image of the display area in the direction specified by the viewer from the 360-degree image being streamed in real time by the broadcaster, based on the information transmitted by the information processing device 100.

[0039] (Control Unit 24) The control unit 24 is, for example, a controller, and is implemented by a CPU or MPU, which executes various programs stored in the memory device inside the viewer terminal 20 using RAM as a working area. For example, these various programs include application programs installed on the viewer terminal 20. For example, these various programs include application programs that, based on information transmitted by the information processing device 100, display images of the direction specified by the viewer from the 360-degree images being distributed in real time by the broadcaster terminal 10. The control unit 24 is also implemented by an integrated circuit such as an ASIC or FPGA.

[0040] As shown in Figure 6, the control unit 24 includes a receiving unit 241, a display control unit 242, and a transmitting unit 243, and realizes or executes the information processing operations described below.

[0041] (Receiver 241) The receiving unit 241 receives information transmitted by the information processing device 100. Specifically, the receiving unit 241 receives the 360-degree image being streamed in real time by the broadcasting terminal 10, and information for displaying the image in the direction specified by the viewer from within the 360-degree image (for example, coordinate information of the 360-degree image).

[0042] (Display control unit 242) Based on the information received by the receiving unit 241, the display control unit 242 displays the image on the output unit 23 that corresponds to the direction specified by the viewer from the 360-degree image being streamed in real time by the broadcasting terminal 10. The direction of the image displayed on the output unit 23 can be changed by the viewer specifying a new direction for the display area.

[0043] Here, if the 360-degree camera 14 rotates while the viewer is watching the video displayed on the output unit 23, the terminal coordinate system rotates in accordance with the rotation of the 360-degree camera 14. Therefore, even if the video displayed was in a specific direction in the physical space coordinate system before the rotation of the 360-degree camera 14, the video displayed will be in a different direction from that specific direction in the physical space coordinate system after the rotation of the 360-degree camera 14. Consequently, even if the viewer wants to continue watching the video in a specific direction in the physical space coordinate system, they will no longer be able to continue watching the video in that direction once the 360-degree camera 14 rotates.

[0044] For example, if viewer U11 is watching the video in display area IM11 and broadcaster T11 bows, the 360-degree camera 14 rotates forward along with broadcaster T11's movement. As a result, the terminal coordinate system, which is the coordinate system of the 360-degree camera 14, also rotates forward, and the video from display area IM13 (Figure 4) is displayed on the output unit 23 instead of the video from display area IM11.

[0045] (Transmitter 243) The transmitting unit 243 transmits information received from the viewer via the input unit 22 to the information processing device 100. For example, the transmitting unit 243 transmits specification information indicating the direction specified by the viewer for viewing. For example, if the viewer wants to view the image corresponding to the direction of the object wearing the hat HA11 in the 360-degree image IM1, specifying the direction of the display area IM12 (for example, by tapping or clicking) will cause the transmitting unit 243 to transmit specification information indicating the direction of the display area IM12. If the viewer newly specifies a direction different from the direction of the display area IM12, the transmitting unit 243 will transmit specification information indicating the different direction. In this case, the image of the newly specified direction will be displayed to the viewer. The direction indicated by the specification information corresponds to the direction in the viewer coordinate system. This point will be explained later.

[0046] [4. Configuration of the Information Processing Device] Next, the configuration of the information processing device 100 according to the embodiment will be described with reference to Figure 7. Figure 7 is a diagram showing an example of the configuration of the information processing device 100 according to the embodiment. As shown in Figure 7, the information processing device 100 has a communication unit 110 and a control unit 120. The information processing device 100 may also have an input unit (for example, a keyboard or mouse) that receives various operations from the administrator of the information processing device 100, and a display unit (for example, a liquid crystal display) for displaying various information.

[0047] (Communications Department 110) The communication unit 110 is implemented, for example, by a NIC (Network Interface Card). The communication unit 110 is connected to the network N by wire or wireless connection and transmits and receives information with the broadcaster terminal 10, etc., via the network N.

[0048] (Control unit 120) The control unit 120 is a controller, and is implemented, for example, by a CPU or MPU executing various programs stored in the memory device inside the information processing device 100 using RAM as the working area. Alternatively, the control unit 120 is a controller and can be implemented, for example, by an integrated circuit such as an ASIC or FPGA.

[0049] As shown in Figure 7, the control unit 120 includes an acquisition unit 121, a determination unit 122, and a transmission unit 123, and realizes or executes the information processing operations described below. Note that the internal configuration of the control unit 120 is not limited to the configuration shown in Figure 7, and other configurations are also acceptable as long as they perform the information processing described later.

[0050] (Acquisition part 121) The acquisition unit 121 acquires various information from other information processing devices such as the broadcaster terminal 10. Specifically, the acquisition unit 121 acquires change information related to changes in the imaging direction of the 360-degree camera 14. For example, as change information, the acquisition unit 121 acquires gyro values ​​indicating changes in the imaging direction of the 360-degree camera 14 from the information processing device 100. Then, based on the gyro values, the acquisition unit 121 acquires the rotation direction and amount of rotation of the 360-degree camera 14. In addition, the acquisition unit 121 acquires specification information from the viewer terminal 20, which is the direction specified by the viewer for viewing.

[0051] (Decision Section 122) The determination unit 122 determines the area in the direction specified by the viewer from the 360-degree spherical image captured by the 360-degree camera 14 as the display area.

[0052] As will be explained in detail later, when the 360-degree camera 14 captures an image in a second imaging direction different from the first imaging direction, the determination unit 122 determines, based on change information regarding the change from the first imaging direction to the second imaging direction, the region of the second 360-degree image that corresponds to the first display region of the first 360-degree image as the second display region. For example, when the 360-degree camera 14 rotates, it captures an image in a second imaging direction different from the first imaging direction. In this case, because the imaging direction changes from the first imaging direction to the second imaging direction, the coordinate information of the direction of object OB11 in the terminal coordinate system before the change in imaging direction is different from the coordinate information of the direction of object OB11 after the change in imaging direction. Based on the change in imaging direction information, the determination unit 122 determines, based on change information regarding the change in imaging direction, the region of the second 360-degree image that corresponds to the first display region of the first 360-degree image as the second display region.

[0053] Figure 8 shows an example of how the displayed image on the viewer's side changes in response to a change in the imaging direction. When the 360-degree camera 14 rotates, the terminal coordinate system rotates in accordance with the rotation of the 360-degree camera 14. For example, even if the 360-degree camera 14 is imaging in the south direction as the first imaging direction in the physical space coordinate system, and viewer U11 is viewing the image in the direction of object OB11, when the 360-degree camera 14 rotates in the east direction (step S11), the terminal coordinate system AX1 rotates, and the image in the direction of object OB11 that viewer U11 was viewing is no longer displayed (step S12).

[0054] The determination unit 122 determines a region in a specific direction from the first 360-degree image as the first display region. For example, the determination unit 122 determines a region in the direction specified by the viewer from the first 360-degree image as the first display region. The determination unit 122 also determines a region in the direction of the first display region from the second 360-degree image, which has been rotated in the opposite direction to the change from the first imaging direction to the second imaging direction, as the second display region. For example, when the imaging direction changes from the first imaging direction to the second imaging direction, the coordinate information of the direction of object OB11 in the terminal coordinate system before the change in imaging direction is different from the coordinate information of the direction of object OB11 after the change in imaging direction. Therefore, based on the information of the change in imaging direction, the determination unit 122 determines a region in the direction of object OB11 from the second 360-degree image, which has been rotated in the opposite direction to the change from the first imaging direction to the second imaging direction, as the second display region.

[0055] (Transmitter 123) The transmitting unit 123 transmits information (for example, coordinate information of the 360-degree image) for displaying the image of a first display area in a specific direction from the 360-degree image, based on the specified information acquired by the acquisition unit 121. Specifically, the transmitting unit 123 transmits information for displaying the image of a first display area identified based on the coordinate information of the direction specified by the viewer.

[0056] Furthermore, the transmission unit 123 transmits information for displaying the image of the second display area determined by the determination unit 122. The size of the first display area (or second display area) is predetermined for each viewer terminal 20. Based on the predetermined size, the image of the first display area (or second display area) of the 360-degree image is displayed on the viewer terminal 20. When the viewer terminal 20 receives the information transmitted by the transmission unit 123, it displays the image on the output unit 23 based on the received information.

[0057] For example, in the example shown in Figure 4, viewer U11 specifies the direction of the display area IM11 for viewing, so the transmission unit 123 transmits information to display the image IM1 corresponding to the coordinate information of the specified direction on the viewer terminal 201. At the same time, viewer U12 specifies the direction of the display area IM12 for viewing, so the transmission unit 123 transmits information to display the image IM1 corresponding to the coordinate information of the specified direction on the viewer terminal 202.

[0058] The following describes the processing performed by the information processing device 100. Prior to describing the processing performed by the information processing device 100, Figures 9 to 11 will be used to explain the case in which the direction of the image provided to the viewer changes according to the rotation of the 360-degree camera 14. Subsequently, Figures 12 and 13 will be used to explain the processing performed by the information processing device 100.

[0059] Figure 9 shows the relationship between the terminal coordinate system before the change in imaging direction and the surrounding objects. Coordinate system AX11 is the physical space coordinate system. For example, it shows that object OB11 is in the south direction relative to the broadcaster T11. Objects OB11, OB12, and OB13 are objects that appear in the 360-degree image captured by the 360-degree camera 14. Coordinate system AX12 is the terminal coordinate system.

[0060] Figure 10 shows the relationship between the viewer coordinate system and the image displayed on the viewer's side. The determination unit 122 expands the 360-degree image into the viewer coordinate system AX13 based on the coordinate information of the 360-degree image. Specifically, each coordinate of the 360-degree image corresponds to each coordinate of the viewer coordinate system AX13. For example, the front direction of the 360-degree camera 14 corresponds to the coordinates (x1, y1, z1) of the 360-degree image, and furthermore, the coordinates (x1, y1, z1) of the 360-degree image correspond to the coordinates (X1, Y1, Z1) of the viewer coordinate system AX13. In this case, the determination unit 122 expands the pixels corresponding to the coordinates (x1, y1, z1) of the 360-degree image captured in the front direction of the 360-degree camera 14 into the coordinates (X1, Y1, Z1) of the viewer coordinate system AX13. In the example shown in Figure 10, the coordinates of the spherical image corresponding to the front direction of the spherical camera 14 (the direction in which object OB11 was captured) are assumed to correspond to the downward direction of the viewer coordinate system AX13 (the southern hemisphere direction of the sphere on which the spherical image is unfolded). In this way, the determination unit 122 unfolds the spherical image into the viewer coordinate system AX13 based on the correspondence between each coordinate of the spherical image (i.e., the coordinates of the terminal coordinate system) and each coordinate of the viewer coordinate system AX13.

[0061] The camera CA11 shown in Figure 10 is a virtual camera (viewer's field of view) representing the viewer's viewpoint. In the example shown in Figure 10, the designated information specified by viewer U11 indicates the direction of object OB11, so camera CA11 is oriented towards object OB11. In this case, a predetermined area (mesh portion) in the direction of object OB11 becomes the display area, and the image is displayed on the viewer terminal 20 according to the display area. Although not shown, if the designated information specified by viewer U12 at the same time indicates the direction of object OB13, camera CA11 is oriented towards object OB13. In this case, a predetermined area (not shown) in the direction of object OB13 becomes the display area, and the corresponding image is displayed. Areas other than the display area are hidden areas. Just as the display area differs for each viewer, the hidden area also differs. The image is displayed on the viewer terminal 20 based on the 360-degree image and the orientation of camera CA11. Thus, the designated information specified by the viewer indicates a specific direction in the viewer coordinate system.

[0062] Figure 11 shows the relationship between the terminal coordinate system after a change in imaging direction and the viewer's displayed image when there is no field of view fixation (i.e., when the image is not fixed in the direction indicated by the specified information). Explanations similar to those for Figures 9-10 are omitted as appropriate.

[0063] In Figure 11, initially, the broadcaster T11 (i.e., facing the 360-degree camera 14) was facing south. However, after broadcaster T11 discovered object OB13, which is a ramen shop, they changed direction to face west of object OB13 (step S21). In this case, because broadcaster T11 changed direction from south to west, the direction of the 360-degree camera 14 also rotates from south to west. As the terminal coordinate system rotates clockwise, the viewer terminal 201 of viewer U11, which was viewing the image in the direction of object OB11, no longer displays the image in the direction of object OB11. Furthermore, because broadcaster T11 changed direction by 90 degrees from south to west, the image in the direction of object OB13 is now displayed on the viewer terminal 201. As a result, viewer U11 wonders, "Where's the tower?"

[0064] Let's explain this in detail using the example in Figure 10. As explained in Figure 10, the front direction of the 360-degree camera 14 corresponds to the downward direction of the viewer coordinate system AX13 (the southern hemisphere direction of the sphere on which the 360-degree image is unfolded). That is, if the front direction of the 360-degree camera 14 rotates from south to west, the image of the west direction captured by the 360-degree camera 14 will be unfolded downward in the viewer coordinate system AX13. In this case, since the direction specified by the viewer U11 is downward in the viewer coordinate system AX13, the viewer terminal 201 receives the image in the direction of object OB13, and viewer U11 sees the image in the direction of object OB13.

[0065] In the example shown in Figure 11, at the same time, viewer U12, who was viewing the video in the direction of object OB13, will no longer see the video in the direction of object OB13 on their viewer terminal 202, and the video in the direction of object OB12 will be displayed on the viewer terminal 202. As a result, viewer U12 will think, "I can no longer see the ramen shop."

[0066] Thus, if there is no fixed field of view to keep the viewer watching images in a specific direction, the rotation of the terminal coordinate system causes images in directions corresponding to the change in the imaging direction to be displayed, giving the viewer the experience of their field of view rotating.

[0067] Figures 12 and 13 are diagrams illustrating the processing performed by the information processing device 100 according to this embodiment. Note that explanations similar to those given in Figures 9 to 11 will be omitted as appropriate.

[0068] In Figure 12, in step S21, the region of the second 360-degree image obtained after rotating the 360-degree camera 14 in the opposite direction (counterclockwise) to cancel out the change in the imaging direction of the 360-degree camera 14 is determined as the second display region, so that the image in a specific direction continues to be displayed. For example, the viewer terminal 201 continues to see the image in the direction of object OB11. Therefore, viewer U11 can watch while thinking, "I'd like to go to object OB11 someday."

[0069] Furthermore, viewer terminal 202 of viewer U12, who was viewing the video in the direction of object OB13, continues to display the video in the direction of object OB13. As a result, viewer U11 can watch while letting their imagination run wild, thinking, "The ramen here looks delicious."

[0070] This point will be explained in detail using Figure 13.

[0071] First, as a premise for explaining Figure 13, it is assumed that viewer U11 wishes to view in the south direction (the direction in which object OB11 is located) as shown in Figure 9. That is, the direction specified by viewer U11 corresponds to the south direction in the physical space coordinate system and corresponds to the downward direction (the area indicating the display area) in the viewer coordinate system AX13 shown in Figure 10. In this case, as long as the front of the 360-degree camera 14 is facing south, the image that viewer U11 wants to view will be displayed in the downward direction of the viewer coordinate system AX13. For this reason, the determination unit 122 determines the area of ​​the 360-degree image (corresponding to the first 360-degree image) displayed in the viewer coordinate system AX13 that corresponds to the direction specified by the viewer as the display area (corresponding to the first display area), as shown in the example in Figure 10. Thus, if the 360-degree camera 14 is capturing images in the same direction as the imaging direction of the 360-degree camera 14 (corresponding to the first imaging direction) at the time the viewer specifies the viewing direction, the determination unit 122 can determine the area corresponding to the specified information from the 360-degree image unfolded in the viewer coordinate system AX13 as the display area, thereby determining the image in the direction desired by the viewer as the display area.

[0072] Next, let's assume that the imaging direction of the 360-degree camera 14 is rotated, as shown in the example in Figure 12. That is, let's assume that the 360-degree camera 14 is imaged in an imaging direction (corresponding to the second imaging direction) that is different from the imaging direction of the 360-degree camera 14 at the time the viewer specifies the viewing direction (corresponding to the first imaging direction). In this case, as shown in the example on the left side of Figure 13, the 360-degree image captured in the second imaging direction (corresponding to the second 360-degree image) is rotated and unfolded in the viewer coordinate system AX13 in accordance with the rotation of the 360-degree camera 14. Specifically, the 360-degree image shown in the left example of Figure 13 is rotated 90 degrees in direction R2 and unfolded in the viewer coordinate system AX13 compared to the 360-degree image shown in Figure 10.

[0073] Therefore, based on the rotation direction and amount of the 360-degree camera 14, which are change information from the first imaging direction to the second imaging direction, the 360-degree image shown on the left side of Figure 13 is rotated in the viewer coordinate system. Specifically, as shown in the middle of Figure 13, the determination unit 122 rotates the 360-degree image in direction R1 to cancel out the rotation of the 360-degree image in direction R2 corresponding to the rotation of the 360-degree camera 14.

[0074] Then, as shown on the right side of Figure 13, the determination unit 122 determines the region of the rotated 360-degree image corresponding to the direction specified by the viewer as the display region (mesh portion). At this time, the determination unit 122 identifies the coordinate information of the 360-degree image corresponding to the display region. As a result, the transmission unit 123 transmits the coordinate information corresponding to the display region to the viewer terminal 20. Note that the coordinate information of the 360-degree image corresponding to the display region referred to here is not the coordinates in the viewer coordinate system, but the coordinates of the 360-degree image (x1, y1, z1, etc.) as described above.

[0075] [5. Information Processing Flow] Next, the information processing procedure by the information processing system 1 according to the embodiment will be explained using Figure 14. Figure 14 is a flowchart showing the information processing procedure by the information processing system 1 according to the embodiment.

[0076] As shown in Figure 14, the information processing device 100 acquires change information regarding the change in the imaging direction of the 360-degree camera 14 (step S101).

[0077] The information processing device 100 determines, as the first display area, from the first 360-degree image captured by the 360-degree camera 14 in a first imaging direction, the area corresponding to the direction specified by the viewer (step S102).

[0078] When the 360-degree camera 14 captures an image in a second imaging direction different from the first imaging direction, the information processing device 100 determines, based on change information regarding the change from the first imaging direction to the second imaging direction, the region of the second 360-degree image corresponding to the second imaging direction that corresponds to the first display region of the first 360-degree image as the second display region (step S103).

[0079] [6. Variant Example] The information processing system 1 according to the above embodiment may be implemented in various other forms besides the embodiment described above. Therefore, other embodiments of the information processing system 1 will be described below.

[0080] In the above embodiment, the information processing device 100 transmits the rotated second 360-degree image to the viewer terminal 20, and the viewer terminal 20 extracts an image from the received second 360-degree image that corresponds to the second display area and displays the video. However, the embodiment is not limited to this example. For example, the information processing device 100 may transmit to the viewer terminal 20 an image extracted from the rotated second 360-degree image based on the second display area. The information processing device 100 may then continue to transmit the extracted image in real time so that the viewer terminal 20 can continuously display the video corresponding to the second display area.

[0081] In the above embodiment, the case where the information processing device 100 and the viewer terminal 20 are separate devices is shown, but the information processing device 100 and the viewer terminal 20 may be integrated. In this case, the viewer terminal 20 receives the 360-degree image distributed via the broadcasting terminal 10. The viewer terminal 20 then rotates the second 360-degree image in the opposite direction to the rotation of the 360-degree camera 14 in accordance with the gyro value to cancel out the rotation of the 360-degree camera 14. The viewer terminal 20 then displays the image cropped from the rotated second 360-degree image based on the second display area. Alternatively, the broadcasting terminal 10 may continue to distribute the 360-degree image in real time so that the viewer terminal 20 continues to display the image corresponding to the second display area.

[0082] In the above embodiment, the case in which the broadcaster terminal 10 and the 360-degree camera 14 are integrated is shown, but the embodiment is not limited to this example. For example, if the broadcaster terminal 10 and the 360-degree camera 14 are not integrated, the 360-degree camera 14 shall be attached to the broadcaster separately from the broadcaster terminal 10. In this case, the mounting position (physical arrangement) of the 360-degree camera 14 shall be fixed.

[0083] In the above embodiment, the case in which the broadcaster terminal 10 and the gyro sensor 15 are integrated is shown, but the example is not limited to this. For example, if the broadcaster terminal 10 and the gyro sensor 15 are not integrated, the gyro sensor 15 shall be attached to the broadcaster separately from the broadcaster terminal 10. In this case, the mounting position of the gyro sensor 15 shall be fixed.

[0084] In the above embodiment, the broadcaster using the broadcaster terminal 10 is shown to be a person. However, the broadcaster is not limited to a person, as long as it is a mobile body to which the 360-degree camera 14 and the gyro sensor 15 can be connected. For example, the broadcaster may be a robot, a car, or the like. If the broadcaster is not a person, the broadcaster and the broadcaster terminal 10 may be integrated. In this case, the change in imaging direction according to the above embodiment can be interpreted as a change in the direction of travel.

[0085] [7. Effects] As described above, the information processing device 100 according to the embodiment includes an acquisition unit 121 and a determination unit 122. The acquisition unit 121 acquires change information relating to the change in the imaging direction of the 360-degree camera 14 installed on the moving object. The determination unit 122 determines the region of the first 360-degree image captured by the 360-degree camera 14 in the first imaging direction that corresponds to the direction specified by the viewer as the first display region. Furthermore, when the 360-degree camera 14 captures an image in a second imaging direction different from the first imaging direction, the determination unit 122 determines the region of the second 360-degree image corresponding to the second imaging direction that corresponds to the first display region of the first 360-degree image as the second display region, based on the change information relating to the change from the first imaging direction to the second imaging direction.

[0086] As a result, even if the imaging direction of the 360-degree camera 14 changes, the information processing device 100 according to the embodiment can determine the area of ​​the second 360-degree image that corresponds to the first display area of ​​the first 360-degree image as the first display area, thereby meeting the demands of viewers who want to continue viewing images in a specific direction.

[0087] Furthermore, the determination unit 122 determines that the image in the first imaging direction in the physical spatial coordinate system where the 360-degree camera is located, from among the second 360-degree images, will be used as the second display area.

[0088] As a result, even if the imaging direction of the 360-degree camera 14 changes, the information processing device 100 according to the embodiment can identify the image from the second 360-degree image in the first imaging direction based on the physical spatial coordinate system, thus meeting the demands of viewers who want to continue viewing images in a specific direction.

[0089] Furthermore, the determination unit 122 determines a region in a specific direction from the first 360-degree image as the first display region, and determines a region in a specific direction from the second 360-degree image, which has been rotated in the opposite direction to the change from the first imaging direction to the second imaging direction, as the second display region.

[0090] As a result, the information processing device 100 according to the embodiment can be rotated in the opposite direction to the change in the imaging direction of the 360-degree camera 14, so that the viewer coordinate system can be fixed with respect to the physical space coordinate system, and as a result it is possible to meet the demands of viewers who want to continue viewing images in a specific direction.

[0091] Furthermore, the determination unit 122 determines the display area from the 360-degree image corresponding to each frame, based on the change information, for each frame of the video captured by the 360-degree camera 14.

[0092] As a result, the information processing device 100 according to the embodiment can determine the display area for each frame of the video, and even if the imaging direction of the 360-degree camera 14 changes, it can perform the processing according to the embodiment for each frame, and as a result it can meet the demands of viewers who want to continue viewing video in a specific direction.

[0093] [8. Hardware Configuration] Furthermore, the information processing device 100 according to the above-described embodiment is realized by a computer 1000 having a configuration such as that shown in Figure 15. Figure 15 is a hardware configuration diagram showing an example of a computer that realizes the functions of the information processing device 100. The computer 1000 includes a CPU 1100, RAM 1200, ROM 1300, HDD 1400, communication interface (I / F) 1500, input / output interface (I / F) 1600, and media interface (I / F) 1700.

[0094] The CPU 1100 operates based on programs stored in the ROM 1300 or HDD 1400, controlling various components. The ROM 1300 stores boot programs executed by the CPU 1100 when the computer 1000 starts up, as well as programs that depend on the computer 1000's hardware.

[0095] The HDD1400 stores programs executed by the CPU1100, as well as data used by such programs. The communication interface1500 receives data from other devices via a predetermined communication network and sends it to the CPU1100, and transmits data generated by the CPU1100 to other devices via the predetermined communication network.

[0096] The CPU 1100 controls output devices such as displays and printers, and input devices such as keyboards and mice, via the input / output interface 1600. The CPU 1100 acquires data from input devices via the input / output interface 1600. The CPU 1100 also outputs the generated data to output devices via the input / output interface 1600.

[0097] The media interface 1700 reads a program or data stored in the recording medium 1800 and provides it to the CPU 1100 via the RAM 1200. The CPU 1100 loads the program from the recording medium 1800 onto the RAM 1200 via the media interface 1700 and executes the loaded program. The recording medium 1800 is, for example, an optical recording medium such as a DVD (Digital Versatile Disc) or PD (Phase Change Rewritable Disk), a magneto-optical recording medium such as an MO (Magneto-Optical disk), a tape medium, a magnetic recording medium, or a semiconductor memory.

[0098] For example, when computer 1000 functions as an information processing device 100, the CPU 1100 of computer 1000 realizes the functions of the control unit 130 by executing programs loaded onto RAM 1200. The CPU 1100 of computer 1000 reads and executes these programs from the recording medium 1800, but as another example, these programs may be obtained from other devices via a predetermined communication network.

[0099] Although some embodiments of the present invention have been described in detail above with reference to the drawings, these are illustrative examples, and the present invention can be implemented in various other forms with modifications and improvements based on the knowledge of those skilled in the art, starting with the embodiments described in the disclosure section of the invention.

[0100] [9. Other] Furthermore, among the processes described in the above embodiments and modifications, all or part of the processes described as being performed automatically can be performed manually, or all or part of the processes described as being performed manually can be performed automatically by known methods. In addition, the processing procedures, specific names, and information including various data and parameters shown in the above document and drawings can be changed at will unless otherwise specified. For example, the various information shown in each figure is not limited to the information shown.

[0101] Furthermore, the components of each illustrated device are functionally conceptual and do not necessarily need to be physically configured as shown. In other words, the specific forms of distribution and integration of each device are not limited to those shown, and all or part of them can be functionally or physically distributed and integrated in any unit according to various loads and usage conditions.

[0102] Furthermore, the embodiments and modifications described above can be combined as appropriate, provided that the processing content is not inconsistent.

[0103] Furthermore, the terms "section, module, unit" mentioned above can be replaced with "means" or "circuit," etc. For example, a specific section can be replaced with a specific means or a specific circuit. [Explanation of Symbols]

[0104] 1. Information Processing System 10. Streamer's device 20 Viewer terminals 100 Information Processing Devices 110 Communications Department 121 Acquisition Department 122 Decision Section 123 Transmitter N Network

Claims

1. An acquisition unit that acquires change information regarding changes in the imaging direction of a 360-degree camera installed on a moving object, The system includes a determination unit that determines a region of the first 360-degree image captured by the aforementioned 360-degree camera in a first imaging direction, corresponding to a direction specified by the viewer, as the first display region. The aforementioned determination unit, When the imaging direction of the 360-degree camera changes from the first imaging direction, which is the imaging direction when the first display area is determined, to a second imaging direction different from the first imaging direction, and a second 360-degree image is captured by the 360-degree camera, the second 360-degree image is rotated in the opposite direction to the change from the first imaging direction to the second imaging direction based on the relationship between change information regarding the change from the first imaging direction to the second imaging direction, the coordinate system of the 360-degree camera which is the terminal coordinate system on the side of the moving object that rotates according to the movement of the moving object, and the coordinate system for rendering the 360-degree image captured by the 360-degree camera onto the viewer's screen, which is the viewer coordinate system on the side of the viewer whose viewpoint rotates according to the viewer's operation, and the region of the rotated second 360-degree image corresponding to the first display area of ​​the first 360-degree image is determined as the second display area. An information processing device characterized by the following:

2. The aforementioned determination unit, Based on the change information relating to the change from the first imaging direction to the second imaging direction, the second 360-degree image is rotated and corrected so that there is no discrepancy between the coordinate information of a specific object in the first 360-degree image and the coordinate information of the same specific object in the second 360-degree image, and the region corresponding to the first display region of the first 360-degree image is determined to be the second display region. The information processing apparatus according to feature 1.

3. The aforementioned determination unit, For each frame of the image captured by the aforementioned 360-degree camera, the display area of ​​the 360-degree image corresponding to each frame is determined based on the change information. The information processing apparatus according to claim 1 or 2.

4. A method of information processing performed by a computer, An acquisition process to acquire change information regarding changes in the imaging direction of a 360-degree camera installed on a moving object, The process includes a determination step of determining a region of the first spherical image captured by the spherical camera in a first imaging direction, corresponding to a direction specified by the viewer, as the first display region. The aforementioned decision-making process is, When the imaging direction of the 360-degree camera changes from the first imaging direction, which is the imaging direction when the first display area is determined, to a second imaging direction different from the first imaging direction, and a second 360-degree image is captured by the 360-degree camera, the second 360-degree image is rotated in the opposite direction to the change from the first imaging direction to the second imaging direction based on the relationship between change information regarding the change from the first imaging direction to the second imaging direction, the coordinate system of the 360-degree camera which is the terminal coordinate system on the side of the moving object that rotates according to the movement of the moving object, and the coordinate system for rendering the 360-degree image captured by the 360-degree camera onto the viewer's screen, which is the viewer coordinate system on the side of the viewer whose viewpoint rotates according to the viewer's operation, and the region of the rotated second 360-degree image corresponding to the first display area of ​​the first 360-degree image is determined as the second display area. An information processing method characterized by the following:

5. A procedure for acquiring change information regarding changes in the imaging direction of a 360-degree camera installed on a moving object, Of the first spherical image captured by the spherical camera in a first imaging direction, the region corresponding to the direction specified by the viewer is determined as the first display region, and when the spherical camera captures a second spherical image while the imaging direction of the spherical camera has changed from the first imaging direction, which is the imaging direction when the first display region was determined, to a second imaging direction different from the first imaging direction, change information regarding the change from the first imaging direction to the second imaging direction and the moving body rotating in accordance with the movement of the moving body A determination procedure for determining the second 360-degree image as the second display area, based on the relationship between the coordinate system of the 360-degree camera, which is the terminal coordinate system on the user's side, and the coordinate system for rendering the 360-degree image captured by the 360-degree camera onto the user's screen, which is the viewer coordinate system on the user's side, where the viewer's viewpoint rotates according to the user's operation, by rotating the second 360-degree image in the opposite direction to the change from the first imaging direction to the second imaging direction, and determining the area of ​​the rotated second 360-degree image corresponding to the first display area of ​​the first 360-degree image as the second display area, An information processing program characterized by causing a computer to execute it.

Images