Information processing device, information processing method, reproduction system, and program

Abstract

To provide high presence to a viewer of a moving image.SOLUTION: An information processing device 200 acquires data of a moving image obtained by imaging using an imaging apparatus, time sequential angle-of-view data indicating a time sequential angle of view of the moving image and time sequential detection data indicating time sequential detection results during imaging of the moving image by a detection device for detecting elements corresponding to a stimulus with respect to at least any one of olfactory sense, skin tactile sense and hearing sensor that a natural person may feel, in an imaging environment of the imaging apparatus. On the basis of the time sequential angle-of-view data, the information processing device changes a part of or all a plurality of time stamps included in the time sequential detection data.SELECTED DRAWING: Figure 6

Description

Technical Field

[0001] The present disclosure relates to a technique for pseudo-reproducing a physical sensation in an imaging environment of a moving image in synchronization with the reproduction of the moving image.

Background Art

[0002] There is a technique for reproducing a physical sensation other than a stimulus to vision received by a natural person, such as vibration, wind, smell, or voice, when playing a moving image. According to such a technique, a higher sense of presence can be given to a person (hereinafter referred to as a "viewer") who views the moving image. Patent Document 1 discloses a technique for adjusting the level of an audio signal in an imaging environment acquired at the time of imaging a moving image based on the zoom amount of an imaging device also used for imaging. According to the technique disclosed in Patent Document 1, it is possible to eliminate the discomfort of the viewer caused by the difference between the distance from a sound collector arranged near the imaging device to the sound source and the distance from a pseudo-viewing position in the imaging environment recognized by the viewer based on the angle of view of the moving image to the sound source.

Prior Art Documents

Patent Documents

[0003]

Patent Document 1

Summary of the Invention

Problems to be Solved by the Invention

[0004] Patent Document 1 discloses adjusting the level of an audio signal, but does not disclose a technique for adjusting the intensity of stimuli other than hearing, such as smell or skin sensation. Also, Patent Document 1 discloses adjusting the level of an audio signal, but does not disclose a technique for adjusting the timing of the generation of elements that stimulate hearing, smell, and skin sensation. For example, sound, smell, or wind has a slower transmission speed compared to the transmission speed of light. Due to this difference in transmission speed, there is a difference between the period until sound, smell, or wind reaches various sensors arranged near the imaging device and the period until it reaches a pseudo-viewing position in the imaging environment recognized by the viewer based on the viewing angle of the moving image. Therefore, the conventional technology has a problem of giving the viewer a sense of discomfort and not being able to provide the viewer with a sufficient sense of presence.

Means for Solving the Problem

[0005] In order to solve the above problems, an information processing apparatus according to the present disclosure includes moving image data obtained by imaging with an imaging device, time-series viewing angle data indicating the time-series viewing angles of the moving image, and time-series detection data indicating time-series detection results during imaging of the moving image by a detection device that detects elements corresponding to stimuli for at least any one of smell, skin sensation, and hearing that a natural person can feel in the imaging environment of the imaging device. An acquisition unit that acquires the data, and a change unit that changes some or all of a plurality of timestamps included in the time-series detection data based on the time-series viewing angle data.

Advantages of the Invention

[0006] According to the present disclosure, a high sense of presence can be provided to a viewer of a moving image.

Brief Description of the Drawings

[0007]

Figure 1

Figure 2

Figure 3

Figure 4

Figure 5

Figure 6

Figure 7

Figure 8

Figure 9

Figure 10

Figure 11

Mode for Carrying Out the Invention

[0008] Hereinafter, embodiments according to the present disclosure will be described with reference to the drawings. The following embodiments do not limit the configuration of the solution according to the present disclosure, and not all of the combinations of features described in this embodiment are essential for the solution of the present disclosure.

[0009] [Conventional Reproduction System] Before describing embodiments according to the present disclosure, a conventional playback system will be described with reference to FIG. 1. FIG. 1 is a diagram showing an example of the configuration and data flow of a conventional playback system. The conventional playback system includes an imaging device 120, and a playback device 111, a display device 112, and a generation device 113 arranged in a playback environment. The imaging device 120 includes a detection device 103 that detects an element (hereinafter referred to as a "somatosensory element") corresponding to a stimulus to at least one of the hearing, smell, and skin touch of a natural person.

[0010] Here, when the detection device 103 detects a somatosensory element corresponding to a stimulus to the hearing of a natural person, the detection device 103 is a sound collector such as a microphone. When the detection device 103 detects a somatosensory element corresponding to a stimulus to the smell of a natural person, the detection device 103 is an odor sensor. When the detection device 103 detects a somatosensory element corresponding to a stimulus to the skin touch of a natural person, the detection device 103 is a wind direction and wind speed sensor or a water droplet sensor. The imaging device 120 does not necessarily have to have the detection device 103. For example, a configuration in which the imaging device 120 and the detection device 103 are communicably connected to each other may be used. Hereinafter, the imaging device 120 will be described as having the detection device 103.

[0011] The imaging device 120 captures an object 101 to be imaged in an imaging environment and generates and outputs moving image data 105. The detection device 103 detects a somatosensory element to be detected during part or all of the period in which the imaging device 120 captures the object 101, and outputs time-series detection data (hereinafter referred to as "time-series detection data 106") as a detection result. The imaging device 120 and the detection device 103 are synchronized with each other, and the same time stamp is assigned to the frame data and the detection data corresponding to the same time.

[0012] The detection device 103 is not limited to outputting time-series detection data 106 as a detection result. For example, it may output the detection result as a waveform by an electrical signal. In this case, the imaging device 120 first acquires the electrical signal output by the detection device 103 and digitizes it by performing A / D conversion on the acquired electrical signal. Subsequently, the imaging device 120 generates time-series detection data 106 by associating the digitized detection result with a time stamp similar to the time stamp given to the frame data, and outputs the generated time-series detection data 106. Hereinafter, the imaging device 120 will be described as outputting a data set 104 including moving image data 105 and time-series detection data 106. Note that the data set 104 may be output as one moving image file with detection data including the moving image data 105 and the time-series detection data 106.

[0013] The playback device 111 acquires the data set 104 output from the imaging device 120 and plays back the moving image data 105 and the time-series detection data 106 included in the data set 104 based on the time stamps included in these data. The playback device 111 transmits an image signal obtained by playing back the moving image data 105 to the display device 112. The display device 112 receives the image signal transmitted from the playback device 111 and displays the image indicated by the received image signal. Further, the playback device 111 transmits a detection signal obtained by playing back the time-series detection data 106 to the generating device 113. The generating device 113 receives the detection signal transmitted from the playback device 111 and generates a somatosensory element based on the received detection signal. Specifically, for example, when the time-series detection data 106 is based on a detection device 103 that detects a somatosensory element corresponding to a stimulus to a natural person's sense of smell, the generating device 113 releases a chemical substance that reproduces the smell corresponding to the received detection signal into the playback environment. The viewer 114 can obtain a higher sense of immersion compared to the case of receiving only the visual stimulus by the image, by receiving not only the visual stimulus by the image displayed on the display device 112 but also the stimulus by the somatosensory element generated from the generating device 113.

[0014] [Embodiment 1] Embodiment 1 according to the present disclosure will be described with reference to FIGS. 2 to 11. FIG. 2 is a diagram showing an example of the configuration and data flow of a playback system (hereinafter simply referred to as the "playback system") according to Embodiment 1. The playback system includes an imaging device 220, an information processing device 200, and a playback device 111, a display device 112, and a generation device 113 arranged in a playback environment. Hereinafter, in FIG. 2, the same components as those in FIG. 1 are given the same reference numerals and the description thereof is omitted. The imaging device 220 has a detection device 103 in the same manner as the imaging device 120, and has the following functions in addition to the same functions as the imaging device 120 shown in FIG. 1. Specifically, the imaging device 220 has a zoom mechanism for changing the focal length, and acquires a zoom value in the zoom mechanism during part or all of the period of imaging the object 101, and outputs time-series angle-of-view data including angle-of-view data corresponding to the zoom value at each time point. Hereinafter, the time-series angle-of-view data will be described as time-series angle-of-view data 207.

[0015] That is, the imaging device 220 outputs a data set 204 including moving image data 105, time-series detection data 106, and time-series angle-of-view data 207. Note that the angle-of-view data at each time point included in the time-series angle-of-view data 207 is associated with the time stamp at the corresponding time point in the frame data included in the moving image data. Note that the data set 204 may be output as one moving image file with angle-of-view and detection data including the moving image data 105, the time-series detection data 106, and the time-series angle-of-view data 207. Details of the imaging device 220 will be described later.

[0016] The information processing apparatus 200 acquires the data set 204 output from the imaging apparatus 220, and changes the time-series detection data 106 using the time-series viewing angle data 207 included in the data set 204. The information processing apparatus 200 outputs a data set 209 including the moving image data 105 and the time-series detection data 210 which is the changed time-series detection data 106. Note that the data set 209 may be output as a moving image file with one detection data including the moving image data 105 and the time-series detection data 210. Details of the information processing apparatus 200 will be described later.

[0017] The playback apparatus 111 acquires the data set 209 output from the information processing apparatus 200, and plays back the moving image data 105 and the time-series detection data 210 included in the data set 209 based on the time stamps included in these data. Since the playback apparatus 111 is the same as the conventional playback apparatus 111, the description thereof will be omitted.

[0018] With reference to FIGS. 3 to 6, the configurations of the information processing apparatus 200 and the imaging apparatus 220 will be described. FIG. 3 is a block diagram showing an example of the hardware configuration of the information processing apparatus 200 according to Embodiment 1. The information processing apparatus 200 is configured by a personal computer or the like, and includes a CPU 301, a ROM 302, a RAM 303, a storage device 304, and an external I / F 305, and each is communicably connected to each other via a bus 306.

[0019] The CPU 301 is a control unit composed of at least one processor or circuit, and controls the entire information processing apparatus 200. The ROM 302 is a memory capable of electrically erasing and storing data, and stores constant data and programs for the operation of the CPU 301. The program referred to here is a computer program for executing the processes of various flowcharts described later in the present embodiment. The RAM 303 is a memory used as a work area of the CPU 301, and stores constant and variable data for the operation of the CPU 301, and programs read from the ROM 302.

[0020] The memory device 304 is a device that can write and read various information, which is built-in or externally attached to the information processing device 200, and is composed of a hard disk drive, a solid disk, a removable memory card, or the like. The memory device 304 stores a data set 204 acquired from the imaging device 120, a data set 209 including the time-series detection data 210, and a database of various parameters used for the change processing of the time-series detection data 106. The external I / F 305 is an interface for communicating with a computer network or an external device. The information processing device 200 acquires the data set 204 via the external I / F 305 and outputs the data set 209 to the outside.

[0021] FIG. 4 is a block diagram showing an example of the hardware configuration of the imaging device 220 according to Embodiment 1. The imaging device 220 includes a CPU 401, a ROM 402, a RAM 403, a memory device 404, an imaging unit 405, a detection device 103, and an external I / F 407, and each is communicably connected to each other via a bus 408. The CPU 401 is a control unit composed of at least one processor or circuit, and controls the entire imaging device 220. The ROM 402 is a memory that can electrically erase and store data, and stores constant data and programs for the operation of the CPU 401. The RAM 403 is a memory used as a work area for the CPU 401, and stores constant and variable data for the operation of the CPU 401, and programs read from the ROM 402.

[0022] The memory device 404 is a storage medium composed of a semiconductor memory such as a memory card, which stores the moving image data 105 obtained by imaging. The imaging unit 405 has an image sensor and an optical system that captures external light up to the image sensor, and outputs an electrical signal indicating the optical image formed by the image sensor. The image sensor is composed of an element such as a CCD (Charge Coupled Device) or CMOS (Complementary Metal-Oxide-Semiconductor) that converts the optical image into an electrical signal. The optical system has a plurality of microlenses and a zoom mechanism that changes the focal length by moving a specific microlens. The external I / F 407 outputs an external data set 204 including the moving image data 105 obtained by imaging, the time-series detection data 106 indicating the detection results by the detection device 103, and the angle-of-view data corresponding to the zoom value at each time point.

[0023] FIG. 5 is a block diagram showing an example of the functional configuration of the imaging device 220 according to Embodiment 1. The imaging device 220 includes, as functional components, a moving image generation unit 503, an angle-of-view data generation unit 504, a detection data generation unit 505, and a transmission unit 508. The moving image generation unit 503 generates the moving image data 105. Specifically, first, the moving image generation unit 503 generates an image corresponding to a frame in the moving image based on the electrical signal indicating the optical image output from the imaging unit 405. Subsequently, the moving image generation unit 503 generates the moving image data 105 including a plurality of frame data by encoding the generated plurality of images into a standardized format. The moving image data 105 generated by the moving image generation unit 503 is stored in the memory device 404.

[0024] The angle-of-view data generation unit 504 generates time-series angle-of-view data 207. Specifically, for example, the angle-of-view data generation unit 504 first samples and digitizes a signal indicating the focal length of the zoom mechanism output from the imaging unit 405 during part or all of the period of imaging the moving image. Subsequently, the angle-of-view data generation unit 504 generates the time-series angle-of-view data 207 by specifying the angle of view corresponding to each focal length digitized by sampling. Specifically, for example, a user of the imaging device 220 (hereinafter referred to as the "imager") manually operates the zoom mechanism to change the focal length. The imaging unit 405 outputs a signal corresponding to the change in the focal length in the zoom mechanism to the angle-of-view data generation unit 504. The time-series angle-of-view data 207 generated by the angle-of-view data generation unit 504 is stored in the storage device 404.

[0025] The detection data generation unit 505 generates time-series detection data 106. Specifically, for example, the detection data generation unit 505 first generates the time-series detection data 106 by sampling and digitizing an electrical signal output as a detection result from the detection device 103. The time-series detection data 106 generated by the detection data generation unit 505 is stored in the storage device 404. The transmission unit 508 transmits an external data set 204 including the moving image data 105 generated by the moving image generation unit 503, the time-series angle-of-view data 207 generated by the angle-of-view data generation unit 504, and the time-series detection data 106 generated by the detection data generation unit 505.

[0026] As shown in FIG. 5 as an example, the imaging device 220 may further include a file generation unit 506 in addition to the above-described configuration. In this case, the file generation unit 506 generates a moving image file with one angle of view and detection data including the moving image data 105, the time-series angle-of-view data 207, and the time-series detection data 106. Specifically, for example, the imaging device 220 generates a moving image file with angle-of-view and detection data by performing a process of adding the time-series angle-of-view data and the time-series detection data to the time-specifying metadata of the moving image file. The moving image file with angle-of-view and detection data generated by the file generation unit 506 is transmitted to the outside by the transmission unit 508 as a data set 204.

[0027] FIG. 6 is a block diagram showing an example of the functional configuration of the information processing apparatus 200 according to Embodiment 1. The information processing apparatus 200 includes a generation unit 601, a reception unit 602, and an output unit 611. The reception unit 602 receives a data set 204 including the moving image data 105, the time-series angle-of-view data 207, and the time-series detection data 106 from an external device such as the imaging device 220. The data set 204 received by the reception unit 602 is held in the moving image holding unit 603 in the storage device 304. The generation unit 601 performs a change process on the time-series detection data 106 based on the time-series angle-of-view data 207 included in the data set 204 received by the reception unit 602, and generates time-series detection data 210 which is the changed time-series detection data 106. Details of the process in the generation unit 601 will be described later. The moving image data 105 and the time-series detection data 210 generated by the generation unit 601 are held in the moving image holding unit 603 in the storage device 304 as a data set 209. The output unit 611 acquires the data set 209 from the moving image holding unit 603, and outputs the acquired data set 209 to an external device such as the playback device 111.

[0028] The generation unit 601 includes an angle-of-view data acquisition unit 604, a detection data acquisition unit 605, a moving image analysis unit 606, an adjustment amount acquisition unit 609, and a modification unit 610. The detection data acquisition unit 605 acquires the time-series detection data 106 included in the data set 204 held in the moving image holding unit 603 by reading it from the storage device 404. The time-series detection data 106 acquired by the detection data acquisition unit 605 is output to the adjustment amount acquisition unit 609 and the modification unit 610. The angle-of-view data acquisition unit 604 acquires the time-series angle-of-view data 207 included in the data set 204 held in the moving image holding unit 603 by reading it from the storage device 404. The time-series angle-of-view data 207 acquired by the angle-of-view data acquisition unit 604 is output to the adjustment amount acquisition unit 609.

[0029] The moving image analysis unit 606 acquires the moving image data 105 included in the data set 204 held in the moving image holding unit 603 by reading it from the storage device 404, and performs image analysis processing on the acquired moving image data 105. Specifically, the moving image analysis unit 606 performs object detection processing on each frame included in the moving image data 105, and acquires the type and distance of the object existing as an image in the frame as the processing result of the object detection processing. For example, the moving image analysis unit 606 uses the object database 608 to identify an object of a predetermined type from the objects existing as images in the frame. Further, the moving image analysis unit 606 acquires the type of the identified object and the distance from the imaging device 220 to the object.

[0030] The object database 608 is information in which information indicating the type of object and the physical sensation elements that the object can generate are associated with each other. The object database 608 is pre-stored in the storage device 404, and the moving image analysis unit 606 acquires the object database 608 by reading the object database 608 from the storage device 404. That is, the moving image analysis unit 606 identifies an object that can generate a physical sensation element from among the objects existing as images in the frame, and acquires the type of the identified object, the distance to the object, etc. as the processing result of the object detection process. Information indicating the processing result of the object detection process acquired by the moving image analysis unit 606 (hereinafter referred to as "object information") is output to the adjustment amount acquisition unit 609.

[0031] The adjustment amount acquisition unit 609 acquires the adjustment amount of the time-series detection data 106, which is the amount of change when changing the time-series detection data 106. Specifically, the adjustment amount acquisition unit 609 first acquires the time-series viewing angle data 207, the time-series detection data 106, and the object information output from the viewing angle data acquisition unit 604, the detection data acquisition unit 605, or the moving image analysis unit 606. In addition, the adjustment amount acquisition unit 609 acquires it by reading the physical sensation database 607 pre-stored in the storage device 404 from the storage device 404. Subsequently, the adjustment amount acquisition unit 609 acquires the adjustment amount of the time-series detection data 106 by determining it using the acquired viewing angle data acquisition unit 604, object information, and physical sensation database 607. The physical sensation database 607 holds data on the transmission speed of physical sensation elements for each type of physical sensation. Information indicating the adjustment amount acquired by the adjustment amount acquisition unit 609 is output to the change unit 610.

[0032] The change unit 610 acquires the time-series detection data 106 output by the detection data acquisition unit 605 and the information indicating the adjustment amount output by the adjustment amount acquisition unit 609, and changes the time-series detection data 106 based on the adjustment amount. The change unit 610 outputs the time-series detection data 210, which is the changed time-series detection data 106 generated by the change process of the time-series detection data 106, to the moving image holding unit 603. Specifically, the change unit 610 outputs the time-series detection data 210 to the moving image holding unit 603, and causes the moving image holding unit 603 to hold it as a data set 209 including the time-series detection data 210 and the moving image data 105 corresponding to the time-series detection data 106.

[0033] The change unit 610 may generate a moving image file with detection data corresponding to the data set 209, and output the generated moving image file with detection data to the moving image holding unit 603. In this case, the change unit 610 reads the moving image data 105 corresponding to the time-series detection data 106 from the moving image holding unit 603, and adds the time-series detection data 210 generated by the above-described change process to the read moving image data 105 as time-specifying metadata. The change unit 610 may read the angle of view and the moving image file with detection data in which the time-series detection data 106 is added as time-specifying metadata from the moving image holding unit 603, and replace the time-specifying metadata with the time-series detection data 210.

[0034] FIG. 7 is a flowchart showing an example of a processing flow in the information processing apparatus 200 according to Embodiment 1. Hereinafter, it is assumed that when the information processing apparatus 200 starts the processing of the flowchart, a moving image file with an angle of view and detection data is held in the moving image holding unit 603 as the data set 204. FIG. 8 is a diagram showing an example of a moving image file 800 with an angle of view and detection data held in the moving image holding unit 603 as the data set 204 according to Embodiment 1. The moving image file 800 with an angle of view and detection data includes moving image data 801 encoded in a standardized format and converted into data, and time-specifying metadata associated with the moving image data 801. The time-specifying metadata includes time-series detection data 802 and time-series data 803 of the focal length of a lens convertible into an angle of view. In the following description, the symbol "S" means step (process).

[0035] In S701, the detection data acquisition unit 605 acquires time-series detection data 106 that has not yet been acquired from among one or more time-series detection data 106 included in the data set 204 stored in the moving image holding unit 603. Here, the one or more time-series detection data 106 included in the data set 204 are, for example, time-series detection data 106 based on detections by one or more detection devices that detect different types of somatosensory elements. Specifically, the detection data acquisition unit 605 acquires the time-series detection data 106 by reading it from the time-series detection data 802 included in the time-specifying metadata of the moving image file 800 with an angle of view and detection data. Hereinafter, it is assumed that the time-series detection data 106 first acquired in S701 is based on detection by an odor sensor. Subsequently in S701, the detection data acquisition unit 605 acquires the type of the somatosensory element indicated by the acquired time-series detection data 106 and the time stamp included in the time-series detection data, and holds information indicating the type of the acquired somatosensory element and the time stamp in the RAM 403 or the like.

[0036] Next, at S702, the detection data acquisition unit 605 determines whether the time-series detection data 106 could be acquired at S701. If it is determined at S702 that the time-series detection data 106 could not be acquired, the information processing apparatus 200 ends the processing of the flowchart shown in FIG. 7, assuming that there is no time-series detection data 106 to be acquired. If it is determined at S702 that the time-series detection data 106 could be acquired, the information processing apparatus 200 executes the processing of S703. Specifically, at S703, the angle-of-view data acquisition unit 604 acquires the time-series angle-of-view data 207 corresponding to the time-series detection data 106 acquired at S701 from the data set 204 held in the moving image holding unit 603. Specifically, first, the angle-of-view data acquisition unit 604 acquires the time-series data of the focal length convertible to the time-series angle-of-view data 207 by reading it from the time-specifying metadata of the moving image file 800 with angle-of-view and detection data. Subsequently, the angle-of-view data acquisition unit 604 converts the acquired time-series data of the focal length into the time-series angle-of-view data 207, thereby acquiring the time-series angle-of-view data 207 corresponding to the time-series detection data 106 acquired at S701.

[0037] Next, at S704, the angle-of-view data acquisition unit 604 determines whether the angle-of-view indicated by all the angle-of-view data included in the time-series angle-of-view data acquired at S703 is the standard angle-of-view. Here, the standard angle-of-view is assumed to be approximately 47°, which is substantially equivalent to the viewing angle of a natural person and is the width of the field of view when a natural person views normally. If it is determined at S704 that the angle-of-view indicated by all the angle-of-view data coincides (here, the term "coincides" includes the case of "substantially coincides") with the standard angle-of-view, the information processing apparatus 200 executes the processing of S906 described later. If it is determined at S704 that the angle-of-view indicated by at least some of the angle-of-view data does not coincide with the standard angle-of-view, at S705, the moving image analysis unit 606 executes the process of acquiring the virtual position deviation value. The virtual position deviation value will be described later. For example, the information processing apparatus 200 may perform the processing of S705 for each of one or more periods corresponding to the time stamp at which it is determined that the angle-of-view indicated by the angle-of-view data among the time-series angle-of-view data 207 does not coincide with the standard angle-of-view.

[0038] FIG. 9 is a flowchart showing an example of a processing flow in the moving image analysis unit 606 according to Embodiment 1, and is a flowchart showing an example of the flow of the virtual position deviation value acquisition process of S705 shown in FIG. 7. In S901, the moving image analysis unit 606 first acquires, from the moving image data 105 included in the data set 204 held in the moving image holding unit 603, one or more frame data corresponding to the time stamps of the time-series detection data 106 acquired in S701. Specifically, the moving image analysis unit 606 acquires the frame data by reading the frame data from the moving image data 801 of the moving image file 800 with the viewing angle and detection data. Subsequently, in S901, the moving image analysis unit 606 executes an object detection process on each of the acquired frame data to detect an object included as an image in the image indicated by each frame data. The object detection process can be realized by using a method such as a method using a learned model obtained by deep learning, but the object detection method is not limited to the method using a learned model.

[0039] After S901, in S902, the moving image analysis unit 606 identifies an object 101 (hereinafter referred to as "target object 101") that can generate a predetermined somatosensory element from among the one or more objects detected as images in S902. Here, it is assumed that the object database 608 manages a list of objects that can generate somatosensory elements that can be handled in the reproduction system. The moving image analysis unit 606 can identify one or more target objects 101 from among the one or more objects detected in S705 by searching the object database 608.

[0040] FIG. 10 is a diagram showing an example of the somatosensory database 607 and the object database 608 according to Embodiment 1. Specifically, FIG. 10(b) shows an example of the object database 608. FIG. 10(a) will be described later. As shown in FIG. 10(b), in the object database 608, for example, information indicating the type of object that can be detected by the moving image analysis unit 606, information indicating the somatosensory elements that can generate the object, and information indicating the size of the object are associated with each other. In the present embodiment, the object database 608 is described as being held in the storage device 304 of the information processing apparatus 200, but it may be held in an external device such as a cloud server that can be accessed via an external I / F.

[0041] After S902, in S903, the moving image analysis unit 606 determines whether or not one or more target objects 101 have been specified in S902. If it is determined in S903 that at least one target object 101 has been specified, the moving image analysis unit 606 executes a series of processes from S904 to S906. Hereinafter, it is described that one target object 101 has been specified in S902, but if a plurality of target objects 101 have been specified in S902, a series of processes from S904 to S906 can be appropriately executed for each specified target object 101.

[0042] First, in S904, the moving image analysis unit 606 acquires the distance from the imaging device 220 to the target object 101 specified in S706 (hereinafter referred to as the "imaging distance"). Next, in S905, the moving image analysis unit 606 acquires the distance from the virtual position (hereinafter referred to as the "virtual position") of the viewer 114 in the imaging environment to the target object 101 specified in S706 (hereinafter referred to as the "appreciation distance"). Next, in S906, the moving image analysis unit 606 acquires a virtual position deviation value based on the imaging distance acquired in S904 and the appreciation distance acquired in S905. The virtual position deviation value is a value corresponding to the deviation amount between the position of the imaging device 220 and the virtual position of the viewer 114.

[0043] FIG. 11 is a diagram for explaining an example of the process of acquiring the virtual position of the moving image analysis unit 606 according to Embodiment 1. First, with reference to FIG. 11(a), the principle for estimating the distance from the imaging device 220 to the object will be described. If the size of the object or the size of other objects existing around the object is known, the distance from the imaging device 220 to the object included as an image in the still image obtained by imaging by the imaging unit 405 can be estimated. For example, as shown in FIG. 11(a), when the object 1101 with a size of Hr is imaged using a lens with a focal length of f, the distance from the imaging unit 405 to the object 1101 can be calculated using the following formula (1). In formula (1), Dr is the distance from the imaging unit 405 to the object 1101, and Hi is the size of the object image 1104 in which the object 1101 is captured as an image on the image sensor 1103 of the imaging unit 405. Dr = (Hr / Hi) · f ··· Formula (1)

[0044] The moving image analysis unit 606 calculates the imaging distance RL, which is the imaging distance from the imaging device 220 to the target object 101, using formula (1), thereby acquiring the imaging distance (RL). For example, the imaging distance (RL) can be calculated based on the focal length data acquired at S703, the information indicating the size of the object in the object database 608 shown as an example in FIG. 10(b), and the size of the image of the target object 101 in the frame.

[0045] Similarly, the moving image analysis unit 606 obtains the viewing distance (VL) by calculating the viewing distance VL from the virtual position of the viewer 114 to the target object 101 using Equation (1). Here, the viewing distance (VL) can be calculated by substituting the focal length corresponding to the standard angle of view into f in Equation (1). Specifically, the viewing distance (VL) can be calculated based on the focal length corresponding to the standard angle of view, the information indicating the size of the object in the object database 608 described above, and the size of the image of the target object 101 in the frame. Note that the focal length corresponding to the standard angle of view depends on the size of the image sensor 1103. When the image sensor 1103 is a so-called full-size sensor of 35 millimeters (mm), the focal length corresponding to the standard angle of view is 50 mm.

[0046] Further, the moving image analysis unit 606 calculates a virtual position deviation value VD by subtracting the viewing distance (VL) from the imaging distance (RL), and obtains the virtual position deviation value (VD). When the virtual position deviation value (VD) is a positive value, it means that the virtual position of the viewer 114 is closer to the target object 101 compared to the imaging position. When the virtual position deviation value (VD) is a negative value, it means that the virtual position of the viewer 114 is farther from the target object 101 compared to the imaging position.

[0047] If it is determined in S902 that no target object 101 can be identified, the moving image analysis unit 606 obtains 0 (zero) as the virtual position deviation value, assuming that the viewing distance (VL) is the imaging distance (RL) in S906. This is because it can be assumed that the somatosensory elements indicated by the detection data are generated from objects existing outside the viewing range in the virtual imaging environment of the viewer 114 based on the image displayed on the display device 112. The information (object information) indicating the type of the target object 101 identified in S902 and the virtual position deviation value obtained in S906 are output to the adjustment amount acquisition unit 609. After S906, the moving image analysis unit 606 ends the processing of the flowchart shown in FIG. 9, that is, the processing of S705 shown in FIG. 7.

[0048] After S705, at S706, the adjustment amount acquisition unit 609 acquires the propagation speed (also referred to as the "diffusion speed") of the somatosensory elements that can be generated from the target object 101 based on the object information of the target object 101 output from the moving image analysis unit 606. Specifically, at S706, first, the adjustment amount acquisition unit 609 searches the object database 608 for the type of the target object 101 indicated by the object information to identify the somatosensory elements that can be generated from the target object 101. Hereinafter, as an example, it is assumed that the type of the target object 101 is "roasted turkey", and the somatosensory element "odor component A" is identified as the somatosensory element that can be generated from the target object 101 and will be described.

[0049] Subsequently, at S706, the adjustment amount acquisition unit 609 searches the somatosensory database 607 for the identified somatosensory elements to acquire information indicating the propagation speed (also referred to as the "diffusion speed") of the somatosensory elements. FIG. 10(a) shows an example of the somatosensory database 607. As shown in FIG. 10(a), in the somatosensory database 607, the information indicating the somatosensory elements and the information indicating the propagation speed are associated with each other. Since the identified somatosensory element is "odor component A", "0.4" (millimeters per second (mm / s)) is acquired as the information indicating the propagation speed of the somatosensory element. In this embodiment, the somatosensory database 607 is described as being held in the storage device 304 of the information processing apparatus 200, but it may be held in an external device such as a cloud server accessible via an external I / F.

[0050] After S706, at S707, the adjustment amount acquisition unit 609 acquires the adjustment amount of the time-series detection data 106 based on the information indicating the virtual position deviation value output from the moving image analysis unit 606 and the information indicating the propagation speed acquired at S706. Specifically, for example, the adjustment amount acquisition unit 609 acquires an adjustment amount for adjusting the timing of generating the somatosensory element from the generator 113 based on the virtual position deviation value and the propagation speed. The information indicating the acquired adjustment amount is output to the change unit 610. For the adjustment amount a (seconds (s)), it can be calculated, for example, using the following formula (2) with the time for the somatosensory element to propagate the distance between the virtual position corresponding to the virtual position deviation value and the imaging position. In formula (2), pv is the propagation speed of the somatosensory element (mm / s), and VD is the virtual position deviation value (mm). a = VD / pv ··· Formula (2)

[0051] Next, at S708, the change unit 610 generates the time-series detection data 210 by changing the time-series detection data 106 based on the information indicating the adjustment amount output from the adjustment amount acquisition unit 609. Specifically, for example, the change unit 610 changes the time-series detection data 106 by changing the time stamps included in the time-series detection data 106 based on the adjustment amount acquired at S707. For example, each time stamp is changed by subtracting the adjustment amount from each of all the time stamps included in the time-series detection data 106. The generated time-series detection data 210 is output to the moving image holding unit 603, and is associated with the moving image data 105 corresponding to the time-series detection data 106 in the moving image holding unit 603, and is held as the data set 209.

[0052] When the virtual position of the viewer 114 is closer to the target object 101 compared to the imaging position, VD becomes a positive value, and thus the adjustment value also becomes a positive value. Therefore, when the dataset 209 is reproduced as compared to when the dataset 104 is reproduced by the playback device 111 in the playback environment, the timing at which the physical sensation element is generated from the generation device 113 is earlier. Conversely, when the virtual position of the viewer 114 is farther from the target object 101 compared to the imaging position, VD becomes a negative value, and thus the adjustment value also becomes a negative value. Therefore, when the dataset 209 is reproduced as compared to when the dataset 104 is reproduced by the playback device 111 in the playback environment, the timing at which the physical sensation element is generated from the generation device 113 is later. At this time, the moving image analysis unit 606 may detect the timing of scene switching and prevent the change of the time stamp across the scene switching. Thereby, even though the scene is switched during playback, it is possible to suppress the generation of a physical sensation element that does not match the scene from the generation device 113, and it is possible to suppress giving a sense of discomfort to the viewer 114.

[0053] After S708, in S709, the detection data acquisition unit 605 determines whether all the time-series detection data 106 included in the dataset 204 held in the moving image holding unit 603 have been acquired. Specifically, the detection data acquisition unit 605 determines whether all the time-series detection data 106 have been read out from the angle of view and the time-specifying metadata of the moving image file with detection data. If it is determined in S709 that at least a part of the time-series detection data 106 has not been acquired (not read out), the information processing apparatus 200 returns to the process of S701. Thereafter, the information processing apparatus 200 repeatedly executes the processes from S701 to S709 until it is determined in S709 that all the time-series detection data 106 have been acquired (read out). If it is determined in S709 that all the time-series detection data 106 have been acquired (read out), the information processing apparatus 200 ends the process of the flowchart shown in FIG. 7.

[0054] In the above description, as an example, the form of adjusting the timing for generating a somatosensory element corresponding to the stimulus to the sense of smell of the viewer 114 has been described. However, the adjustment of the timing is not limited to only the somatosensory element corresponding to the stimulus to the sense of smell. For example, the information processing apparatus 200 according to the present disclosure is also applicable when adjusting the timing for generating a somatosensory element corresponding to the stimulus to the cutaneous tactile sensation felt when wind or water droplets touch the skin of the viewer 114. Further, for example, the information processing apparatus 200 according to the present disclosure is also applicable when adjusting the timing for generating a somatosensory element corresponding to the stimulus to the sense of hearing of the viewer 114, such as a sound generated at a position relatively far from the imaging position, such as thunder. According to the reproduction system configured as described above, it is possible to reduce the discomfort of the viewer 114 caused by the timing deviation between the stimulus to the vision of the viewer 114 and the stimulus to at least any one of hearing, smell, and cutaneous tactile sensation. As a result, according to the reproduction system configured as described above, it is possible to provide a high sense of presence to the viewer 114 of the moving image.

[0055] Also, in the above description, as an example, the form of adjusting the timing for generating a somatosensory element has been described. However, the information processing apparatus 200 may adjust the magnitude of the detected value in the time-series detection data 106 based on the virtual position deviation value. For example, the farther away from the generation source of the somatosensory element corresponding to the stimulus to the sense of smell, the weaker the stimulus caused by the somatosensory element felt by a natural person. Also, for wind blowing from a certain direction or water droplets scattering from a certain direction, the farther away from the generation source of the wind or water droplets, the weaker the stimulus to the cutaneous tactile sensation felt by a natural person. By adjusting the magnitude of the detected value based on the virtual position deviation value, it is possible to suppress the deviation between the virtual position imagined by the viewer 114 based on the stimulus to the vision and the virtual position imagined by the viewer 114 based on the intensity of the stimuli to the smell and cutaneous tactile sensation. As a result, it is possible to reduce the discomfort of the viewer 114 and provide a high sense of presence to the viewer 114 of the moving image.

[0056] As described above, the discomfort of the viewer 114 during the playback of the moving image caused by the deviation between the imaging position resulting from the difference between the focal length of the lens of the imaging unit 405 during the imaging of the moving image and the focal length corresponding to the standard angle of view and the virtual position of the viewer 114 was mentioned. In the case of a digital moving image, during the imaging or editing of the moving image, the angle of view may change due to digital zoom or crop editing in which a partial area in the image is cut out and the area is enlarged. Also in this case, the same phenomenon as the zoom-in of the zoom mechanism by the lens of the imaging unit 405 occurs. Therefore, the virtual position deviation value can be calculated based on the difference between the size of the image before the area is cut out and the size of the cut-out area, or the amount of change in the size of the attention object 101 resulting from the difference. Accordingly, by converting the information indicating the operation of the digital zoom or crop editing into the time-series angle-of-view data 207, the information processing apparatus 200 can perform the same processing as described above using the time-series angle-of-view data 207. As a result, the discomfort felt by the viewer 114 due to the digital zoom or crop editing can also be reduced, and a high sense of presence can be provided to the viewer 114 of the moving image.

[0057] [Other Embodiments] The present disclosure can also be realized by supplying a program that realizes one or more functions of the above-described embodiments to a system or apparatus via a network or a storage medium, and causing one or more processors in a computer of the system or apparatus to read and execute the program. Further, it can also be realized by a circuit (for example, ASIC) that realizes one or more functions.

[0058] Note that within the scope of the present disclosure, any combination of the embodiments, any modification of any component of each embodiment, or any omission of any component in each embodiment is possible.

[0059] [Configuration of the Present Disclosure] The present disclosure includes the following configuration, method, and program.

[0060] <Configuration 1> Acquisition means for acquiring time-series detection data indicating a time series of detection results during imaging of the moving image by a detection device that detects an element corresponding to a stimulus to at least one of the sense of smell, skin touch, and hearing that a natural person can feel in the imaging environment of the imaging device, data of the moving image obtained by imaging with the imaging device, and time-series angle-of-view data indicating the angle of view in the time series of the moving image; Changing means for changing some or all of a plurality of timestamps included in the time-series detection data based on the time-series angle-of-view data; An information processing apparatus, characterized by comprising:

[0061] <Configuration 2> The imaging device is capable of changing the angle of view during imaging of the moving image, The time-series detection data indicates the time-series angle of view that changes in the imaging device during imaging of the moving image, The information processing apparatus according to Configuration 1, characterized by comprising:

[0062] <Configuration 3> When the angle of view corresponding to each timestamp included in the time-series angle-of-view data is smaller than a standard angle of view, the changing means changes the time of the timestamp in a direction to advance it; The information processing apparatus according to Configuration 1 or 2, characterized by comprising:

[0063] <Configuration 4> When the angle of view corresponding to each timestamp included in the time-series angle-of-view data is larger than a standard angle of view, the changing means changes the time of the timestamp in a direction to delay it; The information processing apparatus according to any one of Configurations 1 to 3, characterized by comprising:

[0064] <Configuration 5> The changing means Changing amount acquisition means for acquiring a changing amount of the timestamp based on a difference between the angle of view corresponding to each timestamp included in the time-series angle-of-view data and a standard angle of view, and comprising: The changing means changes the time stamp by using the change amount acquired by the change amount acquisition means. The information processing apparatus according to any one of Configurations 1 to 4, characterized by the above.

[0065] <Configuration 6> The change amount acquisition means Based on the viewing angle and the standard viewing angle, the imaging distance, which is the distance from the imaging position, where the moving image is captured, to the object that generates the element, and the viewing distance, which is the distance from the virtual position, which is the virtual position of the viewer in the imaging environment recognized by the viewer when viewing the moving image, to the object, are obtained, and the difference therebetween is obtained. Based on the difference between the imaging distance and the viewing distance and the propagation speed of the element, the change amount is obtained by calculating the difference between the period until the element generated from the object reaches the imaging position and the period until it reaches the virtual position. The information processing apparatus according to Configuration 5, characterized by the above.

[0066] <Configuration 7> The changing means further includes analysis means for identifying, by image analysis, an object that is included as an image in a frame of the moving image and that can generate the element, and outputting information indicating the type of the identified object. The change amount acquisition means acquires information indicating the propagation speed of the element that can be generated by the object based on the information indicating the type of the object identified by the analysis means, and acquires the change amount based on the difference between the imaging distance and the viewing distance and the information indicating the propagation speed of the acquired element. The information processing apparatus according to Configuration 6, characterized by the above.

[0067] <Configuration 8> The time-series detection data includes the time-series detection results by a detection device that detects the element corresponding to the stimulus to the sense of smell. The information processing apparatus according to any one of Configurations 1 to 7, characterized by the above.

[0068] <Configuration 9> The time-series detection data includes the time-series detection results by a detection device that detects the elements corresponding to the stimulus to the skin tactile sensation. The information processing apparatus according to any one of Configurations 1 to 8, characterized in that.

[0069] <Configuration 10> The time-series detection data includes the time-series detection results by a detection device that detects the elements corresponding to the stimulus to the auditory sensation. The information processing apparatus according to any one of Configurations 1 to 9, characterized in that.

[0070] <Configuration 11> The changing means further changes detection values corresponding to some or all of the plurality of timestamps included in the time-series detection data based on the time-series viewing angle data. The information processing apparatus according to any one of Configurations 1 to 10, characterized in that.

[0071] <Configuration 12> An acquisition means for acquiring data of a moving image obtained by imaging with an imaging device, time-series viewing angle data indicating the time-series viewing angle of the moving image, and time-series detection data indicating time-series detection results during imaging of the moving image by a detection device that detects elements corresponding to stimuli to at least one of the olfactory and skin tactile sensations that a natural person can feel in the imaging environment of the imaging device; Changing means for changing detection values corresponding to some or all of the plurality of timestamps included in the time-series detection data based on the time-series viewing angle data; An information processing apparatus, characterized by comprising.

[0072] <Configuration 13> The information processing apparatus according to any one of Configurations 1 to 12; An element generator for generating the element; A playback device that plays back the data of the moving image and causes the display device to display the moving image, the playback device controlling the element generator so that the element occurs in synchronization with the playback of the data of the moving image based on the time-series detection data after being changed by the changing means of the information processing device. A playback system characterized by having the above.

[0073] <Method 1> An acquisition step of acquiring data of a moving image obtained by imaging with an imaging device, time-series angle-of-view data indicating the time-series angle of view of the moving image, and time-series detection data indicating the time-series detection results during the imaging of the moving image by a detection device that detects elements corresponding to stimuli to at least one of smell, skin touch, and hearing that a natural person can feel in the imaging environment of the imaging device; A changing step of changing some or all of a plurality of timestamps included in the time-series detection data based on the time-series angle-of-view data; An information processing method characterized by including the above.

[0074] <Method 2> An acquisition step of acquiring data of a moving image obtained by imaging with an imaging device, time-series angle-of-view data indicating the time-series angle of view of the moving image, and time-series detection data indicating the time-series detection results during the imaging of the moving image by a detection device that detects elements corresponding to stimuli to at least one of smell and skin touch that a natural person can feel in the imaging environment of the imaging device; Changing means for changing detection values corresponding to some or all of a plurality of timestamps included in the time-series detection data based on the time-series angle-of-view data; An information processing device characterized by having the above.

[0075] <Program> A program for causing a computer to function as the information processing device according to any one of Configurations 1 to 12.

Description of Signs

[0076] 200 Information processing apparatus 604 Angle-of-view data acquisition unit 605 Detection data acquisition unit 609 Adjustment amount acquisition unit 610 Modifying unit

Claims

1. an acquisition means for acquiring data of a moving image obtained by imaging using an imaging device, time-series angle-of-view data indicating a time-series angle of view of the moving image, and time-series detection data indicating a time-series detection result during imaging of the moving image by a detection device that detects an element corresponding to a stimulus to at least one of the senses of smell, touch, and hearing that a natural person can sense in an imaging environment of the imaging device; a change means for changing a part or all of a plurality of time stamps included in the time-series detection data based on the time-series angle of view data; 13. An information processing device comprising:

2. the imaging device is capable of changing the angle of view during imaging of the moving image, the time-series detection data indicates a time-series change in the angle of view of the imaging device while the moving image is being captured; 2. The information processing apparatus according to claim 1,

3. the changing means changes the time of the timestamp in a direction to advance the time of the timestamp when the angle of view corresponding to each of the timestamps included in the time-series angle of view data is smaller than a standard angle of view; 2. The information processing apparatus according to claim 1,

4. the changing means changes the time of the timestamp in a direction to delay the time of the timestamp when the angle of view corresponding to each of the timestamps included in the time-series angle of view data is larger than a standard angle of view; 2. The information processing apparatus according to claim 1,

5. The change means is a change amount acquisition means for acquiring a change amount of the timestamp based on a difference between the angle of view corresponding to each of the timestamps included in the time-series angle of view data and a standard angle of view; having the change means changes the timestamp using the amount of change acquired by the amount of change acquisition means; 2. The information processing apparatus according to claim 1,

6. The change amount acquisition means acquiring a difference between an imaging distance, which is a distance from an imaging position where the moving image is captured to an object generating the element, and a viewing distance, which is a distance from a virtual position, which is a virtual position of the viewer in an imaging environment, recognized by a viewer when viewing the moving image, to the object, based on the angle of view and the standard angle of view; acquiring the change amount by calculating a difference between a period until the element generated from the object reaches the imaging position and a period until the element reaches the virtual position based on a difference between the imaging distance and the viewing distance and a propagation speed of the element; 6. The information processing device according to claim 5,

7. the modification means further includes an analysis means for identifying the object that is included as an image in a frame of the video and that may cause the element by image analysis, and outputting information indicating a type of the identified object; the change amount acquisition means acquires information indicating a propagation speed of the element that may be generated by the object based on information indicating a type of the object specified by the analysis means, and acquires the change amount based on a difference between the imaging distance and the viewing distance and the acquired information indicating the propagation speed of the element; The information processing device according to claim 6 .

8. the time-series detection data includes the time-series detection results by a detection device that detects the element corresponding to a stimulus to the olfactory sense; 2. The information processing apparatus according to claim 1,

9. the time-series detection data includes the time-series detection results by a detection device that detects the element corresponding to a stimulus to the skin tactile sense; 2. The information processing apparatus according to claim 1,

10. the time-series detection data includes the time-series detection results by a detection device that detects the element corresponding to a stimulus to the auditory sense; 2. The information processing apparatus according to claim 1,

11. the change means further changes detection values ​​corresponding to a part or all of the plurality of time stamps included in the time-series detection data based on the time-series angle of view data; 2. The information processing apparatus according to claim 1,

12. an acquisition means for acquiring data of a moving image obtained by imaging using an imaging device, time-series angle-of-view data indicating a time-series angle of view of the moving image, and time-series detection data indicating a time-series detection result during imaging of the moving image by a detection device that detects an element corresponding to a stimulus to at least one of the sense of smell and the sense of touch of the skin that a natural person can sense in an imaging environment of the imaging device; a change means for changing detection values ​​corresponding to some or all of a plurality of time stamps included in the time-series detection data based on the time-series angle of view data; 13. An information processing device comprising:

13. An information processing device according to any one of claims 1 to 12, an element generator for generating the elements; a playback device that plays back the data of the moving image and displays the moving image on a display device, the playback device controlling the element generator so that the elements are generated in synchronization with the playback of the data of the moving image based on the time-series detection data after it has been changed by the change means of the information processing device; A playback system comprising:

14. an acquisition step of acquiring data of a moving image obtained by imaging using an imaging device, time-series angle-of-view data indicating a time-series angle of view of the moving image, and time-series detection data indicating a time-series detection result during imaging of the moving image by a detection device that detects an element corresponding to a stimulus to at least one of the senses of smell, touch, and hearing that a natural person can sense in an imaging environment of the imaging device; a modification step of modifying a part or all of a plurality of time stamps included in the time-series detection data based on the time-series angle of view data; 13. An information processing method comprising:

15. an acquisition step of acquiring data of a moving image obtained by imaging using an imaging device, time-series angle-of-view data indicating a time-series angle of view of the moving image, and time-series detection data indicating a time-series detection result during imaging of the moving image by a detection device that detects an element corresponding to a stimulus to at least one of the sense of smell and the sense of touch of the skin that a natural person can sense in an imaging environment of the imaging device; a change means for changing detection values ​​corresponding to some or all of a plurality of time stamps included in the time-series detection data based on the time-series angle of view data; 13. An information processing device comprising:

16. A program for causing a computer to function as the information processing device according to any one of claims 1 to 12.

Images