Information processing apparatus, information processing system, and information processing method

Abstract

This information processing device comprises: a playback data reception unit which receives first playback data and control information pertaining to the first playback data; a first buffer memory which stores the received first playback data and the received control information; a synchronization information management unit which receives synchronization information from another playback device that plays back second playback data in synchronization with the first playback data; a playback data management unit which reproduces or discards the first playback data that is stored in the first buffer memory; and a playback processing unit which plays back the first playback data on the basis of the received synchronization information.

Description

Technical Field

[0001] The present disclosure relates to an information processing apparatus, an information processing system, and an information processing method. In particular, it relates to a technique for playing back playback data in synchronization with other playback devices.

Background Art

[0002] In a playback device that plays back playback data received via wireless communication, if the communication quality deteriorates, the consumption of playback data by the playback process may exceed the securing of playback data by the reception process, and symptoms such as sound dropout may occur due to insufficient playback data.

[0003] In response to such a problem, Patent Document 1 below proposes a technique for suppressing sound dropout on the receiving side.

Prior Art Document

Patent Document

[0004]

Patent Document 1

Summary of the Invention

Problems to be Solved by the Invention

[0005] However, in the configuration of Patent Document 1, although it is for suppressing audible sound dropout on the receiving side, that is, intermittent sound dropout, when an underflow occurs in the buffer memory, the output of audio data is stopped until recovery from the underflow state, so a long silent period or the like may occur.

[0006] In view of such problems, an object of the present disclosure is to play back playback data while suppressing the occurrence of sound dropout and ensuring synchronization among a plurality of playback devices.

Means for Solving the Problems

[0007] The information processing apparatus of the present technology includes a reproduction data receiving unit that receives first reproduction data, a synchronization information management unit that receives synchronization information from another reproduction device that reproduces second reproduction data in synchronization with the first reproduction data, and a reproduction processing unit that reproduces the first reproduction data based on the received synchronization information. The first reproduction data and the second reproduction data are, for example, tactile data, acoustic data, or video data. Also, the combination thereof does not have to be a combination of the same type of data. For example, the first reproduction data may be acoustic data and the second reproduction data may be tactile data. Of course, both the first reproduction data and the second reproduction data may be tactile data, or both may be acoustic data. According to this configuration, it is possible to reproduce the first reproduction data in synchronization with the second reproduction data reproduced by another reproduction device. The above-described information processing apparatus may include a first buffer memory that stores the received first reproduction data. By accumulating the first reproduction data in the first buffer memory, it is possible to reproduce the first reproduction data without interruption even if the reception of the first reproduction data stops for a certain period of time.

[0008] The above-described information processing apparatus may include a reproduction data management unit that duplicates and deletes the first reproduction data stored in the first buffer memory. By the reproduction data management unit duplicating the first reproduction data in the first buffer memory, the accumulation amount of the first buffer memory can be increased. Also, by the reproduction data management unit deleting the first reproduction data in the first buffer memory, the accumulation amount of the first buffer memory can be decreased.

[0009] The reproduction data receiving unit in the above-described information processing apparatus may receive control information about the first reproduction data, and the reproduction data management unit may duplicate and delete the first reproduction data based on the control information. Thereby, it is possible to manage the accumulation amount of the first reproduction data in the first buffer memory based on the control information.

[0010] The synchronization information received by the synchronization information management unit in the above-described information processing apparatus may be information regarding the storage amount of a second buffer memory provided in the other playback device for temporarily storing the second playback data. Thereby, the playback processing unit can change the playback mode according to the degree to which the second playback data played back by another playback device is stored in the second buffer memory.

[0011] The synchronization information received by the synchronization information management unit in the above-described information processing apparatus may be information on the ratio of the storage amount to the memory size of the second buffer memory. Thereby, the synchronization information management unit can specifically know the amount of the second playback data stored by another playback device.

[0012] The synchronization information received by the synchronization information management unit in the above-described information processing apparatus may be flag information indicating whether the storage amount of the second buffer memory is equal to or greater than a threshold value or whether it has fallen below the threshold value. Thereby, the synchronization information management unit can know whether the second playback data played back by another playback device is small.

[0013] When the storage amount of the first buffer memory is less than a first threshold value, the playback data management unit in the above-described information processing apparatus may duplicate any of the first playback data based on the control information. Thereby, it is avoided that the storage amount of the first playback data in the first buffer memory falls below the first threshold value.

[0014] When the storage amount of the first buffer memory is equal to or greater than a second threshold value, the playback data management unit in the above-described information processing apparatus may discard any of the first playback data based on the control information. This avoids the situation where the accumulation amount of the first reproduction data in the first buffer memory exceeds the second threshold value.

[0015] In the reproduction data management unit in the above-described information processing apparatus, when the synchronization information management unit receives, as the synchronization information, information indicating that the accumulation amount of the second buffer memory is less than the third threshold value, among the first reproduction data, a copy of any one of the first reproduction data may be made based on the control information. This enables the reproduction of the first reproduction data in response to the situation where the accumulation amount of the second reproduction data in the second buffer memory has fallen below the third threshold value.

[0016] In the reproduction data management unit in the above-described information processing apparatus, when the synchronization information management unit receives, as the synchronization information, information indicating that the accumulation amount of the second buffer memory is equal to or greater than the fourth threshold value, among the first reproduction data, any one of the first reproduction data may be discarded based on the control information. This enables the reproduction of the first reproduction data in response to the situation where the accumulation amount of the second reproduction data in the second buffer memory has become equal to or greater than the fourth threshold value.

[0017] In the reproduction data management unit in the above-described information processing apparatus, the reproduction data management unit receives the copy information and discard information of the second reproduction data received from the other reproduction device, and when receiving the copy information, makes a copy of the first reproduction data to be reproduced in synchronization with the second reproduction data targeted for copying, and when receiving the discard information, discards the first reproduction data to be reproduced in synchronization with the second reproduction data targeted for discarding. This enables the copying and discarding of the first reproduction data to be performed in synchronization with the second reproduction data.

[0018] In the above-described information processing apparatus, the control information may be information indicating permission / non - permission for at least one of copying or discarding of the first reproduction data. The first playback data is data having a playback time length of a predetermined time (for example, several μsec to several hundred msec). With this configuration, it is determined whether duplication or deletion is appropriate for each piece of the first playback data.

[0019] In the information processing apparatus described above, the control information may be information based on a perception feature amount representing the degree of a user's perception of duplication or deletion of the first playback data. Thereby, among a plurality of pieces of the first playback data, the first playback data that is difficult for the user to perceive even when duplication or deletion is performed is selected as a duplication target or a deletion target.

[0020] The synchronization information management unit in the information processing apparatus described above may transmit, as synchronization information, information regarding the accumulation amount of the first playback data in the first buffer memory to the other playback device. Thereby, in the other playback device, playback processing based on the accumulation amount of the first playback data can be performed.

[0021] The playback data management unit in the information processing apparatus described above may reuse the first playback data instead of duplicating the first playback data. Thereby, the duplication process of the first playback data is not performed.

[0022] The information processing system of the present technology includes a transmission device that transmits first playback data and second playback data, a first playback device that plays back the first playback data, and a second playback device that plays back the second playback data. The first playback device includes a first playback data reception unit that receives the first playback data and control information about the first playback data, a first buffer memory that stores the received first playback data and control information about the first playback data, a first synchronization information management unit that receives synchronization information from the second playback device, a first playback data management unit that duplicates or deletes the first playback data stored in the first buffer memory, and a first playback processing unit that plays back the first playback data based on the synchronization information received by the first synchronization information management unit. The second playback device includes a second playback data reception unit that receives the second playback data and control information about the second playback data, a second buffer memory that stores the received second playback data and control information about the second playback data, a second synchronization information management unit that receives synchronization information from the first playback device, a second playback data management unit that duplicates or deletes the second playback data stored in the second buffer memory, and a second playback processing unit that plays back the second playback data based on the synchronization information received by the second synchronization information management unit. Thereby, in each playback device, it is possible to duplicate or delete playback data based on synchronization information.

[0023] The transmission device in the above-described information processing system may include a control information generation unit that generates first control information that is control information regarding whether to permit duplication and deletion of the first playback data, and second control information that is control information regarding whether to permit duplication and deletion of the second playback data. Thereby, permission / non-permission of duplication and deletion of the first playback data and the second playback data is set individually.

[0024] In the above-described information processing system, when the control information generation unit determines that at least one of the first reproduction data and the second reproduction data reproduced synchronously cannot be permitted to be replicated or discarded, the control information generation unit generates control information indicating that replication and discard are not permitted as the first control information of the first reproduction data, and may generate control information indicating that replication and discard are not permitted as the second control information of the second reproduction data reproduced synchronously with the first reproduction data. Accordingly, when information that enables replication and discard is stored in the first control information of the first reproduction data, information that enables replication and discard will also be stored in the second control information of the second reproduction data reproduced synchronously.

[0025] The information processing method executed by the computer device of the present technology includes a process of receiving the first reproduction data and control information about the first reproduction data, a process of storing the received first reproduction data and the control information, a process of receiving synchronization information from another playback device that plays back the second reproduction data in synchronization with the first reproduction data, a process of replicating or discarding the stored first reproduction data, and a process of playing back the first reproduction data based on the received synchronization information. By such an information processing method, playback processing can be performed so that the playback statuses of the first reproduction data and the second reproduction data are kept synchronized.

Brief Description of the Drawings

[0026]

Figure 1

Figure 2

Figure 3

Figure 4

Figure 5

Figure 6

Figure 7

Figure 8

Figure 9

Figure 10

Figure 11

Figure 12

Figure 13

Figure 14

Figure 15

Mode for Carrying Out the Invention

[0027] Hereinafter, the embodiments will be described in the following order. <1. Configuration of Synchronous Reproduction System> <2. Configuration of Computer Device> <3. Duplication and Deletion of Reproduction Data> <3-1. First Example of Duplication> <3-2. Second Example of Duplication> <3-3. First Example of Deletion> <3-4. Third Example of Duplication> <4. Processing Examples> <4-1. Processing Example of Transmitter Device> <4-2. Processing Example 1 of the Recycling Device> <4-3. Processing Example 2 of the Recycling Device> <4-4. Processing Example 3 of the Recycling Device> <4-5. Processing Example 4 of the Recycling Device> <5. Modification Example> <6. Summary> <7. This Technology>

[0028] <1. Configuration of the Synchronous Playback System> The configuration of the synchronous playback system 1 as an information processing system that synchronously plays back a plurality of data will be described with reference to FIG. 1. The synchronous playback system 1 is configured to include at least two or more playback devices. The data to be played back by the playback device may be acoustic data, tactile data, video data, or a mixed data thereof.

[0029] The synchronous playback system 1 in the present embodiment includes a transmission device 2 that generates and transmits playback data, an acoustic playback device 3 as a first playback device that plays back the received acoustic data, and a tactile playback device 4 as a second playback device that plays back the tactile data.

[0030] The transmission device 2 acquires the original data of the data provided to the user from the data source SC that is the data supply source, generates the playback data (acoustic data or tactile data) to be played back by the acoustic playback device 3 or the tactile playback device 4, and transmits it to the acoustic playback device 3 or the tactile playback device 4.

[0031] For this purpose, the transmission device 2 includes a playback data generation unit 21, a control information generation unit 22, a buffer memory 23 for transmission data, a transmission data generation unit 24, and a transmission unit 25.

[0032] Note that the reproduction data generation unit 21, the control information generation unit 22, the transmission data buffer memory 23, and the transmission data generation unit 24 may be provided separately for audio data and for tactile data. Here, those for audio data processing and those for tactile data processing are described together.

[0033] The reproduction data generation unit 21 acquires original data for generating reproduction data from the data source SC. The original data is, for example, LPCM (Linear Pulse Code Modulation) data or the like if it is audio data. In addition to this, the data source SC supplies original data for generating tactile data and original data for generating video data. The reproduction data generation unit 21 generates reproduction data by performing encoding processing or the like for generating audio data to be reproduced by the audio reproduction device 3 and tactile data to be reproduced by the tactile reproduction device 4. Note that already encoded data may be supplied from the data source SC. In that case, the transmission device 2 may not include the reproduction data generation unit 21.

[0034] The reproduction data generated by the reproduction data generation unit 21 is, for example, divided data with a reproduction time of a very short time (for example, several μsec to several hundred msec). One reproduction data is processed into one transmission data (packet data) by the transmission data generation unit described later and transmitted to the reproduction device. Also, the original data acquired by the reproduction data generation unit 21 is data corresponding to one reproduction data.

[0035] The reproduction data generated by the reproduction data generation unit 21 is output to the control information generation unit 22 and the transmission data buffer memory 23.

[0036] The control information generation unit 22 receives reproduction data from the reproduction data generation unit and performs analysis processing for each piece of reproduction data. The analysis processing is a process of determining whether or not to permit the duplication or deletion of the fragmented reproduction data (permit / non-permit) for each piece of reproduction data (transmission data). Also, when there are multiple types of target data for the analysis processing, for example, when there is acoustic data and tactile data, it is performed for each type of target data. Note that when there is acoustic data for the left ear and acoustic data for the right ear, they may be treated as different types of data. Specifically, when devices that receive and reproduce the reproduction data for the left ear and devices that receive and reproduce the reproduction data for the right ear are provided separately, the control information for the acoustic data for the left ear and the control information for the acoustic data for the right ear may be generated separately.

[0037] When a "permission flag" that permits duplication or deletion is set for the transmission data, duplication or deletion of the reproduction data extracted from the reproduction data can be performed in the acoustic reproduction device 3 or the tactile reproduction device 4. Specifically, duplication of the reproduction data is performed to prevent the reproduction of the data from being interrupted when the reception process in the reproduction device that receives the transmission data has not been performed normally for a certain period of time. Thereby, symptoms such as dropout can be prevented by continuously reproducing the same reproduction data.

[0038] Also, when the reproduction process in the reproduction device stalls and the reproduction data stored in the buffer memory provided in the reproduction device overflows, there is a possibility that a part of the received reproduction data may be lost. In such a case, by deleting the reproduction data that can be deleted without problems, loss of important reproduction data can be prevented.

[0039] The control information generation unit 22 performs a process of determining for each piece of reproduction data whether such duplication or deletion may be performed in the reproduction data. The determination of whether the reproduction data may be copied or discarded is made based on a perception feature amount indicating how much the user will perceive it when the reproduction data is copied or discarded. The perception feature amount is calculated based on, for example, the reproduction level of the reproduction data (volume in the case of audio data) and the data waveform. Taking audio data as an example, for silent data or audio data with a volume lower than a predetermined level (e.g., 70 dB), even if it is discarded without reproduction or the audio data is copied and reproduced again, there is little risk of giving the user a sense of discomfort or unease. In other words, even if such reproduction data is copied or discarded, it is difficult for the user to perceive. For such reproduction data, a "permission flag" indicating that copying or discarding may be performed is set.

[0040] Also, for audio data with a large gain only at a predetermined frequency in the frequency characteristics, there is a high possibility that copying or discarding will give the user a sense of unease or discomfort. For such reproduction data, a "non - permission flag" indicating that copying or discarding is not to be performed is set. In other words, for reproduction data in which various frequencies are distributed complexly in the frequency characteristics, a "permission flag" indicating that copying or discarding may be performed is set. In addition, for audio data with a high temporal discontinuity such as noise, clapping sounds, white noise, or wave sounds, or highly random audio data, a "permission flag" indicating that copying or discarding may be performed is set.

[0041] The control information generated by the control information generation unit 22 is output to the reproduction data generation unit 21. Note that the control information generation unit 22 may output the generated control information to the transmission data generation unit 24, and the transmission data generation unit 24 may generate transmission data including the reproduction data and the control information.

[0042] The reproduction data generation unit 21 that has acquired the control information from the control information generation unit 22 outputs the reproduction data and the control information as a pair to the transmission data buffer memory 23.

[0043] The buffer memory 23 for transmission data is a buffer memory that temporarily stores the reproduction data generated by the reproduction data generation unit 21 and the control information as a set. Each reproduction data is temporarily stored in the buffer memory 23 for transmission data and then read out by the transmission data generation unit 24 as appropriate.

[0044] The transmission data generation unit 24 reads out the set of reproduction data and control information from the buffer memory for transmission data, and further adds information necessary for communication such as information for specifying the reproduction device at the transmission destination to generate transmission data. The transmission data is generated as, for example, packet data.

[0045] The transmission data generation unit 24 outputs the generated transmission data to the transmission unit 25.

[0046] The transmission unit 25 transmits the transmission data to each reproduction device such as the audio reproduction device 3 and the tactile reproduction device 4. When the communication is wireless communication, for example, the transmission unit 25 performs a process of modulating the transmission data into an RF (Radio Frequency) signal and performs a transmission process using wireless communication.

[0047] The audio reproduction device 3 and the tactile reproduction device 4 receive the transmission data (packet data) including the audio data and tactile data as reproduction data, and reproduce the reproduction data to provide the user with an integrated user experience. Specifically, the audio reproduction device 3 provides the user with reproduced sound by reproducing the audio data. Also, the tactile reproduction device 4 provides the user with a tactile stimulus by reproducing the tactile data.

[0048] It is important that the reproduced sound and tactile stimulus provided by the audio reproduction device 3 and the tactile reproduction device 4 are synchronized with each other, that is, timed correctly. For example, by providing the user with the sound heard when the user hits an obstacle in the virtual space and the tactile stimulus when hitting the obstacle at the same timing, the user can obtain a virtual experience with little difference from the real space. However, if this timing is off, a satisfactory experience cannot be provided to the user.

[0049] Therefore, the audio playback device 3 and the tactile playback device 4 are configured to play back the playback data synchronously.

[0050] The audio playback device 3 includes a communication processing unit 31, a playback data acquisition unit 32, a buffer memory 33 for playback data, a control unit 34, and a playback processing unit 35.

[0051] The communication processing unit 31 performs reception processing of transmission data (packet data) transmitted from the transmission device 2. Further, the communication processing unit 31 performs transmission and reception processing of synchronization information, which will be described later, with the tactile playback device 4. The received data (packet data) received by the communication processing unit 31 is output to the playback data acquisition unit 32.

[0052] The playback data acquisition unit 32 acquires playback data and control information from the received data and outputs them to the buffer memory 33 for playback data.

[0053] The buffer memory 33 for playback data is a buffer memory that temporarily stores the playback data and control information extracted by the playback data acquisition unit 32 from the received data. The playback data and control information are temporarily stored in the buffer memory 33 for playback data and then read out by the playback processing unit 35 as needed.

[0054] The control unit 34 monitors the amount of playback data stored in the buffer memory 33 for playback data. Specifically, it determines whether the amount of playback data stored in the buffer memory 33 for playback data is less than a first threshold value. And when it is determined that the amount is less than the first threshold value, synchronization information is generated. The synchronization information is information transmitted to other playback devices to synchronize the timing of duplicating or discarding playback data. For example, it may be information indicating the storage ratio (0% - 100%) of the playback data in the buffer memory 33 for playback data, or it may be flag information indicating that the storage amount has become less than the first threshold value. Alternatively, the synchronization information may be information (duplication information) indicating that duplication of the playback data is to be performed. The first threshold value is a threshold value for determining whether there is a possibility that the reproduction data stored in the reproduction data buffer memory 33 becomes insufficient.

[0055] Further, the control unit 34 determines whether or not the amount of reproduction data stored in the reproduction data buffer memory 33 is equal to or greater than a second threshold value (> the first threshold value). When it is determined that the amount is equal to or greater than the second threshold value, synchronization information is generated. The synchronization information may be, for example, information indicating the accumulation ratio (0% to 100%) of the reproduction data in the reproduction data buffer memory 33, or flag information indicating that the accumulated amount has become equal to or greater than the second threshold value. Alternatively, the synchronization information may be information (discard information) indicating that the reproduction data is to be discarded. The second threshold value is a threshold value for determining whether the reproduction data stored in the reproduction data buffer memory 33 is likely to overflow.

[0056] The information on the accumulated amount or the flag information generated by the control unit 34 is transmitted as synchronization information to the tactile reproduction device 4 via the communication processing unit 31. That is, the tactile reproduction device 4 performs reproduction synchronized with the acoustic reproduction device 3 by appropriately replicating or discarding the reproduction data using the synchronization information received from the acoustic reproduction device 3. Note that synchronization information may be transmitted from the tactile reproduction device 4 to the acoustic reproduction device 3, and the acoustic reproduction device 3 may perform reproduction synchronized with the tactile reproduction device 4.

[0057] When the control unit 34 calculates information indicating the accumulation ratio (or information indicating the number of reproduced data stored) as synchronization information, the accumulation ratio may be calculated as synchronization information at regular intervals and transmitted to the tactile reproduction device 4 via the communication processing unit 31.

[0058] The reproduction processing unit 35 appropriately acquires acoustic data as reproduction data from the reproduction data buffer memory 33 and performs reproduction processing. Thereby, the sound reproduced for the user is emitted. Note that the reproduction processing unit 35 performs reproduction processing according to the control of the control unit 34.

[0059] Specifically, when the reproduction data stored in the reproduction data buffer memory 33 is less than the first threshold value, the control unit 34 duplicates the reproduction data. The reproduction processing unit 35 reproduces the duplicated reproduction data, so that the same audio data is reproduced. That is, it is possible to prevent the sound from being interrupted. When duplicating the reproduction data, the control unit 34 refers to the control information about the reproduction data. If the control information of the reproduction data is a "non - permission flag" indicating that duplication is not allowed, the control unit 34 does not duplicate the reproduction data. Then, the control unit 34 duplicates the reproduction data whose control information is set as a "permission flag" indicating that duplication is possible. This makes it possible to perform reproduction control to prevent sound interruption without giving the user a sense of discomfort or uneasiness.

[0060] Also, when the reproduction data stored in the reproduction data buffer memory 33 provided in the audio reproduction device 3 is equal to or greater than the second threshold value, the control unit 34 discards the reproduction data. This prevents the reproduction data buffer memory 33 from overflowing.

[0061] When discarding the reproduction data, the control unit 34 refers to the control information about the reproduction data. If the control information of the reproduction data is a "non - permission flag" indicating that discarding is not allowed, the control unit 34 does not discard the reproduction data. Then, the control unit 34 discards the reproduction data whose control information is set as a "permission flag" indicating that discarding is possible. This makes it possible to perform reproduction control without giving the user a sense of discomfort or uneasiness.

[0062] The control unit 34 performs reproduction control in consideration of not only the amount of reproduction data stored in the reproduction data buffer memory 33 provided in the audio reproduction device 3 but also the amount of reproduction data stored in the reproduction data buffer memory provided in the tactile reproduction device 4.

[0063] Specifically, the control unit 34 acquires synchronization information from the tactile playback device 4 via the communication processing unit 31. And when the synchronization information received from the tactile playback device 4 indicates that replication of the playback data is required, replication of the playback data is performed even if the playback data stored in the playback data buffer memory 33 is not less than the first threshold value. Thereby, for example, it is possible to prevent only the playback data in the tactile playback device 4 from being replicated and the playback of the acoustic data from proceeding first. Note that the playback data targeted for the replication process performed by the control unit 34 at this time has a "permission flag" set as control information.

[0064] Also, when the synchronization information received from the tactile playback device 4 indicates that deletion of the playback data is required, deletion of the playback data is performed even if the playback data stored in the playback data buffer memory 33 is not more than the second threshold value. Thereby, for example, it is possible to prevent only the playback data in the tactile playback device 4 from being deleted and the playback of the acoustic data from being delayed. Note that the playback data targeted for the deletion process performed by the control unit 34 at this time has a "permission flag" set as control information.

[0065] The tactile playback device 4 includes a communication processing unit 41, a playback data acquisition unit 42, a playback data buffer memory 43, a control unit 44, and a playback processing unit 45.

[0066] The processing content of each unit performs the same processing as each unit included in the acoustic playback device 3. The difference is that the playback target is tactile data instead of acoustic data. Therefore, detailed description is omitted to avoid redundant description. However, the threshold value used by the control unit 44 included in the tactile playback device 4 to determine whether the accumulation amount of the playback data stored in the playback data buffer memory 43 is appropriate is set to the third threshold value instead of the first threshold value, and the fourth threshold value instead of the second threshold value. Note that the first threshold value and the third threshold value may be the same numerical value (for example, 40%). Also, the second threshold value and the fourth threshold value may be the same numerical value (for example, 80%).

[0067] Note that the acoustic playback device 3 may be configured to play acoustic data for the left ear and acoustic data for the right ear. For example, the acoustic playback device 3 may include a communication processing unit 31, a playback data acquisition unit 32, a buffer memory 33 for playback data, and a control unit 34, and may also include a playback processing unit 35 for the right ear and a playback processing unit 35 for the left ear. Also, when the acoustic playback device 3 is a device in which the left ear earphone and the right ear earphone are separated, the acoustic playback device 3 may separately include a communication processing unit 31, a playback data acquisition unit 32, a buffer memory 33 for playback data, a control unit 34, and a playback processing unit 35 for each of the left ear and the right ear. Further, an acoustic playback device 3 for playing acoustic data for the left ear and an acoustic playback device 3 for playing acoustic data for the right ear may be provided separately.

[0068] Similarly, when the tactile playback device 4 includes a plurality of tactile presentation units connected by wire, the tactile playback device 4 may commonly include a communication processing unit 41, a playback data acquisition unit 42, a buffer memory 43 for playback data, and a control unit 44 for each tactile presentation unit, and may also include a playback processing unit 45 for each tactile presentation unit. Also, when the tactile playback device 4 for the upper body and the tactile playback device 4 for the lower body are provided and each tactile playback device 4 is wirelessly connected, each tactile playback device 4 may include a communication processing unit 41, a playback data acquisition unit 42, a buffer memory 43 for playback data, a control unit 44, and a playback processing unit 45.

[0069] <2. Configuration of the computer device> The transmitter device 2, the acoustic playback device 3, and the tactile playback device 4 included in the synchronous playback system 1 are configured as computer devices in order to have the above-described respective functions. Note that each device does not necessarily have all the configurations shown below, and may have only some of them. As shown in Fig. 2, the CPU 71 of each device executes various processes according to programs stored in the ROM 72 and the non-volatile memory section 74 such as an EEP-ROM (Electrically Erasable Programmable Read-Only Memory), or programs loaded from the storage section 79 into the RAM 73. The RAM 73 also appropriately stores data and the like necessary for the CPU 71 to execute various processes. The CPU 71, ROM 72, RAM 73, and non-volatile memory section 74 are interconnected via a bus 83. An input / output interface 75 is also connected to this bus 83.

[0070] An input section 76 composed of an operator and an operation device is connected to the input / output interface 75. For example, as the input section 76, various operators and operation devices such as a keyboard, mouse, keys, dial, touch panel, touch pad, remote controller, etc. are assumed. The operation of the user is detected by the input section 76, and the signal corresponding to the input operation is interpreted by the CPU 71.

[0071] In addition, a display section 77 composed of an LCD or an organic EL panel, etc., and an audio output section 78 composed of a speaker, etc. are connected to the input / output interface 75 either integrally or separately. The display section 77 is a display section that performs various displays, and is composed of, for example, a display device provided on the housing of a computer device, a separate display device connected to the computer device, etc. The display section 77 executes displays of images for various image processes, moving images to be processed, etc. on the display screen based on the instructions of the CPU 71. The display section 77 also performs displays as a GUI (Graphical User Interface), such as various operation menus, icons, messages, etc., based on the instructions of the CPU 71.

[0072] The input / output interface 75 may be connected to a storage unit 79 composed of a hard disk, a solid-state memory, etc., and a communication unit 80 composed of a modem, etc.

[0073] The communication unit 80 performs communication processing via a transmission path such as the Internet, wired / wireless communication with various devices, and bus communication.

[0074] The input / output interface 75 may also be connected to a drive 81 as needed, and a removable storage medium 82 such as a magnetic disk, an optical disk, a magneto-optical disk, or a semiconductor memory is appropriately mounted. The drive 81 can read data files such as image files and various computer programs from the removable storage medium 82. The read data files can be stored in the storage unit 79, or the images and sounds included in the data files can be output by the display unit 77 and the audio output unit 78. Also, computer programs etc. read from the removable storage medium 82 are installed in the storage unit 79 as needed.

[0075] In this computer device, for example, software for the processing of this embodiment can be installed via network communication by the communication unit 80 or via the removable storage medium 82. Alternatively, the software may be stored in the ROM 72, the storage unit 79, etc. in advance.

[0076] When the CPU 71 performs a processing operation based on various programs, necessary information processing and communication processing as the transmitter device 2, the audio playback device 3, and the tactile playback device 4 are executed. Note that the information processing devices constituting the transmitter device 2, the audio reproduction device 3, and the tactile reproduction device 4 are not limited to being configured by a single computer device as shown in FIG. 2, and a plurality of computer devices may be systematized and configured. The plurality of computer devices may be systematized by a LAN or the like, or may be arranged at a remote location by a VPN or the like using the Internet or the like. The plurality of information processing devices may include information processing devices as a server group (cloud) that can be used by cloud computing services.

[0077] <3. Duplication and Deletion of Reproduction Data> <3-1. First Example of Duplication> The variation in the accumulation amount of audio data stored in the reproduction data buffer memory 33 of the audio reproduction device 3 and the first example of the duplication of audio data will be described.

[0078] FIG. 3 shows the change in the accumulation amount of audio data stored in the reproduction data buffer memory 33 at each time from time T0, time T1 after a predetermined time has elapsed from time T0, time T2 after a predetermined time has elapsed from time T1, ··· time T7.

[0079] At time T0, six pieces of audio data (DataA1 to DataA6) of audio data A1 to A6 are stored in the reproduction data buffer memory 33. Among the audio data A1 to A6, the one to be reproduced earliest is the audio data A1. That is, the audio data A6 is the reproduction data that the audio reproduction device 3 received last.

[0080] Also, for the sake of simplifying the explanation, the capacity occupied by one piece of audio data in the storage capacity of the reproduction data buffer memory 33 is assumed to be 10%. Note that the aforementioned first threshold value in the reproduction data buffer memory 33 is 40%. That is, when the accumulation amount of audio data in the reproduction data buffer memory 33 becomes less than 40%, the duplication of audio data is performed.

[0081] In FIG. 3, the information of "OK" or "NG" attached to the left of the acoustic data indicates a flag showing permission / non - permission of duplication. That is, the acoustic data marked as "OK" is the acoustic data with the aforementioned "permission flag" set, and the acoustic data marked as "NG" is the acoustic data with the aforementioned "non - permission flag" set.

[0082] At time T1 when the time has elapsed by the reproduction time length of one acoustic data as a predetermined time from time T0, the acoustic data A1 has been used for reproduction and deleted from the reproduction data buffer memory 33.

[0083] At time T2 when a further predetermined time has elapsed from time T1, the acoustic data A2 has been used for reproduction and deleted from the reproduction data buffer memory 33.

[0084] At time T3 when a further predetermined time has elapsed from time T2, the acoustic data A3 has been used for reproduction and deleted from the reproduction data buffer memory 33. Here, the accumulation amount of the acoustic data in the reproduction data buffer memory 33 is 30%, which is less than 40% that is the first threshold value.

[0085] The control unit 34 determines that the accumulation amount of the acoustic data in the reproduction data buffer memory 33 is insufficient. Also, the acoustic data A3 used for this reproduction is the acoustic data with the "permission flag" set. Therefore, the control unit 34 duplicates the acoustic data A3 (the state at time T3' in FIG. 3).

[0086] At time T4 when a further predetermined time has elapsed from time T3, the acoustic playback device 3 newly receives the acoustic data A7, and the duplicated acoustic data A3 is used for reproduction and deleted from the reproduction data buffer memory 33 again. The accumulation amount of the acoustic data at time T4 is 40%, which is not less than 40% that is the first threshold value. Therefore, the duplication of the acoustic data is not performed.

[0087] At time T5 when a further predetermined time has elapsed since time T4, the acoustic data A4 has been used for playback and deleted from the playback data buffer memory 33. Here, the accumulation amount of the acoustic data in the playback data buffer memory 33 is 30%, which is less than 40% that is the first threshold value.

[0088] However, a "disallow flag" is set in the acoustic data A4 used for the current playback. Therefore, the control unit 34 does not duplicate the acoustic data A4. That is, the accumulation amount of the acoustic data in the playback data buffer memory 33 remains 30%.

[0089] At time T6 when a further predetermined time has elapsed since time T5, the acoustic data A5 has been used for playback and deleted from the playback data buffer memory 33. Here, the accumulation amount of the acoustic data in the playback data buffer memory 33 is 20%, which is less than 40% that is the first threshold value.

[0090] However, a "disallow flag" is set in the acoustic data A5 used for the current playback. Therefore, the control unit 34 does not duplicate the acoustic data A5. That is, the accumulation amount of the acoustic data in the playback data buffer memory 33 remains 20%.

[0091] At time T7 when a further predetermined time has elapsed since time T6, the acoustic data A6 has been used for playback and deleted from the playback data buffer memory 33. Here, the accumulation amount of the acoustic data in the playback data buffer memory 33 is 10%, which is less than 40% that is the first threshold value.

[0092] In addition, the acoustic data A6 used for the current playback is the acoustic data with a "permit flag" set. Therefore, the control unit 34 duplicates the acoustic data A6 and restores the accumulation amount of the acoustic data in the playback data buffer memory 33 from 10% to 20% (the state at time T7' in FIG. 3).

[0093] Note that, by replicating a plurality of acoustic data A6 at the timing of time T7', the accumulation amount may be made to be equal to or greater than the first threshold value. Alternatively, at the timing of time T8 (not shown), it may be determined that the accumulation amount is again less than the first threshold value, and one acoustic data A6 may be replicated at the timing of time T8'. Also, when the accumulation amount becomes less than the first threshold value, replication of the acoustic data may be performed until it recovers to a value different from the first threshold value (for example, a reference value). For example, the first threshold value is set to 40% and the reference value is set to 60%. This prevents the accumulation amount near the first threshold value from being maintained for a long time and the determination process and replication process from being executed frequently.

[0094] Note that instead of replicating the acoustic data, it can also be substituted by ceasing the deletion process of the acoustic data. That is, when the acoustic data A3 is used for reproduction at the timing of time T3 and the deletion process of the acoustic data A3 is not executed from the reproduction data buffer memory 33, the same effect can be obtained.

[0095] <3-2. Second Example of Replication> The second example of replication is an example in which replication of the acoustic data is performed earlier than the first example of replication. Note that, although the same state as the first example of replication appears in FIG. 4 used to explain the second example of replication, the explanation thereof is omitted.

[0096] At time T0, six acoustic data A1 to A6 are accumulated in the reproduction data buffer memory 33. Also, one acoustic data is data corresponding to 10% of the accumulation amount in the reproduction data buffer memory 33.

[0097] Since the movement is the same as that of the first example of replication until time T4, the explanation is omitted.

[0098] At time T5 when a predetermined time has elapsed since time T4, the acoustic data A4 has been subjected to playback and deleted from the playback data buffer memory 33. Here, the accumulation amount of the acoustic data in the playback data buffer memory 33 is 30%, which is less than 40% that is the first threshold value.

[0099] Here, the control unit 34 executes the duplication process of the acoustic data. Among the acoustic data stored in the playback data buffer memory 33, the acoustic data in which a "permission flag" is set as control information and which is the playback data to be subjected to playback earliest is selected as the target of the duplication process.

[0100] A "non - permission flag" is set for the acoustic data A5, and a "permission flag" is set for the acoustic data A6. Therefore, the control unit 34 selects the acoustic data A6 as the duplication target.

[0101] The duplicated acoustic data A6 is stored in the playback data buffer memory 33 so as to be subjected to playback immediately before or after the original acoustic data A6. That is, the acoustic data stored in the playback data buffer memory 33 at the time point of time T5' is subjected to playback in the order of acoustic data A5, acoustic data A6, acoustic data A6, acoustic data A7.

[0102] At time T6 when a further predetermined time has elapsed since time T5, the acoustic data A5 has been subjected to playback and deleted from the playback data buffer memory 33. Here, the accumulation amount of the acoustic data in the playback data buffer memory 33 is 30%, and it is in a state of falling below 40% that is the first threshold value again.

[0103] Therefore, the control unit 34 executes the duplication process of the acoustic data again. Among the acoustic data stored in the playback data buffer memory 33, since the acoustic data in which a "permission flag" is set as control information and which is the playback data to be subjected to playback earliest is the acoustic data A6, the acoustic data A6 is selected as the target of the duplication process. At time T6', the acoustic data stored in the playback data buffer memory 33 is provided for playback in the order of acoustic data A6, acoustic data A6, acoustic data A6, and acoustic data A7.

[0104] Note that at time T6', acoustic data A6 is scheduled to be played three times. However, if new acoustic data is received before these playbacks are performed and the amount of acoustic data stored in the playback data buffer memory 33 has recovered to the first threshold value or more, the duplicated acoustic data A6 may be deleted without being played. As a result, it is not necessary to use the temporarily duplicated acoustic data A6 for playback.

[0105] <3-3. First Example of Discarding> The variation in the amount of acoustic data stored in the playback data buffer memory 33 of the acoustic playback device 3 and the first example of discarding acoustic data will be described.

[0106] FIG. 5 shows the changes in the amount of acoustic data stored in the playback data buffer memory 33 at times T1, T2, and T3, which are times after a predetermined time has elapsed from time T0.

[0107] Also, similar to the example of duplication, the capacity occupied by one piece of acoustic data in the storage capacity of the playback data buffer memory 33 is assumed to be 10%. Note that the second threshold value described above for the playback data buffer memory 33 is 80%. That is, when the amount of acoustic data stored in the playback data buffer memory 33 reaches 80% or more, the acoustic data is discarded.

[0108] In this example, "discarding" means that the acoustic data is deleted without being used for playback, and "deletion" means that the data becomes unnecessary and is deleted when the acoustic data is used for playback.

[0109] In Fig. 5, the information of "OK" or "NG" attached to the left of the acoustic data indicates a flag showing permission / non - permission for deletion. That is, the acoustic data marked as "OK" is the acoustic data with the aforementioned "permission flag" set, and the acoustic data marked as "NG" is the acoustic data with the aforementioned "non - permission flag" set.

[0110] At time T1 when a predetermined time has elapsed from time T0, the acoustic data A1 has been used for playback and deleted from the playback data buffer memory 33. Also, two newly received acoustic data A7 and A8 by the acoustic playback device 3 have been added to the playback data buffer memory 33. At this point, the accumulated amount of acoustic data in the playback data buffer memory 33 is 70%, which is less than 80% which is the second threshold value. Therefore, the deletion of acoustic data is not performed.

[0111] At time T2 when a further predetermined time has elapsed from time T1, the acoustic data A2 has been used for playback and deleted from the playback data buffer memory 33. Also, two newly received acoustic data A9 and A10 by the acoustic playback device 3 have been added to the playback data buffer memory 33. At this point, the accumulated amount of acoustic data in the playback data buffer memory 33 is 80%, which is 80% or more which is the second threshold value. Therefore, the control unit 34 determines to delete the acoustic data.

[0112] Here, the control unit 34 selects, as the target for deletion processing, the acoustic data with the "permission flag" set as control information among the acoustic data accumulated in the playback data buffer memory 33 and which is the earliest playback data to be used for playback. That is, the acoustic data A3 is selected as the deletion target and deleted (the state at time T2' in Fig. 5).

[0113] At time T3 when a further predetermined time has elapsed since time T2, since the acoustic data A3 has been discarded, the acoustic data A4 to be reproduced next is used for reproduction and deleted from the reproduction data buffer memory 33. Also, the three acoustic data A11, A12, and A13 newly received by the acoustic playback device 3 are added to the reproduction data buffer memory 33. At this point, the accumulation amount of the acoustic data in the reproduction data buffer memory 33 is 90%, which is 80% or more, which is the second threshold value. Therefore, the control unit 34 determines to discard the acoustic data.

[0114] Note that, in order to make the accumulation amount of the acoustic data in the reproduction data buffer memory 33 less than the second threshold value, it is necessary to discard two pieces of acoustic data. Therefore, the control unit 34 selects, as the objects of the discard process, in order, two pieces of the acoustic data in which the "permission flag" is set as control information and which are reproduced earlier among the acoustic data stored in the reproduction data buffer memory 33. That is, the acoustic data A6 and the acoustic data A7 are selected as the discard targets and discarded (the state at time T3' in FIG. 5).

[0115] Note that, in the first example of discarding, the case where the accumulation amount of the acoustic data stored in the reproduction data buffer memory 33 is compared with the threshold value each time the acoustic data is used for reproduction has been described. However, the accumulation amount and the threshold value may be compared more frequently, and the acoustic data may be discarded as appropriate. Thereby, it is possible to reduce the possibility of receiving a large amount of acoustic data at one time and losing the acoustic data unintentionally.

[0116] <3-4. Third Example of Duplication> The third example of duplication is an example in which duplication of the reproduction data is performed in both the acoustic playback device 3 and the tactile playback device 4 in response to a decrease in the accumulation amount of the acoustic data in the reproduction data buffer memory 33 of the acoustic playback device 3.

[0117] The upper part in FIG. 6 shows the accumulation state of acoustic data A1 to A6 in the reproduction data buffer memory 33 of the acoustic reproduction device 3. Also, the lower part in FIG. 6 shows the accumulation state of tactile data B1 to B8 in the reproduction data buffer memory 43 of the tactile reproduction device 4.

[0118] At time T0, six acoustic data are accumulated in the reproduction data buffer memory 33 of the acoustic reproduction device 3, and six tactile data are accumulated in the reproduction data buffer memory 43 of the tactile reproduction device 4.

[0119] At time T2 when a time that is twice the predetermined time has elapsed from time T0, the acoustic data A1 and A2 used for reproduction are deleted from the reproduction data buffer memory 33 of the acoustic reproduction device 3, and the tactile data B1 and B2 used for reproduction are deleted from the reproduction data buffer memory 43 of the tactile reproduction device 4.

[0120] Also, at time T2, the tactile reproduction device 4 is in a state where new tactile data B7 and B8 are added to the reproduction data buffer memory 43. Note that the acoustic reproduction device 3 is assumed to have failed to receive new acoustic data A7 and A8.

[0121] Since the accumulation amount of the acoustic data accumulated in the reproduction data buffer memory 33 of the acoustic reproduction device 3 is 40% or more, which is the first threshold value, the control unit 34 does not duplicate the acoustic data. Similarly, since the accumulation amount of the acoustic data accumulated in the reproduction data buffer memory 43 of the tactile reproduction device 4 is 40% or more, which is the third threshold value, the control unit 44 does not duplicate the tactile data.

[0122] At time T3 when a predetermined time has elapsed since time T2, the acoustic data A3 used for reproduction has been deleted from the reproduction data buffer memory 33 of the acoustic reproduction device 3, and the tactile data B3 used for reproduction has been deleted from the reproduction data buffer memory 43 of the tactile reproduction device 4. Also, reception of new acoustic data A7 and A8 has not yet been performed.

[0123] Since the amount of acoustic data stored in the reproduction data buffer memory 33 of the acoustic reproduction device 3 is less than 40%, which is the first threshold value, the control unit 34 determines to duplicate the acoustic data. Therefore, the control unit 34 notifies the tactile reproduction device 4 via the communication processing unit 31 of information such as the amount of stored acoustic data (30%) as synchronization information and flag information indicating that duplication of the acoustic data is to be performed. That is, the acoustic reproduction device 3 transmits synchronization information to the tactile reproduction device 4.

[0124] On the other hand, since the amount of acoustic data stored in the reproduction data buffer memory 43 of the tactile reproduction device 4 is 40% or more, which is the third threshold value, the control unit 44 does not originally need to duplicate the tactile data. However, since it has been found that the reproduction data is duplicated in the acoustic reproduction device 3 based on the synchronization information received from the acoustic reproduction device 3, the reproduction data buffer memory 43 of the tactile reproduction device 4 determines to perform the duplication process of the tactile data.

[0125] The control unit 34 of the acoustic reproduction device 3 selects, as the acoustic data to be duplicated, the acoustic data for which the "permission flag" is set as control information and which is the earliest to be used for reproduction. Similarly, the control unit 44 of the tactile reproduction device 4 selects, as the tactile data to be duplicated, the tactile data for which the "permission flag" is set as control information and which is the earliest to be used for reproduction.

[0126] That is, the control unit 34 of the acoustic reproduction device 3 selects the acoustic data A3 as the duplication target, and the control unit 44 of the tactile reproduction device 4 selects the tactile data B3 as the duplication target. As a result, the control unit 34 and the control unit 44 are in the state shown at time T3'.

[0127] Note that in the state shown in FIG. 6, since the "permission flag" is set for both the audio data A3 and the tactile data B3 that are used for reproduction at the same timing, even if the reproduction data is duplicated or discarded, the reproduction of the audio reproduction device 3 and the tactile reproduction device 4 is in a synchronized state.

[0128] However, there may be a case where the "permission flag" is set for only one of the audio data A3 and the tactile data B3. In that case, it may be possible to duplicate the reproduction data with the closest reproduction timing.

[0129] For example, as shown in FIG. 7, the audio data A3, A4, A5, A6 are stored in the reproduction data buffer memory 33, the "permission flag" is set only for the audio data A3, and the tactile data B3, B4, B5, B6, B7, B8, B9 are stored in the reproduction data buffer memory 43, and the "permission flag" is set for the tactile data B4, B8, B9. Consider this case. When the audio data A3 is used for reproduction in this state, duplication of the audio data A3 is determined based on the above-described conditions. Then, in the tactile reproduction device 4, it is desirable that duplication of the tactile data B4, which is the tactile data closest to the audio data A3 in terms of reproduction timing among the tactile data for which the "permission flag" is set, is determined.

[0130] Also, as described in the second example of duplication, when the duplication of the reproduction data is performed earlier, a state as shown in FIG. 8 can be considered. Specifically, the audio data A3, A4, A5, A6 are stored in the reproduction data buffer memory 33, the "permission flag" is set only for the audio data A6, and the tactile data B3, B4, B5, B6, B7, B8, B9 are stored in the reproduction data buffer memory 43, and the "permission flag" is set for the tactile data B3, B5, B7, B8, B9. When the acoustic data A3 is used for playback in this state, replication is determined based on the above-described conditions of the acoustic data A3. Then, in the tactile playback device 4, it is desirable that replication of either the tactile data B5 or the tactile data B7, which is the tactile data with the closest playback timing to the acoustic data A6 among the tactile data for which the "permission flag" is set, be determined.

[0131] Note that several methods can be considered to prevent the playback data subject to replication processing or deletion processing from being played back at different timings. One method is that when the control information generation unit 22 of the transmission device 2 attaches a permission flag or a non-permission flag as control information to the playback data, if it is possible to set the permission flag for all of the playback data played back at the same timing, the permission flag is set. If it is necessary to set the non-permission flag for at least one of the playback data played back at the same timing, it is conceivable to set the non-permission flag for all of the playback data uniformly. As a result, the states shown in FIGS. 7 and 8 cannot exist, and the playback-side devices (the acoustic playback device 3 and the tactile playback device 4) can perform replication or deletion of the playback data without considering the control information regarding each other's playback data.

[0132] As another method, when a certain playback device determines to perform replication or deletion of the playback data, it is conceivable to include information specifying the playback data subject to replication or deletion in the synchronization information transmitted to other playback devices. As a result, it becomes possible to select the playback data subject to replication or deletion from among the playback data played back at the same timing.

[0133] Note that when there is no playback data for which replication or deletion is permitted among the playback data played back at the same timing, as described above, it is desirable to select the playback data with the closest possible playback timing as the target.

[0134] In addition, for example, there are a playback device for playing acoustic data for the left ear, a playback device for playing acoustic data for the right ear, and a playback device for playing tactile data, and when playing back playback data by synchronizing them, permission flags are set for all flags, and it may be difficult to select the playback data to be played back at the same timing. In particular, it is obvious that it becomes more difficult as the number of playback devices increases. In that case, it is desirable to select the playback data to be duplicated or discarded according to whether or not the user is likely to perceive a deviation in the playback timing. For example, acoustic data for the left ear and acoustic data for the right ear, for which the user is relatively likely to perceive a deviation in the playback timing, are selected as the playback data to be duplicated or discarded that are played back at the same timing, and for tactile data for which it is difficult to perceive a slight deviation, the playback data (tactile data) whose deviation in the playback timing is within a predetermined time may be selected as the data to be duplicated or discarded. Thereby, it is possible to duplicate or discard the playback data so as not to give the user as much discomfort or unpleasantness as possible.

[0135] <4. Processing Example> <4-1. Processing Example of Transmitting Device> An example of the processing executed by the transmitting device 2 to realize each of the functions described above will be described with reference to FIG. 9. Note that each process shown in FIG. 9 is a process executed by an arithmetic processing unit such as the CPU 71 provided in the transmitting device 2. In this example, the CPU 71 of the transmitting device 2 is simply described as an “arithmetic processing unit”.

[0136] In addition, similar to the example described above, an example in which the transmitting device 2 acquires the original data of the acoustic data and the original data of the tactile data from the data supply source SC, converts them into small data units (packet data), and transmits them to each playback device will be described.

[0137] In step S201, the arithmetic processing unit determines whether the original data supplied from the data source SC has ended. If there is no more original data supplied from the data source SC, it is determined that it has ended. If there is still original data supplied from the data source SC, it is determined that it has not ended.

[0138] If it is determined that the original data supplied from the data source has ended, the arithmetic processing unit ends the series of processes shown in FIG. 9. On the other hand, if it is determined in step S201 that the original data has not ended, the arithmetic processing unit identifies the data type of the acquired original data in step S202. In the above example, it is identified whether the original data is original acoustic data or original tactile data. Then, the arithmetic processing unit performs each process after step S203 according to the data type of the acquired original data.

[0139] In step S203, the arithmetic processing unit extracts (or calculates) the perceptual feature amount of the acquired original data. The perceptual feature amount is information indicating how easy (or difficult) it is for the user to perceive when the reproduced data generated from the original data is copied or discarded.

[0140] In step S204, the arithmetic processing unit determines whether the perceptual feature amount is less than the threshold value. The original data with a perceptual feature amount less than the threshold value is regarded as reproduced data that is difficult for the user to perceive even when the generated reproduced data is copied or discarded. In the case of acoustic data, it is silent data or acoustic data with a volume below a predetermined level. Also, in the case of tactile data, it is tactile data with a presentation intensity of the tactile stimulus below a predetermined intensity. Also, in the case of video data, it is an image with an overall low luminance or data with a small difference from the previous video data.

[0141] If the perceptual feature amount is less than the threshold value, the arithmetic processing unit sets a "permission flag" as control information for the original data in step S205. On the other hand, when the sensed feature amount is equal to or greater than the threshold value, the arithmetic processing unit sets a "non - permission flag" as control information for the original data in step S206.

[0142] Next, in step S207, the arithmetic processing unit determines whether control information has been set for all of the original data of the reproduction data reproduced at the same timing. For example, in the example shown in FIG. 6, the reproduction data reproduced at the same timing is acoustic data A1 and tactile data B1. Also, acoustic data A2 and tactile data B2 are also reproduction data reproduced at the same timing.

[0143] Here, an explanation will be given by taking the acoustic data A1 and the tactile data B1 as examples. When each process from step S201 to step S206 is executed with the original data of the acoustic data A1 as the processing target and the process proceeds to the process of step S207, the control information has not yet been set for the original data of the tactile data B1. In this case, the determination in step S207 is a "No" determination.

[0144] When the determination in step S207 is a "No" determination, the arithmetic processing unit proceeds to step S208 and selects the data for which the control information has not been set, that is, the original data of the tactile data B1, from the reproduction data reproduced at the same timing. Then, the arithmetic processing unit performs the processes from step S202 to step S206 on the original data of the tactile data B1 and sets either a "permission flag" or a "non - permission flag" as control information.

[0145] When either a "permission flag" or a "non - permission flag" is set as control information for the original data of both the acoustic data A1 and the tactile data B1, the determination in step S207 is a "Yes" determination.

[0146] If the determination in step S207 is "Yes", the arithmetic processing unit proceeds to step S209 and determines whether all of the control information set in the original data of the reproduction data reproduced at the same timing is the "permission flag". In the above example, if the "permission flag" is set for the original data of each of the acoustic data A1 and the tactile data B1, the determination in step S209 is a "Yes" determination, and if the "non - permission flag" is set for at least one of them, the determination in step S209 is a "No" determination.

[0147] In the case of a "Yes" determination in step S209, since the "permission flag" has already been set as the control information for all of the target original data (the original data of the acoustic data A1 and the tactile data B1 respectively), that is, since there is no need to unify the control information, the process proceeds to step S211 without performing processing on the control information.

[0148] On the other hand, in the case of a "No" determination in step S209, since there is original data in which the "non - permission flag" is set as the control information among the target original data, the process proceeds to step S210 to change the control information.

[0149] In step S210, the arithmetic processing unit resets the "non - permission flag" as the control information for all of the target original data. That is, for the original data in which the "permission flag" is set as the control information, a process of changing the control information to the "non - permission flag" is performed.

[0150] In step S211, the arithmetic processing unit generates reproduction data from the original data. Specifically, the acoustic data A1 and the tactile data B1 reproduced at the same timing are each generated as reproduction data. Each of the generated reproduction data (acoustic data A1 and tactile data B1) is stored in the transmission - data - use buffer memory 23 in combination with the control information respectively.

[0151] Next, in step S212, the arithmetic processing unit acquires a set of reproduction data and control information from the transmission data buffer memory 23, generates packet data as transmission data in step S213, and performs transmission processing in step S214. By this transmission processing, the packet data is transmitted to a reproduction device such as the audio reproduction device 3 or the tactile reproduction device 4.

[0152] By the transmission device 2 performing each process from step S201 to step S214, audio data and tactile data as transmission data are transmitted to the reproduction device together with control information.

[0153] <4-2. Example of Processing of Reproduction Device 1> An example of the processing executed by the reproduction device to realize each of the above-described functions will be described with reference to FIG. 10 and the like. Note that each process described here will be described as an example of the process executed by an arithmetic processing unit such as the CPU 71 of the audio reproduction device 3, but these processes are also the processes executed by an arithmetic processing unit such as the CPU 71 of the tactile reproduction device 4. That is, the flowcharts of FIGS. 10 and 11 perform each process using information received from the transmission device 2 or another reproduction device by a certain reproduction device. When a certain reproduction device is the audio reproduction device 3, another reproduction device is the tactile reproduction device 4, and when a certain reproduction device is the tactile reproduction device 4, another reproduction device is the audio reproduction device 3.

[0154] When the process is executed by the CPU 71 of the tactile reproduction device 4, the audio reproduction data may be changed to tactile reproduction data, the reproduction data buffer memory 33 may be changed to the reproduction data buffer memory 43, and the control unit 34 may be changed to the control unit 44. In the description, the CPU 71 of the audio reproduction device 3 is simply described as the "arithmetic processing unit".

[0155] In step S301, the arithmetic processing unit receives packet data which is the transmission data transmitted from the transmission device 2. In step S302, the arithmetic processing unit acquires acoustic data, which is reproduction data, and control information from the received packet data.

[0156] In step S303, the arithmetic processing unit stores the reproduction data and the control information as a set in the reproduction data buffer memory 33.

[0157] In step S304, the arithmetic processing unit acquires the reproduction data from the reproduction data buffer memory 33 and executes reproduction processing. The reproduction data used for reproduction is deleted from the reproduction data buffer memory 33, but since it may be replicated in subsequent processing, it is stored together with the control information in any storage unit in the acoustic reproduction device 3.

[0158] In step S305, the arithmetic processing unit acquires (calculates) the amount of reproduction data stored in the reproduction data buffer memory 33.

[0159] In step S306, the arithmetic processing unit generates synchronization information based on the acquired amount of storage information. Note that the example shown in FIG. 10 is an example of periodically transmitting synchronization information to another reproduction device (for example, the tactile reproduction device 4) regardless of the amount of storage. Therefore, the synchronization information is generated unconditionally. When the amount of storage is less than the first threshold or greater than or equal to the second threshold, that is, when replication or deletion of reproduction data is performed in the acoustic reproduction device 3, only when synchronization information is transmitted, before executing step S306, a process of comparing the amount of storage with each threshold may be performed, and the synchronization information generation process of step S306 may be appropriately performed according to the comparison result.

[0160] In step S307, the arithmetic processing unit transmits the synchronization information to another reproduction device (for example, the tactile reproduction device 4).

[0161] Also, in step S308, the arithmetic processing unit receives synchronization information from another reproduction device.

[0162] The arithmetic processing unit that has obtained the synchronization information of itself (the audio playback device 3) and the synchronization information of another playback device (the tactile playback device 4) determines, in step S309 of FIG. 11, whether the accumulation amount in the buffer memory of any of the playback devices (the audio playback device 3 or the tactile playback device 4) including itself is less than the threshold value. For the accumulation amount in the audio playback device 3, a comparison process using the first threshold value is performed, and for the accumulation amount in the tactile playback device 4, a comparison process using the third threshold value instead of the first threshold value is performed. However, the first threshold value and the third threshold value may be the same value (for example, 40%).

[0163] When the accumulation amount in the audio playback device 3 is less than the first threshold value, or when the accumulation amount in the tactile playback device 4 is less than the third threshold value, or when both conditions are satisfied, it means that the accumulation amount of the playback data in any of the buffer memories has decreased, and there is a concern about the depletion of the playback data. In this case, the arithmetic processing unit proceeds to step S310 and determines whether a "permission flag" is set as the control information of the playback data that was last used for playback.

[0164] When the "permission flag" is set as the control information of the playback data, in step S311, the arithmetic processing unit duplicates the playback data that was last used for playback, that is, the playback data that was deleted from the playback data buffer memory 33 in step S304 and stored in another storage unit, and stores it again in the playback data buffer memory 33. When storing the duplicated playback data in the playback data buffer memory 33, the duplicated playback data is set to be selected as the next playback data.

[0165] On the other hand, if it is determined in step S310 that a "non - permission flag" is set as the control information of the playback data, the arithmetic processing unit does not execute the duplication process in step S311 and returns to the process of step S301 again. That is, the arithmetic processing unit receives the packet data from the transmitter device 2, performs the playback of the playback data, and appropriately duplicates or discards the playback data.

[0166] Depending on the communication quality between the transmitter device 2 and the audio playback device 3, there may be cases where packet data cannot be received from the transmitter device 2. In such cases, the playback process in step S304 and the like are executed without performing the reception process in step S301. In order to implement such processing, the reception process of packets, the playback process of the playback data, the determination process as to whether the duplication or deletion of the playback data is necessary, and the duplication process and the deletion process may each be configured to be executed as multitasks.

[0167] Return to the description of FIG. 11. In step S309, when it is determined that the accumulated amount in the buffer memory of all the playback devices (the audio playback device 3 or the tactile playback device 4) is equal to or greater than the threshold values (the first threshold value, the third threshold value), the arithmetic processing unit performs a process of determining whether to discard the playback data in step S312. Specifically, the arithmetic processing unit determines whether the accumulated amount in the buffer memory of any of the playback devices including itself is equal to or greater than the threshold value. Note that, for the accumulated amount in the audio playback device 3, a comparison process using the second threshold value is performed, and for the accumulated amount in the tactile playback device 4, a comparison process using the fourth threshold value instead of the second threshold value is performed. However, the second threshold value and the fourth threshold value may be the same value (for example, 80%).

[0168] When the accumulated amount in the audio playback device 3 is equal to or greater than the second threshold value, or when the accumulated amount in the tactile playback device 4 is equal to or greater than the fourth threshold value, or when both conditions are satisfied, it means that the accumulated amount of the playback data has increased in any of the buffer memories, and there is a possibility that the playback data may be lost unintentionally. In this case, the arithmetic processing unit proceeds to step S313, and discards the playback data that is earliest used for playback among those for which the permission flag is set as control information. Thereby, the accumulated amount in the buffer memory can be reduced.

[0169] Note that the number of pieces of playback data discarded in step S313 does not have to be one. For example, a plurality of pieces of playback data may be selected so that the accumulated amount in the buffer memory 33 for the playback data becomes less than the second threshold value.

[0170] After executing step S313, or after determining in step S312 that the accumulated amount in the buffer memory of all playback devices (the audio playback device 3 or the tactile playback device 4) is less than the threshold value (the second threshold value, the fourth threshold value), the arithmetic processing unit returns to the processing of step S301 again.

[0171] By executing each process shown in FIGS. 10 and 11, it is possible to realize a change in the accumulated amount as shown in FIGS. 3 and 5 in the buffer memory 33 for playback data of the audio playback device 3.

[0172] Note that the examples shown in FIGS. 10 and 11 are examples where no replication of the playback data is performed when a "disallow flag" is set in the playback data that was finally used for playback even when replication of the playback data is required. In this case, after looping through each process several times, the replication process for the playback data is executed at the timing when the playback data with the "allow flag" set is played back. However, as shown in FIG. 4, when replication of the playback data is required, the replication process may be performed on playback data other than the playback data that was finally used for playback. For example, instead of steps S310 and S311 in FIG. 11, the replication process may be performed by selecting, as the replication target, the playback data with the earliest playback timing among the playback data with the "allow flag" set.

[0173] <4-3. Processing Example 2 of Playback Device> In processing example 1, after playing back the playback data, it was determined whether replication or deletion of the playback data was necessary, and if replication was necessary, replication of the played-back playback data was performed. In processing example 2, it is determined whether replication or deletion of the playback data is necessary, and after appropriately performing the replication or deletion process, the playback data is played back.

[0174] Note that for processes similar to those in processing example 1, the same step numbers are assigned, and the description is omitted as appropriate. Also, although the arithmetic processing unit of the audio playback device 3 is taken as an example of the processing unit that executes each process, the arithmetic processing unit of the tactile playback device 4 may also execute it.

[0175] By executing each process from step S301 to step S303 in FIG. 12, the arithmetic processing unit acquires reproduction data and control information from the received packet data and stores them in the reproduction data buffer memory 33.

[0176] By executing each process from step S305 to step S307, the arithmetic processing unit acquires (calculates) information on the accumulation amount in the reproduction data buffer memory 33, generates synchronization information, and transmits it to other playback devices (haptic playback device 4). Also, by executing the process of step S308, the arithmetic processing unit receives synchronization information from other playback devices.

[0177] Based on the synchronization information acquired from the reproduction data buffer memory 33 and the synchronization information received from other playback devices, the arithmetic processing unit executes each process shown in FIG. 13. Specifically, in step S309, the arithmetic processing unit determines whether the accumulation amount in the buffer memory of any playback device (audio playback device 3 or haptic playback device 4) including itself is less than the threshold value.

[0178] If the accumulation amount in the audio playback device 3 is less than the first threshold value, or if the accumulation amount in the haptic playback device 4 is less than the third threshold value, or if both conditions are met, the arithmetic processing unit determines in step S321 whether a "permission flag" is set as the control information for the reproduction data to be played back next.

[0179] If the "permission flag" is set in the control information of the reproduction data to be played back next, the arithmetic processing unit executes a process of copying the reproduction data to be played back next and storing it in the reproduction data buffer memory 33 in step S322. Then, in step S304, the arithmetic processing unit acquires the reproduction data from the reproduction data buffer memory 33 and executes the reproduction process. That is, the reproduction data to be replicated temporarily increases by one (for example, from 1 to 2) when replicated in step S322, and decreases by one (for example, from 2 back to 1) when used for reproduction in step S304 and deleted from the reproduction data buffer memory 33.

[0180] Note that a "disallow flag" is set as the control information for the next reproduction data. If the replication of the reproduction data is not appropriate, the replication process of the reproduction data (the process in step S322) is not performed, and the reproduction process in step S304 is executed.

[0181] Also, in step S309, when it is determined that the accumulated amount in the buffer memory of all reproduction devices (the audio reproduction device 3 or the tactile reproduction device 4) is equal to or greater than the threshold values (the first threshold value, the third threshold value), since there is no need to replicate the reproduction data, the arithmetic processing unit performs a process of determining whether to discard the reproduction data in step S312. And when it is determined that the reproduction data needs to be discarded, the arithmetic processing unit discards the reproduction data in step S313.

[0182] On the other hand, when it is determined that neither replication nor discard of the reproduction data is necessary, that is, when the determination in step S309 is "No" and the determination in step S312 is "No", only the reproduction of the reproduction data is executed.

[0183] <4-4. Processing Example 3 of Reproduction Device> Processing example 3 is an example of preventing the replication and discard of reproduction data from being performed multiple times in a short period. By not continuously replicating and discarding the reproduction data frequently, it is possible to prevent giving the user a sense of discomfort or unease.

[0184] For the processes similar to those described in Process Example 1 and Process Example 2, the same step numbers are assigned, and the explanations are omitted as appropriate. Also, as in the precedent, each process is executed by an arithmetic processing unit such as the CPU 71 of the audio playback device 3.

[0185] The arithmetic processing unit executes the processes of steps S301, S302, S303, S305, S306, S307, and S308 in FIG. 12, extracts reproduction data and control information from the packet data, stores them in the reproduction data buffer memory 33, generates synchronization information from the accumulation amount information, and transmits it to other playback devices. Also, it receives synchronization information from other playback devices.

[0186] Subsequently, the arithmetic processing unit performs the process of step S309 in FIG. 14 to determine whether replication of the reproduction data is necessary. If at least one of the accumulation amounts of the reproduction data buffer memory 33 and the reproduction data buffer memory 43 is less than the threshold value, replication of the reproduction data is determined. That is, in step S321, the arithmetic processing unit checks the control information of the reproduction data to be reproduced next and determines whether the "permission flag" is set.

[0187] When the "permission flag" is set, the arithmetic processing unit proceeds to step S331 and determines whether the counter value is equal to or greater than a predetermined value.

[0188] Here, the counter value will be explained. The initial value of the counter value is set to 0, and it is incremented by 1 each time the reproduction data is reproduced without replication or deletion. That is, the counter value can be regarded as the elapsed time since the previous replication or deletion.

[0189] When the counter value is equal to or greater than the predetermined value, that is, when the predetermined time has elapsed since the previous replication or deletion, the arithmetic processing unit proceeds to step S332, replicates the reproduction data to be reproduced next, and stores it in the reproduction data buffer memory 33. Then, in step S333, the arithmetic processing unit resets the counter value to 0. Subsequently, in step S304, the arithmetic processing unit acquires reproduction data from the reproduction data buffer memory 33, executes reproduction processing, and deletes the reproduced reproduction data from the reproduction data buffer memory 33. As a result, the accumulation state of the reproduction data in the reproduction data buffer memory 33 returns to the state before the duplication process in step S332 and the reproduction process in step S304.

[0190] If it is determined in step S331 that the counter value is less than a predetermined value, that is, if not much time has elapsed since the duplication or deletion of the previous reproduction data, the arithmetic processing unit proceeds to step S334, adds 1 to the counter value, and performs the reproduction process of the reproduction data in step S304. As a result, the amount of reproduction data accumulated in the reproduction data buffer memory 33 decreases.

[0191] If it is determined in step S309 that duplication of the reproduction data is not necessary, the arithmetic processing unit determines in step S312 whether it is necessary to delete the reproduction data.

[0192] If it is determined that the reproduction data needs to be deleted in any of the buffer memories, the arithmetic processing unit determines in step S335 whether the "permission flag" is set as the control information of the reproduction data to be reproduced next.

[0193] If the "permission flag" is set, the arithmetic processing unit determines in step S336 whether the counter value is greater than or equal to a predetermined value. If the counter value is greater than or equal to the predetermined value, since it is difficult for the user to perceive even if the reproduction data is deleted here, the arithmetic processing unit proceeds to step S337 and deletes the reproduction data to be reproduced next. Then, the arithmetic processing unit resets the counter value to 0 in step S338 and performs the reproduction process in step S304. That is, one reproduction data is deleted and the next reproduction data is reproduced.

[0194] Also, when it is determined in step S312 that it is not necessary to discard the reproduction data, or when it is determined in step S335 that a "non - permission flag" is set in the control information of the reproduction data to be used for the next reproduction, or when it is determined in step S336 that the counter value is less than a predetermined value, the counter value is incremented by 1 in step S334 without discarding the reproduction data, and then the reproduction process is performed in step S304.

[0195] This prevents the reproduction data from being frequently copied or discarded, and also prevents the same reproduction data from being copied multiple times.

[0196] <4 - 5. Processing Example 4 of the Reproduction Device> In processing example 4, reuse is performed instead of copying the reproduction data. In the following description, an example obtained by modifying processing example 2 of the reproduction device will be described. Similar to the precedent, each process is executed by an arithmetic processing unit such as the CPU 71 of the audio reproduction device 3.

[0197] The arithmetic processing unit obtains reproduction data and control information from the packet data, stores them in the reproduction - data buffer memory 33, and transmits and receives synchronization information with other reproduction devices by performing the series of processes shown in FIG. 12.

[0198] Subsequently, in step S309 of FIG. 15, the arithmetic processing unit determines whether the accumulated amount of the buffer memory of each reproduction device is decreasing. When it is determined that the accumulated amount of the buffer memory is decreasing and corresponding measures are required (step S309: "Yes" determination), the arithmetic processing unit determines in step S321 whether a "permission flag" is set in the control information of the reproduction data to be used for the next reproduction.

[0199] When the "permission flag" is set, the arithmetic processing unit sets a "reuse flag" in the reproduction data to be used for the next reproduction in step S341. The reuse flag is a flag attached to reproduction data determined to be reproduced multiple times. The reuse flag can be attached only to reproduction data for which a "permission flag" is set as control information.

[0200] In step S304, the arithmetic processing unit acquires reproduction data from the reproduction data buffer memory 33 and performs reproduction processing.

[0201] In step S342, the arithmetic processing unit determines whether a reuse flag is set for the reproduction data reproduced immediately before. If the reuse flag is set, in step S343, the arithmetic processing unit deletes the reuse flag set for the reproduction data reproduced immediately before.

[0202] On the other hand, in step S321, if it is determined that a "non - permission flag" is set as the control information of the reproduction data to be reproduced next, the arithmetic processing unit executes the reproduction processing in step S304 without setting the reuse flag. In that case, in the determination process of step S342, the arithmetic processing unit determines that the reuse flag is not set for the reproduction data reproduced immediately before, and in step S344, performs a process of deleting the reproduction data reproduced immediately before from the reproduction data buffer memory 33. Note that the deletion process in step S344 is also executed when it is determined in step S309 that the accumulated amount of the buffer memory has not decreased (step S309: "No" determination).

[0203] In this way, by using the reuse flag, it is not necessary to duplicate the reproduction data. Note that such a processing mode can be realized even without using the reuse flag. For example, when it is determined as "Yes" in both the determination processes of step S309 and step S321 shown in FIG. 15, the reproduction processing may be performed so that the reproduction data is not deleted from the reproduction data buffer memory 33, and in other cases, the reproduction data may be deleted from the reproduction data buffer memory 33 after the reproduction processing.

[0204] Note that the mode of reusing instead of replicating the reproduction data as in this example can also be applied to Processing Example 1 or Processing Example 3 of the playback device by using a reuse flag.

[0205] <5. Modification Example> In each of the above-described examples, the transmission device 2 includes a control information generation unit 22, and the control information generation unit 22 analyzes the reproduction data and generates control information for the reproduction data. That is, in each playback device, the control information for each piece of reproduction data is confirmed, and it is determined whether the reproduction data can be replicated or discarded. However, the transmission device 2 does not include a control information generation unit 22, and it is not necessary to generate control information. In this case, each playback device can be selected as an object of replication processing or deletion processing according to the amount of reproduction data accumulated in the buffer memory regardless of the reproduction data. Even with such a configuration, it is possible to perform synchronized playback processing while eliminating problems such as sound skipping due to insufficient reproduction data in a plurality of playback devices.

[0206] In each of the above-described examples, no upper limit was set on the number of times the reproduction data to be replicated is replicated. However, in order to prevent the same reproduction data from being replicated excessively, an upper limit on the number of replications may be provided. For example, between the processes of step S310 and step S311 in FIG. 11, a determination process for checking the number of replications may be executed, and if the number of replications has not reached the upper limit value, the replication process of step S311 may be executed. Further, when the replication process of step S311 is executed, a process of adding 1 to a counter value for counting the number of replications is performed.

[0207] In the above-described embodiments, an example in which there are two playback devices has been described. However, even if there are three or more playback devices, the same operational effects can be obtained. In that case, for example, in the process of step S307 in FIG. 10, it is preferable to transmit synchronization information to two or more playback devices as other playback devices, and in the process of step S308, to receive synchronization information from two or more playback devices as other playback devices. Also, in each process of step S309 and step S312 in FIG. 11, a process of comparing the accumulation amount and the threshold value is performed for each playback device. By doing so, it is possible to perform the playback process while ensuring the synchronization state of a plurality of playback devices.

[0208] The above-described configuration is suitable when the playback data transmitted and received between the transmission device 2, the audio playback device 3, and the tactile playback device 4 is performed by wireless communication. However, it can also be applied even when the transmission and reception of the playback data is performed by wired communication. That is, when performing replication or deletion of the playback data in order to prevent sound dropout or the like caused by deterioration of the quality of the wired communication, by applying the configuration of the present disclosure, it is possible to perform the playback process by a plurality of playback devices while ensuring the synchronization state.

[0209] <6. Summary> As described in each of the above examples, the playback device (audio playback device 3) as the information processing device includes a playback data reception unit (communication processing unit 31) that receives the first playback data (audio data), and another playback device (tactile playback device 4) that synchronizes with the first playback data and plays back the second playback data (tactile data). It includes a synchronization information management unit (control unit 34) that receives synchronization information, and a playback processing unit 35 that plays back the first playback data based on the received synchronization information (information indicating the accumulation amount or flag information). The first playback data and the second playback data are, for example, tactile data, acoustic data, or video data. Also, the combination thereof does not have to be a combination of the same type of data. For example, the first playback data may be tactile data and the second playback data may be acoustic data. Of course, both the first playback data and the second playback data may be tactile data, or both may be acoustic data. According to this configuration, it is possible to play back the first playback data in synchronization with the second playback data played back by another playback device. Therefore, for example, it is possible to reduce the deviation between the first playback data and the second playback data and provide it to the user, and it is possible to provide an integrated user experience. Note that this configuration can be applied to the tactile playback device 4. Specifically, a playback device (tactile playback device 4) as an information processing device may include a playback data reception unit (communication processing unit 41) that receives the first playback data (tactile data), and a synchronization information management unit (control unit 44) that receives synchronization information from another playback device (acoustic playback device 3) that plays back the second playback data (acoustic data) in synchronization with the first playback data, and a playback processing unit 45 that plays back the first playback data based on the received synchronization information. The description in this summary can be applied to the tactile playback device 4 by making similar substitutions.

[0210] As described in the configuration of the synchronous playback system, a playback device (acoustic playback device 3) as an information processing device may include a first buffer memory (playback data buffer memory 33) that stores the received first playback data (acoustic data). By accumulating the first playback data in the first buffer memory, it is possible to play back the first playback data smoothly even if the reception of the first playback data is interrupted for a certain period of time. Therefore, it is possible to provide a good playback environment with few data interruptions.

[0211] As described in the configuration of the synchronous reproduction system, in the reproduction device (audio reproduction device 3) as the information processing device, the synchronous information received by the synchronous information management unit (control unit 34) may be information regarding the storage amount of the second buffer memory (reproduction data buffer memory 43) provided in another reproduction device (tactile reproduction device 4) for temporarily storing the second reproduction data (tactile data). Thereby, the reproduction processing unit 35 can change the reproduction mode according to the degree to which the second reproduction data reproduced by another reproduction device is stored in the second buffer memory. Thereby, for example, when the reproduction of data is interrupted in another reproduction device, by similarly interrupting the reproduction of the first reproduction data, the synchronization of the reproduction of the first reproduction data and the second reproduction data can be maintained, and it is possible to prevent giving the user a sense of discomfort or strangeness.

[0212] As described in the configuration of the synchronous reproduction system, the synchronous information received by the synchronous information management unit (control unit 34) in the reproduction device (audio reproduction device 3) as the information processing device may be information on the ratio of the storage amount to the memory size of the second buffer memory (reproduction data buffer memory 43) provided in another reproduction device (tactile reproduction device 4). Thereby, the synchronous information management unit can specifically know the amount of the second reproduction data reproduced by another reproduction device. Therefore, the reproduction of the first reproduction data can be performed based on the amount of the second reproduction data stored.

[0213] As described in the configuration of the synchronous reproduction system, the synchronous information received by the synchronous information management unit (control unit 34) in the reproduction device (audio reproduction device 3) as the information processing device may be flag information indicating whether or not the storage amount of the second buffer memory (reproduction data buffer memory 43) provided in another reproduction device (tactile reproduction device 4) is below a threshold value (third threshold value). Thereby, the synchronous information management unit can know whether or not the second reproduction data reproduced by another reproduction device is small. Since the flag information has a small data volume, the time required for transmitting and receiving synchronization information can be shortened, and the area necessary for storing the received synchronization information can be reduced, thus reducing the burden on the information processing apparatus.

[0214] As described in the configuration of the synchronous reproduction system, in the reproduction device (audio reproduction device 3) as the information processing apparatus, a reproduction data management unit (control unit 34) that duplicates and discards the first reproduction data (audio data) stored in the first buffer memory (reproduction data buffer memory 33) may be provided. By the reproduction data management unit duplicating the first reproduction data in the first buffer memory, the accumulation amount of the first buffer memory can be increased. Also, by the reproduction data management unit discarding the first reproduction data in the first buffer memory, the accumulation amount of the first buffer memory can be decreased. That is, it becomes possible to maintain the first reproduction data accumulated in the first buffer memory in a certain amount by the processing of the reproduction data management unit. Thereby, it is possible to prevent the reproduction of the first reproduction data from being delayed.

[0215] As described in each example, in the reproduction device (audio reproduction device 3) as the information processing apparatus, the reproduction data management unit (control unit 34) may duplicate the first reproduction data (audio data) when the accumulation amount of the first buffer memory (reproduction data buffer memory 33) is less than the first threshold value. Thereby, it is avoided that the accumulation amount of the first reproduction data in the first buffer memory falls below the first threshold value. Therefore, it is prevented that the reproduction of the first reproduction data is delayed.

[0216] As described in each example, in the reproduction device (audio reproduction device 3) as the information processing apparatus, the reproduction data management unit (control unit 34) may discard the first reproduction data (audio data) when the accumulation amount of the first buffer memory (reproduction data buffer memory 33) is equal to or more than the second threshold value. This avoids the situation where the accumulation amount of the first playback data in the first buffer memory exceeds the second threshold value. This avoids the overflow of the first playback data in the first buffer memory, and enables appropriate playback of the first playback data.

[0217] As described in each example, in the playback device (audio playback device 3) as the information processing device, the synchronization information received by the synchronization information management unit (control unit 34) is information regarding the accumulation amount of the second buffer memory (playback data buffer memory 43) provided in another playback device (haptic playback device 4) for temporarily storing the second playback data (haptic data). When the synchronization information management unit receives, as synchronization information, information indicating that the accumulation amount of the second buffer memory is less than the third threshold value, the playback data management unit (control unit 34) may duplicate the first playback data (audio data). This enables the playback of the first playback data in response to the accumulation amount of the second playback data in the second buffer memory falling below the third threshold value. Therefore, playback can be performed in accordance with the playback status of other playback devices.

[0218] As described in each example, in the playback device (audio playback device 3) as the information processing device, the synchronization information received by the synchronization information management unit (control unit 34) is information regarding the accumulation amount of the second buffer memory (playback data buffer memory 43) provided in another playback device (haptic playback device 4) for temporarily storing the second playback data (haptic data). When the synchronization information management unit receives, as synchronization information, information indicating that the accumulation amount of the second buffer memory is equal to or greater than the fourth threshold value, the playback data management unit (control unit 34) may discard the first playback data (audio data). This enables the playback of the first playback data in response to the accumulation amount of the second playback data in the second buffer memory becoming equal to or greater than the fourth threshold value. Therefore, playback can be performed in accordance with the playback status of other playback devices.

[0219] As described in each example, in the playback device (audio playback device 3) as the information processing device, the playback data management unit (control unit 34) receives the replication information and deletion information of the second playback data (haptic data) received from another playback device (haptic playback device 4), and when receiving the replication information, replicates the first playback data to be synchronized with the second playback data targeted for replication and plays it back, and when receiving the deletion information, deletes the first playback data to be synchronized with the second playback data targeted for deletion. Thereby, the replication and deletion of the first playback data are performed in synchronization with the second playback data. Therefore, even when one of the first playback data and the second playback data becomes insufficient or is about to overflow in the buffer memory, it is possible to maintain the synchronization state of the first playback data and the second playback data and play them back, and it is possible to prevent giving discomfort or a sense of incongruity to the user.

[0220] As described in each example, in the playback device (audio playback device 3) as the information processing device, the playback data reception unit (communication processing unit 31) receives control information about the first playback data (audio data), and the playback data management unit (control unit 34) may replicate and delete the first playback data based on the control information. Thereby, it is possible to manage the accumulation amount of the first playback data in the first buffer memory based on the control information, and it is possible to contribute to appropriate playback processing.

[0221] As described in each example, in the playback device (audio playback device 3) as the information processing device, the control information may be information indicating permission / non - permission for at least one of replication or deletion of the first playback data (audio data). The first playback data is data having a playback time length of a predetermined time (for example, several μsec to several hundred msec). With this configuration, it is determined whether replication or deletion is appropriate for each piece of the first playback data. That is, only the first playback data determined to be okay to duplicate or discard becomes the target of duplication or discard. Therefore, for example, when duplicating the first playback data due to a decrease in the first playback data stored in the first buffer memory (playback data buffer memory 33), duplication is performed while avoiding the first playback data that would cause discomfort or a sense of incongruity during playback. Therefore, the first playback data can be played back without giving discomfort or a sense of incongruity to the user.

[0222] As described in each example, in the playback device (audio playback device 3) as the information processing device, the synchronization information management unit (control unit 34) may transmit information regarding the accumulation amount of the first playback data (audio data) in the first buffer memory (playback data buffer memory 33) to another playback device (tactile playback device 4) as synchronization information. Thereby, in another playback device, playback processing based on the accumulation amount of the first playback data can be performed. In particular, duplication or discard of the second playback data can be performed in another playback device in accordance with the duplication or discard status of the first playback data. Therefore, even when the amount of the first playback data stored in the first buffer memory fluctuates, the state in which the first playback data (audio data) and the second playback data (tactile data) are played back synchronously can be continued.

[0223] As described in the processing example 4 of the playback device, in the playback device (audio playback device 3) as the information processing device, the playback data management unit (control unit 34) may reuse the first playback data instead of duplicating the first playback data. Thereby, the duplication process of the first playback data is not performed. Therefore, the processing burden associated with duplicating the first playback data can be eliminated.

[0224] As described in each of the above examples, the synchronous playback system 1 as an information processing system may include a transmission device 2 that transmits first playback data (audio data) and second playback data (haptic data), a first playback device (audio playback device 3) that plays back the first playback data, and a second playback device (haptic playback device 4) that plays back the second playback data (haptic data). The first playback device may include a first playback data reception unit (communication processing unit 31) that receives the first playback data, a first synchronization information management unit (control unit 34) that receives synchronization information from the second playback device, and a first playback processing unit (playback processing unit 35) that plays back the first playback data based on the received synchronization information. The second playback device may include a second playback data reception unit (communication processing unit 41) that receives the second playback data, a second synchronization information management unit (control unit 44) that receives synchronization information from the first playback device, and a second playback processing unit (playback processing unit 45) that plays back the second playback data based on the received synchronization information. With such a configuration, the respective storage amounts of the first playback data and the second playback data are appropriately managed. Also, synchronization information about the storage amount is shared among the playback devices. And in each playback device, replication and deletion of the playback data are performed based on the synchronization information, and a synchronized three-dimensional playback environment can be provided to the user while preventing sound skipping and the like. As a result, the user can obtain a good user experience.

[0225] In the synchronous playback system 1 as an information processing system, the transmission device 2 may include a control information generation unit 22 that generates first control information (control information about audio data) which is control information regarding whether to permit replication and deletion of the first playback data (audio data), and second control information (control information about haptic data) which is control information regarding whether to permit replication and deletion of the second playback data (haptic data). Thereby, permission / non - permission of replication and deletion of the first playback data and the second playback data is set individually. Therefore, when duplicating or deleting the first reproduction data, it is only necessary to check the first control information regarding the first reproduction data, thereby improving the processing efficiency.

[0226] As described in the processing example of the transmitting device, in the synchronous reproduction system 1 as an information processing system, when it is determined that at least one of the first reproduction data (acoustic data) and the second reproduction data (tactile data) that are synchronously reproduced cannot be duplicated or deleted, the control information generation unit 22 generates control information indicating that duplication and deletion are not permitted as the first control information for the first reproduction data, and may also generate control information indicating that duplication and deletion are not permitted as the second control information for the second reproduction data that is reproduced synchronously with the first reproduction data. Thereby, when information that enables duplication or deletion is stored in the first control information of the first reproduction data, information that enables duplication or deletion will also naturally be stored in the second control information of the second reproduction data that is reproduced synchronously. Therefore, since it is avoided that only one of the first reproduction data and the second reproduction data that are synchronously reproduced is duplicated or deleted, even when the amount of the first reproduction data stored in the first buffer memory fluctuates or the amount of the second reproduction data stored in the second buffer memory fluctuates, the state in which the first reproduction data and the second reproduction data are reproduced synchronously can be maintained.

[0227] The information processing method executed by the reproduction device (acoustic reproduction device 3) as an information processing apparatus includes a process of receiving the first reproduction data (acoustic data), a process of receiving synchronization information from another reproduction device (tactile reproduction device 4) that reproduces the second reproduction data (tactile data) in synchronization with the first reproduction data, and a process of reproducing the first reproduction data based on the received synchronization information. With such a configuration, the above-described acoustic reproduction device 3 and tactile reproduction device 4 can be configured, and the synchronous reproduction system 1 including the acoustic reproduction device 3 and the tactile reproduction device 4 can be realized. Therefore, various operational effects described above can be obtained.

[0228] Note that the effects described in this specification are merely illustrative and not limiting, and there may be other effects. In addition, each of the above examples can be combined in any way as long as the combination is possible.

[0229] <7. The present technology> Note that the audio playback device 3 and the tactile playback device 4 as the information processing devices of the present technology can also adopt the following configurations. (1) A playback data receiving unit that receives first playback data and control information about the first playback data; A first buffer memory that stores the received first playback data and the control information; A synchronization information management unit that receives synchronization information from another playback device that plays second playback data in synchronization with the first playback data; A playback data management unit that duplicates or deletes the first playback data stored in the first buffer memory; A playback processing unit that plays the first playback data based on the received synchronization information, and an Information processing device. (2) The synchronization information received by the synchronization information management unit is information regarding the accumulation amount of a second buffer memory provided in the other playback device for temporarily storing the second playback data. The information processing device according to (1) above. (3) The synchronization information received by the synchronization information management unit is information regarding the ratio of the accumulation amount to the memory size of the second buffer memory. The information processing device according to (2) above. (4) The synchronization information received by the synchronization information management unit is flag information indicating whether the accumulation amount of the second buffer memory is equal to or greater than a threshold value, or whether it has fallen below the threshold value. The information processing device according to (2) above. (5) When the accumulated amount in the first buffer memory is less than the first threshold value, the playback data management unit replicates any of the first playback data based on the control information among the first playback data. The information processing apparatus according to any one of (1) to (4) above. (6) When the accumulated amount in the first buffer memory is greater than or equal to the second threshold value, the playback data management unit deletes any of the first playback data based on the control information among the first playback data. The information processing apparatus according to any one of (1) to (5) above. (7) When the synchronization information management unit receives, as the synchronization information, information indicating that the accumulated amount in the second buffer memory is less than the third threshold value, the playback data management unit replicates any of the first playback data based on the control information among the first playback data. The information processing apparatus according to any one of (2) to (4) above. (8) When the synchronization information management unit receives, as the synchronization information, information indicating that the accumulated amount in the second buffer memory is greater than or equal to the fourth threshold value, the playback data management unit deletes any of the first playback data based on the control information among the first playback data. The information processing apparatus according to any one of (2) to (4) above. (9) The playback data management unit receives the replication information and deletion information of the second playback data received from the other playback device, when receiving the replication information, replicates the first playback data to be synchronized with the second playback data targeted for replication, and when receiving the deletion information, deletes the first playback data to be synchronized with the second playback data targeted for deletion. The information processing apparatus according to any one of (1) to (8) above. (10) The control information is information indicating permission / non - permission for at least one of replication or deletion of the first playback data. The information processing apparatus according to any one of (1) to (9) above. (11) The control information is information based on a perception feature amount representing the degree of user perception of duplication or deletion of the first reproduction data. The information processing apparatus according to any one of (1) to (10) above. (12) The synchronization information management unit transmits information regarding the accumulation amount of the first reproduction data in the first buffer memory as synchronization information to the other reproduction device. The information processing apparatus according to any one of (1) to (11) above. (13) The reproduction data management unit performs reuse of the first reproduction data instead of duplication of the first reproduction data. The information processing apparatus according to any one of (1) to (13) above. (14) A transmission device that transmits first reproduction data and second reproduction data, A first reproduction device that reproduces the first reproduction data, A second reproduction device that reproduces the second reproduction data, and The first reproduction device includes A first reproduction data reception unit that receives the first reproduction data and control information about the first reproduction data, A first buffer memory that stores the received first reproduction data and control information about the first reproduction data, A first synchronization information management unit that receives synchronization information from the second reproduction device, A first reproduction data management unit that duplicates or deletes the first reproduction data stored in the first buffer memory, A first reproduction processing unit that reproduces the first reproduction data based on the synchronization information received by the first synchronization information management unit. The second reproduction device includes A second reproduction data reception unit that receives the second reproduction data and control information about the second reproduction data, A second buffer memory that stores the received second reproduction data and control information about the second reproduction data; A second synchronization information management unit that receives synchronization information from the first playback device; A second reproduction data management unit that duplicates or deletes the second reproduction data stored in the second buffer memory; A second reproduction processing unit that reproduces the second reproduction data based on the synchronization information received by the second synchronization information management unit, and an Information processing system. (15) The transmission device is provided with a control information generation unit that generates first control information that is control information regarding whether to permit duplication and deletion of the first reproduction data and second control information that is control information regarding whether to permit duplication and deletion of the second reproduction data; The information processing system according to (14) above. (16) The control information generation unit When it is determined that duplication or deletion of at least one of the first reproduction data and the second reproduction data reproduced synchronously is not permitted, generates control information indicating that duplication and deletion are not permitted as the first control information of the first reproduction data, and generates control information indicating that duplication and deletion are not permitted as the second control information of the second reproduction data reproduced synchronously with the first reproduction data. The information processing system according to (15) above. (17) A process of receiving the first reproduction data and control information about the first reproduction data, a process of storing the received first reproduction data and the control information, a process of receiving synchronization information from another playback device that reproduces the second reproduction data in synchronization with the first reproduction data, a process of duplicating or deleting the stored first reproduction data, and a process of reproducing the first reproduction data based on the received synchronization information An information processing method executed by a computer device.

Explanation of symbols

[0230] 1 Synchronous playback system (information processing system) 2 Transmitting device 3 Acoustic playback device (information processing device) 3 Acoustic playback device (first playback device) 4 Tactile playback device (second playback device) 22 Control information generation unit 31 Communication processing unit (playback data receiving unit) 31 Communication processing unit (first playback data receiving unit) 33 Buffer memory for playback data (first buffer memory) 34 Control unit (synchronization information management unit) 34 Control unit (first synchronization information management unit) 34 Control unit (playback data management unit) 34 Control unit (first playback data management unit) 35 Playback processing unit (first playback processing unit) 41 Communication processing unit (second playback data receiving unit) 43 Buffer memory for playback data (second buffer memory) 44 Control unit (second synchronization information management unit) 44 Control unit (second playback data management unit) 45 Playback processing unit (second playback processing unit)

Claims

1. A reproduction data receiving unit that receives first reproduction data and control information about the first reproduction data; A first buffer memory that stores the received first reproduction data and the control information; A synchronization information management unit that receives synchronization information from another playback device that plays back second reproduction data in synchronization with the first reproduction data; A reproduction data management unit that duplicates or deletes the first reproduction data stored in the first buffer memory; A reproduction processing unit that reproduces the first reproduction data based on the received synchronization information, comprising: The synchronization information includes information for specifying the second reproduction data that has been duplicated or deleted in the other playback device; The reproduction data management unit: When receiving the synchronization information including information for specifying the duplicated second reproduction data, duplicates the first reproduction data to be reproduced in synchronization with the specified second reproduction data; When receiving the synchronization information including information for specifying the deleted second reproduction data, deletes the first reproduction data to be reproduced in synchronization with the specified second reproduction data An information processing apparatus.

2. The synchronization information received by the synchronization information management unit is information regarding the storage amount of a second buffer memory provided in the other playback device for temporarily storing the second reproduction data The information processing apparatus according to claim 1.

3. The synchronization information received by the synchronization information management unit is information on the ratio of the storage amount to the memory size of the second buffer memory The information processing apparatus according to claim 2.

4. The synchronization information received by the synchronization information management unit is flag information indicating whether the storage amount of the second buffer memory is equal to or greater than a threshold value, or whether it has fallen below the threshold value The information processing apparatus according to claim 2.

5. When the storage amount of the first buffer memory is less than a first threshold value, the reproduction data management unit duplicates any of the first reproduction data based on the control information The information processing apparatus according to claim 1.

6. When the storage amount of the first buffer memory is equal to or greater than a second threshold value, the reproduction data management unit deletes any of the first reproduction data based on the control information The information processing apparatus according to claim 1.

7. When the synchronization information management unit receives, as the synchronization information, information indicating that the accumulation amount of the second buffer memory is less than a third threshold value, the reproduction data management unit replicates any of the first reproduction data based on the control information among the first reproduction data The information processing apparatus according to claim 2.

8. When the synchronization information management unit receives, as the synchronization information, information indicating that the accumulation amount of the second buffer memory is equal to or more than a fourth threshold value, the reproduction data management unit deletes any of the first reproduction data based on the control information among the first reproduction data The information processing apparatus according to claim 2.

9. The control information is information indicating permission / non-permission regarding at least one of replication or deletion of the first reproduction data The information processing apparatus according to claim 1.

10. The control information is information based on a perception feature amount representing the degree of user perception regarding replication or deletion of the first reproduction data The information processing apparatus according to claim 1.

11. The synchronization information management unit transmits, as the synchronization information, information regarding the accumulation amount of the first reproduction data in the first buffer memory to the other reproduction device The information processing apparatus according to claim 1.

12. The reproduction data management unit re-uses the first reproduction data instead of replicating the first reproduction data The information processing apparatus according to claim 1.

13. A transmission device that transmits first reproduction data and second reproduction data, A first reproduction device that reproduces the first reproduction data, A second reproduction device that reproduces the second reproduction data, and includes: The first reproduction device is A first reproduction data reception unit that receives the first reproduction data and control information regarding the first reproduction data, A first buffer memory that stores the received first reproduction data and control information regarding the first reproduction data, A first synchronization information management unit that receives synchronization information from the second reproduction device, A first reproduction data management unit that replicates or deletes the first reproduction data stored in the first buffer memory, A first reproduction processing unit that reproduces the first reproduction data based on the synchronization information received by the first synchronization information management unit, and includes: The second reproduction device is A second reproduction data reception unit that receives the second reproduction data and control information regarding the second reproduction data, A second buffer memory that stores the received second reproduction data and control information regarding the second reproduction data, A second synchronization information management unit that receives synchronization information from the first playback device; A second playback data management unit that duplicates or deletes the second playback data stored in the second buffer memory; A second playback processing unit that plays back the second playback data based on the synchronization information received by the second synchronization information management unit, and includes: The synchronization information received by the first synchronization information management unit includes information for specifying the second playback data duplicated or deleted in the second playback device; The first playback data management unit: When receiving the synchronization information including information for specifying the duplicated second playback data, duplicates the first playback data to be played back in synchronization with the specified second playback data; When receiving the synchronization information including information for specifying the deleted second playback data, deletes the first playback data to be played back in synchronization with the specified second playback data. An information processing system.

14. The transmitting device: A control information generation unit that generates first control information that is control information regarding whether to permit duplication and deletion of the first playback data, and second control information that is control information regarding whether to permit duplication and deletion of the second playback data. The information processing system according to claim 13, comprising: The information processing system according to claim 13.

15. The control information generation unit: When determining that duplication or deletion of at least one of the first playback data and the second playback data to be played back synchronously is not permitted, Generates control information indicating that duplication and deletion are not permitted as the first control information for the first playback data; Generates control information indicating that duplication and deletion are not permitted as the second control information for the second playback data that is played back synchronously with the first playback data. The information processing system according to claim 14, comprising: The information processing system according to claim 14.

16. A process of receiving the first playback data and control information regarding the first playback data; A process of storing the received first playback data and the control information; A process of receiving synchronization information from another playback device that plays back the second playback data in synchronization with the first playback data; A process of duplicating or deleting the stored first playback data; A process of playing back the first playback data based on the received synchronization information; A process of duplicating the first playback data to be played back in synchronization with the specified second playback data when receiving the synchronization information including information for specifying the duplicated second playback data; When receiving the synchronization information including information specifying the discarded second reproduction data, performing a process of discarding the first reproduction data to be reproduced in synchronization with the specified second reproduction data An information processing method executed by a computer device.

Images