Content data processing method and content data processing device

Abstract

This content data processing method includes: generating a first timecode indicating the amount of time that has elapsed since the time at which a live event began; receiving data related to the live event, and generating first content data in which the time of the first timecode is attached to the live-event-related data; and generating a second timecode indicating the amount of time that has elapsed since the time at which said data was received.

Description

[Technical field]

[0001] An embodiment of the present invention relates to a content data processing method and a content data processing device for processing content data. [Background technology]

[0002] Patent Document 1 discloses a terminal device that synchronizes the progress of a video image reproduced on a television receiver with the progress of a content.

[0003] The synchronization of multiple data is performed using a time code, as shown in, for example, Patent Document 2. [Prior art documents] [Patent documents]

[0004] [Patent Document 1] Patent Publication No. 2016-111492 [Patent Document 2] JP 2004-21844 A Summary of the Invention [Problem to be solved by the invention]

[0005] In a live event, various data such as the voice of the presenter, singing sound, musical performance sound, video, or performance are generated at various times. Therefore, management for editing or playing these data related to the live event becomes complicated. For example, if a performance occurs at a certain timing in the second performance after the start of the live event, it is difficult to determine how much time has passed since the start of the second performance only from the time code indicating the elapsed time from the start of the live event.

[0006] An object of one embodiment of the present invention is to provide a content data processing method and a content data processing device that facilitates management of multiple data related to a live event. [Means for solving the problem]

[0007] A content data processing method according to one embodiment of the present invention generates a first time code indicating the elapsed time from the start of a live event, accepts data related to the live event, generates first content data by adding the time of the first time code to the data, and generates a second time code indicating the elapsed time from the time the data was accepted. Effect of the Invention

[0008] According to one embodiment of the present invention, multiple data relating to a live event can be easily managed. [Brief description of the drawings]

[0009] [Figure 1] 1 is a block diagram showing a configuration of a content data processing system 1. FIG. [Diagram 2] 1 is a block diagram showing the main configuration of a content data processing device 10. FIG. [Diagram 3] 4 is a flowchart showing the operation of a CPU 104. [Figure 4] FIG. 2 is a diagram showing the relationship between a time code, sound data, and time. [Diagram 5] FIG. 2 is a diagram showing the relationship between a time code, sound data, and time. [Figure 6] FIG. 11 is a diagram showing an example of a structure of third sound data. [Figure 7] 13 is a diagram showing the relationship between the time code, sound data, and time when the third sound data is reproduced. FIG. [Figure 8] 13 is a diagram showing the relationship between time codes, sound data, and time when reaction data is collected. FIG. [Figure 9] FIG. 13 is a diagram showing an example of a structure of sound data according to an application example. [Figure 10] FIG. 13 is a conceptual diagram showing related information according to an application example. [Figure 11]FIG. 13 is a diagram showing an example of a structure of related information according to an application example. [Figure 12] FIG. 13 is a diagram illustrating an example of a structure of related information. [Figure 13] FIG. 13 is a conceptual diagram showing a hierarchical structure of related information according to an application example. [Figure 14] FIG. 13 is a diagram showing an example of a structure of sound data according to an application example. [Figure 15] 1 is a block diagram showing a configuration of a content data processing system 1. FIG. [Figure 16] FIG. 13 is a diagram showing the relationship between time code, sound data, and time according to Modification 1. [Figure 17] FIG. 11 is a diagram showing an example of sound data acquired by the content data processing device 10 in Modification 2. [Figure 18] FIG. 11 is a diagram showing an example of a combination of sound data. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0010] 1 is a block diagram showing a configuration of a content data processing system 1. The content data processing system 1 includes a content data processing device 10, a content data processing device 10A, a content data processing device 10B, and a content data processing device 10C. The content data processing device 10, the content data processing device 10A, the content data processing device 10B, and the content data processing device 10C are connected to each other via the Internet 5.

[0011] Each of the content data processing device 10, the content data processing device 10A, the content data processing device 10B, and the content data processing device 10C is an information processing device such as a personal computer.

[0012] The content data processing device 10 is installed, for example, at a venue of a live event. The content data processing device 10A, the content data processing device 10B, and the content data processing device 10C are installed at a venue different from the venue of the live event, or at the homes of participants who watch the live event in remote locations.

[0013] 2 is a block diagram showing the main configuration of the content data processing device 10. The content data processing device 10 includes a display 101, a user interface (I / F) 102, a flash memory 103, a CPU 104, a RAM 105, and a communication interface (I / F) 106. The content data processing device 10A, the content data processing device 10B, and the content data processing device 10C have the same or similar hardware configuration as the content data processing device 10. Therefore, a description of the hardware configuration of the content data processing device 10A, the content data processing device 10B, and the content data processing device 10C will be omitted.

[0014] The display 101 is, for example, a liquid crystal display (LCD) or an organic light-emitting diode (OLED), and displays various information. The user I / F 102 is, for example, a switch, a keyboard, a mouse, a trackball, a touch panel, or the like, and accepts user operations. When the user I / F 102 is a touch panel, the user I / F 102 and the display 101 form a GUI (Graphical User Interface, hereinafter abbreviated).

[0015] The communication I / F 106 is connected to various devices via a communication line such as a USB cable, HDMI (registered trademark), Ethernet (registered trademark), or MIDI. For example, the content data processing device 10 is connected to an audio mixer that processes the sound of a live event via the communication I / F 106. The content data processing device 10 receives sound data related to the live event from the audio mixer. The sound data related to the live event includes sound signals such as the voice of a presenter, singing sounds, performance sounds, or performance sounds. The performance sounds also include sound effects such as applause, cheers, or explosion sounds.

[0016] The sound data may be sound data including all sound signals generated in a live event, such as the sound signals of singing sounds, the sound signals of performance sounds, and the sound signals of performance sounds, or may be sound data including the sound signals of singing sounds, the sound signals of performance sounds, and the sound signals of performance sounds separately. The sound data of singing sounds may be sound data including the singing sounds of all singers, or may be sound data including the singing sounds of each singer individually. The sound data of performance sounds may be sound data including the performance sounds of all musical instruments, or may be sound data including the performance sounds of each musical instrument individually.

[0017] CPU 104 reads out a program stored in flash memory 103, which is a storage medium, into RAM 105 to realize a predetermined function. For example, CPU 104 displays an image for accepting user operations on display 101, and accepts selection operations and the like for the image via user I / F 102, thereby realizing a GUI.

[0018] It should be noted that the program read by CPU 104 does not need to be stored in flash memory 103 in the device itself. For example, the program may be stored in a storage medium of an external device such as a server. In this case, CPU 104 may read the program from the server into RAM 105 each time and execute it.

[0019] 3 is a flowchart showing the operation of CPU 104. CPU 104 first determines whether or not a live event has started (S11). For example, when an operator who manages the live event instructs the start of the live event via user I / F 102, CPU 104 determines that the live event has started (S11: Yes).

[0020] When the CPU 104 determines that the live event has started, it generates a first time code (S12). FIG. 4 is a diagram showing the relationship between time codes, sound data, and time. The first time code indicates the elapsed time from the start timing of the live event. In other words, the first time code is a relative time based on the start timing of the live event. In the example of FIG. 4, the live event starts at absolute time 13:00:00. The first time code is the reference time (00:00:00) at absolute time 13:00:00. In this example, the CPU 104 generates the first time code every second for ease of explanation, but the CPU 104 may generate the first time code at shorter time intervals. For example, when the sampling frequency of the sound signal is 48 kHz, the first time code may also be generated in units of 48 kHz (20.83 μsec).

[0021] Next, CPU 104 judges whether or not a performance in the live event has started (S13). For example, when an operator who manages the live event instructs the start of a performance via user I / F 102, CPU 104 judges that the performance has started (S13: Yes). CPU 104 repeats generating the first time code until a performance in the live event starts (S13: No).

[0022] When the CPU 104 determines that the performance has started, it generates a second time code (S14). As shown in Fig. 4, the second time code indicates the elapsed time from the start timing of each performance in the live event. In other words, the second time code is a relative time based on the start timing of each performance in the live event. In the example of Fig. 4, the performance of the emcee's speech starts at absolute time 13:03:30. At this time, the first time code is 00:03:30, and the second time code is 00:00:00. The second time code is also generated in units of one second, but may be generated in units of 48 kHz (20.83 μsec), for example.

[0023] CPU 104 receives sound data at the same time as generating the second time code (S15). That is, CPU 104 starts recording sound data (sound signals such as voice, singing sound, musical performance sound, or performance sound) related to the live event received from the audio mixer. CPU 104 then generates first content data in which the time of the first time code is added to the recorded sound data (S16). The first content data is data in which a header of time information is added to sound data (sound signals such as voice, singing sound, musical performance sound, or performance sound) related to the live event.

[0024] Next, CPU 104 determines whether the performance has ended (S17). For example, when an operator managing a live event instructs the end of the performance via user I / F 102, CPU 104 determines that the performance has ended (S17: Yes). When CPU 104 determines that the performance is continuing (S17: No), it repeats the process from S12. When CPU 104 determines that the performance has ended, it ends the generation of the second time code (S18) and stops recording of the sound data (S19).

[0025] 4, the performance of the MC's speech ends at absolute time 13:06:30. At this time, the first time code becomes 00:06:30, and the second time code becomes 00:03:00. This completes the generation of the first content data related to the MC's speech.

[0026] Thereafter, CPU 104 determines whether the live event has ended (S20). For example, when an operator managing the live event instructs the end of the live event via user I / F 102, CPU 104 determines that the live event has ended (S20: Yes). When CPU 104 determines that the live event is continuing (S20: No), it repeats the process from S12. When CPU 104 determines that the live event has ended (S20: Yes), it ends generation of the first time code (S21) and ends the operation.

[0027] In the example of FIG. 4, the performance of the first song starts at absolute time 13:11:00 as the performance following the speech of the presenter. When CPU 104 repeats the process from S12, it generates first content data related to the performance sound of the first song and generates the second time code. At this time, the first time code becomes 00:11:00 and the second time code becomes 00:00:00. CPU 104 starts generating the second time code again and accepts sound data (S15). Then, CPU 104 generates first content data by adding the time of the first time code to the recorded sound data (S16). The first content data generated here is data in which a header of time information is added to the sound data of the performance sound of the first song. The performance of the first song ends at absolute time 13:16:00. At this time, the first time code becomes 00:16:00, and the second time code becomes 00:05:00. This completes the generation of the first content data related to the performance sound of the first piece of music, to which the time information of the first time code with the start time of 00:11:00 and the end time of 00:16:00 has been added.

[0028] In this way, the content data processing device 10 generates a first time code indicating the elapsed time from the start timing of the live event, generates first content data to which the time of the first time code is added, and further generates a second time code indicating the elapsed time from the start timing of the performance corresponding to each first content data. Although only two performances are shown in the example of FIG. 4, various data such as the voice of the presenter, singing sound, musical performance sound, video, or performance occurs at various times in an actual live event. Therefore, if there is no second time code, management for editing or playing these data related to the live event becomes complicated. For example, if a performance occurs at a certain timing in the performance of the first song in FIG. 4, it is difficult to determine how much time has passed since the start timing of the performance of the first song when the performance occurred, based only on the first time code from the start of the live event. However, the content data processing device 10 of this embodiment generates a second time code indicating the elapsed time from the start timing of each performance, so that it is easy to determine how much time has passed since the start timing of the performance of the first song when the performance occurred.

[0029] In the example of Fig. 4, the sound data of the performance of the first song is sound data including a sound signal of a singing sound, a sound signal of a performance sound, and a sound signal of a performance sound. However, the sound data may separately include, for example, a first sound data including a sound signal of a singing sound and a sound signal of a performance sound, and a second sound data including a sound signal of a performance sound. Fig. 5 is a diagram showing the relationship between the time code, the sound data, and the time when the performance sound is accepted as the second sound data.

[0030] In the example of FIG. 5, the sound effect is generated one minute after the start of the performance of the first song. For example, an operator who manages a live event instructs the start of the sound effect via the user I / F 102. The CPU 104 of the content data processing device 10 receives the second sound data of the sound effect and generates second content data by adding the time of the second time code to the second sound data of the sound effect. This generates second content data related to the sound effect, to which time information of the second time code of the start time 00:01:00 and the end time 00:04:30 is added. Note that, if another sound effect (second sound effect) related to the sound effect (first sound effect) is generated while the sound effect is being generated, the content data processing device 10 may further receive the second sound data of the second sound effect and generate second content data by adding the time of the second time code to the sound data of the second sound effect. Alternatively, the content data processing device 10 may further generate a third time code indicating the elapsed time from the start timing of the first production sound, and generate third content data by adding the time of the third time code to the sound data of the second production sound.

[0031] The content data processing device 10 distributes the first time code, the first content data (the voice data of the presenter and the audio data of the first song), and the second time code shown in Fig. 4. Alternatively, the content data processing device 10 distributes the first time code, the first content data (the voice data of the presenter and the audio data of the first song), the second time code, and the second content data shown in Fig. 5.

[0032] In the example configuration of FIG. 1, the content data processing device 10 distributes various data via the Internet 5 to a content data processing device 10A, a content data processing device 10B, and a content data processing device 10C.

[0033] The content data processing device 10A, the content data processing device 10B, and the content data processing device 10C receive the first time code, the first content data, and the second time code, respectively. Alternatively, the content data processing device 10A, the content data processing device 10B, and the content data processing device 10C receive the first time code, the first content data, the second time code, and the second content data, respectively.

[0034] The content data processing device 10A, the content data processing device 10B, and the content data processing device 10C reproduce the voice data of the presenter and the sound data of the first piece of music, which are included in the first content data, based on the received first time code.

[0035] The content data processing device 10A, the content data processing device 10B, and the content data processing device 10C have different communication environments. Therefore, the content data processing device 10A, the content data processing device 10B, and the content data processing device 10C receive data at different times. Therefore, the absolute times of the playback start timings of the live events are different. However, the content data processing device 10A, the content data processing device 10B, and the content data processing device 10C each play back each of the sound data related to the live event based on a first time code, which is the time elapsed from the start timing of the live event. Therefore, the sound data of each performance is played back at the same time elapsed from the start of the live event in all the devices. In addition, when the content data processing device 10A, the content data processing device 10B, and the content data processing device 10C receive the second content data, they play back the second sound data based on a second time code, which is the time elapsed from the start timing of each performance. Therefore, the sound data of the performance sound is played back at the same time elapsed from the start of the performance in all the devices.

[0036] The content data processing device 10A, the content data processing device 10B, and the content data processing device 10C may play back the third sound data stored therein together with or instead of the second sound data. The third sound data may be sound data such as a performance sound related to each performance, or may be sound data including a sound signal of a performance sound. The content data processing device 10 distributes the third sound data to the content data processing device 10A, the content data processing device 10B, and the content data processing device 10C in advance before the start of the live event.

[0037] Fig. 6 is a diagram showing an example of the structure of the third sound data. The third sound data is sound data to which time information of the second time code is added. In the example of Fig. 6, the third sound data includes time data and performance sound data. The time data includes the time information of the second time code and related information indicating a related destination. The related information is information for identifying the related destination of each third sound data, and in the example of Fig. 6, information of the first song is included as the related information.

[0038] Fig. 7 is a diagram showing the relationship between the time code, the sound data, and the time when the third sound data is reproduced. In the example of Fig. 7, the content data processing device 10 distributes the first content data related to the voice of the presenter, the first time code, the first content data related to the vocal voice of the first song, the second time code related to the first content data related to the voice of the presenter, and the second time code related to the first content data related to the vocal voice of the first song.

[0039] The content data processing device 10A, the content data processing device 10B, and the content data processing device 10C download the third sound data in advance. The third sound data is, for example, sound data of the performance sound excluding the vocal voice of the first song. The third sound data includes information of the first content data related to the vocal voice of the first song as related information. Therefore, the content data processing device 10A, the content data processing device 10B, and the content data processing device 10C play the vocal voice of the first song included in the first content data based on the first time code, and play the third sound data of the performance sound downloaded in advance based on the second time code of the first song.

[0040] 6, as an example, playback of the third sound data of the performance sound starts at 00:00:30 of the first song, and ends at 00:04:30 of the first song. This allows the content data processing device 10A, the content data processing device 10B, and the content data processing device 10C to each play the performance sound in time with the vocal voice of the first song. Therefore, in all the devices, the sound data of each performance is played at the same time elapsed from the start of the live event, and the sound data downloaded in advance is played at the same time elapsed from the start of the performance.

[0041] In the examples of Figures 6 and 7, the content data processing device 10A, the content data processing device 10B, and the content data processing device 10C do not play back the second sound data (the performance sound data shown in Figure 5). However, the content data processing device 10 may further distribute second content data including the second sound data. The content data processing device 10A, the content data processing device 10B, and the content data processing device 10C may further play back the second sound data based on a second time code included in the second sound content data.

[0042] The content data processing device 10A, the content data processing device 10B, and the content data processing device 10C may store the sound data of the distributed live event. The users of the content data processing device 10A, the content data processing device 10B, and the content data processing device 10C can play back the live event that was held in the past at any time they like. The users of the content data processing device 10A, the content data processing device 10B, and the content data processing device 10C can also edit the sound data of the live event. For example, if the start time of the sound data of the first song shown in FIG. 5 is changed to 00:15:00, the performance of the first song starts at 00:15:00 from the start of the live event. Even if the start time of the sound data of the first song is changed, the sound data of the performance sound related to the first song starts at 00:01:00 from the start of the playback of the first song. The user may also cut out and play only an arbitrary part of an arbitrary song. For example, the user can cut out only a part of the performance of the sound data of the first song by specifying 00:01:00 to 00:03:00 in the second time code of the sound data of the first song in the example of FIG. 5. The user can also change the order of the songs by swapping the sound data of the first song with the sound data of the second song. In this case, the sound data of the performance sound related to each of the first and second songs is played at a timing when a predetermined time has passed from the playback start timing of each song. The user can also extract any songs from multiple live events and combine the sound data of the songs he / she likes. Therefore, the content data processing system 1 of this embodiment can easily manage and edit each data of the live event. The content data processing system 1 of this embodiment can also create a live event unique to the user by extracting any songs from multiple live events or cutting out only any part.

[0043] The content data processing device 10A, the content data processing device 10B, and the content data processing device 10C may receive reactions from their respective users (participants in the live event). The content data processing device 10A, the content data processing device 10B, and the content data processing device 10C generate reaction data by adding a first time code or a second time code to information corresponding to the received reaction.

[0044] The content data processing device 10A, the content data processing device 10B, and the content data processing device 10C display icon images such as "cheers," "applause," "calls," and "bustling noise" on the display device 101, and accept selection operations on these icon images from the user via the user I / F 102 to accept reactions.

[0045] The content data processing device 10A, the content data processing device 10B, and the content data processing device 10C generate information corresponding to the received reactions, and generate reaction data by adding a first time code or a second time code to the generated reaction information. The content data processing device 10A, the content data processing device 10B, and the content data processing device 10C transmit the reaction data to the content data processing device 10 or another aggregation server, etc. The content data processing device 10 or the aggregation server receives and aggregates reaction data from multiple participants. The event organizer can use the device that aggregates reaction data to analyze at what time in a live event, or at what time in which performance, excitement occurs.

[0046] The reaction data may be text information, but may also be, for example, location information, audio information, biometric information, or motion information. When the reaction data includes text information, the device that collects the reaction data can extract, for example, a specific word (keyword) from the text information. Furthermore, when the device that collects the reaction data extracts the same word from multiple devices, it can extract the word as a trend word. This allows the event organizer to refer to the text content and analyze what emotions participants felt at what time during a live event, or at what time during which performance.

[0047] Location information includes GPS data of the device that generated the reaction, the IP address of the access point for wireless communication, etc. Live event managers can analyze what time of the live event, or what time of performance, and in what area, is the excitement occurring.

[0048] The audio information is sound data recorded from the voices of the participants, such as "cheering," "applause," "calling out," and "bustling noise." The device that collects the reaction data can analyze changes in the volume, quality, and tone of the participants' voices.

[0049] Biometric information is information such as sweat rate, heart rate, blood pressure, etc. acquired by sensors worn by participants. A device that collects reaction data can analyze at what time during a live event, or at what time during a performance, excitement occurred.

[0050] The motion information is, for example, viewpoint information detected by a head tracker or a camera of a mobile device owned by a user, or participant movement information detected by an acceleration sensor of the mobile device. Participant movement information may be detected by an acceleration sensor built into a penlight used at a live event. The device that collects reaction data can analyze what movement each participant made at what time during a live event, or at what time during which performance.

[0051] 8 is a diagram showing the relationship between the time code, sound data, and time when the reaction data is collected. In the example of FIG. 8, the content data processing device 10 receives four reactions A, B, C, and D.

[0052] Reaction A occurs at 00:01:20 after the performance of the presenter's speech starts. Therefore, the content data processing device 10 accepts the reaction data of reaction A including the time information of 00:01:20 as the time information of the second time code of the presenter's speech. Reaction B occurs at 00:09:30 after the live event starts. Therefore, the content data processing device 10 accepts the reaction data of reaction B including the time information of 00:09:30 as the time information of the first time code. Reaction C occurs at 00:02:00 after the performance of the first song starts. Therefore, the content data processing device 10 accepts the reaction data of reaction C including the time information of 00:02:00 as the time information of the second time code of the first song. Reaction D occurs at 00:03:00 after the performance of the first song starts. However, in this case, the content data processing device 10 accepts reaction data for reaction D that includes time information of 00:14:00 as time information of the first time code. In this way, when a reaction occurs while a performance is occurring, the time information included in the reaction data may be time information of the first time code or time information of the second time code. Also, when a reaction occurs while a performance is occurring, the time information of the reaction data may include both time information of the first time code and time information of the second time code.

[0053] Because the reaction data includes time information of the first time code or the second time code, it is possible to accurately determine at what time in the live event or at what time in which performance the reaction data occurred even if there is a communication delay between the devices. This allows the live event manager to determine at what time in the live event or at what time in which performance the excitement was high by referring to the reaction data compilation results. The live event manager can use this compilation of reaction data to consider the organization or direction of the next live event.

[0054] The content data processing device 10A, the content data processing device 10B, and the content data processing device 10C may record the absolute time of when the first sound data is played, when the second sound data is played, when the third sound data is played, or when the reaction data is received, and transmit it to the content data processing device 10 or the tallying server. The content data processing device 10 or the tallying server can estimate the communication delay time from when the data is delivered to each device until it is received by calculating the difference between the absolute time when the data is delivered and the absolute time received from the content data processing device 10A, the content data processing device 10B, and the content data processing device 10C.

[0055] Next, an application example of related information will be described. Fig. 9 is a diagram showing an example of the structure of sound data according to the application example. As described above, the sound data is provided with time information of the first time code or time information of the second time code. The sound data is downloaded in advance from the content data processing device 10 installed at the venue of the live event. The sound data shown in Fig. 6 includes related information indicating the related destination as a header. However, in the application example, the related information is recorded as separate data.

[0056] FIG. 10 is a diagram showing an example of the structure of related information according to an application example. FIG. 11 is a conceptual diagram showing a hierarchical structure of related information according to an application example. As shown in FIG. 11, for example, the related information of the sound data of the first song has a live event related to the first time code registered as a related destination. The related information of the sound data of the second song has a live event related to the first time code registered as a related destination. Meanwhile, the related information of the sound data of the first performance has the sound data of the first song related to the second time code registered as a related destination. The related information of the sound data of the second performance has the sound data of the first song related to the second time code registered as a related destination. The related information of the sound data of the third performance has the sound data of the second song related to the second time code registered as a related destination. Therefore, the related information has a hierarchical structure as shown in FIG. 11.

[0057] In the example of FIG. 10, a device that plays content data (e.g., content data processing device 10A, content data processing device 10B, and content data processing device 10C) plays the sound data of the first song at 00:01:00 from the start timing of the live event, and ends the playback of the sound data of the first song at 00:04:30. A device that plays content data (hereinafter referred to as a playback device) plays the sound data of performance 1 at 00:00:30 from the start timing of playback of the first song, and ends the playback of the sound data of performance 1 at 00:01:30. The playback device plays the sound data of performance 2 at 00:01:40 from the start timing of playback of the first song, and ends the playback of the sound data of performance 2 at 00:02:40. The playback device plays the sound data of the second song at 00:05:30 from the start timing of the live event, and ends the playback of the sound data of the second song at 00:08:30. The playback device plays back the sound data of production 3 at 00:00:30 from the start timing of playback of the second song, and ends playback of the sound data of production 3 at 00:01:00.

[0058] The playback device can accept a change in the associated destination of the related information and change the associated destination of the sound data based on the accepted associated destination. For example, the user of the playback device changes the associated destination of the sound data of the second performance to the sound data of the second song, as shown in Fig. 12. Then, as shown in Fig. 13, the sound data of the second performance is changed to a lower layer of the sound data of the second song.

[0059] As a result, the sound data of effect 2 is played back at 00:01:40 from the start of the playback of the second song, and playback ends at 00:02:40 from the start of the playback of the second song.

[0060] In this way, the related information in the application example is recorded separately from the sound data. The playback device can easily edit the playback timing of sound data such as sound effects by simply changing the related destination of the related information without the need to edit the sound data. In addition, for example, when changing the order of songs, the user of the playback device only needs to change the start time and end time of the sound data of the first song and the start time and end time of the sound data of the second song, as shown in FIG. 14. In this case, the related destination, start time, and end time of the performance 1 and performance 2, which are lower layers of the first song, are not changed. Therefore, even if the order of songs is changed, the playback timing of the sound data such as sound effects related to each song does not change.

[0061] In the examples of Fig. 9 to Fig. 14, the playback device downloads sound data in advance. However, the playback device may download sound data in real time from the venue of the live event. Even if the order of songs in the live event is changed, the associated destination of the sound effects and the start time of the second time code are distributed, so that sound data such as the sound effects are played back at the appropriate timing. An operator who manages the live event may change the associated destination of the sound data of the sound effects in accordance with the progress of the live event.

[0062] The second time code may be musical time information instead of time information. The musical time information is, for example, information on measures, beats, and tempo (BPM: Beats-Per-Minute). Alternatively, the musical time information may include information indicating the structure of a song, such as "intro," "A verse," "B verse," "chorus 1," or "last chorus." In this case, the second sound data or third sound data, such as a special sound, is specified as the time information of the second time code, indicating at which measure and beat the playback should start.

[0063] This allows the user of the playback device and the operator who manages the live event to easily manage the playback timing of the sound effects, etc. For example, if the start timing of the sound data of the second performance shown in FIG. 9 is the first beat of the third bar of the second time code, even if the associated destination of the sound data of the second performance is changed to the sound data of the second song, the sound data of the second performance is played back on the first beat of the third bar of the second song. Or, even if the performer of the live event changes the tempo of the second song to a speed different from the original tempo, the sound data of the second song is always played back on the first beat of the third bar of the second song. The operator who manages the live event can play back the sound effects at the same musical timing even if the associated destination of the sound data of the sound effects is changed to a song with a completely different tempo in accordance with the progress of the live event.

[0064] Also, when the content data processing device 10A, the content data processing device 10B, and the content data processing device 10C store sound data of a live event and a user edits a live event that was held in the past, the editing can be performed based on the above musical time information. For example, a user can reproduce the same sound effect at the beginning of the chorus of the second song by simply copying the sound effect that starts at the beginning of the chorus of the first song to the beginning of the chorus of the second song.

[0065] The data generated and distributed by the content data processing device 10 is not limited to sound data. Moreover, the data reproduced by the content data processing device 10A, the content data processing device 10B, and the content data processing device 10C is not limited to sound data. For example, as shown in Fig. 15, the data generated and distributed by the content data processing device 10 includes at least any one of effect parameter data related to sound data, video data of a live event, and motion data.

[0066] The effect parameters include, for example, information regarding the localization of a special sound. When the second content data includes data of the effect parameters, the playback device plays a certain special sound (for example, the sound of fireworks) at a time specified by the second time code, and localizes the sound of the fireworks at a specified position. This allows an operator of a live event to generate special sound at any timing of any song and localize the special sound at any position in accordance with the progress of the live event.

[0067] The video data includes not only video data of the entire live event, but also video data of individual performers, or pre-recorded performance images (e.g., fireworks images). An operator managing the live event generates performance images (e.g., fireworks images) at any timing of any song in accordance with the progress of the live event. The playback device combines the performance images with the video data of the live event and displays them on the display 101.

[0068] If the operator further specifies effect parameters and distributes them, the effect image can be positioned at any position. Furthermore, the image data is not limited to the actual venue where the live event is being held, but may be pre-recorded image data of another venue. Furthermore, for example, the video data of the venue may change for each song. The video data may be computer graphics that mimic the image of the venue. The playback device may combine the video data of the actual venue with computer graphic effect images (for example, images of fireworks). In this case, too, the operator running the live event can play back the effect images at a specified timing for a specified song.

[0069] The effect parameters may include information about the reverberation of the space. The information about the reverberation of the space may be a filter coefficient of a digital filter such as an FIR filter, or may be information indicating the shape of a venue such as a concert hall (information indicating the size of the space, position information indicating the shape of the wall, information indicating the sound absorption rate of the material of the wall, etc.). The effect parameters may also include position information of a sound source such as a singing sound or a performance sound. The playback device may perform a process of localizing the singing sound or the performance sound based on the position information of the sound source. The playback device may generate an indirect sound such as an early reflection sound or a reverberation sound of the venue based on the information about the reverberation of the space. The playback device generates the indirect sound by, for example, convolving the above filter coefficient with a sound signal of the sound data. Alternatively, the playback device may calculate the level and delay amount of an indirect sound generated in the venue after a direct sound based on the information indicating the shape of the venue and the position information of the sound source, and generate a pseudo indirect sound.

[0070] The information about the spatial reverberation may be related to the video data of the venue. The content data processing device 10 distributes the spatial reverberation information corresponding to the video data of each venue. When the video data of the venue changes, the information about the spatial reverberation also changes. This allows the playback device to generate indirect sounds such as early reflection sounds and reverberation sounds that match the changes in the venue.

[0071] The motion data is data related to position information (two-dimensional or three-dimensional position information) of the motion capture worn by the performer. The playback device plays back the virtual character image, and changes the character image in accordance with the progression of the second time code based on the motion data included in the second content data. This replaces the performer's image with the virtual character image. An operator managing the live event can also change the character image in accordance with the progress of the live event, for example, between the performance of the first song and the performance of the second song. In this case, the second content data includes information specifying the character image.

[0072] The above-mentioned collected reaction data can be applied to performance. For example, in a live event where a certain performance sound (sound of fireworks) and a certain performance image (image of fireworks) are played at a certain timing of a certain performance, the content data processing device 10 distributes the performance sound and the performance image and plays the performance sound and the performance image when the collected number of texts corresponding to trending words is equal to or greater than a predetermined threshold. When the collected number of texts corresponding to trending words is less than a predetermined threshold, the content data processing device 10 does not distribute the performance sound and the performance image and does not play the performance sound and the performance image. Alternatively, the content data processing device 10 may change the performance sound and the performance image according to the collected number of texts corresponding to trending words. For example, the content data processing device 10 may increase the number of firework sounds and firework images as the number of texts corresponding to trending words increases. The performance image may also be an image of flying silver tape with text corresponding to trending words written thereon.

[0073] The content data processing device 10 may generate a sound effect according to the position information. For example, when the content data processing device 10 detects excitement on the east side of the venue of a live event, the content data processing device 10 may generate a sound effect of cheers localized on the east side of the venue of the live event.

[0074] The content data processing device 10 may generate a special sound and a special image according to the motion information. For example, the content data processing device 10 counts the movements of the participants detected by the acceleration sensors of the penlights, and generates a special sound (the sound of fireworks) and a certain special image (an image of fireworks) when it determines that the number of penlights has moved more than a predetermined threshold value.

[0075] The reaction data may be color information detected by photographing the penlights of the participants with a camera of a mobile terminal owned by the user. In this case, the content data processing device 10 generates a fireworks display image that matches the color of the penlights in the reaction data. For example, the content data processing device 10 collects the reaction data and generates a fireworks display image that matches the most popular color.

[0076] When the reaction data includes biological information, the content data processing device 10 generates a fireworks display image that matches the biological information of the reaction data. For example, the content data processing device 10 collects reaction data of heart rates, and generates many display sounds (fireworks sounds) and a certain display image (fireworks image) when the collected heart rate exceeds a threshold value.

[0077] The content data processing device 10A, the content data processing device 10B, and the content data processing device 10C may receive motion information by, for example, displaying an image of a penlight on the display device 101 and detecting a touch operation on the image of the penlight. Also, the content data processing device 10A, the content data processing device 10B, and the content data processing device 10C may receive color information by, for example, displaying a color image of a penlight on the display device 101 and detecting a touch operation on the color image of the penlight.

[0078] (Variation 1) The content data processing device 10A, the content data processing device 10B, and the content data processing device 10C of the first modification accept a reaction from a user, and the content data processing device 10 distributes an effect sound corresponding to the accepted reaction.

[0079] In other words, the content data processing method of variant example 1 accepts specific reactions from participants in a live event, generates reaction data to which musical time information is added as a second time code, and distributes sound data corresponding to the reaction data based on the second time code.

[0080] For example, the content data processing device 10A, the content data processing device 10B, and the content data processing device 10C display an icon image of "clap hands" on the display device 101, and accept a selection operation of the "clap hands" icon image from the user via the user I / F 102, thereby accepting a clap reaction. Alternatively, the content data processing device 10A, the content data processing device 10B, and the content data processing device 10C may accept a clap reaction when the selection operation of the "clap hands" icon image is accepted continuously at a certain interval. In particular, the content data processing device 10A, the content data processing device 10B, and the content data processing device 10C may accept a clap reaction when the selection operation of the "clap hands" icon image is accepted at an interval that matches the rhythm (BPM) of the content being played. The tempo (BPM) of the content being played may be acquired by analyzing the received content data, or the content data processing device 10 may distribute information indicating the tempo (BPM) of the content.

[0081] The content data processing device 10A, the content data processing device 10B, and the content data processing device 10C generate information corresponding to the received reaction, and generate reaction data by adding a second time code to the generated reaction information. The content data processing device 10 receives the reaction data from the content data processing device 10A, the content data processing device 10B, and the content data processing device 10C.

[0082] The content data processing device 10 distributes a performance sound corresponding to the received reaction based on the second time code of the received reaction data. Fig. 16 is a diagram showing the relationship between the time code, sound data, and time according to the first modification.

[0083] In this example, the second time code is not time information but musical time information. The content data processing device 10A, the content data processing device 10B, or the content data processing device 10C generates reaction data of "handclap" to which, for example, "the first beat of the third bar" is added as the second time code for the performance of the first piece of music.

[0084] When the content data processing device 10 receives the reaction data, it distributes the "clap" sound effect. The content data processing device 10A, the content data processing device 10B, and the content data processing device 10C play the "clap" sound effect. As a result, all users listening to the sound data of the live event hear the clap sound effect. If a user other than the user who made the "clap" reaction makes a "clap" reaction to the distributed clap sound effect, further reaction data is transmitted to the content data processing device 10. The content data processing device 10 distributes the "clap" sound effect based on each of the received reaction data.

[0085] On the other hand, when the content data processing device 10A, the content data processing device 10B, or the content data processing device 10C detects the end of the "clap" reaction, it generates reaction data by adding a second time code to the reaction information indicating "clap end". When the content data processing device 10 receives the reaction data of "clap end", it stops the distribution of the "clap" sound effect.

[0086] This allows the delivery of sound effects that correspond to the "clap" reactions of many users, allowing users to experience the feeling of clapping along with the other participants, creating a more immersive feeling of being at a live event.

[0087] Also, in this example, the second time code is musical time information. Therefore, even if different communication delays occur in the content data processing device 10A, the content data processing device 10B, or the content data processing device 10C at the distribution destination, the handclap effect sound can be reproduced at the same musical timing and tempo. Also, when reproducing sound data of a past live event, the handclap effect sound can be reproduced at the same musical timing and tempo in any reproduction device. However, it is preferable to reproduce multiple "handclap" effect sounds at slightly different timings randomly. This allows the user to hear a more natural handclap effect sound.

[0088] (Variation 2) The content data processing method of variant 2 acquires multiple sound data relating to the sounds of multiple performers at a live event, assigns a first time code or a second time code to each of the acquired sound data and records it, accepts a selection of any of the multiple sound data, and plays back the selected sound data based on the first time code or the second time code assigned to the selected sound data.

[0089] Specifically, the content data processing device 10 of the second modification first acquires sound signals of the performers of each performance (e.g., a sound signal of a singing sound, a sound signal of a musical performance, and a performance sound) separately, and records content data (sound data for each performer) in which a first time code or a second time code is added to the sound data related to the acquired sound signals. A user of the playback device (content data processing device 10A, content data processing device 10B, and content data processing device 10C) selects any sound data from the sound data for each performer. The playback device reproduces the selected sound data based on the first time code or the second time code.

[0090] 17 is a diagram showing an example of sound data acquired by the content data processing device 10 in the second modification. For example, in the first live event, the content data processing device 10 acquires sound signals of vocal 1 data, guitar 1 data, and bass 1 data for the sound data of the first song. In the second live event, the content data processing device 10 acquires sound signals of vocal 2 data, guitar 2 data, and bass 2 data for the sound data of the first song. Each data is assigned a first time code or a second time code. The second time code may be time information or musical time information.

[0091] The first and second live events each perform the same song, but with different performers. In the first song of the first live event, a first singer sings, a first guitar player plays guitar, and a first bass player plays bass. In the second song of the first live event, the same performers sing or play. In the first song of the second live event, a second singer sings, a second guitar player plays guitar, and a second bass player plays bass, the same song as in the first live event, while a second singer sings, a second guitar player plays guitar, and a second bass player plays bass. In the second song of the second live event, a third singer sings the same song as in the second song of the first live event.

[0092] As described above, the users of the content data processing device 10A, the content data processing device 10B, and the content data processing device 10C can edit the sound data of a live event. The users of the content data processing device 10A, the content data processing device 10B, and the content data processing device 10C of the second modification can arbitrarily select the sounds of each live event and each performer (for example, the sound signals of singing sounds, the sound signals of performance sounds, and the performance sounds) and combine the sound data of any performer of any song of a plurality of live events.

[0093] 18 is a diagram showing an example of a combination of sound data. In this example, for the first song, the user selects sound data of a vocalist 1 in a first live event, selects sound data of a guitarist 2 in a second live event, and selects sound data of a bassist 2 in the second live event.

[0094] As described above, each data is assigned a first time code or a second time code. The second time code may be time information or musical time information. If it is musical time information, the playback device can play back the sound data at the same musical timing and tempo even if the user selects different live events or performance sounds by different performers. The user of the second modification can create his or her own unique live event through editing by selecting any performance by any performer from multiple live events.

[0095] The description of the present embodiment should be considered as illustrative in all respects and not restrictive. The scope of the present invention is indicated by the claims, not by the above-described embodiments. Furthermore, the scope of the present invention includes the scope equivalent to the claims. [Explanation of symbols]

[0096] 1. Content data processing system 5. Internet 10, 10A, 10B, 10C...Content data processing device 101...Indicator 102...User I / F 103...Flash memory 104…CPU 105…RAM 106…Communication I / F

Claims

1. generating a first time code indicating an elapsed time from a start timing of the live event; receiving data related to the live event and generating first content data by adding a time of the first time code to the data; generating a second time code indicating an elapsed time from the timing of receiving the data; Content data processing method.

2. Distributing the first time code, the first content data, and the second time code; The content data processing method according to claim 1 .

3. the data includes first data and second data related to the first data; the first content data includes the first data as the data; receiving the second data and generating second content data by adding the time of the second time code to the second data; 3. The content data processing method according to claim 1.

4. further distributing the second content data; The content data processing method according to claim 3 .

5. a first time code indicating an elapsed time from a start timing of the live event; first content data in which the first time code is added to data related to the live event; A second time code indicating an elapsed time from a start timing of the data; A content data processing method for receiving the data includes first data and third data related to the first data; The content data processing method includes reproducing the first data included in the first content data based on the first time code; Accepting the third data, reproducing the third data based on the second time code; Content data processing method.

6. Accepting reactions from participants of said live event; generating reaction data to which the first time code or the second time code is assigned; The content data processing method according to claim 5 .

7. accepting and aggregating the reaction data from a plurality of participants; The content data processing method according to claim 6.

8. the data includes second data related to the first data; receiving second content data in which the second time code is added to the second data; reproducing the second data based on the second time code; The content data processing method according to any one of claims 5 to 7.

9. Generate related information indicating a related destination of the data; The content data processing method according to any one of claims 1 to 8.

10. Accepting a change to the related destination of the related information; changing the associated destination of the data based on the accepted associated destination; The content data processing method according to claim 9.

11. The data includes sound data. The content data processing method according to any one of claims 1 to 10.

12. the second time code includes musical time information; The content data processing method according to claim 11.

13. Accepting certain reactions from participants of said live event; generating reaction data by adding the musical time information as the second time code to information indicating the specific reaction; delivering sound data corresponding to the reaction data based on the second time code; The content data processing method according to claim 12.

14. The data includes video data. The content data processing method according to any one of claims 1 to 13.

15. The data includes motion data. The content data processing method according to any one of claims 1 to 14.

16. acquiring a plurality of pieces of sound data relating to sounds of a plurality of performers at the live event; assigning the first time code or the second time code to each of the acquired sound data and recording the data; Accepting a selection of any one of the plurality of recorded sound data, reproducing the selected sound data based on the first time code or the second time code assigned to the selected sound data; The content data processing method according to any one of claims 1 to 15.

17. generating a first time code indicating an elapsed time from a start timing of the live event; receiving data related to the live event and generating first content data by adding a time of the first time code to the data; generating a second time code indicating an elapsed time from the timing of receiving the data; A processing unit is provided. Content data processing device.

18. the processing unit distributes the first time code, the first content data, and the second time code; The content data processing device according to claim 17.

19. the data includes first data and second data related to the first data; the first content data includes the first data as the data; the processing unit receives the second data and generates second content data by adding the time of the second time code to the second data; 19. A content data processing device according to claim 17 or 18.

20. The processing unit further distributes the second content data. The content data processing device according to claim 19.

21. a first time code indicating an elapsed time from a start timing of the live event; first content data in which the first time code is added to data related to the live event; A second time code indicating an elapsed time from a start timing of the data; A content data processing device having a processing unit for receiving the data includes first data and third data related to the first data; The processing unit reproduces the first data included in the first content data based on the first time code; Accepting the third data, reproducing the third data based on the second time code; Content data processing device.

22. The processing unit receives reactions from participants of the live event, generating reaction data to which the first time code or the second time code is assigned; The content data processing device according to claim 21 .

23. The processing unit receives and aggregates the reaction data from a plurality of participants. The content data processing device according to claim 22.

24. the data includes second data related to the first data; The processing unit receives second content data in which the second data is assigned with the time of the second time code, reproducing the second data based on the second time code; 24. A content data processing device according to any one of claims 21 to 23.

25. The processing unit generates association information indicating an association destination of the data. A content data processing device according to any one of claims 17 to 24.

26. The processing unit accepts a change to the associated destination of the associated information, changing the associated destination of the data based on the accepted associated destination; The content data processing device according to claim 25.

27. The data includes sound data.

27. A content data processing device according to any one of claims 17 to 26.

28. the second time code includes musical time information; 28. A content data processing device according to claim 27.

29. The processing unit receives specific reactions from participants of the live event, generating reaction data by adding the musical time information as the second time code to information indicating the specific reaction; delivering sound data corresponding to the reaction data based on the second time code; 29. A content data processing device according to claim 28.

30. The data includes video data.

30. A content data processing device according to any one of claims 17 to 29.

31. The data includes motion data. A content data processing device according to any one of claims 17 to 30.

32. the processing unit acquires a plurality of pieces of sound data relating to sounds of a plurality of performers in the live event, assigning the first time code or the second time code to each of the acquired sound data and recording the data; Accepting a selection of any one of the plurality of recorded sound data, reproducing the selected sound data based on the first time code or the second time code assigned to the selected sound data; A content data processing device according to any one of claims 17 to 31.

Images