Signal processing device, signal processing system, and signal processing method

Abstract

To allow a user to appropriately perform operation according to target timing.SOLUTION: A signal processor includes: a reception part for receiving first time series data including sound data and second time series data generated on the basis of the first time series data and including at least data indicating operation timing of a person; a generation part for generating third time series data informing the operation timing on the basis of the second time series data; and a processing part for synchronizing and outputting an output signal based on the first time series data and an output signal based on the third time series data so that the operation timing in the first time series data is matched with the operation timing informed by the third time series data.SELECTED DRAWING: Figure 1

Description

Technical Field

[0001] The present invention relates to a signal processing device , signal processing system, and signal processing method and is concerned with it.

Background Art

[0002] In accordance with time-series data such as performance data, for example, a user may perform operations such as playing a musical instrument to perform an ensemble. Also, techniques for efficiently performing such an ensemble are known (see, for example, Patent Document 1).

Prior Art Documents

Patent Documents

[0003]

Patent Document 1

Summary of the Invention

Problems to be Solved by the Invention

[0004] However, when performing an operation such as playing a musical instrument to perform an ensemble in accordance with time-series data such as performance data, it may be difficult to perform an operation such as playing at an appropriate timing without knowing the operation timing such as the start timing of the performance.

[0005] The present invention has been made to solve the above problems, and an object thereof is to provide a signal processing device that can appropriately perform an operation in accordance with a target timing for a user and enable appropriate operation according to the target timing. , signal processing system, and signal processing method to provide.

Means for Solving the Problems

[0006] To solve the above problem, one aspect of the present invention includes a generation unit that generates video data indicating the operation timing based on beat data indicating the operation timing generated from performance data, and a processing unit that synchronizes and outputs an output signal based on the performance data and an output signal based on the video data so that the operation timing associated with the performance data and the operation timing associated with the video data match. The aforementioned operation timing includes the start timing or end timing of the performance. It is a signal processing device. [Effects of the Invention]

[0007] According to the present invention, users can perform actions appropriately at the desired timing. [Brief explanation of the drawing]

[0008] [Figure 1] This is a block diagram showing an example of a signal processing system according to the first embodiment. [Figure 2] This figure shows an example of a video of a conductor. [Figure 3] This figure shows an example of the operation of the signal processing system according to the first embodiment. [Figure 4] This is a block diagram showing an example of a signal processing system according to a second embodiment. [Figure 5] This figure shows an example of the operation of the signal processing system according to the second embodiment. [Modes for carrying out the invention]

[0009] Hereinafter, a signal processing system, a signal processing device, and a signal processing method according to one embodiment of the present invention will be described with reference to the drawings.

[0010] [First Embodiment] Figure 1 is a block diagram showing the signal processing system 1 according to this embodiment. The signal processing system 1 comprises a signal processing device 10 and a transmitting device 20. The signal processing device 10 and the transmitting device 20 can be connected via the network NW1.

[0011] The transmitting device 20 generates second time-series data, which includes at least data indicating the timing of human actions, based on first time-series data including sound data, and transmits both the first and second time-series data. The first time-series data may be, for example, performance data such as a musical performance, or video data of gymnastics or dance. In this embodiment, the case where the first time-series data is performance data will be described.

[0012] Furthermore, the second time series data is data generated based on the first time series data. The second time series data is, for example, beat data containing time information of beat points in performance data, and is time series data with a smaller amount of data than the first time series data. In this embodiment, the case in which the second time series data is beat data will be described. Furthermore, the timing of human actions refers to the timing of actions performed by user U1 (for example, actions such as playing a musical instrument or singing) in accordance with the output signal based on the first time-series data (for example, the sound signal of a performance based on performance data). Action timing includes, for example, the start timing of the performance (the start of the whole performance, the start of a solo part, etc.) and the end timing of the performance.

[0013] The transmitting device 20 includes a network communication unit 21, an input unit 22, a display unit 23, a storage unit 24, and a control unit 40. The NW communication unit 21 is an interface unit that can be connected to the network NW1, and communicates with the signal processing unit 10 via the network NW1.

[0014] The input unit 22 is, for example, an input device such as a keyboard, mouse, or touch panel, and accepts various inputs from the user. The input unit 22 is used for various operations when using the transmission device 20, such as selecting and specifying performance data. The display unit 23 is, for example, a display device such as a liquid crystal display, and displays video of performance data, beat data, images of various operation screens, etc.

[0015] The storage unit 24 stores various information used by the transmission device 20. The storage unit 24 includes a performance data storage unit 241 and a beat data storage unit 242.

[0016] The performance data storage unit 241 stores performance data that is the source data for generating beat data. The performance data is, for example, sound data such as musical instrument performances or singing voices.

[0017] The beat data storage unit 242 stores beat data indicating the times of the beat points of the performance data. The beat data is data corresponding to the performance data. Note that the beat data may include data indicating the strength and weakness of the movement (for example, the strength and weakness of the performance).

[0018] The control unit 40 is a processor including, for example, a CPU (Central Processing Unit), and controls the transmission device 20. The control unit 40 includes a beat data generation unit 41 and a transmission processing unit 42.

[0019] The beat data generation unit 41 generates beat data based on the performance data. The beat data generation unit 41 stores the generated beat data in the beat data storage unit 242. Note that the beat data can also be said to be compressed data generated based on the performance data.

[0020] The transmission processing unit 42 transmits the performance data and the beat data to the signal processing device 10. The transmission processing unit 42 transmits the performance data and the beat data to the signal processing device 10 via, for example, the NW communication unit 21 and a network. The transmission processing unit 42 includes a performance data transmission unit 421 and a beat data transmission unit 422.

[0021] The performance data transmission unit 421 transmits the performance data to the signal processing device 10 via the NW communication unit 21. The beat data transmission unit 422 transmits the beat data to the signal processing device 10 via the NW communication unit 21.

[0022] The signal processing device 10 is a receiving device connected to the network NW1 and capable of communicating with the transmitting device 20. The signal processing device 10 receives first time-series data and second time-series data from the transmitting device 20 via the network NW1. The signal processing device 10 generates third time-series data that indicates the operation timing based on the second time-series data. The signal processing device 10 synchronizes and outputs an output signal based on the first time-series data and an output signal based on the third time-series data so that the operation timing in the first time-series data matches the operation timing in the third time-series data. In this embodiment, the case where the third time-series data is video data will be described. The signal processing unit 10 receives performance data and beat data. Based on the beat data, the signal processing unit 10 generates video data showing the conductor's image and outputs the conductor's image in accordance with the sound signal based on the performance data. The signal processing device 10 comprises a network communication unit 11, an input unit 12, a display unit 13, a speaker 14, a storage unit 15, and a control unit 30.

[0023] The NW communication unit 11 has the same configuration and functions as the NW communication unit 21. The NW communication unit 11 communicates with the transmitting device 20 via the network NW1.

[0024] The input unit 12 and the display unit 13 have the same configuration and functions as the input unit 22 and the display unit 23. The input unit 12 is used for various operations when using the signal processing device 10, such as selecting and specifying various settings for generating video data of the conductor. The display unit 13 displays, for example, the conductor's video, images of various operation screens, etc.

[0025] Speaker 14 outputs various sound signals used by the signal processing unit 10. Speaker 14 outputs, for example, sound signals based on performance data.

[0026] The memory unit 15 stores various types of information used by the signal processing unit 10. The memory unit 15 includes a performance data storage unit 151, a beat data storage unit 152, a generation information storage unit 153, and a video data storage unit 154.

[0027] The performance data storage unit 151 stores the performance data received from the transmission device 20. The beat data storage unit 152 stores the beat data received from the transmitting device 20.

[0028] The generation information storage unit 153 stores data used when generating video data. For example, the generation information storage unit 153 stores a trained model, which is the result of training built in advance using machine learning, and various setting information for generating video data, such as video data showing the conductor. The trained model is a model that outputs a third time series data when the input is a second time series data. In this embodiment, the trained model is a model that outputs video data showing the conductor when beat data is input.

[0029] The video data storage unit 154 stores video data that indicates the timing of actions such as musical performance. The video data of the conductor may include actual video footage of the conductor, or it may be an illustration or computer graphic animation. The video data of the conductor may also display the movements of the baton and hands.

[0030] The control unit 30 is a processor, such as a CPU, and controls the signal processing device 10. The control unit 30 comprises a receiving processing unit 31, a video data generation unit 32, and an output processing unit 33.

[0031] The receiving processing unit 31 is an example of a receiving unit. The receiving processing unit 31 receives performance data and beat data from the transmitting device 20 via the network NW1 and the NW communication unit 11. The receiving processing unit 31 comprises a performance data receiving unit 311 and a beat data receiving unit 312.

[0032] The performance data receiving unit 311 receives performance data transmitted from the transmitting device 20 via the network NW1. The performance data receiving unit 311 stores the received performance data in the performance data storage unit 151.

[0033] The beat data receiving unit 312 receives beat data transmitted from the transmitting device 20 via the network NW1. The beat data receiving unit 312 stores the received beat data in the beat data storage unit 152.

[0034] The video data generation unit 32 is an example of a generation unit. The video data generation unit 32 generates video data of a conductor that indicates the timing of actions based on beat data. The video data generation unit 32 acquires beat data stored in the beat data storage unit 152 and generates video data of the conductor in video M1 as shown in Figure 2, based on the acquired beat data and the data stored in the generation information storage unit 153. For example, the video data generation unit 32 uses the learning model information stored in the generation information storage unit 153 to generate video data of the conductor based on beat data using machine learning.

[0035] Furthermore, if the beat data includes data indicating the intensity of movement (for example, the intensity of the performance), the video data generation unit 32 generates video data of the conductor that includes data indicating the intensity of movement. The video data generation unit 32 stores the generated video data of the conductor in the video data storage unit 154.

[0036] The output processing unit 33 is an example of a processing unit. The output processing unit 33 synchronizes and outputs the performance sound signal based on the performance data and the conductor's video signal based on the conductor's video data so that the timing of the actions in the performance data matches the timing of the actions in the conductor's video data. The output processing unit 33 outputs the performance sound signal based on the performance data from the speaker 14, and synchronizes the output of the conductor's video signal from the display unit 13 with the sound signal (see the conductor's video M1 in Figure 2).

[0037] Next, with reference to Figure 3, the operation of the signal processing system 1 according to this embodiment will be described. Figure 3 shows an example of the operation of the signal processing system 1 according to this embodiment. First, the transmitting device 20 generates beat data for the performance data (step S101). The beat data generation unit 41 obtains the performance data specified by the operation of the input unit 22 from the performance data storage unit 241. Based on the performance data, the beat data generation unit 41 generates beat data including time information of the beat points. The beat data generation unit 41 stores the generated beat data in the beat data storage unit 242.

[0038] The beat data generation unit 41 may also display the performance data and the generated beat data on the display unit 23 or otherwise make them visible to the user of the transmitting device 20, allowing the user to confirm the beat data and adjust or correct the beat data based on the user's instructions.

[0039] Next, the transmitting device 20 transmits the performance data to the signal processing device 10 (step S102). The performance data transmitting unit 421 transmits the performance data acquired from the performance data storage unit 241 to the signal processing device 10. As a result, the performance data receiving unit 311 receives the performance data from the transmitting device 20 and stores the received performance data in the performance data storage unit 151.

[0040] Next, the transmitting device 20 transmits the beat data to the signal processing device 10 (step S103). The beat data transmitting unit 422 transmits the beat data to the signal processing device 10. As a result, the beat data receiving unit 312 receives the beat data from the transmitting device 20 and stores the received beat data in the beat data storage unit 152.

[0041] Next, the signal processing device 10 accepts the selection of video data (step S104). The video data generation unit 32 accepts the selection of video data (such as specifying settings) by the user U1 via the input unit 12. The signal processing device 10 may accept, for example, the specification of video data of the conductor that corresponds only to the timing of the performance, or the specification of video data of the conductor that includes instructions on the intensity of the movements, etc.

[0042] Next, the signal processing device 10 generates video data showing the conductor's image based on the beat data (step S105). The video data generation unit 32 generates video data from the beat data and a trained model, which is the learning result of machine learning stored in the generation information storage unit 153, based on the video data selection (setting specification, etc.) by the user U1. The video data generation unit 32 stores the generated video data in the video data storage unit 154.

[0043] Next, the signal processing unit 10 outputs the video data and the performance data in synchronization (step S106). The output processing unit 33 outputs an audio signal based on the performance data from the speaker 14. The output processing unit 33 also outputs a video signal of the conductor based on the video data, such as video M1 in Figure 2, from the display unit 13 in synchronization with the audio signal.

[0044] User U1 performs actions such as playing a musical instrument or singing in accordance with the conductor's video output by the signal processing device 10. This allows user U1 to appropriately perform actions such as playing a musical instrument in accordance with the sound signals based on the performance data.

[0045] In the example shown in Figure 3, the signal processing device 10 generates video data representing the conductor corresponding to the beat data in step S105, and outputs a video signal of the conductor based on the video data in step S106. This can be implemented in various ways. For example, the signal processing device 10 may perform step S106 once for the entire beat data. Alternatively, the signal processing device 10 may implement this by repeating the processes in steps S105 and S106.

[0046] As described above, the signal processing device 10 according to this embodiment comprises a receiving processing unit 31, a video data generation unit 32, and an output processing unit 33. The receiving processing unit 31 receives first time-series data including sound data and second time-series data generated based on the first time-series data and including at least data indicating the timing of human movement. The video data generation unit 32 generates third time-series data indicating the timing of movement based on the second time-series data. The output processing unit 33 synchronizes and outputs an output signal based on the first time-series data and an output signal based on the third time-series data so that the timing of movement in the first time-series data matches the timing of movement indicated in the third time-series data.

[0047] As a result, in the signal processing device 10 according to this embodiment, the output signal based on the third time-series data and the output signal based on the first time-series data are output in synchronization, so that the user U1 can appropriately recognize the timing of operation and perform the operation in accordance with the desired timing.

[0048] Furthermore, since the signal processing device 10 according to this embodiment receives beat data from the transmitting device 20, which has a smaller data volume than the first time-series data, the amount of communication data can be reduced, and the delay of the output signal based on the third time-series data can be reduced.

[0049] In this embodiment, the first time-series data is performance data, and the operation timing includes the operation start timing. The second time-series data is beat data indicating a beat that includes at least the operation start timing. The video data generation unit 32 generates video data as a third time-series data based on the beat data. The output processing unit 33 outputs a video signal based on the video data, synchronized with an output signal based on the performance data.

[0050] As a result, with the signal processing device 10 according to this embodiment, user U1 can appropriately recognize, for example, the start timing of a performance. Therefore, user U1 can appropriately play an instrument or sing a song, etc., in time with the output signal based on the performance data.

[0051] Furthermore, in this embodiment, the video data of the third time-series data is video data of the conductor. Since the conductor has the role of communicating the timing of various actions, using the conductor's video makes it easier to recognize the start time of an action. Therefore, by visualizing the video signal based on the conductor's video data, user U1 can appropriately play their instrument or other instruments in accordance with the timing of the performance data received from the transmitting device 20.

[0052] Furthermore, in this embodiment, the video data generation unit 32 generates video data of the conductor based on beat data using machine learning. As a result, the signal processing device 10 according to this embodiment can easily create video data of the conductor from beat data.

[0053] In this embodiment, the receiving processing unit 31 receives a second time-series data, which has a smaller data volume than the first time-series data. The video data generation unit 32 decodes the second time-series data to generate a third time-series data.

[0054] As a result, the signal processing device 10 in this embodiment receives second time-series data, which has a smaller data volume than the first time-series data, thus reducing the amount of communication data. This allows it to generate and output third time-series data prior to outputting the first time-series data. Therefore, it can output a signal based on the third time-series data without delay following the output of the sound signal based on the first time-series data.

[0055] In this embodiment, the second time-series data includes beat point data indicating the beat of the action and data indicating the intensity of the action. The video data generation unit 32 generates a third time-series data which includes data indicating the intensity of the action.

[0056] As a result, in the signal processing device 10 according to this embodiment, user U1 can recognize the timing of the intensity of the operation and perform more appropriate operations, including the intensity of the operation, in accordance with the first time-series data. In other words, in the signal processing device 10 according to this embodiment, user U1 can perform expressive performances, including the intensity of the operation, in accordance with the output of the performance data.

[0057] Furthermore, the signal processing system 1 according to this embodiment includes a transmitting device 20 that generates a second time series data including at least data indicating the timing of human actions based on a first time series data including data, and transmits the first time series data and the second time series data, and a signal processing device 10. As a result, the signal processing system 1 according to this embodiment achieves the same effect as the signal processing device 10, allowing the user U1 to operate appropriately in accordance with the desired timing.

[0058] [Second Embodiment] Next, with reference to the drawings, a signal processing system 1a and a signal processing device 10a according to a second embodiment will be described.

[0059] Figure 4 is a block diagram showing an example of a signal processing system 1a according to this embodiment. The signal processing system 1a comprises a plurality of signal processing devices 10a (10a-1, 10a-2, ...) and a transmitting device 20a. The plurality of signal processing devices 10a and the transmitting device 20a can be connected via a network NW1.

[0060] In this embodiment, an example of how multiple users can perform an ensemble performance is described, in which a transmitting device 20a transmits beat data to multiple signal processing devices 10a, and each of the multiple signal processing devices 10a outputs a video of the conductor based on the beat data. In Figure 4, components identical to those in Figure 1 are given the same reference numerals, and their explanations are omitted. Also, in Figure 4, signal processing devices 10a-1, 10a-2, ... each have a similar configuration, and when they represent any signal processing device included in the signal processing system 1a, or when no particular distinction is made, they are described as signal processing device 10a.

[0061] The transmitting device 20a has the same basic functions as the transmitting device 20, but differs from the first embodiment in that it does not transmit performance data to the signal processing device 10a. In this embodiment, it is assumed that the musical piece of performance data is shared between the transmitting device 20a and the signal processing device 10a. The transmitting device 20a comprises a network communication unit 21, an input unit 22, a display unit 23, a storage unit 24, and a control unit 40a.

[0062] The control unit 40a has the same basic configuration and function as the control unit 40. The control unit 40a includes a beat data generation unit 41 and a transmission processing unit 42a. The transmission processing unit 42a differs from the transmission processing unit 42 in that it includes a beat data transmission unit 422 and does not include the performance data transmission unit 421 described above. The transmission processing unit 42a transmits beat data generated based on performance data of a predetermined musical piece to a plurality of signal processing units 10a.

[0063] The signal processing device 10a has the same basic functions as the signal processing device 10a, but differs in that it does not receive performance data from the transmitter 20a and has an added function for ensemble performance among multiple signal processing devices 10a. The signal processing device 10a comprises a network communication unit 11, an input unit 12, a display unit 13, a speaker 14, a storage unit 15a, a microphone 16, and a control unit 30. In this embodiment, user U1 is a user of signal processing device 10a-1, and user U2 corresponds to a user of signal processing device 10a-2.

[0064] The memory unit 15a differs from the memory unit 15 in that it includes a beat data storage unit 152, a generation information storage unit 153, and a video data storage unit 154, but does not include a performance data storage unit 151. Microphone 16 picks up sounds from the vicinity of the signal processing device 10a and outputs the picked-up sound signal. During ensemble playing, microphone 16 outputs the sound signal of the user's performance to the control unit 30a.

[0065] The control unit 30a has the same basic configuration and functions as the control unit 30. The control unit 30a comprises a receiving processing unit 31a, a video data generation unit 32, an output processing unit 33a, and an ensemble processing unit 34.

[0066] The receiving processing unit 31a differs from the receiving processing unit 31 in that it includes a beat data receiving unit 312 and does not include the performance data receiving unit 311 described above. The output processing unit 33a outputs a video signal based on the conductor's video data generated by the video data generation unit 32, so that the timing of the performance can be visually recognized.

[0067] The ensemble processing unit 34 receives performance sound signals from other devices (for example, other signal processing devices 10a, etc.) via the NW communication unit 11 and the network NW1, and outputs the received performance sound signals to the speaker 14. The ensemble processing unit 34 also transmits the performance sound signals picked up by the microphone 16 to other devices (for example, other signal processing devices 10a, etc.) via the NW communication unit 11 and the network NW1.

[0068] Next, with reference to Figure 5, the operation of the signal processing system 1a according to this embodiment will be described. Figure 5 illustrates an example of an operation in which users U1 and U2 perform a musical duet using beat data transmitted from the transmitter 20a.

[0069] In Figure 5, the process in step S201 is the same as the process in step S101 shown in Figure 3, so its explanation is omitted here. Next, the transmitting device 20a transmits the beat data to each signal processing device 10a (10a-1, 10a-2) (step S202). The beat data transmitting unit 422 transmits the beat data to each signal processing device 10a. As a result, the beat data receiving unit 312 of the signal processing device 10a receives the beat data from the transmitting device 20a and stores the received beat data in the beat data storage unit 152.

[0070] The subsequent processing in step S203 by signal processing device 10a-1 and the processing in step S205 by signal processing device 10a-2 are the same as the processing in step S105 shown in Figure 3, so their explanation is omitted here.

[0071] In step S204, the signal processing unit 10a-1 picks up the sound of user U1's performance from the microphone 16, which has output the conductor's video. The output processing unit 33a outputs a video signal from the display unit 13 based on the conductor's video data stored in the video data storage unit 154, such as the video M1 in Figure 2. The ensemble processing unit 34 also acquires the sound signal (performance sound signal) of user U1's performance picked up by the microphone 16.

[0072] Furthermore, in step S206, the signal processing unit 10a-2 picks up the performance of user U2, which has output the conductor's video, from the microphone 16. The output processing unit 33a outputs a video signal from the display unit 13 based on the conductor's video data stored in the video data storage unit 154, such as the video M1 in Figure 2. The ensemble processing unit 34 also acquires the performance sound signal of user U2 picked up by the microphone 16.

[0073] Next, signal processing devices 10a-1 and 10a-2 transmit their respective performance sound signals to each other and perform ensemble processing (step S207). The ensemble processing unit 34 of signal processing device 10a-1 transmits the performance sound signal of user U1 to signal processing device 10a-2, receives the performance sound signal of user U2 from signal processing device 10a-2, and causes the speaker 14 of signal processing device 10a-1 to output the performance sound signal of user U1. The ensemble processing unit 34 of signal processing device 10a-2 also transmits the performance sound signal of user U2 to signal processing device 10a-1, receives the performance sound signal of user U1 from signal processing device 10a-1, and causes the speaker 14 of signal processing device 10a-2 to output the performance sound signal of user U1. Thus, the signal processing system 1a according to this embodiment allows for ensemble playing among multiple users located in different locations.

[0074] As described above, the signal processing device 10a according to this embodiment comprises a receiving processing unit 31a (receiving unit), a video data generation unit 32 (generation unit), and an output processing unit 33a (processing unit). The receiving processing unit 31a receives second time-series data (e.g., beat data) which includes at least data indicating the timing of the performance, generated based on performance data (first time-series data) of a predetermined musical piece being played. The video data generation unit 32 generates third time-series data (e.g., video data of the conductor) based on the second time-series data (e.g., beat data) received by the receiving processing unit 31a, making the timing of the performance visually recognizable. The output processing unit 33a outputs an output signal (e.g., video signal) based on the third time-series data (e.g., video data of the conductor) so that the timing of the performance can be visually recognized.

[0075] As a result, the signal processing device 10a and signal processing system 1a according to this embodiment enable users to perform appropriately in accordance with the desired timing. Furthermore, by having the transmitting device 20a transmit beat data to multiple signal processing devices 10a, the signal processing device 10a and signal processing system 1a according to this embodiment enable multiple users in different locations to synchronize their performance timing and perform together without meeting in person.

[0076] It should be noted that the present invention is not limited to the embodiments described above, and can be modified without departing from the spirit of the invention. The second time-series data, while described as beat data, is not limited to this. The second time-series data may also be data indicating positional reference information. Examples of positional reference information include beat point time (including advance notice), beat point data, measure boundary time, measure number, time signature, rehearsal mark, movement number, and movement name. In addition to positional reference information, it may also be data indicating other information. Examples of other information include standard tempo, tempo markings, tempo change markings, dynamics, dynamic changes, expression markings, articulation, conductor's facial expression, conductor's gestures, conductor's dialogue, phrase start for each part, phrase end for each part, start of solo part, end of solo part, and ending signal.

[0077] Furthermore, the third time-series data may include sound data in addition to video data. An example of sound data would be the sound of a metronome synchronized with the conductor's video. In this case, the video data may be video of the metronome instead of video of the conductor, or video of the metronome may be shown in addition to video of the conductor. For example, in each of the embodiments described above, an example of using the signal processing system 1(1a) for playing a musical instrument was explained, but it is not limited to this, and the actions performed by the user may be other than playing a musical instrument, such as dancing, gymnastics, reciting poetry to background music, or radio drama.

[0078] Furthermore, in each of the embodiments described above, an example was applied in which video data of the conductor was used as the third time-series data. However, the invention is not limited to this. For example, if the first time-series data is video data of gymnastics or dance, the third time-series data may be sound data of cues or lines indicating the start timing of the dance or gymnastics (for example, shouts or cues such as "Bigger!" or "Slower!") or video data of the instructor (for example, video showing facial expressions and gestures).

[0079] Furthermore, in the first embodiment described above, similar to the second embodiment, beat data may be transmitted to multiple signal processing devices 10 to perform ensemble playing or the like. In this case, the signal processing device 10 may also include the ensemble playing processing device 34 described above.

[0080] Furthermore, in the first embodiment described above, the signal processing device 10 may set a predetermined shift range when outputting an output signal based on the third time-series data in synchronization with the output signal based on the first time-series data. The signal processing device 10 may also be able to change the speed, pause, or otherwise control the synchronized output.

[0081] Furthermore, in each of the above embodiments, the signal processing device 10(10a) may be configured to change the main part in the conductor's video. In this case, the main part may be a bass instrument or a rhythm instrument.

[0082] Each component of the signal processing system 1(1a) described above has an internal computer system. The processing steps of each component of the signal processing system 1(1a) described above are stored in program form on a computer-readable recording medium, and the above processing is performed when the computer reads and executes this program. Here, a computer-readable recording medium refers to a magnetic disk, magneto-optical disk, CD-ROM, DVD-ROM, semiconductor memory, etc. Alternatively, this computer program may be distributed to a computer via a communication line, and the computer that receives the distribution may execute the program. [Explanation of symbols]

[0083] 1,1a...Signal processing system, 10,10a,10a-1,10a-2...Signal processing device, 11,21...NW communication unit, 12,22...Input unit, 13,23...Display unit, 14...Speaker, 15,15a,24...Storage unit, 16...Microphone, 30,30a,40,40a...Control unit, 31,31a...Receiving processing unit, 32...Video data generation unit, 33,33a...Output processing unit, 34 ...Ensemble processing unit, 41...Beat data generation unit, 42,42a...Transmission processing unit, 151,241...Performance data storage unit, 152,242...Beat data storage unit, 153...Generation information storage unit, 154...Video data storage unit, 311...Performance data receiving unit, 312...Beat data receiving unit, 421...Performance data transmission unit, 422...Beat data transmission unit, NW1...Network, U1,U2...Users

Claims

1. A generation unit that generates video data indicating the timing of the operation based on beat data indicating the timing of the operation generated from performance data, A processing unit that synchronizes and outputs an output signal based on the performance data and an output signal based on the video data so that the operation timing associated with the performance data and the operation timing associated with the video data coincide. Equipped with, The aforementioned operation timing includes the start timing or end timing of the performance. Signal processing device.

2. The aforementioned video data is video data of the conductor. The signal processing apparatus according to claim 1.

3. The generation unit generates video data of the conductor based on the beat data using machine learning. The signal processing apparatus according to claim 2.

4. The generation unit decodes the beat data, which is smaller in data size than the performance data, and generates the video data. The signal processing apparatus according to any one of claims 1 to 3.

5. The generation unit generates the video data which includes data indicating the intensity of the movement. The signal processing apparatus according to any one of claims 1 to 4.

6. The system includes a transmitting device that generates beat data indicating the timing of operations based on performance data and transmits the performance data and the beat data, and a signal processing device. The signal processing device is A generation unit that generates video data indicating the timing of the operation based on the beat data, A processing unit that synchronizes and outputs an output signal based on the performance data and an output signal based on the video data so that the operation timing associated with the performance data and the operation timing associated with the video data coincide. Equipped with, The aforementioned operation timing includes the start timing or end timing of the performance. Signal processing system.

7. The generation unit generates video data that indicates the timing of the operation based on beat data that indicates the timing of the operation, generated from the performance data. A processing step in which the processing unit synchronizes and outputs an output signal based on the performance data and an output signal based on the video data so that the operation timing associated with the performance data and the operation timing associated with the video data coincide. Includes, The aforementioned operation timing includes the start timing or end timing of the performance. Signal processing method.

Images