Signal processing device, signal processing system and signal processing method

Abstract

The signal processing device comprises: a receiving unit, which receives first time series data and second time series data, wherein the first time series data includes sound data, and the second time series data is generated based on the first time series data and includes at least data representing the timing of a person's action; a generating unit, which generates third time series data for notifying the action timing based on the second time series data; and a processing unit, which synchronizes and outputs an output signal based on the first time series data and an output signal based on the third time series data so that the action timing in the first time series data is consistent with the action timing notified by the third time series data.

Description

Technical Field

[0001] The invention relates to a signal processing device, a signal processing system and a signal processing method. Background Art

[0002] The user may perform an action such as playing a musical instrument to perform an ensemble in coordination with the time-series data such as the performance data, etc. A technique for efficiently performing such an ensemble is known (for example, refer to Patent Document 1).

[0003] Prior art literature

[0004] Patent Literature

[0005] Patent Document 1: Japanese Patent Application Publication No. 2004-145221 Summary of the invention

[0006] Problems to be solved by the invention

[0007] However, when playing a musical instrument in ensemble in coordination with time-series data such as performance data, for example, the timing of the action such as the start timing of the performance is unknown, and it may be difficult to perform the action such as the performance at an appropriate timing.

[0008] The present invention has been made to solve the above-mentioned problems, and an object of the present invention is to provide a signal processing device, a signal processing system, and a signal processing method that allow a user to appropriately perform operations in accordance with a target timing.

[0009] Means for solving problems

[0010] In order to solve the above-mentioned problem, one embodiment of the present invention is a signal processing device, comprising: a receiving unit, receiving first time series data and second time series data, the first time series data including sound data, the second time series data being generated based on the first time series data and at least including data representing the action timing of a person; a generating unit, generating third time series data for notifying the action timing based on the second time series data; and a processing unit, synchronizing an output signal based on the first time series data with an output signal based on the third time series data so that the action timing in the first time series data is consistent with the action timing notified by the third time series data.

[0011] Effects of the Invention

[0012] According to the present invention, the user can perform actions appropriately in accordance with target timing. BRIEF DESCRIPTION OF THE DRAWINGS

[0013] Figure 1 This is a block diagram showing an example of a signal processing system according to the first embodiment.

[0014] Figure 2 This is a diagram showing an example of an image of a commander.

[0015] Figure 3 This is a diagram showing an example of the operation of the signal processing system according to the first embodiment.

[0016] Figure 4 This is a block diagram showing an example of a signal processing system according to the second embodiment.

[0017] Figure 5 This is a diagram showing an example of the operation of the signal processing system according to the second embodiment. DETAILED DESCRIPTION

[0018] Hereinafter, a signal processing system, a signal processing device, and a signal processing method according to an embodiment of the present invention will be described with reference to the drawings.

[0019] [First embodiment]

[0020] Figure 1 : is a block diagram showing a signal processing system 1 according to the present embodiment.

[0021] The signal processing system 1 includes a signal processing device 10 and a transmission device 20 .

[0022] The signal processing device 10 and the transmission device 20 are connectable via a network NW1 .

[0023] The transmitting device 20 generates second time series data including at least data indicating the timing of a person's movements based on first time series data including sound data, and transmits the first time series data and the second time series data. The first time series data is, for example, performance data such as a musical performance, image data of gymnastics or dance, etc. In this embodiment, the case where the first time series data is performance data is described.

[0024] In addition, the second time series data is data generated based on the first time series data. The second time series data is, for example, beat data containing the moment information of the beat points of the performance data, and is time series data with a smaller data volume than the first time series data. In this embodiment, the case where the second time series data is beat data is described.

[0025] In addition, the human action timing indicates the timing of the action (e.g., the performance of a musical instrument or singing) performed by the user U1 in coordination with the output signal based on the first time series data (e.g., the sound signal of the performance based on the performance data). The action timing includes, for example, the start timing of the performance (the start of the whole performance, the start of the solo part, etc.), the end timing of the performance, etc.

[0026] The transmission device 20 includes an NW (network) communication unit 21 , an input unit 22 , a display unit 23 , a storage unit 24 , and a control unit 40 .

[0027] The NW communication unit 21 is an interface unit that can be connected to the network NW1 , and communicates with the signal processing device 10 via the network NW1 .

[0028] The input unit 22 is an input device such as a keyboard, a mouse, or a touch panel, and receives various inputs from the user. The input unit 22 is used for various operations when the transmission device 20 is used, such as selection and designation of performance data.

[0029] The display unit 23 is a display device such as a liquid crystal display, and displays a video of performance data, beat data, images of various operation screens, and the like.

[0030] The storage unit 24 stores various information used by the transmission device 20. The storage unit 24 includes a performance data storage unit 241 and a beat data storage unit 242.

[0031] The performance data storage unit 241 stores performance data that is raw data for generating beat data. The performance data is, for example, sound data such as a performance of a musical instrument or a singing voice.

[0032] The beat data storage unit 242 stores beat data indicating the timing of the beats of the performance data. The beat data is data corresponding to the performance data. In addition, the beat data may also include data indicating the strength of the action (for example, the strength of the performance).

[0033] The control unit 40 is a processor including a CPU (Central Processing Unit) and the like, and controls the transmission device 20. The control unit 40 includes a beat data generation unit 41 and a transmission processing unit 42.

[0034] The beat data generation unit 41 generates beat data based on the performance data. The beat data generation unit 41 stores the generated beat data in the beat data storage unit 242. In addition, the beat data can also be regarded as compressed data generated based on the performance data.

[0035] The transmission processing unit 42 transmits the performance data and the beat data to the signal processing device 10. The transmission processing unit 42 transmits the performance data and the beat data to the signal processing device 10 via the NW communication unit 21 and the network, for example. The transmission processing unit 42 includes a performance data transmission unit 421 and a beat data transmission unit 422.

[0036] The performance data transmission unit 421 transmits the performance data to the signal processing device 10 via the NW communication unit 21 .

[0037] The beat data transmitting unit 422 transmits the beat data to the signal processing device 10 via the NW communication unit 21 .

[0038] The signal processing device 10 is a receiving device connected to the network NW1 and capable of communicating with the sending device 20. The signal processing device 10 receives the first time series data and the second time series data from the sending device 20 via the network NW1. The signal processing device 10 generates the third time series data for notifying the action timing based on the second time series data. The signal processing device 10 synchronizes and outputs the output signal based on the first time series data and the output signal based on the third time series data so that the action timing in the first time series data is consistent with the action timing in the third time series data. In this embodiment, the case where the third time series data is image data is described.

[0039] The signal processing device 10 receives the performance data and the beat data, generates video data representing the image of the conductor based on the beat data, and outputs the image of the conductor in conjunction with the sound signal based on the performance data.

[0040] The signal processing device 10 includes a NW communication unit 11 , an input unit 12 , a display unit 13 , a speaker 14 , a storage unit 15 , and a control unit 30 .

[0041] The NW communication unit 11 has the same configuration and function as the NW communication unit 21. The NW communication unit 11 communicates with the transmission device 20 via the network NW1.

[0042] The input unit 12 and the display unit 13 have the same structure and function as the input unit 22 and the display unit 23. The input unit 12 is used for various operations when using the signal processing device 10, such as selecting and designating various settings for generating the image data of the commander. In addition, the display unit 13 displays, for example, the image of the commander, images of various operation screens, and the like.

[0043] The speaker 14 outputs various sound signals used by the signal processing device 10. The speaker 14 outputs, for example, a sound signal based on performance data.

[0044] The storage unit 15 stores various information used by the signal processing device 10. The storage unit 15 includes a performance data storage unit 151, a beat data storage unit 152, a generation information storage unit 153, and a video data storage unit 154.

[0045] The performance data storage section 151 stores the performance data received from the transmission device 20 .

[0046] The beat data storage unit 152 stores the beat data received from the transmission device 20 .

[0047] The generation information storage unit 153 stores data used when generating image data. For example, in order to generate image data representing an image of a conductor, the generation information storage unit 153 stores a learned model that is a learning result constructed in advance using machine learning, various setting information when generating image data, etc. In addition, the learned model is a model that outputs third time series data when the input is the second time series data. In this embodiment, the learned model is a model that outputs image data representing an image of a conductor when the beat data is input.

[0048] The image data storage unit 154 stores image data for notifying the timing of movements of a performance, etc. In addition, the image data of the conductor may include an image of the conductor actually taken, or an animation based on illustrations or computer graphics. In addition, the image data of the conductor may also show the movements of the baton and the hands.

[0049] The control unit 30 is a processor including a CPU, for example, and controls the signal processing device 10. The control unit 30 includes a reception processing unit 31, a video data generating unit 32, and an output processing unit 33.

[0050] The reception processing unit 31 is an example of a reception unit. The reception processing unit 31 receives the performance data and the beat data from the transmission device 20 via the network NW1 and the NW communication unit 11. The reception processing unit 31 includes a performance data reception unit 311 and a beat data reception unit 312.

[0051] The performance data receiving unit 311 receives the performance data transmitted from the transmission device 20 via the network NW1 . The performance data receiving unit 311 stores the received performance data in the performance data storage unit 151 .

[0052] The beat data receiving unit 312 receives the beat data transmitted from the transmitting device 20 via the network NW1 . The beat data receiving unit 312 stores the received beat data in the beat data storage unit 152 .

[0053] The image data generating unit 32 is an example of a generating unit. The image data generating unit 32 generates image data for notifying the conductor of the action timing based on the beat data. The image data generating unit 32 obtains the beat data stored in the beat data storage unit 152, and generates the image data based on the obtained beat data and the data stored in the generating information storage unit 153. Figure 2 The image data of the conductor is the image M1 shown. For example, the image data generating unit 32 generates the image data of the conductor based on the beat data by machine learning using the learning model information stored in the generation information storage unit 153 .

[0054] Furthermore, when the beat data includes data indicating the strength of a movement (for example, the strength of a performance), the video data generating unit 32 generates video data of a conductor including data indicating the strength of a movement.

[0055] The image data generation unit 32 stores the generated image data of the commander in the image data storage unit 154 .

[0056] The output processing unit 33 is an example of a processing unit. The output processing unit 33 synchronizes and outputs a performance sound signal based on the performance data and a conductor's image signal based on the conductor's image data so that the action timing in the performance data and the action timing in the conductor's image data are consistent. The output processing unit 33 outputs the performance sound signal based on the performance data from the speaker 14, and outputs the conductor's image signal from the display unit 13 in synchronization with the sound signal (see FIG. 1 ). Figure 2 Image of the commander M1).

[0057] Next, refer to Figure 3 , the operation of the signal processing system 1 based on this embodiment is described.

[0058] Figure 3 It is a diagram showing an example of the operation of the signal processing system 1 according to the present embodiment.

[0059] First, the transmitting device 20 generates beat data for the performance data (step S101). The beat data generating unit 41 obtains the performance data specified by the operation of the input unit 22 from the performance data storage unit 241. The beat data generating unit 41 generates beat data including the timing information of the beat points based on the performance data. The beat data generating unit 41 stores the generated beat data in the beat data storage unit 242.

[0060] Furthermore, the beat data generating unit 41 may display the performance data and the generated beat data on the display unit 23 so that the user of the transmitting device 20 can check the beat data and adjust or modify the beat data based on the user's instructions.

[0061] Next, the transmitting device 20 transmits the performance data to the signal processing device 10 (step S102). The performance data transmitting unit 421 transmits the performance data acquired from the performance data storage unit 241 to the signal processing device 10. Thus, the performance data receiving unit 311 receives the performance data from the transmitting device 20 and stores the received performance data in the performance data storage unit 151.

[0062] Next, the transmitting device 20 transmits the beat data to the signal processing device 10 (step S103 ). The beat data transmitting unit 422 transmits the beat data to the signal processing device 10 . Thus, the beat data receiving unit 312 receives the beat data from the transmitting device 20 and stores the received beat data in the beat data storage unit 152 .

[0063] Next, the signal processing device 10 receives the selection of the image data (step S104). The image data generating unit 32 receives the selection of the image data (setting designation, etc.) by the user U1 via the input unit 12. For example, the signal processing device 10 may also receive the designation of the conductor's image data corresponding only to the timing of the performance, the designation of the conductor's image data including the indication of the strength of the movement, etc.

[0064] Next, the signal processing device 10 generates image data representing the conductor's image based on the beat data (step S105). The image data generating unit 32 generates image data based on the beat data and the learning result of machine learning stored in the generation information storage unit 153, that is, the learned model, based on the selection of image data (setting designation, etc.) by the user U1. The image data generating unit 32 stores the generated image data in the image data storage unit 154.

[0065] Next, the signal processing device 10 synchronizes the video data with the performance data and outputs them (step S106). The output processing unit 33 outputs a sound signal based on the performance data from the speaker 14. In addition, the output processing unit 33 synchronizes the sound signal with the sound signal. Figure 2 An image signal of the commander based on the image data, such as the image M1, is output from the display unit 13.

[0066] The user U1 performs actions such as playing a musical instrument or singing in coordination with the conductor's image output by the signal processing device 10. Thus, the user U1 can appropriately perform actions such as playing a musical instrument in coordination with the sound signal based on the performance data.

[0067] exist Figure 3 In the example shown, the signal processing device 10 generates image data representing the conductor corresponding to the beat data in step S105, and outputs an image signal of the conductor based on the image data in step S106. This can be achieved in various ways. For example, the signal processing device 10 can also perform step S106 once for the entire beat data. In addition, the signal processing device 10 can also repeatedly implement the processing of step S105 and step S106.

[0068] As described above, the signal processing device 10 according to the present embodiment includes a receiving processing unit 31, an image data generating unit 32, and an output processing unit 33. The receiving processing unit 31 receives first time series data and second time series data, wherein the first time series data includes sound data, and the second time series data is generated based on the first time series data and includes at least data indicating the timing of a person's action. The image data generating unit 32 generates third time series data for notifying the timing of the action based on the second time series data. The output processing unit 33 synchronizes the output signal based on the first time series data with the output signal based on the third time series data and outputs them so that the action timing in the first time series data is consistent with the action timing notified by the third time series data.

[0069] Thus, in the signal processing device 10 according to this embodiment, the output signal based on the third time series data is output synchronously with the output signal based on the first time series data, so the user U1 can appropriately recognize the action timing and can perform the action in accordance with the target timing.

[0070] Furthermore, the signal processing device 10 according to this embodiment receives beat data having a smaller data volume than the first time series data from the transmitting device 20 , thereby being able to reduce the communication data volume and the delay of the output signal based on the third time series data.

[0071] In addition, in the present embodiment, the first time series data is performance data, and the action timing includes the action start timing. In addition, the second time series data is beat data indicating a beat including at least the action start timing. The image data generation unit 32 generates image data as the third time series data based on the beat data. The output processing unit 33 synchronizes and outputs an image signal based on the image data with an output signal based on the performance data.

[0072] Thus, in the signal processing device 10 according to this embodiment, the user U1 can appropriately recognize the start timing of the performance, for example. Therefore, the user U1 can appropriately play a musical instrument, sing, etc. by matching the timing with the output signal based on the performance data.

[0073] Furthermore, in the present embodiment, the image data of the third time-series data is image data of the commander.

[0074] The conductor has the function of conveying the timing of various actions, so by using the conductor's image, it is easier to identify the start timing of the action. Thus, the user U1 can visually confirm the image signal based on the conductor's image data and coordinate with the timing of the performance data received from the transmitting device 20 to appropriately perform the performance of the musical instrument, etc.

[0075] Furthermore, in the present embodiment, the image data generating unit 32 generates image data of the conductor based on the beat data by machine learning.

[0076] Thus, the signal processing device 10 according to the present embodiment can easily create the image data of the conductor based on the beat data.

[0077] In addition, in this embodiment, the reception processing unit 31 receives the second time series data having a smaller data volume than the first time series data. The video data generation unit 32 decodes the second time series data to generate the third time series data.

[0078] Thus, the signal processing device 10 according to the present embodiment receives the second time series data having a smaller data volume than the first time series data, so the communication data volume is reduced, and the third time series data is generated and output before the output of the first time series data. Therefore, the signal based on the third time series data can be outputted in a manner not later than the output of the sound signal based on the first time series data.

[0079] In the present embodiment, the second time series data includes beat data indicating the tempo of the motion and data indicating the strength of the motion. The video data generation unit 32 generates third time series data including data indicating the strength of the motion.

[0080] Thus, in the signal processing device 10 based on this embodiment, the user U1 can recognize the timing of the strength of the action, and can coordinate with the first time series data to perform a more appropriate action including the strength of the action. That is, in the signal processing device 10 based on this embodiment, the user U1 can, for example, coordinate with the output of the performance data to perform a rich performance including the strength of the expression.

[0081] In addition, the signal processing system 1 based on this embodiment includes a sending device 20 and a signal processing device 10, wherein the sending device 20 generates second time series data containing at least data representing the timing of a person's movements based on first time series data containing data, and sends the first time series data and the second time series data.

[0082] Thus, the signal processing system 1 according to the present embodiment achieves the same effect as that of the signal processing device 10 , and the user U1 can appropriately perform operations in accordance with the target timing.

[0083] [Second embodiment]

[0084] Next, a signal processing system 1a and a signal processing device 10a according to a second embodiment will be described with reference to the drawings.

[0085] Figure 41 is a block diagram showing an example of a signal processing system 1 a according to the present embodiment.

[0086] The signal processing system 1a includes a plurality of signal processing devices 10a (10a-1, 10a-2, ...) and a transmission device 20a. The plurality of signal processing devices 10a and the transmission device 20a are connectable via a network NW1.

[0087] In this embodiment, an example is described in which the transmission device 20a transmits beat data to a plurality of signal processing devices 10a, and the plurality of signal processing devices 10a each outputs an image of a conductor based on the beat data, thereby performing an ensemble among a plurality of users.

[0088] In addition, Figure 4 In, with Figure 1 The same structures are marked with the same reference numerals and their descriptions are omitted. Figure 4 In the figure, each of the signal processing device 10a-1, the signal processing device 10a-2, ... has the same structure, and when any signal processing device included in the signal processing system 1a is shown, or when no particular distinction is made, it is described as the signal processing device 10a.

[0089] The basic function of the transmission device 20a is the same as that of the transmission device 20, but it is different from the first embodiment in that the performance data is not transmitted to the signal processing device 10a. In this embodiment, the music of the performance data is shared between the transmission device 20a and the signal processing device 10a.

[0090] The transmission device 20a includes a NW communication unit 21, an input unit 22, a display unit 23, a storage unit 24, and a control unit 40a.

[0091] The basic structure and function of the control unit 40a are the same as those of the control unit 40. The control unit 40a includes a beat data generating unit 41 and a transmission processing unit 42a.

[0092] The transmission processing unit 42a is different from the transmission processing unit 42 in that it includes a beat data transmission unit 422 instead of the performance data transmission unit 421. The transmission processing unit 42a transmits beat data generated based on performance data of a predetermined music piece to the plurality of signal processing devices 10a.

[0093] The basic function of the signal processing device 10a is the same as that of the signal processing device 10a, but the difference is that it does not receive the performance data from the transmission device 20a, and a function for performing an ensemble between a plurality of signal processing devices 10a is added.

[0094] The signal processing device 10 a includes a NW communication unit 11 , an input unit 12 , a display unit 13 , a speaker 14 , a storage unit 15 a , a microphone 16 , and a control unit 30 a .

[0095] In the present embodiment, the user U1 is a user of the signal processing device 10 a - 1 , and the user U2 corresponds to a user of the signal processing device 10 a - 2 .

[0096] The storage unit 15 a is different from the storage unit 15 in that it includes a beat data storage unit 152 , a generation information storage unit 153 , and a video data storage unit 154 , but does not include the performance data storage unit 151 .

[0097] The microphone 16 collects sounds around the signal processing device 10a and outputs a sound signal of the collected sounds. The microphone 16 outputs a sound signal of the sound collected by the user's performance to the control unit 30a during the ensemble.

[0098] The basic structure and function of the control unit 30a are the same as those of the control unit 30. The control unit 30a includes a reception processing unit 31a, a video data generating unit 32, an output processing unit 33a, and an ensemble processing unit 34.

[0099] The reception processing unit 31 a is different from the reception processing unit 31 in that it includes a beat data reception unit 312 instead of the performance data reception unit 311 .

[0100] The output processing unit 33 a outputs a video signal based on the video data of the conductor generated by the video data generating unit 32 so as to visually recognize the timing of the performance.

[0101] The ensemble processing unit 34 receives a performance sound signal from another device (e.g., another signal processing device 10a) via the NW communication unit 11 and the network NW1, and outputs the received performance sound signal to the speaker 14. In addition, the ensemble processing unit 34 transmits the performance sound signal picked up by the microphone 16 to another device (e.g., another signal processing device 10a) via the NW communication unit 11 and the network NW1.

[0102] Next, refer to Figure 5 , the operation of the signal processing system 1a according to this embodiment is described.

[0103] exist Figure 5 In the following, an example of an operation of playing together between the user U1 and the user U2 using the beat data transmitted from the transmission device 20a is described.

[0104] exist Figure 5 In step S201, the processing is the same as Figure 3 The processing of step S101 shown is the same, so its description is omitted here.

[0105] Next, the sending device 20a sends the beat data to each signal processing device 10a (10a-1, 10a-2) (step S202). The beat data sending unit 422 sends the beat data to each signal processing device 10a. As a result, the beat data receiving unit 312 of the signal processing device 10a receives the beat data from the sending device 20a and stores the received beat data in the beat data storage unit 152.

[0106] Next, the processing of step S203 performed by the signal processing device 10a-1 and the processing of step S205 performed by the signal processing device 10a-2 are the same as Figure 3 The processing of step S105 shown is the same, so its description is omitted here.

[0107] In step S204, the signal processing device 10a-1 collects the sound of the performance of the user U1 outputting the conductor's video from the microphone 16. Figure 2 The display unit 13 outputs an image signal of the image data of the conductor stored in the image data storage unit 154, such as the image M1 of the conductor. The ensemble processing unit 34 also acquires a sound signal (performance sound signal) of the performance of the user U1 collected by the microphone 16.

[0108] In step S206, the signal processing device 10a-2 collects the sound of the performance of the user U2 outputting the conductor's image from the microphone 16. Figure 2 The image signal of the image data of the conductor stored in the image data storage unit 154, such as the image M1 of the conductor, is output from the display unit 13. In addition, the ensemble processing unit 34 acquires the performance sound signal of the user U2 picked up by the microphone 16.

[0109] Next, the signal processing device 10a-1 and the signal processing device 10a-2 send each other's performance sound signals to perform ensemble processing (step S207). The ensemble processing unit 34 of the signal processing device 10a-1 sends the performance sound signal of the user U1 to the signal processing device 10a-2, receives the performance sound signal of the user U2 from the signal processing device 10a-2, and causes the speaker 14 of the signal processing device 10a-1 to output the performance sound signal of the user U2. In addition, the ensemble processing unit 34 of the signal processing device 10a-2 sends the performance sound signal of the user U2 to the signal processing device 10a-1, receives the performance sound signal of the user U1 from the signal processing device 10a-1, and causes the speaker 14 of the signal processing device 10a-2 to output the performance sound signal of the user U1.

[0110] As described above, in the signal processing system 1 a according to the present embodiment, it is possible for a plurality of users in remote locations to perform an ensemble.

[0111] As described above, the signal processing device 10a according to the present embodiment includes a receiving processing unit 31a (receiving unit), an image data generating unit 32 (generating unit), and an output processing unit 33a (processing unit). The receiving processing unit 31a receives second time series data (e.g., beat data), which is generated based on performance data (first time series data) of a predetermined music piece, and at least includes data indicating the timing of the performance. The image data generating unit 32 generates third time series data (e.g., image data of a conductor) that can visually identify the timing of the performance based on the second time series data (e.g., beat data) received by the receiving processing unit 31a. The output processing unit 33a outputs an output signal (e.g., image signal) based on the third time series data (e.g., image data of a conductor) so as to visually identify the timing of the performance.

[0112] Thus, in the signal processing device 10a and the signal processing system 1a based on the present embodiment, the user can perform the music appropriately in accordance with the target timing. In addition, by sending the beat data to a plurality of signal processing devices 10a through the sending device 20a, the signal processing device 10a and the signal processing system 1a based on the present embodiment can, for example, coordinate the timing of the performance between a plurality of users in remote places without directly meeting each other, thereby enabling ensemble.

[0113] In addition, the present invention is not limited to the above-described embodiment, and can be modified within a scope not departing from the gist of the present invention.

[0114] The second time series data is described as beat data, but is not limited to this. The second time series data can also be data representing the reference information of the position. As the reference information of the position, the beat data of the moment of the beat (including the one announced in advance), the moment of the measure boundary, the measure number, the beat, the rehearsal mark, the movement number and the movement name, etc. can be listed. In addition, it can also be data representing other information in addition to the reference information of the position. As other information, standard speed (tempo), speed mark, speed change mark, dynamics, dynamic changes, music performance marks, articulation, conductor's expression, conductor's posture information, conductor's lines, the beginning of each part of the phrase, the end of each part of the phrase, the beginning of the solo part, the end of the solo part and the end signal, etc. can be listed.

[0115] In addition, the third time series data may include sound data in addition to the image data. As an example of sound data, the sound of a metronome synchronized with the conductor's image may be cited. In this case, the image data may be an image of the metronome instead of the conductor's image, or may be an image of the metronome in addition to the conductor's image.

[0116] For example, in the above-mentioned embodiments, an example of using the signal processing system 1 (1a) in the performance of a musical instrument is described, but it is not limited to this. The actions performed by the user can be applied to dance, gymnastics, poetry recitation with BGM, radio drama, etc. in addition to the performance of a musical instrument.

[0117] In addition, in each of the above-mentioned embodiments, an example of using image data of a conductor is applied as the third time series data, but the invention is not limited to this. For example, when the first time series data is image data of gymnastics or dance, the third time series data may also be a signal indicating the start timing of the dance or gymnastics, sound data of lines (for example, shouts or signals such as "Big!" and "Slow!"), image data of the instructor (for example, images that can show expressions and postures), etc.

[0118] In the first embodiment, similarly to the second embodiment, the beat data may be transmitted to a plurality of signal processing devices 10 to perform an ensemble, etc. In this case, the signal processing device 10 may include the ensemble processing unit 34 .

[0119] In addition, in the first embodiment, the signal processing device 10 can also set a predetermined offset width when outputting the output signal based on the third time series data synchronously with the output signal based on the first time series data. In addition, the signal processing device 10 can also change the speed of the synchronous output, temporarily stop it, etc.

[0120] Furthermore, in each of the above-mentioned embodiments, the signal processing device 10 (10a) may change the main part in the conductor's image. In this case, the main part may be a bass instrument or a rhythm instrument.

[0121] In addition, each structure of the signal processing system 1 (1a) has a computer system inside. Furthermore, the processing process of each structure of the signal processing system 1 (1a) is stored in a computer-readable recording medium in the form of a program, and the computer reads out the program and executes it to perform the above-mentioned processing. Here, the computer-readable recording medium refers to a magnetic disk, a magneto-optical disk, a CD-ROM, a DVD-ROM, a semiconductor memory, etc. In addition, the computer program can also be distributed to a computer via a communication line, and the computer that receives the distribution executes the program.

[0122] Description of Reference Numerals

[0123] 1, 1a…Signal processing system, 10, 10a, 10a-1, 10a-2…Signal processing device, 11, 21…NW communication unit, 12, 22…Input unit, 13, 23…Display unit, 14…Speaker, 15, 15a, 24…Storage unit, 16…Microphone, 30, 30a, 40, 40a…Control unit, 31, 31a…Reception processing unit, 32…Image data generating unit, 33, 33a…Output processing unit, 34, 35a…Output processing unit, 35, 36a…Output processing unit, 36, 37a…Output processing unit, 37, 38a…Output processing unit, 38, 39a…Output processing unit, 40, 41a…Output processing unit, 41, 42a…Output processing unit, 43, 44a…Output processing unit, 44, 45a…Output processing unit, 46, 47a…Output processing unit, 48, 49a…Output processing unit, 49, 50a…Output processing unit, 51, 52a…Output processing unit, 53, 54a…Output processing unit, 54, 55a…Output processing unit, 56, 57a…Output processing unit, 58, 59a…Output processing unit, 60, 61a…Output processing unit, 62, 63a…Output processing unit, 64, 65a…Output processing unit, 66, 67a…Output processing unit, 68, 69a…Output processing unit, 70, 71a…Output processing unit, 72, 73a…Output processing unit, 74, 7 4…ensemble processing unit, 41…beat data generating unit, 42, 42a…transmission processing unit, 151, 241…performance data storage unit, 152, 242…beat data storage unit, 153…generation information storage unit, 154…image data storage unit, 311…performance data receiving unit, 312…beat data receiving unit, 421…performance data sending unit, 422…beat data sending unit, NW1…network, U1, U2…user.

Claims

1. A signal processing device comprising: a generating unit that generates third time series data for notifying action timing based on second time series data, wherein the second time series data is generated based on the first time series data and includes at least data indicating the action timing of a person, and the first time series data includes sound data; and a processing unit that synchronizes and outputs an output signal based on the first time series data and an output signal based on the third time series data so that the action timing in the first time series data coincides with the action timing notified by the third time series data, The first time series data is performance data, The action timing includes the action start timing, The second time series data is beat data indicating a beat including at least the timing of starting the action. The generating unit generates video data as the third time series data based on the beat data, The processing unit synchronizes and outputs a video signal based on the video data and an output signal based on the performance data.

2. The signal processing device according to claim 1, wherein: The image data is image data of a commander.

3. The signal processing device according to claim 2, wherein: The generating unit generates the image data of the conductor based on the beat data through machine learning.

4. The signal processing device according to any one of claims 1 to 3, wherein: The signal processing device further includes a receiving unit configured to receive the second time series data having a smaller data volume than the first time series data. The generating unit decodes the second time series data to generate the third time series data.

5. The signal processing device according to any one of claims 1 to 3, wherein: The second time series data includes beat data indicating the tempo of an action and data indicating the strength of the action. The generating unit generates the third time-series data including data indicating the strength of the action.

6. A signal processing device comprising: a generating unit that generates third time series data that enables visual recognition of the timing of the performance based on the second time series data, the second time series data being generated based on the first time series data of the performance of a predetermined piece of music and including at least data indicating the timing of the performance; and a processing unit that outputs an output signal based on the third time series data so as to visually recognize the timing of the performance, The first time series data is performance data, The timing of the performance includes the start timing of the performance. The second time series data is beat data indicating a beat including at least the start timing of the performance, The generating unit generates video data as the third time series data based on the beat data, The processing unit synchronizes and outputs a video signal based on the video data and an output signal based on the performance data.

7. A signal processing system comprising a transmitting device and a signal processing device, wherein the transmitting device generates second time series data including at least data indicating a timing of a person's movement based on first time series data including sound data, and transmits the first time series data and the second time series data, The signal processing device comprises: a generating unit that generates third time series data for notifying the action timing based on the second time series data; and a processing unit that synchronizes and outputs an output signal based on the first time series data and an output signal based on the third time series data so that the action timing in the first time series data coincides with the action timing notified by the third time series data, The first time series data is performance data, The action timing includes the action start timing, The second time series data is beat data indicating a beat including at least the timing of starting the action. The generating unit generates video data as the third time series data based on the beat data, The processing unit synchronizes and outputs a video signal based on the video data and an output signal based on the performance data.

8. A signal processing method, comprising: a generating step, wherein the generating unit generates third time series data for notifying action timing based on second time series data, wherein the second time series data is generated based on the first time series data and includes at least data indicating the action timing of the person, and the first time series data includes sound data; and a processing step in which a processing unit synchronizes and outputs an output signal based on the first time series data and an output signal based on the third time series data so that the action timing in the first time series data coincides with the action timing notified by the third time series data, The first time series data is performance data, The action timing includes the action start timing, The second time series data is beat data indicating a beat including at least the timing of starting the action. In the generating step, the generating unit generates video data as the third time series data based on the beat data, In the processing step, the processing unit synchronizes and outputs a video signal based on the video data and an output signal based on the performance data.

Images