Information processing device, rhythm detection device and image processing system

Abstract

The present invention provides an information processing device, a rhythm detection device, and an image processing system, which detect the beat of a played music from a musical point of view. The information processing device of the present invention includes: an acquisition element, which acquires samples of a musical sound signal in time sequence; an evaluation element, which includes an adaptive filter, the adaptive filter uses the acquired samples of the musical sound signal as a reference signal, and uses the samples of the musical sound signal acquired a specified time earlier than the samples of the musical sound signal as an input signal; and a rhythm determination element, which sequentially inputs the samples of the musical sound signal to the adaptive filter, and determines the rhythm corresponding to the musical sound based on the filter coefficients of the adaptive filter when the filter coefficients have converged.

Description

Technical Field

[0001] The present invention relates to a technology for detecting the tempo of a musical instrument, and more particularly to an information processing device, a tempo detection device and an image processing system. Background Art

[0002] The following method is well known: a music video is produced by filming an artist or musician singing or playing from multiple angles and connecting the images obtained. In the method, it is necessary to select an appropriate camera according to the plot of the video content to be produced as the music progresses.

[0003] As a related technology, for example, Patent Document 1 discloses a system capable of performing the following control: switching multiple cameras arranged on a stage based on a pre-stored scenario. Also, Patent Document 2 discloses a technology for identifying the position of a speaker based on voices acquired by multiple microphones and switching multiple cameras to capture the speaker.

[0004] [Prior art literature]

[0005] [Patent Document]

[0006] Patent Document 1: Japanese Patent Application Publication No. 2005-026739

[0007] Patent Document 1: Japanese Patent Application Publication No. 2005-295431 Summary of the invention

[0008] [Problems to be solved by the invention]

[0009] According to the system described in Patent Document 1, the camera can be automatically switched according to a preset intention. In the invention, the camera switching timing must correspond to any position in the music. However, in the case of a live performance of the music, it is impossible to make the correspondence in advance. Although there is also a method of switching the camera autonomously, if the camera is switched at a timing unrelated to the music (such as a beat or a measure), it may cause discomfort to the audience.

[0010] The present invention has been made in view of the above-mentioned problems, and an object of the present invention is to provide a technology for detecting the beat of a played music from a musical perspective.

[0011] [Technical means to solve the problem]

[0012] The information processing device of the present invention includes: an acquisition element, which acquires samples of a musical sound signal in chronological order; an evaluation element, which includes an adaptive filter (adaptive filter), which uses the acquired samples of the musical sound signal as a reference signal and uses the samples of the musical sound signal acquired before a specified time as an input signal; and a rhythm determination element, which sequentially inputs the samples of the musical sound signal to the adaptive filter and determines the rhythm corresponding to the musical sound based on the filter coefficients of the adaptive filter when the filter coefficients have converged.

[0013] An adaptive filter is a digital filter that dynamically updates the filter coefficients in a way that minimizes the error between the input signal (the signal of the evaluation object) and the reference signal (the real signal). Music is composed with beats, so a fixed periodicity can be observed in the musical signal. Therefore, when samples of the musical signal with a certain interval are input into the adaptive filter as the reference signal and the input signal, the filter coefficients converge to values ​​corresponding to their periodicity. Therefore, based on the converged filter coefficients, the rhythm corresponding to the musical sound can be evaluated.

[0014] Furthermore, the rhythm determination element may determine whether the predetermined time is a value corresponding to the rhythm of the musical sound based on the converged filter coefficient.

[0015] When the filter coefficient of the adaptive filter is a single coefficient, the converged filter coefficient becomes a value indicating “how well the set predetermined time matches the true rhythm”.

[0016] Furthermore, it may be characterized in that the filter coefficient includes a plurality of coefficients, and the rhythm determination element inputs a plurality of sample groups of the musical sound signals acquired within a predetermined period as the input signal to the adaptive filter.

[0017] When the filter coefficients of the adaptive filter include a plurality of coefficients, a plurality of samples of musical sound signals acquired within a predetermined period can be used as inputs to the adaptive filter. In this case, the timing corresponding to the true rhythm can be known from the converged values ​​of the coefficients.

[0018] Furthermore, it may also be characterized in that the rhythm determination element determines the value corresponding to the time difference between a sample of an input signal and a sample of the musical sound signal serving as a reference signal as the rhythm corresponding to the music piece, and the input signal is a signal multiplied by a coefficient representing a maximum value among the multiple coefficients that have converged.

[0019] When there is a coefficient with the maximum value among the plurality of coefficients, it means that the sample of the evaluation object of the coefficient is most similar to the sample of the reference signal. Therefore, it can be determined that the time difference between these samples is the rhythm corresponding to the musical sound.

[0020] In addition, it may also be characterized in that the filter coefficient includes multiple coefficients, and the rhythm determination element uses a sample group of multiple musical sound signals acquired in a first period and a sample group of multiple musical sound signals acquired in a second period as the input signal, and inputs them into the adaptive filter, and the second period has a length of n times (n is an integer greater than 2) the length of the first period and is continuous with the first period.

[0021] As described above, the sample group evaluated by the adaptive filter does not need to be a sample group included in a single period. By using the sample group included in the first period and the second period as the evaluation object, it is possible to evaluate a period n times the first period. In other words, it is possible to evaluate from a musical perspective.

[0022] In addition, the image processing system of the present invention includes: the information processing device; and a control device for switching a plurality of image sources (video sources) corresponding to a plurality of cameras respectively in a timing according to a rhythm determined by the information processing device.

[0023] By switching the video source (for example, video obtained by photographing the performer using a plurality of cameras) at a timing in accordance with the detected rhythm of the music, it is possible to obtain a video with less discomfort.

[0024] The rhythm detection device of the present invention comprises:

[0025] A music signal acquisition unit, which acquires a music signal; and

[0026] A rhythm detection unit, the rhythm detection unit comprising

[0027] a sampling unit, which samples the musical sound signal at a predetermined frequency and uses the obtained signal as a sample of the musical sound signal;

[0028] a signal delay section that delays samples of the musical sound signal by a predetermined number of time steps; and

[0029] The adaptive filter unit uses the sample of the latest time step as a reference signal and the sample generated before a specified number of time steps as an input signal, and updates the filter coefficient in a way that the error between the input signal and the reference signal is minimized; and

[0030] The rhythm detection unit sequentially inputs samples of the musical sound signal, and determines a rhythm corresponding to the musical sound based on the filter coefficients of the adaptive filter when the values ​​of the filter coefficients converge.

[0031] In addition, the present invention may be an information processing device or an image processing system including at least a part of the above-mentioned elements. Furthermore, it may be a method executed by the above-mentioned information processing device or image processing system. Furthermore, it may be a program for executing the above-mentioned method, or a non-transitory storage medium having the above-mentioned program recorded thereon. The above-mentioned processes or elements may be freely combined and implemented as long as no technical contradictions arise. BRIEF DESCRIPTION OF THE DRAWINGS

[0032] Figure 1 It is the overall structure diagram of the image processing system.

[0033] Figure 2 This is a diagram for explaining the switching of the image source (camera).

[0034] Figure 3 It is a module structure diagram of a rhythm detection device and an image processing device.

[0035] Figure 4 This is a diagram for explaining an overview of an adaptive filter.

[0036] Figure 5 This is a diagram illustrating musical sound signals to be processed in the first embodiment.

[0037] FIG. 6(A) and FIG. 6(B) are diagrams for explaining the adaptive filter according to the first embodiment.

[0038] Figure 7 This is a diagram for explaining the rhythm detection unit 102 according to the first embodiment in detail.

[0039] Figure 8 It is a diagram illustrating the evaluation result of the tempo according to the first embodiment.

[0040] Fig. 9 This is a flowchart of the processing performed by the image processing device in the first embodiment.

[0041] Fig.10 This is a diagram for explaining the rhythm detection unit 102 according to the second embodiment in detail.

[0042] Fig.11 1 is a diagram illustrating musical sound signals to be processed in the second embodiment.

[0043] Fig.12 It is a diagram for explaining the adaptive filter according to the second embodiment.

[0044] Fig.13This is a diagram for explaining the rhythm detection unit 102 according to the third embodiment in detail.

[0045] Fig.14 1 is a diagram illustrating musical sound signals to be processed in the third embodiment.

[0046] [Explanation of Symbols]

[0047] 100: Rhythm detection device

[0048] 101: Music signal acquisition unit

[0049] 102: Rhythm Detection Department

[0050] 200: Image processing device

[0051] 201: Video Recording Department

[0052] 202: Image source selection unit

[0053] 300: Camera

[0054] 400: Microphone DETAILED DESCRIPTION

[0055] (First Embodiment)

[0056] The image processing system of this embodiment is a system that uses multiple cameras to capture the performance of a musical instrument by a performer, rearranges the captured images, and outputs them. The image processing system of this embodiment includes a rhythm detection device 100, an image processing device 200, multiple cameras 300, and a microphone 400.

[0057] Figure 1 A diagram showing the configuration of an image processing system according to this embodiment.

[0058] The cameras 300 are multiple cameras arranged around a performer who plays a musical instrument. The cameras 300 can capture the performer from different angles. The cameras 300 are connected to the image processing device 200 described later, and transmit image signals to the image processing device 200.

[0059] The sound of the performance performed by the performer is collected by the microphone 400, converted into an electrical signal (hereinafter referred to as a musical sound signal), and then sent to the rhythm detection device 100 and the image processing device 200 described later. In addition, in this example, the example of collecting sound by the microphone 400 is given, but in the case where the musical sound signal can be directly obtained from an electronic musical instrument, etc., the microphone 400 can also be replaced by an element that obtains the musical sound signal.

[0060] The rhythm detection device 100 is a device for detecting the rhythm of a piece of music based on an input musical sound signal. In the present embodiment, the so-called rhythm refers to the number of beats per minute, which is expressed by beats per minute (BPM). For example, when the BPM is 120, there are 120 beats per minute. Information related to the detected rhythm is sent to the image processing device 200 as rhythm information.

[0061] The image processing device 200 is a device that acquires and records image signals from a plurality of connected cameras 300, re-arranges the recorded images according to a predetermined rule, and outputs the images. Figure 2 As shown, the recorded multiple video sources are selected in time order, and the selected video sources are combined and output. By selecting multiple video sources in sequence, the same effect as switching multiple cameras 300 can be obtained. In the following description, "switching of video sources" and "switching of cameras" have the same meaning.

[0062] In addition, the image processing device 200 generates the rhythm information obtained from the rhythm detection device 100 at a timing consistent with the rhythm of the music being performed ( Figure 2 ) to switch the camera.

[0063] According to the above configuration, the cameras can be switched at a natural timing synchronized with the music.

[0064] Next, the rhythm detection device 100 will be described in detail.

[0065] The rhythm detection device 100 is a general-purpose computer, which includes a central processing unit (CPU), an auxiliary storage device, and a main storage device. The auxiliary storage device stores a program executed in the CPU and data used by the control program. The processing described below is performed by loading the program stored in the auxiliary storage device into the main storage device and executing it by the CPU.

[0066] Figure 3 The diagram is a diagram for explaining functional blocks of the rhythm detection device 100 and the image processing device 200 .

[0067] The rhythm detection device 100 is composed of two modules, namely, a musical sound signal acquisition unit 101 and a rhythm detection unit 102. These modules may be implemented as program modules to be executed by a CPU.

[0068] The musical sound signal acquisition unit 101 acquires a musical sound signal which is an analog signal from the microphone 400. In the description of this specification, the musical sound signal is a concept including both an analog signal and a digital signal, and the digital signal is obtained by sampling the analog signal.

[0069] The rhythm detection unit 102 samples the analog signal at a predetermined rate and detects the rhythm based on the obtained digital signal. The specific processing content will be described later. The rhythm detection unit 102 generates information (rhythm information) representing the rhythm of the music and sends it to the image processing device 200. In this embodiment, the so-called rhythm information refers to information including the value of the detected rhythm (e.g., 120 BPM).

[0070] Next, the image processing device 200 will be described.

[0071] The image processing device 200 is a general-purpose computer, which includes a CPU (central processing unit), an auxiliary storage device, and a main storage device. The auxiliary storage device stores the program executed in the CPU and the data used by the control program. The processing described below is performed by loading the program stored in the auxiliary storage device into the main storage device and executing it by the CPU.

[0072] The image recording unit 201 obtains image signals and voice signals from the plurality of cameras 300 and the microphone 400, and records them. For example, when there are four cameras, the image recording unit 201 is connected to the camera 300A, the camera 300B, the camera 300C, and the camera 300D, respectively, to obtain and record a plurality of image signals (image streams). Hereinafter, the recorded image signals are also referred to as image sources. In addition, the image recording unit 201 and the camera 300 may be connected by wire or wirelessly.

[0073] The image source selection unit 202 concatenates (edits) a plurality of image signals recorded by the image recording unit 201 using the rhythm information obtained from the rhythm detection unit 102, and generates an output signal. The image source selection may also be performed according to a predetermined rule set in advance. For example, the image source selection unit 202 holds data describing the correspondence between the number of beats from the start of the performance of the music and the camera 300 (hereinafter referred to as image source selection information), so as to select the image source based on the timing of the rhythm information obtained from the rhythm detection device 100, such as Figure 2 As shown, the image source is continuously switched to generate the output signal. In addition, regardless of the image source, a common voice signal is used.

[0074] Before explaining the principle of rhythm detection by the rhythm detection unit 102, an adaptation algorithm will be explained. Since the adaptation algorithm is known, a detailed explanation will be omitted and only an overview will be given below.

[0075] Figure 4 1 is a diagram showing an example of an adaptive filter including a finite impulse response (FIR) filter. An adaptive filter is a filter that dynamically updates filter coefficients in such a way that the error between a reference signal and an input signal is minimized, and the order in which the filter coefficients are updated is called an adaptive algorithm. According to the example, a plurality of filter coefficients h are automatically updated so that the output signal y(n) is close to the reference signal d(n).

[0076] In addition, n represents a time step. When n=0, it represents the latest time step, and when n=-32, it represents 32 steps ago.

[0077] The rhythm detection device 100 of this embodiment calculates the similarity between the sample to be processed and the past samples using the characteristics of the adaptive filter as described above.

[0078] Figure 5 This is a graph showing a musical sound signal in time sequence. The horizontal axis is time (the right side is the past), and the vertical axis is sound pressure. Time is expressed in time steps corresponding to the sampling rate.

[0079] In this embodiment, after sampling the musical sound signal at 44 Hz and 100 Hz, the sampling unit 1021 performs extraction processing on the obtained signal every 512 samples. That is, the duration of one sample is about 11.6 milliseconds. In this example, it is about 371 milliseconds at 32 steps and about 743 milliseconds at 64 steps. This is equal to the interval of the beat at 160 BPM and 80 BPM, respectively.

[0080] The rhythm detection unit 102 detects the rhythm using an adaptive filter. Specifically, the adaptive algorithm is executed using the latest sample x(0) as a reference signal and the samples x(-32) to x(-63) generated 32 steps ago as input signals.

[0081] Fig. 6(A) is a diagram showing an adaptive filter included in the rhythm detection unit 102. As shown in the figure, the adaptive filter included in the rhythm detection unit 102 uses a musical sound signal delayed by only 32 to 63 steps as an input signal and executes an adaptive algorithm.

[0082] In addition, D in the figure represents a delay of one step. In this embodiment, the adaptive filter is composed of 32 stages. That is, the musical sound signal from 32 steps ago to 63 steps ago becomes the evaluation object. In addition, in this specification, multiple groups (32 groups in the example of FIG. 6 (A)) of musical sound signals including delayed musical sound signals are referred to as input signals.

[0083] Figure 7 2 is a block diagram of the rhythm detection unit 102 for realizing the above operation.

[0084] The sampling unit 1021 is a component that samples the musical sound signal at a predetermined sampling rate.

[0085] The musical sound signal queue 1022 is an element (for example, a first in first out (FIFO) memory) that queues musical sound signals for each sample and delays them by a predetermined number of time steps (32 steps in this example).

[0086] The adaptive filter unit 1023 is a device that includes an adaptive filter and executes an adaptive algorithm. With the above-described structure, the latest musical sound signal and the musical sound signal 32 steps ago can be provided to the adaptive filter.

[0087] Here, if there is a beat of the music in the interval from 32 steps ago to 63 steps ago, there should be a sample with the highest similarity to x(0) at a certain step. In other words, it can be inferred that the step at which the sound pressure most similar to x(0) is observed in the interval from 32 steps ago to 63 steps ago is the step corresponding to the beat of the music.

[0088] In the example of FIG6(A), the output signal y can be expressed as formula (1). And the error between the output and the reference signal is shown in formula (2).

[0089] y(0)=h 32 (0)x(-32)+h 33 (0)x(-33)+……+h 47 (0)x(-47)+……+h 63 (0)x(-63)……Formula (1)

[0090] e(0)=x(0)-y(0)...Formula (2)

[0091] The calculated error is fed back to update the filter coefficients at the next time step. The following is the formula for determining the filter coefficients at the next time step. In addition, μ is a response sensitivity value obtained empirically.

[0092] h 32(1) = h 32 (0)+μe(0)x(-32)

[0093] h 33 (1) = h 33 (0)+μe(0)x(-33)

[0094] …

[0095] h 63 (1) = h 63 (0)+μe(0)x(-63)

[0096] If the rhythm detection unit 102 receives musical sound signals in sequence at each time step, the filter coefficient h is updated at any time. 32 (0)~Filter coefficient h 63 (0) and converges to a certain state.

[0097] The adaptive algorithm updates the filter coefficient h in such a way that the error between the input signal and the reference signal is minimized. Therefore, the filter coefficient h corresponding to the step that observes the sound pressure most similar to the sample at x(0) is the largest. For example, if the step corresponding to the beat of the music is always 47 steps ago, the filter coefficient h is updated from h 32 (0) to h 63 Among the filter coefficients up to (0), h 47 (0) is the largest compared to other filter coefficients. That is, by referring to the filter coefficients in a converged state, the position where the beat exists can be estimated.

[0098] The filter coefficient h represents the similarity of the sound pressure at each time step.

[0099] Figure 8 is a graph showing the relationship between the time step and the converged filter coefficient. In this example, it can be seen that the filter coefficient h corresponding to 47 steps ago is 47 (0) is greater than any of the filter coefficients corresponding to the other steps. This means that a sound pressure similar to x(0) has been observed before step 47, so it can be inferred that the illustrated period t1 corresponds to the beat of the music. For example, when t1 is 500 milliseconds, it can be inferred that the tempo of the music is 120 BPM.

[0100] In this example, the evaluation targets are from 32 steps ago to 63 steps ago. Figure 8T1 in becomes the interval for evaluation. The T1 must be a length that includes the envisioned rhythm. As mentioned above, the duration of 0 steps to 32 steps corresponds to 160 BPM, and the duration of 0 steps to 63 steps corresponds to 80 BPM. The rhythm detection device of this embodiment is capable of detecting the rhythm within the interval (i.e., the range of BPM=80 to 160). The interval T1 can be appropriately set according to the envisioned rhythm of the music. The length of T1 can be adjusted according to the sampling rate of the musical sound signal, the length of the musical sound signal arrangement 1022, the number of stages of the adaptive filter, etc.

[0101] The value (t1) determined by the rhythm detection unit 102 is sent to the video processing device 200 (video source selection unit 202), and an output signal is generated. Fig. 9 2 is a flowchart of the processing performed by the video source selection unit 202. The processing is executed at the timing when the recording of the video signal and the musical sound signal is completed and the rhythm detection processing performed by the rhythm detection device 100 is completed.

[0102] First, in step S11, the tempo information is acquired from the tempo detection unit 102. The tempo information may include, in addition to the value indicating the tempo of the music, information on a time stamp, etc. For example, information indicating the timing of starting the performance of the music may be included.

[0103] Next, in step S12, image source selection information is obtained. The image source selection information may be obtained from pre-stored information or from the user.

[0104] Next, in step S13, the position of the beat of the music is calculated. The position of the beat can be calculated by referring to the time stamp included in the tempo information, for example.

[0105] Next, in step S14 , based on the video source selection information and the beat positions calculated in step S13 , the plurality of recorded video sources are combined to generate a new video signal.

[0106] The generated video signal is output in step S15. The video signal may be transmitted to an external device or recorded in a storage medium.

[0107] As described above, according to the image processing system of the first embodiment, the tempo of the music can be calculated based on the periodicity of the waveform of the musical sound signal, and the images can be combined in synchronization with the beat position, so that camera work with little discomfort can be achieved.

[0108] (Second Embodiment)

[0109] In the first embodiment, the rhythm detection device 100 evaluates the periodicity of the musical sound signal included in the period T1. In contrast, the second embodiment evaluates the periodicity of the musical sound signal included in a plurality of different periods (T1 and T2) and combines them to determine the rhythm of the music.

[0110] In the rhythm detection device 100 of the second embodiment, only the configuration of the rhythm detection unit 102 is different from that of the first embodiment. The differences will be described below.

[0111] Fig.10 1 is a block diagram of the rhythm detection unit 102 of the second embodiment. In the second embodiment, the musical sound signal array 1022 has a length of 64 steps, and the sample delayed by 32 steps is supplied to the adaptive filter unit 1023A, and the sample delayed by 64 steps is supplied to the adaptive filter unit 1023B. DS in the figure means 1 / 2 downsampling (decimating 1 / 2 of the sample).

[0112] The adaptive filter unit 1023A is a unit that evaluates the period T1 of the first embodiment, and the adaptive filter unit 1023B is a unit that evaluates the period T2 having a length twice that of the period T1.

[0113] Fig.11 It is a diagram showing a musical sound signal in time sequence according to the present embodiment.

[0114] According to the above configuration, when the latest sample is x(0), adaptive filter section 1023A processes samples within a section of length T1 indicated by reference numeral 1101. Adaptive filter section 1023B processes samples within a section of length T2 indicated by reference numeral 1102.

[0115] The period indicated by T1 is the first period, and the period indicated by T2 is the second period. In this embodiment, the length of T2 is twice the length of T1. As described above, the timing one beat ago and the timing two beats or more ago can be detected separately.

[0116] Fig.12 is a diagram showing an adaptive filter according to this embodiment. Fig.11As shown, in the adaptive filter unit 1023A, the musical sound signal from 32 steps ago to 63 steps ago (32 steps in total) becomes the evaluation object. And, in the adaptive filter unit 1023B, the musical sound signal from 64 steps ago to 126 steps ago (32 steps in total) becomes the evaluation object. The musical sound signal input to the adaptive filter unit 1023B is downsampled by 1 / 2, so the period of the evaluation object is doubled, and the sample interval is 1 / 2.

[0117] exist Fig.12 In the example of , the output signal of the adaptive filter unit 1023A is set to y1, and the output signal can be expressed as formula (3). In addition, the error between the output signal and the reference signal is shown in formula (4).

[0118] y1(0)=h 32 (0)x(-32)+h 33 (0)x(-33)+……+h 63 (0)x(-63)……Formula (3)

[0119] e1(0)=x(0)-y1(0)……Formula (4)

[0120] In addition, when the output signal in the adaptive filter unit 1023B is set to y2, the output signal can be expressed as shown in equation (5). In addition, the error between the output signal and the reference signal is shown in equation (6).

[0121] y2(0)=h 64 (0)x(-64)+h 66 (0)x(-66)+……+h 126 (0)x(-126)……Formula (5)

[0122] e2(0)=x(0)-y2(0)...Formula (6)

[0123] Here, the filter coefficients of equation (5) are replaced by the filter coefficients of adaptive filter unit 1023A. As a result, the output signal is as shown in equation (7).

[0124] y2(0)=h 32 (0)x(-64)+h 33 (0)x(-66)+……+h 64 (0)x(-126)……Formula (7)

[0125] In the second embodiment, if the adaptive filter unit 1023A updates the filter coefficient h 32 ~Filter coefficient h 63 The formula is as follows. The terms in single brackets are independent terms in this embodiment.

[0126] h 32 (1) = h 32 (0)+μ1e1(0)x(-32)+[μ2e2(0)x(-64)]

[0127] h 33 (1) = h 33 (0)+μ1e1(0)x(-33)+[μ2e2(0)x(-66)]

[0128] …

[0129] h 63 (1) = h 63 (0)+μ1e1(0)x(-63)+[μ2e2(0)x(-126)]

[0130] That is, in the second embodiment, when adaptive filter section 1023A updates the filter coefficients, the result of the correction of the filter coefficients by adaptive filter section 1023B is added. In other words, the result of the determination of the similarity by adaptive filter section 1023B for period T2 is added to the result of the determination of the similarity by adaptive filter section 1023A for period T1.

[0131] In the first embodiment, the value of the rhythm is calculated from a mathematical point of view, but sometimes the value of the mathematically calculated rhythm is not necessarily consistent with the value of the musical rhythm (the original rhythm of the music). For example, depending on the structure of the music, there are sometimes a mixture of intervals where the rhythm sounds like 120BPM and intervals where it sounds like 60BPM. For example, when the sounding method of percussion instruments (percussion) changes before and after the interlude, the inferred result of the rhythm sometimes changes even though the rhythm of the music remains unchanged. In the first embodiment, when a piece of music that is mathematically determined to be 120BPM enters an interval that can be determined as 60BPM, sometimes the converged filter coefficients will change again, and accurate rhythm determination cannot be performed. This is because Figure 8 The shape of the peak represented by symbol 801 in the image changes.

[0132] On the other hand, in the second embodiment, the periodicity of the musical sound signal in the period T1 is evaluated by adding the periodicity of the musical sound signal in the period T2 (whose length is twice that of T1). According to the structure, even if the rhythm temporarily sounds halfway, the cumulatively evaluated filter coefficients will not change significantly. That is, the rhythm of the music can be determined not only from a mathematical perspective, but also from a musical perspective.

[0133] (Third Embodiment)

[0134] In the second embodiment, two adaptive filter units are used to evaluate the periodicity of the musical sound signal in the period T1 and the period T2 . In contrast, in the third embodiment, four adaptive filter units are used to evaluate four periods.

[0135] In the rhythm detection device 100 of the third embodiment, only the configuration of the rhythm detection unit 102 is different from that of the second embodiment. The differences will be described below.

[0136] Fig.13 1 is a block diagram of the rhythm detection unit 102 of the third embodiment. In the third embodiment, the input musical sound signal is separated into two systems, and each is passed through a high pass filter (HPF) and a low pass filter (LPF). The musical sound signal in the high pitch range is input to the sampling unit 1021A, and the musical sound signal in the low pitch range is input to the sampling unit 1021B.

[0137] The sampling unit 1021A samples the musical sound signal at 44 Hz and 100 Hz and then decimates the obtained signal every 512 samples, similarly to the sampling unit 1021. The sampling unit 1021B samples the musical sound signal at 44 Hz and 100 Hz and then decimates the obtained signal every 2048 samples.

[0138] The musical sound signal array 1022A and the musical sound signal array 1022B have a length of 64 steps similarly to the second embodiment. Also, the symbol DS is an element for downsampling similarly to the second embodiment.

[0139] In the third embodiment, the musical sound signals processed as described above are input to the four adaptive filter units 1023A to 1023D, respectively.

[0140] Fig.14 1023A to 1023D are diagrams for explaining the ranges of musical sound signals that are processed by adaptive filter sections 1023A to 1023D, respectively.

[0141] The adaptive filter section 1023A is a section that evaluates from 32 steps ago to 63 steps ago (the range of reference numeral 1401), and the adaptive filter section 1023B is a section that evaluates from 64 steps ago to 126 steps ago (the range of reference numeral 1402). These are the same as those of the second embodiment.

[0142] In addition, the adaptive filter unit 1023C becomes a unit that evaluates the bass range from 32 steps ago to 64 steps ago (the range of symbol 1403. Among them, the sampling rate of the bass range is 1 / 4 of the treble range, so 1 step in the bass range is equivalent to 4 steps in the treble range).

[0143] Similarly, the adaptive filter unit 1023D is a unit that evaluates the bass range from 64 steps ago to 126 steps ago (the range of reference numeral 1404).

[0144] In the following description, the musical sound signal in the bass range is denoted as x L (n), and distinguish it from the musical sound signal x(n) in the high-pitch area.

[0145] Here, if the output signal of the adaptive filter unit 1023C is set to y3, the output signal can be expressed as shown in equation (8). In addition, the error between the output signal and the reference signal is shown in equation (9).

[0146] y3(0)=h L32 (0)x L (-32)+h L33 (0)x L (-33)+……+h L63 (0)x L (-63)……Formula (8)

[0147] e3(0)=x L (0)-y3(0)……Formula (9)

[0148] In addition, if the output signal of the adaptive filter unit 1023D is set to y4, the output signal can be expressed as shown in equation (10). In addition, the error between the output signal and the reference signal is shown in equation (11).

[0149] y4(0)=h L64 (0)x L (-64)+h L66 (0)x L (-66)+……+h L126 (0)x L (-126)……Formula (10)

[0150] e4(0)=x L (0)-y4(0)...Formula (11)

[0151] Here, the filter coefficients in equation (8) are replaced with the filter coefficients of the adaptive filter unit 1023A. As a result, the output signal is as shown in equation (12).

[0152] y3(0)=h 32(0)x L (-32)+h 33 (0)x L (-33)+……+h 63 (0)x L (-63)……Formula (12)

[0153] In addition, the filter coefficients of equation (10) are replaced with the filter coefficients of adaptive filter section 1023A. As a result, the output signal is as shown in equation (13).

[0154] y4(0)=h 32 (0)x L (-64)+h 33 (0)x L (-66)+……+h 63 (0)x L (-126)……Formula (13)

[0155] In the third embodiment, if the adaptive filter unit 1023A updates the filter coefficient h 32 ~Filter coefficient h 63 The formula is as follows. The terms in single brackets are independent terms in this embodiment.

[0156] h 32 (1) = h 32 (0)+μ1e1(0)x(-32)+[μ2e2(0)x(-64)+μ3e3(0)x L (-32)+μ4e4(0)x L (-64)]

[0157] h 33 (1) = h 33 (0)+μ1e1(0)x(-33)+[μ2e2(0)x(-66)+μ3e3(0)x L (-33)+μ4e4(0)x L (-66)]

[0158] …

[0159] h 63 (1) = h 63 (0)+μ1e1(0)x(-63)+[μ2e2(0)x(-126)+μ3e3(0)x L (-63)+μ4e4(0)x L (-126)]

[0160] That is, in the third embodiment, when the adaptive filter unit 1023A updates the filter coefficients, the correction results of the filter coefficients by the adaptive filter units 1023B, 1023C, and 1023D are added. In other words, the similarity determination results by the adaptive filter units 1023B, 1023C, and 1023D for the periods T2, T3, and T4 are added to the similarity determination results by the adaptive filter units 1023A for the period T1.

[0161] In the third embodiment, the period T2, the period T3, and the period T4 correspond to the second period. The lengths of the period T2, the period T3, and the period T4 may be n times (n is an integer greater than or equal to 2) the length of the period T1.

[0162] As described above, in the third embodiment, the periodicity of the musical sound signal in the period T1 is evaluated by adding the periodicity of the musical sound signal in the period T2, period T3, and period T4 (whose length is twice, four times, or eight times that of T1). Then, the high-pitched area and the low-pitched area are separated, and the period T1 and period T2 are evaluated using the musical sound signal in the high-pitched area, and the period T3 and period T4 are evaluated using the musical sound signal in the low-pitched area. Generally, there is a tendency that musical instruments in the high-pitched area (such as a hi-hat) sound at a fast tempo, and musical instruments in the low-pitched area (such as a bass drum) sound at a slow tempo, so that rhythm determination with higher accuracy than that of the second embodiment can be performed.

[0163] (Effect)

[0164] The specific effects of the above-exemplified embodiments are disclosed. Table 1 is a table showing the progression of the music to be evaluated. The tempo of the music is set to 120 BPM.

[0165] [Table 1]

[0166] Music composition Instrument composition Prelude 8 beats Hi-hat (1 note per beat) + bass drum (1 note per beat) A section 4 beats Hi-hat (1 note per beat) + bass drum (1 note per beat) + piano (random rhythm) B section 2 beats Hi-hat (1 note every 2 beats) + bass drum (1 note every 2 beats) + piano (random rhythm) Interlude 4 beats Hi-hat (1 note per beat) + bass drum (1 note per beat) + piano (random rhythm) C section 2 beats Hi-hat (1 note every 2 beats) Ending 8 beats Hi-hat (1 note per beat) + bass drum (1 note per beat) + piano (random rhythm)

[0167] In the prelude section, the tempo is estimated to be 120 BPM. Then, when the music moves to the A section or the B section, the rhythm of the percussion instruments changes after the piano is played randomly, so it is difficult to accurately estimate the tempo using mathematical methods.

[0168] On the other hand, in the method of the embodiment, when estimating the rhythm of the A section or the B section, the estimated result of the rhythm of the prelude section is added to perform cumulative evaluation. Therefore, even after the music progresses to the A section, the estimated result of the rhythm of the music will not deviate significantly from 120 BPM.

[0169] In the C section, the percussion instruments play at a tempo equivalent to 60 BPM. However, even in the evaluation of the C section, the estimated results accumulated so far are added, so as a whole, the evaluation result of 120 BPM is still maintained.

[0170] As described above, the rhythm detection unit of the embodiment performs comprehensive evaluation by accumulating the evaluation results of multiple intervals, and thus can detect the rhythm of the music with higher accuracy than when only mathematical methods are used. In other words, the rhythm of the music can be musically evaluated taking into account the progression of the music.

[0171] (Variation Example)

[0172] The above-described embodiment is merely an example, and the present invention can be implemented with appropriate changes without departing from the gist of the present invention. For example, the illustrated embodiments can also be implemented in combination.

[0173] For example, in the second embodiment, a high-pass filter and a low-pass filter may be used to separate the musical sound signal. In this case, the musical sound signal input to the adaptive filter unit corresponding to the faster tempo only needs to contain a frequency component that is higher than the frequency component of the musical sound signal input to the adaptive filter unit corresponding to the slower tempo.

[0174] In addition, in the description of the implementation method, a plurality of sample groups contained in a prescribed period (for example, from 32 steps before to 63 steps before) are input as input signals to the adaptive filter, but the object evaluated by the adaptive filter may also be a single sample. At this time, the filter coefficient becomes a single value as shown in FIG6(B). In this modification, the converged filter coefficient becomes a value indicating "to what extent the delay amplitude (for example, 32 steps) has deviated from the rhythm of the music". It may also be set to determine whether the delay amplitude corresponds to the rhythm of the music based on the converged filter coefficient. For example, it is also possible to obtain a plurality of filter coefficients while changing the delay amplitude, and determine the delay amplitude with the largest filter coefficient as the delay amplitude corresponding to the rhythm of the music.

[0175] Furthermore, in the second embodiment and the third embodiment, a plurality of adaptive filter units are used, but a single adaptive filter unit may be used in a time-division manner.

[0176] In the description of the embodiment, the image signal is recorded by the image recording unit 201, and the image source selection unit 202 combines the recorded multiple images to generate an output signal. On the other hand, the rhythm detection device 100 can also detect the beat in real time. In this case, the rhythm detection device 100 can also generate rhythm information every time a beat is detected, and send it to the image processing device 200 in real time. In this case, the rhythm information becomes information indicating the timing of the occurrence of the beat. In addition, the image processing device 200 can also be set not to record the image, but to select multiple image sources based on the timing of the occurrence of the beat notified in real time, and output the selected image source.

[0177] In the description of the embodiment, an adaptive filter is used as an element for calculating the similarity of musical sound signals (between samples), but as long as data representing the periodicity of the waveform of the musical sound signal can be obtained, elements other than those illustrated may be used to calculate the similarity between samples.

[0178] In the description of the embodiment, the rhythm detection device 100 and the image processing device 200 are provided as separate devices, but hardware in which both are integrated may be used.

[0179] In the description of the embodiment, a system in which a plurality of cameras are switched by the image processing device 200 is exemplified. However, the image processing device 200 may be omitted and the system may be implemented using a single rhythm detection device 100 .

Claims

1. An information processing device, characterized in that: include: An acquisition component, which acquires samples of a musical sound signal in time sequence, wherein the samples of the musical sound signal include current samples of the musical sound signal and past samples of the musical sound signal; An evaluation element, comprising an adaptive filter, the adaptive filter using the acquired sample of the musical sound signal, namely the current sample, as a reference signal, and using the sample of the musical sound signal acquired before a specified time than the current sample of the musical sound signal, namely the past sample, as an input signal; as well as The rhythm determination element sequentially inputs samples of the musical sound signal to the adaptive filter, and when the value of the filter coefficient of the adaptive filter converges to a predetermined value according to the periodicity of the musical sound signal, determines the rhythm corresponding to the musical sound based on the specified time.

2. The information processing device according to claim 1, characterized in that The rhythm determination element determines whether the predetermined time is a value corresponding to the rhythm of the musical sound based on the converged filter coefficient.

3. The information processing device according to claim 1 or 2, wherein The filter coefficients include a plurality of coefficients. The rhythm determination element inputs a plurality of past sample groups of the musical sound signals acquired within a predetermined period as the input signal to the adaptive filter.

4. The information processing device according to claim 1 or 2, wherein The filter coefficients include a plurality of coefficients. The input signal includes a plurality of first input signals and a plurality of second input signals, The rhythm determination element inputs a sample group of multiple musical sound signals acquired in a first period as the first input signal, and a sample group of multiple musical sound signals acquired in a second period as the second input signal, into the adaptive filter, wherein the second period has a length n times that of the first period and is continuous with the first period, wherein n is an integer greater than 2.

5. The information processing device according to claim 3, wherein The rhythm determination element determines a value corresponding to a time difference between a sample of an input signal multiplied by a coefficient indicating a maximum value among the plurality of coefficients that have converged and a sample of the musical sound signal as a reference signal as a rhythm corresponding to the musical sound.

6. An image processing system comprising: The information processing device according to any one of claims 1 to 5; as well as The control device switches the plurality of image sources corresponding to the plurality of cameras respectively at a rhythm determined by the information processing device.

7. A rhythm detection device, comprising: A music signal acquisition unit, which acquires a music signal; and A rhythm detection unit, the rhythm detection unit comprising: a sampling unit, which samples the musical sound signal at a predetermined frequency and uses the obtained signal as a sample of the musical sound signal; a signal delay section that delays the sample of the musical sound signal by a prescribed number of time steps to generate a sample, that is, a past sample, which is earlier than the latest time step by the prescribed number of time steps; and An adaptive filter unit, taking the sample of the latest time step as a reference signal and one of the past samples as an input signal, and updating the filter coefficient in a manner that the error between the input signal and the reference signal is minimized; and The rhythm detection section sequentially inputs samples of the musical sound signal, and when the value of the filter coefficient of the adaptive filter unit converges to a predetermined value according to the periodicity of the musical sound signal, determines the rhythm corresponding to the musical sound based on the specified number of time steps.

8. The rhythm detection device according to claim 7, characterized in that: The rhythm detection unit determines whether the predetermined number of time steps is a value corresponding to the rhythm of the musical sound based on the converged filter coefficient.

9. The rhythm detection device according to claim 7, wherein The filter coefficients include a plurality of coefficients. The rhythm detection unit inputs a plurality of past sample groups of the musical sound signals acquired within a predetermined period as the input signal to the adaptive filter unit.

10. The rhythm detection device according to claim 7, wherein The filter coefficients include a plurality of coefficients. The input signal includes a plurality of first input signals and a plurality of second input signals, The rhythm detection unit inputs a sample group of multiple musical sound signals acquired in a first period as the first input signal, and a sample group of multiple musical sound signals acquired in a second period as the second input signal, into the adaptive filter unit, wherein the second period has a length n times that of the first period and is continuous with the first period, where n is an integer greater than 2.

Images