Image control device

Abstract

The present invention provides an image control device, comprising: a detection unit for detecting a beat timing of a sound; and a control unit for changing the display form of an image based on the beat timing and change information indicating the change content of the display form of an image displayed by a display device.

Description

Technical Field

[0001] The invention relates to an image control device. Background Art

[0002] In the past, there is a content playback device that corrects the beat position detected from the pulsation of an external musical sound based on input correction information, reads out the lyrics track in synchronization with the external musical sound, and generates lyrics subtitles (for example, Patent Document 1). In addition, there is an image viewing device that sets a threshold value based on the peak value of the high-pitched range and / or low-pitched range of the audio, and switches the display of the image based on the time interval when the audio level exceeds the threshold value (for example, Patent Document 2). In addition, there is a method for mixing a plurality of media including sound and images in synchronization with the beat (for example, Patent Document 3, Patent Document 4). In addition, there is a timing control device for displaying an animation of beating a drum (for example, Patent Document 5).

[0003] Prior art literature

[0004] Patent Literature

[0005] Patent Document 1: Japanese Patent No. 4561735

[0006] Patent Document 2: Japanese Patent No. 4622479

[0007] Patent Document 3: U.S. Patent No. 8269093

[0008] Patent Document 4: U.S. Patent No. 8704069

[0009] Patent Document 5: International Publication No. 2008 / 129837 Summary of the invention

[0010] Problems to be solved by the invention

[0011] Conventionally, when controlling an image in accordance with sound, the user needs to manually control the image while listening to the sound. An object of the present invention is to provide a technique that can change the display content of a display device in accordance with sound.

[0012] Technical means of solving problems

[0013] An embodiment of the present invention is an image control device, comprising:

[0014] a detection unit for detecting a beat timing of a sound; and

[0015] The control unit changes the display form of the video displayed on the display device based on the beat timing and the change information indicating the change content of the display form of the video displayed on the display device.

[0016] The image control device of the present invention may also adopt the following structure, that is, the control unit performs any one of the following as the change of the display mode:

[0017] (i) switching from the image to a different image;

[0018] (ii) the addition of effects to the image;

[0019] (iii) changes to the effects applied to the image; and

[0020] (iv) Removal of the effect applied to the image.

[0021] The image control device of the present invention may also adopt the following structure: when switching from the image to a different image as the change of the display mode,

[0022] The image includes a first image to which the sound is added, and a second image different from the first image,

[0023] The control unit performs a parallel playback process of the first video and the second video, and during playback of one of the first video and the second video, repeatedly plays the other of the first video and the second video that have already been completed.

[0024] The image control device of the present invention may also adopt the following structure: when switching from the image to a different image as the change of the display mode,

[0025] The image includes a first image and a second image different from the first image,

[0026] The control unit performs a process of playing the sound, the first image, and the second image in parallel, and repeats playing the first image or the second image that has been completed during the playing of the sound.

[0027] The image control device of the present invention may be configured such that, when adding an effect to the image as the change in the display mode, the control unit changes a parameter of the image related to the effect.

[0028] The image control device of the present invention may also adopt the following structure, that is, the control unit changes the magnitude of the parameter according to the waveform of a predetermined signal that changes with time. The predetermined signal may be, for example, the sound signal or a signal obtained by mixing the sound signal with a signal different from the sound signal.

[0029] The image control device of the present invention may also adopt the following structure, that is, the control unit changes the display form when the beat timing is detected multiple times in succession. Also, the following structure may be adopted, that is, the control unit changes the display form multiple times within the interval of the beat timing. Also, the following structure may be adopted, that is, the control unit changes the display form at the interval of the last detected beat timing for the non-detection section where the beat timing is not detected.

[0030] Furthermore, the image control device of the present invention may also adopt the following structure, namely: the control unit determines the timing of changing the display form based on the interval of the beat timing before and after the non-detection interval in which the beat timing is not detected. Furthermore, the following structure may also be adopted, namely: the control unit changes the display form for each interval between the first beat timing and the next beat timing, tracing back from the first beat timing, for the interval from the beginning of the music as the sound to the first beat timing.

[0031] A second embodiment of the present invention is an image control device, comprising: a detection unit, detecting a beat timing of a sound;

[0032] and a control unit that controls an image displayed by a display device based on the beat timing. As the control of the image, the control unit, for example, switches the image displayed by the display device to a different image. Alternatively, as the control of the image, the control unit, for example, performs at least one of adding, changing, and canceling an effect of the image displayed by the display device.

[0033] A third embodiment of the present invention is an image control method, comprising: detecting a beat timing of a sound; and

[0034] The image displayed by the display device is controlled based on the beat timing. The image control is, for example, switching the image displayed by the display device to a different image. Alternatively, the image control is at least one of adding an effect to the image displayed by the display device, changing the effect, and canceling the addition of the effect.

[0035] A fourth embodiment of the present invention is an image control device, comprising: a detection unit, which detects the beat timing of the sound according to the image with sound being played;

[0036] The control unit controls the video with sound displayed on the display device based on the beat timing.

[0037] In the first to fourth embodiments of the present invention, the beat timing may be detected in advance or in real time.

[0038] For example, in the first embodiment, the second embodiment, and the fourth embodiment, the following structure may also be adopted, namely: the detection unit performs the following processing: generates timing information responsible for the beat of the sound and intensity data representing the power of the timing from the input sound data; uses the intensity data to calculate the period and phase of the beat of the sound; and detects the beat timing based on the period and phase of the beat.

[0039] Moreover, in the first embodiment, the second embodiment, and the fourth embodiment, the following structure may also be adopted, namely: the detection unit performs a Fourier transform with an attenuation term on the intensity data with respect to multiple beats per minute (BPM), and calculates the BPM when the absolute value of the Fourier transform value reaches the maximum as the period of the beat of the sound.

[0040] Moreover, in the first embodiment, the second embodiment, and the fourth embodiment, the following structure may also be adopted, namely: the detection unit obtains a plurality of wavelet transform values ​​by performing the Fourier transform on the following multiple values, and calculates the phase when the absolute value of the multiple wavelet transform values ​​reaches the maximum as the phase of the beat of the sound, and the multiple values ​​are obtained by multiplying the intensity data by respective window functions staggered in units of 1 / n period of the BPM corresponding to the period of the beat of the sound.

[0041] Moreover, in the first embodiment, the second embodiment, and the fourth embodiment, the following structure may also be adopted, namely: the detection unit obtains a count value representing the period of the beat and the phase of the beat, uses a counter that increments for each sample of the sampling rate to time the count value, and detects the timing when the value of the counter reaches the count value as the beat timing. BRIEF DESCRIPTION OF THE DRAWINGS

[0042] Figure 1 A configuration example of an information processing device (computer) that can function as a video control device is shown.

[0043] Figure 2 A diagram schematically showing an image control device (controller).

[0044] Figure 3A Indicates the timing of the beat, Figure 3B Indicates the amount of change in the parameter that adapts to the beat timing. Figure 3C A sound waveform is shown as an example of a predetermined signal waveform.

[0045] Figure 4This is a diagram for explaining a method for determining the timing for changing the display form of the beat non-detection section.

[0046] Figure 5 This section shows an example of a user interface (UI) of an information processing device.

[0047] Figure 6 Another example of a user interface (UI) of an information processing device is shown.

[0048] Figure 7 This is a flowchart showing a processing example of the image control device (controller).

[0049] Figure 8 A configuration example of a beat timing detection unit is shown.

[0050] Fig. 9 This is a flowchart showing a processing example of the generation unit.

[0051] Fig. 10A An example of a digital signal (also referred to as a music signal) representing a 12-second music piece input to the generator, Fig. 10B Indicated by Fig. 10A An example of Spx data generated from a music signal.

[0052] Fig.11 This is a flowchart showing a processing example of the calculation unit.

[0053] Fig.12 It is a diagram showing an example of a sine wave of Spx data and BPM used for Fourier transform.

[0054] Fig.13 The diagram shows the relationship between the BPM cosine wave and the timing of beat generation.

[0055] Fig.14 This is a flowchart showing an example of a process of detecting a beat generation timing performed by the detection unit.

[0056] Fig.15 The diagram shows the spectrum intensity of 1 times (basic beat) and 2 times beat as examples.

[0057] Fig.16 This is a flowchart showing an example of a calculation process of period data and phase data in the second beat timing detection method.

[0058] Fig.17 This is the loop diagram of formula 3.

[0059] Fig.18 ] represents an example of Spx data and a decaying sine wave with a BPM frequency used for the Fourier transform of Equation 3.

[0060] Fig.19 Schematically represents the calculation of the wavelet transform value w n circuit.

[0061] Fig. 20A , Fig. 20B , Fig. 20C Represents the relationship between Spx data and the periodic Haining window series.

[0062] Fig.21 This is a flowchart showing an example of a calculation process of phase data.

[0063] Fig. 22 This is an illustration of the wavelet transform values.

[0064] Fig.23 This is a graph showing the intensity of beats (IBS) of a piece of music.

[0065] [Explanation of Symbols]

[0066] 1: Information processing device

[0067] 2: Network

[0068] 10: CPU

[0069] 11: ROM

[0070] 12: RAM

[0071] 13: HDD

[0072] 14: Input Device

[0073] 15: Display device

[0074] 16: Communication interface

[0075] 17: Digital to Analog Converter

[0076] 18: Amplifier

[0077] 19: Speaker

[0078] 20: Analog-to-digital converter

[0079] 21: Microphone

[0080] 50: Image control device (controller)

[0081] 51: Playback Department

[0082] 52: Inspection Department

[0083] 53: Control Department

[0084] 54, 55: Video RAM (VRAM)

[0085] 57: Storage device

[0086] 58: Beat information storage unit

[0087] 59: Change information storage

[0088] 100: Beat timing detection unit

[0089] 101: Generation Department

[0090] 102: Buffer

[0091] 103: Computing Department

[0092] 104: Inspection Department

[0093] 105: Playback processing unit DETAILED DESCRIPTION

[0094] As an image control device according to an embodiment, the following image control device will be described.

[0095] [1] An image control device comprising:

[0096] a detection unit for detecting a beat timing of a sound; and

[0097] The control unit changes the display form of the video displayed on the display device based on the beat timing and the change information indicating the change content of the display form of the video displayed on the display device.

[0098] [2] An image control device comprising:

[0099] a detection unit for detecting a beat timing of a sound; and

[0100] The control unit controls the image displayed by the display device based on the beat timing.

[0101] [3] An image control device comprising:

[0102] a detection unit that detects the beat timing of the sound from the video with sound being played; and

[0103] The control unit controls the video with sound displayed on the display device based on the beat timing.

[0104] Here, the sound (audio, also called sound) includes the sound of musical instruments, human voices, music, and various other sounds. The sound can be a played sound or a sound generated in real time. The image (video, movie, video) includes a plurality of static images (frame images) that are switched and displayed at a specified frame rate. The image includes a situation where no difference in motion vectors or pixel values ​​is generated between frame images during a specified period, that is, a situation where a static image is displayed. Therefore, in addition to an image representing the movement of an object or background in the screen, the image also includes an image that continuously displays a static image, or an image that switches and displays multiple static images over time (slideshow). The image can be a played image using information recorded on a medium, or it can be an image that is played in real time using information obtained from a data stream. Moreover, the beat timing can be a beat timing detected in advance, or it can be a beat timing detected in real time for the output sound.

[0105] "Change of display mode" and "control of image" may each include at least one of the following (i) to (iv).

[0106] (i) switching an image displayed by a display device to a different image;

[0107] (ii) adding effects to images displayed by a display device;

[0108] (iii) changes to the effects added to the images displayed by the display device; and

[0109] (iv) Cancellation of the effect applied to the image displayed by the display device.

[0110] When the image displayed by the display device is switched to a different image, the original image and the image of the switching target can be switched alternately, or more than two switching target images can be prepared. "Effect" is a visual effect (image effect) that appears on the display screen. For example, it includes changes in the brightness (on and off) or color (pixel value) of the image, inversion of pixel values, addition of two-dimensional or three-dimensional images (graphics, etc.), changes in the degree of emphasis of edges (contours) in the image, deformation (skewness) of the image, etc. Manage image parameters related to the effect. Parameters related to the effect include image brightness, pixel values, edge emphasis, etc. However, it can also be other than these. By changing the value of the parameter related to the effect, the effect is added or removed. Or, the effect is changed by changing the type of parameter.

[0111] The length of time for changing the display form (controlling the image) (the time for switching to a different image, or the time for adding one or more effects to the image) can be appropriately determined. The start timing of the change in display form (controlling the image) can be set, for example, to the time point of the beat timing, or a specified time point before or after the beat timing. Moreover, the start timing of the change can also be generated at an equal multiple, n times (n is an integer greater than 2), or 1 / n times the interval of the beat timing. The end timing of the change in display form (controlling the image) can be set to a time point after a certain period of time has passed since the start of the change, or a time point when the next beat timing is detected, or a time point at an equal multiple, n times, or 1 / n times the interval of the beat timing.

[0112] The following structure may also be adopted in the image control device, namely: when switching from the image to a different image as a change in the display form of the image (control of the image), it includes a first image to which the sound is attached, and a second image different from the first image, the control unit performs parallel playback processing of the first image and the second image, and during the playback of one of the first image and the second image, repeats the playback of the other of the first image and the second image that has already ended.

[0113] The following structure can also be adopted in the image control device, namely: when switching from the image to a different image as a change in the display form of the image (control of the image), the image includes a first image and a second image different from the first image, and the control unit performs parallel playback processing of the sound, the first image and the second image, and during the playback of the sound, repeats the playback of the first image or the second image that has ended.

[0114] The following structure may also be adopted in the image control device, that is, as the change of the display form (control of the image), the control unit performs at least one of adding, changing, and canceling the effect of the image displayed by the display device. The effect is a visual effect (image effect) that appears in the image. For example, it includes a change in the brightness of the image (including turning on and off) or a change in the color (pixel value) (inversion, etc.), the addition or superposition of a two-dimensional or three-dimensional image or image (signal waveforms such as characters, graphics, signs, icons, sounds, etc.), a change in the emphasis of the edge (contour line) in the image, deformation of the image, the addition of a skew, etc.

[0115] The image control device may also adopt the following structure, namely: when the control unit adds an effect to the image as a change in the display form (control of the image), it changes the parameters of the image related to the effect. The following structure may also be adopted, namely: the control unit changes the size of the parameter according to the waveform of a specified signal that changes over time. The specified signal includes, for example, a sound signal, a signal other than sound, a signal obtained by mixing a sound signal with a signal other than the sound signal, and a signal formed by mixing two or more signals other than the sound signal. The following structure may also be adopted, namely: as the waveform of the signal for changing the size of the parameter, a plurality of waveforms (patterns) are prepared, and the pattern is selected by the user.

[0116] The image control device may also adopt the following structure, that is, the control unit changes the display form (controls the image) when the beat timing is detected multiple times in succession. In addition, the following structure may also be adopted, that is, the control unit changes the display form (controls the image) multiple times within the interval of the beat timing. In this case, the interval of the image control within the interval may not be equal.

[0117] The image control device may also adopt the following structure, that is, the control unit changes the display form (controls the image) with the interval of the beat timing detected last for the non-detection section in which the beat timing is not detected. In addition, the control unit may also adopt the following structure, that is, the control unit determines the timing of changing the display form (controlling the image) based on the interval of the beat timing before and after the non-detection section for the non-detection section in which the beat timing is not detected. As the timing of changing the display form (controlling the image), the average value of the interval of the beat timing before and after may be used, or the value obtained by interpolating between the two may be used.

[0118] The image control device may also adopt the following structure, namely: the control unit controls the image for the section from the beginning of the music as sound to the first beat timing, starting from the first beat timing, each time the interval between the first beat timing and the next beat timing is traced back in time.

[0119] Hereinafter, an image control device and an image control method, a beat sound generation timing generation device and a beat sound generation timing generation method according to an embodiment will be described with reference to the accompanying drawings. The structure of the embodiment is for illustration only and is not limited to the structure of the embodiment.

[0120] [First embodiment]

[0121] <Structure of information processing device>

[0122] Figure 1The following is a structural example of an information processing device (computer) that can operate as an image control device. The information processing device may be a smart media such as a smart phone or a tablet terminal, a general-purpose computer such as a personal computer, or a dedicated computer. The information processing device 1 may or may not be portable. In the following description, the case where the information processing device 1 is a smart phone is described.

[0123] Figure 1 In the embodiment, the information processing device 1 includes a central processing unit (CPU) 10, a read only memory (ROM) 11, a random access memory (RAM) 12, a hard disk drive (HDD) 13, an input device 14, a display device 15, and a communication interface (communication I / F) 16 connected to a bus 3. The information processing device 1 also includes a digital / analog converter (D / A) 17 and an analog / digital converter (A / D) 20 connected to the bus 3. An amplifier (AMP) 18 is connected to the D / A 17, and a speaker 19 is connected to the AMP 18. A microphone (MIC) 21 is connected to the A / D 20.

[0124] The ROM 11 stores various programs executed by the CPU 10 or data used when executing the programs. The RAM 12 is used as a program development area, an operation area of ​​the CPU 10, a data storage area, etc. Furthermore, the RAM 12 is used as a video RAM for drawing frame images of images displayed by the display device 15.

[0125] The HDD 13 stores programs or data used when executing the programs, music data, etc. The music data is, for example, sound (audio) data in a format of a predetermined sound file such as Moving Picture Experts Group Audio Layer III (MP3) or Waveform Audio (WAVE). The sound file format may be other than MP3 or WAVE.

[0126] Furthermore, HDD 13 stores a variety of image files. The format of the image files is not limited. The image files may contain both image data and sound data, or may not contain sound data. Sound files and image files have a schedule (information of the playback time), and the images and sounds are played according to the schedule. Regarding the sounds in the sound files and the sounds in the image files, information (called beat timing information) indicating the timing of the beat generation (called beat timing) is stored in association with the playback time. However, the information indicating the beat timing does not need to be included in the image files or the sound files.

[0127] ROM 11 and RAM 12 are examples of main storage devices, and HDD 13 is an example of auxiliary storage device. The main storage device and the auxiliary storage device are examples of storage devices, storage media, or memories. Hereinafter, ROM 11, RAM 12, and HDD 13 are sometimes collectively described as "storage device 57".

[0128] The input device 14 is a key, a button, a touch screen, etc., and is used to input information (including instructions or commands). The display device 15 is used to display information. The communication I / F 16 is connected to the network 2 and is responsible for communication-related processing. For example, the CPU 10 can download the required music data (music signal) from the network 2 according to the instruction input from the input device 14, and store it in the HDD 13.

[0129] The CPU 10 performs various processes by executing the program. In addition to the above-mentioned processes related to music downloading, the CPU 10 also performs processes related to the playback of sound or images. In addition, the CPU 10 performs the following processes, namely: using the beat timing information in the sound file or the image file, detecting the beat timing; and controlling the image based on the detected beat timing. The control of the image includes switching the image displayed by the display device 15, adding, changing, and canceling the addition of effects to the image.

[0130] For example, when playing music data, the CPU 10 generates digital data (digital signal) representing the sound of the music from the music data read from the HDD 13 to the RAM 12 by executing the program, and supplies the digital data to the D / A 17. The D / A 17 converts the digital data representing the sound into an analog signal by digital-to-analog conversion, and outputs the analog signal to the AMP 18. The analog signal whose amplitude is adjusted by the AMP 18 is output from the speaker 19.

[0131] The MIC 21 collects, for example, singing sounds as accompaniment (backing vocals) with the sound of the music output from the speaker 19. The analog sound signal collected by the MIC 21 is amplified in amplitude by the AMP 18 and amplified from the speaker 19. At this time, the singing sound may be mixed with the music sound, or may be output from different speakers.

[0132] The MIC 21 is also used to collect the sound of a musical instrument (so-called live performance) or the sound of a piece of music played from an external device, amplify the sound (output from the speaker 19), or record it. For example, the signal of the performance sound collected by the MIC 21 is converted into a digital signal by the A / D 20 and sent to the CPU 10. The CPU 10 converts the signal of the performance sound into a format that complies with the format of the sound file to generate a sound file, and stores it in the HDD 13.

[0133] Furthermore, the information processing device 1 may also include a drive device (not shown) for a compact disc (CD), a digital versatile disc (DVD), a blue-ray disc or other optical disc type recording medium. In this case, a digital signal representing the sound of the music read from the optical disc type recording medium using the drive device may be supplied to the D / A 17 to play the music sound or image. In this case, the beat timing of the sound signal of the music read from the optical disc type recording medium may also be detected.

[0134] <Structure of image control device>

[0135] Figure 2 A diagram schematically showing an image control device (controller). Figure 2 In the controller 50, the CPU 10 operates as the controller 50 by executing the program stored in the storage device 57. In the controller 50, the CPU 10 operates as the playback unit 51, the detection unit 52, and the control unit 53. In addition, the storage device 57 is provided with a first video random access memory (VRAM) (VRAM 54), a second VRAM (VRAM 55), a beat information storage unit 58, and a change information storage unit 59.

[0136] Audio data, video data, and video data with audio are input to the playback unit 51. The audio data, video data, and video data with audio that are input may be data related to playback read from a medium (such as a storage device 57 or an optical disk storage device), or may be data related to streaming playback received from the network 2.

[0137] The playback unit 51 performs audio and video playback processing based on the input audio data, video data, and video data with audio. That is, the playback unit 51 generates an audio signal based on the audio data and the audio data contained in the video data with audio and outputs it. The audio signal is transmitted through the D / A 17 and the AMP 18 ( Figure 2The audio signal is processed by the speaker 19 and outputs sound based on the audio signal. The video of the embodiment does not matter whether a frame rate or a difference in pixel value between frames is generated. That is, the video includes a case where only one static image (image) is displayed, or a case where multiple static images (images) are switched over time (slide show).

[0138] The playback unit 51 draws an image (frame image) based on the video data or the video data with sound in the VRAM. The frame image drawn in the VRAM is rewritten at a predetermined frame rate. The frame image signal written in the VRAM is output to the display device 15, and the display device 15 displays the image based on the video signal (i.e., the frame image switched at the frame rate) on the screen of the display device 15.

[0139] Figure 2 In the example shown, a plurality of VRAMs are provided (for example, two VRAMs 54 and 55). However, the number of VRAMs (the number of playbacks that can be processed in parallel) may be greater than 2. The playback unit 51 performs playback processing of a plurality of video data (video files) in parallel, and draws frame images in the corresponding VRAMs. Figure 2 In the example shown, the playback unit 51 draws an image based on the first image data (an example of the first image) on the VRAM 54, and draws an image based on the second image data (an example of the second image) on the VRAM 55. One of the image signal output from the VRAM 54 and the image signal output from the VRAM 55 is output to the display device 15 via the switch 56. The switch 56 is controlled by the control unit 53.

[0140] The detection unit 52 receives the identifier and the data of the playback time of the sound from the playback unit 51. The identifier is, for example, an identifier of the sound data or the image data with sound, but is not limited thereto. The identifier and the data of the playback time of the sound are included in the sound data or the image data with sound input to the playback unit 51.

[0141] The beat information storage unit 58 stores information (called beat information) indicating the timing of beat generation (beat timing) on ​​the time axis of the playback time of the sound or the video with sound in association with the identifier. The detection unit 52 outputs a signal indicating the detection of the beat timing to the control unit 53 when the timing on the playback time coincides with the timing of the beat timing of the beat information. Figure 2 In the illustrated example, the configuration in which the detection unit 52 detects a predetermined beat timing is described. However, the detection unit 52 may receive a sound signal from the playback unit 51 and calculate the beat timing in real time.

[0142] The control unit 53 changes the display mode of the video displayed on the display device 15 (controls the video) based on the beat timing input from the detection unit 52 and the change information stored in the change information storage unit 59 .

[0143] The change information is information related to the change in the display form of the image displayed by the display device 15, including information indicating the type of change in the display form, information indicating the image or effect used for the change, information indicating the parameters of the image used to add the effect, and information indicating the start time of the change.

[0144] The information indicating the type is information indicating whether the change or control is "switching of images", "addition of effects to images", "change of effects added to images", or "cancellation of effects added to images".

[0145] The information indicating the image to be changed is information that determines the image to be switched when the type of change is image switching. One image to be switched may be prepared, or two or more images may be prepared. The information indicating the image to be changed is information that determines the effect to be added, changed, or cancelled when the type of change is addition of an effect, change of an effect, or cancellation of an added effect.

[0146] Information indicating parameters of an image used for an additional effect is information that determines parameters in the image used for the additional effect. For example, the parameters are the brightness of the image or the emphasis of the edge of the image, the degree of inversion of the color (pixel value), the degree of posterization (grayscale change), etc. The parameters are not limited to these. The amount of change in the parameters used for the additional effect can also be predetermined by the user, etc. The change can be an increase, a decrease, or an increase or decrease.

[0147] Furthermore, the magnitude of the parameter may be changed in accordance with the waveform of a predetermined signal during the period of applying the effect. Figure 3A Indicates the timing of the beat, Figure 3B Indicates the amount of parameter change that adapts to the timing of the beat. Figure 3B The waveform shown is an example of a waveform of a predetermined signal, and shows an example of a waveform of a signal other than sound. Figure 3C A waveform of a sound signal is shown as an example of a waveform of a predetermined signal. Figure 3A and Figure 3BIn the example shown, the control unit 53 adapts to each beat opportunity, and increases or decreases the value of the parameter in a mountain shape in a specified interval before and after the beat opportunity. In other intervals, the parameter strength takes a default value (the value when no effect is added). The start timing of the interval of parameter increase or decrease can be at the same time as the beat opportunity, or it can be after the beat opportunity. Moreover, the length of the interval of parameter increase or decrease can be appropriately changed, and even if it is not generated for each beat opportunity, it can be generated once for several beat opportunities, or it can be generated twice or more between beat opportunities.

[0148] Figure 3C An example of a waveform of a signal of sound reproduced by the reproduction unit 51. Figure 3B The signal indicating the strength of the parameter is used to change the size of the parameter according to the amplitude of the sound signal. Figure 3B The signal indicating the strength of the parameter is mixed with the signal of the sound, and the strength of the parameter is changed according to the waveform of the signal (e.g., the amplitude). If set in this way, a certain degree of deviation or irregularity will appear in the strength of the effect with the additional effect, which can improve the visual effect of the image for the listener.

[0149] The information indicating the start timing of the change is information indicating the start timing of the change based on the beat timing. For example, the following settings can be implemented, namely: starting the change (making the change) at equal multiples, n times, or 1 / n times the beat interval. In the case of equal multiples of the beat interval, the display form is changed at each beat interval. In the case of n times the beat interval (n≧2), the display form is changed at intervals of n times the beat interval. Conversely, in the case of 1 / n times the beat interval (n≧2), the display form is changed at intervals of 1 / n times the beat interval. Moreover, the switching or effect can also be continued from the time the beat is detected to the time the next beat is detected. In this case, the first beat timing becomes the start timing of the switching or effect addition, and the next beat timing becomes the release timing of the switching or effect addition.

[0150] Depending on the sound, there may be a section where no beat is generated. For example, in the prelude of a piece of music, there is often no fixed beat. Even in such a section where no beat is detected (non-detection section), the start timing of the change is determined in such a way that a change in the display form is generated. For example, Figure 4 As shown in the figure, it is assumed that there is an interval B where no beats are generated between interval A where the average value of the beat interval is "a" and interval C where the average value of the beat interval is "c". In this case, the control unit 53 obtains the average value "b" of the value "a" and the value "c" for interval B, and generates a change in the display form with the beat interval of the average value "b".

[0151] In addition, the control unit 53 may not necessarily change the display mode for the non-detection interval of the beat. Alternatively, the control unit 53 may change the display mode based on the interval before the non-detection interval ( Figure 4 In the example of interval A), the display form is changed. Alternatively, the following beat interval can be applied, which is based on the beat interval before and after the non-detection interval ( Figure 4 Furthermore, when the prelude portion of the music corresponds to a non-detected interval of the beat, the beat interval initially detected in the music is applied to the prelude portion, and the beat interval is defined in a manner connected to the timing of the initially detected beat.

[0152] Back to Figure 2 If the start timing of the display mode change determined based on the beat timing and the change information arrives, the control unit 53 changes the display mode. For example, assume the following situation: the playback unit 51 uses the VRAM 54 to play the first image and uses the VRAM 55 to play the second image different from the first image. The second image is the image of the switching target.

[0153] At this time, when the timing of changing the display mode comes, the control unit 53 controls the switch 56 according to the change information (type of change: switching of images), and switches the state where the display device 15 is connected to the output of the VRAM 54 to the state where the output of the VRAM 55 is connected to the display device 15 at a predetermined time. As a result, the display mode of the display device 15 is switched from the first image display to the second image display.

[0154] Furthermore, if the type of change in the display form is any one of the addition, change, and removal of effects, the control unit 53 does not control the switch 56 (maintaining the state in which the output of the VRAM 54 is connected to the display device 15), but controls the playback unit 51 to add, change, or remove the effect. For example, the value of the parameter of the image (for example, the emphasis of the edge) that the playback unit 51 refers to when drawing to the VRAM 54 is rewritten to the value when the effect is added or removed. Alternatively, the control unit 53 may remove the first effect (restore the emphasis to the original value) and add a second effect different from the first effect (for example, increasing the brightness value of the image) while the first effect is being applied. Thus, the effect is changed. In addition, sometimes a second effect different from the first effect may be given while the first effect is applied. Moreover, sometimes an effect may be added to the frame image drawn in the VRAM 55 and connected to the display device 15.

[0155] <User Interface>

[0156] Figure 5An example of a user interface (UI) of the information processing device 1 is shown. Figure 5 The figure shows a screen 15A of the display device 15. The display device 15 has a touch screen, and various buttons (operators) displayed on the screen 15A serve as the input device 14. The screen 15A includes a display area 21 for an image (first image), selection buttons 22, 23, and 24 for switching the target image (second image), a slide button 25 for increasing or decreasing the intensity of a parameter, an adjustment unit 26 for the beat interval, and an indicator 27 indicating the playback time of the first image.

[0157] The selection buttons 22, 23, and 24 are image selection buttons that are switched to when changing the display mode. Figure 5 In the example shown, up to three images can be input as the switching target images. Thumbnail images or images representing the switching target images can also be displayed on the selection buttons 22, 23, and 24. The slide button 25 can operate the strength of the parameter by moving the operator (represented by a circle) on the straight line to the left and right. Figure 5 In the example of , the intensity increases as the operator is located to the right, and decreases as the operator is located to the left. The parameter assigned to the slide button 25 can be selected appropriately. In this embodiment, the emphasis of the edge of the image displayed in the display area 21 can be changed by operating the slide button 25.

[0158] The beat interval adjustment unit 26 includes a plurality of buttons for selecting an interval at which the display form is changed with respect to the beat interval detected by the detection unit 52 . Figure 5 In the example shown, the magnification factor with respect to the beat interval can be selected from any one of 1 / 8, 1 / 4, 1 / 2, 1, 2, 3, and 4. Figure 5 Selecting equal magnification (1) will display the selected magnification in reverse.

[0159] The indicator 27 represents a time axis with a straight line, and this time axis represents the playback time of the image shown in the display area 21, and represents the following situation: a pointer (represented by a triangle) representing the current moment of the playback time moves on the time axis. By operating the touch screen to move the pointer in the left and right directions, the image can be fast-forwarded or rewound.

[0160] The user can select or determine the switching target image, parameter strength, and switching execution timing (execution interval) using the selection buttons 22 to 24, the slide button 25, and the adjustment unit 26. The determination result is stored as change information in the change information storage unit 59 (storage device 57).

[0161] Figure 6 Another example of the user interface (UI) of the information processing device 1 is shown. Figure 5The UI (referred to as a first UI) displayed on the illustrated screen 15A is a UI for switching images as a change in display mode. Figure 6 The UI (referred to as a second UI) displayed on the screen 15B shown is a UI in the case where an effect is added as a change in the display form.

[0162] like Figure 6 As shown, in the screen 15B (second UI), the selection buttons 22, 23, 24 for switching the target image are changed to selection buttons 32, 33, 34 for selecting the type of effect to be added to the image. The display area 21, the slide button 25, the adjustment unit 26, the indicator 27 and the first UI ( Figure 5 )same.

[0163] The types of effects can be assigned to selection buttons 32, 33, and 34. For example, pixel value inversion can be assigned to selection button 32, tone separation (grayscale change) can be assigned to selection button 33, and edge emphasis can be assigned to selection button 34. Thumbnail images or videos with effects added can be displayed on selection buttons 32, 33, and 34. In addition, selection buttons 32, 33, and 34 can be used to specify the effect of the change target when changing the effect. For example, selection button 32 can be used to specify inversion as the first effect, and selection button 33 can be used to specify grayscale change as the second effect, and the effect assigned to the object can be changed from inversion to grayscale change when the change is implemented. In addition, Figure 6 The image shown in the display area 21 is to make Figure 5 The image shown in the display area 21 is an inverted image. As an example of another effect, the waveform of the sound may be superimposed on the image displayed in the display area 21 and displayed in real time.

[0164] <Processing Example>

[0165] Figure 7 1 is a diagram showing an example of processing by the controller 50. In S001, the setting of the change information is performed. The user uses the input device 14 and the display device 15 to specify the image (first image) displayed in the display area 21 of the display device 15. In this processing example, the image with sound is specified as the first image. Then, the CPU 10 responds to the user's call-up operation using the input device 14 and displays the screen 15A ( Figure 5 ) or screen 15B( Figure 6 ). The user uses screen 15A or screen 15B to input settings of information (change information) related to changes in the display mode.

[0166] <<When switching images>>

[0167] For example, the user uses the input device 14 to determine the image displayed in the display area 21 of the display device 15. Next, the user uses the selection buttons 22 to 24 of the screen 15A to input the designation of the image to be switched (the second image). Furthermore, the user uses the adjustment unit 26 to designate the timing (implementation interval) of image switching. Information indicating the image to be switched or the implementation interval of switching is stored in the storage unit 59 as change information.

[0168] Then, when the user presses a playback button for the video with sound (e.g., displayed in the display area 21), the playback unit 51 starts the playback process of the video with sound (S002). At this time, in parallel with the playback process of the video with sound, the playback process of the video to be switched is also started, and the drawing to the VRAM 54 and the VRAM 55 is started.

[0169] The control unit 53 controls the switch 56 so that the switch 56 selects the output of the VRAM 54. As a result, the image from the VRAM 54 is displayed in the display area 21 of the display device 15. In addition, in the playback process of the image with sound, the playback unit 51 plays the sound added to the image and outputs the sound from the speaker 19.

[0170] In S003, it is determined whether the end of playback of the video with sound is indicated (for example, whether the playback end button is pressed). Here, if it is determined that the end of playback is indicated, Figure 7 On the other hand, if the end of playback is not indicated, the process proceeds to S004.

[0171] In S004, it is determined whether there is a setting change of the change information. If there is a setting change, the setting change of the change information is reflected (the change information is overwritten). Here, the slide button 25 or the adjustment unit 26 can also be operated during the playback of the image. Moreover, during the playback of the image, the image of the switching target using the selection buttons 22, 23, and 24 or the type of effect using the selection buttons 32, 33, and 34 can be changed. The results of these changes are reflected in the change information stored in the storage unit 59 in S004 (the change information is updated). In addition, instead of S004, the update processing of the change information accompanying the operation of the first or second UI can be performed by interrupt processing at any time.

[0172] As the video with sound is played, the information of the playing time is sent to the detection unit 52. The detection unit 52 compares the information of the playing time with the beat information to detect the beat timing (S005).

[0173] The information indicating the detection of the beat timing is supplied from the detection unit 52 to the control unit 53. Based on the change information (type of change of display form: switching of images) stored in the storage unit 59, the control unit 53 controls the switch 56 using the information indicating the implementation timing (implementation interval) of the change included in the change information, and switches the state of outputting the output of the VRAM 55 (the signal of the switching target image) to the display device 15. As a result, the switching target image is displayed for a predetermined time in the display area 21 instead of the image with sound (S006).

[0174] In S007, a determination is made as to whether the playback time of the image has expired. The determination is made for both the image with sound and the image of the switching target. When it is determined that the playback time of the image has expired, the playback unit 51 repeatedly plays the image whose playback time has expired (S008). For example, when the playback time of the image with sound (first image) is longer than the playback time of the image of the switching target (second image), and the playback time of the second image has expired, if the playback time of the first image has not expired, the second image is repeatedly played. Furthermore, if it is not set to automatically end the playback of the first image due to the expiration of the playback time of the first image, the playback unit 51 performs repeated playback accompanying the expiration of the playback time of the first image (S007, S008). However, as described above, if it is determined that the playback time of the first image has expired, the playback unit 51 does not play the first image. Figure 7 The shown processing ends.

[0175] <<When adding effects>>

[0176] Figure 7 In the process shown, when adding an effect as a change in the display mode, use Figure 6 The selection buttons 32 to 34 of the screen 15B shown in the figure are used to designate the effect of the added object. Furthermore, the strength of the parameter is designated using the slide button 25, and the timing (implementation interval) of the addition of the effect is designated using the adjustment unit 26. Information indicating the designation of the effect of the added object, the strength of the parameter, and the implementation interval of the addition of the effect is stored in the storage unit 59 and is referred to by the control unit 53.

[0177] When the playback unit 51 starts playing the video with sound in response to the user's operation of the play button, the output of the VRAM 54 is connected to the display device 15, and the video is displayed in the display area 21. The control unit 53 changes the parameters related to the drawing of the frame image of the video with sound for the playback unit 51 based on the beat timing detected by the detection unit 52 and the timing of the information indicating the implementation interval of the addition of the effect included in the change information. For example, if the type of the effect is inversion, the control unit 53 changes the pixel value of the image drawn by the playback unit 51 from the pixel value before the addition to the inverted value. As a result, the image with the pixel value inverted is displayed in the display area 21 for a predetermined time. If the type of the effect is grayscale change, the image with the grayscale number reduced is displayed in the display area 21 for a predetermined time. Moreover, if the type of the effect is edge emphasis, the image with the edge emphasized at the edge emphasis set by the slide button 25 is displayed in the display area 21. The type of effect may be changed or the added effect may be canceled at the next implementation timing of the addition of a certain effect. In addition, in the above processing example, the example in which the first image is an image with sound is described, but the first image may also be an image based on image data only, and the sound data is (separate) data independent of the image. In this case, the playback unit 51 plays the sound in synchronization with the playback of the first image.

[0178] In addition, in the embodiment, an example in which the controller 50 is constituted using the CPU 10 and the storage device 57 is shown. The CPU 10 is an example of a processor, and the processing performed by the processor can be performed by multiple CPUs (processors) or by a CPU with a multi-core structure. Moreover, the processing performed by the processor can also be performed by a processor other than the CPU (digital signal processor (DigitalSignal Processor, DSP) or a graphics processor (Graphical Processing Unit, GPU), etc.), an integrated circuit other than the processor (Application Specific Integrated Circuit (Application Specific Integrated Circuit, ASIC) or a Field Programmable Gate Array (Field Programmable Gate Array, FPGA), etc.), or a combination of a processor and an integrated circuit (microprocessor (Micro Processing Unit, MPU), system-on-a-Chip (System-on-a-Chip, SoC), etc.).

[0179] <Functions and Effects of First Embodiment>

[0180] In the embodiment, the display form of the image displayed in the display area 21 of the display device 15 is automatically changed (switching of images or addition of effects, etc.) based on the beat timing of the sound and the change information related to the change of the display form of the image displayed on the display device 15. This can reduce the workload of displaying the status of changing the display form of the image in accordance with the beat of the sound in the display area 21 as in the past.

[0181] [Second Embodiment]

[0182] In the first embodiment, the beat timing for the sound is determined in advance and stored in the storage unit 58 as beat information, and the detection unit 52 uses it to detect the beat timing. In the second embodiment, a beat timing detection method in which the detection unit 52 can detect the beat timing in real time based on the sound signal is described.

[0183] (First method)

[0184] The first beat timing detection method (first method) includes: generating timing information responsible for the beat of the music and a plurality of intensity data indicating the power of the timing from the input music data, calculating the period and phase of the beat of the music using the plurality of intensity data, and detecting the generation timing of the beat sound based on the period and phase of the beat. In the first method, a beat timing detection device is used, and the beat timing detection device includes: a generation unit that generates timing information responsible for the beat of the music and a plurality of intensity data indicating the power of the timing from the input music data; a calculation unit that calculates the period and phase of the beat of the music using the plurality of intensity data; and a detection unit that detects the generation timing of the beat sound based on the period and phase of the beat.

[0185] The beat timing detection device may further include: a playback processing unit configured to perform playback processing of the beat sound according to the generation timing of the beat sound.

[0186] The following structure can also be adopted in the beat timing detection device, namely: the calculation unit determines the beats per minute (BPM) related to the multiple intensity data based on the timing indicated by the multiple intensity data, calculates a period of the BPM as the period of the beat, and calculates the relative position of the timing of the generation of the beat sound in the sine wave representing the BPM as the phase of the beat, the detection unit obtains a count value representing the period of the beat and the phase of the beat, uses a counter that increments for each sample of the sampling rate to time the count value, and detects the timing when the value of the counter reaches the count value as the timing of the generation of the beat sound.

[0187] The following structure can also be adopted in the beat timing detection device, namely: the calculation unit calculates a period of BPM when the value of the following Fourier transform data reaches a maximum as the period of the beat, and the Fourier transform data is obtained by performing Fourier transform on each of the multiple intensity data and each of the multiple BPMs.

[0188] The following structure can also be adopted in the beat timing detection device, namely: when the Fourier transform data is obtained for each of the multiple intensity data and the first BPM among the multiple BPMs, the calculation unit obtains the Fourier transform data related to at least one second BPM having a frequency that is an integer multiple of the frequency of the first BPM, adds the value of the Fourier transform data calculated using the first BPM and the value of the Fourier transform data calculated using the second BPM at a prescribed ratio, and uses the obtained value as the value of the Fourier transform data related to the first BPM.

[0189] The following structure can also be adopted in the beat timing detection device, namely: the generating unit obtains a frame containing a specified number of samples of continuous sound based on the input music data, extracts samples in the frame, performs fast Fourier transform on the extracted samples, and processes the following data at specified intervals, wherein the data represents the sum of the powers of each frequency bandwidth obtained by fast Fourier transform, and on the other hand, the data representing the sum of the powers when the state in which the data representing the sum of the powers representing a value larger than itself does not appear for a specified time is extracted as the intensity data.

[0190] <Structure of the beat timing detection device>

[0191] Figure 1 The information processing device 1 shown can be applied to a beat timing detection device. The CPU 10 executes a program stored in the storage device 57 to perform processing for detecting the beat timing of sound (generating the timing of generating a beat sound).

[0192] Figure 8 1 is a diagram showing a configuration example of a beat timing detection unit. By executing the program, the CPU 10 acts as Figure 8 The beat timing detection unit 100 shown in the figure operates. The beat timing detection unit 100 is included in the detection unit 52 of the first embodiment. Alternatively, it can be replaced with the detection unit 52.

[0193] The beat timing detection unit 100 includes a time sparse data (expressed as "Spx data": equivalent to intensity data) generation unit 101, a buffer 102, a period data and phase data calculation unit 103, and a beat timing detection unit 104. The beat timing is notified to the control unit 53 ( Figure 1). The control unit 53 uses the beat timing for changing the display form (control of the image). The operation and processing of the control unit 53 have been described in the first embodiment, so the re-explanation is omitted. In addition, the beat timing is input to the playback processing unit 105 of the beat as the beat generation timing (output instruction). The playback processing unit 105 performs playback processing of the beat adapted to the generation timing. The operation of the playback processing unit 105 is performed by, for example, the CPU 10. The buffer 102 is provided in, for example, a specified storage area of ​​the RAM 12 or the HDD 13.

[0194] The Spx data generation unit 101 generates and outputs Spx data using digital data representing the sound of the music. The buffer 102 stores at least Spx data (equivalent to a plurality of intensity data) for a predetermined time. In the present embodiment, 6 seconds is exemplified as the predetermined time, but the predetermined time may be longer or shorter than 6 seconds. The calculation unit 103 calculates the period data and phase data of the beat using the set of Spx data for a predetermined time stored in the buffer 102. The generation timing detection unit 104 detects the beat timing using the period data and phase data.

[0195] <Generation of Spx data>

[0196] The generation of Spx data by the generator 101 will be described. A digital signal representing the sound of the music data being played (data sent to the D / A 17 for outputting the sound) is input to the generator 101. The digital signal representing the sound (music signal) may be obtained by the playback processing of the music data stored in the HDD 13, or by A / D conversion of the sound signal collected by the MIC 21.

[0197] The digital data representing the sound is stored in the RAM 12 and is used for processing by the generating unit 101. The digital data representing the sound is a collection of sample (specimen) data (usually the voltage value of the analog signal) collected from the analog signal at a predetermined sampling rate. In this embodiment, as an example, the sampling rate is set to 44100 Hz. However, the sampling rate can be appropriately changed as long as the required Fast Fourier Transform (FFT) resolution can be obtained.

[0198] Fig. 9 Flowchart showing a processing example of the generator 101. Digital data (digital signal) representing the sound of a piece of music, which is sent to the D / A 17 for outputting music (playing), is input to the generator 101. The generator 101 obtains a specified number of samples (referred to as "frames") based on the input digital data (S01). The specified number is 1024 in this embodiment, but it may be more or less than it. The samples are obtained at a specified interval. The specified interval is, for example, 5 ms, but it may be more or less than it.

[0199] In S02, the generator 101 performs a decimation process. That is, the generator 101 performs a 1 / 4 decimation on the 1024 samples to obtain 256 samples. The decimation may be other than the 1 / 4 decimation. In S03, the generator 101 performs a fast Fourier transform (FFT) on the 256 samples, and obtains data (referred to as power data) representing the magnitude of the power per frame unit based on the result of the FFT (power per frequency bandwidth) (S04). In addition, the power is expressed as the square of the amplitude, so the concept of "power" also includes the amplitude.

[0200] The power data is, for example, the sum of powers obtained by performing FFT on 256 samples. However, the power of the corresponding bandwidth of the previous frame may be subtracted from the power of each frequency bandwidth of the current frame, and if the value is positive (power increased), the power value is retained for the sum calculation, and the value that is not so (the value obtained by subtraction is negative (power decreased)) is ignored. The reason for this is that the place where the power increases greatly is likely to be a beat.

[0201] Furthermore, as long as the comparison object is the same as that of other frames, the value used to calculate the sum may be the sum of the powers of the current frame, the sum of the powers whose value is a positive value obtained by subtracting the power of the previous frame from the power of the current frame, or the difference obtained by subtracting the power of the previous frame from the power of the current frame. Furthermore, the difference may be calculated only for frequencies below a specified frequency in the power spectrum obtained by performing FFT. For frequencies above the specified frequency, a low-pass filter may be used to cut off the frequencies.

[0202] The power data is stored in the RAM 12 or the HDD 13 in units of frames. Whenever the power data in units of frames is generated, the generator 101 compares the size of the sum (peak value) of the power, retains the larger value, and discards the smaller value (S05). The generator 101 determines whether a larger sum than the sum retained in S05 has not appeared within a specified time (S06). The specified time is, for example, 100 ms, but may be larger or smaller than 100 ms. In the case where the state in which the data representing the larger sum does not appear continues for a specified time, the generator 101 extracts the data representing the sum of the power as Spx data, and stores (saves) it in the buffer 102 (S07). In this way, the Spx data is data in which the peak value of the digital data representing the musical sound is extracted at intervals of 100 ms, and is data representing the following content, namely: information (timing information) representing the timing of the beat of the music, and the power at the timing. A plurality of Spx data are stored in the buffer 102. The generator 101 repeats the processing of S01 to S06.

[0203] Fig. 10Ais a digital signal of a 12-second music piece input to the generator 101, Fig. 10B Indicated by Fig. 10A An example of Spx data generated from the digital signal of the music shown. Fig. 10B The horizontal axis of the graph shown is time, and the vertical axis is power. Fig. 10A For each Spx data obtained from the digital signal of the music shown, the position of the horizontal axis (time axis) represents the timing, and the length of the vertical line represents the power. When the Spx data is generated at 100 ms intervals, about 10 Spx data are generated per second.

[0204] <Calculation of period data and phase data>

[0205] Fig.11 Flowchart showing an example of processing by the calculation unit 103. In S10, the new Spx data generated by the generation unit 101 arrives at the buffer 102 and is stored. In S11, Spx data (equivalent to a plurality of intensity data) of a specified time in the Spx data stored in the buffer 102 is obtained from the buffer 102. The specified time is, for example, 6 seconds, but as long as the period and phase of the beat can be obtained, it can be longer or shorter than 6 seconds. The subsequent processing of S12 to S16 is the processing performed using the 6-second Spx data obtained in S11. In S12, a Fourier transform corresponding to a specified number (for example, 20) of BPM (Beats Per Minute, tempo) is performed on the 6-second Spx data to calculate the period of the beat (one cycle of BPM) and the phase of the beat (the timing of the generation of the beat sound).

[0206] Specifically, for 6 seconds of Spx data, a predetermined number, for example, 20 frequencies corresponding to BPM (BPM frequency) f = {86, 90, 94, ..., 168} / 60 corresponding to BPM 86 to 168 are used to calculate the product sum of Exp (2πjft) (a sine wave vibrating at the BPM frequency, with the same amplitude regardless of the frequency). That is, Fourier transform is performed. The result of Fourier transform is Fourier transform data c (i) (i = 0, 1, 2, 3, ..., 19).

[0207] Fig.12 : is a diagram showing an example of Spx data and a sine wave with a BPM frequency used for Fourier transform. Fig.12 In the example, the sine wave of BPM72 (indicated by a solid line), the sine wave of BPM88 (indicated by a dotted line), and the sine wave of BPM104 (indicated by a dashed line) are illustrated. The value of the Fourier transform data c(i) is obtained by the following formula 1. In addition, the value of BPM and its number can be changed appropriately.

[0208] [Number 1]

[0209]

[0210] Here, t(k) in Formula 1 is the time position in the past 6 seconds where Spx data exists, and the unit is seconds. k is the index of the Spx data, which is k=1, ..., M (M is the number of Spx data). Moreover, x(t(k)) represents the value of the Spx data at the instant (the size of the peak value). j is the imaginary unit (j 2 =-1). f(i) is the BPM frequency, for example, BPM120 is 2.0 Hz.

[0211] The calculation unit 103 determines the BPM whose absolute value corresponds to the maximum value among c(i)=(c0, c1, c2, c3, ..., c19) as the BPM of the Spx data (beat) (S13). The beat timing of the Spx data is assumed to be 6 seconds. The beat timing indicates a relative position with respect to the timing of generation of a beat that comes periodically.

[0212] Phase value is the complex deflection angle, and c = c re +jc im (c re is the real part and c im is the imaginary part), it is obtained by the following formula 2.

[0213] [Number 2]

[0214]

[0215] By calculating the phase value This makes it possible to know the relative position of the beat generation timing with respect to the sine wave of the BPM, that is, to what extent the beat generation timing is delayed with respect to one cycle of the BPM.

[0216] Fig.13 The diagram shows the relationship between the cosine wave of BPM (real part of EXP(2πjft)) and the timing of beat generation. Fig.13 In the example shown, the number of Spx data is 4 and its BPM is 72. Fig.13 The Spx data shown are values ​​(phases) of c(i) obtained using equation 2, and indicate the timing of beat generation. The intervals between the Spx data are the intervals of beat generation timing. Fig.13 In the example shown, the phase value is calculated by The timing obtained in this way, which is delayed by π / 2 from the cosine wave having the BPM frequency, becomes the timing of the beat generation. The calculation unit 103 uses the number of samples in one cycle of the BPM as the cycle data (S15).

[0217] For example, when the BPM is 104 and the sampling rate is 44100 Hz, the periodic data (number of samples) becomes 44100 [pieces] / (104 / 60)=25442 [pieces]. Furthermore, when the periodic data is 25442 [pieces], when the phase value When Spx is 0.34 [rad], the phase data (number of samples) becomes 25442 [samples] × 0.34 [rad] / 2π [rad] = 1377 [samples]. Next, the calculation unit 103 outputs the period data and the phase data (S16). In addition, the calculation unit 103 repeats the processing of S11 to S16 every time it stores 6 seconds of Spx data. In this way, it is possible to follow the change of the rhythm of the music.

[0218] <Detection of beat timing>

[0219] Fig.14 1 is a flowchart showing an example of the detection process of the beat timing by the detection unit 104. In S21, the detection unit 104 determines whether new period data and phase data are provided from the calculation unit 103. If new period data and phase data are provided, the process proceeds to S22, and if not, the process proceeds to S23.

[0220] In S22, the detection unit 104 uses the new period data and phase data for the detection of the beat generation timing, and discards the old period data and phase data. At this time, since the sample of the frame of the Spx data is given a 100ms delay when the Spx data is generated, the time adjustment (phase adjustment) is performed here so that the music and rhythm being played or broadcast are consistent with the handclap sound described later. Then, the process enters S23.

[0221] In S23, a counter is set using the number of samples of the period data and the number of samples of the phase data. For example, the detection unit 104 has a counter that counts up (increments) for each sample of the sampling rate (intervals of voltage verification of the analog signal that follows the sampling rate), and the count value of the counter is incremented for each sample. Wait until the count value reaches a predetermined value (a value representing the sum of the number of samples (count value) of the phase data and the number of samples (count value) of the period data) from zero (S24).

[0222] If the count value of the counter reaches or exceeds the specified value, the detection unit 104 detects the timing of the generation of the beat sound (beat timing) based on the prediction (S25). The detection unit 104 notifies the control unit 53 of the timing of the generation of the beat and outputs an output instruction of the beat sound (S25). The control unit 53 performs the operation (display form change) described in the first embodiment based on the beat timing. The playback processing unit 105 sends the digital data of the beat sound (e.g., clapping sound) pre-stored in the ROM 11 or the HDD 13 to the D / A 17 according to the output instruction. The digital data is converted into an analog signal by the D / A 17, and the amplitude is amplified by the AMP 18 and then output from the speaker 19. As a result, the clapping sound is output superimposed on the music being played or performed.

[0223] According to the first method, a (previous) music that has been played or performed is input to the generation unit 101, and the generation unit 101 generates Spx data. This Spx data is stored in the buffer 102, and the calculation unit 103 calculates the period and phase of the beat from a plurality of Spx data of a predetermined time (6 seconds), and the detection unit 104 detects the beat timing that is suitable for the music (sound) being played or performed and outputs it. As a result, the control unit 53 can change the display form based on the beat timing. In addition, the playback processing unit 105 can output a clapping sound that matches the rhythm of the music being played or performed. The automatic output of the clapping sound can be performed by a simple algorithm with a small amount of calculation, such as the generation of the Spx data, the calculation of the period and phase of the beat based on Fourier transform data, and the counting of the counter value. As a result, an increase in the load on the execution body (CPU 10) of the processing or an increase in memory resources can be avoided. In addition, since the processing amount is small, a beat sound with no delay (even if there is a delay, a person cannot recognize the delay) relative to the playback sound or performance sound can be output.

[0224] In addition, the processing performed by the beat timing detection unit 100 can be performed by multiple CPUs (processors) or a CPU with a multi-core structure. In addition, the processing performed by the beat timing detection unit 100 can also be performed by a processor other than the CPU 10 (DSP or GPU, etc.), an integrated circuit other than the processor (ASIC or FPGA, etc.), or a combination of a processor and an integrated circuit (MPU, SoC, etc.).

[0225] <Modification of the first method>

[0226] In the above embodiment, an example of using BPM86 to 168 as the BPM for calculating period data is shown. In contrast, the absolute value (spectral intensity) of c(i) can be obtained not only for BPM86 to 168 (each corresponding to the first BPM), but also for BPM172 to 336 or BPM344 to 672 (equivalent to at least one second BPM having a frequency that is an integral multiple of the frequency of the first BPM) that is twice that of BPM86 to 168 (each corresponding to the first BPM). Fig.15 The figure shows the spectrum intensity of 1 times (basic beat) and 2 times the beat. In addition, the spectrum intensity of 1 times, 2 times, and 4 times is added at a predetermined ratio, and the obtained value is used to determine the BPM. For example, the spectrum intensity of BPM91 (an example of the first BPM) and the spectrum intensity of BPM182 and BPM364 (an example of at least one second BPM) are added at a ratio of 0.34:0.33:0.33, and the numerical value is used as the absolute value of c(i) related to BPM91.

[0227] Depending on the music, the power of the BPM corresponding to the subdivided eighth note or sixteenth note may be greater than that of the basic beat represented by the quarter note, so the optimal BPM can be selected by reflecting the power of 2 or 4 times in the strength of the basic beat. In the above example, 2 or 4 times is illustrated as an example of an integer multiple, but the same effect can be obtained by an integer multiple of 3 or 5 or more. The structures shown in the embodiment can be appropriately combined within the range that does not deviate from the purpose.

[0228] (Second method)

[0229] The second beat timing detection method (second method) is described below. In the second beat timing detection method (second method), the beat timing detection unit 100 performs the following processing, namely: generating the timing information responsible for the beat of the sound and the intensity data representing the power of the timing from the input sound data; using the intensity data, calculating the period and phase of the beat of the sound; and detecting the beat timing based on the period and phase of the beat. The second method is an improvement of the first method. The second method is the same as the first method in terms of the generation of Spx data, but is different in terms of the calculation of period data and phase data as follows.

[0230] Fig.16 4 is a flowchart showing an example of calculation processing of period data and phase data in the second beat timing detection method. In S50 , the new Spx data generated by the generator 101 arrives at the buffer 102 .

[0231] In S51, the calculation unit 103 obtains Fourier transform data corresponding to a specified number of BPMs. In the first method, regarding the calculation of period data and phase data, a Fourier transform (for example, 20 to 40) of BPMs (Beats Per Minute: indicating the speed of the beat) is performed on 6 seconds of Spx data. Fig.11 , S12).

[0232] In contrast, in the second method (S51), instead of the Fourier transform used in the first method, a method with an attenuation term U is used. k The Fourier transform formula (Formula 3) is shown below.

[0233] [Number 3]

[0234]

[0235] In formula 3, U represents the attenuation of each sample, which is a number close to 1. U represents the rate of gradually forgetting past data. The interval is infinite. Fig.17 is the loop diagram of equation 3. For the current delay block (Z -1 )61 gives the delayed previous signal f n－1 (m), multiplied by the attenuation term Ue using multiplier 62 -jωm , is added to the current signal f(n) by adder 63. Thus, the Fourier transform value of each sample is obtained.

[0236] The Fourier transform value of Formula 3 can be expressed by the following Formulas 4 and 5.

[0237] [Number 4]

[0238]

[0239]

[0240] For the interval (empty interval) where the Spx data value has not reached but L (L is a positive integer) samples have passed through, Formula 3 ( Fig.17 The Fourier transform value of L samples is obtained by using the following equations 6 and 7. m L The value of can be easily calculated using Formula 7. In addition, f(n) is the value of Spx data, L is the arrival interval of Spx data, U is the attenuation coefficient, ω m is the angular frequency of each sample corresponding to the BPM.

[0241] [Number 5]

[0242]

[0243]

[0244] Fig.18 ] represents an example of Spx data and a decaying sine wave with a BPM frequency used for the Fourier transform of Equation 3. Fig.18 In the example of , the wave with the longest period is the wave of BPM72, the wave with the second longest period is the wave of BPM88, and the wave with the shortest period is the wave of BPM104. In the second method, a predetermined number of BPMs (e.g., 20) are also prepared, and the Fourier transform value using the above formula 3 is obtained for each BPM. The number of BPMs may be more or less than 20.

[0245] In the second method, unlike the first method, it is not necessary to store Spx data for a predetermined period (6 seconds). Therefore, the storage area of ​​the memory (storage device 57) for storing Spx data can be effectively used. In addition, in the first method, the product and sum calculation of a plurality of BPM×Spx data numbers is performed, while in the second method, the calculation of equation 3 is performed for each BPM, thereby significantly reducing the amount of calculation.

[0246] In S52 , the calculation unit 103 obtains a predetermined number (eg, 5) of wavelet transform values ​​corresponding to a predetermined number (eg, 20) of BPMs. Fig.19 Schematic representation of the calculation of the wavelet transform value w n The circuit becomes Fig.17 The circuit for calculating the Fourier transform value shown in the figure is additionally configured with a multiplier 64. The multiplier 64 multiplies the Spx data by a periodic Henning window sequence having a period corresponding to the number of samples of the BPM value. The output of the multiplier 64 is subjected to the Fourier transform of the equation 3 to obtain the wavelet transform value w n The Haining window is an example of a window function, and in addition to the Haining window, a triangular window or a Hamming window can be applied.

[0247] Wavelet transform value w n It is obtained for each BPM at a time when the BPM is shifted by 1 / 5 of the period. That is, a periodic Haining window sequence is prepared that is shifted by 1 / 5 of the period of BPM, and the wavelet transform value {w n}0≦n<5.

[0248] Fig. 20A , Fig. 20B and Fig. 20C Represents the relationship between Spx data and the periodic Haining window series. Fig. 20A The decaying sine wave representing the periodic Haining window sequence at timing 0 at a certain BPM is shown as a thick line, and the decaying sine wave representing the periodic Haining window sequence other than timing 0 is shown as a thin line. Fig. 20B The thick line shows the decaying sine wave of the periodic Haining window train at a certain BPM at timing 1 (1 / 5 period ahead of timing 0), and the thin line shows the decaying sine wave of the periodic Haining window train other than timing 1. Fig. 20C The thick line shows the decaying sine wave of the periodic Haining window train at a certain BPM at timing 2 (1 / 5 period ahead of timing 1), and the thin line shows the decaying sine wave of the periodic Haining window train other than timing 1.

[0249] In S53, the calculation unit 103 determines the BPM corresponding to the Fourier transform value whose absolute value reaches the maximum among the Fourier transform values ​​corresponding to the plurality of BPMs as the BPM of the Spx data (beat) in the same manner as in S13. Furthermore, the calculation unit 103 determines the number of samples of one cycle of the beat of the determined BPM as the period data of the beat (S54).

[0250] In S55, the calculation unit 103 calculates the phase value from the predetermined number of wavelet transform values ​​corresponding to the BPM, and transforms it into a sample value for the periodic data. That is, the calculation unit 103 obtains the wavelet transform value w n When the absolute value of n reaches the maximum ( Fig.21 S551), find the phase value Arg(w corresponding to n n )( Fig.21 The calculation unit 103 converts the phase value into a sample value (phase data) for the periodic data (S55), and outputs the periodic data and the phase data (S56).

[0251] Fig. 22 : is an explanatory diagram of wavelet transform values. Wavelet transform values ​​have complex phase information with time deviation. That is, wavelet transform values ​​have a curve of the Hening window, a curve of the product of the Hening window and the real part (cosine), and a curve of the product of the Hening window and the imaginary part (sine). In this embodiment, by using multiple wavelet transform values ​​with a timing offset of 1 / 5 period (n=5) for one Spx data (beat), the phase of the beat can be detected with high accuracy. In addition, the detection process of the beat timing is different from the first method ( Fig.14 ) are the same, so the description is omitted.

[0252] According to the second method, compared with the first method, the storage capacity and the amount of calculation required for processing can be reduced, and the detection accuracy of the phase (beat timing) can be improved.

[0253] <Calculation of beat intensity>

[0254] The following describes a method for calculating the beat intensity applicable to the first embodiment and the second embodiment. The beat intensity can be calculated by, for example, the controller 50 of the first embodiment or the beat timing detection unit 100 (eg, the calculation unit 103) of the second embodiment.

[0255] For example, the average of the past five beat intervals (number of samples) is set to m. For m, the change rate Qrate of the period is calculated from the new beat interval I using the following formula 8. It can be said that the smaller the value of the change rate Qrate, the more stable the beat.

[0256] [Number 6]

[0257]

[0258] As shown in the following formula 9, if the change rate Qrate is less than 2%, the beat strength is considered to be "strong", and a change of more than 5% can be considered to be "weak". The larger the value of ibt shown in the following formula 9, the more stable the beat. For example, it is conceivable to use the following formula 10 to smooth the ibt and suppress sudden changes in the beat cycle.

[0259] [Number 7]

[0260] ibt

[0261]

[0262] [Current IBS] = [Previous IBS] + 0.1 * (ibt - [Previous IBS]) ··· (10)

[0263] Fig.23 This is a graph showing the IBS of a certain piece of music. The beat is not clear from the beginning of the performance to about 12 seconds. At this time, the IBS starts with a small value and rises to around 1.0 when the beat becomes clear. It can indicate how credible the detected beat timing or beat interval is in music. For example, in the first embodiment, the interval where the IBS is less than a specified value (for example, 0.6) can be judged as an interval where the beat is not detected. The above-described structure is an example and can be appropriately combined.

Claims

1. An image control device, comprising: The detection unit detects the beat timing of the sound; The control unit changes the display mode based on the beat timing and the change information indicating the change content of the display mode of the image displayed by the display device, The display form of the image is changed by switching the display of one of a first image to which the sound is attached and at least one second image different from the first image to the display of the other, the control unit performs parallel playback processing of depicting the first image and the second image in the video memory, and when playback of one of the first image and the second image is completed during playback of the other, the playback of the other image that has ended is repeated.

2. The image control device according to claim 1, wherein: The control unit adds an effect to the image displayed by the display device as a change in the display mode, and changes a parameter of the image related to the effect.

3. The image control device according to claim 2, wherein: The control unit changes the magnitude of the parameter according to the waveform of a predetermined signal that changes with time.

4. The image control device according to claim 3, wherein: The predetermined signal is the audio signal, or a signal obtained by mixing the audio signal with a signal different from the audio signal.

5. The image control device according to claim 1, wherein: The control unit further performs at least one of the following as a change in the display mode: (i) adding an effect to the image displayed on the display device; (ii) changing the effect added to the image displayed on the display device; and (iii) cancelling the effect applied to the image displayed on the display device.

6. The image control device according to claim 1 or 5, wherein: The control unit changes the display mode when the beat timing is detected a plurality of times in succession.

7. The image control device according to claim 1 or 5, wherein: The control unit changes the display mode a plurality of times within the interval of the beat timing.

8. The image control device according to claim 1 or 5, wherein: The control unit changes the display mode at an interval of the last detected beat timing in a non-detection section in which the beat timing is not detected.

9. The image control device according to claim 1 or 5, wherein: The control unit determines the timing of changing the display mode based on an interval between the beat timings before and after the non-detection section in which the beat timing is not detected.

10. The image control device according to claim 1 or 5, wherein: The control unit changes the display mode for a section from the beginning of the music as the sound to a first beat timing, each time the interval between the first beat timing and the next beat timing is traced back from the first beat timing.

11. The image control device according to claim 1, wherein: The detection unit performs the following processing: generates timing information responsible for the beat of the sound and intensity data representing the power of the timing from the input sound data; uses the intensity data to calculate the period and phase of the beat of the sound; and detects the beat timing based on the period and phase of the beat.

12. An image control device, comprising: The detection unit detects the beat timing of the sound; The control unit changes the display mode based on the beat timing and the change information indicating the change content of the display mode of the image displayed by the display device, The display mode of the image is changed by switching the display of one of a first image and at least one second image different from the first image to the other, The control unit performs parallel playback processing of playing the sound, drawing the first image in the first video memory, and drawing the second image in the second video memory, and during the playback of the sound, and during the playback of one of the first image and the second image when the playback of the other image has ended, the first image or the second image that has ended is repeatedly played.

13. The image control device according to claim 12, wherein: The control unit adds an effect to the image displayed by the display device as a change in the display mode, and changes a parameter of the image related to the effect.

14. The image control device according to claim 13, wherein: The control unit changes the magnitude of the parameter according to the waveform of a predetermined signal that changes with time.

15. The image control device according to claim 14, wherein: The predetermined signal is the audio signal, or a signal obtained by mixing the audio signal with a signal different from the audio signal.

16. The image control device according to claim 12, wherein: The control unit further performs at least one of the following as a change in the display mode: (i) adding an effect to the image displayed on the display device; (ii) changing the effect added to the image displayed on the display device; and (iii) cancelling the effect applied to the image displayed on the display device.

17. The image control device according to claim 12 or 14, wherein: The control unit changes the display mode when the beat timing is detected a plurality of times in succession.

18. The image control device according to claim 12 or 14, wherein: The control unit changes the display mode a plurality of times within the interval of the beat timing.

19. The image control device according to claim 12 or 14, wherein: The control unit changes the display mode at an interval of the last detected beat timing in a non-detection section in which the beat timing is not detected.

20. The image control device according to claim 12 or 14, wherein: The control unit determines the timing of changing the display mode based on an interval between the beat timings before and after the non-detection section in which the beat timing is not detected.

21. The image control device according to claim 12 or 14, wherein: The control unit changes the display mode for a section from the beginning of the music as the sound to a first beat timing, each time the interval between the first beat timing and the next beat timing is traced back from the first beat timing.

22. The image control device according to claim 12, wherein: The detection unit performs the following processing: generates timing information responsible for the beat of the sound and intensity data representing the power of the timing from the input sound data; uses the intensity data to calculate the period and phase of the beat of the sound; and detects the beat timing based on the period and phase of the beat.

Images