Karaoke equipment

Abstract

To provide a Karaoke device enabling effect processing corresponding to the emotion of a user using the Karaoke device.SOLUTION: A Karaoke device having an effector for performing effect processing on a voice signal of singing voice includes: an acquisition part for acquiring emotion information indicating the emotion of a user on the basis of a face image signal of the user, which is output from photographing means; and a performance control part for controlling the effector to perform effect processing corresponding to acquired emotion information.SELECTED DRAWING: Figure 2

Description

[Technical Field]

[0001] The present invention relates to a karaoke machine. [Background technology]

[0002] Karaoke machines are equipped with effectors that apply effect processing to the audio signals of singing voices, which causes the singing voices to be amplified with effects such as echo and other effects, and are then emitted from the speakers of the karaoke machine.

[0003] Patent Document 1 discloses a karaoke device that measures the duration of sound contained in an audio signal acquired by a microphone, determines the content of effect processing based on the measured duration of sound, applies the determined effect processing to the audio signal, and outputs the audio signal after the effect processing. [Prior art documents] [Patent documents]

[0004] [Patent Document 1] Japanese Patent Application Laid-Open No. 2011-197235 Summary of the Invention [Problem to be solved by the invention]

[0005] SUMMARY OF THE INVENTION An object of the present invention is to provide a karaoke machine that allows effect processing according to the emotions of the user using the karaoke machine. [Means for solving the problem]

[0006] One invention for achieving the above object is a karaoke device having an effector that performs effect processing on the audio signal of the singing voice, and including an acquisition unit that acquires emotional information indicating the user's emotions based on the user's facial image signal output from a photographing means, and a performance control unit that controls the effector to perform effect processing corresponding to the acquired emotional information. Other features of the present invention will become apparent from the following description and drawings. [Effects of the Invention]

[0007] According to the present invention, effect processing can be performed according to the emotions of the user using the karaoke machine. [Brief explanation of the drawings]

[0008] [Figure 1] 1 is a diagram showing a karaoke device according to an embodiment; [Figure 2] 1 is a diagram showing a karaoke machine main body according to an embodiment; [Figure 3] FIG. 10 is a diagram showing the correspondence between emotion types and effects according to the embodiment. [Figure 4] 4 is a flowchart showing the process of the karaoke device according to the embodiment. [Figure 5] FIG. 10 is a diagram showing the correspondence between the intensity of emotion and an effect according to Modification 2. DETAILED DESCRIPTION OF THE INVENTION

[0009] <Embodiment> A karaoke device according to an embodiment will be described with reference to Figures 1 to 4. In the following description, among users of the karaoke device, users who sing karaoke will be referred to as "singers," and users other than singers (users who listen to the karaoke singing) will be referred to as "audience."

[0010] ==Karaoke Equipment== The karaoke device K is a device for performing karaoke songs and for singers to sing karaoke. As shown in Fig. 1, the karaoke device K includes a karaoke main unit 10, a speaker 20, a display device 30, a microphone 40, a remote control device 50, and a photographing means 60.

[0011] The karaoke machine main unit 10 performs various controls related to karaoke performance and singing, such as controlling the karaoke performance of the selected song, controlling the display of lyrics and background images, and processing audio signals input through the microphone 40. The speaker 20 is configured to emit karaoke performance sounds and singing voices based on signals from the karaoke machine main unit 10. The display device 30 is configured to display videos and images on a screen based on signals from the karaoke machine main unit 10. The microphone 40 is configured to convert the singing voice of the singer singing karaoke into an analog audio signal and input it to the karaoke machine main unit 10. The remote control device 50 is a device for performing various operations on the karaoke machine main unit 10. The photographing means 60 is a camera for photographing the user of the karaoke machine K. Multiple photographing means 60 may be provided.

[0012] 2, the karaoke machine 10 according to this embodiment includes a storage unit 10a, a communication unit 10b, an input unit 10c, a performance unit 10d, and a control unit 10e. Each component is connected to a bus B via an interface (not shown).

[0013] [Storage means] The storage means 10a is a large-capacity storage device that stores various types of data. The storage means 10a stores song data. Song data is assigned song identification information. Song identification information is information unique to each song, such as a song ID for identifying the song. Song data includes karaoke performance data, reference data, section information, etc. Karaoke performance data is MIDI format data that is the source of karaoke performance sounds. Reference data is data that indicates the main melody of the song performed karaoke. Section information indicates a performance section. A performance section is a section in which karaoke performance is performed. A performance section includes a singing section and a non-singing section. A singing section is a section in which lyrics to be sung are set in a certain song (for example, the A-melody, B-melody, and chorus of the first verse). A non-singing section is a section in which lyrics to be sung are set in a certain song, such as an introduction, interlude, or postlude.

[0014] The storage means 20 also stores, for each song, background image data corresponding to the background image displayed during karaoke performance, and song attribute information (song title, singer name, genre, etc.).

[0015] [Communication means / input means] The communication means 10b provides an interface for communicating with the remote control device 50. The input means 10c is configured to allow the user to input various instructions. The input means 10c is a button or the like provided on the karaoke main unit 10. Alternatively, the remote control device 50 may function as the input means 10c.

[0016] [Means of performance] Under the control of the control means 10e, the performance means 10d performs karaoke performance of the music piece and processes the audio signal of the singing voice input through the microphone 40. In this embodiment, the performance means 10d includes a sound source 101d, an effector 102d, a mixer 103d, and an amplifier 104d.

[0017] The sound source 101d generates a performance sound signal based on the karaoke performance data and outputs the generated performance sound signal to the mixer 103d.

[0018] The effector 102d applies effect processing to the audio signal of the singing voice input through the microphone 40. The effector 102d outputs the audio signal that has undergone the effect processing to the mixer 103d.

[0019] Effect processing is a process that changes the singing voice by applying processing such as changing the volume level or frequency characteristics or delaying the audio signal depending on the type of effect. Types of effects include distortion, delay, chorus, echo, reverb, phase shifter, etc. Multiple effectors 102d may be provided depending on the type of effect, or a single effector such as a multi-effector may be provided.

[0020] The mixer 103d mixes the input audio signal and the performance sound signal. The amplifier 104d amplifies the signal output from the mixer 103d, and outputs the karaoke performance sound and singing voice from the speaker 20.

[0021] [Control means] The control means 10e performs various controls in the karaoke device K. The control means 10e includes a CPU and a memory (neither of which is shown). The CPU executes programs stored in the memory to realize various functions.

[0022] In this embodiment, the CPU executes a program stored in the memory, and the control means 10e functions as the acquisition unit 100 and the performance control unit 200.

[0023] (Acquisition Department) The acquisition unit 100 acquires emotion information based on the face image signal of the user output from the image capture means 60.

[0024] When the karaoke performance starts, the acquisition unit 100 controls the photographing means 60 to photograph the user's face. The photographing means 60 photographs the user's face and generates facial image data. The photographing means 60 outputs a facial image signal corresponding to the facial image data to the karaoke unit 10.

[0025] Emotional information indicates the user's emotions. The acquisition unit 100 acquires emotional information by processing a facial image signal using known technology. Emotional information can be acquired at a predetermined timing and a predetermined number of times. For example, the acquisition unit 100 can acquire emotional information only once at a certain timing during a karaoke performance of a song. The certain timing can be, for example, one minute after the start of the karaoke performance or during the karaoke performance corresponding to the singing section of the first chorus. The acquisition unit 100 outputs the acquired emotional information to the performance control unit 200.

[0026] Alternatively, the acquisition unit 100 may acquire emotion information multiple times during a karaoke performance of a song, such as at predetermined intervals (for example, every 40 seconds after the start of the karaoke performance) or at the start of each singing section. In this case, the acquisition unit 100 outputs emotion information to the performance control unit 200 each time it acquires it.

[0027] In this embodiment, the acquiring section 100 acquires emotion types (for example, "joy," "sadness," and "anger") as emotion information. The emotion types acquired by the acquiring section 100 are set in advance.

[0028] To acquire the emotion type, a deep learning framework that predicts emotions in real time from facial expressions in a video ("Estimation of continuous valence and arousal levels from faces in naturalistic conditions" Samsung AI, Imperial College London) or software that uses AI such as "Facial Expression Estimation AI" (User Local Co., Ltd.) can be used. The acquisition unit 100 can acquire the emotion type expressed in the facial image data corresponding to the facial image signal by inputting the facial image signal output from the imaging means 60 into the software.

[0029] In this embodiment, the acquisition section 100 also acquires emotion information that indicates the emotion of the singer.

[0030] For example, if there is only one user of the karaoke device K, that user is the singer. The photographing means 60 photographs the face of the singer and generates facial image data. The acquiring unit 100 acquires emotion information indicating the emotion of the singer based on the facial image signal of the singer.

[0031] On the other hand, when there are multiple users of the karaoke device K, there will be a singer and an audience. In this case, the imaging means 60 generates facial image data of the singer and the audience and outputs corresponding facial image signals to the karaoke main unit 10. The acquisition unit 100 identifies the singer's facial image signal by analyzing the facial image signal output from the imaging means 60. The identification of the facial image signal can be achieved using known techniques. For example, by analyzing the facial image data corresponding to the facial image signal, the acquisition unit 100 can identify the user holding the microphone 40 from among multiple users as the "singer" and the other users as the "audience." The acquisition unit 100 acquires emotion information indicating the singer's emotion based on the identified singer's facial image signal.

[0032] (Performance control unit) The performance control unit 200 controls the performance means 10d to perform a karaoke performance of the music piece.

[0033] Specifically, the performance control unit 200 acquires the corresponding karaoke performance data from the storage unit 10a based on the song ID of the song registered in the reservation queue. The performance control unit 200 then outputs the acquired karaoke performance data to the performance unit 10d, causing the karaoke performance to be performed. The singer sings along with the karaoke performance.

[0034] The performance control unit 200 controls the effector 102d to execute a preset normal effect process on the audio signal of the singing voice input through the microphone 40. Note that the effector 102d may be set not to execute the effect process normally.

[0035] Here, the performance control unit 200 according to this embodiment controls the effector 102d to execute effect processing corresponding to the acquired emotion information.

[0036] Specifically, the performance control unit 200 identifies an effect that corresponds to the emotion information acquired by the acquisition unit 100, and controls the effector 102d to execute processing that corresponds to the identified effect.

[0037] The effect can be specified by referring to a table stored in advance in the storage means 10a, for example.

[0038] Figure 3 is a table showing the relationship between emotion types and effects. In this example, the "Happy" type is linked to the "Deep Echo" effect, the "Anger" type is linked to the "Distortion" effect, and the "Sadness" type is linked to the "Phase Shifter" effect.

[0039] "Deep Echo" is an effect that has a longer reverberation time and more repeats than a normal echo, thereby creating a splendor that suits the emotion of "joy." "Distortion" is an effect that severely distorts the singing voice, thereby emphasizing the intensity that accompanies the emotion of "anger." "Phase Shifter" is an effect that makes the singing voice waver and undulate, thereby evoking the emotion of "sadness."

[0040] For example, suppose that the acquisition unit 100 outputs the acquired emotion type "anger" to the performance control unit 200. In this case, the performance control unit 200 refers to the table shown in FIG. 3 and identifies the effect "distortion" corresponding to the emotion type "anger." The performance control unit 200 controls the effector 102d to execute processing corresponding to "distortion." The effector 102d executes processing corresponding to "distortion" on the audio signal of the singing voice.

[0041] The mixer 103d mixes the audio signal that has been processed by the effector 102d to correspond to the "distortion" effect and the karaoke performance sound signal. The amplifier 104d amplifies the signal output from the mixer 103d and outputs the karaoke performance sound and the singing voice to which the "distortion" effect has been applied from the speaker 20.

[0042] ==About Karaoke Equipment Processing== Next, the processing in the karaoke device K according to this embodiment will be described with reference to Fig. 4. Fig. 4 is a flowchart showing the processing in the karaoke device K. In this example, a user U uses the karaoke device K alone. It is also assumed that processing corresponding to "normal echo" is preset as the normal effect processing. It is also assumed that the table shown in Fig. 3 is stored in the storage means 10a.

[0043] A user U uses a remote control device 50 to select a piece of music X that he or she wishes to sing as karaoke.

[0044] The performance control unit 200 controls the performance means 10d to perform karaoke performance of the song X. The performance means 10d starts the karaoke performance of the song X (start of karaoke performance; step 10). The user U sings karaoke along with the karaoke performance of the song X. In this case, the singing voice of the user U to which the "normal echo" effect has been applied is emitted from the speaker 20.

[0045] The photographing means 60 photographs the face of the user U who will be singing karaoke, and generates facial image data. The photographing means 60 outputs a facial image signal corresponding to the facial image data to the karaoke main unit 10 (outputting a facial image signal corresponding to the facial image data of the user; step 11).

[0046] The acquisition unit 100 acquires emotion information indicating the emotion of the user U based on the facial image signal of the user U output in step 11 (acquisition of emotion information; step 12). The acquisition unit 100 outputs the acquired emotion information to the performance control unit 200.

[0047] The performance control section 200 controls the effector 102d to execute effect processing corresponding to the emotion information acquired in step 12 (execute effect processing corresponding to the emotion information; step 13).

[0048] For example, suppose that the imaging means 60 captures an image of the face of user U at the timing of the first chorus of song X. Then, the acquisition unit 100 acquires "anger" as the type of emotion of user U based on a facial image signal corresponding to the facial image data generated by the imaging means 60. In this case, the acquisition unit 100 outputs the acquired type of emotion, "anger," to the performance control unit 200 as emotion information.

[0049] The performance control unit 200 refers to the table shown in Fig. 3 and identifies "distortion," which is an effect corresponding to the emotion type "anger." The performance control unit 200 controls the effector 102d to execute processing corresponding to "distortion" instead of the preset "normal echo."

[0050] Thereafter, the imaging means 60 captures an image of the face of the user U at the timing of the verse of the second verse of the song X. Then, the acquisition unit 100 acquires "sadness" as the type of emotion of the user U based on a facial image signal corresponding to the facial image data generated by the imaging means 60. In this case, the acquisition unit 100 outputs the acquired type of emotion, "sadness," to the performance control unit 200 as emotion information.

[0051] The performance control unit 200 refers to the table shown in Fig. 3 and identifies "phase shifter," an effect that corresponds to the emotion type "sadness." The performance control unit 200 controls the effector 102d to execute processing that corresponds to "phase shifter" instead of "distortion."

[0052] Furthermore, suppose that the imaging means 60 captures an image of the face of the user U at the timing of the bridge of the third verse of the song X. Then, suppose that the acquisition unit 100 acquires "joy" as the type of emotion of the user U based on the facial image signal corresponding to the facial image data generated by the imaging means 60. In this case, the acquisition unit 100 outputs the acquired type of emotion, "joy," to the performance control unit 200 as emotion information.

[0053] The performance control unit 200 refers to the table shown in Fig. 3 and identifies the effect "deep echo" that corresponds to the emotion type "joy." The performance control unit 200 controls the effector 102d to execute processing corresponding to "deep echo" instead of "phase shifter."

[0054] The performance control unit 200 controls the mixer 103d to mix the sound signal that has been effected by the effector 102d and the karaoke performance sound signal. The performance control unit 200 controls the amplifier 104d to amplify the signal output from the mixer 103d and emit the karaoke performance sound and the singing voice with the effect applied from the speaker 20 (emit the karaoke performance sound and the singing voice with the effect applied; step 14).

[0055] The karaoke device K repeats the processes from step 11 to step 14 until the karaoke performance of the piece of music X is completed (if Y in step 15).

[0056] The performance control unit 200 may control the effector 102d for each singing section. For example, the performance control unit 200 controls the effector 102d to execute a process corresponding to "phase shifter" at the timing of the verse of the second verse of song X. Thereafter, when the singing section of the verse of the second verse of song X ends (i.e., when the karaoke performance for that singing section ends), the performance control unit 200 stops the execution of the process corresponding to "phase shifter" and controls the effector 102d to execute a normal effect process (a process corresponding to "normal echo" in the above example).

[0057] As is clear from the above, the karaoke machine K according to this embodiment has an effector 102d that applies effect processing to the audio signal of the singing voice. The karaoke machine K also has an acquisition unit 100 that acquires emotion information indicating the emotion of the user based on the facial image signal of the user output from the image capture means 60, and a performance control unit 200 that controls the effector 102d to apply effect processing corresponding to the acquired emotion information.

[0058] According to the karaoke device K, an effect corresponding to the emotion of the user can be specified based on the emotion information of the user, and a process corresponding to the specified effect can be performed on the audio signal. Therefore, the karaoke device K can produce a singing voice that reflects the emotion of the user. In other words, the karaoke device K according to this embodiment can perform effect processing according to the emotion of the user using the karaoke device K.

[0059] Furthermore, the acquiring section 100 according to this embodiment can acquire the type of emotion as emotion information, in which case the karaoke device K can execute appropriate effect processing according to the type of emotion.

[0060] The acquiring unit 100 according to the present embodiment can acquire emotion information based on a facial image signal of a singer who is a user singing karaoke. In this case, the karaoke device K can emit a singing voice that reflects the emotion of the singer.

[0061] <Variation 1> The acquisition unit 100 can acquire the type of emotion and the intensity of the emotion as emotion information.

[0062] To acquire the type and intensity of emotion, known software such as emotion recognition software "Affdex" (Affectiva) can be used. The acquisition unit 100 inputs the facial image signal output from the imaging means 60 into the software, thereby acquiring the type and intensity of emotion expressed in the facial image data corresponding to the facial image signal. The intensity of emotion can be indicated by a numerical value (for example, a value from 0 to 10) for each type of emotion.

[0063] As in the example embodiment, it is assumed that the imaging means 60 captures an image of the face of the user U at the timing of the first chorus of the song X. It is assumed that the acquisition unit 100 acquires "anger (7), joy (1), sadness (3)" as the types and intensities of the emotions of the user U based on the facial image signal corresponding to the facial image data generated by the imaging means 60. In this case, the acquisition unit 100 outputs the acquired types and intensities of the emotions "anger (7), joy (1), sadness (3)" to the performance control unit 200 as emotion information.

[0064] The performance control unit 200 identifies the strongest emotion type (highest numerical value) of "anger" from among the multiple emotion types based on the emotion information output from the acquisition unit 100. The performance control unit 200 controls the effector 102d to execute effect processing corresponding to the emotion type of "anger."

[0065] In this way, the acquisition section 100 according to this modification can acquire the type and intensity of emotion as emotion information, and in this case, can execute effect processing suited to the type and intensity of emotion.

[0066] <Variation 2> Alternatively, the acquisition unit 100 may acquire the intensity of emotion alone as emotion information. For example, the acquisition unit 100 can acquire the intensity of emotion expressed in the facial image data corresponding to the facial image signal by inputting the facial image signal output from the imaging means 60 into known software. The acquisition unit 100 outputs the acquired intensity of emotion to the performance control unit 200 as emotion information. The intensity of emotion can be indicated by a numerical value (for example, a value from 0 to 10).

[0067] The performance control unit 200 identifies an effect that corresponds to the intensity of the acquired emotion, and controls the effector 102d to execute processing that corresponds to the identified effect.

[0068] The effect can be specified by referring to a table stored in advance in the storage means 10a, for example.

[0069] Figure 5 is a table showing the relationship between emotional intensity and effects. In this example, emotional intensity is expressed as a number from 0 to 10. Numbers "0 to 2" are associated with the "robot voice" effect, numbers "3 to 6" are associated with the "normal echo" effect, and numbers "7 to 10" are associated with the "live sound" effect.

[0070] "Robot Voice" is an effect that processes singing voices to sound impersonal and emotionless, thereby emphasizing that the user is not expressing emotion on their face. On the other hand, "Live Sound" is an effect that simulates the acoustic characteristics (such as reverberation) specific to large stages like stadiums, thereby further emphasizing the user's emotional expression.

[0071] As in the example embodiment, it is assumed that the imaging means 60 captures an image of the face of the user U at the timing of the first chorus of the song X. It is assumed that the acquisition unit 100 acquires a numerical value "7" as the intensity of the emotion of the user U based on a facial image signal corresponding to the facial image data generated by the imaging means 60. In this case, the acquisition unit 100 outputs the acquired emotional intensity "7" to the performance control unit 200 as emotion information.

[0072] 5, the performance control unit 200 identifies the effect "live sound" that corresponds to the emotional strength "7." The performance control unit 200 controls the effector 102d to execute processing that corresponds to "live sound."

[0073] In this way, the acquisition section 100 according to this modification can acquire the intensity of emotion as emotion information, and in this case, can execute effect processing suited to the intensity of emotion.

[0074] <Variation 3> In the above embodiment, an example has been described in which emotion information indicating the emotion of the singer is acquired, but the acquisition unit 100 may also acquire emotion information indicating the emotion of the audience.

[0075] As described in the embodiment, the acquisition unit 100 can identify audience members, who are users other than the singer, by analyzing images corresponding to the facial image signals of multiple people. The acquisition unit 100 acquires emotion information indicating the emotions of the audience based on the facial image signals of the identified audience members. The emotion information indicating the emotions of the audience can be acquired in the same way as the emotion information indicating the emotions of the singer.

[0076] There may be multiple audience members. In this case, the acquisition unit 100 may output all of the emotional information acquired for each audience member to the performance control unit 200, or may select one piece of emotional information from the multiple pieces of emotional information and output it to the performance control unit 200.

[0077] When all of the emotional information acquired for each audience member has been output, the performance control unit 200 determines one effect process to be executed based on a predetermined condition. The predetermined condition may be a preset condition such as "the most numerous emotional information."

[0078] For example, suppose that the plurality of pieces of emotion information output from the acquisition unit 100 include two emotion types of "anger" and one emotion type of "sadness." In this case, the performance control unit 200 controls the effector 102d to execute effect processing corresponding to the emotion type of "anger" that is the most frequently found.

[0079] In this way, the acquisition unit 100 according to this modification can acquire emotion information based on the facial image signals of the audience, i.e., users listening to the karaoke singing. By using the facial image signals of the audience, singing voices that reflect the emotions of the audience can be emitted.

[0080] <Variation 4> The acquiring section 100 may acquire emotion information indicating the emotions of the singer and the audience.

[0081] As described in the embodiment and variant example 3, the acquisition unit 100 can acquire emotional information indicating the emotions of the singer based on the facial image signal of the identified singer, and can acquire emotional information indicating the emotions of the audience based on the facial image signal of the identified audience.

[0082] The acquisition unit 100 outputs the acquired emotional information of the singer and emotional information of the audience to the performance control unit 200.

[0083] When the singer's emotional information and the audience's emotional information are output, the performance control unit 200 compares the singer's emotional information with the audience's emotional information. If the acquired singer's emotional information and audience's emotional information match, the performance control unit 200 controls the effector 102d to execute effect processing corresponding to the emotional information. On the other hand, if the acquired singer's emotional information and audience's emotional information do not match, the performance control unit 200 does not execute any particular processing.

[0084] With this karaoke device K, only when the acquired emotional information of the singer and the emotional information of the audience match, effect processing corresponding to the emotional information can be executed, thereby enabling the singing voice to be emitted that reflects the common emotion of the singer and the audience.

[0085] <Other> The above-described embodiments are presented as examples and do not limit the scope of the invention. The above configurations can be implemented in appropriate combinations, and various omissions, substitutions, and modifications can be made without departing from the spirit of the invention. The above-described embodiments and their modifications are included in the scope and spirit of the invention, as well as in the inventions described in the claims and their equivalents. [Explanation of symbols]

[0086] 60 Filming Methods 100 Acquisition Department 102d Effector 200 Performance control unit K Karaoke equipment

Claims

1. A karaoke device having an effector that performs effect processing on a voice signal of a singing voice, an acquisition unit that acquires emotion information indicating the emotion of the singer and emotion information indicating the emotion of the audience based on face image signals of the singer who sings karaoke and the audience who listen to the karaoke singing, output from the imaging means; a performance control unit that controls the effector to execute effect processing corresponding to the acquired emotional information when the acquired emotional information of the singer and the acquired emotional information of the audience match; A karaoke device having:

2. 2. The karaoke apparatus according to claim 1, wherein the acquisition unit acquires the type of emotion and / or the intensity of emotion as the emotion information.

Images