Information processing system, keyboard instrument, information processing method, and recording medium

Abstract

An information processing system has an operation detecting section that detects operations by a user with respect to a plurality of operation pieces corresponding to different pitches, and an operation analyzing section that determines a chord corresponding to a combination of two or more operation pieces of the plurality of operation pieces for which operations are detected, and including a root note defined in correspondence with operation amounts or operation strengths of the two or more operation pieces, respectively.

Description

Technical Field

[0001] This invention relates to a technique for determining chords (chords) composed of multiple different pitches. Background Technology

[0002] Techniques have been proposed to infer chords (harmonics) based on the results of a user playing an instrument. For example, Patent Document 1 discloses a structure for inferring chords based on performance information indicating a performance by a user.

[0003] Patent Document 1: Japanese Patent Application Publication No. 2015-31738 Summary of the Invention

[0004] For example, among multiple chords in inversion, even if the combination of multiple pitches is common, sometimes the root note is different. Based on conventional techniques, it is difficult to accurately determine the chord while also taking into account the difference in the root note. Considering the above, one objective of this invention is to accurately determine the chord while also taking into account the difference in the root note.

[0005] To address the above issues, one aspect of the present invention relates to an information processing system comprising: an operation detection unit that detects the user's operation on each of a plurality of operation devices corresponding to different pitches; and an operation analysis unit that determines a chord corresponding to a combination of two or more operation devices from the plurality of operation devices on which the operation was detected, and which includes a root note defined in accordance with the operation amount or intensity of each of the two or more operation devices.

[0006] One aspect of the present invention relates to a keyboard musical instrument comprising: a keyboard including a plurality of keys corresponding to different pitches; an operation detection unit that detects operations performed by a user on each of the plurality of keys; a playback control unit that causes a playback device to play a musical tone corresponding to a pitch of the plurality of keys whose operation was detected by the operation detection unit; and an operation analysis unit that determines a chord corresponding to a combination of two or more keys whose operation was detected, and which includes a root note defined in accordance with the amount or intensity of operation of each of the two or more keys.

[0007] One aspect of the present invention relates to an information processing method that detects the user's operation of multiple operating devices corresponding to different pitches, and determines a chord that corresponds to a combination of two or more operating devices whose operation has been detected, and includes a root note defined in accordance with the amount or intensity of operation of each of the two or more operating devices.

[0008] One aspect of the present invention relates to a program that enables a computer to function as: an operation detection unit that detects the user's operation on each of a plurality of operation pieces corresponding to different pitches; and an operation analysis unit that determines a chord corresponding to a combination of two or more operation pieces from the plurality of operation pieces whose operation has been detected, and which includes a root note defined in accordance with the operation amount or intensity of each of the two or more operation pieces. Attached Figure Description

[0009] Figure 1 This is a block diagram illustrating the structure of the keyboard musical instrument according to the first embodiment.

[0010] Figure 2 This is an explanatory diagram related to the displacement of each key.

[0011] Figure 3 This is a block diagram illustrating the functional structure of a keyboard musical instrument.

[0012] Figure 4 This is a schematic diagram of the reference table.

[0013] Figure 5 This is an explanatory diagram related to the method of determining the playing chords.

[0014] Figure 6 This is a flowchart illustrating the specific process of parsing and processing.

[0015] Figure 7 This is a block diagram illustrating the functional structure of the keyboard instrument according to the second embodiment.

[0016] Figure 8 This is a flowchart illustrating the specific process of the parsing procedure in the second embodiment.

[0017] Figure 9 This is an explanatory diagram of the operation analysis section of the third embodiment.

[0018] Figure 10 This is an explanatory diagram related to machine learning of a trained model. Detailed Implementation

[0019] A: Implementation Method 1

[0020] Figure 1 This is a block diagram illustrating the structure of a keyboard musical instrument 100 according to a first embodiment of the present invention. The keyboard musical instrument 100 is an electronic musical instrument that produces musical notes corresponding to the playing performed by the user. The keyboard musical instrument 100 includes a keyboard 10, a detection device 20A, an information processing system 30, a playback device 40, and a display device 50.

[0021] The keyboard 10 consists of multiple keys 12 corresponding to different pitches. These multiple keys 12 are arranged horizontally along the user's side when playing the keyboard instrument 100, and include multiple white keys and multiple black keys. Each key 12 is an actuating element that shifts in accordance with the user's operation (pressing or releasing a key).

[0022] Figure 2 This is an explanatory diagram related to the displacement of key 12. Each key 12 is displaced vertically between the starting position E1 and the ending position E2, corresponding to the operation performed by the user. The starting position E1 is the position of the surface of the key 12 in the released state, where the user's finger is not in contact with the key 12. On the other hand, the ending position E2 is the position of the surface of the key 12 in the pressed state, where the user has fully pressed down the key 12. The user can operate each key 12 at any position between the starting position E1 and the ending position E2.

[0023] Figure 1 The detection device 20A detects the displacement of each of the multiple keys 12. Specifically, the detection device 20A generates a detection signal Da corresponding to the position of each key 12 in the vertical direction. The detection signal Da is an electrical signal whose signal level changes periodically or continuously in conjunction with the movement of each key 12 in the vertical direction. For example, the detection device 20A may be a magnetic sensor that generates the detection signal Da by utilizing the change in magnetic field in conjunction with the movement of each key 12, or an optical sensor that generates the detection signal Da by utilizing the change in the amount of light received in conjunction with the movement of each key 12. However, the structure and method of the detection device 20A for detecting the displacement of each key 12 are not limited to the examples described above.

[0024] The information processing system 30 determines the chords (hereinafter referred to as "playing chords") played by the user. A playing chord is a combination of multiple pitches played side by side. The information processing system 30 determines the playing chords by analyzing the detection signal Da. The determination of the playing chords based on the information processing system 30 is repeated in parallel with the playing by the user. The information processing system 30 is implemented by a computer system having a control device 31 and a storage device 32.

[0025] The control device 31 consists of one or more processors that control various elements of the keyboard instrument 100. For example, the control device 31 consists of one or more processors such as CPU (Central Processing Unit), SPU (Sound Processing Unit), DSP (Digital Signal Processor), FPGA (Field Programmable Gate Array), or ASIC (Application Specific Integrated Circuit).

[0026] Storage device 32 is a single or multiple memory devices that store the program executed by control device 31 and the data used by control device 31. Storage device 32 is constructed of known recording media such as magnetic recording media or semiconductor recording media. Alternatively, storage device 32 may be constructed from a combination of various recording media. In addition, portable recording media that can be attached to or detached from keyboard instrument 100, or external recording media (e.g., network hard drive) that can communicate with keyboard instrument 100 may be used as storage device 32.

[0027] The playback device 40 plays musical sounds corresponding to the user's operations on the keyboard 10, based on the control of the control device 31. Specifically, the playback device 40 has a sound source device 41 and a sound output device 42. Furthermore, the playback device 40 can be integrated into the information processing system 30, or it can be configured as a separate device from the information processing system 30.

[0028] The sound source device 41 generates an acoustic signal V (acoustic signal) representing the waveform of a musical tone corresponding to the user's operation on the keyboard 10. Specifically, it generates an acoustic signal V representing the pitch of a musical tone among a plurality of keys 12 corresponding to the key 12 pressed by the user. The playback device 42 plays the musical tone represented by the acoustic signal V. For example, a speaker or headphones are used as the playback device 42. Alternatively, the sound source device 41 can also function as a software sound source by executing a program stored in the storage device 32 by the control device 31.

[0029] The display device 50 displays images based on the control of the control device 31. For example, a display panel such as a liquid crystal display panel or an organic EL display panel is used as the display device 50. The display device 50 of the first embodiment displays the chord names of the playing chords determined by the control device 31. Furthermore, the display device 50 can be mounted on the information processing system 30, or it can be configured as a separate device from the information processing system 30.

[0030] Figure 3 This is a block diagram illustrating the functional structure of the information processing system 30. The control device 31 implements multiple functions (operation detection unit 61A, playback control unit 62, and operation analysis unit 63A) for analyzing user operations on the keyboard 10 by executing programs stored in the storage device 32.

[0031] The operation detection unit 61A detects the user's operations on each of the multiple keys 12. Specifically, the operation detection unit 61A determines the operation amount Z of each key 12 by analyzing the detection signal Da generated by the detection device 20A. The operation amount Z is as follows: Figure 2As illustrated, this is the displacement of key 12 generated by the user's operation. Specifically, as... Figure 2 As illustrated, the operation amount Z is the displacement of key 12 based on the starting position E1. That is, the depth to which key 12 is pressed by the user is determined as the operation amount Z. Furthermore, the operation detection unit 61A determines whether an operation has occurred on each key 12 in accordance with the operation amount Z of each key 12. Specifically, the operation detection unit 61A determines that a key 12 has been operated by the user if the operation amount Z is greater than a predetermined threshold.

[0032] Figure 3 The playback control unit 62 controls the playback of musical tones from the playback device 40. Specifically, the playback control unit 62 causes the playback device 40 to play musical tones corresponding to the pitches of the keys 12 that are operated and detected by the operation detection unit 61A among the multiple keys 12 constituting the keyboard 10.

[0033] The operation analysis unit 63A determines the chords played by the user through the operation of the keyboard 10. The operation analysis unit 63A determines the chords based on the detection results of the operation detection unit 61A. Specifically, the determination of the chords by the operation analysis unit 63A utilizes a reference table T stored in the storage device 32.

[0034] Figure 4 This is a schematic diagram of reference table T. Reference table T is a data table that records the chord name, constituent notes, and root note for each of several candidate chords used in playing a chord. The chord name is a string representing the name of the chord. The constituent notes of each chord are a combination of the multiple pitches that make up the chord. The root note of each chord is the single pitch among the multiple pitches that make up the chord, serving as the foundation of the chord. For example, the lowest pitch among the multiple pitches that make up the chord is equivalent to the root note.

[0035] Figure 3 The operation analysis unit 63A determines the playable chords corresponding to the combinations of multiple keys 12 detected by the operation detection unit 61A by referring to the reference table T. Specifically, the operation analysis unit 63A determines the playable chords as those among the multiple chords registered in the reference table T that are composed of multiple pitches corresponding to the multiple keys 12 detected by the operation detection unit 61A.

[0036] However, for example, among multiple chords that are inverted, sometimes even if the combination of multiple pitches is common, the root note is different. Therefore, it is conceivable that the user's intended chord cannot be uniquely determined based solely on the combination of multiple pitches. Taking the above into consideration, the operation analysis unit 63A determines the root note of the chord played among the multiple pitches played by the user.

[0037] Users tend to operate more forcefully on one key 12 corresponding to the root note of a desired chord among the multiple pitches that constitute that chord, compared to more than one key 12 corresponding to other pitches. Therefore, there is a tendency for the amount of operation Z on the one key 12 corresponding to the root note of the intended chord among the multiple keys 12 operated by the user to be greater than the amount of operation Z on more than one other key 12.

[0038] Considering the above tendencies, the operation analysis unit 63A of the first embodiment determines the root note of the chord to be played in accordance with the operation amount Z of each of the plurality of keys 12. Specifically, the operation analysis unit 63A determines the pitch corresponding to the key 12 with the largest operation amount Z among the plurality of keys 12 operated by the user as the root note. Moreover, the operation analysis unit 63A determines the chord containing this root note as the chord to be played. As understood from the above description, the operation analysis unit 63A of the first embodiment determines the chord that corresponds to the combination of the plurality of keys 12 operated by the operation detection unit 61A and contains the root note determined according to the operation amount Z of each of the plurality of keys 12. In addition, the operation analysis unit 63A displays the chord name registered in the reference table T for the chord to be played on the display device 50.

[0039] Figure 5 This is a specific example of the action of determining the chords to be played using the analytical unit 63A. Figure 5 In Examples A to C, we imagine a user operating the three keys 12 corresponding to the pitches of “C (do)”, “E (mi)”, and “G (sol)”. As chords with the three pitches shown above as the constituent notes, there are “C”, “Em (+5)”, and “G6sus4”.

[0040] In Example A, the operation amount Z of the key 12 corresponding to the pitch "C" is greater than the operation amount Z of the keys 12 corresponding to the pitches "E" and "G" respectively. Therefore, the operation analysis unit 63A determines the pitch "C" as the root note and determines the chord "C" containing the root note "C" as the chord to be played.

[0041] In Example B, the operation amount Z of the key 12 corresponding to the pitch "E" is greater than the operation amount Z of the keys 12 corresponding to the pitches "C" and "G" respectively. Therefore, the operation analysis unit 63A determines the pitch "E" as the root note and determines the chord "Em(+5)" containing the root note "E" as the chord to be played.

[0042] In example C, the operation amount Z of the key 12 corresponding to the pitch "G" is greater than the operation amount Z of the keys 12 corresponding to the pitches "C" and "E" respectively. Therefore, the operation analysis unit 63A determines the pitch "G" as the root note and determines the chord "G6sus4" containing the root note "G" as the chord to be played.

[0043] Figure 6 This is a flowchart illustrating the specific flow of the process (hereinafter referred to as "parsing process") Sa performed by the control device 31. For example, the parsing process Sa is initiated based on an instruction from the user.

[0044] If the parsing process Sa begins, the operation detection unit 61A acquires the detection signal Da (Sa1) supplied from the detection device 20A. The operation detection unit 61A parses the detection signal Da to determine the operation amount Z (Sa2) of each of the multiple keys 12. That is, the operation detection unit 61A detects the user's operation on each of the multiple keys 12. The operation detection unit 61A determines whether the user has played a chord (Sa3). For example, the operation detection unit 61A determines whether the operation amount Z of two or more keys 12 is greater than a threshold. If the operation amount Z of two or more keys 12 is greater than the threshold, the operation detection unit 61A determines that the user has played a chord (Sa3: YES); if the operation amount Z of one or fewer keys 12 is greater than the threshold, the operation detection unit 61A determines that the user has not played a chord (Sa3: NO). The operation detection unit 61A repeatedly acquires the detection signal Da (Sa1) and determines the operation amount Z (Sa2) (Sa3: NO) of each key 12 until the user plays a chord. In addition, the playback control unit 62 causes the playback device 40 to play musical notes of a pitch corresponding to the key 12 operated by the user.

[0045] When the user plays a chord (Sa3: YES), the operation analysis unit 63A determines the root note of the chord played by the user (Sa4). Specifically, the operation analysis unit 63A determines the pitch corresponding to the key 12 with the largest operation amount Z among the multiple keys 12 detected by the operation detection unit 61A as the root note.

[0046] The operation analysis unit 63A identifies the chord corresponding to the combination of multiple keys 12 operated by the user and containing the root note determined in accordance with the operation amount Z as the performance chord (Sa5). Specifically, the operation analysis unit 63A retrieves from the reference table T two or more chords that use the multiple pitches played by the user as constituent notes, and identifies the chord containing the root note determined in step Sa4 as the performance chord. The operation analysis unit 63A displays the chord name registered in the reference table T for the performance chord on the display device 50 (Sa6).

[0047] Control device 31 determines whether the specified termination condition is met (Sa7). The termination condition is, for example, that the user has indicated the end, or that the user's performance has ended. If the termination condition is not met (Sa7: NO), control device 31 causes the process to proceed to step Sa1. That is, until the termination condition is met, the operation quantity Z of each key 12 is repeatedly checked (Sa1-Sa3), the chord is determined (Sa4, Sa5), and the chord name of the chord is displayed (Sa6). If the termination condition is met (Sa7: YES), control device 31 causes the parsing process Sa to end.

[0048] As explained above, in the first embodiment, the root note of the chord is determined corresponding to the operation amount Z of each of the multiple keys 12 that are detected to have been operated by the user. Therefore, compared to determining the structure of the chord based solely on the combination of the multiple keys 12 operated by the user, an accurate chord that also takes into account the difference in the root note can be determined. In particular, in the first embodiment, it is possible to determine a chord that includes the pitch corresponding to the key 12 with the largest operation amount Z among the multiple keys 12 operated by the user as the root note. Therefore, based on the user's strong tendency to operate the key 12 corresponding to the root note of the desired chord, a chord with the root note accurately distinguished can be determined.

[0049] B: Implementation Method 2

[0050] The second embodiment will now be described. Furthermore, in the embodiments illustrated below, elements that function the same as those in the first embodiment will retain the reference numerals used in the description of the first embodiment, with appropriate omitting of their detailed descriptions.

[0051] Figure 7 This is a block diagram illustrating the functional structure of the control device 31 in the second embodiment. For example... Figure 7 As illustrated, in the second embodiment, the detection device 20A of the first embodiment is replaced by a detection device 20B. The detection device 20B generates a detection signal Db that corresponds to the intensity (operation intensity) of the user's operation of each key 12. Specifically, the detection device 20B is a pressure sensor that generates a detection signal Db that corresponds to the pressure applied by the user to each key 12.

[0052] In the second embodiment, the operation detection unit 61A is replaced by the operation detection unit 61B, and the operation analysis unit 63A is replaced by the operation analysis unit 63B. The operation detection unit 61B detects the user's operation on each of the multiple keys 12. The operation detection unit 61B of the second embodiment analyzes the detection signal Db generated by the detection device 20B, thereby determining the intensity (hereinafter referred to as "operation intensity") X of the user's operation on each key 12. The operation intensity X is, for example, the pressure applied by the user when pressing the key 12. Furthermore, the playback control unit 62 causes the playback device 40 to play musical notes of the pitch corresponding to the key 12 detected by the operation detection unit 61A, and the structure and operation are the same as in the first embodiment.

[0053] The operation analysis unit 63B determines the playing chords based on the detection results of the operation detection unit 61B. The determination of the playing chords by the operation analysis unit 63B utilizes the same reference table T as in the first embodiment.

[0054] As mentioned above, users tend to operate the key 12 corresponding to the root note of the desired chord more forcefully than any other key 12 corresponding to other pitches. Therefore, there is a tendency for the intensity X of the operation on the key 12 corresponding to the root note of the intended chord to be greater than the intensity X of the operation on any other key 12.

[0055] Considering the above tendencies, the operation analysis unit 63B of the second embodiment determines the root note of the chord to be played in accordance with the operation intensity X of each of the plurality of keys 12. Specifically, the operation analysis unit 63B determines the pitch corresponding to the key 12 with the highest operation intensity X among the plurality of keys 12 operated by the user as the root note. Moreover, the operation analysis unit 63A determines the chord containing this root note as the chord to be played. As understood from the above description, the operation analysis unit 63B of the second embodiment determines the chord that corresponds to the combination of the plurality of keys 12 operated by the operation detection unit 61B and contains the root note determined according to the operation intensity X of each of the plurality of keys 12.

[0056] In the second embodiment, the parsing process Sa is performed instead of the parsing process in the first embodiment. Figure 8 The parsing and processing of Sb. For example, the parsing and processing of Sb is initiated by an instruction from the user.

[0057] If the analysis and processing of Sb begins, the operation detection unit 61B analyzes the detection signal Db supplied from the detection device 20B, thereby determining the operation intensity X of each key 12 (Sb1, Sb2). That is, the operation detection unit 61B detects the operation for each of the plurality of keys 12. Similar to the first embodiment, the operation detection unit 61B determines whether the user has played a chord (Sb3). The operation detection unit 61B repeatedly performs the process of determining the operation intensity X of each key 12 (Sb1, Sb2) (Sb3: NO) until the user plays a chord.

[0058] When the user plays a chord (Sb3: YES), the operation analysis unit 63B determines the root note of the chord played by the user (Sb4). Specifically, the operation analysis unit 63B determines the pitch corresponding to the key 12 with the largest operation intensity X among the multiple keys 12 detected by the operation detection unit 61B as the root note.

[0059] The operation analysis unit 63B determines the chord corresponding to the combination of multiple keys 12 operated by the user and including the root note determined according to the operation intensity X as the performance chord (Sb5). Similar to the first embodiment, the operation analysis unit 63B displays the chord name registered in reference table T for the performance chord on the display device 50 (Sb6). The above process is repeated until the predetermined end condition is met (Sb7: YES) (Sb1-Sb6).

[0060] In the second embodiment, the same effect as in the first embodiment is achieved. Furthermore, in the second embodiment, a chord is determined that includes the pitch corresponding to the key 12 with the highest operational intensity X among the multiple keys 12 operated by the user as the root note. Therefore, based on the user's strong tendency to operate the key 12 corresponding to the root note of the desired chord, it is possible to accurately determine the chord with the root note distinguished.

[0061] C: Third Implementation

[0062] In the third embodiment, the operation parsing unit 63A of the first embodiment is replaced with... Figure 9 The operation analysis unit 63C of the first embodiment. As described above, the operation analysis unit 63A of the first embodiment determines the playing chords using reference table T. The operation analysis unit 63C of the third embodiment uses a trained model M to determine the playing chords.

[0063] Specifically, the operation analysis unit 63C inputs the input data Q1, representing the detection result of the operation detection unit 61A, into the trained model M, thereby generating output data Q2. The input data Q1 represents the operation quantity Z determined by the operation detection unit 61A for each of the multiple keys 12. On the other hand, the output data Q2 represents the data representing the played chords.

[0064] The trained model M is a statistical inference model that has learned (trained) the relationship between the operands Z of each key 12 and the chord progressions (the relationship between input data Q1 and output data Q2) through machine learning. Specifically, the trained model M is, for example, composed of a deep neural network (DNN). For example, any form of neural network, such as a recurrent neural network (RNN) or a convolutional neural network (CNN), can be used as the trained model M. Additionally, additional elements such as Long Short-Term Memory (LSTM) can be incorporated into the trained model M. Recognition models such as Hidden Markov Models (HMMs) or Support Vector Machines (SVMs) can also be used as the trained model M.

[0065] The trained model M is achieved by a combination of a program that causes the control device 31 to perform a calculation to generate output data Q2 based on input data Q1, and multiple variables (specifically, weighted values ​​and biases) applied to this calculation. The program for implementing the trained model M and the multiple variables are stored in the storage device 32. The values ​​of each of the multiple variables are preset through machine learning.

[0066] Figure 10 This is an illustrative diagram related to machine learning of the trained model M. The control device 31 acts as a controller by executing programs stored in the storage device 32. Figure 10 The learning processing unit 64 is operational. The learning processing unit 64 builds a trained model M using teacher-guided machine learning with multiple training data τ. Multiple training data τ are stored in the storage device 32. Alternatively, a trained model M can be built using machine learning performed by a different machine learning system than the keyboard instrument 100, and this trained model M can be transferred to the keyboard instrument 100.

[0067] Multiple training data sets τ are each composed of a combination of input data Q1t and output data Q2t. Output data Q2t represents the data for playing chords. The input data Q1t of each training data set τ specifies the combination of multiple keys 12 corresponding to the playing chord represented by the output data Q2t of that training data set τ. That is, the combination of multiple keys 12 operated when playing a playing chord is specified by the input data Q1t. Furthermore, the input data Q1t specifies the operation amount Z for each of the multiple keys 12 corresponding to the playing chord. Even when the combination of multiple keys 12 specified by the input data Q1t is common across the multiple training data sets τ, the playing chord represented by the output data Q2t of each training data set τ will be different if the operation amounts Z of each key 12 are different. Specifically, the output data Q2t specifies the playing chord with the pitch corresponding to the key 12 with the largest operation amount Z among the multiple keys 12 represented by the input data Q1t as the root note.

[0068] The learning processing unit 64 repeatedly updates multiple variables of the trained model M in a manner that reduces the difference between the output data Q2 generated by inputting the input data Q1t of each training data τ to the initial or provisional model and the output data Q2t (positive solution value) of the same training data τ. For example, backpropagation is used to update the multiple variables. As understood from the above description, the trained model M outputs statistically appropriate output data Q2 relative to the unknown input data Q1 based on the potential relationship between the input data Q1t and the output data Q2t of the multiple training data τ.

[0069] As described above, the chord represented by the output data Q2t of each training data τ is a chord containing the root note corresponding to the operation amount Z specified by the input data Q1t for each key 12. Therefore, the operation analysis unit 63C, like the operation analysis unit 63A in the first embodiment, determines the chord (output data Q2) that corresponds to the combination of the multiple keys 12 operated by the user and contains the root note determined corresponding to the operation amount Z of each of the multiple keys 12. As understood from the above description, the same effect as in the first embodiment is achieved in the third embodiment.

[0070] Furthermore, in the above description, an example is given of determining the playing chord in accordance with the operation amount Z of each key 12. However, for the method of determining the playing chord in accordance with the operation intensity X of each key 12, as in the second embodiment, the same trained model M as in the third embodiment can also be used. Specifically, input data Q1 and input data Q1t each specify the operation intensity X for multiple keys 12. In addition, the output data Q2t of each training data τ specifies the playing chord with the pitch corresponding to the key 12 with the largest operation intensity X among the multiple keys 12 represented by the input data Q1t of the training data τ as the root note. The machine learning process and reference of the trained model M performed by the learning processing unit 64. Figure 10 The aforementioned process is the same. Based on the above structure, similar to the second embodiment, a playing chord (output data Q2) corresponding to the combination of multiple keys 12 operated by the user and including the root note determined in accordance with the operating intensity X of each of the multiple keys 12 is determined.

[0071] D: Variation Example

[0072] The following examples illustrate specific variations that can be added to the methods illustrated above. Two or more methods selected from the examples below may be combined, provided they do not contradict each other.

[0073] (1) In the aforementioned methods, the chord names of the played chords are displayed on the display device 50, but the method for determining the result of the played chords is not limited to the examples above. For example, the time sequence of the played chords resolved by the operation resolution unit 63 (63A, 63B, or 63C) may also be stored in the storage device 32. Furthermore, it is envisioned that the playback device 40 plays musical notes corresponding to the played chords determined by the operation resolution unit 63. For example, the playback control unit 62 causes the playback device 40 to play the accompaniment note corresponding to the played chords determined by the operation resolution unit 63 from among multiple accompaniment notes corresponding to different chords. In addition, the played chords resolved by the operation resolution unit 63 may also be transmitted to other information devices via a communication network such as the Internet.

[0074] (2) In the aforementioned embodiments, each key 12 constituting the keyboard 10 is exemplified as an operating element, but the specific form of the operating element is not limited to the key 12. Operating elements of any form that are displaced in accordance with the user's operation can be used in the same way as the keys 12 in the aforementioned embodiments. Furthermore, the direction of the operation detected by the operation detection unit 61 (61A, 61B) is not limited to the vertical direction. For example, the operation detection unit 61 can also detect displacements in rotational directions related to the operating element, such as pitch, yaw, or rolling direction.

[0075] (3) For example, the information processing system 30 can also be implemented via a server device that communicates between terminal devices such as smartphones or tablets. The terminal device sends detection signals D (Da, Db) corresponding to user operations on the keyboard 10 to the information processing system 30 via a communication network such as the Internet. The information processing system 30, in the same manner as described above, analyzes the detection signals D to determine the chords to be played, and sends the chords to the terminal device. The terminal device performs various processing using the chords received from the information processing system 30.

[0076] (4) The functions illustrated above, as described above, are achieved through the coordinated operation of one or more processors constituting the control device 31 and the program stored in the storage device 32. The program involved in this invention can be provided and installed in a computer in the form of a computer-readable recording medium. The recording medium is, for example, a non-transitory recording medium, preferably an optical recording medium (optical disc) such as a CD-ROM, and also includes any known form of recording medium such as a semiconductor recording medium or a magnetic recording medium. Furthermore, as a non-transitory recording medium, it includes any recording medium other than a transient propagating signal, and may also include volatile recording media. In addition, in a structure in which a transmission device transmits a program via a communication network, the storage device that stores the program in the transmission device is equivalent to the aforementioned non-transitory recording medium.

[0077] E: Appendix

[0078] Based on the examples above, one can, for instance, grasp the following structure.

[0079] One aspect (Aspect 1) of the present invention relates to an information processing system comprising: an operation detection unit that detects the user's operations on multiple operation devices corresponding to different pitches; and an operation analysis unit that determines a chord corresponding to a combination of two or more operation devices whose operations have been detected, and which includes a root note defined in accordance with the operation amount or intensity of each of the two or more operation devices. In the above aspect, the root note of the chord is distinguished in accordance with the operation amount or intensity of each of the two or more operation devices that have detected the user's operation. Therefore, compared to determining the chord structure solely based on the combination of two or more operation devices that have detected the user's operation, an accurate chord that also takes into account the difference in root note can be determined.

[0080] "Operating element" is, for example, a key on a keyboard instrument. Therefore, operation on the operating element is, for example, pressing or releasing a key by the user. Furthermore, "operation amount" refers to the amount of movement of the operating element caused by the user's operation, such as the depth to which the operating element is pressed in by the user's operation. Additionally, "operation intensity" refers to the intensity of the operation on the operating element; a representative example is the pressure applied to the operating element by the operation (e.g., key pressure).

[0081] "A chord containing a root note that is specified in accordance with the amount or intensity of operation of each of two or more operating elements" means that even if the combination of two or more operating elements is common, the resulting chord will be different if the amount or intensity of operation of each operating element is different.

[0082] In a specific example of Method 1 (Method 2), the operation analysis unit determines a chord that includes the pitch corresponding to the operation with the largest operation volume among the two or more operation elements as the root note. Users tend to operate the operation element corresponding to the root note of the desired chord more forcefully. Therefore, by using the aforementioned method of taking the pitch corresponding to the operation element with the largest operation volume among two or more operation elements as the root note, an accurate chord containing the root note intended by the user can be determined.

[0083] In a specific example of Method 1 (Method 3), the operation analysis unit determines a chord whose root note is the pitch corresponding to the operation with the greatest intensity among the two or more operation elements. Users tend to operate the operation element corresponding to the root note of the desired chord more forcefully. Therefore, by using the aforementioned method of determining the root note as the pitch corresponding to the operation element with the greatest intensity among the two or more operation elements that detect the user's operation, a chord with an accurately distinguished root note can be determined.

[0084] In a specific example of Method 1 (Method 4), the operation analysis unit, having learned the relationship between the operation amount or intensity associated with each of two or more operation elements and the chord, inputs input data containing the operation amount or intensity of each of the two or more operation elements detected by the operation detection unit, thereby determining the chord. Based on this structure, compared to determining the chord solely based on the combination of two or more operation elements detected as being operated by the user, an accurate chord that also considers the root note difference can be determined. Furthermore, the trained model is, for example, a statistical inference model constructed through machine learning.

[0085] One aspect (aspect 5) of the present invention relates to a keyboard instrument comprising: a keyboard including a plurality of keys corresponding to different pitches; an operation detection unit that detects operations performed by a user on each of the plurality of keys; a playback control unit that causes a playback device to play a musical tone corresponding to a pitch of the plurality of keys whose operation was detected by the operation detection unit; and an operation analysis unit that determines a chord corresponding to a combination of two or more keys whose operation was detected, and which includes a root note defined in accordance with the amount or intensity of operation of each of the two or more keys.

[0086] One aspect (aspect 6) of the present invention relates to an information processing method that detects the operation of a user on each of a plurality of operating elements corresponding to different pitches, and determines a chord that corresponds to a combination of two or more operating elements whose operation has been detected, and includes a root note that is specified in accordance with the amount or intensity of operation of each of the two or more operating elements.

[0087] One aspect of the present invention (aspect 7) involves a program that causes a computer to function as: an operation detection unit that detects the user's operation on each of a plurality of operation pieces corresponding to different pitches; and an operation analysis unit that determines a chord that corresponds to a combination of two or more operation pieces among the plurality of operation pieces that have detected the operation, and includes a root note defined in accordance with the operation amount or intensity of each of the two or more operation pieces.

[0088] Explanation of the label

[0089] 100…Keyboard instrument, 10…Keyboard, 12…Keys, 20A, 20B…Detection device, 30…Information processing system, 31…Control device, 32…Storage device, 40…Playback device, 41…Sound source device, 42…Sound output device, 50…Display device, 61A, 61B…Operation detection unit, 62…Playback control unit, 63A, 63B, 63C…Operation analysis unit, 64…Learning processing unit.

Claims

1. An information processing system, comprising: The operation detection unit detects the user's operation of multiple control components corresponding to different pitches; and The operation analysis unit determines a chord that corresponds to a combination of two or more operation elements from the plurality of operation elements that have detected the operation, and that includes a root note specified in accordance with the operation amount or intensity of each of the two or more operation elements. wherein The amount of operation is the amount of movement of the operating component caused by the user's operation, and the intensity of operation is the intensity of the operation on the operating component caused by the user's operation.

2. The information processing system according to claim 1, wherein, The operation analysis unit determines the pitch corresponding to the operation with the largest operation amount among the two or more operation elements as the chord including the root note.

3. The information processing system according to claim 1, wherein, The operation analysis unit determines the pitch corresponding to the operation with the greatest operation intensity among the two or more operation elements as the chord including the root note.

4. The information processing system according to claim 1, wherein, The operation analysis unit, having learned the relationship between the operation amount or intensity associated with two or more operation elements and the chord, inputs input data containing the operation amount or intensity of the two or more operation elements detected by the operation detection unit, thereby determining the chord.

5. A keyboard musical instrument, comprising: A keyboard, which contains multiple keys corresponding to different pitches; An operation detection unit detects the user's operations on each of the plurality of keys; The playback control unit causes the playback device to play a musical tone with a pitch corresponding to the key among the plurality of keys that has been operated by the operation detection unit. as well as The operation analysis unit determines a chord corresponding to a combination of two or more keys from the plurality of keys that have detected the operation, and includes a root note specified in accordance with the operation amount or intensity of each of the two or more keys. Wherein, the operation amount is the amount of key movement caused by the user's operation, and the operation intensity is the intensity of the operation on the key caused by the user's operation.

6. The keyboard instrument according to claim 5, wherein, The operation analysis unit determines that the pitch corresponding to the key with the largest operation volume among the two or more keys is included as the root note in the chord.

7. The keyboard instrument according to claim 5, wherein, The operation analysis unit determines that the pitch corresponding to the key with the greatest operation intensity among the two or more keys is included as the root note in the chord.

8. The keyboard instrument according to claim 5, wherein, The operation parsing unit, having learned the relationship between the operation quantity or intensity associated with each of two or more keys and the chord, inputs input data containing the operation quantity or intensity of each of the two or more keys whose operation is detected by the operation detection unit, thereby determining the chord.

9. An information processing method implemented by a computer. The user's operation of multiple control components corresponding to different pitches was tested. A chord is determined that corresponds to a combination of two or more operating elements from the plurality of operating elements in which the operation is detected, and includes a root note specified in accordance with the operation amount or intensity of each of the two or more operating elements. in, The amount of operation is the amount of movement of the operating component caused by the user's operation, and the intensity of operation is the intensity of the operation on the operating component caused by the user's operation.

10. The information processing method according to claim 9, wherein, In determining the chord, the pitch corresponding to the operation with the largest operation amount among the two or more operation operations is determined as the root note of the chord.

11. The information processing method according to claim 9, wherein, In determining the chord, the pitch corresponding to the operating element with the greatest operating intensity among the two or more operating elements is determined as the chord including the root note.

12. The information processing method according to claim 9, wherein, The chord is determined by taking input data containing the operation amount or operation intensity of the two or more operation elements that are detected, and the relationship between the operation amount or operation intensity of the operation element and the chord learned from the trained model that has learned the relationship between the operation amount or operation intensity of the two or more operation elements that are detected.

13. A recording medium having a program recorded thereon that enables a computer to function as a functional unit: The operation detection unit detects the user's operation of multiple control components corresponding to different pitches; and The operation analysis unit determines a chord that corresponds to a combination of two or more operation elements from the plurality of operation elements that have detected the operation, and that includes a root note specified in accordance with the operation amount or intensity of each of the two or more operation elements. in, The amount of operation is the amount of movement of the operating component caused by the user's operation, and the intensity of operation is the intensity of the operation on the operating component caused by the user's operation.

Images