Information processing device, information processing method, and information processing program

Abstract

An information processing device according to the present disclosure comprises: a reception unit for receiving designation of a range within which candidates for a calibration location in first performance data are extracted, and receiving feedback regarding the candidates for the calibration location; and a generation unit for inputting the first performance data, including the range within which candidates for the calibration location are extracted, to a first trained model that accepts input of performance data and outputs candidates for calibration locations, thereby generating candidates for the calibration location in the first performance data and generating second performance data in which the designated calibration location is calibrated on the basis of the feedback.

Description

[Supplement under Rule 26 07.02.2025] Information processing device, information processing method and information processing program

[0001] The present disclosure relates to an information processing device, an information processing method, and an information processing program.

[0002] In recent years, techniques for generating music and the like have become known. For example, Non-Patent Document 1 discloses a technique for generating music for missing parts of music. In Non-Patent Document 1, the music is generated using a machine learning model.

[0003] In machine learning models, efforts are being made to improve the accuracy of the objects to be generated. For example, in the field of image synthesis, it is known that the accuracy of image synthesis can be improved by iteratively improving the generated images (e.g., Non-Patent Document 2).

[0004] “The Piano Inpainting Application” Gaetan Hadjeres, Leopold Crestel “Improved Masked Image Generation with Token-Critic” Jose Lezama, Huiwen Chang, Lu Jiang, Irfan Essa

[0005] In conventional technology, when proofreading a song, the selection of the range to be proofread is accepted as input, and the machine learning model outputs the proofreading of the song within that range. As a result, with conventional technology, there are cases where the proofreading of the song is not what the user intended.

[0006] Therefore, the present disclosure proposes an information processing device, an information processing method, and an information processing program that enable proofreading of music as intended by the user.

[0007] In order to solve the above problem, an information processing device according to one embodiment of the present disclosure includes a generation unit that generates candidates for calibration points for first performance data by inputting first performance data, the first performance data including a range specified by a user from which candidates for calibration points are extracted, into a first trained model that takes performance data as input and outputs candidates for calibration points, and a reception unit that receives feedback on the candidates for calibration points, and the generation unit generates second performance data in which the specified calibration points have been calibrated based on the feedback.

[0008] FIG. 1 is a diagram illustrating an example of information processing according to the present disclosure. FIG. 2 is a diagram illustrating proofreading of performance data in the prior art. FIG. 3 is a diagram illustrating a process for proofreading performance data. FIG. 4 is a diagram illustrating an example configuration of an information processing device according to an embodiment. FIG. 5 is a diagram illustrating details of an inpainting model according to an embodiment. FIG. 6 is a diagram illustrating details of a feedback model according to an embodiment. FIG. 7 is a diagram illustrating an inference phase according to an embodiment. FIG. 8 is a diagram illustrating an information processing procedure according to an embodiment. FIG. 9 is a diagram illustrating an information processing procedure for repeated proofreading. FIG. 10 is a hardware configuration diagram illustrating an example of a computer that realizes the functions of an information processing device.

[0009] Hereinafter, embodiments of the present disclosure will be described in detail with reference to the drawings. In the following embodiments, the same components are designated by the same reference numerals, and redundant description will be omitted.

[0010] The present disclosure will be described in the following order of items: 1. Embodiment 1-1. Overview of information processing according to an embodiment of the present disclosure 1-2. Configuration of an information processing device according to an embodiment 2. Details of the learning phase according to an embodiment 2-1. Details of learning of an inpainting model 2-2. Details of learning of a feedback model 3. Details of the inference phase according to an embodiment 4. Information processing procedure according to an embodiment 5. Other embodiments 6. Effects of an information processing device according to the present disclosure 7. Hardware configuration

[0011] (1. Embodiment) (1-1. Overview of Information Processing According to an Embodiment of the Present Disclosure) First, an overview of information processing according to an embodiment of the present disclosure will be described using FIG. 1. FIG. 1 is a diagram showing an example of information processing according to the embodiment of the present disclosure. The information processing according to the embodiment of the present disclosure is realized by an information processing device 100 shown in FIG. 1.

[0012] The information processing device 100 is a device that executes information processing according to the present disclosure, and is, for example, an information processing terminal or a server device.

[0013] In an embodiment, the information processing device 100 has a feedback model for outputting candidates for calibration points in performance data. The feedback model is a learned model that is trained to input performance data and output candidates for calibration points. For example, the performance data is information related to a piece of music or the like. Specifically, the performance data is MIDI (Musical Instruments Digital Interface) data that records information such as which pitches are played, how hard they are played, and how long they are played.

[0014] The information processing device 100 also has an inpainting model for outputting calibrated performance data. The inpainting model is a learned model that is trained to receive performance data as input and output calibrated performance data.

[0015] Next, a flow of information processing according to an embodiment of the present disclosure will be described. As shown in Fig. 1, the information processing device 100 receives, from a user, a specification of a range 15 for extracting candidates for calibration points in MIDI data 11. Note that in the MIDI data 11, MIDI data 12, MIDI data 13, and MIDI data 14, the vertical axis indicates pitch and the horizontal axis indicates time.

[0016] The information processing device 100 displays MIDI data 12 including notes 21, 22, and 23 that are candidates for calibration points (step S1). Specifically, the information processing device 100 uses the MIDI data 11 in the specified range 15 as input to a feedback model, and outputs MIDI data 12 that includes candidates for calibration points in the specified range 15. The information processing device 100 displays the output MIDI data 12.

[0017] The information processing device 100 displays the candidates for calibration points included in the displayed MIDI data 12 in a ranking format in the order in which they should be calibrated. For example, if the candidates for calibration points are musical notes (sounds), the information processing device 100 changes the display format of the sounds of each note for each ranking and displays them.

[0018] Specifically, the information processing device 100 displays the top 15 candidate notes 21 in the ranking of the proofreading portion to be proofread. The information processing device 100 also displays the top 15 to 30 candidate notes 22 in the ranking, and displays the top 30 to 45 candidate notes 23 in the ranking.

[0019] The process of generating the rankings is performed using, for example, a feedback model, specifically an encoder-only Transformer architecture that tokenizes MIDI data and classifies the resulting musical tokens as fake or authentic.

[0020] The feedback model outputs, for example, music tokens classified as fakes as candidates for proofreading. The information processing device 100 ranks the music tokens classified as fakes by the feedback model in order of the highest probability of being fake to the lowest probability of being fake.

[0021] The information processing device 100 receives from the user a specification of a portion to be proofread from among the candidate portions to be proofread (step S2). For example, the information processing device 100 receives the top 15 to 30 candidate notes 22 and the top 30 to 45 candidate notes 23 in the ranking of the portion to be proofread as the notes 24 to be proofread.

[0022] The information processing device 100 receives the MIDI data 13 including the designation of the portion to be calibrated as an input to the inpainting model, and outputs the calibrated MIDI data 14 (step S3). Specifically, the information processing device 100 receives the MIDI data 13 including the notes 24 of the portion to be calibrated as an input to the inpainting model, and outputs the MIDI data 14 including the notes 25 obtained by calibrating the notes 24 of the portion to be calibrated. Note that the portions not accepted as portions to be calibrated are output as they are without being calibrated.

[0023] Next, we will explain the proofreading of performance data in the prior art. It is conceivable that a person can proofread each desired part of performance data one by one. However, proofreading each part by one by a person is cumbersome and difficult in terms of the amount of work time.

[0024] Therefore, in conventional technology, performance data is sometimes calibrated using a machine learning model. Figure 2 is a diagram illustrating the calibration of performance data in conventional technology. As shown in Figure 2, the conventional technology inputs MIDI data 61 including a range 63 to be calibrated into a trained machine learning model, thereby outputting calibrated MIDI data 62.

[0025] In the conventional technology, when a range to be calibrated is selected, calibrated MIDI data is output, and therefore it is not possible to make fine adjustments such as presenting candidate calibration points or accepting the user's choice of whether or not to calibrate. For this reason, in the conventional technology, it is difficult to perform calibration that incorporates the user's intentions, and there are cases where performance data is calibrated that differs from the user's intentions. Note that, for example, if the range to be calibrated is multiple bars, such as 10 or 20 bars, the calibration points are each note included in those multiple bars.

[0026] On the other hand, the information processing device 100 of the present disclosure presents candidates for correction to the user during the process of proofreading performance data, and the user can specify the parts to be proofread, thereby reflecting the user's intentions in the proofreading of the performance data.

[0027] Next, returning to the description of the information processing device 100 of the present disclosure, Fig. 3 is a diagram showing the process of correcting performance data. In the following description, an example of repeatedly executing the process of correcting performance data will be described with reference to Fig. 3.

[0028] First, a first calibration process 51 will be described. The information processing device 100 receives from the user a range 35 for extracting calibration location candidates from the performance data 41 (step S11). For example, the information processing device 100 receives, as the range 35 for extracting calibration location candidates, a range of music tokens 42 obtained by tokenizing MIDI data, which is performance data.

[0029] The information processing device 100 inputs the music token 42 including the range 35 from which candidates for the calibration part are extracted to the inpainting model 31 (step S12). The information processing device 100 outputs the music token 43 obtained by calibrating the music token 42 in the range 35 from which candidates for the calibration part are extracted from the inpainting model 31 (step S13).

[0030] The information processing device 100 inputs the music token 43 into the feedback model 32 (step S14). The information processing device 100 outputs a music token 44 including a calibration portion candidate 36 from the feedback model 32 (step S15). Note that the calibration portion candidate may be specified from the MIDI data corresponding to the music token.

[0031] The information processing device 100 receives from the user a designation of a portion to be proofread from the proofreading portion candidates 36. For example, the information processing device 100 receives from the user a designation of all of the proofreading portion candidates 36 as a designation of a portion to be proofread.

[0032] The information processing device 100 inputs the music token 44, for which the portion to be proofread has been specified, into the inpainting model 31 (step S16). The information processing device 100 outputs the music token 45, for which the portion to be proofread has been proofread, from the inpainting model 31 (step S17).

[0033] Next, the second proofreading process 52 will be described. As in the first proofreading process, the information processing device 100 inputs the proofread music tokens 45 into the feedback model 32 (step S18). The information processing device 100 outputs proofreading location candidates 37 for the input music tokens 46 from the feedback model 32 (step S19).

[0034] The information processing device 100 receives from the user a designation of a portion to be proofread from the candidate proofreading portion 37. For example, the information processing device 100 receives from the user a designation of all of the candidate proofreading portion 37 as a designation of a portion to be proofread.

[0035] The information processing device 100 inputs the music token 46, for which the designation of the portion to be proofread has been accepted, into the inpainting model 31 (step S20). The information processing device 100 outputs the music token 47, for which the proofread portion has been proofread, from the inpainting model 31 (step S21).

[0036] The information processing device 100 repeats the above-described proofreading process using the output music tokens. Note that, for example, the information processing device 100 reduces and displays fewer candidates for proofreading parts each time the process of proofreading the performance data is repeated. Specifically, in the second or subsequent proofreadings, the information processing device 100 may display to the user music tokens that include fewer candidates for proofreading parts than the music tokens displayed to the user in the first proofreading. Furthermore, in the third or subsequent proofreadings, the information processing device 100 may display to the user music tokens that include fewer candidates for proofreading parts than the music tokens displayed to the user in the second proofreading, or MIDI data corresponding to the music tokens.

[0037] For example, the information processing device 100 reduces and displays the candidates for calibration locations using data that ranks the candidates for calibration locations output by the feedback model. Specifically, if 50 candidates for calibration locations are displayed in the first calibration, the information processing device 100 reduces and displays the candidates for calibration locations, for example, by displaying 40 candidates for calibration locations in the second calibration. Note that the number of candidates for calibration locations shown in the example is merely an example and is not limited to this number.

[0038] In this way, the information processing device 100 can correct the performance data as intended by the user by reflecting the user's feedback in the correction of the performance data. Furthermore, by repeatedly correcting the performance data, the information processing device 100 can provide correction of the performance data that better reflects the user's intentions.

[0039] (1-2. Configuration of Information Processing Apparatus According to Embodiment) Next, the configuration of the information processing apparatus 100 according to the embodiment will be described with reference to Fig. 4. Fig. 4 is a diagram showing an example of the configuration of the information processing apparatus 100 according to the embodiment.

[0040] 4, the information processing device 100 includes a communication unit 110, a storage unit 120, and a control unit 130. The information processing device 100 may also include an input unit (e.g., a touch panel) that accepts various operations from a user operating the information processing device 100, and a display unit (e.g., a liquid crystal display) that displays various information.

[0041] The communication unit 110 is realized by, for example, a network interface card (NIC), etc. The communication unit 110 is connected to a network (such as the Internet, near field communication (NFC), or Bluetooth (registered trademark)) via a wired or wireless connection, and transmits and receives information to and from other devices, etc. via the network.

[0042] The storage unit 120 is realized by, for example, a semiconductor memory element such as a random access memory (RAM) or a flash memory, or a storage device such as a hard disk or an optical disk. As shown in FIG. 4 , the storage unit 120 includes a trained model storage unit 121.

[0043] The storage unit 120 stores data used in this embodiment, such as performance data.

[0044] The trained model storage unit 121 stores trained models. For example, the trained model storage unit 121 stores an inpainting model that receives performance data as input and outputs corrected performance data. The trained model storage unit 121 also stores a feedback model that receives performance data as input and outputs candidates for correction points.

[0045] The control unit 130 is realized, for example, by a central processing unit (CPU), a micro processing unit (MPU), or the like executing a program (for example, an information processing program according to the present disclosure) stored inside the information processing device 100 using a random access memory (RAM) or the like as a working area. The control unit 130 is a controller, and may be realized, for example, by an integrated circuit such as an application specific integrated circuit (ASIC) or a field programmable gate array (FPGA).

[0046] The control unit 130 has a receiving unit 131, a generating unit 132, a display control unit 133, and a learning unit 134, and realizes or executes the functions and actions of the information processing described below. Note that the internal calibration of the control unit 130 is not limited to the calibration shown in FIG. 4, and may be any other calibration that performs the information processing described below.

[0047] In the following explanation, an example will be given in which the first trained model is a feedback model and the second trained model is an inpainting model.

[0048] In this disclosure, for the sake of distinction, the performance data may be referred to as first performance data, second performance data, third performance data, and fourth performance data. When no distinction is made, the performance data may simply be referred to as performance data. Note that when performance data is proofread multiple times, the first performance data, second performance data, third performance data, and fourth performance data may each be performance data that has been proofread once.

[0049] The receiving unit 131 receives from the user a designation of a range for extracting candidates for calibration parts of the first performance data. For example, the receiving unit 131 receives candidates for calibration parts of the first performance data from MIDI data. Specifically, the receiving unit 131 receives from the user a designation of a range for calibration of a music token corresponding to the MIDI data for which the designation of the range for calibration has been received. Note that the MIDI data includes information such as pitch, sound intensity, and sound duration. The first performance data is performance data, such as MIDI data or music tokens obtained by tokenizing the MIDI data.

[0050] The receiving unit 131 receives feedback on the candidate calibration points. The candidate calibration points may be points to be calibrated within a specified range of MIDI data, such as candidate notes. The candidate calibration points may also be points to be calibrated, such as candidates that suggest candidates for what kind of notes to calibrate.

[0051] For example, the receiving unit 131 receives, as feedback, a designation of a calibration portion. Specifically, the receiving unit 131 receives a designation of whether or not to calibrate a portion presented as a candidate for calibration. The receiving unit 131 may also receive a designation of how much to change a portion presented as a candidate for calibration compared with the first performance data before calibration.

[0052] For example, the receiving unit 131 receives, as feedback, a designation of the percentage of the first performance data to be retained. Specifically, the receiving unit 131 receives a range of 0% to 100% of the degree to which the pre-proofreaded first performance data is to be retained. If the receiving unit 131 receives 0% of the degree, it accepts all of the candidate calibration points as the calibration points. If the receiving unit 131 receives 50% of the degree, it accepts half of the candidate calibration points as the calibration points. For example, the receiving unit 131 receives, within a range of 0% to 100% of the degree to which the pre-proofreaded first performance data is to be changed for the points presented as the candidate calibration points.

[0053] For example, the receiving unit 131 receives feedback according to the ranking. Specifically, the receiving unit 131 receives, as the correction points, the top k points in the ranking of the points to be corrected, generated by the generating unit 132 (described later). Note that the number k may be any value selected by the user.

[0054] For example, the accepting unit 131 accepts a specification of a calibration portion of the second performance data from candidates for the calibration portion of the second performance data. For example, the second performance data is calibrated performance data, such as MIDI data or musical tokens obtained by tokenizing the MIDI data.

[0055] For example, the receiving unit 131 receives a designation of a calibration portion of the second performance data from among candidates for calibration portions of the second performance data for the second or subsequent calibration. Furthermore, when performing multiple calibrations, the receiving unit 131 may receive a designation of a calibration portion each time. Note that the calibrated performance data is not limited to performance data obtained by calibrating the first performance data, but may be performance data calibrated by the second or subsequent calibration.

[0056] The receiving unit 131 receives feedback on candidates for calibration parts for the performance data to be repeatedly proofread. For example, the receiving unit 131 receives designation of calibration parts from the user in order to proofread the performance data that has been proofread for the second time.

[0057] The receiving unit 131 may receive from the user the number of times calibration is to be repeated when the calibration of the performance data is to be repeated. Note that the number of times calibration of the performance data is to be repeated may be set to an arbitrary number by default or the like.

[0058] The generation unit 132 generates candidates for calibration points of the first performance data by inputting the first performance data, which includes a range specified by the user and from which candidates for calibration points are extracted, into a first trained model that takes the performance data as input and outputs candidates for calibration points.

[0059] For example, the generation unit 132 inputs performance data within a range from which candidates for calibration points received from the user are extracted into a feedback model, and generates candidates for calibration points within the range from which candidates for calibration points received from the user are extracted.

[0060] Furthermore, the generating unit 132 generates second performance data in which the designated calibration points have been calibrated based on the feedback. For example, the designated calibration points include calibration points designated by the user, which will be described later, and calibration points designated as a result of determining the points to be calibrated according to a designated ratio.

[0061] For example, the generating unit 132 generates second performance data in which the designated calibration portion has been calibrated. Specifically, the generating unit 132 inputs the calibration portion designated by the user into a trained model or the like that uses performance data as input and outputs calibrated performance data, thereby generating the calibrated second performance data.

[0062] For example, the generation unit 132 determines the parts to be corrected according to the specified percentage and generates the second performance data. Specifically, when the generation unit 132 receives 0 percent, it determines all of the candidates for correction as the parts to be corrected and generates the second performance data. When the generation unit 132 receives 50 percent, it determines half of the candidates for correction as the parts to be corrected and generates the second performance data. Note that the parts to be corrected may be determined by any method, such as randomly. For example, the parts to be corrected may be determined based on a ranking of the candidates for correction according to the percentage of the first performance data received by the receiving unit 131 to be retained. Specifically, when the candidates for correction are 50 percent, the generation unit 132 may determine the top 50 percent of the candidates in the ranking as the candidates for correction.

[0063] For example, the generation unit 132 generates second performance data by inputting the first performance data for which feedback has been received into a second trained model that uses the performance data as input and calibrated performance data as output.

[0064] Specifically, the generating unit 132 inputs performance data including the calibration points designated by the user into an inpainting model, thereby generating performance data in which the designated calibration points have been calibrated.

[0065] For example, the generation unit 132 generates a ranking of candidates for calibration locations. Specifically, the generation unit 132 generates the candidates for calibration locations as a heat map. The generation unit 132 generates the heat map with the calibration locations to be calibrated ranked high.

[0066] For example, the generation unit 132 generates a ranking of the calibration location candidates based on the probability of whether or not the candidates correspond to the calibration location candidates output by the first trained model. Specifically, the generation unit 132 generates the ranking of the calibration location candidates using a feedback model that classifies music tokens obtained by tokenizing MIDI data as fake or real.

[0067] The generation unit 132 generates music tokens classified as fakes as candidates for proofreading portions. The generation unit 132 generates music tokens classified as fakes by the feedback model by ranking them in order of high probability of being fake to low probability of being fake.

[0068] For example, the generation unit 132 generates candidates for the calibration portion of the second performance data by inputting the second performance data to a first trained model, and the generation unit 132 generates candidates for the calibration portion of the second performance data by inputting the second performance data to a feedback model.

[0069] For example, the generation unit 132 generates third performance data by inputting the second performance data, for which the calibration portion has been specified, into a second trained model. Similarly to generating the second performance data, the generation unit 132 generates the third performance data by inputting the portion of the second performance data to be calibrated into an inpainting model. In this way, the generation unit 132 further calibrates the calibrated performance data to generate performance data.

[0070] Furthermore, the generated performance data as the third performance data may be the first performance data or the second performance data.

[0071] For example, the generation unit 132 generates new performance data by repeating the process of generating fourth performance data from the second trained model based on candidate calibration points obtained by further inputting the third performance data into the first trained model, and the process of further inputting the fourth performance data into the first trained model to obtain candidate calibration points.

[0072] Specifically, the generation unit 132 generates candidates for calibration parts by further inputting the calibrated performance data into a feedback model. The generation unit 132 also generates calibrated performance data by inputting the candidates for calibration parts into an inpainting model without receiving feedback such as a designation of a calibration part from the user. In this way, the generation unit 132 generates performance data for the calibrated performance data without receiving feedback from the user.

[0073] The generating unit 132 may also execute other processes to calibrate the performance data. For example, when repeating the process of calibrating the performance data, the generating unit 132 may determine whether the calibration has been repeated the number of times accepted by the accepting unit 131, and generate the calibrated performance data.

[0074] The display control unit 133 displays candidates for calibration parts in the first performance data. For example, the display control unit 133 highlights and displays the candidates for calibration parts in the first performance data by ranking. Specifically, the display control unit 133 displays the generated heat map. For example, the display control unit 133 displays the candidates in darker colors in order of the candidates that need to be calibrated.

[0075] For example, the display control unit 133 controls the number of candidates for proofreading parts to be displayed based on the ranking. Specifically, the display control unit 133 changes the number of candidates for proofreading parts to be displayed using a slider bar or the like. For example, when displaying the top k rankings, the display control unit 133 hides all but the top k rankings.

[0076] The training unit 134 uses the performance data set to train a second trained model so that the second trained model receives performance data as input and outputs corrected performance data. For example, the training unit 134 trains an inpainting model so that the second trained model can correct performance data that is partially missing. Details of the training of the inpainting model will be described later.

[0077] For example, the learning unit 134 uses the calibrated performance data output by the second trained model to train the first trained model so that the first trained model outputs candidates for calibration parts using the performance data as input. Specifically, the learning unit 134 trains the feedback model so that the first trained model can determine, from the calibrated performance data output by the inpainting model, uncalibrated parts as "genuine" and calibrated parts as "fake." Details of the learning of the feedback model will be described later.

[0078] (2. Details of the Learning Phase According to the Embodiment) In the following description, details of the learning of the feedback model as the first trained model described above and details of the learning of the inpainting model as the second trained model will be described.

[0079] (2-1. Details of Inpainting Model Training) The inpainting model is a trained model that takes performance data as input and outputs calibrated performance data. The inpainting model also includes an encoder that converts input data into a latent representation and a decoder that predicts the final output from the latent representation.

[0080] The training unit 134 trains the inpainting model so that missing parts of the performance data indicated by the music tokens can be predicted or supplemented by masking the music tokens. The music tokens include information related to the pitch, the start time of the note (sound), and the end time of the note (sound).

[0081] 5 is a diagram illustrating details of an inpainting model 151 according to an embodiment. As shown in FIG. 5, the training unit 134 trains an inpainting model 503 using musical tokens 501, which are performance data randomly sampled from a performance data set.

[0082] The learning unit 134 corrects the missing portion 511 contained in the masked music token 502, and learns the inpainting model 503 so as to output a music token 504 including a music token 512 with the missing portion corrected. Note that a missing portion refers to a portion where a part of a music token is missing.

[0083] The music token 502 is a randomly masked music token from the music token 501, which is performance data randomly sampled from the performance data set.

[0084] The learning unit 134 uses the cross entropy (CE LOSS) 505 calculated from the output of the inpainting model 503 to learn the inpainting model by gradient descent.

[0085] (2-2. Details of training the feedback model) The feedback model is a trained model that takes performance data as input and outputs candidates for calibration points. The feedback model includes an encoder that classifies musical tokens as fake or real.

[0086] In the following description, an example of the flow of learning a feedback model will be described with reference to FIG. 6. FIG. 6 is a diagram for explaining details of a feedback model 605 according to an embodiment. FIG. 6 is a diagram showing an overview 152 of feedback model learning. The learning unit 134 learns the feedback model using a learned inpainting model 603. The inpainting model 603 receives as input a music token 601 including a music token 611 of a portion to be proofread specified from a range 602 for extracting a proofread portion, and outputs a music token 604 including a proofread music token 612.

[0087] The musical tokens 604 output by the inpainting model 603 include proofread musical tokens 612 and unproofread musical tokens. For example, the inpainting model 603 labels the proofread musical tokens 612 as "fake" and the unproofread musical tokens as "real."

[0088] The feedback model 605 evaluates the music tokens 604 output by the inpainting model 603 and classifies them as "real" or "fake." The feedback model outputs the resulting music tokens 606 as "real" (R in the figure) or "fake" (F in the figure).

[0089] The learning unit 134 calculates a loss function from the classified output results and learns a feedback model to minimize the loss function using the gradient descent method.

[0090] The feedback model can generate a heat map for the designated calibration points through learning. The heat map indicates the probability that each music token is "genuine" or "fake" calculated using a binary cross entropy (BCE) loss function. For example, the generation unit 132 may generate a heat map with a darker color for the designated calibration points, indicating that the higher the probability of the music token being "fake." Furthermore, the generation unit 132 may generate a heat map with a lighter color for the designated calibration points, indicating that the lower the probability of the music token being "fake."

[0091] (3. Details of the Inference Phase According to the Embodiment) Next, the inference phase according to the embodiment will be described with reference to FIG. 7. FIG. 7 is a diagram illustrating the inference phase according to the embodiment. FIG. 7 is a diagram illustrating an outline 153 of inference between a feedback model and an inpainting model. Note that, although the following description will be given of an example in which the calibration process of performance data is repeated multiple times, calibration of performance data may be performed only once.

[0092] The receiving unit 131 receives from the user a range 702 to be extracted as a proofreading portion. The range 702 to extract the proofreading portion includes a music token 751 to be proofread. The generating unit 132 inputs the music token 701 including the music token 751 in the range to be extracted as a proofreading portion to an inpainting model 721, and generates a music token 703 including a proofread music token 752.

[0093] The feedback model 722 receives as input music tokens 703 including music tokens 752 generated by the inpainting model 721, and outputs music tokens 704 including music tokens containing candidates for proofreading portions.

[0094] The feedback model 722 outputs the music token 704 as either "real" (R in the figure) or "fake" (F in the figure). For example, the generation unit 132 generates "fake" as a candidate for the proofreading portion.

[0095] The receiving unit 131 receives a designation of a calibration portion from the calibration portion candidates. Note that the receiving unit 131 may receive the designation of a calibration portion from a user using MIDI data corresponding to a music token. The generating unit 132 inputs a music token 705 including a music token 753 of the designated calibration portion to the inpainting model 721, and generates a music token 706 including a calibrated music token 754.

[0096] The generation unit 132 inputs the music tokens 706, including the proofread music tokens 754, into the feedback model 722 to generate music tokens 707, including music tokens containing candidates for proofreading. In this way, in the inference phase, the process of proofreading the performance data may be repeated. Note that when the generation unit 132 repeatedly proofreads the performance data, it may generate the proofread performance data without receiving feedback from the user.

[0097] The generation unit 132 may generate and gradually reduce the number of candidates for calibration locations each time the process of calibrating the candidates for calibration locations output by the feedback model is repeated. Alternatively, the display control unit 133 may display and gradually reduce the number of candidates for calibration locations each time the process of calibrating the number of candidates for calibration locations output by the feedback model is repeated.

[0098] After repeating the above-described process of proofreading the performance data multiple times, the inpainting model receives as input a music token 708 including a music token 755 of the specified proofread portion, and outputs a music token 709 including a proofread music token 756. The proofreading process may be repeated any number of times predetermined by default settings, or may be repeated a number of times received from the user.

[0099] (4. Information Processing Procedure According to the Embodiment) Next, an information processing procedure of the information processing device 100 according to the embodiment will be described. Fig. 8 is a diagram showing the information processing procedure according to the embodiment.

[0100] 8, the receiving unit 131 of the information processing device 100 determines whether a selection of a portion to be corrected from the performance data has been received (step S101). The receiving unit 131 repeats the process of step S101 until a selection of a portion to be corrected is received (step S101: No).

[0101] When the generation unit 132 receives the selection of the portion to be calibrated (step S101: Yes), the generation unit 132 uses a feedback model to generate candidates for the portion to be calibrated from the range selected as the portion to be calibrated (step S102).

[0102] The display control unit 133 displays the generated candidates for the calibration part (step S103). The receiving unit 131 determines whether or not a designation of a part to be calibrated has been received from the displayed candidates for the calibration part (step S104).

[0103] The receiving unit 131 repeats the process of step S104 until it receives a designation of a portion to be calibrated (step S104: No). When it receives a designation of a portion to be calibrated (step S104: Yes), the generating unit 132 uses the inpainting model to generate performance data in which the designated portion to be calibrated is calibrated (step S105).

[0104] Next, the process of repeated calibration performed by the information processing device 100 will be described with reference to Fig. 9. Fig. 9 is a diagram showing the information processing procedure when repeated calibration is performed.

[0105] The processes from step S101 to step S105 are the same as those described with reference to FIG. 8, and therefore will not be described again.

[0106] 9, the generation unit 132 of the information processing device 100 determines whether the iterative process has been executed a predetermined number of times (step S201). If the iterative process has been executed a predetermined number of times (step S201: Yes), the generation unit 132 ends the process.

[0107] On the other hand, if the repetitive process has not been executed a predetermined number of times (step S201: No), the accepting unit 131 of the information processing device 100 accepts the same part as in the previous calibration process as the part to be calibrated (step S202).

[0108] The processing from step S101 to step S201 after step S202 is a process of repeating the same processing as described above, and therefore the description thereof will be omitted.

[0109] (5. Other Embodiments) The processing according to each of the above-described embodiments may be implemented in various different forms other than the above-described embodiments.

[0110] Furthermore, among the processes described in the above embodiments, all or part of the processes described as being performed automatically can be performed manually, or all or part of the processes described as being performed manually can be performed automatically using known methods. Furthermore, the information, including the processing procedures, specific names, various data, and parameters shown in the above documents and drawings, can be changed as desired unless otherwise specified. For example, the various information shown in each drawing is not limited to the information shown in the drawings.

[0111] Furthermore, the components of each device shown in the figure are conceptual functional components and do not necessarily have to be physically configured as shown in the figure. In other words, the specific form of distribution and integration of each device is not limited to that shown in the figure, and all or part of them can be functionally or physically distributed and integrated in any unit depending on various loads, usage conditions, etc.

[0112] Furthermore, the above-described embodiments and modifications can be combined as appropriate within the scope of not causing any contradiction in the processing content.

[0113] Furthermore, the effects described in this specification are merely examples and are not limiting, and other effects may also be present.

[0114] (6. Effects of Information Processing Device According to the Present Disclosure) As described above, the information processing device according to the present disclosure (information processing device 100 in the embodiment) includes a receiving unit (receiving unit 131 in the embodiment), a generating unit (generating unit 132 in the embodiment), a display control unit (display control unit 133 in the embodiment), and a learning unit (learning unit 134 in the embodiment). The generating unit generates candidates for the calibration location of the first performance data by inputting the first performance data, which is a range specified by a user and includes a range from which candidates for calibration are extracted, to a first trained model that receives the performance data as input and outputs candidates for calibration location. The receiving unit receives feedback on the candidates for calibration location. Furthermore, the generating unit generates second performance data in which the specified calibration location is calibrated based on the feedback.

[0115] In this way, the information processing device according to the present disclosure presents candidates for correction and corrects the specified correction points, thereby reducing the burden on the user for proofreading. The information processing device is characterized by receiving feedback during the process of proofreading performance data. This allows the user to reflect their intentions in the proofreading of the performance data. This allows the information processing device to proofread a piece of music as intended by the user.

[0116] The receiving unit receives, as feedback, a designation of a calibration portion, and the generating unit generates second performance data in which the designated calibration portion is calibrated.

[0117] In this way, the information processing device can accept fine adjustments to the parts to be proofread, and can reflect the user's intentions regarding the proofreading.

[0118] The display control unit displays candidates for the calibration part in the first performance data. In this way, by presenting candidates for the calibration part, the information processing device can improve the accuracy of feedback on the calibration from the user.

[0119] The generating unit generates a ranking of the candidates for the proofreading portion, and the receiving unit receives feedback according to the ranking.

[0120] In this way, the information processing device can improve usability for proofreading music by ranking candidates for proofreading parts.

[0121] The generation unit also generates a ranking of the candidates for the correction portion based on the probability of whether or not the candidate corresponds to a candidate for the correction portion output by the first trained model.

[0122] In this way, the information processing device can generate a ranking of candidates for calibration points with high accuracy by using the output of the first trained model (e.g., feedback model).

[0123] The generation unit also generates second performance data by inputting the first performance data for which feedback has been received into a second trained model that uses the performance data as input and calibrated performance data as output.

[0124] In this way, the information processing device can generate highly accurate calibrated performance data by using the second trained model (e.g., the inpainting model in this embodiment). Furthermore, by using the inpainting model, the information processing device can quickly provide the user with calibrated performance data.

[0125] The generating unit generates candidates for the calibration portion of the second performance data by inputting the second performance data to the first trained model, and the accepting unit accepts a designation of the calibration portion of the second performance data from the candidates for the calibration portion of the second performance data.

[0126] In this way, the information processing device can receive further proofreading of a piece of music that has already been proofread, and by repeating the proofreading, it is possible to proofread the piece of music with higher accuracy.

[0127] The generating unit also generates third performance data by inputting the second performance data, for which the calibration portion has been specified, into the second trained model.

[0128] In this way, the information processing device can proofread a piece of music with higher accuracy by repeatedly proofreading the piece of music that has already been proofread.

[0129] The generation unit also generates new performance data by repeating the process of generating fourth performance data from the second trained model based on candidate calibration points obtained by further inputting the third performance data into the first trained model, and the process of further inputting the fourth performance data into the first trained model to obtain candidate calibration points.

[0130] In this way, the information processing device can reduce the burden on the user and perform highly accurate proofreading of music by omitting the reception of feedback from the user in the process of repeating proofreading.

[0131] The display control unit also highlights and displays the candidates for the calibration part in the first performance data by ranking.

[0132] In this way, the information processing device can easily allow the user to specify the part to be proofread by highlighting the part to be proofread.

[0133] Furthermore, the display control unit controls and displays the number of candidates for the proofreading portion based on the ranking.

[0134] In this way, the information processing device controls and displays the number of candidates for correction points, thereby enabling even a user who is unfamiliar with music proofreading to use the music proofreading function.

[0135] The learning unit uses the performance data set to learn the second trained model so that the second trained model receives the performance data as input and outputs calibrated performance data.

[0136] In this way, the information processing device can use the performance data set to learn how to generate various performance data, thereby enabling highly accurate proofreading of music.

[0137] In addition, the learning unit uses the calibrated performance data output by the second trained model to train the first trained model so that it takes the performance data as input and outputs candidates for calibration points.

[0138] In this way, the information processing device can efficiently perform learning by using the output of the inpainting model for learning the feedback model.

[0139] The receiving unit also receives, as feedback, a designation of a proportion of the first performance data to be retained. The generating unit determines a portion to be calibrated in accordance with the designated proportion, and generates the second performance data.

[0140] In this way, the information processing device can provide a UI that allows easy proofreading and proofreading of music that reflects the user's intentions.

[0141] (7. Hardware Configuration) The information processing device 100 according to the embodiment of the present disclosure described above is realized by, for example, a computer 1000 configured as shown in FIG. 10 . The information processing device 100 will be described as an example. FIG. 10 is a hardware configuration diagram showing an example of the computer 1000 that realizes the functions of the information processing device 100. The computer 1000 has a processing circuitry 1100, a RAM 1200, a ROM 1300, a secondary storage device 1400, a communication interface 1500, an input / output interface 1600, a display unit 1700, a camera unit 1800, a microphone 1900, and a speaker 2000. The components of the computer 1000 are connected by a bus 1050.

[0142] The processing circuit 1100 operates and controls each unit based on programs stored in the ROM 1300 or the secondary storage device 1400. For example, the processing circuit 1100 loads the programs stored in the ROM 1300 or the secondary storage device 1400 into the RAM 1200 and executes processing corresponding to the various programs.

[0143] The ROM 1300 stores boot programs such as a Basic Input Output System (BIOS) that is executed by the processing circuit 1100 when the computer 1000 is started up, and programs that depend on the hardware of the computer 1000 .

[0144] The secondary storage device 1400 is a computer-readable recording medium that non-temporarily records programs executed by the processing circuit 1100 and data used by such programs. Specifically, the secondary storage device 1400 is a recording medium that records programs for each process of the information processing device 100 according to an embodiment of the present disclosure, which are examples of program data 1450.

[0145] The communication interface 1500 is an interface for connecting the computer 1000 to an external network 1550. The communication interface 1500 corresponds to the communication unit 110 provided in the information processing device 100. For example, the processing circuit 1100 receives data from other devices and transmits data generated by the processing circuit 1100 to other devices via the communication interface 1500.

[0146] The input / output interface 1600 is an interface for connecting the input / output device 1650 and the computer 1000. For example, the processing circuit 1100 receives data from an input device such as a microphone 1900 or a touch panel via the input / output interface 1600. The processing circuit 1100 also transmits data to an output device such as a display unit 1700 or a speaker 2000 via the input / output interface 1600. The input / output interface 1600 may also function as a media interface for reading programs and the like recorded on a predetermined recording medium. Examples of the media include optical recording media such as a DVD (Digital Versatile Disc) or a PD (Phase Change Rewritable Disc), magneto-optical recording media such as an MO (Magneto-Optical Disk), tape media, magnetic recording media, and semiconductor memories.

[0147] The display unit 1700 is an interface for displaying information processed by the computer 1000. The display unit 1700 is, for example, a liquid crystal display or an organic electroluminescence display (EL display). The display unit 1700 may also be a touch panel display device or a video projection device.

[0148] The camera unit 1800 is an interface through which the computer 1000 captures images. The microphone 1900 is an interface through which the computer 1000 captures audio. The speaker 2000 is an interface through which the computer 1000 outputs audio processed by the computer 1000. The components of the computer 1000 are connected by a bus 1050. The interfaces do not necessarily need to be provided inside the computer 1000, but may be provided outside the computer 1000 via a network or the like. Furthermore, the components constituting the computer 1000 may be controlled by a circuit different from the processing circuit 1100. For example, the display unit 1700 may be controlled not by the processing circuit 1100 but by a circuit dedicated to display processing provided in the display unit 1700.

[0149] For example, when the computer 1000 functions as the information processing device 100 according to an embodiment of the present disclosure, the processing circuit 1100 of the computer 1000 functions as the control unit 130 by executing a program loaded onto the RAM 1200. The secondary storage device 1400 stores the information processing program according to the present disclosure and various data stored in the storage device 120. The processing circuit 1100 reads and executes program data 1450 from the secondary storage device 1400. Alternatively, the processing circuit 1100 may obtain these programs from another device via an external network 1550. That is, the secondary storage device 1400 does not need to be located inside the computer 1000, but may also be located outside the computer 1000. The processing circuit 1100 is an example of an integrated circuit, and a CPU, an MPU, a GPU, an APU, an ASIC, and an FPGA can all be considered to be integrated circuits.

[0150] The present technology can also be configured as follows. (1) An information processing device comprising: a generation unit that generates candidates for calibration locations for the first performance data by inputting first performance data, the first performance data including a range specified by a user and from which candidates for calibration locations are extracted, to a first trained model that takes performance data as input and outputs candidates for calibration locations; and a reception unit that receives feedback on the candidates for calibration locations, wherein the generation unit generates second performance data in which the specified calibration locations have been corrected based on the feedback. (2) The information processing device described in (1), wherein the reception unit receives designation of calibration locations as the feedback, and the generation unit generates the second performance data in which the specified calibration locations have been corrected. (3) The information processing device described in (1) or (2), further comprising a display control unit that displays candidates for calibration locations in the first performance data. (4) The information processing device described in (3), wherein the generation unit generates a ranking of the candidates for calibration locations, and the reception unit receives feedback in accordance with the ranking. (5) The information processing device according to (4), wherein the generation unit generates a ranking of the candidates for calibration location based on a probability of whether or not the candidates correspond to the candidates for calibration location output by a first trained model. (6) The information processing device according to any one of (1) to (5), wherein the generation unit generates the second performance data by inputting the first performance data for which the feedback has been received into a second trained model that outputs performance data calibrated using performance data as input. (7) The information processing device according to (6), wherein the generation unit generates candidates for calibration location of the second performance data by inputting the second performance data into the first trained model, and the receiving unit receives designation of calibration location of the second performance data from the candidates for calibration location of the second performance data. (8) The information processing device according to (7), wherein the generation unit generates third performance data by inputting the second performance data for which the designation of calibration location has been received into the second trained model.(9) The information processing device according to (8), wherein the generation unit generates new performance data by repeating a process of generating fourth performance data from the second trained model based on candidates for calibration locations obtained by further inputting the third performance data into the first trained model, and a process of further inputting the fourth performance data into the first trained model to obtain candidates for calibration locations. (10) The information processing device according to (4), wherein the display control unit highlights and displays the candidates for calibration locations among the first performance data by ranking. (11) The information processing device according to (4), wherein the display control unit controls and displays the number of candidates for calibration locations based on the ranking. (12) The information processing device according to (6), further comprising a learning unit that uses a performance dataset to train the second trained model so as to input performance data and output calibrated performance data. (13) The information processing device according to (12), wherein the learning unit uses the calibrated performance data output by the second trained model to train the first trained model so as to input performance data and output candidates for calibration parts. (14) The information processing device according to any one of (1) to (13), wherein the receiving unit receives, as the feedback, a designation of a proportion of the first performance data to be retained, and the generating unit determines parts to be calibrated according to the designated proportion, and generates second performance data. (15) An information processing method including: a computer generates candidates for calibration parts of the first performance data by inputting first performance data, the first performance data including a range designated by a user from which candidates for calibration parts are extracted, to a first trained model that takes performance data as input and outputs candidates for calibration parts; receiving feedback on the candidates for calibration parts; and generating second performance data in which the designated calibration parts are calibrated based on the feedback.(16) An information processing program for causing a computer to function as an information processing device, comprising: a generation unit that generates candidates for calibration points of first performance data by inputting first performance data, the first performance data including a range specified by a user and from which candidates for calibration points are extracted, to a first trained model that takes performance data as input and outputs candidates for calibration points; and a reception unit that receives feedback on the candidates for calibration points, wherein the generation unit generates second performance data in which the specified calibration points have been corrected based on the feedback.

[0151] REFERENCE SIGNS LIST 100 Information processing device 110 Communication unit 120 Storage unit 121 Trained model storage unit 130 Control unit 131 Reception unit 132 Generation unit 133 Display control unit 134 Learning unit

Claims

1. An information processing device comprising: a generation unit that generates candidates for calibration points for first performance data by inputting first performance data, the first performance data including a range specified by a user from which candidates for calibration points are extracted, into a first trained model that takes performance data as input and outputs candidates for calibration points; and a reception unit that receives feedback on the candidates for calibration points, wherein the generation unit generates second performance data in which the specified calibration points have been corrected based on the feedback.

2. The information processing device according to claim 1, wherein the receiving unit receives, as the feedback, a designation of a calibration portion, and the generating unit generates second performance data in which the designated calibration portion has been calibrated.

3. The information processing device according to claim 1, further comprising a display control unit that displays candidates for calibration points in the first performance data.

4. The information processing device according to claim 3, wherein the generating unit generates a ranking of the candidates for the proofreading portion, and the receiving unit receives feedback in accordance with the ranking.

5. The information processing device according to claim 4, wherein the generation unit generates a ranking of the candidates for the correction location based on the probability of whether or not the candidate corresponds to a candidate for the correction location output by the first trained model.

6. The information processing device according to claim 1, wherein the generation unit generates the second performance data by inputting the first performance data for which the feedback has been received into a second trained model that uses the performance data as input and calibrated performance data as output.

7. The information processing device of claim 6, wherein the generation unit generates candidates for calibration points of the second performance data by inputting the second performance data into the first trained model, and the reception unit receives a specification of the calibration points of the second performance data from the candidates for calibration points of the second performance data.

8. The information processing device according to claim 7, wherein the generation unit generates third performance data by inputting the second performance data, for which the calibration portion has been specified, into the second trained model.

9. The information processing device described in claim 8, wherein the generation unit generates new performance data by repeating a process of generating fourth performance data from the second trained model based on candidates for calibration points obtained by further inputting the third performance data into the first trained model, and a process of further inputting the fourth performance data into the first trained model to obtain candidates for calibration points.

10. The information processing device according to claim 4, wherein the display control unit displays the candidates for the calibration part in the first performance data in an emphasized manner according to ranking.

11. The information processing device according to claim 4, wherein the display control unit controls and displays the number of candidates for the proofreading portion based on the ranking.

12. The information processing device according to claim 6, further comprising a learning unit that uses a performance dataset to learn the second trained model so as to input performance data and output calibrated performance data.

13. The information processing device described in claim 12, wherein the learning unit uses the calibrated performance data output by the second trained model to train the first trained model so as to input the performance data and output candidates for calibration points.

14. The information processing device according to claim 1, wherein the receiving unit receives, as the feedback, a specification of a proportion of the first performance data to be retained, and the generating unit determines the parts to be calibrated according to the specified proportion and generates second performance data.

15. An information processing method comprising: a computer inputting first performance data, the first performance data including a range specified by a user from which candidates for calibration are to be extracted, into a first trained model that takes performance data as input and outputs candidates for calibration, thereby generating candidates for calibration of the first performance data; receiving feedback on the candidates for calibration; and generating second performance data in which the specified calibration points have been calibrated based on the feedback.

16. An information processing program for causing a computer to function as an information processing device, comprising: a generation unit that generates candidates for calibration points for first performance data by inputting first performance data, the first performance data including a range specified by a user from which candidates for calibration points are to be extracted, into a first trained model that takes performance data as input and outputs candidates for calibration points; and a reception unit that receives feedback on the candidates for calibration points, wherein the generation unit generates second performance data in which the specified calibration points have been corrected based on the feedback.

Images