Musical score recognition device, musical score recognition method, and program

Abstract

To provide a musical score recognition device, a musical score recognition method and a program for recognizing a musical score of a Japanese musical instrument.SOLUTION: A musical score recognition device 10 includes an image acquisition section 130 for acquiring image data indicating a musical score of a Japanese instrument, an image extraction section 140 for extracting a partial image indicating one sound from the image data, an image recognition section 150 for recognizing one or more symbols included in the partial image, and a scale identification section 160 for identifying a scale on the basis of a recognition result of the one or more symbols.SELECTED DRAWING: Figure 5

Description

[Technical Field]

[0001] The present disclosure relates to a musical score recognition device, a musical score recognition method, and a program. [Background technology]

[0002] There are known techniques for recognizing images of musical scores. For example, Patent Document 1 discloses a musical score recognition device that recognizes symbols from musical score image data and generates musical score data based on the recognized symbols. Patent Document 2 discloses a musical score recognition device that includes a normalization means that converts musical score figures into a fixed binary matrix, and a neural network that accepts the binary matrix output by the normalization means as input data and analyzes musical score symbols that fit the musical score figures. [Prior art documents] [Patent documents]

[0003] [Patent Document 1] Patent No. 5030144 [Patent Document 2] Japanese Patent Application Publication No. 3-249799 Summary of the Invention [Problem to be solved by the invention]

[0004] However, conventional techniques are targeted at widely used musical scores such as staff notation, and it is difficult to recognize musical scores for Japanese musical instruments.

[0005] One aspect of the present disclosure aims to provide a technology that enables recognition of musical scores for Japanese musical instruments. [Means for solving the problem]

[0006] A musical score recognition device according to one aspect of the present disclosure includes an image acquisition unit that acquires image data representing musical scores for Japanese musical instruments, an image extraction unit that extracts a partial image representing one note from the image data, an image recognition unit that recognizes one or more symbols included in the partial image, and a scale identification unit that identifies a scale based on the recognition results of the one or more symbols. [Effects of the Invention]

[0007] According to one aspect of the present disclosure, musical scores for Japanese musical instruments can be recognized. [Brief explanation of the drawings]

[0008] [Figure 1] FIG. 1 is a diagram showing an example of shamisen musical score. [Figure 2] FIG. 10 is a diagram showing an example of a grid in a musical score. [Figure 3] 1 is a block diagram showing an example of the overall configuration of a musical score recognition system. [Figure 4] FIG. 1 is a block diagram illustrating an example of a computer. [Figure 5] FIG. 2 is a block diagram illustrating an example of a functional configuration of the musical score recognition device. [Figure 6] 10 is a flowchart illustrating an example of a learning process. [Figure 7] 10 is a flowchart illustrating an example of a recognition process. DETAILED DESCRIPTION OF THE INVENTION

[0009] Hereinafter, embodiments of the present disclosure will be described with reference to the accompanying drawings. In this specification and drawings, components having substantially the same functional configurations are designated by the same reference numerals, and redundant description will be omitted.

[0010] [Embodiment] An embodiment of the present disclosure is an example of an information processing system that recognizes musical scores. Hereinafter, the information processing system according to this embodiment will be referred to as a "musical score recognition system." The musical score recognition system has a function of recognizing musical scores used to play Japanese musical instruments. Hereinafter, the musical scores to be recognized in this embodiment will also be referred to as "Japanese musical instrument scores."

[0011] Japanese musical instruments are instruments that have traditionally been used in Japan. Examples of Japanese musical instruments include the shamisen, koto, koto, biwa, and kokyu. Japanese musical instruments are not limited to stringed instruments, but may also include wind instruments such as the shakuhachi, and percussion instruments such as the taiko drum. In this embodiment, a configuration for recognizing sheet music for the shamisen, which is an example of a Japanese musical instrument, will be described.

[0012] FIG. 1 is a diagram showing an example of shamisen musical score. FIG. 1 shows an example of musical score written in a format called kateishikifu, which is a vertically written musical score also known as tatefu. As shown in FIG. 1, the Japanese musical instrument musical score 9 includes a title 91, a key 92, and notes 93. The Japanese musical instrument musical score 9 may also include lyrics 94.

[0013] The title 91 is the title of the piece of music written in the Japanese musical instrument score 9. The key 92 is the standard for the pitch of each of the three strings of a shamisen. The notes 93 are an area that describes the notes that make up the piece of music in chronological order. The lyrics 94 are the lyrics of the piece of music written in the Japanese musical instrument score 9, and are written in positions that correspond to the notes written in the notes 93. The title 91 and key 92 may also be written in the margins of the Japanese musical instrument score 9.

[0014] In home music, the notes 93 are separated by grids. Each grid in the notes 93 contains one or more symbols representing a note. The number of grids included in a sheet of music is arbitrary and may vary depending on the type of instrument, school, notation, composer, etc.

[0015] FIG. 2 is a diagram showing an example of a grid in a musical score. As shown in FIG. 2, a grid 95 representing a note includes one or more symbols. The one or more symbols include at least one of pitch, rest, playing technique, time signature, fingering instructions, or singing. Each symbol is represented by Chinese numerals, Arabic numerals, Chinese characters, symbols, or a combination of these. A note shown in the grid is represented by a combination of symbols included in the grid and the positional relationship of the symbols within the grid. The meaning of each symbol varies depending on the type of instrument, notation, school, composer, etc., but is written according to a single rule within a single musical score.

[0016] Figure 2, from the top left to the bottom right, contains the symbols "chi," "4," "naka," "ri," "∧," and "3." "Chi" and "ri" are symbols used when verbally describing the tone of the shamisen. "Chi" and "ri" indicate which string is plucked and how. "Naka" is a symbol indicating which finger to press the string with. "Naka" means pressing the string with the middle finger. "4" and "3" are symbols indicating the type of string to press with the finger and the kansho. The strings are called the first string, the second string, and the third string from the side closest to the performer. The kansho is the place on the shamisen neck where the string is pressed. If the Chinese numeral with "i" (ninben) is used, it means pressing the first string. If the Chinese numeral without "i" (ninben) is used, it means pressing the second string. If it is an Arabic numeral, it means pressing the third string. "∧" is a symbol indicating the playing technique. The "∧" signifies plucking the string. Note that "Chi", "Ri", and "Chu" are not essential information for identifying the scale, so they do not need to be recognized.

[0017] The musical scale of the koto is determined by the position of the pillars supporting the 13 strings. In koto music scores, the Chinese numerals from "ichi" to "ju" and the characters "do", "me", and "kin" are used to indicate the 13 strings. In koto music scores, the symbols "o" and "triangle" are used as rests, and katakana is used as performance symbols. For example, the performance symbol "o" means to press the string.

[0018] The task of recognizing musical scores for Japanese instruments faces the following challenges. The first challenge is that many existing scores for Japanese instruments are handwritten. The second challenge is that Japanese instrument scores contain a variety of symbols representing musical scales. The third challenge is that Japanese instrument scores depend on the personal writing style of the creator, and there is a large amount of variation in the writing style for each score. The fourth challenge is that there are few performers of Japanese instruments, and therefore there are few existing scores. These challenges make Japanese instrument scores more difficult to recognize than widely used musical scores such as staff notation.

[0019] The purpose of this embodiment is to recognize musical scores for Japanese musical instruments. To this end, this embodiment extracts a partial image representing one note from image data representing the musical score for a Japanese musical instrument, recognizes one or more symbols contained in the partial image, and identifies the musical scale based on the recognition results of the one or more symbols.

[0020] In one aspect, this embodiment recognizes the musical scores of Japanese musical instruments by recognizing each note included in the musical scores, thereby reducing the difficulty of recognition, and thus making it possible to recognize the musical scores of Japanese musical instruments. In another aspect, this embodiment can output electronic data that can play music based on the identified scale, thereby making it possible to play music written in musical scores based on the musical scores of Japanese musical instruments.

[0021] <Overall structure> The overall configuration of the musical score recognition system according to this embodiment will be described with reference to Fig. 3. Fig. 3 is a block diagram showing an example of the overall configuration of the musical score recognition system.

[0022] 3, the score recognition system 1000 includes a score recognition device 10. The score recognition device 10 is an example of an information processing device, such as a personal computer, workstation, or server, that recognizes Japanese musical instrument scores. The score recognition device 10 receives a score image as input and outputs the recognition result of the score image.

[0023] The musical score image is image data showing a Japanese musical instrument score. The musical score image may be image data obtained by capturing an image of a Japanese musical instrument score written on paper with an imaging device such as a camera or scanner. The musical score image may also be image data created electronically using software capable of creating Japanese musical instrument scores.

[0024] The recognition result of the musical score image may include electronic data in which the scales of multiple notes written in a Japanese musical instrument score are arranged in chronological order. The electronic data in which the scales are arranged in chronological order may be image data representing a musical score written in another notation method, such as staff notation. The recognition result of the musical score image may also include electronic data capable of playing a musical piece written in a Japanese musical instrument score. The electronic data capable of playing a musical piece may be audio data playable through a speaker or music data conforming to the MIDI (Musical Instruments Digital Interface) standard.

[0025] The musical score recognition device 10 may include a recognition model M. The recognition model M may be a machine learning model trained to recognize symbols included in musical score images. The recognition model M may be any model capable of image recognition, and may be a deep neural network based on deep learning. For example, the recognition model M may be configured with a convolutional neural network or a Vision Transformer.

[0026] The overall configuration of the score recognition system 1000 shown in FIG. 3 is an example, and various system configurations are possible depending on the application and purpose. For example, the score recognition system 1000 may include multiple score recognition devices 10. For example, the score recognition devices 10 may be implemented by multiple computers, or as a cloud computing service. For example, the score recognition system 1000 may be implemented by a standalone computer, or by a client-server system including terminal devices for inputting and outputting data. The classification of devices such as the score recognition device 10 shown in FIG. 3 is an example.

[0027] <Hardware configuration> The hardware configuration of the musical score recognition system 1000 will be described with reference to Fig. 4. The musical score recognition device 10 may be realized by, for example, a computer. Fig. 4 is a block diagram showing an example of a computer.

[0028] 4, the computer 500 includes a CPU (Central Processing Unit) 501, a ROM (Read Only Memory) 502, a RAM (Random Access Memory) 503, a HDD (Hard Disk Drive) 504, an input device 505, a display device 506, a communication I / F (Interface) 507, and an external I / F 508. The CPU 501, the ROM 502, and the RAM 503 form a so-called computer. The hardware components of the computer 500 are connected to each other via a bus line 509. The input device 505 and the display device 506 may be connected to the external I / F 508 for use.

[0029] The CPU 501 is a computing device that reads programs and data from a storage device such as the ROM 502 or the HDD 504 onto the RAM 503 and executes the processes, thereby realizing the overall control and functions of the computer 500. The computer 500 may have a GPU (Graphics Processing Unit) in addition to or instead of the CPU 501.

[0030] The ROM 502 is an example of a non-volatile semiconductor memory (storage device) that can retain programs and data even when the power is turned off. The ROM 502 functions as a main storage device that stores various programs, data, etc. required for the CPU 501 to execute various programs installed in the HDD 504. Specifically, the ROM 502 stores boot programs such as a Basic Input Output System (BIOS) and an Extensible Firmware Interface (EFI) that are executed when the computer 500 starts up, as well as data such as OS (Operating System) settings and network settings.

[0031] The RAM 503 is an example of a volatile semiconductor memory (storage device) in which programs and data are erased when the power is turned off. The RAM 503 is, for example, a dynamic random access memory (DRAM) or a static random access memory (SRAM). The RAM 503 provides a working area in which various programs installed in the HDD 504 are expanded when executed by the CPU 501.

[0032] The HDD 504 is an example of a non-volatile storage device that stores programs and data. The programs and data stored in the HDD 504 include an OS, which is basic software that controls the entire computer 500, and applications that provide various functions on the OS. Note that the computer 500 may use a storage device that uses flash memory as a storage medium (e.g., an SSD (Solid State Drive)) instead of the HDD 504.

[0033] The input device 505 includes a touch panel, operation keys and buttons, a keyboard and mouse, a microphone for inputting sound data such as voice, and the like, which are used by the user to input various signals.

[0034] The display device 506 is configured with a display such as a liquid crystal display or organic EL (Electro-Luminescence) display for displaying a screen, a speaker for outputting sound data such as voice, and the like.

[0035] The communication I / F 507 is an interface that connects to a communication network and enables the computer 500 to perform data communication.

[0036] The external I / F 508 is an interface with external devices, such as a drive device 510.

[0037] The drive device 510 is a device for loading a recording medium 511. The recording medium 511 here includes media that record information optically, electrically, or magnetically, such as a CD-ROM, a flexible disk, or a magneto-optical disk. The recording medium 511 may also include semiconductor memories that record information electrically, such as ROMs and flash memories. This allows the computer 500 to read from and / or write to the recording medium 511 via the external I / F 508.

[0038] The various programs to be installed in the HDD 504 are installed, for example, by setting the distributed recording medium 511 in a drive device 510 connected to the external I / F 508 and reading out the various programs recorded on the recording medium 511 by the drive device 510. Alternatively, the various programs to be installed in the HDD 504 may be installed by being downloaded via the communication I / F 507 from a network different from the communication network.

[0039] <Functional configuration> The functional configuration of the score recognition device 10 will be described with reference to Fig. 5. Fig. 5 is a block diagram showing an example of the functional configuration of the score recognition device.

[0040] 5, the musical score recognition device 10 includes a data storage unit 101, a model storage unit 102, a teacher data generation unit 110, a model learning unit 120, an image acquisition unit 130, an image extraction unit 140, an image recognition unit 150, a scale identification unit 160, and an output unit 170. The musical score recognition device 10 functions as the data storage unit 101, the model storage unit 102, the teacher data generation unit 110, the model learning unit 120, the image acquisition unit 130, the image extraction unit 140, the image recognition unit 150, the scale identification unit 160, and the output unit 170 by executing a pre-installed musical score recognition program.

[0041] For example, the data storage unit 101 and the model storage unit 102 are realized by the HDD 504 shown in Fig. 4. For example, the teacher data generation unit 110, the model learning unit 120, the image acquisition unit 130, the image extraction unit 140, the image recognition unit 150, the scale identification unit 160, and the output unit 170 are realized by the processing that the CPU 501 executes by a program loaded from the HDD 504 onto the RAM 503 shown in Fig. 4.

[0042] The data storage unit 101 stores training data. The training data is electronic data used to train the recognition model M. The training data may be stored in advance in the data storage unit 101. The data storage unit 101 may also store training data generated by the training data generation unit 110.

[0043] The training data includes images used to train the recognition model M (hereinafter referred to as "training images") and correct answer data. The training images include one or more symbols. The symbols included in the training images include symbols used in Japanese musical instrument scores, and may also include symbols not used in Japanese musical instrument scores. The correct answer data includes information indicating each of the one or more symbols included in the training image and information indicating the position of the symbol. The information indicating the position may be information indicating the range of the area in which the symbol is shown in the training image. The information indicating the range of the area may, for example, include the coordinates of the top left and bottom right of the area.

[0044] The training data may be generated based on an image of a musical score representing an existing Japanese musical instrument. Specifically, the training data may be generated by extracting images separated by a grid from the musical score image, and assigning to each training image correct answer data including information indicating the symbols contained in each training image and information indicating the positions of the symbols. The correct answer data may also be assigned manually by a user.

[0045] The data storage unit 101 may store symbol data. The symbol data is electronic data used to generate training data. The symbol data includes an image showing one symbol and information showing the symbol. The image included in the symbol data may include a handwritten symbol. The symbol shown in the image may be a symbol used in Japanese musical instrument scores, or may be a symbol not used in Japanese musical instrument scores.

[0046] The model storage unit 102 stores a recognition model M. The model storage unit 102 may store a trained recognition model M generated by the model training unit 120. The recognition model M may be trained using training data stored in the data storage unit 101. The recognition model M may be trained using training data generated by the training data generation unit 110.

[0047] The teacher data generation unit 110 generates teacher data. The teacher data generation unit 110 may generate the teacher data based on symbol data read from the data storage unit 101. The teacher data generation unit 110 may generate training images by randomly arranging one or more images included in the symbol data. The teacher data generation unit 110 may generate correct answer data to be assigned to the training images according to the arrangement of the images.

[0048] The teacher data generation unit 110 may generate new training images by randomly arranging grid lines on the training images. The teacher data generation unit 110 may randomly arrange grid lines on training images read out from the data storage unit 101. The teacher data generation unit 110 may randomly arrange grid lines on training images generated using symbol data. When generating new training images, the teacher data generation unit 110 may assign the correct answer data that was assigned to the original training images to the new training images.

[0049] The model learning unit 120 learns the recognition model M. The model learning unit 120 may learn the recognition model M based on training data. The training data may include at least one of training data read from the data storage unit 101 and training data generated by the training data generation unit 110.

[0050] The recognition model M may be a machine learning model that receives an image showing one or more symbols as input and outputs information indicating the symbols included in the image and information indicating the positions of each symbol. The information indicating the positions may be, for example, coordinates of a region of interest (ROI) or coordinates of a bounding box. The coordinates of the bounding box may include the coordinates of the upper left and lower right of the bounding box.

[0051] The recognition model M may be a machine learning model that receives an array of images showing one or more symbols as input and outputs an array of information showing the symbols included in the images and an array of information showing the positions of each symbol. In other words, the recognition model M may be a machine learning model that can recognize time-series data of images.

[0052] The recognition model M may be a machine learning model that receives an image and information indicating the structure of a symbol shown in the image as input, and recognizes the symbol shown in the image using the information indicating the structure of the symbol. The information indicating the structure of the symbol may be, for example, edge information indicating edges detected from the image. Any method may be used to detect edges, and as an example, a Sobel filter may be used for detection.

[0053] The model learning unit 120 may learn the recognition model M based on an error backpropagation method or the like. Specifically, the model learning unit 120 may input a training image included in the training data to the recognition model M during training and acquire the output of the recognition model M. The model learning unit 120 may calculate the error between the acquired output of the recognition model M and the correct data included in the training data, and update the parameters of the recognition model M based on the error. Any error measure used in the image recognition task may be used as the error. The model learning unit 120 may repeatedly update the parameters of the recognition model M until a predetermined convergence condition is satisfied. The predetermined convergence condition may be that the amount of parameter update is equal to or less than a threshold, or that the number of parameter updates is equal to or greater than a threshold.

[0054] The image acquisition unit 130 acquires a score image. The image acquisition unit 130 may accept a score image input by a user via the input device 505 of the score recognition device 10. The image acquisition unit 130 may acquire a score image from an imaging device such as a camera or a scanner. The image acquisition unit 130 may also accept a score image received from another information processing device.

[0055] The image extraction unit 140 extracts a partial image showing one note from the musical score image acquired by the image acquisition unit 130. The image extraction unit 140 may detect grid lines from the musical score image and extract the area surrounded by the detected grid lines as the partial image. The image extraction unit 140 may also extract the partial image by acquiring information indicating the area in which notes are written and dividing the area into a predetermined number of parts. The area in which notes are written and the number of grids included in the area may be input by the user. Hereinafter, the partial image extracted by the image extraction unit 140 will also be referred to as a "grid image."

[0056] The image extraction unit 140 may extract multiple grid images from one musical score image. When the image extraction unit 140 extracts multiple grid images, the image extraction unit 140 may generate an array of the grid images. In this case, the image extraction unit 140 may align the grid images according to a predetermined rule. The image extraction unit 140 may generate the array of grid images by aligning the grid images downward starting from the top right of the musical score image, and repeating this process in the leftward direction.

[0057] The image recognition unit 150 recognizes the grid image extracted by the image extraction unit 140. The image recognition unit 150 may recognize one or more symbols included in the grid image. The image recognition unit 150 may recognize the grid image based on a recognition model M read out from the model storage unit 102.

[0058] The image recognition unit 150 may input a grid image to the recognition model M, and acquire the symbols and position information output by the recognition model M. The image recognition unit 150 may input the arrangement of the grid image to the recognition model M, and acquire the arrangement of the symbols and the arrangement of the position information output by the recognition model M. The image recognition unit 150 may detect edges from the grid image, and input the edge information together with the grid image to the recognition model M, and acquire the symbols and position information output by the recognition model M.

[0059] The scale identification unit 160 identifies a scale corresponding to one note shown in the grid image based on the recognition result of the grid image recognized by the image recognition unit 150. The scale identification unit 160 may identify a scale corresponding to one note shown in the grid image based on one or more symbols recognized from the grid image.

[0060] The scale identification unit 160 may identify the scale according to a predetermined specific rule. The specific rule may be a rule that associates an arrangement of symbols with a scale. The arrangement of symbols included in the specific rule may be an arrangement in which one or more symbols representing one note are arranged according to a predetermined rule. The arrangement of symbols included in the specific rule may be an arrangement in which symbols are arranged based on the position at which the symbols are written in a grid. As an example, the arrangement of symbols included in the specific rule may be an arrangement in which one or more symbols written in a grid are arranged from the upper left to the lower right.

[0061] The scale identification unit 160 may have a plurality of specific rules. The plurality of specific rules may be created corresponding to the type of instrument, school, notation, composer, etc. The scale identification unit 160 may identify the scale according to a specific rule selected by the user. Information indicating the specific rule selected by the user may be input by the user together with the musical score image.

[0062] The scale identification unit 160 may align the symbols included in the recognition result based on the position information included in the recognition result of the grid image, and identify the scale based on the arrangement of the aligned symbols. The scale identification unit 160 may also compare the aligned arrangement of the symbols with the arrangement of symbols included in the identification rule, and identify the scale associated with the same arrangement of symbols.

[0063] The rules for identifying musical scales will be explained in detail using the grid image shown in Figure 2 as an example. The symbols "4," "∧," and "3" are placed in the center of the grid image. Because "chi," "ri," and "chu" are not essential information for identifying musical scales, we will explain them as not being recognized or discarding the recognition results. Because "4" appears first in the recognition results of the grid image, it identifies the note that will be produced when the fourth key point on the san no ito is pressed. Furthermore, because "∧" appears next in the recognition results of the grid image, it identifies the note that will be produced when the third key point on the san no ito is pressed. Furthermore, because "3" appears after "∧" in the recognition results of the grid image, it identifies the note that will be produced when the third key point on the san no ito is pressed.

[0064] The scale has a one-to-one correspondence with the note that is produced when a particular string is pressed and a particular key point is pressed, based on the open note. An open note is a note that is produced when the string is not pressed. The open note is identified by the key (key 92 in Figure 1) written on the Japanese musical instrument score. In the recognition result of the grid image shown in Figure 2, "4" appears first, so if the scale of the third string is D, the note D+4 is identified. In addition, in the recognition result of the grid image, "3" appears after "4", so the note D+3 is identified after the note D+4.

[0065] The output unit 170 outputs the recognition result of the musical score image. The output unit 170 may generate scale data indicating the arrangement of the identified scale for each of the multiple grid images and include it in the recognition result. The output unit 170 may convert the scale data into image data indicating a musical score written in another notation system, such as staff notation, and include it in the recognition result. The output unit 170 may convert the scale data into electronic data that can reproduce a musical piece and include it in the recognition result. The output unit 170 may convert the scale data into audio data that can be reproduced by a speaker. The output unit 170 may convert the scale data into music data that complies with the MIDI standard.

[0066] The output unit 170 may play the music described in the score image based on the recognition result of the score image. The output unit 170 may output the music from a speaker based on the audio data included in the recognition result. The output unit 170 may play the music data included in the recognition result using music editing software that complies with the MIDI standard.

[0067] <Processing Procedure> The score recognition method executed by the score recognition system 1000 will be described with reference to Figures 6 and 7. The score recognition method includes a learning process (see Figure 6) and a recognition process (see Figure 7).

[0068] <Learning process> 6 is a flowchart showing an example of the learning process, which is a process for learning the recognition model M.

[0069] In step S1, the teacher data generation unit 110 of the musical score recognition device 10 reads symbol data from the data storage unit 101. The teacher data generation unit 110 generates a training image by randomly arranging one or more images included in the symbol data. The teacher data generation unit 110 may also generate a new training image by randomly arranging grid lines in the generated training image.

[0070] The teacher data generation unit 110 may read out the teacher data from the data storage unit 101. The teacher data generation unit 110 may generate a new training image by randomly arranging grid lines in the training image included in the training data.

[0071] In step S2, the teacher data generation unit 110 of the score recognition device 10 assigns correct answer data to the training images generated in step S1. The teacher data generation unit 110 assigns correct answer data generated according to the image arrangement to training images generated using symbol data. The teacher data generation unit 110 assigns the correct answer data that was assigned to the original training image to training images generated by arranging grid lines. In this way, teacher data including the training images and correct answer data is generated. The teacher data generation unit 110 sends the generated teacher data to the model training unit 120.

[0072] In step S3, the model learning unit 120 of the score recognition device 10 receives the teacher data from the teacher data generation unit 110. The model learning unit 120 reads the teacher data from the data storage unit 101. The model learning unit 120 integrates the teacher data received from the teacher data generation unit 110 and the teacher data read from the data storage unit 101. The model learning unit 120 learns the recognition model M based on the integrated teacher data.

[0073] In step S4, the model learning unit 120 of the score recognition device 10 stores the learned recognition model M in the model storage unit 102. If the learned recognition model M is already stored in the model storage unit 102, the model learning unit 120 may update the recognition model M stored in the model storage unit 102 with the recognition model M learned in step S3.

[0074] <Recognition processing> 7 is a flowchart showing an example of the recognition process, which is a process for recognizing a Japanese musical instrument score based on a trained recognition model M.

[0075] In step S11, a user of the score recognition system 1000 inputs a score image to the score recognition device 10. The image acquisition unit 130 of the score recognition device 10 acquires the score image input by the user. The image acquisition unit 130 sends the score image to the image extraction unit 140.

[0076] In step S12, the image extraction unit 140 of the score recognition device 10 receives the score image from the image acquisition unit 130. The image extraction unit 140 extracts a grid image from the score image. The image extraction unit 140 sends the grid image to the image recognition unit 150.

[0077] In step S13, the image recognition unit 150 of the score recognition device 10 receives the grid image from the image extraction unit 140. The image recognition unit 150 reads out the recognition model M from the model storage unit 102. The image recognition unit 150 inputs the grid image to the recognition model M. The recognition model M recognizes one or more symbols from the input grid image and outputs the recognized one or more symbols and position information indicating the position at which each symbol was recognized. The image recognition unit 150 acquires the symbols and position information output by the recognition model M.

[0078] The image recognition unit 150 sends the recognition result of the lattice image to the scale identification unit 160. The recognition result of the lattice image includes the symbols and position information output by the recognition model M. In other words, the recognition result of the lattice image includes one or more symbols included in the lattice image and position information indicating the position where each symbol is recognized.

[0079] In step S14, the scale identification unit 160 of the musical score recognition device 10 receives the recognition result of the grid image from the image recognition unit 150. The scale identification unit 160 aligns the symbols included in the recognition result based on the position information included in the recognition result. The scale identification unit 160 compares the aligned symbol arrangement with the symbol arrangement included in the specific rule, and identifies the scale associated with the same symbol arrangement. The scale identification unit 160 sends information indicating the identified scale to the output unit 170.

[0080] The score recognition device 10 repeatedly executes steps S13 and S14 for each grid image extracted in step S12. In other words, the score recognition device 10 repeatedly recognizes one or more symbols and specifies a musical scale based on the recognition results for the multiple grid images extracted from the score image, the number of times corresponding to the number of extracted grid images. As a result, the output unit 170 receives a musical scale arrangement corresponding to each grid image extracted from the score image.

[0081] In step S15, the output unit 170 of the musical score recognition device 10 receives information indicating the scale from the scale identification unit 160. The output unit 170 receives information indicating the scale identified for each of the multiple grid images and generates scale data indicating the arrangement of the scale. The output unit 170 may convert the scale data into electronic data that can play the music. The output unit 170 generates a recognition result that includes at least one of the scale data and the electronic data that can play the music. The output unit 170 outputs the recognition result of the musical score image.

[0082] <Effects of the embodiment> The musical score recognition device 10 according to this embodiment extracts a partial image representing one note from image data representing a musical score for a Japanese musical instrument, recognizes one or more symbols contained in the partial image, and identifies a scale based on the recognition results of the one or more symbols. In one aspect, this embodiment enables recognition of musical scores for Japanese musical instruments.

[0083] The musical score may be a vertical staff in which one or more symbols corresponding to one note are written within one grid. The musical score recognition device 10 may extract an image within a grid included in the image data as a partial image. In one aspect, this embodiment can recognize a vertical staff in which one note is written within one grid.

[0084] The score recognition device 10 may output one or more symbols included in the partial image and their respective positional information. According to one aspect, this embodiment allows for accurate identification of a musical scale based on the positional relationships of the symbols in the partial image.

[0085] The score recognition device 10 may output an arrangement of symbols and an arrangement of position information based on the arrangement of the partial images. The score recognition device 10 may also recognize each symbol based on edge information detected from the partial images. In one aspect, according to this embodiment, symbols included in the partial images can be recognized with high accuracy.

[0086] The score recognition device 10 may identify a scale based on a symbol arrangement in which one or more symbols are aligned based on position information. In one aspect, this embodiment allows for robust identification of a scale against variations in the positions of symbols within a grid.

[0087] The score recognition device 10 may recognize one or more symbols by inputting a partial image into a trained recognition model. The score recognition device 10 may train the recognition model based on training data including the partial image and correct answer data for the symbols. In one aspect, according to this embodiment, symbols included in the partial image can be recognized with high accuracy.

[0088] The training data may include a partial image in which one or more images showing one symbol are randomly arranged. The training data may include a partial image in which grid lines are randomly arranged. In one aspect, according to this embodiment, the symbol included in the partial image can be recognized with high accuracy.

[0089] The musical score recognition device 10 may output audio data or music data based on the musical scale. According to one aspect, the present embodiment can reproduce music written in a musical score based on a musical score for a Japanese musical instrument.

[0090] The Japanese musical instrument may include a shamisen. According to one aspect of the present embodiment, shamisen music notation can be recognized.

[0091] [supplement] Each function of the above-described embodiments can be realized by one or more processing circuits. Here, the term "processing circuit" in this specification includes a processor programmed to perform each function by software, such as a processor implemented by an electronic circuit, as well as devices such as an ASIC (Application Specific Integrated Circuit), a DSP (Digital Signal Processor), an FPGA (Field Programmable Gate Array), and conventional circuit modules designed to perform each of the above-described functions.

[0092] Although the embodiments of the present invention have been described in detail above, the present invention is not limited to these embodiments, and various modifications and changes are possible within the scope of the gist of the present invention described in the claims.

[0093] The disclosed technology may take the following forms as described below.

[0094] (Appendix 1) an image acquisition unit configured to acquire image data representing a musical score of a Japanese musical instrument; an image extraction unit configured to extract a partial image representing one sound from the image data; an image recognition unit configured to recognize one or more symbols included in the partial image; a scale identification unit configured to identify a scale based on the recognition result of the one or more symbols; A musical score recognition device comprising: (Appendix 2) the musical score is a vertical staff in which the one or more symbols corresponding to the one note are written in one grid; the image extraction unit is configured to extract an image within the grid included in the image data as the partial image. 2. The musical score recognition device according to claim 1. (Appendix 3) the image recognition unit is configured to output each of the one or more symbols included in the partial image and position information of each of the symbols. 3. The musical score recognition device according to claim 2. (Appendix 4) the image recognition unit is configured to output the arrangement of the symbols and the arrangement of the position information based on the arrangement of the partial images. 4. The musical score recognition device according to claim 3. (Appendix 5) the image recognition unit is configured to recognize each of the symbols based on edge information detected from the partial images. 5. The musical score recognition device according to claim 3 or 4. (Appendix 6) the scale identification unit is configured to identify the scale based on an arrangement of the one or more symbols arranged based on the position information. 6. A musical score recognition device according to any one of appendices 3 to 5. (Appendix 7) The image recognition unit is configured to recognize the one or more symbols by inputting the partial image into a trained recognition model. 7. A musical score recognition device according to any one of appendices 1 to 6. (Appendix 8) further comprising a model learning unit configured to learn the recognition model based on training data including the partial image and correct answer data of the symbol; 8. The musical score recognition device according to claim 7. (Appendix 9) The training data includes the partial image in which one or more images showing one symbol are randomly arranged. 9. The musical score recognition device according to claim 8. (Appendix 10) The training data includes the partial image in which grid lines are randomly arranged. 10. The musical score recognition device according to claim 8 or 9. (Appendix 11) an output unit configured to output sound data or music data based on the musical scale; 11. A musical score recognition device according to any one of appendices 1 to 10. (Appendix 12) The Japanese musical instrument includes a shamisen. 12. A musical score recognition device according to any one of appendices 1 to 11. (Appendix 13) The computer A step of acquiring image data representing musical scores for Japanese musical instruments; extracting a partial image representing one sound from the image data; Recognizing one or more symbols contained in the partial image; identifying a scale based on the recognition result of said one or more symbols; A music score recognition method that performs the following. (Appendix 14) On the computer, A step of acquiring image data representing musical scores for Japanese musical instruments; extracting a partial image representing one sound from the image data; Recognizing one or more symbols contained in the partial image; identifying a scale based on the recognition result of said one or more symbols; A program to execute. [Explanation of symbols]

[0095] 10: Musical score recognition device 101: Data storage unit 102: Model memory unit 110: Teacher data generation unit 120: Model learning section 130: Image acquisition unit 140: Image extraction unit 150: Image recognition unit 160: Scale specific part 170: Output section 1000: Musical Score Recognition System

Claims

1. an image acquisition unit configured to acquire image data representing a musical score of a Japanese musical instrument; an image extraction unit configured to extract a partial image representing one sound from the image data; an image recognition unit configured to recognize one or more symbols included in the partial image; a scale identification unit configured to identify a scale based on the recognition result of the one or more symbols; A musical score recognition device comprising:

2. the musical score is a vertical staff in which the one or more symbols corresponding to the one note are written in one grid, the image extraction unit is configured to extract an image within the grid included in the image data as the partial image.

2. The musical score recognition device according to claim 1.

3. the image recognition unit is configured to output each of the one or more symbols included in the partial image and position information of each of the symbols.

3. The musical score recognition device according to claim 2.

4. the image recognition unit is configured to output the arrangement of the symbols and the arrangement of the position information based on the arrangement of the partial images.

4. The musical score recognition device according to claim 3.

5. the image recognition unit is configured to recognize each of the symbols based on edge information detected from the partial images.

4. The musical score recognition device according to claim 3.

6. the scale identification unit is configured to identify the scale based on an arrangement of the one or more symbols arranged based on the position information.

4. The musical score recognition device according to claim 3.

7. The image recognition unit is configured to recognize the one or more symbols by inputting the partial image into a trained recognition model.

7. A musical score recognition device according to claim 1.

8. further comprising a model learning unit configured to learn the recognition model based on training data including the partial image and correct answer data of the symbol; 8. The musical score recognition device according to claim 7.

9. The training data includes the partial image in which one or more images representing one symbol are randomly arranged.

9. The musical score recognition device according to claim 8.

10. The training data includes the partial image in which grid lines are randomly arranged.

9. The musical score recognition device according to claim 8.

11. an output unit configured to output sound data or music data based on the musical scale; 7. A musical score recognition device according to claim 1.

12. The Japanese musical instrument includes a shamisen.

7. A musical score recognition device according to claim 1.

13. The computer A step of acquiring image data representing musical scores for Japanese musical instruments; extracting a partial image representing one sound from the image data; Recognizing one or more symbols contained in the partial image; identifying a scale based on the recognition result of the one or more symbols; A music score recognition method that performs the following.

14. On the computer, A step of acquiring image data representing musical scores for Japanese musical instruments; extracting a partial image representing one sound from the image data; Recognizing one or more symbols contained in the partial image; identifying a scale based on the recognition result of the one or more symbols; A program to execute.

Images