Music score image to braille translation method based on multi-modal large model and location perception

Abstract

A method for translating sheet music images into Braille based on a multimodal large model and position awareness includes: collecting the MusicXML dataset, converting MusicXML into Braille using MuseScore software to obtain a Braille sheet music dataset; using the BPE algorithm to gradually expand the initial Braille vocabulary to the target size by statistically analyzing the frequencies of adjacent Braille characters and iteratively merging high-frequency pairs, thus expanding the Braille vocabulary of the large model; binarizing the staff image, calculating pixel summation through vertical projection, detecting the position of the staff lines, calculating the average spacing, and dividing the sheet music system according to the spacing threshold to determine the upper and lower boundary coordinates of each sheet music system; extracting features from the preprocessed sheet music image using a visual encoder and fusing the position codes of the detected staff music system boundaries to generate fused embedded features; and achieving automatic translation of sheet music images into Braille by constructing multimodal training samples and fine-tuning the pre-trained model. This invention improves translation quality by integrating the coordinate information of the musical score system in the musical score image structure, relying on the graphic and textual understanding capabilities of a multimodal large model, and mapping the staff image into a semantic expression that conforms to Braille standards, thereby achieving automatic completion from the original image input to the final Braille output.

Description

Technical Field

[0001] This invention relates to the fields of image processing, music information processing, and accessibility technology. It is based on a multimodal large model and position-aware automatic conversion method from musical score images to Braille, aiming to solve the complexity and challenges in the process of translating musical scores into Braille. Background Technology

[0002] In the field of optical music score recognition, traditional recognition processes typically rely on manually designed feature extraction methods and multi-stage processing modules, such as image preprocessing, note detection, symbol classification, and structural analysis. These methods often exhibit problems such as low recognition accuracy and poor generalization ability when faced with complex or low-quality music score images.

[0003] The creation of both standard musical notation and Braille sheet music typically requires specialized musical knowledge and a deep understanding of Braille rules. This results in a very limited number of standard musical notation-Braille pairings that accurately represent the same musical information. Furthermore, currently, there are no mature automatic translation tools capable of converting standard musical notation images into high-quality Braille output.

[0004] The above-mentioned problems urgently need to be solved. Summary of the Invention

[0005] This invention aims to overcome the aforementioned shortcomings of existing technologies and, based on the advantages of multimodal large model image and text understanding, provides a method for translating musical scores into Braille based on multimodal large model and position awareness.

[0006] To address the aforementioned technical challenges, a method for translating musical scores from images to Braille based on a multimodal large model and position awareness is proposed, comprising the following steps:

[0007] S110: Collect MusicXML dataset, use MuseScore software to convert MusicXML to Braille, and obtain Braille music score dataset.

[0008] S120 utilizes the BPE algorithm to gradually expand the initial Braille vocabulary to the target size (e.g., 1024 words) by statistically analyzing the frequencies of adjacent Braille characters and iteratively merging high-frequency pairs, thereby expanding the Braille vocabulary of the large model.

[0009] S130 uses a binarized five-line staff image, vertical projection to calculate pixel sum, detect spectral line positions, calculate average spacing, and divides the musical score system according to the spacing threshold to determine the upper and lower boundary coordinates of each musical score system.

[0010] S140: The preprocessed musical score image is used to extract features through a visual encoder, and the positional codes of the detected staff system boundary are fused to generate fused embedded features.

[0011] S150 enables automatic translation of sheet music images into Braille by constructing multimodal training samples and fine-tuning the pre-trained model.

[0012] Furthermore, step S110 involves collecting the MusicXML dataset and using MuseScore software to convert the MusicXML into Braille, thereby obtaining a Braille music score dataset. Specifically, this includes:

[0013] S1101, collect sheet music images and MusicXML dataset;

[0014] S1102, use MuseScore software to convert MusicXML into Braille.

[0015] Furthermore, step S120, which utilizes the BPE algorithm to statistically analyze the frequencies of adjacent Braille characters and iteratively merge high-frequency pairs, gradually expands the initial Braille vocabulary to the target size (e.g., 1024 words), thereby expanding the Braille lexicon of the large model. Specifically, this includes:

[0016] S1201, Construct an initial vocabulary starting from individual Braille characters, with each Braille character serving as a separate marker.

[0017] S1202, Traverse the Braille dataset and count the frequency of each pair of adjacent Braille characters.

[0018] S1203: Find the pair of characters that appear most frequently and merge them into a new subword and add it to the vocabulary.

[0019] S1204, continuously repeats the process of merging the most frequently occurring character / subword pairs until the new dictionary size reaches 1024.

[0020] Furthermore, step S130 involves binarizing the five-line staff image, calculating pixel summation through vertical projection, detecting spectral line positions, calculating the average spacing, and dividing the musical score system according to the spacing threshold to determine the upper and lower boundary coordinates of each musical score system. Specifically, this includes:

[0021] S1301, binarize the musical staff image and project it vertically to calculate the total pixel sum of each line, P(y) = I(x,y). Here, I(x,y) is the pixel value at coordinates (x,y), and W is the image width.

[0022] S1302, based on the maximum value of the pixel array P(y), set an appropriate threshold T1, and filter out extreme points below max(P(y))*T1. The remaining points are used as the spectral coordinate positions of the five-line staff, and the row coordinate array A={y1,y2,…,yn} represents the remaining row coordinate positions.

[0023] S1303, perform a first-order difference operation on the detected spectral line position sequence A = {y1, y2, ..., yn}, and take the arithmetic mean of the difference results to obtain the average spacing D between spectral lines.

[0024]

[0025] S1304, further, for the spectral line distance D, a threshold T is set. d =1.5*D. The spacing between the staff lines in A is less than T. d Divide the spectral lines into groups and simultaneously determine the upper and lower boundaries of each group. Let y be the smallest row number of a certain group of spectral lines. min The maximum row number is y max The upper and lower boundaries of the musical notation system are Top. b =y min -T d Bottom b =y max +T d .

[0026] Step S140, which involves extracting features from the preprocessed musical score image using a visual encoder and fusing the detected positional codes of the staff system boundaries to generate fused embedded features, specifically includes:

[0027] S1401, Input the preprocessed music score image into the visual encoder ViT to extract image representation;

[0028] S1402, Top the music score system boundary information detected in the previous step b and Bottom b This is converted into a spatial location code and fused with image features to obtain the embedding E = MLP(V+P).

[0029] S1403 uses the embedded feature E as the embedding layer of the multimodal model.

[0030] Step S150, which describes the automatic translation of sheet music images into Braille by constructing multimodal training samples and fine-tuning the pre-trained model, specifically includes:

[0031] S1501, construct training sample pairs, with the input being a preprocessed musical stave image and position coordinates, and the output being Braille.

[0032] S1502 employs a pre-trained model that supports image and text comprehension (such as Qwen2.5-VL, InternVL2), whose encoder receives images and location embeddings, and whose decoder is responsible for generating Braille sequences.

[0033] S1503, parameter initialization based on extended vocabulary: the Braille vocabulary constructed by the BPE algorithm is loaded into the model decoder, and the word embedding matrix is ​​adjusted to adapt to the new vocabulary.

[0034] S1504 uses LoRA technology to freeze parameters and fine-tune large models.

[0035] S1505 uses a finely tuned large model to translate sheet music images and output Braille.

[0036] A second aspect of the present invention relates to a music score image to Braille translation system based on a multimodal large model and position awareness, comprising a memory and one or more processors, wherein the memory stores executable code, and the one or more processors execute the executable code to implement the music score image to Braille translation method of the present invention based on a multimodal large model and position awareness.

[0037] The beneficial effects of this invention are that by integrating the coordinate information of the musical score system in the musical score image structure, relying on the graphic and textual understanding capabilities of the multimodal large model, and mapping the staff image into a semantic expression that conforms to Braille standards, the translation quality is effectively improved, and the process from original image input to final Braille output is completed automatically. Attached Figure Description

[0038] Figure 1 This is a flowchart of the method of the present invention. Detailed Implementation

[0039] The present invention will now be described in further detail with reference to the accompanying drawings. These drawings are simplified schematic diagrams, illustrating only the basic structure of the invention, and therefore only show the components relevant to the invention.

[0040] Example 1

[0041] like Figure 1 As shown, this embodiment 1 provides a method for translating musical scores into Braille based on a multimodal large model and position awareness. It uses the coordinate position information of the calculated musical score system and the embedded information generated from the musical score image to fine-tune the multimodal large model. Utilizing the graphic understanding capabilities of the multimodal large model, it understands the logical relationship between the visual layout of the musical staff and the musical symbols, accurately generating musical Braille.

[0042] Specifically, the method includes:

[0043] S110, collects and converts Braille music score datasets;

[0044] Specifically, collect sheet music images and MusicXML datasets. Install the MuseScore software. Use MuseScore to convert MusicXML into Braille, obtaining a paired dataset of images and Braille.

[0045] S120 utilizes the BPE algorithm to expand the Braille lexicon of large models;

[0046] Specifically, an initial vocabulary is built starting with individual Braille characters, each character serving as a separate marker. The Braille dataset is traversed, and the frequency of each pair of adjacent Braille characters is counted. The pair of characters with the highest frequency is found and merged into a new subword, which is then added to the vocabulary. This process of merging the most frequent character / subword pairs is repeated until the new vocabulary reaches a size of 1024.

[0047] S130, the coordinates of each musical score system in the musical score image are calculated;

[0048] Specifically, the musical staff image is binarized and projected vertically to calculate the total pixel sum P(y) = I(x,y) for each row. Here, I(x,y) is the pixel value at coordinates (x,y), and W is the image width. Based on the maximum value of the P(y) pixel array, a suitable threshold T1 is set, and extreme points below max(P(y))*T1 are filtered out. The row coordinates of the remaining points are used as the spectral line coordinates, resulting in a row coordinate array A = {y1,y2,…,yn}. The detected spectral line row coordinates A = {y1,y2,…,yn} are subjected to a first-order difference operation, and the arithmetic mean of the difference results is taken to obtain the average spacing D between the spectral lines. Further, a threshold T is set for the spectral line distance D. d =1.5*D. The spacing between the staff lines in A is less than T. d Divide the spectral lines into groups and simultaneously determine the upper and lower boundaries of each group. Let y be the smallest row number of a certain group of spectral lines. min The maximum row number is y max The upper and lower boundaries of the musical notation system are Top. b =y min -T d Bottom b =y max +T d This allows us to obtain the coordinate positions of each musical notation system within the musical score image.

[0049] S140 integrates musical stave images and coordinate information as model embeddings.

[0050] Specifically, the preprocessed music score image is input into the visual encoder ViT to extract image representations; the music score system boundary information Top detected by S130 is then processed. b and Bottom b The feature is converted into a spatial location code and fused with image features to obtain the embedding E = MLP(V+P). The embedding feature E is then used as the embedding layer of the multimodal model.

[0051] S150, fine-tuning the large model to generate Braille text.

[0052] Specifically, training sample pairs are constructed, with preprocessed musical notation images and position coordinates as input, and Braille as output. A pre-trained model supporting image-text understanding (such as Qwen2.5-VL, InternVL2) is employed, whose encoder receives the image and position embeddings, and whose decoder generates the Braille sequence. Parameter initialization is performed based on an expanded vocabulary: the Braille vocabulary constructed using the BPE algorithm is loaded into the model decoder, and the word embedding matrix is ​​adjusted to fit the new vocabulary. The LoRA technique is used to freeze parameters and fine-tune the large model. The fine-tuned large model is then used to transcribe the musical notation image and output Braille.

[0053] In summary, this embodiment provides a method for translating musical scores from images to Braille based on a multimodal large model and position-aware technology. By fusing musical score images and coordinate information to fine-tune the multimodal large model, the complexity of staff notation recognition is effectively reduced, and recognition accuracy and efficiency are improved. This method integrates the advantages of object detection and a multimodal large model, realizing an end-to-end automated processing flow from image input to musical Braille output.

[0054] Example 2

[0055] This embodiment provides a music score image to Braille translation system based on a multimodal large model and position awareness, including a memory and one or more processors. The memory stores executable code, and when the one or more processors execute the executable code, they are used to implement the music score image to Braille translation method based on a multimodal large model and position awareness of Embodiment 1.

[0056] Based on the above-described preferred embodiments of the present invention, and through the foregoing description, those skilled in the art can make various changes and modifications without departing from the inventive concept. The technical scope of this invention is not limited to the contents of the specification, but must be determined according to the scope of the claims.

Claims

1. A method for translating musical scores from images to Braille based on a multimodal large model and position awareness, comprising the following steps: S110: Collect MusicXML dataset, use MuseScore software to convert MusicXML into Braille, and obtain Braille music score dataset; S120 utilizes the BPE algorithm to gradually expand the initial Braille vocabulary to the target size by statistically analyzing the frequencies of adjacent Braille characters and iteratively merging high-frequency pairs, thereby expanding the Braille lexicon of the large model. S130, binarized five-line staff image, vertical projection to calculate pixel sum, detect spectral line positions, calculate average spacing, and divide the musical score system according to the spacing threshold to determine the upper and lower boundary coordinates of each musical score system; S140, the preprocessed musical score image is used to extract features through a visual encoder, and the positional codes of the detected staff system boundary are fused to generate fused embedded features; S150, by constructing multimodal training samples and fine-tuning the pre-trained model, achieves automatic translation of musical scores from images to Braille; Specifically, it includes: S1501, construct training sample pairs, with the input being a preprocessed musical stave image and position coordinates, and the output being Braille; S1502 employs a pre-trained model that supports image and text comprehension. Its encoder receives images and location embeddings, while the decoder is responsible for generating Braille sequences. S1503, parameter initialization based on extended vocabulary: the Braille vocabulary constructed by the BPE algorithm is loaded into the model decoder, and the word embedding matrix is ​​adjusted to adapt to the new vocabulary; S1504, using LoRA technology to freeze parameters and fine-tune a large model; S1505 uses a finely tuned large model to translate sheet music images and output Braille.

2. The method for translating musical scores into Braille based on a multimodal large model and position awareness as described in claim 1, characterized in that, Step S110, which involves collecting the MusicXML dataset and using MuseScore software to convert the MusicXML into Braille to obtain a Braille music score dataset, specifically includes: S1101, collect sheet music images and MusicXML dataset; S1102, use MuseScore software to convert MusicXML into Braille.

3. The method for translating musical scores into Braille based on a multimodal large model and position awareness as described in claim 1, characterized in that, Step S120, which utilizes the BPE algorithm to statistically expand the initial Braille vocabulary to the target size by counting the frequencies of adjacent Braille characters and iteratively merging high-frequency pairs, thereby expanding the Braille lexicon of the large model, specifically includes: S1201, Construct an initial vocabulary starting from individual Braille characters, with each Braille character serving as a separate marker; S1202, Traverse the Braille dataset and count the frequency of each pair of adjacent Braille characters; S1203: Find the pair of characters that appear most frequently and merge them into a new subword and add it to the vocabulary; S1204, continuously repeats the process of merging the most frequently occurring character and subword pairs until the new dictionary size reaches 1024.

4. The method for translating musical scores into Braille based on a multimodal large model and position awareness as described in claim 1, characterized in that, Step S130 specifically includes: S1301: Binarize the musical staff image and project it vertically to calculate the total number of pixels in each line. ;in, , is the pixel value of the image at coordinates (x, y), and W is the width of the image; S1302, according to Find the maximum value of the pixel array, set an appropriate threshold T1, and filter out values ​​below (max). The extreme points of T1 are used as the remaining spectral line coordinates, resulting in a row coordinate array A = {y1,y2,…,yi,…,yn}, where yi represents the remaining row coordinate positions. S1303, Perform a first-order difference operation on the detected spectral line position sequence A = {y1,y2,…,yi,…,yn}, and take the arithmetic mean of the difference results to obtain the average spacing D between spectral lines; that is... S1304, further, sets a threshold for the spectral line distance D. ; The spacing between the staffs in A is less than Divide the spectral lines into groups and simultaneously determine the upper and lower boundaries of each group; let the minimum row number of a certain group be . The maximum line number is The upper and lower boundaries of the musical notation system are .

5. The method for translating musical scores into Braille based on a multimodal large model and position awareness as described in claim 1, characterized in that, Step S140, which involves extracting features from the preprocessed musical score image using a visual encoder and fusing the detected positional codes of the staff system boundaries to generate fused embedded features, specifically includes: S1401, Input the preprocessed music score image into the visual encoder ViT to extract image representation; S1402, the music score system boundary information detected in the previous step This is converted into a spatial location code and fused with image features to obtain embedded feature E; S1403 uses the embedded feature E as the embedding layer of the multimodal model.

6. A music score image-to-Braille translation system based on a multimodal large model and position awareness, characterized in that: The method includes a memory and one or more processors, wherein the memory stores executable code, and the one or more processors execute the executable code to implement the method for translating musical scores into Braille based on a multimodal large model and position awareness, as described in any one of claims 1-5.

Images