An embedded music / Braille text conversion system capable of synchronizing visual, auditory, and tactile presentations

Abstract

The present invention discloses an embedded music / Braille text conversion system capable of visually, audibly and tactilely synchronously presenting, which includes an acquisition module for acquiring musical scores in MusicXML format, a conversion module for converting the musical scores in MusicXML format into Braille musical scores consisting of Braille dots, and an output module for simultaneously outputting the Braille musical scores in three ways: visually, audibly and tactilely. The conversion system has a fast calculation speed and high accuracy, smoothly provides a new feasible means for Braille musical score information, and can realize the embedded function of visually, audibly and tactilely synchronously presenting Braille musical scores, with clear and accurate visual display, complete auditory reproduction, clear tactile sensation, and better presentation effect, which can further improve the efficiency of visually impaired people learning musical scores, and provide convenience for visually impaired people to acquire and touch-read a large amount of music materials.

Description

[Technical Field]

[0001] The present invention belongs to the technical field of computer science Braille music score processing and analysis, and in particular to an embedded music / Braille text conversion system that can be presented visually, audibly, and tactilely in sync. [Background technology]

[0002] Braille, also known as raised braille or raised characters, is a tactile system specially designed for the visually impaired. It is made by applying different combinations of raised dots to paper using a braille board, braille machine, or braille printer. Generally, each block of braille consists of six dots, with 123 dots from top to bottom on the left and 456 dots on the right, called a square.

[0003] Braille is at the core of the linguistic and written rights of the visually impaired and a key factor in improving their cultural and educational level. It is also an important means of ensuring that the visually impaired can share their cultural and educational rights. While research into Chinese-to-Braille conversion systems has achieved some success in China, Braille music scores and related materials primarily rely on human translation, failing to meet the vast demand for Braille music scores among the visually impaired. At the same time, most research results only allow electronic Braille sequences to be obtained through conversion systems, requiring the use of a Braille display or other device to touch-read Braille music scores, which does not achieve the synchronized presentation of visual, auditory, and tactile sensations. This means that when visually impaired people learn Braille music scores, they can only perceive the notes and rhythm by touching a Braille display, but cannot simultaneously hear the music or feel the rhythm, limiting their learning effectiveness and experience.

[0004] Patent document number CN105931621A discloses a method and system for translating MIDI into Braille music notation. This technical solution can only translate from MIDI to Braille music notation. However, it is difficult to clearly and accurately record music notation information in MIDI that can be read and understood by humans, and the reproducibility of music notation is low, and notes are easily lost, making it difficult to meet the needs of music notation.

[0005] Patent document number CN110390859A discloses a method and system for converting MusicXML sheet music into Braille, but this technical solution only reads and analyzes MusicXML data, and does not convert it into staff notation or reproduce the sounds in the sheet music. It is therefore unable to display a comparison between staff notation and Braille sheet music, facilitating the publication and use of Braille sheet music.

[0006] Converting sheet music into Braille is an important step in digitizing Braille. All of the above methods are implemented using software and cloud computing, and are not integrated into embedded environments, making them unable to be applied conveniently and efficiently in public spaces. Summary of the Invention [Problem to be solved by the invention]

[0007] In view of the above, the object of the present invention is to provide an embedded music / Braille text conversion system that can present music in a visually, auditory, and tactilely synchronized manner, which can automatically convert musical notation into Chinese Braille music in an embedded environment and has the function of presenting music in a visually, auditory, and tactilely synchronized manner, thereby meeting the needs of visually impaired people for learning music and realizing a multimodal digital presentation of Braille music. [Means for solving the problem]

[0008] In order to achieve the above object of the invention, the embedded music / Braille text conversion system according to the present invention, which can be presented visually, aurally, and tactilely in sync, comprises: an acquisition module for acquiring scores in MusicXML format; a conversion module for converting a musical score in MusicXML format into a Braille musical score consisting of a sequence of Braille dots; It also has an output module for simultaneously outputting Braille music scores in three ways: visually, aurally, and tactilely.

[0009] Preferably, the step of converting the MusicXML formatted musical score into a Braille musical score consisting of a sequence of Braille dots comprises: Step 1: reading and parsing a MusicXML file representing a musical score to obtain a tree model corresponding to the MusicXML file content; Step 2: obtaining a root node of the MusicXML file structure from the tree model, and using the root node as a node in the initial traversal tree model, searching for nodes to be converted to Braille and obtaining node label contents; and step 3, converting the node label content into a braille dot sequence based on a Chinese braille music symbol correspondence library to obtain the braille music score.

[0010] Preferably, the step of searching for a node to be converted into Braille and obtaining node label contents using a root node as a node of the initial traverse tree model includes: The method includes determining whether the current node is a movement node, and if the condition is met, further determining whether the child node of the movement is empty, and if so, traversing the next sibling node of the movement; if the child node of the movement is not empty and is a measure node, sequentially analyzing the attribute node and dynamic mark node of the measure node to obtain the attribute node label content and the dynamic mark node label content; if the child node of the measure node is a note node, analyzing the descendant node of the note node to obtain the descendant node label content; and traversing the next sibling node of the measure node after completing the traversal of the descendant node of the measure node.

[0011] Preferably, when converting the metronome information in the dynamic mark node label content into a Braille dot sequence, the method includes obtaining the metronome value of the metronome node, converting the metronome value into an integer number, converting the ones digit, tens digit, and hundreds digit of the integer number into individual numbers, and converting each individual number into a corresponding Braille dot sequence.

[0012] Preferably, the process of converting musical notes into a sequence of Braille dots comprises: For the input note node, determine whether there is a changed note node in the child node of the note node, and if there is a changed note, record the related information of the changed note; if there is no changed note, continue to determine whether there is a pitch node; If there is a pitch node, obtain and temporarily store the attribute values ​​of the scale node and octave node of the note, then obtain and temporarily store the attribute values ​​of the note value node, perform tone group symbol processing, then determine whether to add a changed note type or not to add a braille dot sequence corresponding to the changed note according to the recording status of the changed note, then determine a braille dot sequence corresponding to the note based on the scale and note value temporarily stored, further determine whether to add a braille dot sequence corresponding to the augmentation dot, and finally return the braille dot sequence result; If there is no pitch node, it is determined whether there is a rest node, and if there is a rest node, jump to a step of recording the attribute value of the note value node if there is a pitch node, and continue to execute the subsequent steps, and if there is no rest, perform an error reporting process.

[0013] Preferably, said step of performing tone group symbol processing comprises: Obtaining the scale and octave of the previous note and the scale and octave of the current note, and then determining whether the two adjacent notes belong to the same tone group based on whether the octaves of the two adjacent notes are equal; If two adjacent notes belong to the same key group, determine whether the absolute value of the interval between the two notes is greater than 5, and if it is greater than 5, add a key group symbol before the current note; otherwise, do not add a key group symbol; When two adjacent notes belong to different pitch groups, if the difference between the two notes is two pitch groups or more, add a pitch group symbol in front of the current note; if it is determined that the two notes belong to two adjacent pitch groups and the note interval is greater than 3, add a pitch group symbol in front of the current note; otherwise, do not add a pitch group symbol, and finally output a braille dot sequence of the pitch group symbol.

[0014] Preferably, the process of converting chords in musical notes into sequences of Braille dots comprises: Determine whether the next note has a chord node, and if so, obtain the scale node, octave node, and note value node of the current note, and record the corresponding attribute values ​​in temporary variables, then put the temporary variables in which the scale attribute value and octave attribute value are recorded into stacks s1 and s2 respectively, set the chord marker bit to 1, and repeat this process; If there is no chord node, it is necessary to determine whether the chord marker bit is 1; if the chord marker bit is not 1, convert into a braille dot sequence according to the normal note; if the chord marker bit is 1, mark the current note as a root note, and convert the root note into a braille dot sequence in combination with the attribute value of the note value; extract notes in the stack from the stack in order, while calculating the interval between the notes in the stack and the root note; convert the notes extracted from the stack into a braille dot sequence according to the interval; and set the chord marker bit to 0; repeatedly extract from the stack until it is empty, and output the braille dot sequence.

[0015] Preferably, the process of converting identical note connections in a note into a sequence of Braille dots comprises: Determine whether there is a same note connection line node in the child node of the notation node, and if there is a same note connection line node, traverse the next node, and if the next node is not empty but a note node, if it is determined that a same note connection line node and a chord node coexist in the note node, set both the chord marker bit and the double connection line marker bit to 1, and add a double connection line marker, and if it is determined that a same note connection line node and a chord node do not coexist in the note node, it is necessary to add a same note connection line marker; If the next node is empty, set the chord marker bit to 0, add a same note connection line marker, and finally output the braille dot sequence results to the same note connection line marker and double connection line marker.

[0016] Preferably, the process of converting arpeggios and fermatas in musical notes into a sequence of braille dots comprises: When a chord and an arpeggio coexist, an arpeggio marker needs to be added before the root note. Specifically, if there is no chord node in the next note, an arpeggio marker is added to the arpeggio of the current node, and a braille dot sequence of the arpeggio marker is output. When a chord and a fermata coexist, a fermata marker needs to be added before the root note; specifically, if the next note does not have a chord node, a fermata marker is added to the fermata of the current node, and a braille dot sequence of the fermata marker is output.

[0017] Preferably, the output module performs score presentation and Braille dot presentation when outputting the Braille music score in a visual manner; The music score presentation process involves matching the Braille music score with the Braille music symbol library to obtain the musical sequence of each measure. When the current measure is being read, the musical sequence is input, and the program is run to output the musical score image of the measure and display it synchronously on the screen. The braille dot presentation process is to supplement the length of the braille dot sequence in the braille music score, cut and store a string of characters from the supplemented braille dot sequence at a certain length, and create a figure braille dot for each line of the string of characters and display it synchronously on the screen; When outputting the Braille music score in an auditory manner, the output module matches the Braille music score with a Braille music symbol library to obtain the musical sequence of each bar. When the current bar is being read, the output module takes the musical sequence as input and runs the program to output the audio of the bar, which is saved in MIDI audio format, played back, and displayed synchronously on the screen.

[0018] Preferably, when the output module outputs the Braille music score in a tactile manner, the communication between the upper computer and the lower computer of the embedded system is via a serial port, and the data packet in the serial port communication protocol of the upper computer is 43 byte Each consists of 1 byte Data Head High byte , 1 byte Data Headline byte , 1 byte CRC check of byte and 40 byte Data byte In the serial port communication protocol of the lower-level computer, the data packet is 4 byte Each consists of 1 byte Data Head High byte , 1 byte Data Headline byte , 1 byte CRC check of byte and 1 byte Data byte is.

[0019] Preferably, the conversion module is further for converting the sequence of Braille dots into Braille ASCII code, and providing Braille dot presentation based on the Braille ASCII code. [Effects of the Invention]

[0020] Compared with the prior art, the beneficial effects of the present invention include at least the following:

[0021] The music / Braille text conversion system of the present invention has fast calculation speed and high accuracy, smoothly providing a new and viable means for Braille music score information, and can realize the built-in function of synchronized visual, auditory and tactile presentation of Braille music score, with clear and accurate visual display, complete auditory reproduction, clear tactile sensation and a better presentation effect, which can further improve the efficiency of visually impaired people learning music score and provide convenience for visually impaired people to obtain and touch-read a large amount of music material. [Brief explanation of the drawings]

[0022] In order to more clearly explain the technical solutions of the embodiments of the present invention or the prior art, the drawings necessary for describing the embodiments or the prior art will be briefly described below. Obviously, the drawings described below are only examples of the embodiments of the present invention, and those skilled in the art can further obtain other drawings based on these drawings without any creative efforts.

[0023] [Figure 1] FIG. 1 is a structural diagram of a music / Braille character conversion system according to an embodiment. [Figure 2] FIG. 2 is a block diagram showing the conversion process of the music / Braille character conversion system according to the embodiment. [Figure 3] FIG. 3 is a diagram showing the structure of a small node according to an embodiment. [Figure 4] FIG. 4 is a flowchart illustrating traversal of a MusicXML file according to an embodiment. [Figure 5] FIG. 5 is a flowchart showing a process of converting metronome information values ​​according to the embodiment. [Figure 6] FIG. 6 is a schematic diagram showing the extraction order of some child nodes of a measure according to the embodiment. [Figure 7] FIG. 7 is a flowchart showing conversion from a note node to a Braille dot sequence according to an embodiment. [Figure 8] FIG. 8 is a flowchart showing conversion from a tone group symbol to a Braille dot matrix according to an embodiment. [Figure 9] FIG. 9 is a flowchart showing the conversion from chords to a Braille dot matrix according to an embodiment. [Figure 10] FIG. 10 is a flowchart showing conversion from identical note connecting lines to a Braille dot matrix according to an embodiment. [Figure 11] FIG. 11 is a flowchart showing the conversion from arpeggios and fermatas to a Braille dot matrix according to an embodiment. [Figure 12]FIG. 12 is a flowchart illustrating a visual presentation according to an embodiment. [Figure 13] FIG. 13 is a flowchart for creating a Braille dot matrix according to the embodiment. [Figure 14] FIG. 14 is a flowchart illustrating an auditory presentation according to an embodiment. [Figure 15] FIG. 15 is a flowchart illustrating tactile presentation according to the embodiment. [Figure 16] FIG. 16 is a diagram showing the display result of the first measure of <<Canon in D>> according to the embodiment. DETAILED DESCRIPTION OF THE INVENTION

[0024] In order to make the objectives, technical solutions and advantages of the present invention clearer, the present invention will be further described in detail below by examples with reference to the drawings. It should be understood that the specific embodiments described herein are merely for interpreting the present invention, and do not limit the protection scope of the present invention.

[0025] Figure 1 is a structural diagram of an embedded music / Braille text conversion system capable of synchronized visual, auditory, and tactile presentation according to an embodiment. As shown in Figure 1, the music / Braille text conversion system comprises an acquisition module, a conversion module, and an output module. The acquisition module acquires music scores in MusicXML format, the conversion module converts the music scores in MusicXML format into Braille music scores, and finally the output module outputs the Braille music scores simultaneously in three ways: visually, auditorily, and tactilely. These three modules realize the conversion process shown in Figure 2.

[0026] In an embodiment, the acquisition module is for acquiring sheet music in MusicXML format, which can be directly downloaded from a website or converted from other music formats to MusicXML format using software such as Artinfuser Exercise.

[0027] In an embodiment, the conversion module is for converting the MusicXML format into a Braille score consisting of a sequence of Braille dots, and further converting the converted Braille dots into a Braille ASCII code. The process of converting the MusicXML format into a Braille score consisting of a sequence of Braille dots includes the following steps 1 to 5.

[0028] Step 1: Read and parse the MusicXML file to obtain a tree model corresponding to the MusicXML file structure. Specifically, use the xmlReadFile function to read the file pointer of the MusicXML file, and parse it using a DOM parser to establish a logical tree model.

[0029] Step 2: Obtain the root node of the MusicXML file structure from the tree model. Specifically, use the xmlDocGetRootElement function to obtain the root node curNode of the MusicXML file structure.

[0030] Step 3: Using the root node as the node of the initial traversal tree model, search for the node to be converted to Braille and obtain the node label content. Specifically, use the xmlNodeGetContent function to obtain the node label content for the node to be traversed. curNode->xmlChildrenNode is the set of child nodes of the root node.

[0031] Step 4: Convert the node label content into a Braille dot sequence based on the Chinese Braille Music Symbol Conversion Library. Specifically, convert the label content in the MusicXML file into a corresponding Braille dot sequence based on the correspondence between the node elements and Braille dot sequences in the MusicXML file. Then, process the traversed nodes according to simplified rules as needed. Some of the converted node results need to be placed at the very beginning of the entire score. For example, a certain score has metronome information for the entire score at the start. This metronome information needs to be placed before the start of the score to provide a playing metronome for visually impaired people. Furthermore, chords need to be processed separately. The traversal order of the MusicXML file for high-pitched music scores and the order of the Braille dot sequences to be displayed are exactly opposite, i.e., the root note is at the end and processed in a stack. The traversal order of the MusicXML file for medium / high-pitched music scores and the order of the Braille dot sequences to be displayed are the same, so no separate processing is required; they can simply be traversed sequentially.

[0032] Step 5: Free the file pointer using the xmlFreeDoc function to clear the memory dynamically claimed by all nodes in the MusicXML file.

[0033] In a MusicXML file structure, the outermost root node is the score element (score-partwise), which contains information such as copyright information and structural distribution, as well as at least one movement (part). Each movement primarily contains at least one measure. A measure contains many labels, such as score attribute information, dynamic markers, left / right hand exchange markers, barlines, and notes, as shown in Figure 3. A piece of music is primarily composed of measures, each of which primarily contains at least one note or rest. Therefore, the entire piece is basically composed of movements, measures, and note labels, which together form the layer-by-layer nested structure of a MusicXML file. Specifically, the outermost label is the movement, and the middle label is the measure, with each measure containing a note label. Therefore, the traversal process first traverses the outer labels, and then traverses the nested inner labels. As shown in Figure 4, the specific traversal process includes the following: first, determine whether the current node is a movement node; if the condition is met, further determine whether the child node of the movement is empty; if so, traverse the next sibling node of the movement; next, if the child node of the movement is not empty and is a measure node, analyze the attribute node and dynamic mark node of the measure node in order to obtain the attribute node label content and dynamic mark node label content; and if the child node of the measure node is a note node, analyze the descendant node of the note node to obtain the descendant node label content; and after completing the traversal of the descendant node of the measure node, traverse the next sibling node of the measure node.

[0034] The attribute node determines the key of the staff, including musical notation labels such as the standard quarter note value (divisions), key signature (key), time signature (time), number of staves (staves), and clef for each note. The standard quarter note value is used to measure the note value. The standard quarter note value element itself uses the quarter note as its basic unit, and each note has one standard quarter note value element. The value of the standard quarter note value element is used to calculate the duration of every note. For example, if divisions = 2 and the duration of a note = 1, the note is an eighth note. If a note is a rest, the program mainly relies on the standard quarter note value and duration values ​​to calculate the length of the rest. The key signature is placed adjacent to the standard quarter note value element and indicates the pitch of the tonic note of the scale using seven note names and ascending / descending symbols. When indicating a certain key signature on a musical staff, a few sharp or flat signs may be added before the time signature. Fifths are child nodes of the key signature and are used to indicate ascending or descending key positions. Positive values ​​indicate ascending keys, negative values ​​indicate descending keys, and 0 indicates neither ascending nor descending keys. Time signatures are generally expressed as fractions. In MusicXML files, time signatures have two sublabels: the number of beats and the beat type. The beats are represented by the numerator and the beat type by the denominator, intuitively describing how many beats (numerator) there are in a measure of music and how many quarter notes (denominator) each beat is. In the Chinese Braille Music Symbol Conversion Library, time signatures are generally expressed as Braille fractions, with the numerator written normally according to the Braille Arabic numerals and the denominator written according to the Braille Arabic numerals' drop points. If the staff number element is not present in the attribute, it is initially set to a single-line staff. Clefs determine the pitch placement of notes, and in MusicXML files, clefs are represented by two labels: a sign and a line. For example, the treble clef is commonly represented by G and the line on which it is located is 2, the bass clef is commonly represented by F and the line on which it is located is 4, and the c clef is commonly represented by C and the line on which it is located is 3.

[0035] Dynamic mark nodes can be multiple and do not necessarily have to be attached to notes. Their primary child node is the dynamic mark type (direction-type), which contains multiple child nodes such as coda, segno, dynamics, wedge, metronome, and pedal. Regarding codas, if a piece of music has a coda, there are typically two symbols in a code paragraph: the first ellipsis indicates the chord on the score, and the second indicates the chord, indicating that the paragraph within will be skipped after the repeat. Crescendos and decrescendos are a type of dynamic information and contain 13 symbols. These symbols are typically attached before notes, with wedge symbols in the wedge label written at the beginning and end of the note. There are two ways to display metronome information: first, by placing Italian words as numbers in the word label; second, by using "quarter note = number"; these formats are commonly used with metronome label identifiers. The program's metronome information numeric conversion process, shown in Figure 5, involves obtaining the metronome value of the metronome node, converting it to an integer value, converting the ones, tens, and hundreds digits of the integer value into individual numbers, and then converting each individual digit into a corresponding sequence of Braille dots. Pedal "pressing" and "lifting" are both displayed in the pedal label and are distinguished by different type attribute values.

[0036] There are many barline representation styles within the barline node, and different bar-style values ​​indicate these different barline styles, including whole end (light-heavy), dashed barline, and system end (light-light). Multiple endings in a score may be indicated by a single ending (e.g., "1.") or multiple endings separated by commas (e.g., "1., 2."). The type attribute value for an ending in a MusicXML file indicates the beginning and end of an ending. If any repeat signs appear in a score, the first note in each case should be marked with a key group symbol. When repeat signs and endings are used in combination, they are distinguished by a decrement number in the different parts, and if the decrement number and the following symbol are obfuscated, a "(3)" is inserted between them. Repeat start / end signs are indicated in a MusicXML file with the barline style heavy-light. In the MusicXML file, the ascending and descending sign information, the time signature information, and the dynamic marks all appear in order. In addition to the metronome information being placed first, the order of the corresponding braille dots also matches the order in the MusicXML file.

[0037] It should be emphasized here that, first, in addition to the note elements, there is an almost one-to-one correspondence between the other child nodes of the measure node and the braille symbols, and second, in addition to the metronome information, the order of the braille dot sequence and the order of node appearance in the braille score match. Therefore, the bar label content can be extracted using the simple method shown in Figure 6.

[0038] For note nodes, the notes are processed according to the Chinese Braille Music Symbol Conversion Library until the traversal of the child nodes of the measure is completed. Within notes, several child nodes are commonly used to indicate the characteristics of the current note. Note nodes include basic labels such as rest, pitch, chord, accidental, note value, dot, duration, voice, stem direction, staff, and beam. To obtain the corresponding Braille dot sequence, notes must be used in conjunction with the attribute values ​​of the scale and note value nodes in the pitch. The scale determines which note it is, and the note value determines the note's note value (how many quarter notes it is). All of these must be determined before the Braille dot sequence can be matched. In addition to basic labels, notes also include note modification labels, which are embodied in notation labels. The extraction and conversion of note nodes can be divided into three main parts: (1) extraction and conversion of key labels such as pitches that are closely related to notes; (2) extraction and realization of note chord parts; and (3) extraction and conversion of other nodes such as arpeggiates, fermatas, tied / double note connections, and slurs.

[0039] The pitch label indicates the current note's position on the staff, and its sublabels include the scale (step), octave, and alternation. The scale contains seven notes: C, D, E, F, G, A, and B. The octave corresponds to the nine key groups on a piano and is represented by the numbers 0 through 8, with 4 indicating the octave starting from middle C. The alternation indicates a semitone change; negative numbers indicate a descending key and positive numbers indicate an ascending key. While the pitch of a note can be clearly sensed from the staff, visually impaired people cannot sense this. To help visually impaired people distinguish between high and low notes, a key group symbol must be placed before each note. However, it is not necessary to attach a key group symbol to each note. The Chinese Braille Music Symbols stipulate how to use tone group symbols: first, a tone group symbol must be placed before the first note in a score or line; second, there must be no other symbols between the tone group symbol and the note; third, for two adjacent notes in the same tone group, if the distance between them is within five degrees, no tone group symbol is required; but if the distance is six degrees or more, a tone group symbol must be added; fourth, for two adjacent notes in different tone groups, if the distance between them is within three degrees, no tone group symbol is required; but if the distance is four degrees or more, a tone group symbol must be added. The addition of a tone group symbol is related to the pitch of the note, and must be determined based on the difference between the scale of the previous note and the scale of the current note, as well as whether the octave in which the previous note is located is the same as the octave in which the current note is located.

[0040] As shown in Figure 7, the process of converting the key point elements of a note node into a braille dot sequence starts from the acquired note node. First, it determines whether the child node of the note has a note node. If there is a note node, it records the related information of the note. If there is no note node, it continues to determine whether there is a pitch node. Here, it is divided into two situations: (1) If there is a pitch node, it is proven that there is one note among C, D, E, F, G, A, and B. The attribute values ​​of the note's scale node and octave node are obtained and temporarily saved; then the attribute value of the note value node is obtained and temporarily saved; and then the tone group symbol processing is performed according to the flowchart of Figure 8; then the pitch node is changed according to the recording situation of the note. (2) If there is no pitch node, it is necessary to determine whether there is a rest. If there is a rest, it is necessary to jump to the step of recording the attribute value of the pitch node and continue to execute the subsequent steps. If there is no rest, it is necessary to perform error reporting processing.

[0041] As shown in Figure 8, the process of processing a tone group symbol includes the following: First, the scale and octave of the previous note are marked as step_before and octave_before, respectively, and the scale and octave of the current note are marked as step_after and octave_after. Next, determine whether octave_before and octave_after are equal, i.e., whether they belong to the same tone group. If they are equal, the two notes are said to belong to the same tone group; if not, they belong to different tone groups. Here, two situations are distinguished: (1) two notes belong to the same tone group. According to the rule, tone group symbols are not added within the fifths of the same tone group. Therefore, it is necessary to determine whether the absolute value of the interval between the two notes is greater than 5. If it is greater than 5, the corresponding tone group symbol is added before the current note; if not, no tone group symbol is added. (2) two notes belong to different tone groups. Here, the situation of belonging to different tone groups can be further divided into two situations: a) if the octave difference between the two notes is greater than 1, that is, they do not belong to adjacent tone groups, the degree of the notes must be greater than 4, and the corresponding tone group symbol must be placed before the current note; b) if the two notes belong to two adjacent tone groups, and the interval between the notes is less than a third, it is necessary to determine whether the interval between the notes is greater than 3, and if so, the corresponding tone group symbol must be placed before the current note; if not, the tone group symbol is not placed, and finally the braille dot sequence of the tone group symbol is output.

[0042] Regarding rests, if a note has a rest node, it must not have a pitch node either. This is because pitches and rests cannot coexist, and rests directly substitute for pitch in MusicXML files. Like notes, there are different types of rests, such as whole rests, half rests, quarter rests, and eighth rests. Also, when an entire measure is at rest, a whole rest must be used. Similarly, when a note has a rest, the note value of the rest must be determined by the note value attribute value.

[0043] The note value of a note or rest is determined by its type, as well as its standard quarter note value (divisions) and duration attribute values. Divisions indicates the standard quarter note value units, while duration indicates the note value of the current note or rest. If the value of duration is twice the value of divisions, the current note or rest is a half note, and if the value of duration is equal to the value of divisions, the current note is a quarter note, and so on.

[0044] A dot is typically placed after a note. The actual duration of a dotted note is a half-note extension of the current note, while the actual duration of a double-dotted note is a three-quarter extension of the current note. Two dotted notes are called a double-dotted note. While we mentioned alterations (sharp, flat, double sharp, and double flat) above when describing attributes, rising and falling signs and naturals can also appear on notes, and are indicated by different values ​​of the alteration label. In braille music, a sharp, flat, or natural sign placed before a note indicates a temporary semitone rise or fall. Each sharp and flat sign is valid for identical notes within a measure. If a note in the same measure does not rise or fall, a natural sign must be placed before the note. Therefore, when processing a note's child nodes, we first determine whether it contains alterations.

[0045] Chord processing is a challenge for the system. A chord refers to a combination of stacked pitches according to interval relationships, either thirds or other than thirds. These notes may be played one after the other or simultaneously. A chord contains at least three notes and at most seven. While visually impaired people can distinguish between triads and heptads when learning chords, visually impaired people rely on the number of intervals to calculate chords. An interval refers to the difference in pitch between two notes, and this difference is expressed as a "degree." In other words, the degree is used to determine the pitch relationship. A degree is the number of notes contained in an interval, and is expressed on a staff as the number of tones and intervals contained between the two notes. For example, the degree of the two chords CE (from do to mi) is a third. The two tones may contain one interval, or the two intervals may contain one tone, but both represent a distance of three degrees. The degree of an interval only describes the degree difference; it does not represent exact equality. For example, DE and FG are both second intervals, but they are two completely different intervals, and the only difference is the number of degrees. Therefore, the name of the interval cannot be determined simply by the number of degrees; it must also be combined with the number of notes.

[0046] Chord notes are the individual notes that make up a chord. In a chord, the highest note is called the crown note, the lowest note is called the root note, and the remaining notes are named according to their interval relationship to the root note. In a triad, the notes are arranged according to a third interval, with the lowest note called the root note (written as 1), the middle note, because it is in a third interval relationship with the root note, called a triad (written as 3), and the highest note, because it is in a fifth interval relationship with the root note, called a quintet (written as 5).

[0047] When chords appear in braille music, one note is selected as the root note of the chord and is indicated based on the degree of the interval between the other notes and the root note. In staff notation, intervals are generally calculated from bottom to top, with the lowest note being the root note. However, when notating by voice, the bass staff calculates the degree from bottom to top, while the treble staff calculates the degree from top to bottom. That is, in left-hand / right-hand music, the treble staff calculates the degree from top to bottom using the highest note as the root note, while the bass staff calculates the degree from bottom to top using the lowest note as the root note. Generally, interval symbols are written after the notes. Interval symbols for ninths and intervals within a ninth do not require a key group symbol, but for intervals greater than a ninth, a pitch symbol is required to indicate compound intervals greater than a ninth.

[0048] As shown in Figure 9, the process of converting a note chord portion into a Braille dot matrix involves inputting the label content information of the current note node. Determining a note requires a combination of scale and note value, while determining a key group symbol requires an octave value. Therefore, if there is a chord, the scale label, octave label, and note value label of the current note are obtained and the corresponding values ​​are recorded in temporary variables step_before, octave_before, and t. Then, the temporary variables step_before and octave_before are respectively put into stacks s1 and s2, and the chord marker bit chord is set to 1. This process is repeated until the next note has no chord. If there is no chord, it is necessary to determine whether the chord marker bit is 1. If the chord marker bit is not set to 1, it is said that there is no chord in the entire process. Then, the process proceeds according to the normal note processing steps shown in Figure 7. If the chord marker bit is 1, it means that a chord exists and has been extracted from the stack, and the top note on the stack is the note after the root note, so the current note is marked as the root note, and the value of step_before is assigned to the temporary variable root_step, the value of octave_before is assigned to the temporary variable root_octave, and the root note is converted into a braille dot sequence by referring to the value of t. Finally, the elements in the stack are extracted from the stack in order, and at the same time, the interval between the note in the stack and the root note is calculated, and the notes extracted from the stack are converted into a braille dot sequence according to the interval, and the chord marker bit is set to 0, and the stack is repeatedly extracted until it is empty, and the braille dot sequence is output, and the whole process is completed. Methods for calculating interval include: for notes in the same pitch group, interval is equal to the absolute value of the difference between step_before and root_step plus one, and for notes in different pitch groups, interval is equal to the absolute value of the difference between the highest note and root_step plus one, plus the absolute value of the difference between step_before and lowest note.

[0049] All chords in a MusicXML file are described in order from bottom to top, with the lowest note being the chord.<chord / > Since there are no markers, when the chords are notated by voice on the staff, the chords in the high-pitched staves are in the opposite order, while the chords in the low-pitched staves are in the same order. Therefore, the chords in the high-pitched staves must be processed separately in the braille music. In the present invention, the first-in, last-out feature of the stack is used to stack the chords from the lowest note to the top, and then the last chord is stacked and then extracted in order, with the first note extracted from the stack being the root note.

[0050] What needs to be explained here is that the first note that starts the chord <chord / > Since there is no marker, the present invention determines whether the next note has a chord and then puts it into the stack. If the next note has a chord, it puts the current note information into the stack. That is, if it is the last note of the chord, the last note is not put into the stack, but the last note is exactly the root note. Therefore, the present invention directly marks the last note as the root note, and after processing the root note according to a normal note, extracts the note information in the stack from the stack and calculates the degree of the root note of the chord.

[0051] Tied note connections, also called note extension lines, are used to connect notes of the same pitch. When two or more notes of the same pitch (or chords) are connected, double connections must be used. Therefore, before determining whether a tied note connection or double connection exists, it is necessary to determine whether the next note node contains a chord node or a tied note node. If a tied note node exists, the marker bit is marked as tied_notation, and if there is a chord, the marker bit is marked as fermata_chord.

[0052] As shown in Figure 10, the process of converting identical note connections in a note into a braille dot sequence involves traversing the input "notations" child node. If an identical note connection / double connection node is found, the next node is traversed. If the next node is not empty and is a note node, it is determined whether an identical note connection / double connection node and a chord node coexist in the note node. If so, the chord marker bit "fermata_chord" and the double connection marker bit "tied_notation" are both set to 1 and a double connection marker is added. If not, an identical note connection marker needs to be added. If the next node is empty, this indicates that the next node may contain a chord, but the chord has already ended or there is no chord. In both cases, the chord marker bit "fermata_chord" is set to 0 and an identical note connection marker is added. Finally, the braille dot sequence results are output to the identical note connection marker and double connection marker, and the process ends.

[0053] As shown in Figure 11, when a chord and an arpeggio coexist, or when a chord and a fermata coexist, a corresponding braille marker, i.e., a marker corresponding to the arpeggio or the fermata, can be added before the root note. This means that the fermata_chord marker bit must be used to determine whether to add arpeggio and extended note braille markers. <chord / > If there is no braille symbol, add arpeggio and fermata correctly, otherwise do not add them.

[0054] A slur, also called an arc, is a line that connects notes, allowing multiple notes in a musical score to be played consecutively. In a MusicXML file, a slur is defined by the type attribute, where "start" indicates the beginning and "stop" indicates the end of the slur. In Braille music, a single arc is used to connect two notes. JPEG0007774735000001.jpg46166(14) Markers must be placed, and two lines are needed to connect the three notes. JPEG0007774735000002.jpg46167(14) marker must be added. Therefore, adding a slur marker must be combined with the type attribute, and slur appears and type="start" after each note. JPEG0007774735000003.jpg46167(14) A marker must be added after the note only if the type attribute value is "stop". JPEG0007774735000004.jpg45158(14) No markers are added. In addition, in the Chinese Braille Music Symbols, when connecting 2 to 4 notes, a slur is used, and when connecting 5 or more notes, a large slur is used. JPEG0007774735000005.jpg42158(14,14)...(14). If a slur or large slur spans two measures, leave one square between the measures, i.e. JPEG0007774735000006.jpg45158 (empty square) needs to be added.

[0055] The articulations node contains child nodes such as accent, staccatissimo, staccato, tenuto, doit, falloff, and plop. These nodes have a one-to-one correspondence with the Braille dots, allowing direct matching and conversion in the program. Note their placement: staccato, staccatissimo, substaccato, tenuto, and strong accent are all written before the note; trill, upper trill, and lower trill are all written before the note; plop and slip symbols are also placed before the note, while doit and falloff symbols are placed after the note.

[0056] The articulation node also has a commonly used child node technical, whose child node fingering indicates fingering symbols, denoted by 1, 2, 3, 4, and 5. The first finger is the thumb and is marked as 1, the second finger is the index finger and is marked as 2, the third finger is the middle finger and is marked as 3, the fourth finger is the ring finger and is marked as 4, and the fifth finger is the little finger and is marked as 5. Braille markers for these fingering symbols are all written after the notes, and there is also a one-to-one correspondence between the nodes and the Braille dots.

[0057] When notating music for separate hands, the left-hand symbol is inserted before the beginning of the left-hand music. JPEG0007774735000007.jpg34124(456,345) must be added, and the right hand symbol must be added before the beginning of the right hand music. JPEG0007774735000008.jpg34124(46,345) should be added. Please note that the left and right hand symbols should be added before the note, and the key group symbol should be added before the first note of the series of left and right hand symbols. JPEG0007774735000009.jpg38124(1) dots, JPEG0007774735000010.jpg39125(2) dots, JPEG0007774735000011.jpg40124(3)If a dot is included, JPEG0007774735000012.jpg35124(3) symbols should be added. The notation method used in this invention is to notate one measure at a time, that is, first record one measure of bass music, then record one measure of treble music. Therefore, before writing the note information for a measure, it is necessary to first write the left-hand symbols, then write the right-hand symbols, and finally write the note information for the right-hand music. This process continues one measure at a time until one musical line or the entire piece is completed.

[0058] In addition, it is necessary to traverse some descendant nodes of the measure node more deeply. For example, the label fifths is in the label key, the labels beats and beat-type are in the label time, and the labels sign and line are in the label clef, but key, time, and clef are in the attributes label of the measure. In this case, it is necessary to first access the related labels in the measure to correctly obtain the musical score information and correctly represent it according to the Chinese Braille Music Symbols, and then return to the previous step and start traversing the next measure node until the traversal of the measure in the part node is completed.

[0059] The process continues until the part node has been traversed and there are no more part nodes, thereby completing the traversal of the entire score.

[0060] In an embodiment, each musical score S consists of at least one movement P, as shown in equation (1-1), where F is a composition function. Each movement P consists of at least one measure M, as shown in equation (1-2), where P i denotes the ith part of the song, and M i indicates the ith measure, and the musical score M of each measure is given by formula (1-3), and M attributes denotes the set of musical score attributes, and M direction denotes the set of score dynamics information, and M note denotes the set of musical note information, and M barline indicates the end of the score, and M backup indicates the first movement of musical time, i.e., the left / right hand exchange mark, and M others denotes the set of other nodes in the score, and these sets are shown in formula (1-4).

[0061]

number

[0062] The conversion rule that is followed to convert from MusicXML format to Braille dot sequence is denoted as R, the conversion relationship from music score S to Braille dot sequence Bri_Dot is shown in formula (1-5), the conversion rule is shown in formula (1-6), and M write indicates the writing rule, and M conversion shows the rules for converting musical scores into Braille scores, and M simplify These are simplified rules for converting sheet music into Braille notation.

number

[0063] Research into the MusicXML file structure and the converted Braille dot sequences has revealed that the correspondence between MusicXML nodes and Braille dot sequences is not only one-to-one, but also one-to-many, many-to-one, and even many-to-many. The converted sequence structure can be explained by formula (1-7). (< >) indicates the parent-child relationship between nodes, p is the set of parent nodes in the MusicXML file, and p i denotes one parent node in set p, and s i Ga p i indicates the child nodes of b i , b i ' denotes the generated Braille dot sequence, where i=1, 2, 3, ...

number

[0064] The conversion of the piano into braille dot sequences for the left and right hands is explained as formula (1-8),

number

number

number

number

[0065] In the embodiment, after acquiring a sequence of Braille dots of a musical score, the sequence of Braille dots is converted into Braille ASCII code by comparing it with a comparison table of Braille dots and Braille ASCII code.

[0066] In this embodiment, the output module is designed to simultaneously output three types of senses: visual, auditory, and tactile, in an embedded system. For visual sense, a 10-inch IPSHDMI screen displays musical scores in units of bars, and corresponding electronic Braille dots are displayed below the scores, making it easier for visually impaired users to assist visually impaired users in their learning. For auditory sense, audio files are generated for each bar of musical score and played in dual channels on the ALC5640 audio chip. This adds an aural sensation when visually impaired users touch the Braille dots to learn, making it easier for them to understand and feel the music. For tactile sense, a 40-square Braille dot matrix is ​​displayed, allowing users to learn the music by touching and reading the raised Braille dots. To facilitate learning Braille music scores on other devices, the present invention uses two output formats: Braille dots and Braille ASCII code. At the same time, a correspondence table between Braille dots and Braille ASCII code can be constructed to convert Braille dots into Braille ASCII code.

[0067] The visual presentation is mainly divided into two parts: music score presentation and braille dot presentation. A 10-inch IPSHDMI® screen is used as the medium for music score display, with the music score displayed in picture format, with each picture displaying one measure of music score information. The Linux® system drives the IPSHDMI® screen, allowing the music score content to be displayed on the screen, achieving high resolution and good color reproduction, making the music score clearer and easier to read. Below the music score, the system displays corresponding braille dots, making it easier for visually impaired users to understand the content of the score. The coordinated presentation of braille dots and music score information provides visually impaired users with a more complete understanding and experience of music scores.

[0068] Regarding sheet music presentation, there are currently many notation software programs that can edit MusicXML files. For example, MuseScore is open-source notation software that can not only derive sheet music in MusicXML format, but also save sheet music in formats such as PNG and MP3. Importantly, MuseScore is compatible with Linux® systems and is open-source. To achieve automation and accurately obtain sheet music information for each measure, the system takes advantage of MuseScore's advantages and incorporates the music21 library into MuseScore. It is written in Python, allowing users to view sheet music in graphical format.

[0069] Forming a musical score image requires a sequence of time signatures, notes, etc., which are generated by the Chinese Braille music score conversion system based on the MusicXML file. As shown in Figure 12, the visual presentation process includes the following: First, the MusicXML file is parsed using DOM to generate a tree structure. When traversing the DOM tree structure, a Chinese Braille music symbol library is matched and the musical sequence required for visually presenting the musical score is output in units of bars. The musical sequence includes information such as time signatures, ascending / descending keys, and pitches. The musical sequence is saved in a temporary array. When the current bar is viewed, the Python program takes the musical sequence as input and runs the program to output the musical score for that bar. The score is saved in picture format and synchronously displayed on the IPSHDMI screen. The system continues to traverse each bar in turn until the traversal of the musical score file is completed.

[0070] Regarding Braille dot presentation, the Braille dot sequence or corresponding Braille ASCII code result obtained by the music / Braille character conversion system is primarily displayed as a Braille dot matrix, allowing visually impaired individuals to accurately understand the meaning of the output. However, there are two types of visual impairment: total blindness and low vision. To facilitate use by low-vision individuals, this system utilizes their remaining vision to display the Braille dot sequence converted from the music score in Blair Braille format on a visualization screen, while also contributing to Braille instruction for visually impaired individuals. The Braille dot matrix is ​​created using QT mapping events. The system converts the acquired Braille dot sequence into the corresponding dot position. Braille music scores are displayed in units of measures in embedded systems, so one measure of music score may contain fewer than 40 Braille squares. However, because the system arranges 40 squares as one row, some Braille dot strings may not be fully utilized. To solve this problem, braille dot strings shorter than 40 squares must be supplemented with "0"s to reach a length of 240.

[0071] As shown in Figure 13, the process of creating a Braille dot matrix involves creating a QPainter object, initializing the paintbrush and brush, converting the MusicXML music score to obtain a Braille dot sequence, padding Braille dot sequences with lengths less than 240 with zeros, cutting the strings in the Braille dot sequence to lengths of 6 and storing them in a string array list, and if the length of the string in the Braille dot sequence is less than 40, taking the corresponding string array and creating a Braille dot graphic based on the string. The Braille dot creation function is drawBraille(QPoint d, QPainter *p, QString s), which returns an empty value. This function first takes the starting point coordinate QPoint value, the QPainter object, and the Braille dot sequence to be created. This function then executes 40 Braille drawing operations, thereby displaying the Braille dots on the visualization screen.

[0072] For auditory presentation, this embodiment selects the ALC5640 audio chip, which is a dual I2S interface audio decoder and can be used in an audio hub. It integrates a dynamic range controller and a 7-band parametric equalizer, as well as a digital-to-analog converter with a 100dBA signal-to-noise ratio and an analog-to-digital converter with a 94dBA signal-to-noise ratio, offering the advantages of high performance and low power consumption. Inside the embedded system, the I2S pin of the ALC5640 audio chip is connected to the I2S pin of the main control chip RK3399, and the Class-D output can directly drive 1.5W single-channel or 2.5W single-channel audio. Audio can be output via an external 3.5mm earphone interface or dual-channel audio output. The auditory output of music is different from the audio output of text, so different processing is required. The music21 library is introduced and used in conjunction with MuseScore, and the programming is done in Python. The music sequence obtained from the MusicXML to Chinese Braille music notation conversion system is input into the program, and the program generates a MIDI format music notation audio file.

[0073] As shown in Figure 14, the auditory presentation process includes the following: First, a MusicXML file is input. Next, a tree structure is generated through DOM analysis. During the DOM tree traversal, node information is matched to a Chinese Braille music symbol translation library. At the same time, the musical sequence required for visual presentation of the score is output in units of bars and saved in a temporary array. When playback reaches the current bar, the Python program takes this sequence as input, runs the program, and outputs the audio of one bar of the score, which is saved in MIDI audio format and played back. The system continues to traverse each bar in turn until the MusicXML score file is traversed.

[0074] For tactile presentation, two 40-square Braille dot display devices are used, driving a total of 640 Braille dots. The Braille output from the music Braille conversion program is divided into six squares, with the fourth and eighth dots used to identify the Braille input position. Therefore, the Braille 01 dot sequence acquired by the conversion program must be subjected to secondary processing to convert the 6-dot Braille into 8-dot Braille. Of the 6-dot and 8-dot Braille, the 8-square Braille has four and eight more dots per square than the 6-square Braille. To convert the 6-dot Braille into 8-dot Braille, the fourth and eighth dots must be supplemented with 0 when acquiring the Braille dots for each square. Thus, the 6-dot Braille can be converted into the corresponding 8-dot Braille.

[0075] The serial port is the main interface for communication between the host computer and the subordinate computer, and is also an important interface for transmitting protocol data between the host computer and the subordinate computer. To ensure reliable data transmission and efficient communication, the present invention defines a serial port communication protocol. Braille dot sequences are stored as hexadecimal data via the UART serial port and sent to the subordinate computer. An error detection mechanism must be installed to determine whether the received data is correct. The present invention selects a CRC8 mechanism for checking, which can completely detect all odd random errors and sudden errors of length 8 or less.

[0076] In the serial port communication protocol of the host computer, the data packet is 43 byte Each consists of 1 byte Data Head High byte , 1 byte Data Headline byte , 1 byte CRC check of byte and 40 byte Data byte That's it. Start byteindicates the start of a protocol data frame, and the serial port communication protocol specifies that the data frame sequence is 0x7A, 0x6A. byte This is the result of checking the data byte indicates the specific content of the data. The lower-level computer only needs to check whether the data is correct after receiving it, and does not need to send excessive information, so the data packet of the lower-level computer is simpler, and byte Data Head High byte , 1 byte Data Headline byte , 1 byte CRC check of byte and 1 byte Data byte Contains only.

[0077] As shown in Figure 15, the tactile presentation process includes the following: First, an array protocolArray is defined for transmitting data via serial port communication, and an array dataArray is defined for containing Braille dot sequence data and calculating a CRC8 check value. When serial port communication starts, the high-level data of the data header is added to the protocolArray array. byte / low byte The information is stored, i.e., byte Give the value 0x7A to byte is given a value of 0x6A, the length of the braille dot sequence is set as len, the variable i is set to 0, and the braille dot sequence is traversed in order. If i is less than len, only 6-bit braille strings are processed each time, and converted to 8-dot braille by adding 0 to the 3rd and 7th bits. Next, the 8-dot braille string is read in binary, converted to hex data, and saved in the dataArray array in hexadecimal format. Traversing backward in order, the braille dot sequence is converted to hexadecimal, and the pointer movement is the marker for the length of the braille dot sequence this time. byteIf it is greater than , the system will calculate the CRC8 check value by table lookup based on the data in dataArray and store the check value in the protocolArray array. Then, it will combine the value in dataArray and the value in protocolArray and send it to the subordinate computer. After receiving the data, the subordinate computer will byte / low byte Then, the data byte The calculated value is compared with the CRC8 check value sent from the upper computer, and if the check results are different, the data sent from the lower computer is byte If the check result is the same, the data sent from the lower computer is returned as 0x00. byte is set to 0x01, and the device is driven to display a Braille dot matrix.

[0078] In this embodiment, the integrated overall effect of visually, audibly, and tactilely synchronized presentation is shown in Figure 16, taking the first measure of "Canon in D" as an example. The display results of each part of the integrated system, including the musical score, electronic dot sequence, audio waveform, and braille dot sequence, are shown in Figure 16.

[0079] The embedded music / Braille text conversion system capable of synchronized visual, auditory, and tactile presentation of the above embodiment seamlessly provides a new and viable means for Braille music score information and can be widely applied in various embedded scenarios and environments used by the visually impaired. It conveniently and efficiently generates a large amount of Braille digital resources, effectively reduces development costs, shortens the development process, improves development efficiency, and has greater practicality. At the same time, the developed embedded function of synchronized visual, auditory, and tactile presentation of Braille music score can further improve the efficiency of musical score learning for the visually impaired and demonstrates a new perspective for the application of artificial intelligence in the field of visual impairment. At the same time, it can also provide a model and reference for research in other fields of visual impairment.

[0080] Although the specific embodiments described above have described in detail the technical solutions and beneficial effects of the present invention, it should be understood that the above description is merely the best embodiment of the present invention and is not intended to limit the present invention, and any modifications, supplements, equivalent replacements, etc. made within the principle scope of the present invention shall all be included in the protection scope of the present invention.

Claims

1. An embedded music / Braille text conversion system capable of visually, aurally, and tactilely synchronized presentation, comprising: an acquisition module for acquiring a score in MusicXML format; a conversion module for converting a musical score in MusicXML format into a Braille musical score consisting of a sequence of Braille dots; an output module for simultaneously outputting the Braille music score in three ways: visually, aurally, and tactilely; The step of converting a musical score in MusicXML format into a Braille musical score consisting of a sequence of Braille dots includes: Step 1: reading and analyzing a MusicXML file representing a musical score to obtain a tree model corresponding to the MusicXML file content; Step 2: obtaining a root node of the MusicXML file structure from the tree model, and using the root node as a node in the initial traversal tree model, searching for a node to be converted into Braille and obtaining the node label content; Step 3: converting the node label content into a Braille string based on a Chinese Braille music symbol correspondence library to obtain the Braille music score; The step of searching for a node to be converted into Braille and obtaining node label contents using a root node as a node of the initial traverse tree model includes: determining whether the current node is a movement node; if the condition is met, further determining whether the child node of the movement is empty; if it is empty, traversing the next sibling node of the movement; if the child node of the movement is not empty and is a measure node, analyzing the attribute node and dynamic mark node of the measure node in order to obtain the attribute node label content and the dynamic mark node label content; if the child node of the measure node is a note node, analyzing the descendant node of the note node to obtain the descendant node label content; and traversing the next sibling node of the measure node after completing the traversal of the descendant node of the measure node; The process of converting chords in musical notation into a sequence of Braille dots is Determine whether the next note has a chord node, and if so, obtain the scale node, octave node and note value node of the current note, and record the corresponding attribute values ​​in temporary variables, then put the temporary variables in which the scale attribute value and octave attribute value are recorded into stacks s1 and s2 respectively, set the chord marker bit to 1, and repeat in this manner; If there is no chord node, it is necessary to determine whether the chord marker bit is 1; if the chord marker bit is not 1, convert into a braille dot sequence according to the normal note; if the chord marker bit is 1, the top note in the stack is the note after the root note, and the current note is marked as the root note, and combined with the attribute value of the note value, convert the root note into a braille dot sequence; extract the notes in the stack from the stack in order, and at the same time calculate the interval between the note in the stack and the root note; the interval calculation method includes: for notes in the same pitch group, the interval is equal to the absolute value of the difference between the note in the stack and the root note plus 1; for notes in different pitch groups, the interval is equal to the absolute value of the difference between the highest note and the root note plus 1, and then the absolute value of the difference between the note in the stack and the lowest note plus 1; convert the note extracted from the stack into a braille dot sequence according to the interval, and set the chord marker bit to 0; repeatedly extract from the stack until it is empty, and output the braille dot sequence; When the output module outputs the Braille music score in a tactile manner, the communication between the upper and lower units of the embedded system is via a serial port interface. In the serial port communication protocol of the upper unit, a data packet consists of 43 bytes of hexadecimal data, each of which is composed of 1 data head high byte, 1 data head low byte, 1 CRC check byte, and 40 data bytes. In the serial port communication protocol of the lower unit, a data packet consists of 4 bytes of hexadecimal data, each of which is composed of 1 data head high byte, 1 data head low byte, 1 CRC check byte, and 1 data byte. To present using the tactile method, the embedded music / Braille character conversion system uses two Braille dot display devices as the lower-level machine, and the upper-level machine performs secondary processing on the Braille dot string to convert 6-dot Braille into 8-dot Braille, which can be presented in a synchronized manner visually, audibly, and tactilely.

2. The embedded music / Braille character conversion system as described in claim 1, characterized in that when converting metronome information in the dynamic mark node label content into a Braille dot sequence, the system includes obtaining the metronome value of the metronome node, converting the metronome value into an integer number, converting the ones digit, tens digit, and hundreds digit of the integer number into individual numbers, and converting each individual number into a corresponding Braille dot sequence.

3. The process of converting musical notes into Braille dot sequences is For the input note node, determine whether there is a changed note node in the child node of the note node, and if there is a changed note, record the related information of the changed note; if there is no changed note, continue to determine whether there is a pitch node; If there is a pitch node, obtain and temporarily store the attribute values ​​of the scale node and octave node of the note, then obtain and temporarily store the attribute values ​​of the note value node, perform tone group symbol processing, then determine whether to add a changed note type or not to add a braille dot sequence corresponding to the changed note according to the recording status of the changed note, then determine a braille dot sequence corresponding to the note based on the scale and note value temporarily stored, further determine whether to add a braille dot sequence corresponding to the augmentation dot, and finally return the braille dot sequence result; 2. The embedded music / Braille text conversion system as described in claim 1, characterized in that if there is no pitch node, it determines whether there is a rest node, and if there is a rest node, jumps to a step of recording the attribute value of the pitch node if there is a rest node, and continues to execute subsequent steps, and if there is no rest, performs error reporting processing.

4. The step of performing tone group symbol processing is: Obtaining the scale and octave of the previous note and the scale and octave of the current note, and then determining whether the two adjacent notes belong to the same tone group based on whether the octaves of the two adjacent notes are equal; If two adjacent notes belong to the same pitch group, determine whether the absolute value of the pitch of the two notes is greater than 5, and if it is greater than 5, add a pitch group symbol in front of the current note, otherwise do not add a pitch group symbol; 4. The embedded music / Braille character conversion system capable of presenting visually, aurally, and tactilely synchronized according to claim 3, further comprising: when two adjacent notes belong to different pitch groups, if the difference between the two notes is two pitch groups or more, adding a pitch group symbol in front of the current note; if it is determined that the two notes belong to two adjacent pitch groups and the note interval is greater than 3, adding a pitch group symbol in front of the current note; otherwise, not adding a pitch group symbol, and finally outputting a braille dot sequence of the pitch group symbol.

5. The process of converting identical note connections in a note into a sequence of braille dots is: Determine whether there is a same note connection line node in the child node of the notation node, and if there is a same note connection line node, traverse the next node, and if the next node is not empty but a note node, if it is determined that a same note connection line node and a chord node coexist in the note node, set both the chord marker bit and the double connection line marker bit to 1, and add a double connection line marker, and if it is determined that a same note connection line node and a chord node do not coexist in the note node, it is necessary to add a same note connection line marker; 2. The embedded music / Braille character conversion system as described in claim 1, which can be presented visually, audibly, and tactilely synchronized, further comprising: if the next node is empty, setting the chord marker bit to 0, adding a same note connection line marker, and finally outputting the Braille dot sequence results to the same note connection line marker and the double connection line marker.

6. The process of converting arpeggios and fermatas in musical notes into braille dot sequences is: When a chord and an arpeggio coexist, an arpeggio marker needs to be added before the root note. Specifically, if there is no chord node in the next note, an arpeggio marker is added to the arpeggio of the current node, and a braille dot sequence of the arpeggio marker is output.

2. The embedded music / Braille text conversion system as claimed in claim 1, characterized in that when a chord and a fermata coexist, a fermata marker needs to be added before the root note; specifically, if the next note does not have a chord node, a fermata marker is added to the fermata of the current node, and a sequence of Braille dots of the fermata marker is output.

7. the output module presents the Braille score and the Braille dots when outputting the Braille score in a visual manner; The music score presentation process involves matching the Braille music score with the Braille music symbol library to obtain the musical sequence of each measure. When the current measure is being read, the musical sequence is input, and the program is run to output the musical score image of the measure and display it synchronously on the screen. The braille dot presentation process is to supplement the length of the braille dot sequence in the braille music score, cut and store a string of characters from the supplemented braille dot sequence at a certain length, and create a figure braille dot for each line of the string of characters and display it synchronously on the screen; 2. The embedded music / Braille text conversion system as claimed in claim 1, characterized in that, when outputting the Braille music score in an auditory manner, the output module matches the Braille music score with a Braille music symbol library to obtain the musical sequence of each bar, and when reading the current bar, takes the musical sequence as input, executes a program to output the audio of the bar, saves it in MIDI audio format, plays it, and displays it synchronously on the screen.

8. The embedded music / Braille character conversion system capable of visually, aurally, and tactilely synchronized presentation, as described in claim 1, characterized in that the conversion module is further configured to convert the Braille dot sequence into Braille ASCII code and present Braille dots based on the Braille ASCII code.

Images