An embedded music / Braille conversion system capable of synchronizing visual, auditory and tactile presentations

Abstract

The present invention discloses an embedded music / Braille text conversion system capable of visually, audibly and tactilely synchronously presenting, which includes an acquisition module for acquiring musical scores in MusicXML format, a conversion module for converting the musical scores in MusicXML format into Braille musical scores consisting of Braille dots, and an output module for simultaneously outputting the Braille musical scores in three ways: visually, audibly and tactilely. The conversion system has a fast calculation speed and high accuracy, smoothly provides a new feasible means for Braille musical score information, and can realize the embedded function of visually, audibly and tactilely synchronously presenting Braille musical scores, with clear and accurate visual display, complete auditory reproduction, clear tactile sensation, and better presentation effect, which can further improve the efficiency of visually impaired people learning musical scores, and provide convenience for visually impaired people to acquire and touch-read a large amount of music materials.

Description

[Technical field]

[0001] The present invention relates to the technical field of computer science Braille music score processing and analysis, and in particular to an embedded music / Braille text conversion system that can be presented synchronously visually, audibly, and tactilely. [Background technology]

[0002] Braille, also called raised braille or raised characters, is a tactile character designed specifically for the visually impaired. It is made by applying different combinations of raised dots to paper using a braille board, braille machine, braille printer, etc. Generally, each block of braille has 6 dots, 123 from top to bottom on the left side and 456 from top to bottom on the right side, which is called 1 square.

[0003] Braille is the core of the language and character rights of the visually impaired, a key point for improving the cultural and educational level of the visually impaired, and an important means for ensuring that the visually impaired can share their cultural and educational rights. At present, many Chinese / Braille conversion systems have been researched in China, but Braille music scores and related materials mainly rely on human translation, which cannot meet the large-scale needs of the visually impaired. At the same time, most research results can only obtain electronic Braille dots through conversion systems, and require the placement of Braille displays or other devices to touch and read Braille music scores, which does not achieve the effect of visual, auditory and tactile synchronous presentation. This means that when the visually impaired learn Braille music scores, they can only sense the notes and rhythm by touching the Braille display, but cannot hear the music or feel the rhythm at the same time, which limits their learning effect and experience.

[0004] The patent document with disclosure number CN105931621A discloses a method and system for translating MIDI into Braille music score. This technical solution can only realize translation from MIDI to Braille music score. However, it is difficult to clearly and accurately record music score information in MIDI that can be read and understood by humans, the reproducibility of the music score is low, and notes are easily lost, making it difficult to meet the needs of music notation.

[0005] The patent document with disclosure number CN110390859A discloses a method and system for converting MusicXML sheet music into Braille, but this technical solution only reads and analyzes MusicXML data, and does not convert it into musical staff notation or reproduce the sounds in the musical score. It is unable to display the musical staff notation in comparison with the Braille sheet music, facilitating the publication and use of Braille sheet music.

[0006] Converting sheet music into Braille is an important step in digitizing Braille. All of the above methods are realized by software and cloud computing, and are not integrated into embedded environments, so they cannot be applied conveniently and efficiently in public places. Summary of the Invention [Problem to be solved by the invention]

[0007] In view of the above, the objective of the present invention is to provide an embedded music / Braille text conversion system that can present visually, auditorily and tactilely synchronously, which can automatically convert musical notation into Chinese Braille music notation in an embedded environment, and has the function of presenting visually, auditorily and tactilely synchronously, thereby meeting the needs of visually impaired people in learning music notation, and realizing multi-modal digital presentation of Braille music notation. [Means for solving the problem]

[0008] In order to achieve the above object of the invention, the embedded music / Braille text conversion system according to the present invention is capable of presenting visually, aurally, and tactilely synchronized, an acquisition module for acquiring scores in MusicXML format; a conversion module for converting a score in MusicXML format into a Braille score consisting of a sequence of Braille dots; It also has an output module for simultaneously outputting Braille music scores in three ways: visual, auditory, and tactile.

[0009] Preferably, the step of converting the score in MusicXML format into a Braille score consisting of a sequence of Braille dots comprises: Step 1 of reading and parsing a MusicXML file representing a musical score to obtain a tree model corresponding to the MusicXML file content; Step 2: obtaining a root node of the MusicXML file structure from the tree model, and using the root node as a node in the initial traversal tree model, searching for nodes to be converted to Braille and obtaining node label contents; and step 3 of converting the node label content into a braille dot sequence based on a Chinese braille music symbol correspondence library to obtain the braille music score.

[0010] Preferably, the step of searching for a node to be converted to Braille and obtaining node label contents using a root node as a node of an initial traverse tree model includes: The method includes: determining whether the current node is a movement node; if the condition is satisfied, further determining whether the child node of the movement is empty; if it is empty, traversing the next sibling node of the movement; if the child node of the movement is not empty and is a measure node, sequentially analyzing the attribute node and dynamic mark node of the measure node to obtain the attribute node label content and the dynamic mark node label content; if the child node of the measure node is a note node, analyzing the descendant node of the note node to obtain the descendant node label content; and traversing the next sibling node of the measure node after completing the traversal of the descendant nodes of the measure node.

[0011] Preferably, when converting the metronome information in the dynamic mark node label content into a Braille dot sequence, the method includes obtaining a metronome value of the metronome node, converting the metronome value into an integer-type numeric value, converting the ones digit, tens digit, and hundreds digit of the integer-type numeric value into independent numbers, and converting each independent number into a corresponding Braille dot sequence.

[0012] Preferably, the process of converting musical notes into a sequence of Braille dots comprises: For an input note node, determine whether the child node of the note node has a changed note node, and if so, record the related information of the changed note; if not, continue to determine whether the child node of the note node has a pitch node; If there is a pitch node, obtain and temporarily store the attribute value of the scale node and the attribute value of the octave node of the note, then obtain and temporarily store the attribute value of the note value node, perform tone group symbol processing, then determine whether to add a changed note type or not to add a braille dot sequence corresponding to the changed note according to the recording status of the changed note, then determine a braille dot sequence corresponding to the note based on the temporarily stored scale and note value, further determine whether to add a braille dot sequence corresponding to the dot, and finally return the braille dot sequence result; If there is no pitch node, it is determined whether there is a rest node, and if there is a rest node, jumping to a step of recording the attribute value of the note value node if there is a pitch node and continuing to execute the subsequent steps, and if there is no rest, performing an error reporting process.

[0013] Preferably, said step of performing tone group symbol processing comprises: Obtaining the scale and octave of the previous note and the scale and octave of the current note, and then judging whether the two adjacent notes belong to the same tone group based on whether the octaves of the two adjacent notes are equal; If two adjacent notes belong to the same pitch group, determine whether the absolute value of the pitch of the two notes is greater than 5, and if it is greater than 5, add a pitch group symbol in front of the current note; otherwise, do not add a pitch group symbol; When two adjacent notes belong to different pitch groups, if the difference between the two notes is equal to or greater than two pitch groups, a pitch group symbol is added in front of the current note, if it is determined that the two notes belong to two adjacent pitch groups and the note interval is greater than 3, a pitch group symbol is added in front of the current note, otherwise, no pitch group symbol is added, and finally, a braille dot sequence of the pitch group symbol is output.

[0014] Preferably, the process of converting chords in a musical note into a sequence of Braille dots comprises: determining whether the next note has a chord node, if so, obtaining the scale node, octave node and note value node of the current note, and recording the corresponding attribute values ​​in temporary variables, then putting the temporary variables in which the scale attribute value and octave attribute value are recorded into stacks s1 and s2 respectively, and setting the chord marker bit to 1, and repeating in this manner; If there is no chord node, it is necessary to determine whether the chord marker bit is 1; if the chord marker bit is not 1, convert to a braille dot sequence according to a normal note; if the chord marker bit is 1, mark the current note as a root note, combine with the attribute value of the note value to convert the root note to a braille dot sequence, extract notes in the stack from the stack in order, while calculating the interval between the notes in the stack and the root note, convert the notes extracted from the stack to a braille dot sequence according to the interval, and set the chord marker bit to 0; repeatedly extract from the stack until the stack is empty, and output the braille dot sequence.

[0015] Preferably, the process of converting identical note connecting lines in a note into a sequence of Braille dots comprises: Determine whether the child node of the notation node has a same note connection line node, and if there is a same note connection line node, traverse the next node. If the next node is not empty and is a note node, if it is determined that a same note connection line node and a chord node coexist in the note node, set both the chord marker bit and the double connection line marker bit to 1, and add a double connection line marker; if it is determined that a same note connection line node and a chord node do not coexist in the note node, it is necessary to add a same note connection line marker; If the next node is empty, set the chord marker bit to 0, add a same note connection line marker, and finally output the braille dot sequence result to the same note connection line marker and the double connection line marker.

[0016] Preferably, the process of converting arpeggios and fermatas in a musical note into a sequence of Braille dots comprises: When a chord and an arpeggio coexist, it is necessary to add an arpeggio marker before the root note. Specifically, if there is no chord node in the next note, add an arpeggio marker to the arpeggio of the current node, and output the braille dot sequence of the arpeggio marker. When a chord and a fermata coexist, a fermata marker needs to be added before the root note; specifically, if there is no chord node in the next note, a fermata marker is added to the fermata of the current node, and a braille dot sequence of the fermata marker is output.

[0017] Preferably, the output module performs score presentation and Braille dot presentation when outputting the Braille music score in a visual manner; The process of presenting the musical score is to match the braille musical score with the braille musical symbol library to obtain the musical sequence of each bar, and when the current bar is being read, to input the musical sequence and execute the program to output the musical score image of the bar and display it synchronously on the screen; The braille dot presentation process is to supplement the length of the braille dot sequence in the braille music score, cut and store a certain length of the supplemented braille dot sequence into a string, and generate a graphic braille dot for each string and display it on the screen in synchronization; When the output module outputs the Braille music score in an auditory manner, it matches the Braille music score with a Braille music symbol library to obtain the musical sequence of each bar. When the current bar is being read, it takes the musical sequence as input and executes a program to output the audio of the bar, which is then saved in MIDI audio format, played back, and displayed on the screen in sync.

[0018] Preferably, when the output module outputs Braille music score in a tactile manner, communication between the upper computer and the lower computer of the embedded system is via a serial port interface, in the serial port communication protocol of the upper computer, a data packet consists of 43 bits of hexadecimal data, which respectively include 1 data head high bit, 1 data head low bit, 1 CRC check bit and 40 data bits, and in the serial port communication protocol of the lower computer, a data packet consists of 4 bits of hexadecimal data, which respectively include 1 data head high bit, 1 data head low bit, 1 CRC check bit and 1 data bit.

[0019] Preferably, the conversion module is further for converting the Braille dot sequence into a Braille ASCII code, and providing a Braille dot presentation based on the Braille ASCII code. Effect of the Invention

[0020] Compared with the prior art, the beneficial effects of the present invention include at least the following:

[0021] The music / Braille character conversion system of the present invention has a fast calculation speed and high accuracy, and smoothly provides a new and viable means for Braille music score information. It also realizes the built-in function of synchronized visual, auditory and tactile presentation of Braille music score, with clear and accurate visual display, complete auditory reproduction, clear tactile sensation and better presentation effect, which can further improve the efficiency of visually impaired people learning music score and provide convenience for visually impaired people to obtain and touch-read large amounts of music materials. [Brief description of the drawings]

[0022] In order to more clearly describe the technical solutions of the embodiments of the present invention or the prior art, the drawings necessary for describing the embodiments or the prior art are briefly described below. Obviously, the drawings described below are only examples of the embodiments of the present invention, and those skilled in the art can further obtain other drawings based on these drawings without any creative efforts.

[0023] [Figure 1] FIG. 1 is a structural schematic diagram of a music / Braille character conversion system according to an embodiment. [Diagram 2] FIG. 2 is a block diagram showing a conversion process of the music / Braille character conversion system according to the embodiment. [Diagram 3] FIG. 3 is a structural diagram of a subnode according to the embodiment. [Figure 4] FIG. 4 is a flow chart illustrating traversal of a MusicXML file according to an embodiment. [Diagram 5] FIG. 5 is a flowchart showing a conversion process of metronome information values ​​according to the embodiment. [Figure 6] FIG. 6 is a schematic diagram showing the order of extraction of some child nodes of a bar according to the embodiment. [Figure 7] FIG. 7 is a flow chart showing conversion from a note node to a Braille dot sequence according to an embodiment. [Figure 8] FIG. 8 is a flow chart showing conversion from a tone group symbol to a Braille dot matrix according to an embodiment. [Figure 9] FIG. 9 is a flow chart showing conversion from chords to a Braille dot matrix according to an embodiment. [Figure 10] FIG. 10 is a flow chart showing conversion from identical note connecting lines to a Braille dot matrix according to an embodiment. [Figure 11] FIG. 11 is a flow chart illustrating a conversion from arpeggios and fermatas to a Braille dot matrix according to an embodiment. [Figure 12] FIG. 12 is a flow chart illustrating a visual presentation according to an embodiment. [Figure 13]FIG. 13 is a flow chart for creating a Braille dot matrix according to the embodiment. [Figure 14] FIG. 14 is a flow chart illustrating an auditory presentation according to an embodiment. [Figure 15] FIG. 15 is a flowchart showing tactile presentation according to the embodiment. [Figure 16] FIG. 16 is a diagram showing a display result of the first bar in <<Canon in D>> according to the embodiment. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0024] In order to make the objectives, technical solutions and advantages of the present invention clearer, the present invention will be further described in detail below by examples with reference to the drawings. It should be understood that the specific embodiments described herein are merely for interpreting the present invention, and do not limit the protection scope of the present invention.

[0025] Fig. 1 is a structural schematic diagram of an embedded music / Braille text conversion system capable of visually, audibly and tactilely synchronously presenting according to an embodiment. As shown in Fig. 1, the music / Braille text conversion system includes an acquisition module, a conversion module and an output module, where the acquisition module acquires a score in MusicXML format, the conversion module converts the score in MusicXML format into a Braille score, and finally the output module outputs the Braille score in three ways, visually, audibly and tactilely, simultaneously, and the conversion process shown in Fig. 2 is realized by these three modules.

[0026] In the embodiment, the acquisition module is for acquiring sheet music in MusicXML format, specifically, sheet music in MusicXML format can be directly downloaded from a website, or other music formats can be converted into MusicXML format by software such as Artinfuser Exercise.

[0027] In an embodiment, the conversion module is for converting the MusicXML format into a Braille score consisting of a sequence of Braille dots, and further converting the converted Braille dots into a Braille ASCII code. The process of converting the MusicXML format into a Braille score consisting of a sequence of Braille dots includes the following steps 1 to 5.

[0028] Step 1: Read and parse the MusicXML file to obtain a tree model corresponding to the MusicXML file structure. Specifically, use the xmlReadFile function to read the file pointer of the MusicXML file, and parse it through a DOM parser to establish a logical tree model.

[0029] Step 2: Obtain the root node of the MusicXML file structure from the tree model, specifically, obtain the root node curNode of the MusicXML file structure using the xmlDocGetRootElement function.

[0030] Step 3: Using the root node as the node of the initial traverse tree model, search for the node to be converted to Braille and obtain the node label content. Specifically, use the xmlNodeGetContent function to obtain the node label content for the node to be traversed. curNode->xmlChildrenNode is the set of child nodes of the root node.

[0031] Step 4: Convert the node label content into a braille dot sequence based on the Chinese braille music symbol correspondence library. Specifically, convert the label content in the MusicXML file into the corresponding braille dot sequence based on the correspondence between the node elements and the braille dot sequence in the MusicXML file, and then perform processing for the traversed nodes according to the simplified rules as necessary. The conversion results of some of the nodes need to be placed at the very front of the entire score, for example, at the start of a score, there is metronome information for the entire score, and this metronome information needs to be placed before the start of the score to present the metronome to the visually impaired. In addition, the chord part needs to be processed separately, and the traversal order of the MusicXML file for the high-pitched music score and the order of the braille dot sequence to be displayed are just the opposite, that is, the root note is at the end position, and is processed in stacks, while the traversal order of the MusicXML file for the medium / high-pitched music score and the order of the braille dot sequence to be displayed are the same, so no separate processing is required, and they can be traversed sequentially.

[0032] Step 5: Free the file pointer using the xmlFreeDoc function to clear the memory dynamically claimed by all nodes in the MusicXML file.

[0033] In the MusicXML file structure, the root node of the outermost layer is a score element (score-partwise), which contains information such as copyright information, structure distribution, etc., and mainly contains at least one movement (part). Each movement mainly contains at least one measure (measure). A measure contains many labels such as score attribute information (attributes), dynamic marks (direction), left / right hand exchange marks (backup), barlines (barlines), notes (notes), etc., as shown in Figure 3. A piece of music is mainly composed of measures, and a measure mainly consists of at least one note or rest (rest). Therefore, the entire piece of music is basically composed of movements, measures, and note labels, which jointly form a one-level nested structure of a MusicXML file. Specifically, the outermost label is the movement, the middle label is the measure, and each measure contains a note label. Therefore, the traversal process is to first traverse the outer label, and then traverse the nested inner label. As shown in Figure 4, the specific traversal process includes the following: first, determine whether the current node is a movement node, if the condition is met, further determine whether the child node of the movement is empty, if it is empty, traverse the next sibling node of the movement; then, if the child node of the movement is not empty and the child node is a measure node, analyze the attribute node and dynamic mark node of the measure node in order to obtain the attribute node label content and the dynamic mark node label content, and if the child node of the measure node is a note node, analyze the descendant node of the note node to obtain the descendant node label content, and traverse the next sibling node of the measure node after completing the traversal of the descendant node of the measure node.

[0034] The attribute node is for determining the key of the musical staff, including the musical score labels such as the standard quarter note value (divisions), key signature (key), time signature (time), number of staves, and clef for each note. The standard quarter note value is for measuring the value of a note. The standard quarter note value element itself uses the quarter note as the basic unit, and each note has one standard quarter note value element, and the value of the standard quarter note value element is used to calculate the duration of every note. For example, if divisions=2 and duration=1 for a note, the note is an eighth note. If a note is a rest, the program mainly relies on the standard quarter note value and duration values ​​to calculate how many minutes the rest is. The key signature is placed adjacent to the standard quarter note value element, and indicates the pitch of the tonic note of the scale with seven note names and ascending and descending symbols. When indicating a certain key signature on a musical staff, the time signature may be preceded by some sharps and flats. The fifths are child nodes of the key signature, and are used to indicate ascending or descending key, with positive values ​​indicating ascending key, negative values ​​indicating descending key, and 0 indicating neither ascending nor descending. Time signatures may generally be indicated as fractions, and in MusicXML files, time signatures have two sub-labels, the number of beats and the beat-type, with the beats indicated in the numerator and the beat-type indicated in the denominator, which intuitively describes how many beats (numerator) there are in a measure in the music score, and how many half notes (denominator) each beat is. In the Chinese Braille Music Symbol Conversion Library, time signatures are generally indicated as Braille fractions, with the numerator written normally according to the Braille Arabic numerals, and the denominator written according to the drop point of the Braille Arabic numerals. If the attribute does not have a staff number element, it is defaulted to a single staff. Clefs determine the pitch placement of notes, and in MusicXML files, clefs are represented by two labels: a sign and a line. For example, the treble clef is commonly represented by G and the line it is on is 2, the bass clef is commonly represented by F and the line it is on is 4, and the c clef is commonly represented by C and the line it is on is 3.

[0035] Dynamic mark nodes can be multiple and do not necessarily have to be attached to notes. Its main child node is dynamic mark type (direction-type), and dynamic mark type node contains multiple child nodes such as coda, segno, dynamics, wedge, metronome, pedal, etc. Regarding coda, if there is a coda in the music, there are generally two symbols in one code paragraph, the previous repeat ellipsis symbol indicates that the music is to code on the music, and the latter symbol indicates that the paragraph within it will be repeated and then skipped. Crescendo-decrescendo is a kind of dynamic information, and contains 13 symbols, and these symbols are generally attached before the note, but the wedge symbol in the wedge label is written at the beginning and end of the note. There are two ways to display metronome information. The first is to place Italian words in the word label as numbers, and the second is "quarter note = number", and this format is generally used as a metronome label identifier. The conversion process of metronome information numbers in the program is shown in Figure 5, which includes obtaining the metronome value of the metronome node, converting the metronome value to an integer number, converting the ones, tens, and hundreds digits of the integer number into individual numbers, and converting each individual number into a corresponding braille dot sequence. Both "pressing" and "lifting" the pedal are displayed on the pedal label, and are distinguished by different type attribute values.

[0036] There are many barline representation styles in the barline node, and these different barline styles are indicated by different bar-style values, including light-heavy, dashed, and light-light. If a score has multiple endings, they may be indicated by only a single ending (e.g., "1.") or multiple endings separated by commas (e.g., "1.,2."). The type attribute value in the ending indicates the start and end of an ending in the MusicXML file. If any repeat signs appear in the score, the first note in each case is given a key group symbol. If a repeat sign and an ending are used in combination, they are distinguished by a reduced number in the different parts, and if the reduced number and the following symbol are obfuscated, they are separated by "(3)". Repeat start / end signs are indicated in the MusicXML file with the barline style heavy-light. In the MusicXML file, the ascending and descending sign information, the time signature information, and the dynamic marks all appear in order. In addition to placing the metronome information at the front, the corresponding Braille dot sequence order also matches the front-to-back order in the MusicXML file.

[0037] What should be emphasized here is that, first, other than the note elements, other child nodes in the measure node and the braille symbols have almost one-to-one correspondence, and second, other than the metronome information, the order of the braille dot sequence and the order of the nodes appearing in the braille music score match. Therefore, the label contents of the measure may be extracted using the simple means shown in Figure 6.

[0038] For the note node, the note is processed according to the Chinese Braille Music Symbol Conversion Library until the traversal of the child nodes of the measure is completed. In the note, some child nodes are commonly used to indicate the characteristics of the current note. The note node includes basic labels such as rest, pitch, chord, accidental, type, dot, duration, voice, stem direction, staff and beam, and the note needs to be used together with the attribute values ​​of the scale and value nodes in the pitch to obtain the corresponding Braille dot sequence. The scale is to determine which note it is, and the value is to determine the note's value (how many notes), and they cannot be matched to the Braille dot sequence unless they are all determined. In addition to the basic labels, the note also includes the modification labels of the note, which are embodied in the notation label. The extraction and conversion of note nodes is mainly divided into three parts: (1) extraction and conversion of key labels such as pitches that are closely related to notes, (2) extraction and realization of note chord parts, and (3) extraction and conversion of other nodes such as arpeggiates, fermatas, tied / double note connections, slurs, etc.

[0039] The pitch label indicates the position of the current note on the staff, and its sub-labels include step, octave, and alter. The scale includes seven notes, C, D, E, F, G, A, and B, while the octave corresponds to the nine tone groups on the piano and is represented by numbers 0 to 8, of which 4 indicates the octave starting from middle C. The alter is intended to indicate a semitone change, with negative numbers indicating an alter and positive numbers indicating an alter. The pitch of a note can be clearly sensed from the staff, but the visually impaired cannot sense such pitch. To allow the visually impaired to distinguish between high and low notes, it is necessary to attach an alter symbol before the note. However, it is not necessary to attach an alter symbol to each note. The method of using the tone group symbol is stipulated in the Chinese Braille Music Symbols: first, the tone group symbol must be placed before the first note of the score or musical section; second, there must be no other symbols between the tone group symbol and the note; third, for two adjacent notes in the same tone group, if the distance between them is within 5 degrees, there is no need to add a tone group symbol, but if the distance is 6 degrees or more, a tone group symbol must be added; fourth, for two adjacent notes in different tone groups, if the distance between them is within 3 degrees, there is no need to add a tone group symbol, but if the distance is 4 degrees or more, a tone group symbol must be added. The addition of a tone group symbol is related to the pitch of the note, and must be determined based on the difference between the scale of the previous note and the scale of the current note, and must also consider whether the octave in which the previous note is located is the same as the octave in which the current note is located.

[0040] As shown in FIG. 7, the process of converting the key point elements of a note node into a braille dot sequence starts from the acquired note node, firstly, determine whether the child node of the note has a change note node, and if there is a change note, record the related information of the change note; if there is no change note, continue to determine whether there is a pitch node, here, it is divided into two situations, that is, (1) if there is a pitch node, it is proved that there is one note among C, D, E, F, G, A, B, obtain the attribute values ​​of the scale node and octave node of the note and temporarily store them, then obtain the attribute value of the note value node and temporarily record them, and carry out the tone group symbol processing according to the flowchart of FIG. 8, and then perform the change according to the recording situation of the change note. (2) performing a process of converting the pitch node to Braille, i.e., performing a step of recording the attribute value of the pitch node if there is a rest, and continuing to execute the subsequent steps; and (3) performing a step of recording the attribute value of the pitch node if there is a rest, and continuing to execute the subsequent steps; and (4) performing a step of converting the pitch node to Braille, i.e., performing a step of recording the attribute value of the pitch node if there is a rest, and continuing to execute the subsequent steps; and (5) performing a step of converting the pitch node to Braille, i.e., performing a step of recording the attribute value of the pitch node if there is a rest, and continuing to execute the subsequent steps; and (6) performing a step of converting the pitch node to Braille, i.e., performing a step of converting the pitch node to Braille, i.e., performing a step of converting the pitch node to Braille,

[0041] As shown in FIG. 8, the process of processing the tone group symbol includes the following: first, the scale and octave of the previous note are respectively written as step_before and octave_before, and the scale and octave of the current note are respectively written as step_after and octave_after, then, judge whether octave_before and octave_after are equal, that is, judge whether they belong to the same tone group, if they are equal, the two notes are described as belonging to the same tone group, otherwise they belong to different tone groups. Here, two situations are distinguished, that is, (1) the two notes belong to the same tone group. According to the rule, tone group symbols are not added within the fifth of the same tone group, therefore, here, it is necessary to judge whether the absolute value of the two note intervals is greater than 5, if it is greater than 5, the corresponding tone group symbol is added before the current note, otherwise, the tone group symbol is not added, and (2) the two notes belong to different tone groups. Here, the situation of belonging to different pitch groups can be further divided into two situations: a) if the octave difference between the two notes is greater than 1, that is, they do not belong to adjacent pitch groups, then the degree of the notes must be greater than 4, and the corresponding pitch group symbol must be placed in front of the current note; b) if the two notes belong to two adjacent pitch groups, and the pitch of the notes may be less than a third, then it is necessary to determine whether the pitch of the notes is greater than 3, and if so, the corresponding pitch group symbol must be placed in front of the current note; otherwise, the pitch group symbol is not placed, and finally the Braille dot sequence of the pitch group symbol is output.

[0042] Regarding rests, if a note has a rest node, it must not have a pitch node, because pitch and rests cannot coexist and rests directly substitute for pitch in MusicXML files. Like notes, rests come in different types, such as whole rests, half rests, quarter rests, and eighth rests. Also, when the entire measure is at rest, a whole rest must be used. When a note has a rest, the note value of the rest must be determined by the note value attribute value in the same way.

[0043] The value of a note or rest is determined by its type, as well as its divisions and duration attributes. The divisions indicate the units of standard quarter note value, while the duration indicates the value of the current note or rest. If the value of duration is twice the divisions, then the current note or rest is a half note, if the value of duration is equal to the divisions, then the current note is a quarter note, and so on.

[0044] A dot is generally placed after a note. The actual value of a dotted note is 1 / 2 the current note, and the actual value of a double dotted note is 3 / 4 the current note, so two dotted notes are called double dots. Although we mentioned altered notes (sharp, flat, double sharp, double flat) in the above explanation of attributes, in fact, rising and falling signs and natural signs also appear on notes, and they are indicated by different values ​​of the altered note label. In braille music, when a sharp, flat, or natural sign is written before a note, it indicates a temporary rise or fall of a semitone. Each sharp and flat sign is effective for the same note in a measure, and if a note in the same measure does not rise or fall, a natural sign must be placed before the note. Therefore, when processing the child nodes of a note, we first determine whether there is an altered note.

[0045] Processing chords is a difficult part of the system. A chord refers to a combination of pitches stacked according to a third or interval relationship other than a third. These notes may be spoken one after the other or simultaneously. A chord has at least three notes and at most seven notes. When learning chords, visually healthy people can understand the difference between a triad and a seventh chord, while visually impaired people rely on the degree of the interval to calculate when learning chords. An interval refers to the pitch difference between two notes, and this difference is indicated by the "degree". That is, the degree is used to determine the pitch relationship, and the degree is the number of notes contained in the interval, and is expressed on the staff as the number of tones and intervals contained in the two notes. For example, the degree of two chords CE (from do to mi) is a third, and it may be in the form of one tone contained in the two tones, or one tone contained in the two intervals, but both represent a distance of three degrees. The degree of an interval can only describe how many degrees there is, and does not represent that they are completely the same. For example, DE and FG are both second degrees, but they are two completely different intervals, and the only thing they have in common is the number of degrees. Therefore, the name of the interval cannot be determined simply by the number of degrees, but must also be combined with the number of notes.

[0046] Chord notes are the notes that make up a chord. In a chord, the highest note is called the crown note, the lowest note is called the root note, and the remaining notes are named according to their interval relationship to the root note. In a triad, the notes are arranged according to a third interval, so the lowest note is called the root note (written as 1), the middle note is called a triad (written as 3) because it is in a third interval relationship with the root note, and the highest note is called a quintuplet (written as 5) because it is in a fifth interval relationship with the root note.

[0047] In braille music, when a chord appears, one note is selected as the root note of the chord, and the interval between the other notes and the root note is indicated based on the degree of the interval. In the staff, the interval is generally calculated from bottom to top, that is, the lowest note is the root note. However, when notating by voice, the low music score calculates the degree from bottom to top, and the high music score calculates the degree from top to bottom. That is, in the left hand / right hand separate music score, the high music score part calculates the degree from top to bottom with the highest note as the root note, and the low music score part calculates the degree from bottom to top with the lowest note as the root note. Generally, the interval symbol is written after the note, and there is no need to attach a tone group symbol to the interval symbol of 9ths and intervals within 9ths, but if it is 9ths or more, an interval symbol must be attached to indicate a compound interval of 9ths or more.

[0048] As shown in Figure 9, the process of converting the note chord part into the braille dot matrix includes inputting the label content information of the current note node. The determination of the note needs to be determined by combining the scale and the note value, while the determination of the tone group symbol needs the octave value. Therefore, if there is a chord, obtain the scale label, octave label and note value label of the current note, and record the corresponding values ​​in the temporary variables step_before, octave_before and t. Then, put the temporary variables step_before and octave_before into the stacks s1 and s2 respectively, and set the chord marker bit chord to 1, and repeat in this manner until the next note has no chord. If there is no chord, it is necessary to determine whether the chord marker bit is 1. If the chord marker bit is not set to 1, it is explained that there is no chord in the entire process, and then process according to the normal note processing steps shown in Figure 7. It is explained that only when the chord marker bit is 1, a chord exists and is extracted from the stack, and the top note in the stack is the note next to the root note, so the current note is marked as the root note, the value of step_before is assigned to the temporary variable root_step, the value of octave_before is assigned to the temporary variable root_octave, and the root note is converted to a braille dot sequence by referring to the value of t. Finally, the elements in the stack are extracted from the stack in order, and at the same time, the interval between the note in the stack and the root note is calculated, and the note extracted from the stack is converted to a braille dot sequence according to the interval, and the chord marker bit is set to 0, and the stack is repeatedly extracted until it is empty, and the braille dot sequence is output, and the whole process is completed. The calculation methods for interval include, for notes in the same pitch group, interval is equal to the absolute value of the difference between step_before and root_step plus 1, and for notes in a different pitch group, interval is equal to the absolute value of the difference between the highest note and root_step plus 1, and then the absolute value of the difference between step_before and the lowest note plus 1.

[0049] The chords in a MusicXML file are all described in order from bottom to top, with the lowest note being the chord.<chord / > There are no markers, so when the chords are written by voice on the staff, the chords in the high-pitched staves are exactly the opposite order, and the chords in the low-pitched staves are the same order. Therefore, the chords in the high-pitched staves need to be processed separately in the braille music. In the present invention, the first-in, last-out feature of the stack is used to stack the lowest note in order from bottom to top, and the last chord is stacked and then extracted in order, and the first note extracted from the stack is the root note.

[0050] What needs to be explained here is the first note that starts the chord. <chord / > Since no marker is attached, the present invention puts the next note into the stack by determining whether the next note has a chord. If the next note has a chord, the current note information is put into the stack. That is, if it is the last note of the chord, the last note is not put into the stack, but the last note is exactly the root note. Therefore, the present invention directly marks the last note as the root note, and after processing the root note according to a normal note, extracts the note information in the stack from the stack to calculate the degree of the root note of the chord.

[0051] Tied notes are also called note extension lines and are used to connect notes of the same pitch. When two or more notes of the same pitch (or chords) are to be connected, double connections must be used. Therefore, before determining whether a tied note node is a chord node or a tied note node, it is necessary to determine whether the next note node contains a tied note node. If a tied note node is included, the marker bit is written as tied_notation, and if there is a chord, the marker bit is written as fermata_chord.

[0052] As shown in FIG. 10, the process of converting the same note connection line in a note into a braille dot sequence includes traversing the input notations child node, and if there is a same note connection line / double connection line node, traversing the next node, and if the next node is not empty but a note node, judging whether the same note connection line / double connection line node and the chord node coexist in the note node, and if they coexist, the chord marker bit fermata_chord and the double connection line marker bit tied_notation are both set to 1, and a double connection line marker is added, and if they do not coexist, a same note connection line marker needs to be added. If the next node is empty, it is described that there may be a chord in the next node, but the chord has already ended or there is no chord, and both of these two situations are described as the chord marker bit fermata_chord being set to 0, and a same note connection line marker is added. Finally, the braille dot sequence result is output to the same note connection line marker and the double connection line marker, and the process ends.

[0053] As shown in Figure 11, when a chord and an arpeggio coexist, or when a string and a fermata coexist, we can add a corresponding braille marker before the root, i.e., a marker corresponding to the arpeggio or the fermata. This means that we need to use the fermata_chord marker bit to determine whether to add the arpeggio and extended note braille markers. <chord / > If not present, braille symbols for arpeggios and fermatas are added successfully, otherwise they are not added.

[0054] A slur, also called an arch, is a line that connects notes so that multiple notes in a musical score can be played consecutively. In MusicXML files, slurs are identified by the type attribute, where "start" indicates the beginning and "stop" indicates the end of the slur. In Braille music, a single arc is used to connect two notes. JPEG2025505976000002.jpg46166(14) A marker needs to be placed, and two are needed to connect the three notes. JPEG2025505976000003.jpg46167(14) markers must be added. Therefore, adding a slur marker must be combined with the type attribute, and slur must appear and be added after each note from type="start". JPEG2025505976000004.jpg46167(14) A marker must be added after the note only if the type attribute value is "stop". JPEG2025505976000005.jpg45158(14) No markers are added. In addition, in the Chinese Braille Music Symbols, when connecting 2 to 4 notes, a slur is used, and when connecting 5 or more notes, a large slur is used. JPEG2025505976000006.jpg42158(14,14)...(14). If a slur or large slur spans two measures, leave one square between the measures, i.e. JPEG2025505976000007.jpg45158 (Empty Square) needs to be added.

[0055] The articulations node contains child nodes accent, staccatissimo, staccato, tenuto, doit, falloff, plop, etc. These nodes have a one-to-one correspondence with the braille dots, and can be directly matched and converted in the program. It should be noted that their placement is: staccato, staccatissimo, substaccato, tenuto and strong accent are all written before the note, the three symbols trill, upper trill and lower trill are all written before the note, the symbols plop and slip are also written before the note, but the symbols doit and falloff are written after the note.

[0056] The articulation node also has a commonly used child node technical, whose child node fingering indicates fingering symbols, denoted as 1, 2, 3, 4, and 5. The first finger is the thumb and is marked as 1, the second finger is the index finger and is marked as 2, the third finger is the middle finger and is marked as 3, the fourth finger is the ring finger and is marked as 4, and the fifth finger is the pinky finger and is marked as 5. The braille markers for these fingering symbols are all written after the notes, and there is also a one-to-one correspondence between the nodes and the braille dots.

[0057] When notating music for separate hands, the left hand symbol is inserted before the beginning of the left hand music. JPEG2025505976000008.jpg34124(456,345) must be added, and the right hand symbol must be added before the beginning of the right hand music. JPEG2025505976000009.jpg34124(46,345) should be added. Note that the left and right hand symbols should be added before the note, and the first note with the left and right hand symbols should be added before the key group symbol. JPEG2025505976000010.jpg38124(1)dots, JPEG2025505976000011.jpg39125(2)dots, JPEG2025505976000012.jpg40124(3)If dots are included, JPEG2025505976000013.jpg35124(3) symbols should be added. The notation method used in the present invention is to notate one bar at a time, that is, first record one bar of bass music, and then record one bar of treble music. Therefore, before writing the note information of a bar, it is necessary to first write the left hand symbol, then write the right hand symbol, and finally write the note information in the right hand music. This is done one bar at a time until one musical line or the entire piece is completed.

[0058] In addition, some descendant nodes in the measure node need to be traversed deeper. For example, the label fifths is in the label key, the labels beats and beat-type are in the label time, and the labels sign and line are in the label clef, but key, time and clef are in the attributes labels in measure. In this case, it is necessary to first access the related labels in measure to correctly obtain the music score information and correctly represent it according to Chinese Braille Music Symbols, and then go back to the previous step and start traversing the next measure node until the traversal of the measure in the part node is completed.

[0059] The process continues until the part nodes have been traversed and there are no more part nodes, thereby completing the traversal of the entire score.

[0060] In an embodiment, each musical score S is composed of at least one movement P, as shown in equation (1-1), where F is a composition function. Each movement P is composed of at least one measure M, as shown in equation (1-2), where P i denotes the ith part of a piece of music, and M i indicates the i-th measure, and the musical score M of each measure is shown in formula (1-3), and M attributes Let denote the set of musical score attributes, and M direction denotes the set of score dynamics information, and M note denotes a set of musical note information, and M barline indicates the end of the score, and M backup indicates the first movement of musical time, i.e. the left / right hand exchange mark, and M others Let denote the other sets of nodes in the score, which are shown in equations (1-4).

[0061]

number

[0062] The conversion rule that is followed to convert from MusicXML format to a Braille dot sequence is denoted as R. The conversion relationship from the music score S to the Braille dot sequence Bri_Dot is shown in formula (1-5), and the conversion rule is shown in formula (1-6). write indicates the writing rule, and M conversion shows the rules for converting musical scores into Braille scores, and M simplify These are simplified rules for converting sheet music into Braille music.

number

[0063] Research into the MusicXML file structure and the converted Braille dot sequence has revealed that the relationship between MusicXML nodes and Braille dot sequences is not only one-to-one, but also one-to-many, many-to-one, and even many-to-many. The converted sequence structure can be expressed as formula (1-7). (< >) indicates the parent-child relationship between nodes, p is the set of parent nodes in the MusicXML file, and p i Let denote one parent node in set p, and s i ga p i indicates the child node of b i , b i ′ denotes the generated Braille dot sequence, (i=1, 2, 3, ...).

number

[0064] The conversion of the piano to braille dot sequences for the left and right hands is explained as formula (1-8),

number

number

number

number

[0065] In the embodiment, after obtaining a sequence of Braille dots of a musical score, the sequence of Braille dots is converted into Braille ASCII code by comparing it with a comparison table of Braille dots and Braille ASCII code.

[0066] In the embodiment, the output module is for realizing the simultaneous output of three kinds of senses, namely, vision, hearing and touch, in an embedded system. Vision is displayed on a 10-inch IPSHDMI screen as a unit of music score, and a corresponding electronic braille dot sequence is presented below the music score, making it easy for visually healthy people to assist visually impaired users in learning. Hearing is generated as an audio file as a unit of music score, and played in dual channels in the ALC5640 audio chip, adding the feeling of hearing when visually impaired users touch the braille dots to learn, making it easy to understand and feel the music. Tactile is presented on a 40-square braille dot matrix, and the raised braille dots are touched to read the music score to learn it. In order to facilitate learning braille music score in other devices, the present invention uses two output formats, namely, braille dot sequence and braille ASCII code. At the same time, a correspondence table between the braille dot sequence and the braille ASCII code can be built to convert the braille dot sequence into the braille ASCII code.

[0067] The visual presentation is mainly divided into two parts: music score presentation and braille dot presentation. A 10-inch IPSHDMI screen is used as the medium for music score display, with the music score displayed in picture format, and each picture displays one measure of music score information. The Linux system drives the IPSHDMI screen to display the music score content on the screen, which can achieve high resolution and good color reproduction effect, making the music score clearer and easier to read. Below the music score, the system displays the corresponding braille dot sequence, making it easier for visually impaired users to understand the music score content. The coordinated presentation of the braille dot sequence and music score information provides visually impaired users with a more complete understanding and experience of music scores.

[0068] Regarding the presentation of music scores, there are currently many notation software that can edit MusicXML files. For example, MuseScore is an open source notation software that can not only derive music scores in MusicXML format, but also save music scores in formats such as PNG and MP3. What is important is that MuseScore is compatible with Linux systems and is open source. In order to realize automation and accurately obtain the music score information of each bar, the system takes advantage of this advantage of MuseScore and is associated with MuseScore by introducing the music21 library, and is programmed in the Python language, allowing users to view the music score in the form of a graphic.

[0069] The formation of the music score image requires a sequence of time signatures, notes, etc., which are generated by the Chinese Braille music score conversion system based on the MusicXML file. As shown in Figure 12, the visual presentation process includes the following: First, parse the MusicXML file by DOM to generate a tree structure. When traversing the DOM tree structure, it matches the Chinese Braille music symbol library, and outputs the music sequence required for visually presenting the music score in units of bars, and the music sequence includes information such as time signatures, ascending and descending keys, and pitches, and the music sequence is saved in a temporary array. When the current bar is read, the Python program takes the music sequence as input, and outputs the music score of the bar by running the program, and saves it in a picture format, which is synchronously displayed on the IPSHDMI screen. Until the traversal of the music score file is completed, the system repeatedly traverses each bar in order.

[0070] Regarding the Braille dot presentation, the Braille dot sequence or the corresponding Braille ASCII code result obtained by the music / Braille character conversion system is mainly displayed in a Braille dot matrix, so that the visually impaired can accurately understand the meaning of the output result. However, there are two types of visual impairment: total visual impairment and low vision. In order to facilitate the use by the low vision group in the visually impaired group, the present system uses the remaining visual acuity to display the Braille dot sequence converted from the musical score in the form of Blair Braille on the visualization screen, and at the same time, it also contributes to the guidance of the visually healthy person in Braille. The creation of the Braille dot matrix is ​​realized by the QT mapping event. The system creates it by converting it into the corresponding protrusion position based on the acquired Braille dot sequence. Braille musical scores are displayed in units of bars in the embedded system, so that the Braille of one bar of the musical score may be less than 40 squares. However, because the system arranges 40 squares as one row, some Braille dot strings may not be fully utilized. To solve this problem, braille dot strings shorter than 40 squares must be supplemented with "0"s to make their length reach 240.

[0071] As shown in Figure 13, the process of creating a Braille dot matrix includes: creating a QPainter object to initialize the paintbrush and the paintbrush; converting the MusicXML formatted music score to obtain a Braille dot sequence; then supplementing the Braille dot sequence whose length is less than 240 with 0; cutting the string in the Braille dot sequence with length 6 and storing it in the string array list; if the length i of the string in the Braille dot sequence is less than 40, taking the corresponding string array and creating a Braille dot figure according to the string. The Braille dot creation function is drawBraille(QPoint d, QPainter *p, QString s), and its return value is empty. This function first introduces the coordinate QPoint value of the starting point, the object QPainter and the Braille dot sequence to be created this time; this function executes the Braille drawing operation 40 times, thereby realizing the display of the Braille dot on the visualization screen.

[0072] For auditory presentation, the embodiment selects the ALC5640 audio chip, which is a dual I2S interface audio decoder and may be used in an audio hub. It integrates a dynamic range controller and a 7-band parametric equalizer, and has a digital-to-analog converter with a signal-to-noise ratio of 100 dBA and an analog-to-digital converter with a signal-to-noise ratio of 94 dBA, and has the advantages of high performance and low power consumption. Inside the embedded system, the I2S pin of the ALC5640 audio chip is connected to the I2S pin of the main control chip RK3399, and the Class-D output can directly drive 1.5w single channel or 2.5w single channel audio. The audio may be output through an external 3.5mm earphone interface or through a dual channel audio output. The auditory output of music is different from the audio output of text, so it needs to be processed differently. The music21 library is introduced and used in the method associated with MuseScore, and the programming is done in Python language. The music sequence obtained by the MusicXML to Chinese Braille music score conversion system is input into the program, and the program generates a music score audio file in MIDI format.

[0073] As shown in Figure 14, the auditory presentation process includes the following: First, a MusicXML file needs to be input, then a tree structure is generated by DOM parsing, and during the process of traversing the DOM tree, matching is performed with the Chinese Braille music symbol correspondence library based on the node information. At the same time, the music sequence required for visually presenting the score is output in units of bars and saved in a temporary array. When playing up to the current bar, the python program takes this sequence as input, and the program runs to output the audio of one bar of the score, which is saved in MIDI audio format and played. The system will repeatedly traverse each bar in turn until the traversal of the MusicXML score file is completed.

[0074] For tactile presentation, two 40-square braille dot displays are used, and a total of 640 braille dots need to be driven. The braille output from the music braille conversion program has 6 dots per square, and the 4th and 8th dots are for identifying the braille input position. Therefore, the braille 01 dot sequence acquired by the conversion program needs to be secondarily processed to convert the 6-dot braille into 8-dot braille. Among the 6-dot braille and 8-dot braille, the 8-square braille has 4 dots and 8 dots more per square than the 6-square braille. In order to convert the 6-dot braille into the 8-dot braille, the 4th and 8th dots need to be supplemented with 0 when acquiring the braille dots of each square, and thus the 6-dot braille can be converted into the corresponding 8-dot braille.

[0075] The serial port is the main interface for communication between the upper and lower computers, and is also an important interface for the upper and lower computers to transmit protocol data. In order to ensure the reliability of data transmission and communication efficiency, the present invention defines a serial port communication protocol. The Braille dot sequence is stored as hexadecimal data through the UART serial port and sent to the lower computer, and an error detection mechanism must be set up to determine whether the received data is correct. The present invention selects to use a CRC8 mechanism for checking, and the CRC8 check can completely detect all odd random errors and sudden errors with a length of 8 or less.

[0076] In the serial port communication protocol of the upper computer, the data packet is composed of 43 bits of hexadecimal data, which respectively include 1 data head high bit, 1 data head low bit, 1 CRC check bit and 40 data bits. The start bit indicates the start of the protocol data frame, and the serial port communication protocol specifies that the data frame sequence is 0x7A, 0x6A in sequence. The CRC8 check is the result of checking the data bits, and the data bits indicate the specific content of the data. The lower computer only needs to check whether the data is correct after receiving it, and does not need to send excessive information, so the data packet of the lower computer is simpler, and only includes 1 data head high bit, 1 data head low bit, 1 CRC check bit and 1 data bit.

[0077] As shown in Figure 15, the tactile presentation process includes the following: First, an array protocolArray is defined for transmitting data through serial port communication, and an array dataArray is defined for calculating a CRC8 check value containing the braille dot sequence data. When serial port communication starts, the high bit / low bit information of the data head is stored in the protocolArray array, i.e., the first bit is given a value of 0x7A, the second bit is given a value of 0x6A, the length of the braille dot sequence is set as len, the variable i=0, and the braille dot sequence is traversed in sequence, and if i is less than len, only a braille string with a length of 6 bits is processed each time, and the third and seventh bits are added with 0 to convert it into 8-dot braille. Then, the 8-dot braille string is read in binary, converted into hex data, and stored in the dataArray array in hexadecimal format. Traverse backwards in order, and when the conversion of the braille dot sequence to hexadecimal is completed and the movement of the pointer is greater than the marker bit of the length of the current braille dot sequence, the system calculates a CRC8 check value by table lookup based on the data in dataArray, and stores the check value in the protocolArray array. Then, combine the value of the dataArray array with the value in the protocolArray array and send it to the lower computer. After receiving the data, the lower computer compares the high bit / low bit of the data head, and then calculates the CRC8 value according to the data bits. It compares the calculated value with the CRC8 check value sent from the upper computer, and if the check results are different, it returns the data bits sent from the lower computer as 0x00. If the check results are the same, it sets the data bits sent from the lower computer to 0x01 and drives to display the braille dot matrix.

[0078] In the embodiment, the integrated overall effect of visually, audibly and tactilely synchronized presentation is shown in Figure 16, taking the first measure of "Canon in D" as an example. The display results of each part in the integrated system of the musical score, electronic dot sequence, audio waveform and Braille dot sequence are shown in Figure 16.

[0079] The embedded music / Braille character conversion system capable of visually, audibly and tactilely synchronously presenting according to the above embodiment provides a new feasible means for Braille music score information without hindrance, and can be widely applied to various embedded scenarios and environments used by visually impaired people, conveniently and efficiently generating a large amount of Braille digital resources, effectively reducing development costs, shortening the development process, improving development efficiency, and having higher practicality. At the same time, the embedded function of visually, audibly and tactilely synchronously presenting the developed Braille music score can further improve the efficiency of visually impaired people learning music scores, showing a new perspective on the application of artificial intelligence in the field of visually impaired people. At the same time, it can also provide an example and reference for research on other fields of visually impaired people.

[0080] The specific embodiments described above have described in detail the technical solutions and beneficial effects of the present invention, but it should be understood that the above description is merely the best embodiment of the present invention, and is not intended to limit the present invention. Any modifications, supplements, equivalent replacements, etc. made within the principle of the present invention should be included within the protection scope of the present invention.

Claims

1. An embedded music / Braille text conversion system capable of synchronized visual, auditory and tactile presentation, comprising: an acquisition module for acquiring a score in MusicXML format; a conversion module for converting a score in MusicXML format into a Braille score consisting of a sequence of Braille dots; An embedded music / Braille text conversion system capable of presenting Braille music in a synchronized visual, auditory, and tactile manner, comprising: an output module for simultaneously outputting Braille music scores in three ways: visual, auditory, and tactile.

2. The step of converting the score in MusicXML format into a Braille score consisting of a sequence of Braille dots comprises: Step 1: reading and analyzing a MusicXML file representing a musical score to obtain a tree model corresponding to the MusicXML file content; Step 2: obtaining a root node of the MusicXML file structure from the tree model, and using the root node as a node of the initial traversal tree model, searching for a node to be converted to Braille and obtaining node label contents; and (3) converting the node label content into a braille dot sequence based on a Chinese braille music symbol correspondence library to obtain the braille music score.

3. The step of searching for a node to be converted to Braille and obtaining node label contents using a root node as a node of an initial traverse tree model includes:

3. The embedded music / Braille text conversion system capable of being presented visually, aurally, and tactilely synchronized as claimed in claim 2, further comprising: determining whether a current node is a movement node; if the condition is satisfied, further determining whether a child node of the movement is empty; if it is empty, traversing a next sibling node of the movement; if the child node of the movement is not empty and is a measure node, sequentially analyzing the attribute node and dynamic mark node of the measure node to obtain attribute node label content and dynamic mark node label content; if the child node of the measure node is a note node, analyzing descendant nodes of the note node to obtain descendant node label content; and traversing a next sibling node of the measure node after completing traversal of the descendant nodes of the measure node.

4. The embedded music / Braille text conversion system capable of being presented visually, aurally, and tactilely synchronized, as described in claim 3, characterized in that when converting the metronome information in the dynamic mark node label content into a Braille dot sequence, the system includes obtaining the metronome value of the metronome node, converting the metronome value into an integer-type numerical value, converting the ones, tens, and hundreds digits of the integer-type numerical value into independent numbers, and converting each independent number into a corresponding Braille dot sequence.

5. The process of converting musical notes into Braille dots is For an input note node, determine whether the child node of the note node has a changed note node, and if so, record the related information of the changed note; if not, continue to determine whether the child node of the note node has a pitch node; If there is a pitch node, obtain and temporarily store the attribute value of the scale node and the attribute value of the octave node of the note, then obtain and temporarily store the attribute value of the note value node, perform tone group symbol processing, then determine whether to add a changed note type or not to add a braille dot sequence corresponding to the changed note according to the recording status of the changed note, then determine a braille dot sequence corresponding to the note based on the temporarily stored scale and note value, further determine whether to add a braille dot sequence corresponding to the dot, and finally return the braille dot sequence result; The embedded music / Braille text conversion system as described in claim 3, characterized in that if there is no pitch node, it is determined whether there is a rest node, and if there is a rest node, jumping to a step of recording the attribute value of the note value node if there is a pitch node, and continuing to execute subsequent steps, and if there is no rest, performing an error reporting process.

6. The step of performing tone group symbol processing comprises: Obtaining the scale and octave of the previous note and the scale and octave of the current note, and then judging whether the two adjacent notes belong to the same tone group based on whether the octaves of the two adjacent notes are equal; If two adjacent notes belong to the same pitch group, determine whether the absolute value of the pitch of the two notes is greater than 5, and if it is greater than 5, add a pitch group symbol in front of the current note, or do not add a pitch group symbol otherwise; 6. The embedded music / Braille character conversion system capable of being presented synchronously visually, aurally, and tactilely as claimed in claim 5, further comprising: when two adjacent notes belong to different pitch groups, if the difference between the two notes is equal to or greater than two pitch groups, adding a pitch group symbol in front of the current note, if it is determined that the two notes belong to two adjacent pitch groups and the note interval is greater than 3, adding a pitch group symbol in front of the current note, otherwise not adding a pitch group symbol, and finally outputting a Braille dot sequence of the pitch group symbol.

7. The process of converting chords in musical notation into sequences of Braille dots is determining whether the next note has a chord node, if so, obtaining the scale node, octave node and note value node of the current note, and recording the corresponding attribute values ​​in temporary variables, then putting the temporary variables in which the scale attribute value and octave attribute value are recorded into stacks s1 and s2 respectively, and setting the chord marker bit to 1, and repeating in this manner; 4. The embedded music / Braille text conversion system as claimed in claim 3, characterized in that if there is no chord node, it is necessary to determine whether the chord marker bit is 1; if the chord marker bit is not 1, convert to a braille dot sequence according to a normal note; if the chord marker bit is 1, write the current note as a root note, combine with the attribute value of the note value to convert the root note to a braille dot sequence; extract notes in the stack from the stack in order, while calculating the interval between the notes in the stack and the root note, convert the notes extracted from the stack to a braille dot sequence according to the interval, and set the chord marker bit to 0, repeatedly extract from the stack until the stack is empty, and output the braille dot sequence.

8. The process of converting the same note connection lines in a note into a sequence of braille dots is: Determine whether the child node of the notation node has a same note connection line node, and if there is a same note connection line node, traverse the next node, and if the next node is not empty but a note node, set both the chord marker bit and the double connection line marker bit to 1, and add a double connection line marker, if it is determined that the same note connection line node and the chord node coexist in the note node, it is necessary to add a same note connection line marker, and The embedded music / Braille text conversion system as described in claim 3, characterized in that if the next node is empty, the chord marker bit is set to 0, a same note connection line marker is added, and finally the Braille dot sequence result is output to the same note connection line marker and the double connection line marker.

9. The process of converting arpeggios and fermatas in musical notes into braille dot sequences is: When a chord and an arpeggio coexist, it is necessary to add an arpeggio marker before the root note. Specifically, if there is no chord node in the next note, add an arpeggio marker to the arpeggio of the current node, and output the braille dot sequence of the arpeggio marker. The embedded music / Braille text conversion system capable of being presented visually, aurally, and tactilely synchronized as described in claim 3, characterized in that when a chord and a fermata coexist, a fermata marker needs to be added before the root note, and specifically, if there is no chord node in the next note, a fermata marker is added to the fermata of the current node, and a Braille dot sequence of the fermata marker is output.

10. The output module performs score presentation and Braille dot presentation when outputting the Braille score in a visual manner; The process of presenting the musical score is to match the braille musical score with the braille musical symbol library to obtain the musical sequence of each bar, and when the current bar is being read, to input the musical sequence and execute the program to output the musical score image of the bar and display it synchronously on the screen; The braille dot presentation process is to supplement the length of the braille dot sequence in the braille music score, cut and store a certain length of the supplemented braille dot sequence into a string, and generate a graphic braille dot for each string and display it on the screen in synchronization; The embedded music / Braille text conversion system as described in claim 1, characterized in that, when outputting the Braille music score in an auditory manner, the output module matches the Braille music score with a Braille music symbol library to obtain the musical sequence of each bar, and when reading the current bar, takes the musical sequence as input, executes a program to output the audio of the bar, saves it in MIDI audio format, plays it, and displays it on the screen in sync.

11. 2. The embedded music / Braille character conversion system capable of being presented visually, aurally and tactilely synchronized as claimed in claim 1, characterized in that when the output module outputs the Braille music score in a tactile manner, communication between the upper computer and the lower computer of the embedded system is via a serial port as an interface, in which in the serial port communication protocol of the upper computer, a data packet consists of 43 bits of hexadecimal data, each of which is one data head high bit, one data head low bit, one CRC check bit and 40 data bits, and in the serial port communication protocol of the lower computer, a data packet consists of 4 bits of hexadecimal data, each of which is one data head high bit, one data head low bit, one CRC check bit and one data bit.

12. The embedded music / Braille text conversion system capable of being presented synchronously visually, aurally, and tactilely, as described in claim 1, characterized in that the conversion module is further configured to convert the Braille dot sequence into a Braille ASCII code and present Braille dots based on the Braille ASCII code.

Images