Music score playing data interactive processing display method for multiple chapters and segments and computer program

Abstract

The invention discloses a music score playing data interactive processing and displaying method for multiple chapters and music segments, which comprises the following steps of: processing, analyzing and identifying a scanning image of a music score document through an optical music score identification technology, finally obtaining digital expressions of a music score graph and music semantics, recording position information, and displaying the position information of the music score document; traversing all the position information of the stamps, the music segments and the bars; performing element extraction on notes of the music score, including stem extraction, head extraction and channel extraction, to obtain core and key data information of music score identification; according to single-sound and multi-sound music theory rules, setting a repeated starting point position, a repeated final point position and a playing sequence; and loading multimedia resources, and calculating the playing sequence of the single-sound part and the multi-sound part according to the starting point, the ending point and the sequence position information (X, Y, Z) of the steps. According to the invention, interaction processing of multi-row music score and multi-sound data can be realized, and display and playing can be realized after integration.

Description

Technical Field

[0001] The present invention relates to the field of intelligent teaching technology, and in particular to a method for interactively processing and displaying music score playback data of multiple chapters and sections, and a computer program. Background Art

[0002] Music teaching in the traditional teaching model is mainly achieved through on-site demonstration by teachers, which not only requires high quality of teachers, but also often lacks students' participation in practical learning. Under the integration of information technology, some music teaching software and smart devices with interactive functions have appeared on the market, allowing students to participate more in music learning and experience different learning experiences.

[0003] In the existing interactive intelligent devices for music teaching, the music score display data is usually generated by the music score playback data. The current interactive intelligent devices cannot combine the traditional music score data with multimedia audio and video for data interaction and positioning for playback and display, cannot realize the interactive processing of multi-line music scores and multi-part data, and cannot perform integrated display and playback. Summary of the invention

[0004] Based on the technical problems existing in the background technology, the present invention proposes a method for interactively processing and displaying music score playback data of multiple chapters and music sections and a computer program.

[0005] The present invention provides a method for interactively processing and displaying music score playing data of multiple chapters and sections, comprising the following steps:

[0006] S1 processes, analyzes and recognizes the scanned image of the music score document through optical music score recognition technology, and finally obtains the digital expression of the music score graphics and music semantics, records the position information, and traverses the position information of all movements, sections, and measures;

[0007] S2 extracts primitives from the musical notes, including stem extraction, note head extraction and note channel extraction, to obtain the core and key data information of musical score recognition;

[0008] The stem extraction comprises the following steps:

[0009] S211 scans in the vertical direction and performs vertical run encoding on the score image. If the length of the black pixel run exceeds a certain threshold, the run is marked as a "seed run";

[0010] S212 scans along the horizontal direction, performs horizontal run encoding on the image, retrieves the horizontal run where the image reference point in the "seed run", and obtains the contour points of the left and right boundaries of the "seed run";

[0011] S213 If the contour point of the "seed run" satisfies the following conditions, the run is determined to be a candidate symbol stem;

[0012] Conditions: The distance between the contour points on both sides of the point is less than a certain threshold; the contour point pair is symmetrical relative to the "seed run"; the contour points that meet the above conditions must be continuous and reach a certain length;

[0013] The note head extraction comprises the following steps:

[0014] S221, scanning in the horizontal direction, performing horizontal run encoding on the image, and deleting horizontal black pixel runs whose length is less than a certain threshold;

[0015] S222, marking connected domains of the image, for each connected domain, if the width is greater than a certain threshold and any upper and lower edge lines are close to a straight line, then delete the connected domain;

[0016] S223, using the stem as a dividing line to divide all connected domains, and using the spectrum line or the spectrum line between the lines as the dividing line to divide the connected domains with a height greater than a certain threshold;

[0017] S224, detecting whether the segmented connected domain has the characteristics of a symbol head through three indicators: length, height and area;

[0018] The symbol channel extraction comprises the following steps:

[0019] S231, scanning in the horizontal direction, performing horizontal run encoding on the image, deleting horizontal black pixel runs whose length is less than a certain threshold, and separating the symbol head and the symbol beam;

[0020] S232, detecting whether the connected domain in the image has the characteristics of a beam through thickness, length, and straightness of edge lines, the thickness of the connected domain is not less than 2 times the thickness of the spectrum line, the length is not less than the spacing between the spectrum lines, and at least one of the upper and lower edge lines of the beam is close to a straight line;

[0021] S233, calculating the number of note beams, the number of beams is obtained by dividing the thickness of the connected domain of the beams by the thickness of a single beam;

[0022] S3 sets the starting and ending positions of repetitions and the order of playback according to the music theory rules of monophonic and polyphonic parts;

[0023] S4 loads the multimedia resources and calculates the single-part and multi-part playback sequence according to the starting and ending points and the sequence position information (X, Y, Z) of steps S1, S2 and S3.

[0024] Preferably, the note head features are: the length of the note head connected domain is about 1.5 times the spectral line spacing, the height is 1 times the spectral line spacing, and the ratio of the pixel area to the connected domain bounding box area is about 0.8.

[0025] Preferably, the method for automatically calculating the thickness of a single beam by using the frequency of vertical run length is as follows: performing run encoding in the vertical direction on the segmented beam image; counting the frequency of the run length changes in the run encoding, generating a frequency histogram, and selecting the run length corresponding to the maximum peak in the histogram as the thickness of the single beam; if multiple peaks appear in the frequency histogram, it indicates that the beam is adhered to the top and bottom, and at this time, the run length corresponding to the leftmost peak is selected as the thickness of the single beam.

[0026] Preferably, after the step S1 is identified, the position information (X, Y), the starting point and the end point of each line of the score, and the position information of each key note are recorded, and the identified position information is stored in a temporary cache.

[0027] Preferably, in the process of adding the position information, the core and key data information is associated with a sequential number for each position information until the entire chapter or paragraph is completed.

[0028] Preferably, in the step S3, for multiple voices, a switch needs to be established, which is turned on to record a voice and a cache data, and different voices are marked with different colors on the basis of multiple voices;

[0029] Preferably, in the entire music score data collection process of step S3, the voice part is first determined, and then the voice part information is recorded, and a sequence is created for it, a temporary cache data pool is established, and then each polyphonic part includes multiple monophonic parts, each monophonic part includes multiple chapters, each chapter includes multiple paragraphs, each paragraph includes starting point information and end point information, and key note position information, when all movements, music sections, key note information (treble symbols, alto symbols, tenor symbols, half notes, quarter notes, sixteenth notes, sharps and flats) or simplified notation notes (do, re, mi, fa, so, la, si) key notes can be used as position information to store the data chain, and create a sequence for each information loaded into the data chain, and this sequence process is a playback and display order process.

[0030] Preferably, in the step S4, during the loading process of the media resource, a time axis is obtained, and the overall position information is evenly divided into the movement, the music section, and the note position a and position b of the key point are evenly divided through the time axis;

[0031] Preferably, the step S4 is auditioned, during which the operator's actual situation is perfectly combined to compare the key notes, and the corresponding notes heard are progressively approached through numerical compensation, and so on for the polyphonic part, completing the traversal and counting of the entire data pool, thereby achieving the purpose of spectral sound synchronization.

[0032] A computer program for interactively processing and displaying music score playing data of multiple chapters and sections includes a program and / or instructions. When the program and / or instructions are executed by a processor, the steps of the above-mentioned method are implemented.

[0033] In the present invention, the method for interactively processing and displaying music score playback data of multiple chapters and sections has a high accuracy rate in music score note recognition, combines traditional music score data with multimedia audio and video for data interaction and positioning, thereby determining a new playback mode and display method, and can realize interactive processing of multi-line music scores and multi-part data, and display and playback after integration. BRIEF DESCRIPTION OF THE DRAWINGS

[0034] Figure 1 A diagram showing the operation steps of a method for interactively processing and displaying music score playback data of multiple chapters and sections proposed by the present invention;

[0035] Figure 2 A schematic flow chart of a method for interactively processing and displaying music score playback data of multiple chapters and sections proposed by the present invention;

[0036] Figure 3 A schematic diagram of a stem extraction process of a method for interactively processing and displaying music score playback data of multiple chapters and sections proposed by the present invention;

[0037] Figure 4 A schematic diagram of a note head extraction process of a method for interactively processing and displaying music score playback data of multiple chapters and sections proposed by the present invention;

[0038] Figure 5 The present invention provides a schematic diagram of the process flow of extracting musical symbols for a method of interactively processing and displaying music score playback data with multiple chapters and sections. DETAILED DESCRIPTION

[0039] The technical solutions in the embodiments of the present invention will be described clearly and completely below in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, rather than all the embodiments.

[0040] Reference Figure 1-2 A method for interactively processing and displaying music score playing data of multiple chapters and sections comprises the following steps:

[0041] S1 processes, analyzes and recognizes the scanned image of the music score document through optical music score recognition technology, and finally obtains the digital expression of the music score graphics and music semantics, records the position information, and traverses the position information of all movements, sections, and measures;

[0042] S2 extracts primitives from the musical notes, including stem extraction, note head extraction and note channel extraction, to obtain the core and key data information of musical score recognition;

[0043] After recognition, the position information (X, Y), the starting point and end point of each line of the score, and the position information of each key note are recorded, and the recognized position information is stored in a temporary cache. In the process of adding the core and key data information to the position information, a sequence number is associated with each position information until the entire chapter or paragraph is completed;

[0044] Reference Figure 3 , the stem extraction comprises the following steps:

[0045] S211 scans in the vertical direction and performs vertical run encoding on the score image. If the length of the black pixel run exceeds a certain threshold, the run is marked as a "seed run";

[0046] S212 scans along the horizontal direction, performs horizontal run encoding on the image, retrieves the horizontal run where the image reference point in the "seed run", and obtains the contour points of the left and right boundaries of the "seed run";

[0047] S213 If the contour point of the "seed run" satisfies the following conditions, the run is determined to be a candidate symbol stem;

[0048] Conditions: The distance between the contour points on both sides of the point is less than a certain threshold; the contour point pair is symmetrical relative to the "seed run"; the contour points that meet the above conditions must be continuous and reach a certain length;

[0049] Reference Figure 4 , the note head extraction comprises the following steps:

[0050] S221, scanning in the horizontal direction, performing horizontal run encoding on the image, and deleting horizontal black pixel runs whose length is less than a certain threshold;

[0051] S222, marking connected domains of the image, for each connected domain, if the width is greater than a certain threshold and any upper and lower edge lines are close to a straight line, then delete the connected domain;

[0052] S223, using the stem as a dividing line to divide all connected domains, and using the spectrum line or the spectrum line between the lines as the dividing line to divide the connected domains with a height greater than a certain threshold;

[0053] S224, detecting whether the segmented connected domain has the characteristics of a symbol head through three indicators: length, height and area;

[0054] Reference Figure 5 , the symbol channel extraction comprises the following steps:

[0055] S231, scanning in the horizontal direction, performing horizontal run encoding on the image, deleting horizontal black pixel runs whose length is less than a certain threshold, and separating the symbol head and the symbol beam;

[0056] S232, detecting whether the connected domain in the image has the characteristics of a beam through thickness, length, and straightness of edge lines, the thickness of the connected domain is not less than 2 times the thickness of the spectrum line, the length is not less than the spacing between the spectrum lines, and at least one of the upper and lower edge lines of the beam is close to a straight line;

[0057] S233, calculating the number of note beams, the number of beams is obtained by dividing the thickness of the connected domain of the beams by the thickness of a single beam;

[0058] S3 sets the repeated starting and ending positions and the playback order according to the music theory rules of monophonic and polyphonic music. For polyphonic music, a switch needs to be established. When it is turned on, a part is recorded and a cache data is recorded. On the basis of polyphonic music, different parts are identified by different colors. During the entire score data collection process, the part is first determined, and then the part information is recorded, and a sequence is created for it, and a temporary cache data pool is established. Then each polyphonic part contains multiple monophonic parts, each monophonic part contains multiple chapters, each chapter contains multiple paragraphs, each paragraph contains starting and ending information, and key note position information. When all movements, music sections, key note information (treble symbols, alto symbols, tenor symbols, half notes, quarter notes, sixteenth notes, sharps and flats) or simplified notation notes (do, re, mi, fa, so, la, si) key notes can be used as position information to store the data chain, and create a sequence for each information loaded into the data chain, and this sequence process is a playback and display sequence process.

[0059] S4 loads multimedia resources, and calculates the order of single-part and multi-part playback according to the starting and ending points of step S1 and step S2 and the sequential position information (X, Y, Z); during the loading process of media resources, the time axis is obtained, and the overall position information is equally divided into the movement, the music section, and the note position a and position b of the key point are equally divided through the time axis; then, an audition is performed, and during the audition process, the key sound comparison is perfectly combined with the actual situation of the operator, and the corresponding notes heard are gradually approached through numerical compensation, and so on for the polyphonic part, to complete the traversal and counting of the entire data pool, so as to achieve the purpose of spectrum and sound synchronization.

[0060] A computer program for interactively processing and displaying music score playing data of multiple chapters and sections includes a program and / or instructions. When the program and / or instructions are executed by a processor, the steps of the above-mentioned method are implemented.

[0061] The present invention comprises the following steps: firstly, entering the music score editor, judging whether it is opened or not, displaying an error if it is not opened, repeating the loading, continuing to judge whether it is opened or not, and entering the editing section if it is opened, loading audio or video, performing optical music score recognition on the music score, extracting primitives of the music score, including stem extraction, note head extraction and note channel extraction, obtaining the core and key data information of the music score recognition, then recording the position information (X, Y), the starting point and end point of each line of the music score, and the position information of each key note, and storing the recognized position information in a temporary cache library, and in the process of adding the position information of the key note, associating a sequence number for each position information therein, until the entire chapter and paragraph are completed, and performing synchronous editing, setting the repeated starting position and end point position and the playback order according to the music theory rules of single voice and multi-voice; for multi-voice, a switch needs to be established, turned on, a voice part is recorded, a cache data is recorded, and different voice parts are marked by different colors on the basis of multi-voice; in the whole music score data collection process, the voice part is first judged, then the voice part information is recorded, and a Sequence, establish a temporary cache data pool, then each polyphonic part contains multiple monophonic parts, each monophonic part contains multiple chapters, each chapter contains multiple paragraphs, each paragraph contains starting point information and end point information, as well as key note position information, when all movements, sections, key note information (treble symbols, alto symbols, tenor symbols, half notes, quarter notes, sixteenth notes, sharps and flats) or simplified musical notation notes (do, re, mi, fa, so, la, si) key notes can be used as position information to store data chain, and create a sequence for each information loaded into the data chain, and make this sequence process a playback and display order process, load media resources, obtain the time axis, and divide the overall position information into equal parts for the movement, the section, and the key point note position a and position b through the time axis; then, perform audition, and in the audition process, perfect the comparison of key notes in combination with the actual situation of the operator, and gradually approach the corresponding notes heard through numerical compensation, and so on in the polyphonic part, complete the traversal and counting of the entire data pool, so as to achieve the purpose of spectrum and sound synchronization.

[0062] The above description is only a preferred specific implementation manner of the present invention, but the protection scope of the present invention is not limited thereto. Any technician familiar with the technical field can make equivalent replacements or changes according to the technical scheme and inventive concept of the present invention within the technical scope disclosed by the present invention, which should be covered by the protection scope of the present invention.

Claims

1. A method for interactively processing and displaying music score playing data of multiple chapters and sections, characterized in that: The steps include: S1 processes, analyzes and recognizes the scanned image of the music score document through optical music score recognition technology, and finally obtains the digital expression of the music score graphics and music semantics, records the position information, and traverses the position information of all movements, sections, and measures; S2 extracts primitives from the musical notes, including stem extraction, note head extraction and note channel extraction, to obtain the core and key data information of musical score recognition; The stem extraction comprises the following steps: S211 scans in the vertical direction and performs vertical run encoding on the score image. If the length of the black pixel run exceeds a certain threshold, the run is marked as a "seed run"; S212 scans in the horizontal direction, performs horizontal run encoding on the image, retrieves the horizontal run where the image reference point in the "seed run", and obtains the contour points of the left and right boundaries of the "seed run"; S213 If the contour point of the "seed run" satisfies the following conditions, the run is determined to be a candidate symbol stem; Conditions: The distance between the contour points on both sides of the point is less than a certain threshold; the contour point pair is symmetrical with respect to the "seed run"; the contour points that meet the above conditions must be continuous and reach a certain length; The note head extraction comprises the following steps: S221, scanning in the horizontal direction, performing horizontal run encoding on the image, and deleting horizontal black pixel runs whose length is less than a certain threshold; S222, marking connected domains of the image, for each connected domain, if the width is greater than a certain threshold and any upper and lower edge lines are close to a straight line, then delete the connected domain; S223, using the stem as a dividing line to divide all connected domains, and using the spectrum line or the spectrum line between the lines as the dividing line to divide the connected domains with a height greater than a certain threshold; S224, detecting whether the segmented connected domain has the characteristics of a symbol head through three indicators: length, height and area; The symbol channel extraction comprises the following steps: S231, scanning in the horizontal direction, performing horizontal run encoding on the image, deleting horizontal black pixel runs whose length is less than a certain threshold, and separating the symbol head and the symbol beam; S232, detecting whether the connected domain in the image has the characteristics of a beam through thickness, length, and straightness of edge lines, the thickness of the connected domain is not less than 2 times the thickness of the spectrum line, the length is not less than the spacing between the spectrum lines, and at least one of the upper and lower edge lines of the beam is close to a straight line; S233, calculating the number of note beams, the number of beams is obtained by dividing the thickness of the connected domain of the beams by the thickness of a single beam; S3 sets the starting and ending positions of repetitions and the order of playback according to the music theory rules of monophonic and polyphonic parts; S4 loads the multimedia resources and calculates the single-part and multi-part playback order according to the starting and ending points and the sequence position information (X, Y, Z) of steps S1, S2 and S3.

2. The method for interactively processing and displaying music score playing data of multiple chapters and sections according to claim 1, characterized in that: The characteristics of the note head are: the length of the connected domain of the note head is about 1.5 times the spectral line spacing, the height is 1 times the spectral line spacing, and the ratio of the pixel area to the connected domain bounding box area is about 0.

8.

3. The method for interactively processing and displaying music score playing data of multiple chapters and sections according to claim 1, characterized in that: The method for automatically calculating the thickness of a single beam using the frequency of vertical run length is as follows: performing run encoding on the segmented beam image in the vertical direction; counting the frequency of the run changing with length in the run encoding, generating a frequency histogram, and selecting the run length corresponding to the maximum peak in the histogram as the thickness of the single beam; if multiple peaks appear in the frequency histogram, it indicates that the beam is adhered to the top and bottom, and at this time, the run length corresponding to the leftmost peak is selected as the thickness of the single beam.

4. The method for interactively processing and displaying music score playing data of multiple chapters and sections according to claim 1, characterized in that: After the step S1 is identified, the position information (X, Y), the starting point and the end point of each line of the score, and the position information of each key note are recorded, and the identified position information is stored in a temporary cache.

5. The method for interactively processing and displaying music score playing data of multiple chapters and sections according to claim 4, characterized in that: In the process of adding the position information, the core and key data information is associated with a sequential number for each position information until the entire chapter and paragraph is completed.

6. The method for interactively processing and displaying music score playing data of multiple chapters and sections according to claim 1, characterized in that: The step S3 is multi-part, and a switch needs to be established. When it is turned on, a part is recorded, and a cache data is recorded. On the basis of the multi-part, different parts are marked by different colors.

7. The method for interactively processing and displaying music score playing data of multiple chapters and sections according to claim 1, characterized in that: In the entire music score data acquisition process of step S3, the voice part is first determined, and then the voice part information is recorded, and a sequence is created for it, a temporary cache data pool is established, and then each polyphonic part includes multiple monophonic parts, each monophonic part includes multiple chapters, each chapter includes multiple paragraphs, each paragraph includes starting point information and end point information, and key note position information. When all movements, music sections, key note information (treble symbols, alto symbols, tenor symbols, half notes, quarter notes, sixteenth notes, sharps and flats) or simplified notation notes (do, re, mi, fa, so, la, si) key notes can be used as position information to store data chains, and a sequence is created for each information loaded into the data chain, and this sequence process is a playback and display order process.

8. The method for interactively processing and displaying music score playing data of multiple chapters and sections according to claim 1, characterized in that: In the step S4, during the loading process of the media resource, a time axis is obtained, and the overall position information is evenly divided into the movement, the music section, and the note position a and position b of the key point are evenly divided through the time axis.

9. The method for interactively processing and displaying music score playing data of multiple chapters and sections according to claim 1, characterized in that: The step S4 is to perform an audition, during which the operator's actual situation is combined to compare the key notes, and the corresponding notes heard are gradually approached through numerical compensation. The same is true for the polyphonic part, and the traversal and counting of the entire data pool are completed, thereby achieving the purpose of spectrum and sound synchronization.

10. A computer program for interactively processing and displaying music score playing data of multiple chapters and sections according to claim 1, comprising a program and / or instructions, characterized in that: When the program and / or instruction is executed by a processor, the steps of the method according to claims 1 to 9 are implemented.

Images