Music arithmetic logic system and method based on fixed pitch intensity

Abstract

The invention discloses a music arithmetic logic system and method based on fixed pitch intensity. The method comprises the following steps: receiving music score data and receiving multi-segment recording data in real time by adopting a callback function; comparing the difference between the recording data and the music score data based on the pitch of the music score data; based on the differentiation, recording scores for the recording data; and sending the recording score of the recording data to the client. According to the system, comparison and scoring are set, so that students can clearly know how many mistakes exist, and know the degree of mistakes according to the score so as to correct the mistakes in real time when the mistakes exist; in addition, the system can also enable the students to replay and compare and learn for many times after finding the mistakes, so that a technology for correcting intonation, pitch and the like of the mistakes is developed from the beginning, and a good foundation is provided for the students to develop music in the future.

Description

Technical Field

[0001] The present invention belongs to the technical field of logic operation, and in particular relates to a music operation logic system and method based on fixed pitch strength. Background Art

[0002] Music education is expanding with a growing number of teaching models, and intelligent music teaching is also beginning to enter people's lives and learning. With the rapid development of internet technology, learning tutorials are being delivered to students in an intelligent way, helping them quickly master skills and providing personalized instruction, turning learning music into playing music.

[0003] Although there are many types of intelligent music teaching systems on the market, due to differences in software functions, learners' experiences vary greatly, resulting in the following problems:

[0004] The teaching only teaches how to get into pitch, but does not have the function of comparing the results of students' actual practice and whether they are right or wrong. Therefore, it is unclear what the results of students' practice are, and it is impossible to correct students' mistakes. In this way, when students make mistakes, it is not very obvious for them to do so. As a result, there are always mistakes in the process of students learning music scores. This will continue. Because they made mistakes at the beginning, it may be difficult to correct them later, which will seriously affect the students' future musical development. Summary of the Invention

[0005] The purpose of the present invention is to overcome the above problems and provide a music operation logic system and method based on fixed pitch strength.

[0006] In order to achieve the above purpose, the technical solution adopted by the present invention is:

[0007] A music operation logic method based on fixed pitch strength, the server side includes the following steps:

[0008] Receive music score data and use a callback function to receive multiple recording data in real time; wherein, using the callback function to receive multiple recording data in real time includes the following steps: generating an audio stream array based on the multiple recording data; the audio stream array includes a recording start time, a recording end time, a pitch, and a volume;

[0009] comparing the difference between the recording data and the music score data based on the pitch of the music score data;

[0010] Based on the differentiation, recording a score for the audio recording data;

[0011] The recording data record score is sent to the client.

[0012] As a further improvement, the music score data includes a note array; the note array includes notes; the notes include:

[0013] Pitch, accompaniment playing time and playing duration; among them,

[0014] Setting the height of the waterfall according to the pitch;

[0015] Setting the horizontal position of the waterfall notes according to the accompaniment playing time;

[0016] Setting the length of the waterfall note according to the playing duration;

[0017] A waterfall is generated based on the height, the horizontal position, and the length.

[0018] As a further improvement, comparing the difference between the recording data and the music score data based on the pitch of the music score data includes the following steps:

[0019] Matching the recording start time and the recording end time of the recording data with the accompaniment playing start time + the playing duration of the music score data;

[0020] Among them, when the recording start time and the recording end time of the recording data have a time intersection with the accompaniment start playback time + the playback duration of the music score data, the notes of the recording data and the note data in the time intersection part are matched.

[0021] As a further improvement, the matching of the recording start time and the recording end time of the recording data with the accompaniment play start time + the play duration of the music score data includes the following steps:

[0022] Determine whether the end time of the note of the recording data is after the end time of the current note of the music score data and the current note is not the last note;

[0023] If so, the note of the recording data matches the next note of the current note of the music score data;

[0024] If not, the note of the recording data matches the current note of the music score data.

[0025] As a further improvement, the step of recording a score for the recording data based on the differentiation comprises the following steps:

[0026] Get the audio stream array within a certain period of time;

[0027] Calculate the pitch score using the first calculation method; or

[0028] The second operation method is used to calculate the pitch score; wherein,

[0029] The method of calculating the pitch score using the first operation method includes the following steps:

[0030] Filter out notes in the audio stream array whose volume is lower than a threshold;

[0031] Excluding the maximum and minimum pitch values, and the maximum and minimum volume values ​​in the audio stream array;

[0032] Based on the calculation of the average value of the remaining pitch and volume, the pitch score is obtained;

[0033] The method of calculating the pitch score using the second operation mode comprises the following steps:

[0034] Calculate the last pitch score in the audio stream array;

[0035] The last pitch score is used as the pitch score.

[0036] As a further improvement, the step of recording a score for the recording data based on the differentiation further includes the following steps:

[0037] Obtaining the pitch of the recording data at regular intervals;

[0038] Subtract the pitch of the recorded data from the pitch of the music score data to obtain a difference A;

[0039] Within the error range, subtract the difference A from the pitch of the music score data as a criterion for judging whether the recording data is sung correctly;

[0040] Receive an adaptation mode signal to start recording data, set the error value to 0, and calculate the appropriate error value for the adapted recording data in real time;

[0041] Determine whether the pitch of the recording data of the recording node is within the upper and lower error value range of the pitch of the current music score data;

[0042] If the pitch is within the upper and lower error range of the current score data, the recording data is in the correct singing state, the note is scored, and the rhythm score is obtained.

[0043] As a further improvement, in the judgment of whether the recording data is sung correctly,

[0044] When the pitch of the recorded data is correct, the pitch ball moves to the height of the pitch of the music score data and the note of the recorded data is highlighted;

[0045] When the pitch of the recorded data is correct, the pitch ball moves to the height of the pitch of the recorded data.

[0046] As a further improvement, the client includes the following steps:

[0047] Panning animation: Set the duration of waterfall panning animation according to the duration of the music score;

[0048] Correct note highlight animation: This includes the waterfall generation style and the timing of the correct note highlight elements, which is determined by the start time of recording when the current recording node and the matching human voice waterfall note reach the target pitch.

[0049] When singing correctly, the note icon will float in the animation: the curve of the note is continuous and opens upwards.

[0050] As a further improvement, the animation of the note icon floating when singing correctly includes the following steps:

[0051] The starting point of the floating animation is the height corresponding to the pitch and the time corresponding to the start time of the recording when the recording node and the matching adaptive waterfall note reach the standard. The end point is the constant of the pitch. Draw four quadrants. When the tangent point of the curve is in the third quadrant, the curve meets the requirements. The coordinate calculation formula of the tangent point of the curve is

[0052] X = (starting point X - ending point X) / 4; Y = (ending point Y - starting point Y) / 4.

[0053] On the other hand, the present invention also provides a music operation logic system based on fixed pitch strength, which includes at least one processor and a memory storing instructions. When the instructions are executed by at least one processor, the method in the above technical solution is implemented.

[0054] A computer-readable storage medium stores a computer program, wherein the computer program is executed by a processor to implement the method described in the steps of the above technical solution.

[0055] A computer program product includes a computer program, and when the computer program is executed by a processor, the method described in the steps of the above technical solution is implemented.

[0056] The beneficial effects of the present invention are as follows: by processing the music score, students can better compare the pitch and find out the differences, and at the same time, a more accurate score is given for the real-time syllables. Based on the score, students can know where they are lacking and can check for omissions and fill in the gaps in a targeted manner; students can use rhythm points, pitch points and completeness points to have a preliminary understanding of these three points, the direction of their own deficiencies, and further consolidate their learning or make targeted breakthroughs; the presentation effect of the animation mode is adopted, and when the notes are sung correctly, the note icons will be animated and displayed, so that the pictures of singing correctly and incorrectly can be seen clearly and intuitively, and the curve of the note floating is coherent and opens upward, which realizes the upward pulling of the real-time note intensity. When singing correctly, a highlighted animation will be presented, and students can clearly see Whether the singing is right or wrong; students can practice many times and get close to the fixed notes again and again, which serves as a good standard; the audio stream array is used to include the contents of the recording node, and then the callback function is used to call multiple recording nodes, and then the volume is obtained. In the scoring process, the volume that is too low is ignored, which improves the accuracy of the scoring; this system sets up comparison and scoring, and students can clearly know whether they are wrong or too much, and the degree of error can be known according to the score, so that real-time correction can be made; this system can also allow students to find that they are wrong, replay, and re-learn by comparison many times, so as to develop a skill to correct wrong intonation and pitch from the beginning, which provides a good foundation for students' future musical development. BRIEF DESCRIPTION OF THE DRAWINGS

[0057] The present invention has the following accompanying drawings:

[0058] Figure 1 The principle of a music operation logic system based on fixed pitch strength in some specific embodiments of the present invention Figure 1 ;

[0059] Figure 2 The principle of a music operation logic system based on fixed pitch strength in some specific embodiments of the present invention Figure 2 ;

[0060] Figure 3 The music score data of some specific embodiments of the present invention is converted into a waterfall flow chart;

[0061] Figure 4 It is a note floating animation diagram of some specific embodiments of the present invention;

[0062] Figure 5 are the coordinate axes of the starting and ending points of the note floating animation in some specific embodiments of the present invention;

[0063] Figure 6 is a flowchart of intonation node processing in some specific embodiments of the present invention;

[0064] Figure 7is a coordinate time comparison diagram of a certain note of the recording data of some specific embodiments of the present invention;

[0065] Figure 8 A schematic diagram of a music operation logic system based on fixed pitch strength of the present invention. DETAILED DESCRIPTION

[0066] The following examples further illustrate the present invention, but should not be construed as limiting the present invention. Without departing from the spirit and substance of the present invention, modifications or substitutions made to the methods, steps or conditions of the present invention are within the scope of the present invention.

[0067] like Figure 1-7 As shown, the present invention provides a music operation logic method based on fixed pitch strength, and the server includes the following steps:

[0068] Step S1: receiving music score data and receiving multiple recording data in real time using a callback function; wherein the receiving multiple recording data in real time using the callback function comprises the following steps: generating an audio stream array based on the multiple recording data; the audio stream array includes a recording start time, a recording end time, a pitch, and a volume;

[0069] Step S2: comparing the difference between the recording data and the music score data based on the pitch of the music score data;

[0070] Step S3: Based on the differentiation, recording a score for the recording data;

[0071] Step S4: Send the recording data recording score to the client.

[0072] In some embodiments of the present invention, the music score data includes a note array; the note array includes notes; and the notes include:

[0073] Pitch, accompaniment playing time and playing duration; among them,

[0074] Setting the height of the waterfall according to the pitch;

[0075] Setting the horizontal position of the waterfall notes according to the accompaniment playing time;

[0076] Setting the length of the waterfall note according to the playing duration;

[0077] A waterfall is generated based on the height, the horizontal position, and the length.

[0078] In this invention, a waterfall visualization method is used, which is mainly used in music-related application scenarios. It includes the following contents:

[0079] The input parameter note contains music-related data, such as note (midiNote), play time (playTime), duration (playDuration), etc.

[0080] Main processing logic:

[0081] Get the corresponding keyboard index (keyEqIndex) through MIDI notes;

[0082] Calculate the display time (showTime) and play duration (playDuration) of the note;

[0083] Calculate the position of the note in the waterfall (right, bottom);

[0084] Set the width (width) and height (height) of the note;

[0085] The return value is an object containing:

[0086] Note style information (width, height, right, bottom, etc.);

[0087] Music-related data (showTime, playTime, playDuration);

[0088] Some status flags (isActive, flashShow, etc.);

[0089] Here, the notes are displayed in the form of a waterfall, so users can intuitively see the playing time and duration of the notes.

[0090] Comparing the difference between the recording data and the music score data based on the pitch of the music score data includes the following steps:

[0091] Matching the recording start time and the recording end time of the recording data with the accompaniment playing start time + the playing duration of the music score data;

[0092] Among them, when the recording start time and the recording end time of the recording data have a time intersection with the accompaniment start playback time + the playback duration of the music score data, the notes of the recording data and the note data in the time intersection part are matched.

[0093] The matching of the recording start time and the recording end time of the recording data with the accompaniment play start time + the play duration of the music score data comprises the following steps:

[0094] Determine whether the end time of the note of the recording data is after the end time of the current note of the music score data and the current note (the note of the music score data) is not the last note;

[0095] If so, the note of the recording data matches the next note of the current note of the music score data;

[0096] If not, the note of the recording data matches the current note of the music score data.

[0097] For example, the rule for singing early: after the start time of the note plus half the playing time of the note, before the start of the playing, if the pitch matches the note pitch, it is recorded as singing early. Figure 7 As shown, there are two coordinate axes. The top (first) one is the coordinate axis for the score data, where the horizontal axis represents the score playback time (including the start time, end time, and duration), and the vertical axis represents the score pitch. The second one is the coordinate axis for the recording data, where the horizontal axis represents the recording playback time (including the start time, end time, and duration), and the vertical axis represents the recording pitch. Comparing the first and second axes, we find that for note B, the end time t1 of the recording data in the second coordinate axis is earlier than the end time T1 of the score data. Therefore, note B in the recording data is note B in the score data. Next, comparing note E in the second coordinate axis with note D in the first coordinate axis, we find that the end time t2 of note E in the recording data is later than the end time T2 of note D in the score data. Since note D in the score data is not the last note (because it contains at least one note E), note E in the recording data is the note E following note D in the score data (i.e., note D+1), thus achieving note coordinate +1. The ABCDE in the diagram represent a sequential relationship.

[0098] In this invention, a complete audio recording and real-time analysis function is mainly used in music-related application scenarios. It includes the following contents:

[0099] Core features:

[0100] Audio recording using MediaRecorder of the Web Audio API

[0101] Real-time analysis of audio data, including volume calculation, pitch detection, and MIDI note conversion.

[0102] Main processing flow:

[0103] Initialize the recording environment (check / stop existing recording);

[0104] Create a pitch tracker (DywaPitchTracker);

[0105] Set the audio processing callback (onaudioprocess);

[0106] Start recording and collecting data;

[0107] Real-time audio analysis:

[0108] For each frame of audio data, the following calculations are performed:

[0109] Current playback time, volume value, audio, and corresponding MIDI note.

[0110] Data Storage:

[0111] The original audio data is stored in the database;

[0112] The analysis results are stored in an array;

[0113] Application scenarios:

[0114] This would likely be a core feature of a music teaching / practice app, used to: record user performances, analyze pitch accuracy in real-time, provide visual feedback, and possibly a grading system for singing / instrument practice.

[0115] The step of recording a score for the recording data based on the differentiation comprises the following steps:

[0116] Get the audio stream array within a certain period of time;

[0117] Calculate the pitch score using the first calculation method; or

[0118] The second operation method is used to calculate the pitch score; wherein,

[0119] The method of calculating the pitch score using the first operation method includes the following steps:

[0120] Filter out notes in the audio stream array whose volume is lower than a threshold;

[0121] Excluding the maximum and minimum pitch values, and the maximum and minimum volume values ​​in the audio stream array;

[0122] Based on the calculation of the average value of the remaining pitch and volume, the pitch score is obtained;

[0123] The method of calculating the pitch score using the second operation mode comprises the following steps:

[0124] Calculate the last pitch score in the audio stream array;

[0125] The last pitch score is used as the pitch score.

[0126] For example, two calculation methods are used:

[0127] Method 1: Calculate the average value of MIDI notes and volume within a time period;

[0128] Method 2: Get the MIDI value at the last moment;

[0129] Filter invalid notes (return 0 when the volume is too low);

[0130] The average is calculated after excluding the maximum and minimum values.

[0131] The step of recording a score for the recording data based on the differentiation further includes the following steps:

[0132] Obtaining the pitch of the recording data at regular intervals;

[0133] Subtract the pitch of the recorded data from the pitch of the music score data to obtain a difference A;

[0134] Within the error range, subtract the difference A from the pitch of the music score data as a criterion for judging whether the recording data is sung correctly;

[0135] Receive an adaptation mode signal to start recording data, set the error value to 0, and calculate the appropriate error value for the adapted recording data in real time;

[0136] Determine whether the pitch of the recording data of the recording node is within the upper and lower error value range of the pitch of the current music score data;

[0137] If the pitch is within the upper and lower error range of the current score data, the recording data is in the correct singing state, the note is scored, and the rhythm score is obtained.

[0138] In some embodiments of the present invention, in determining whether the recorded data is sung correctly,

[0139] When the pitch of the recorded data is correct, the pitch ball moves to the height of the pitch of the music score data and the note of the recorded data is highlighted;

[0140] When the pitch of the recorded data is correct, the pitch ball moves to the height of the pitch of the recorded data.

[0141] Completeness calculation score (completeness score):

[0142] Integrity calculation unit score = Integrity calculation score / number of notes;

[0143] During the performance, if the pitch of the recording data of the recording node is within the upper and lower error values ​​of the pitch of the current score data and the start time of the current fixed note is within the interval from the start time to the end time of the recording, then the completeness calculation score + = completeness calculation unit score; where the calculation formula is

[0144] f=sum / b, where f represents the unit score for completeness calculation, sum represents the completeness calculation score, and b represents the number of notes.

[0145] If the pitch of the recording data of the recording node is within the upper and lower error values ​​of the pitch of the current score data and the start time of the current fixed note is within the interval from the start time to the end time of the recording, then the completeness calculation score + = completeness calculation unit score; where the calculation formula is

[0146] sum+=f, where sum represents the completeness calculation score and f represents the completeness calculation unit score. The recorded score also includes the rhythm score.

[0147] In some embodiments of the present invention, the recorded score also includes a rhythm score:

[0148] Rhythm calculation score (rhythm score):

[0149] Rhythm calculation unit score = rhythm calculation score / number of notes;

[0150] During the performance, if the pitch of the recording data of the recording node is within the upper and lower error values ​​of the pitch of the current score data and the start time of the current fixed note is within the interval from the start time to the end time of the recording, then the rhythm calculation score + = rhythm calculation unit score; where the calculation formula is

[0151] f=sum / b, where f represents the rhythm calculation unit fraction, sum represents the rhythm calculation fraction, and b represents the number of notes.

[0152] If the pitch of the recording data of the recording node is within the upper and lower error value range of the pitch of the current music score data and the start time of the current fixed note is within the interval from the start time to the end time of the recording, then the rhythm calculation score + = rhythm calculation unit score; the calculation formula is sum + = f, where sum represents the rhythm calculation score and f represents the rhythm calculation unit score.

[0153] In some embodiments of the present invention, the recorded score also includes an intonation score:

[0154] Intonation calculation score:

[0155] Intonation calculation unit score = ratio of the product of the total intonation score and the duration of the recording node to the cumulative sum of the standard note durations;

[0156] During the singing process, if the pitch of the recording data of the recording node is within the upper and lower error value range of the pitch of the current music score data and the interval from the start time to the end time of the recording node intersects with the start time to the end time of the current fixed note, then the pitch score + = pitch calculation unit score.

[0157] Unit score calculation formula

[0158]

[0159] Calculate the pitch score during singing:

[0160] Intonation score + = unit score.

[0161] In the present invention, Figure 2 As shown, it is a music teaching system, including a music score library, a recording device, a music operation logic system and a display; the music operation logic system obtains the music score data and the recording data from the music score library and the recording device respectively; the note array is generated in the waterfall flow module and then passed to the scoring module, and the audio stream array is generated in the singing recording module and also sent to the scoring module for scoring; then the waterfall flow data and the scoring data (rhythm score, pitch score and completeness score) are passed to the display; the display displays the received content on the waterfall flow interface after front-end processing.

[0162] On the other hand, the present invention provides a music operation logic method based on fixed pitch strength, the server comprising the following steps:

[0163] Step S10: receiving music score data (the music score data played, which is standard music score data. When students learn, they can record and then compare the music score data to see the difference between the two and then give a score) and using a callback function to receive multiple segments of recording data in real time (the music score data of the singing recording, the recording data received by the callback function used in the present invention); wherein, the use of the callback function to receive multiple segments of recording data in real time includes the following steps: generating an audio stream array based on multiple segments of recording data; the audio stream array includes a recording start time, a recording end time, a pitch and a volume; obtaining the pitch: converting the audio stream array into an audio frequency, and converting the audio frequency into a pitch; obtaining the real-time volume: obtaining the volume of the audio stream array based on the audio stream array.

[0164] For example, the generation of singing recording data: each time interval of recording generates an audio stream array, which I call a recording node here. The attributes of each recording node include: recording start time, recording end time, pitch, and volume. The caller obtains the data of each recording segment in the form of a callback function. Then the singing is scored. 1. Get the pitch. The audio stream array will be obtained through media stream recording. The audio array is converted into audio frequency through the algorithm and then converted into pitch by the algorithm. 2. Get the volume. The volume of the audio stream array is obtained by a specific algorithm. Among them, the specific algorithm is the volume algorithm of the recording node, which includes the following contents:

[0165] Step 1: Get channelData audio data:

[0166] channelData=e.inputBuffer.getChannelData(0);

[0167] channelData is a Float32Array array, and its value range is usually between [-1,1];

[0168] The meaning of this range is that the absolute value of the number represents the volume (amplitude intensity).

[0169] Step 2: Get the volume array of the recording node;

[0170] Get the volume of the audio stream node from the recording node volume array;

[0171] The volume of an audio stream node = the ratio of the sum of the absolute values ​​of each item in the channelData array to the length of the channelData array; among them, the volume array of a recording node = the volume of N audio stream nodes.

[0172] Step 3: Recording node volume = ratio of the sum of each item in the recording node volume array to the length of the recording node volume array.

[0173] Step S20: comparing the difference between the recording data and the music score data based on the pitch of the music score data.

[0174] For example, according to the start time and end time of the recording node and the current waterfall note, such as note A, the waterfall note accompaniment start time of A is matched. When the start time to end time of the recording node is consistent with the waterfall note accompaniment start time, the volume is compared. If the volume is too small, no score is recorded (the value of too small volume depends on the volume of the current noise, so the volume must be higher than the noise during singing). Then calculate whether the pitch value of the recording node and the waterfall pitch meet the standard (meeting the standard is not equal, and a certain error is allowed with the adapted human voice waterfall note pitch. Here you need to pay attention to the adapted human voice waterfall pitch. It refers to the pitch compared by the dynamic calculation algorithm designed according to each person's respective pitch range). Finally, count the scores calculated for all recording nodes.

[0175] Step S30: Based on the differentiation, the score of the recording data is recorded; wherein, the rhythm score is recorded in real time, including: when the recording start time and the recording end time of each segment of the recording data have a time intersection with the accompaniment start playback time + the playback duration of each segment, the notes of the recording data and the note data in the time intersection part are matched; based on the note matching degree of the time intersection part of each segment, the rhythm score of each segment is obtained.

[0176] Step S40: Send the recording score of the recording data to the client. The recorded score also includes the complete score.

[0177] Completeness calculation score (completeness score):

[0178] Integrity calculation unit score = Integrity calculation score / number of notes;

[0179] During the performance, if the pitch of the recording data of the recording node is within the upper and lower error values ​​of the pitch of the current score data and the start time of the current fixed note is within the interval from the start time to the end time of the recording, then the completeness calculation score + = completeness calculation unit score; where the calculation formula is

[0180] f=sum / b, where f represents the unit score for completeness calculation, sum represents the completeness calculation score, and b represents the number of notes.

[0181] If the pitch of the recording data of the recording node is within the upper and lower error values ​​of the pitch of the current score data and the start time of the current fixed note is within the interval from the start time to the end time of the recording, then the completeness calculation score + = completeness calculation unit score; where the calculation formula is

[0182] sum+=f, where sum represents the completeness calculation score and f represents the completeness calculation unit score. The recorded score also includes the rhythm score.

[0183] In some embodiments of the present invention, the recorded score also includes a rhythm score:

[0184] Rhythm calculation score (rhythm score):

[0185] Rhythm calculation unit score = rhythm calculation score / number of notes;

[0186] During the performance, if the pitch of the recording data of the recording node is within the upper and lower error values ​​of the pitch of the current score data and the start time of the current fixed note is within the interval from the start time to the end time of the recording, then the rhythm calculation score + = rhythm calculation unit score; where the calculation formula is

[0187] f=sum / b, where f represents the rhythm calculation unit fraction, sum represents the rhythm calculation fraction, and b represents the number of notes.

[0188] If the pitch of the recording data of the recording node is within the upper and lower error values ​​of the pitch of the current score data and the start time of the current fixed note is within the interval from the start time to the end time of the recording, then the rhythm calculation score + = rhythm calculation unit score; where the calculation formula is

[0189] sum+=f, where sum represents the rhythm calculation score and f represents the rhythm calculation unit score.

[0190] In some embodiments of the present invention, the recorded score also includes an intonation score:

[0191] Intonation calculation score:

[0192] Intonation calculation unit score = the sum of the total intonation score and the ratio of the standard note duration to the recording node duration;

[0193] During the singing process, if the pitch of the recording data of the recording node is within the upper and lower error value range of the pitch of the current music score data and the interval from the start time to the end time of the recording node intersects with the start time to the end time of the current fixed note, then the pitch score + = pitch calculation unit score.

[0194] Unit score calculation formula

[0195]

[0196] Calculate the pitch score during singing:

[0197] Intonation score + = unit score.

[0198] In some embodiments of the present invention, the music score data includes a note array; the note array includes notes; and the notes include:

[0199] Pitch, accompaniment playing time and playing duration; among them,

[0200] Set the height of the waterfall according to the pitch;

[0201] Setting the horizontal position of the waterfall note according to the accompaniment playing time; wherein the accompaniment playing time includes the accompaniment starting playing time;

[0202] The length of the waterfall note is set according to the playing duration; and the waterfall is generated based on the height, the horizontal position and the length.

[0203] For example, in waterfall data generation, the score data consists of an array of notes, whose attributes include the accompaniment playback time, duration, and pitch. The horizontal position of the waterfall notes is set based on the playback time, the duration sets the length of the waterfall notes, and the pitch sets the height of the waterfall, thus generating a waterfall.

[0204] like Figure 6 As shown, based on the pitch of the music score data, comparing the difference between the recording data and the music score data includes the following steps:

[0205] Step S200: Matching the recording start time and the recording end time of the recording data with the accompaniment start playing time + the playing duration of the music score data; wherein, when the recording start time and the recording end time of the recording data and the accompaniment start playing time + the playing duration of the music score data have a temporal intersection, the recording data and the note data are matched at the temporal intersection;

[0206] For example, according to the start time and end time of the recording node and the current waterfall note, such as note A, the waterfall note accompaniment start time of A is matched, and the volume is compared when the start time to the end time of the recording node is matched with the waterfall note accompaniment start time.

[0207] Step S201:.

[0208] For example, if the volume is too low, no score will be given (the value of the volume being too low depends on the volume of the current noise, so the volume must be higher than the noise when singing).

[0209] Step S202:.

[0210] Matching the recording start time and the recording end time of the recording data with the accompaniment play start time + the play duration of the music score data includes the following steps:

[0211] Step S2000: determining whether the end time of the note of the recording data is after the end time of the current note of the music score data and the current note is not the last note;

[0212] Step S2001: If yes, the note of the recording data matches the next note of the current note of the music score data;

[0213] Step S2002: If not, the note of the recording data matches the current note of the music score data.

[0214] Based on the differentiation, a score is recorded for the recorded data, comprising the following steps:

[0215] Step S300: Based on the note matching, volume matching, and pitch matching, a rhythm score, an accuracy score, and a completeness score are calculated. The rhythm score is a real-time recorded score, including: when the recording start time and the recording end time of each segment of the recording data overlap with the accompaniment playback start time + the playback duration of each segment, the recording data and the note data have a note match in the overlapping time portion; and the rhythm score of each segment is calculated based on the note matching in the overlapping time portion of each segment.

[0216] For example, Figure 4As shown in the figure, the number "1.93" on the right side of the figure is the pitch score obtained for singing the correct section. There is a "16" in the upper left corner of the figure, which is the rhythm score obtained for singing the correct section.

[0217] For example, entering singing mode will display the score data in a waterfall flow. Clicking "Sing" will start recording, comparing the singing recording with the played score data, including rhythm, pitch, and completeness. At the same time, the waterfall flow will animate from left to right.

[0218] In step S300, the step of recording a score for the recording data based on the differentiation includes the following steps:

[0219] Step S3000: receiving a signal for starting an adaptation mode for recording data, setting the error value to 0, and calculating in real time the appropriate error value for adapting the recording data;

[0220] Step S3001: Determine whether the pitch of the recording data of the recording node is within the upper and lower error value range of the pitch of the current music score data;

[0221] Step S3002: If the note is within the upper and lower error range, the real-time note score is added, wherein the calculation formula for the score is:

[0222]

[0223] In the formula, f represents the bonus score, a represents the duration of the notes in the recording data, b represents the duration of the notes in the current music score data, and A represents the bonus score corresponding to the duration of the notes in the current music score data.

[0224] The client includes the following steps:

[0225] Step S50: receiving the rhythm score and the pitch score from the server;

[0226] Step S60: Animating the rhythm score and the pitch score in a waterfall flow from left to right.

[0227] The client also includes the following steps:

[0228] Step S70: horizontal motion animation: setting the duration of the waterfall horizontal motion animation according to the duration of the music score;

[0229] Step S80: highlighting animation of the correct note: including the waterfall flow generation style and the generation time of the highlight element of the correct note, which depends on the start time of recording when the current recording node and the matching adaptive human voice waterfall flow note reach the target pitch;

[0230] Step S90: When singing correctly, the note icon performs floating animation: the curve of the note floating is continuous and opens upwards.

[0231] In the present invention, the standard pitch minus the calculated pitch difference is directly used as the pitch for judging whether the singing is correct or incorrect. When any singing is correct, half the score of the note will be added to the current fixed note. When calculating the pitch score, the real-time pitch accuracy = correct number / total number of times. The pitch ball will be presented on the screen on the large screen, and the height of the pitch ball will be the height of the standard pitch. If the note is sung correctly for a time slice, half a pitch point will be added (because a note will be added half a note point if it is sung correctly once). If it is sung incorrectly, the pitch ball will move to the height of the original audio pitch.

[0232] When calculating the rhythm score, the client is adjusted to the vocal mode, and when it is sung correctly, it corresponds to the current note pitch (within the pitch error range).

[0233] For example, animations include: 1. A horizontal animation with the duration of the waterfall flow set according to the duration of the score. 2. A highlight animation for correctly sung notes. The logic is similar to the waterfall flow generation, except that the timing of the highlight element for correctly sung notes depends on the start time of recording when the current recording node and the matching human voice waterfall flow note reach the target pitch. 3. A floating animation of the note icon when correctly sung is implemented. The requirement is that the floating curve of the note is continuous and opens upward.

[0234] The animation of the note icon floating when singing correctly includes the following steps:

[0235] Step S900: The starting point of the floating animation is the height corresponding to the pitch and the time corresponding to the start time of recording when the recording node and the matching adaptive waterfall note reach the standard (when the student sings correctly, the floating animation will appear; the starting point of the floating animation is the horizontal and vertical coordinates composed of these two, time is the horizontal coordinate, and height is the vertical coordinate), and the end point is the constant of the pitch (the constant refers to the horizontal coordinate of the starting point of each floating animation, i.e., time, and the vertical coordinate, i.e., pitch, which is a fixed value, constant), draw four quadrants, and when the tangent point of the curve is in the third quadrant, the curve meets the requirements; wherein, the coordinate calculation formula of the tangent point of the curve is

[0236] X = (starting point X - ending point X) / 4; Y = (ending point Y - starting point Y) / 4.

[0237] like Figure 4 As shown in the figure, the number "1.93" on the right side of the figure is the pitch score obtained for singing the correct section. There is a "16" in the upper left corner of the figure, which is the rhythm score obtained for singing the correct section.

[0238] like Figure 3 、 4 As shown, Figure 4 The content in Figure 3When the note is sung correctly, the note icon will float in an animation (so that the student can see whether the note is sung correctly or not). The requirement is that the note floating curve is continuous and opens upward. The implementation logic is: the starting point of the floating animation is the height corresponding to the pitch, and the end point is a fixed constant. Draw four quadrants (that is, draw the X and Y coordinate axes, and the horizontal and vertical axes will be divided into 4 quadrants, namely the first quadrant, the second quadrant, the third quadrant and the fourth quadrant). When the tangent point of the curve is in the third quadrant, the curve meets the requirement. Figure 5 The formula for calculating the tangent point coordinates (X, Y) is: X = (starting point X - end point X) / 4; Y = (end point Y - starting point Y) / 4.

[0239] In the present invention, the server-side recording real-time data collection and processing module includes the following: creating a pitch tracker to track the strength of the pitch in real time; processing the encapsulated audio array data to obtain the current playback time; checking whether recording is in progress; obtaining audio data from the input buffer; obtaining the maximum volume value in the buffer; calculating the audio frequency and converting the audio frequency into MIDI notes; wherein, during this period, the MIDI number is adjusted by adding 1 to the MIDI note; recording the last recording time; when there is available data, pushing it into the audio stream array; saving the recorded data; capturing and outputting the error message of recording failure;

[0240] The real-time score calculation score of the recording node includes the following: using a callback function for each recording callback; calculating the time interval of audio processing (from the start time to the end time of the recording node) and the average volume; calculating the volume value at the last moment, and when the volume value is lower than the threshold, setting the pitch to 0.

[0241] like Figure 1 As shown, the present invention is a music teaching system, which includes a server and a client. The client includes a teaching end and a learning end. The content of the teaching end is the music score data and the scoring end, and the content of the learning end is the recording data. Students practice through the music score teaching of the teaching end, and find out the gaps based on the scores of the scoring end to adjust their own shortcomings.

[0242] exist Figure 2 In the process, the teaching end first presents the music score data on the big screen. Then, when the recording is turned on, the real-time audio of the students is received and encapsulated through an array. The data is processed on the server and then parsed on the big screen. At this time, the received real-time music score is also presented on the big screen. At the same time, the difference between the music score data and the recording data is compared on the server, and a scoring result is given.

[0243] In summary, by processing the music score, students can better compare the pitch and find out the differences. At the same time, the real-time syllables are given a more accurate score. Based on the score, students can know where they are lacking and can check for omissions in a targeted manner. Students can use rhythm points, pitch points and completeness points to have a preliminary understanding of these three points, the direction of their own deficiencies, and further consolidate their learning or make targeted breakthroughs. The animation mode is used for presentation. When the notes are sung correctly, the note icons will be animated and the pictures of singing correctly and incorrectly will be clearly and intuitively seen. The curve of the note floating is coherent and opens upward, which realizes the upward pull of the real-time note intensity. When it is sung correctly, a highlighted animation will be presented, and students can clearly see the correct singing. Right or wrong; students can practice learning and practicing many times, and get close to the fixed notes again and again, which serves as a good standard; use the audio stream array to include the contents of the recording node, and then use the callback function to call multiple recording nodes, and then get the volume. In the scoring process, the volume that is too low will be ignored, which improves the accuracy of the scoring; this system sets up comparison and scoring, and students can clearly know whether they are wrong or too much, and know the degree of error based on the score, and make real-time corrections; this system can also allow students to find that they are wrong, replay, and re-learn by comparison many times, so as to develop a skill to correct wrong intonation, pitch, etc. from the beginning, which provides a good foundation for students' future musical development.

[0244] In some specific embodiments, Figure 8 As shown, a music operation logic system based on fixed pitch strength is provided, which includes at least one processor and a memory storing instructions for implementing any one of the music operation logic methods based on fixed pitch strength in the above technical solutions.

[0245] In some specific embodiments, a computer-readable storage medium stores a computer program, and the computer program is executed by a processor to implement the steps in the above technical solution.

[0246] In some specific embodiments, a computer program product includes a computer program, and when the computer program is executed by a processor, the method of implementing the steps in the above technical solution is implemented.

[0247] The embodiments and functional operations of the subject matter described in this specification may be implemented in digital electronic circuitry, tangibly implemented computer software or firmware, computer hardware including the structures disclosed in this specification and their structural equivalents, or a combination of more than one of the foregoing. The embodiments of the subject matter described in this specification may be implemented as one or more computer programs, i.e., one or more modules of computer program instructions encoded on one or more tangible, non-transitory program carriers, for execution by, or to control the operation of, a data processing apparatus.

[0248] Alternatively or additionally, the program instructions may be encoded on an artificially generated propagated signal, such as a machine-generated electrical, optical, or electromagnetic signal, which is generated to encode information for transmission to an appropriate receiver device for execution by a data processing device. The computer storage medium may be a machine-readable storage device, a machine-readable storage substrate, a random or serial access memory device, or a combination of one or more of the foregoing.

[0249] A computer program (which may also be referred to or described as a program, software, software application, module, software module, script, or code) may be written in any form of programming language, including compiled or interpreted languages ​​or declarative or procedural languages, and the computer program may be deployed in any form, including as a stand-alone program or as a module, component, subroutine, or other unit suitable for use in a computing environment. A computer program may, but need not, correspond to a file in a file system. A program may be stored as part of a file that stores other programs or data, for example, as one or more scripts in a markup language document, in a single file dedicated to the program in question, or in multiple collaborative files, for example, files storing one or more modules, subroutines, or portions of code. A computer program may be deployed to execute on one or more computers, located in one location or distributed across multiple locations and interconnected by a communications network.

[0250] The contents not described in detail in this specification belong to the prior art known to those skilled in the art.

[0251] Although the present invention has been described in detail above using general explanations, specific embodiments, and experiments, it will be apparent to those skilled in the art that modifications and improvements may be made based on the present invention. Therefore, such modifications and improvements, which do not depart from the spirit of the present invention, are intended to be within the scope of protection claimed herein.

Claims

1. A music operation logic method based on fixed pitch strength, characterized in that: The server side includes the following steps: Receive music score data and use a callback function to receive multiple recording data in real time; wherein, using the callback function to receive multiple recording data in real time includes the following steps: generating an audio stream array based on the multiple recording data; the audio stream array includes a recording start time, a recording end time, a pitch, and a volume; comparing the difference between the recording data and the music score data based on the pitch of the music score data; Based on the differentiation, recording a score for the audio recording data; The recording data record score is sent to the client.

2. The music operation logic method based on fixed pitch strength according to claim 1, characterized in that: The music score data includes a note array; the note array includes notes; the notes include: Pitch, accompaniment playing time and playing duration; among them, Setting the height of the waterfall according to the pitch; Setting the horizontal position of the waterfall notes according to the accompaniment playing time; Setting the length of the waterfall note according to the playing duration; A waterfall is generated based on the height, the horizontal position, and the length.

3. The music operation logic method based on fixed pitch strength according to claim 2, characterized in that: The step of comparing the difference between the recording data and the music score data based on the pitch of the music score data comprises the following steps: Matching the recording start time and the recording end time of the recording data with the accompaniment playing start time + the playing duration of the music score data; Among them, when the recording start time and the recording end time of the recording data have a time intersection with the accompaniment start playback time + the playback duration of the music score data, the notes of the recording data and the note data in the time intersection part are matched.

4. The music operation logic method based on fixed pitch strength according to claim 3, characterized in that: The matching of the recording start time and the recording end time of the recording data with the accompaniment play start time + the play duration of the music score data comprises the following steps: Determine whether the end time of the note of the recording data is after the end time of the current note of the music score data and the current note is not the last note; If so, the note of the recording data matches the next note of the current note of the music score data; If not, the note of the recording data matches the current note of the music score data.

5. The music operation logic method based on fixed pitch strength according to claim 1, characterized in that: The step of recording a score for the recording data based on the differentiation comprises the following steps: Get the audio stream array within a certain period of time; Calculate the pitch score using the first calculation method; or The second operation method is used to calculate the pitch score; wherein, The method of calculating the pitch score using the first operation method includes the following steps: Filter out notes in the audio stream array whose volume is lower than a threshold; Excluding the maximum and minimum pitch values, and the maximum and minimum volume values ​​in the audio stream array; Based on the calculation of the average value of the remaining pitch and volume, the pitch score is obtained; The method of calculating the pitch score using the second operation mode comprises the following steps: Calculate the last pitch score in the audio stream array; The last pitch score is used as the pitch score.

6. The music operation logic method based on fixed pitch strength according to claim 1, characterized in that: The step of recording a score for the recording data based on the differentiation further includes the following steps: Obtaining the pitch of the recording data at regular intervals; Subtract the pitch of the recorded data from the pitch of the music score data to obtain a difference A; Within the error range, subtract the difference A from the pitch of the music score data as a criterion for judging whether the recording data is sung correctly; Receive an adaptation mode signal to start recording data, set the error value to 0, and calculate the appropriate error value for the adapted recording data in real time; Determine whether the pitch of the recording data of the recording node is within the upper and lower error value range of the pitch of the current music score data; If the pitch is within the upper and lower error range of the current score data, the recording data is in the correct singing state, the note is scored, and the rhythm score is obtained.

7. The music operation logic method based on fixed pitch strength according to claim 6, characterized in that: In the judgment of whether the recording data is sung correctly, When the pitch of the recorded data is correct, the pitch ball moves to the height of the pitch of the music score data and the note of the recorded data is highlighted; When the pitch of the recorded data is correct, the pitch ball moves to the height of the pitch of the recorded data.

8. The music operation logic method based on fixed pitch strength according to claim 7, characterized in that: The client includes the following steps: Panning animation: Set the duration of waterfall panning animation according to the duration of the music score; Correct note highlight animation: This includes the waterfall generation style and the timing of the correct note highlight elements, which is determined by the start time of recording when the current recording node and the matching human voice waterfall note reach the target pitch. When singing correctly, the note icon will float in the animation: the curve of the note is continuous and opens upwards.

9. The music operation logic method based on fixed pitch strength according to claim 8, characterized in that: The animation of the note icon floating when singing correctly includes the following steps: The starting point of the floating animation is the height corresponding to the pitch and the time corresponding to the start time of recording when the recording node and the matching adaptive waterfall note reach the standard. The end point is the constant of the pitch. Four quadrants are drawn. When the tangent point of the curve is in the third quadrant, the curve meets the requirements; among them, the coordinate calculation formula of the tangent point of the curve is X = (starting point X-end point X) / 4; Y = (end point Y-starting point Y) / 4.

10. A music operation logic system based on fixed pitch strength, the system comprising at least one processor and a memory storing instructions, characterized in that: When the instruction is executed by at least one processor, a music operation logic method based on fixed pitch strength as described in any one of claims 1 to 9 is implemented.

Images