A music online education demonstration method and system based on AR and artificial intelligence

Abstract

The present invention discloses a music online education demonstration method and system based on AR and artificial intelligence, including: establishing a music score timing database of key states and the strength when the keys are pressed; obtaining the music score timing of the current teaching music score and the time when the keys begin to be pressed; establishing a finger displacement model according to the finger position change trajectory when the keys begin to be pressed under different strengths; obtaining the finger keystroke trajectory when the keys are pressed, and obtaining the key time points according to the finger displacement model; using the key time points to determine the time period from when the key is pressed to when the finger leaves the key; loading the time period from when all keys are pressed to when the fingers leave the key into the corresponding music score timing to obtain new music score data; generating AR special effects according to the new music score data for teaching demonstration. After the finger displacement model is established, there is no need to perform frame-by-frame analysis, which reduces the amount of processor calculations; the generated special effects emphasize the fingering, rhythm and strength of the performance, which helps students learn more intuitively and effectively.

Description

Technical Field

[0001] The present application relates to the field of artificial intelligence, and specifically to an online music education demonstration method and system based on AR and artificial intelligence. Background Art

[0002] Online music education has been welcomed by everyone because it is different from traditional one-on-one teaching and is not limited by time and space. Music education requires a combination of knowledge and skills. In the field of music education, a large proportion of the part focuses on online video teaching. However, due to the limitations of interaction between teachers and students, teachers have no way to correct students through videos during online teaching. On the other hand, during video teaching, students cannot clearly see the teacher's piano playing process, let alone accurately understand the essence of the teacher's teaching.

[0003] The existing technology analyzes the teacher's movements in the video shot by a mobile phone or tablet, and displays AR special effects on the student's mobile phone or tablet according to the teacher's movements. However, this method needs to analyze the video of the teacher's demonstration in advance when it is actually used. In one-on-one teaching, the teacher will provide targeted teaching according to the actual situation of different students. If the teaching video is not prepared in advance, the special effects cannot be generated, which has limitations. Summary of the invention

[0004] In response to the above technical problems, the present invention proposes an online music education method and demonstration system based on AR and artificial intelligence.

[0005] First, this paper proposes a music online education demonstration method based on AR and artificial intelligence, including:

[0006] Establish a music score timing database containing the key states of all music scores and the different strengths of the keys when they are pressed;

[0007] Obtaining the music score timing sequence of the current teaching music score from the music score timing sequence database, and obtaining the moment when the piano key starts to be pressed according to the music score timing sequence of the current teaching music score;

[0008] According to the finger position change trajectory at the moment when the key begins to be pressed under different forces, a finger displacement model is established;

[0009] Obtain the finger keystroke trajectory when the key is pressed during performance, determine the finger displacement model corresponding to the force of pressing the key, and obtain the key time point when the key is pressed according to the finger keystroke trajectory and the corresponding finger displacement model;

[0010] Using the obtained key time point, determine the time period from when the key is pressed to when the finger leaves the key;

[0011] Obtain the time period from when all keys are pressed to when fingers are released during the performance and load it into the corresponding music score sequence to obtain new music score data;

[0012] Based on the new score data, AR special effects are generated when playing according to the score for teaching demonstration.

[0013] Furthermore, the establishment of the finger displacement model includes:

[0014] Acquire the video during the demonstration, and acquire the image during the demonstration frame by frame according to the video during the demonstration;

[0015] Using a segmentation network to obtain the positions of all piano keys in the image during the demonstration;

[0016] Using a neural network to obtain the position coordinates of the fingertip in the image;

[0017] Identifying the finger pressing the key corresponding to the moment when the key begins to be pressed;

[0018] The change of the fingertip position when the key is pressed is standardized, and the finger position change trajectory is obtained according to the change of the standardized fingertip position;

[0019] The finger displacement model of the finger change trajectory under the same force is obtained, and the finger displacement models of all forces are obtained.

[0020] Furthermore, the method for standardizing the change in fingertip position includes:

[0021]

[0022] Where p i+1 refers to the fingertip coordinates of the finger pressing the key in the (i+1)th frame image, v max Refers to the maximum value of the v axis of the piano key in the pixel coordinate system, v min Refers to the minimum value of the v axis of the piano key in the pixel coordinate system, v i Refers to the v-axis value of the fingertip coordinate of the finger pressing the key in the i-th frame image, v i+1 Refers to the v-axis value of the fingertip coordinate of the finger pressing the key in the (i+1)th frame image.

[0023] Furthermore, the method for acquiring the finger displacement model of the finger change trajectory under the same force includes:

[0024] The finger change trajectory at the moment when a key starts to be pressed under a certain force is obtained, and the overlapping part of the finger change trajectories corresponding to different keys and different fingers is the finger displacement model under the force.

[0025] Furthermore, the method for obtaining the key time point includes:

[0026] The standard change vector of the finger at the moment the key is pressed is obtained, the finger displacement model corresponding to the current force is selected, and the key time point is obtained according to the position of the standard change vector of the finger at that moment in the finger displacement model.

[0027] Furthermore, the process of acquiring the finger standard change vector includes:

[0028] A teaching video after the finger displacement model is established is obtained, and a previous frame image at the time when the key is pressed and a subsequent frame image at the time when the key is pressed are extracted according to the teaching video, a neural network is used to obtain the fingertip position in the previous frame image and the fingertip position in the subsequent frame image, a difference vector between the fingertip position in the subsequent frame image and the fingertip position in the previous frame image is calculated, and the difference vector is standardized to obtain a standard finger change vector.

[0029] Furthermore, the AR special effect generation process includes:

[0030] Particles are used to replace fingers for movement, the size of the particles is positively correlated with the force with which the current key is pressed, and the movement of the particles between different keys is presented in the form of a parabola.

[0031] In the second aspect, an online music education demonstration system based on AR and artificial intelligence includes:

[0032] Teacher side, storage side, processing side and student side;

[0033] The teacher terminal is used to record the states of the keys in all the scores and the different strengths when the keys are pressed, obtain the score timing data and send it to the storage terminal; collect the performance soundprint information of the score at the beginning of the performance and send it to the storage terminal; collect the key area video and the video during the performance and send it to the storage terminal;

[0034] The storage end is used to store the data sent by the teacher end and the processing end, and accept the call of the data by the processing end;

[0035] The processing end is used to call the data stored in the storage end;

[0036] Establish a music score time sequence database containing all music scores and send it to the storage end; obtain the music score time sequence of the current teaching music score, and obtain the time when the key starts to be pressed according to the music score time sequence of the current teaching music score;

[0037] A finger displacement model is established according to the finger position change trajectory at the moment when the key begins to be pressed under different forces; the finger pressing trajectory when the key is pressed during performance is obtained, and the finger displacement model corresponding to the force at which the key is pressed is determined, and the key time point when the key is pressed is obtained according to the finger pressing trajectory and the corresponding finger displacement model;

[0038] The obtained key time point is used to determine the time period from when the key is pressed to when the finger leaves the key; the time period from when all keys are pressed to when the finger leaves the key during the performance is obtained and loaded into the corresponding music score sequence to obtain new music score data;

[0039] Generate AR data based on new music score data and send it to the student end;

[0040] The student terminal is used to present AR special effects on the display device according to the processed and sent AR data.

[0041] Compared with the traditional technical solution, the beneficial effects of the present invention are:

[0042] 1. Based on the teacher's finger displacement model, the key time points of the teacher's hand movements are determined, without the need to analyze the video frame by frame, which can reduce the amount of calculation of the processor on the teacher's side;

[0043] 2. The special effects generated on the student side can emphasize the teacher's fingering, rhythm and strength, allowing students to learn more intuitively and effectively. BRIEF DESCRIPTION OF THE DRAWINGS

[0044] In the following, the embodiments of the present invention will be described in more detail by way of example only with reference to the accompanying drawings, in which:

[0045] Figure 1 It is a block diagram of an online music education demonstration method based on AR and artificial intelligence of the present invention.

[0046] Figure 2 It is a schematic diagram of piano key states provided by an embodiment of the present invention.

[0047] Figure 3 It is a block diagram of a process of establishing a finger displacement model provided by an embodiment of the present invention.

[0048] Figure 4 It is a schematic diagram of the superposition of relative positions of different fingers under the same force provided by an embodiment of the present invention.

[0049] Figure 5 Schematic diagram of finger displacement model under different forces in an embodiment of the present invention.

[0050] Figure 6 It is a schematic diagram of determining the critical moment in an embodiment of the present invention.

[0051] Figure 7 It is a block diagram of an online music education demonstration system based on AR and artificial intelligence of the present invention. DETAILED DESCRIPTION

[0052] In order to make the purpose, technical scheme and advantages of the present invention more clearly understood, the present invention is further described in detail below in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present invention and are not intended to limit the present invention. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without making creative work are within the scope of protection of the present invention.

[0053] In the following description, specific details such as specific system structures, technologies, etc. are provided for the purpose of illustration rather than limitation, so as to provide a thorough understanding of the embodiments of the present application. However, it should be clear to those skilled in the art that the present application may also be implemented in other embodiments without these specific details. In other cases, detailed descriptions of well-known systems, devices, circuits, and methods are omitted to prevent unnecessary details from obstructing the description of the present application.

[0054] The terms "first" and "second" are used for descriptive purposes only and should not be understood as indicating or implying relative importance or implicitly indicating the number of the indicated technical features. Therefore, the features defined as "first" and "second" may explicitly or implicitly include one or more of the features; in the description of this embodiment, unless otherwise specified, "plurality" means two or more.

[0055] Figure 1 is a block diagram of a music online education demonstration method based on AR and artificial intelligence in this embodiment, such as Figure 1 The music online education demonstration method based on AR and artificial intelligence includes the following steps:

[0056] Step S101, establishing a music score time series database including the key states of all music scores and the different strengths of the keys when they are pressed, specifically including the following contents:

[0057] 1. Play each piece in the textbook on the electric piano;

[0058] 2. During the performance of each song, record whether different keys in the electric piano are pressed or not pressed at the current moment. If the key is pressed, it is recorded as 1, and if it is not pressed, it is recorded as 0;

[0059] 3. In the timing of each key, mark the intensity of each key when it is pressed (there are 8 types of intensity, very pianissimo ppp, very pianissimo pp, weak p, medium pianissimo mp, medium forte mf, strong f, very strong ff, very forte fff);

[0060] 4. Record the timing of each piano key at different times for each song in the textbook and establish a music score timing database.

[0061] Step S102: obtaining the music score timing of the current teaching music score from the music score timing database, and obtaining the moment when the keys begin to be pressed according to the music score timing of the current teaching music score.

[0062] At the beginning of the demonstration teaching, audio fingerprint recognition is used to determine the current teaching score, the corresponding score timing is obtained in the score timing database, and the moment when the key is pressed is determined according to the rising edge of different keys in the score.

[0063] Figure 2 FIG. 4 shows a schematic diagram of the piano key states in this embodiment. Figure 2 As shown in each song, each key has a corresponding force mark when being pressed, and the entire song is divided into different states. In the figure, key 1 is on the rising edge at time t1, and key 1 is pressed within the time (t1, t2), that is, key 1 starts to be pressed at time t1.

[0064] Step S103: Establish a finger displacement model according to the finger position change trajectory at the moment when the key begins to be pressed under different forces, which specifically includes the following contents:

[0065] The purpose of this step is to obtain the teacher's demonstration video on the teacher's side and obtain the teacher's fingertip position in the selected frame; determine the teacher's finger displacement model by comparing the fingertip positions in each key state change period, and then determine the key moment of playing the current score according to the finger displacement model in subsequent performances.

[0066] In this embodiment, rhythm and dynamics are playing skills that must be mastered for beginner piano learners. At the same time, whether a piece of music is good or not depends largely on whether the teacher can correctly express each beat in the score. However, for piano beginners, rhythm and dynamics are one of the techniques that are difficult to master. The AR special effects generated in this embodiment determine the key position of particles according to the fingertips of the teacher on the teacher side, so as to assist the students on the student side in learning rhythm and dynamics.

[0067] Since the demonstration method provided by the present invention is applied to one-to-one online teaching, its real-time nature needs to be considered, that is, the teacher's actions on the teacher's end must be able to be presented at the student's end with low latency; this embodiment focuses on the actions of the teacher's fingertips when pressing the keys, and establishes a finger displacement model for the teacher's fingertip actions to determine the key moments of the performance; after the modeling is completed, the subsequent key moments do not need to be detected frame by frame, and the finger displacement model can be used to determine the key moments by calculating the finger position in any frame, thereby ensuring the real-time nature of the system solution.

[0068] Figure 3 FIG. 4 is a block diagram showing the process of establishing a finger displacement model in this embodiment. Figure 3 The process of establishing the finger displacement model shown in the figure specifically includes the following contents:

[0069] Step S201, obtaining the video during the demonstration and the segmented video, and obtaining the image of the video during the demonstration frame by frame, specifically includes the following contents:

[0070] The teacher's camera is set diagonally above the key area to ensure that the entire key area can be captured. Video is acquired when the teacher is demonstrating the performance. In the beginning, when the finger displacement model has not been established, images are acquired frame by frame. The camera position is fixed so that the key area only needs to be detected once during each demonstration.

[0071] Step S202: using a segmentation network to obtain the positions of the piano keys in the image, specifically including the following contents:

[0072] In this embodiment, an instance segmentation network is used to select the area of ​​each key in the image, which specifically includes the following contents:

[0073] 1. The DNN used in this embodiment is an Encoder-Decoder structure, and the training set is a piano picture taken from above;

[0074] 2. The labels are divided into two categories: key areas and other areas. In this embodiment, the 88 key areas are marked as 1 to 88, and other non-key areas are marked as zero;

[0075] 3. The loss function adopts the cross entropy function;

[0076] 4. Record the maximum value v of the v axis in the pixel coordinate system for each key area max , minimum value v min .

[0077] Step S203, using a neural network to obtain the position coordinates of the fingertip in the image, specifically includes the following contents:

[0078] According to the segmentation time point of the current teaching score, select the video frames with a total length of T before and after the segmentation time, and detect the teacher's fingertip position in the piano key area in these frames. The size of T can be determined according to the processor performance of the demonstration end, and the empirical value is 0.5 seconds. In this embodiment, the teacher's fingertip position detection network adopts a key point detection network, which is an Encoder-Decoder structure, and the specific training is as follows:

[0079] 1. The dataset uses images taken by a camera diagonally above the piano keys;

[0080] 2. The labels are ten types of key points, i.e., the ten fingertips of a person's hands. The labeling process is: each type of key point corresponds to a single channel, the Diagram position of the pixel corresponding to the key point is marked in the channel, and then Gaussian blur is performed to form a key point hot spot at the marked point. In this embodiment, the label image contains a total of ten channels;

[0081] 3.Loss is the mean square error.

[0082] The fingertip position hot spots obtained by the key point detection network are transformed by soft-argmax to obtain the corresponding fingertip position coordinates in each frame.

[0083] Step S204, identifying the finger pressing the key corresponding to the moment when the key starts to be pressed, specifically includes the following contents:

[0084] In each selected video, get the time corresponding to the second half of the video, determine the state of the keys during this period, and select the area in the image corresponding to each key with state 1. Count the fingertips corresponding to the key points that appear in the selected area in the second half of the video. The finger that appears more times in the specified area of ​​the video presses the key corresponding to this area, and this finger is called the current key pressing finger.

[0085] In this embodiment, the selected video is centered on the moment when the key in the database is pressed. In the first half of the video, the teacher's fingers may just pass through this area without pressing the key; in this embodiment, the teacher's fingers must be pressed on the corresponding key in the second half of the video, causing the key to be pressed.

[0086] Step S205, standardizing the change of the fingertip position when the key is pressed, and obtaining the finger position change trajectory according to the change of the standardized fingertip position, specifically including the following contents:

[0087] 1. Get the fingertip coordinates p of the key in the i-th frame of the selected video i (u i , v i );

[0088] 2. Compare the coordinates in two adjacent frames and calculate the coordinate difference between the two frames to obtain the change vector Δp i ,Right now:

[0089] Δp i =p i+1 -p i ;

[0090] 3. Map the change vector to the end of the key to get the standard change vector, that is:

[0091]

[0092] 4. Superimpose all the standard change vectors in this video in sequence to obtain the finger change trajectory.

[0093] Because the positions where the fingers press on the keys are different, the displacement of the fingers when the keys are pressed to the bottom is also different. This embodiment takes the outer edge of the keys as the standard. Through this step, the displacement of all fingers is mapped to the outer edge of the keys, which facilitates the comparison of the relative positions of the fingers in subsequent steps.

[0094] Step S206, obtaining the finger displacement model of the finger change trajectory under the same force, and obtaining the finger displacement models of all forces, specifically including the following contents:

[0095] The purpose of this step is to extract the force of the key state change in the database, compare the finger change trajectory of the corresponding key in the video near the time point of the key state change with the same force, and generate the finger displacement model of the key state with different forces. The specific steps are as follows:

[0096] 1. Compare the lengths of all the music score segments, i.e. the time intervals, and take the longest time interval x as the delay of the teacher's first teaching, and obtain the finger displacement model in the video frames within x time at the beginning of the teaching;

[0097] 2. Extract the finger changes corresponding to pressing different keys at different times with the same force in the video of x period of time;

[0098] 3. Calculate the longest common subsequence of all finger change trajectories under different forces as the finger movement template under the force.

[0099] Figure 4 FIG. 4 shows a schematic diagram of the relative positions of different fingers under the same force in this embodiment. Figure 4 As shown in FIG. 1 , within the selected 2T time length, the relative positions of different fingers under the same force have overlapping parts and non-overlapping parts. The overlapping k to k′ segments in the displacements of various fingers under the same force are used to obtain the finger displacement model under the force, and then the finger displacement models under all forces are obtained. Figure 5 A schematic diagram of the finger displacement model under different forces in this embodiment is shown. Since there are 8 forces in this embodiment, there are 8 finger displacement models in this embodiment.

[0100] Step S104, determining the key moment of teaching according to the fingertip position at the current moment and the finger displacement model, specifically including the following contents:

[0101] First, after a period of time from the beginning of teaching or after the finger displacement model is determined, the position of the detection frame is determined according to the change of the time series in the database. The specific steps are as follows:

[0102] 1. Extract the time t corresponding to the segmentation point n ;

[0103] 2. Extract the video frame of the teacher playing at time t after the teaching starts;

[0104] 3. Extract the previous frame t at time t after the teaching starts - and the next frame t + ;

[0105] 4. Use the fingertip keypoint detection network to obtain the fingertip position in the extracted frame;

[0106] 5. + Subtract the fingertip position in t from the fingertip position in t to get a vector a1. - Get another vector a2 from the fingertip position in , and take half of the sum of the two vectors as the current finger position change vector a;

[0107] 6. Map the current finger position change vector to the end of the key to obtain the standard finger change vector at time t.

[0108] Because the timing in the database in this embodiment is not necessarily the same as the timing when the music is actually played, the timing in the database is used as a reference, and the specific time point is determined according to the model.

[0109] Step S105, using the obtained key time point to determine the time period from when the key is pressed to when the finger leaves the key, specifically includes the following contents:

[0110] 1. Determine the force of the keys played by your fingers at the current moment;

[0111] 2. Change the current standard change vector to the position of the template under this force;

[0112] 3. Determine the critical moments k and k′ based on the relative position of this template.

[0113] In this embodiment, the current position is compared with the corresponding finger displacement model, and the time t is determined according to the relative position of the current standard change vector in the model under the force. n Relative relationship with critical moments k and k′.

[0114] Figure 6 FIG. 4 shows a schematic diagram of determining the key moment in this embodiment, such as Figure 6 After obtaining the current finger standard change vector a, according to the relative position of a in the finger displacement model, that is, Figure 6 Medium n Relative to the position of p and p', we can get the time of the key moment relative to the current moment, that is, the time of k and k' relative to t n time difference.

[0115] Step S106, obtaining the time period from when all keys are pressed to when fingers are released during the performance and loading it into the corresponding music score sequence to obtain new music score data, specifically including the following contents:

[0116] Because the original music score timing database only contains the key status and the force when the key is pressed, but does not have the specific time of the key being pressed and the specific displacement changes of the fingers during the key being pressed, new music score data is loaded into the existing music score database based on the time period from when all keys are pressed to when the fingers are released to obtain the new music score data. The new music score data includes the key status, the force when the key is pressed, and the time period from when the fingers are pressed to when the keys are released, which provides a basis for the subsequent generation of AR special effects.

[0117] Step S107, generating an AR special effect for playing the music score according to the new music score data for teaching demonstration, specifically including the following contents:

[0118] In this embodiment, each finger has a particle corresponding to it. In this embodiment, the particles are used to reflect the teacher's hand movements, and the movements of the particles always emphasize the teacher's fingering, rhythm and strength. The details are as follows:

[0119] 1. In this embodiment, there are ten particles, each particle represents each finger of the teacher;

[0120] 2. The size of the particle represents the strength of the current finger playing the piano key. In this embodiment, the eight sizes from small to large represent eight strengths from pianissimo to fortissimo;

[0121] 3. The position of the particle represents the position of the finger;

[0122] 4. Each particle is located on the key pressed by the corresponding finger during the time from k to k′. The exact position of the particle during this time is determined according to the finger displacement model;

[0123] 5. The trajectory of each particle from the time k′ when the previous key is pressed to the time k when the next key is pressed is a parabolic trajectory.

[0124] Reference below Figure 7 To describe the system of the present disclosure embodiment, Figure 7 A block diagram of an online music education demonstration system based on AR and artificial intelligence is shown. As shown in the figure, the online music education demonstration system based on AR and artificial intelligence includes the following contents:

[0125] Teacher end 301 , storage end 302 , processing end 303 and student end 304 .

[0126] The teacher terminal 301 is used to record the states of the keys in all the scores and the different strengths when the keys are pressed, obtain the score timing data and send it to the storage terminal 302; collect the performance soundprint information of the score at the beginning of the performance and send it to the storage terminal 302; collect the key area video and the video during the performance and send it to the storage terminal 302;

[0127] The storage end 302 is used to store data sent by the teacher end 301 and the processing end 303, and accept the call of the data by the processing end 303.

[0128] The processing end 303 is used to call the data stored in the storage end;

[0129] Establish a music score time sequence database containing all music scores and send it to the storage end 302; obtain the music score time sequence of the current teaching music score, and obtain the time when the key starts to be pressed according to the music score time sequence of the current teaching music score;

[0130] A finger displacement model is established according to the finger position change trajectory at the moment when the key begins to be pressed under different forces; the finger pressing trajectory when the key is pressed during performance is obtained, and the finger displacement model corresponding to the force at which the key is pressed is determined, and the key time point when the key is pressed is obtained according to the finger pressing trajectory and the corresponding finger displacement model;

[0131] The obtained key time point is used to determine the time period from when the key is pressed to when the finger leaves the key; the time period from when all keys are pressed to when the finger leaves the key during the performance is obtained and loaded into the corresponding music score sequence to obtain new music score data;

[0132] AR data is generated according to the new music score data and sent to the student terminal 304.

[0133] The student terminal 304 is used to present AR special effects on a display device according to the processed and sent AR data.

[0134] To summarize, compared with traditional technical solutions, the present invention determines the key time points of the teacher's hand movements based on the teacher's finger displacement model after the finger displacement model is established. There is no need to analyze the video frame by frame, which reduces the computational complexity of the processor on the teacher's side; the special effects generated on the student side emphasize the teacher's fingering, rhythm and strength, which can help students learn music more intuitively and effectively.

[0135] The words "including", "comprising", "having" and the like in the present invention are open words, meaning "including but not limited to", and can be used interchangeably therewith. The words "or" and "and" used herein mean the words "and / or" and can be used interchangeably therewith, unless the context clearly indicates otherwise. The word "such as" used herein means the phrase "such as but not limited to", and can be used interchangeably therewith.

[0136] It should also be noted that in the method and system of the present invention, each component or each step can be decomposed and / or recombined. Such decomposition and / or recombination should be regarded as equivalent solutions of the present disclosure.

[0137] The above embodiments are merely examples for the purpose of clear explanation and do not constitute a limitation on the protection scope of the present invention. For those skilled in the art, other different forms of changes or modifications can be made based on the above description, and it is not necessary and impossible to list all the implementation methods here. All designs that are the same or similar to the present invention belong to the protection scope of the present invention.

Claims

1. A music online education demonstration method based on AR and artificial intelligence, characterized in that: include: Establish a music score timing database containing the key states of all music scores and the different strengths of the keys when they are pressed; Obtaining the music score timing sequence of the current teaching music score from the music score timing sequence database, and obtaining the moment when the piano key starts to be pressed according to the music score timing sequence of the current teaching music score; According to the finger position change trajectory at the moment when the key begins to be pressed under different forces, a finger displacement model is established; Obtain the finger keystroke trajectory when the key is pressed during performance, determine the finger displacement model corresponding to the force of pressing the key, and obtain the key time point when the key is pressed according to the finger keystroke trajectory and the corresponding finger displacement model; Using the obtained key time point, determine the time period from when the key is pressed to when the finger leaves the key; Obtain the time period from when all keys are pressed to when fingers are released during the performance and load it into the corresponding music score sequence to obtain new music score data; Generate AR special effects based on new music score data for teaching demonstration when playing according to the music score; The establishment of the finger displacement model includes: Acquire the video during the demonstration, and acquire the image during the demonstration frame by frame according to the video during the demonstration; Using a segmentation network to obtain the positions of all piano keys in the image during the demonstration; Using a neural network to obtain the position coordinates of the fingertip in the image; Identifying the finger pressing the key corresponding to the moment when the key begins to be pressed; The change of the fingertip position when the key is pressed is standardized, and the finger position change trajectory is obtained according to the change of the standardized fingertip position; Obtain the finger displacement model of the finger change trajectory under the same force, and obtain the finger displacement model of all forces; The method for obtaining the key time point includes: Obtain the finger standard change vector at the time when the key is pressed, select the finger displacement model corresponding to the current force, and obtain the key time point according to the position of the finger standard change vector at this moment in the finger displacement model; The generation process of the AR special effects includes: Particles are used to replace fingers for movement, the size of the particles is positively correlated with the force with which the current key is pressed, and the movement of the particles between different keys is presented in the form of a parabola.

2. According to claim 1, a music online education demonstration method based on AR and artificial intelligence is characterized in that: The method for standardizing the change of fingertip position when a key is pressed comprises: ； In the formula Refers to The fingertip coordinates of the finger pressing the key in the frame image, Refers to the piano keys in the pixel coordinate system Axis maximum value, Refers to the piano keys in the pixel coordinate system Axis minimum value, Refers to the The coordinates of the fingertip of the finger pressing the key in the frame image Axis value, Refers to the The coordinates of the fingertip of the finger pressing the key in the frame image Axis value.

3. According to claim 1, a music online education demonstration method based on AR and artificial intelligence is characterized in that: The method for acquiring the finger displacement model of the finger change trajectory under the same force includes: The finger change trajectory at the moment when a key starts to be pressed under a certain force is obtained, and the overlapping part of the finger change trajectories corresponding to different keys and different fingers is the finger displacement model under the force.

4. According to claim 1, a music online education demonstration method based on AR and artificial intelligence is characterized in that: The process of acquiring the finger standard change vector includes: A teaching video after the finger displacement model is established is obtained, and a previous frame image at the time when the key is pressed and a subsequent frame image at the time when the key is pressed are extracted according to the teaching video, a neural network is used to obtain the fingertip position in the previous frame image and the fingertip position in the subsequent frame image, a difference vector between the fingertip position in the subsequent frame image and the fingertip position in the previous frame image is calculated, and the difference vector is standardized to obtain a standard finger change vector.

5. A music online education demonstration system based on AR and artificial intelligence, which implements the music online education demonstration method based on AR and artificial intelligence as claimed in claim 1, characterized in that: include: Teacher side, storage side, processing side and student side; The teacher terminal is used to record the states of the keys in all the scores and the different strengths when the keys are pressed, obtain the score timing data and send it to the storage terminal; collect the performance soundprint information of the score at the beginning of the performance and send it to the storage terminal; collect the key area video and the video during the performance and send it to the storage terminal; The storage end is used to store the data sent by the teacher end and the processing end, and accept the call of the data by the processing end; The processing end is used to call the data stored in the storage end; Establish a music score time sequence database containing all music scores and send it to the storage end; obtain the music score time sequence of the current teaching music score, and obtain the time when the key starts to be pressed according to the music score time sequence of the current teaching music score; A finger displacement model is established according to the finger position change trajectory at the moment when the key begins to be pressed under different forces; the finger pressing trajectory when the key is pressed during performance is obtained, and the finger displacement model corresponding to the force at which the key is pressed is determined, and the key time point when the key is pressed is obtained according to the finger pressing trajectory and the corresponding finger displacement model; The obtained key time point is used to determine the time period from when the key is pressed to when the finger leaves the key; the time period from when all keys are pressed to when the finger leaves the key during the performance is obtained and loaded into the corresponding music score sequence to obtain new music score data; Generate AR data based on new music score data and send it to the student end; The student terminal is used to present AR special effects on the display device according to the processed and sent AR data.

Images