A method and device for realizing a virtual performance partner

Abstract

The present application discloses a method for realizing a virtual performance partner, including: collecting audio frame data of a performer's performance in units of audio frames; for each current audio frame data collected, converting the current audio frame data into a current digital score, and matching it with a digital score in a specified range in a performance repertoire, and determining a matching digital score in the specified range that matches the current digital score; locating the position of the matching digital score in the performance repertoire, and determining the start time of the performance partner for the cooperative part of the music in the next measure of the matching digital score in the performance repertoire; determining the performance error between the performer and the performance partner according to the performance time of the current digital score and the performance time of the matching digital score, and adjusting the playing speed of the cooperative part of the performance repertoire by the performance partner according to the performance error. The application can make the music playback adapt to the performance progress of the performer and improve the performance experience of the performer.

Description

Technical Field

[0001] The present application relates to audio processing technology, and in particular to a method and device for realizing a virtual performance partner. Background Art

[0002] With the increasing quality of education in China, there are more and more music lovers. Instrumental performance is a very important part of music skills. In a normal instrumental performance, there will be a performer and a collaborator to play a piece of music. The performer plays a leading role in the music cooperation, and the collaborator needs to follow the performer to play. For example, in a violin performance, there is usually a performer playing the violin, and a symphony orchestra as a collaborator, playing with the violinist.

[0003] In traditional instrument playing practice, performers usually follow CDs to play. However, due to the limited level of the performers, or the music played may be a difficult and virtuosic work, the performers may not be able to keep up with the artist's recording in the CD, making the current playing practice experience poor. Summary of the invention

[0004] The present application provides a method and device for realizing a virtual performance partner, which can make music playback adapt to the performance progress of the performer and improve the performance experience of the performer.

[0005] To achieve the above objectives, this application adopts the following technical solutions:

[0006] A method for realizing a virtual performance partner, comprising:

[0007] Taking audio frames as units, collecting audio frame data played by the performer;

[0008] For each current audio frame data collected, perform the following processing:

[0009] Converting the current audio frame data into a current digital music score, and matching it with a digital music score in a specified range in the performance repertoire, and determining a matching digital music score in the digital music score in the specified range that matches the current digital music score;

[0010] Locating the position of the matching digital music score in the performance repertoire, and determining the start time of the performance partner's playing of the cooperative part of the music in the next measure of the matching digital music score in the performance repertoire;

[0011] The performance error between the performer and the performance partner is determined according to the performance time of the current digital music score and the performance time of the matching digital music score, and the playing speed of the cooperative part of the performance piece by the performance partner is adjusted according to the performance error.

[0012] Preferably, the method further comprises: determining the designated range for the next audio frame to be matched according to the position where the current digital music score is located.

[0013] Preferably, the step of determining the start time of playing the next measure of the music in the matching digital music score by the playing partner comprises:

[0014] Determine the performance speed of the performer according to the located position of the matching digital music score and the located positions of the matching digital music score corresponding to the first N audio frames of the current audio frame, and use the performance speed as the reference playing speed of the performance piece;

[0015] Based on the reference playing speed, the playing start time of the next measure of the music in the matching digital music score by the playing partner is determined.

[0016] Preferably, adjusting the playing speed of the playing partner for the playing music according to the playing error comprises:

[0017] When the playing error is less than one beat, the playing speed is adjusted according to the playing error in the current measure on the basis of the reference playing speed, so that the playing partner and the performer finish playing the current measure of the music at the same time;

[0018] When the performance error is greater than one beat, the performance partner pauses playing in the current measure and plays the next measure of the music according to the playing time of the next measure of the music.

[0019] Preferably, the method further comprises:

[0020] When the performance music includes repeated sections, an input set performance section is received and the performance section is used as the designated range.

[0021] Preferably, the method further comprises:

[0022] When the performance piece starts with the performance of the performance partner, the performance partner plays the part of the performance piece before the performance of the performer at a set playing speed.

[0023] Preferably, the method further comprises:

[0024] When the performance piece switches from the solo part of the performer to the performance part of the performance partner, if the performance speed of the performer changes, the performance partner starts playing the performance piece at the performance speed at the end of the solo part; otherwise, the performance partner starts playing the performance piece at the set playback speed.

[0025] Preferably, when the current digital music score fails to be matched successfully within a first set time, the performance partner ends the playing of the performance piece.

[0026] Preferably, converting the current audio frame data into a current digital music score comprises:

[0027] The current audio frame data is processed using a pre-trained neural network model, and a current digital music score corresponding to the current audio frame data is output.

[0028] Preferably, the digital music score is represented by a binary saliency map, and the neural network model is trained using a binary cross entropy loss function.

[0029] Preferably, the matching operation is implemented using NPU.

[0030] Preferably, the method further comprises: outputting the music score of the corresponding performance piece and the position determined by the positioning.

[0031] Preferably, the method further comprises: determining a current scene according to the position determined by the positioning, and corresponding to the current scene, synthesizing a virtual performance animation corresponding to the current scene using a virtual image pre-selected by the performer.

[0032] Preferably, when there are multiple performing users,

[0033] The performer is a preset performing user; if the matching is not successful within the preset second time, the performer is switched to the next preset performing user.

[0034] Preferably, when there are multiple performing users, the virtual image pre-selected by each user is saved;

[0035] When displaying a virtual performance animation, a virtual performance animation synthesized using a virtual image pre-selected by the current performer is displayed; when the performer switches, the virtual performance animation is switched to a virtual performance animation synthesized using a virtual image pre-selected by the switched performer; alternatively, virtual images pre-selected by all performing users are displayed synchronously, and the desired virtual performance animation is synthesized.

[0036] Preferably, the step of synthesizing a virtual performance animation corresponding to the current scene using a virtual image pre-selected by the performer includes:

[0037] Pre-setting an animation switching position in the performance piece, and when the playing progress of the performance piece by the performance partner reaches the animation switching position, changing the virtual performance animation; and / or,

[0038] If the current digital music score is not matched successfully and / or the performance error corresponding to the current digital music score is greater than a set threshold, the performer's preset virtual image is transformed into a preset action, and the virtual performance animation is synthesized.

[0039] Preferably, the animation switching position is set according to the input of the performing user, or the animation switching position is included in the performing repertoire.

[0040] Preferably, the animation switching position is: the switching position between different instruments in the cooperative part of the performance music;

[0041] The transforming of the virtual performance animation includes: corresponding to the switching positions between the different musical instruments, displaying the virtual performance animation corresponding to the performance settings of the musical instruments after switching.

[0042] A device for realizing a virtual performance partner, comprising: a collection unit, a music score recognition and matching unit, a music score positioning unit and a rhythm tracking unit;

[0043] The collecting unit is used to collect the audio frame data played by the performer in units of audio frames;

[0044] The music score recognition and matching unit is used to convert each current audio frame data collected into a current digital music score, match it with a digital music score in a specified range in the performance repertoire, and determine a matching digital music score in the digital music score in the specified range that matches the current digital music score;

[0045] The music score positioning unit is used to locate the position of the matching digital music score in the performance track for each current audio frame data collected, and determine the start time of the playing of the cooperative part of the music in the next measure of the matching digital music score by the performance partner in the performance track;

[0046] The rhythm tracking unit is used to determine the performance error between the performer and the performance partner according to the performance time of the current digital music score and the performance time of the matching digital music score for each current audio frame data collected, and adjust the playing speed of the cooperative part of the performance piece by the performance partner according to the performance error.

[0047] As can be seen from the above technical solution, in this application, the current audio frame data is converted into the current digital music score, and matched with the digital music score in the specified range in the performance repertoire, and the matching digital music score in the digital music score in the specified range that matches the current digital music score is determined. According to the performance time of the current digital music score and the performance time of the matching digital music score, the performance error between the performer and the performance partner is determined, and the performance speed of the performance repertoire by the performance partner is adjusted or the playback of the current measure is stopped according to the performance error; according to the matching digital music score that matches the current audio frame data and its first N audio frame data, the playback time of the next measure of the music of the current digital music score by the performance partner is determined. In the above manner, the performance partner can track and play according to the audio played by the performer, especially track the rhythm, so that the music playback adapts to the performance progress of the performer and improves the performance experience of the performer. BRIEF DESCRIPTION OF THE DRAWINGS

[0048] Figure 1 A basic flow chart of the method for implementing a virtual playing partner in this application;

[0049] Figure 2 This is a schematic diagram of the system architecture in the embodiment of this application;

[0050] Figure 3 is a schematic diagram of a specific flow chart of the method for implementing a virtual playing partner in this embodiment;

[0051] Figure 4 This is a schematic diagram of the training of the neural network model;

[0052] Figure 5 This is a basic structural diagram of a device for implementing a virtual playing partner in this application. DETAILED DESCRIPTION

[0053] In order to make the objectives, technical means and advantages of the present application more clearly understood, the present application is further described in detail below in conjunction with the accompanying drawings.

[0054] In musical instrument performance, the collaborator needs to follow the performer in playing. At the same time, music is sensitive to rhythm, and a slight delay can cause auditory discomfort. Based on this, the present application provides a method and device for realizing a virtual performance partner, which can adaptively adjust the content and speed of the performance music played in the player according to the performer's audio, and especially adjust the playback according to the performer's rhythm.

[0055] Specifically, a performance partner is a virtual device that controls a player to play music, and a performer is a user who uses a performance partner to accompany his or her own musical instrument performance. The performer performs the actual performance of a certain music A, and the specified part of the corresponding music A is played in the player, which is usually the accompaniment part in the music A played by a certain musical instrument. The entire music played is referred to as a performance track below, and the complete score of the performance track and the corresponding audio file played by the performance partner can be saved in advance, such as saved in a server. The performance partner obtains the score of the performance track saved and the audio file played by the performance partner according to the request of the performance user for a certain performance track, and the complete score includes the score of the performer's performance part and the score corresponding to the audio file played by the performance partner (hereinafter referred to as the score of the cooperation part). In addition, in order to provide more selectivity, corresponding to a performance track, multiple audio files played by the performance partner can be saved, such as audio files corresponding to different musical instruments, so that different versions of the same performance track can be provided for the performance user.

[0056] Figure 1 Schematic diagram of the basic flow of the method for implementing the virtual performance partner in this application. Figure 1 As shown, the method includes:

[0057] Step 101, collecting audio frame data played by a performer in units of audio frames.

[0058] For each current audio frame data collected in step 101, the following processing of steps 102-104 is performed:

[0059] Step 102: convert the current audio frame data into the current digital music score, and match it with the digital music score in a specified range in the performance repertoire, and determine the matching digital music score in the digital music score in the specified range that matches the current digital music score.

[0060] This step is used to perform music score recognition and music score matching operations. First, the collected audio frame data is converted into a digital music score, and the digital music score converted from the current audio frame data is called the current digital music score. The audio data of the performance piece is converted into a digital music score in advance, and the converted current digital music score is matched with the digital music score of the entire performance piece or a specified range in the performance piece, and the part of the digital music score that is successfully matched in the performance piece is called the matching digital music score. It can be seen that each current audio frame data is converted into the current digital music score, and the corresponding matching digital music score is determined. Among them, when matching the digital music score, the digital music score of the entire performance piece that is matched with the current digital music score should be the music score of the performer's performance part.

[0061] Step 103, locate the position of the matching digital music score in the performance track, and determine the start time of playing the cooperative part of the music in the next measure of the matching digital music score in the performance track by the performance partner.

[0062] This step is used to locate the score. For the matching digital score determined in step 102, the position of the matching digital score in the entire performance track is determined, that is, the position of the part currently played by the performer in the entire music. Using this position information, it is determined when the performance partner starts playing the music content of the next measure at the located position. The specific method for determining the time to start playing the next measure will be described in detail in the following embodiments.

[0063] At the same time, if the performance partner supports displaying the music score of the performance, the position of the positioning can also be indicated in the music score of the performance according to the positioning position.

[0064] Step 104, determining the performance error between the performer and the performance partner based on the performance time of the current digital music score and the performance time of the matching digital music score, and adjusting the playing speed of the performance partner for the cooperative part of the performance piece based on the performance error.

[0065] This step is used to perform the process of rhythm tracking. Specifically, the error between the player's performance and the playing of the performance partner is determined, so as to perform rhythm tracking to adjust the playing speed of the performance partner for the performance track.

[0066] At this point, the most basic virtual performance partner implementation method process of this application is completed. Through the above process, the audio played by the performer can be identified, and the position of the performance part in the music can be located, so that the performer can control the playback progress of the player in each measure. At the same time, the performance error between the performer and the performance partner is used to track the rhythm and further adjust the playback speed of the player.

[0067] like Figure 1 The process shown needs to process steps 102-104 for each audio frame, so the processing speed is required to be relatively fast. Preferably, hardware such as NPU can be used as support to implement the above method. At the same time, the performance partner analyzes the audio in real time and matches the music score, which belongs to a high-energy computing scenario. Deployment on a desktop device can ignore power supply and other issues. In addition, the ensemble with musical instruments requires a considerable volume. Therefore, preferably, the television can be used as a suitable deployment platform to implement the specific method of this application.

[0068] The following is a specific example to illustrate the specific implementation of the virtual performance partner implementation method in this application. In this embodiment, a television is selected as the deployment platform of the corresponding method. The entire system architecture is as follows: Figure 2 As shown, it includes a television, a microphone connected to the television, and a speaker connected to the television. Figure 3 FIG. 4 is a schematic diagram of a specific flow chart of the method for implementing a virtual playing partner in this embodiment. The flow chart takes the processing of an audio frame as an example for explanation. Figure 3 As shown, the specific processing of an audio frame includes:

[0069] Step 301, obtaining the current audio frame data of the performer.

[0070] The processing of this step can be performed in various existing ways, such as plugging a microphone into the TV to collect the audio of the performer.

[0071] Step 302: using a pre-trained neural network model, convert the current audio frame data into the current digital music score.

[0072] This step is used to perform music score recognition, that is, converting the audio collected by the microphone into a digital music score that can be processed later. The digital music score can be represented by various existing methods, such as a binary saliency map. Specifically, the pitch of the music ranges from large letter group A to small letter group C, with a total of 88 key values ​​at intervals of semitones. Based on this, the music score can be represented as a two-dimensional matrix, with the X-axis as the time coordinate and the Y-axis as the pitch coordinate, so as to generate a binary saliency map.

[0073] In this step, the current audio frame data is input into the trained neural network model, and the model outputs the current digital music score corresponding to the current audio frame data after processing. Figure 3 Before starting the process shown, the neural network model for converting audio data and digital music scores needs to be trained in advance. The following is a brief introduction to the training of the neural network model.

[0074] The neural network model input is the audio frame data collected by the microphone, and the output is the corresponding digital music score, which is specifically a binary saliency map in this embodiment. Considering the particularity of musical instrument performance and the difference in timbre of different musical instruments, preferably, a corresponding neural network model can be trained for each musical instrument.

[0075] For the training of neural networks, training data can be prepared in advance, specifically including a series of audio data (collected from the process of corresponding musical instruments playing music) and digital music scores corresponding to the audio data. The digital music scores corresponding to the audio data are obtained in the following way: the music scores corresponding to the known audio data A, that is, the scores of the performance content, such as five-line music scores, are represented in the form of digital music scores, for example, the corresponding scores are represented by binary saliency maps, and the digital music scores correspond to the audio data A. In the training of neural networks, if Figure 4As shown, the audio data and its corresponding digital music score constitute paired training data, the audio data is input into the neural network model, the model output is obtained, the output is compared with the digital music score corresponding to the audio data, the loss function is calculated, and the model parameters are updated accordingly, and then the next training audio data is input into the model for processing until the model training is completed. In this embodiment, the digital music score is represented by a binary saliency map. Since the binary saliency map is a binary classification label, preferably, a binary classification cross entropy loss function can be used to train the neural network model.

[0076] Step 303, matching the current digital music score with the digital music score in the specified range in the performance repertoire, and determining a matching digital music score in the specified range that matches the current digital music score.

[0077] This step is used for music score matching. For music score matching, it is necessary to obtain the digital music score of the performance, which can be pre-stored in the database. Compare the current digital music score with the complete digital music score of the performance or the specified partial digital music score, perform window search, find out the part most similar to the current digital music score, which is called matching digital music score. Wherein, as mentioned above, the digital music score of the performance for comparison can be the complete digital music score or the specified partial digital music score, where the specified partial digital music score can be user-specified, or it can be that after the processing of several audio frames, the content of the current performance has been locked in a certain area of ​​the performance, and the area is used as the specified range. Of course, when matching the digital music score, the digital music score of the performer's performance part in the performance should be selected to match with the current digital music score.

[0078] The matching can be implemented by using various existing search and matching algorithms, or by pre-training a network model. In addition, in order to speed up the operation and share the CPU occupancy rate, the above matching process can be performed by the NPU.

[0079] Step 304, locate the position of the matching digital music score in the performance track, and determine the start time of the performance partner's cooperative part of the music in the next measure of the matching digital music score in the performance track.

[0080] As mentioned above, the realization of the virtual playing partner is actually to control the player (the TV in this embodiment) to play the set playing content. The set playing content can be pre-set by the user, such as the cello accompaniment audio of a certain song.

[0081] This step first locates the score. Determine the position of the matching digital score matched in step 303 in the performance repertoire, which is also the part currently played by the performer. According to the located position of the matching digital score and the located position of the matching digital score corresponding to the first N audio frame data of the current audio frame data, calculate the performer's performance speed, that is, the average performance speed within N+1 audio frame time, and use this performance speed as the benchmark playback speed for the player to play the performance repertoire. Next, based on the benchmark playback speed, determine the start playback time of the next measure of the matching digital score in the performance repertoire. In other words, based on the above-mentioned benchmark playback speed (i.e. the average performance speed of the performer), calculate the start playback time of the next measure of the measure where the current audio frame is located, and the player uses this start playback time as the start playback time of the next measure of the music, so that at the starting position of the next measure, the performer's performance and the player's playback are synchronized.

[0082] In addition, as mentioned above, when performing score matching, matching can be performed within a specified range of the music. Preferably, the specific position of the specified range in the processing of the next audio frame data can be determined according to the positioning result of this step.

[0083] The above processing of music score positioning is the most basic processing method, which is called random positioning method here. It is used for processing ordinary audio frames. Specifically, it can include the first time a performer enters the music, and the performer starts playing from any position. On this basis, it can further include music theory positioning method, which refers to processing and playing the cooperation part according to the information marked in the music score. The information marked in the music score can be, for example, the music starts from the solo part, the free speed of the performance makes it difficult to track, the work contains repeated passages, the music starts from the orchestra, and the music changes from the solo to the orchestra.

[0084] For the above random positioning method and music theory positioning method, corresponding processing can be used according to different situations to determine the start time of the next section of the cooperative music. The above determination method is referred to as a random positioning algorithm. The processing allocation of random positioning and music theory positioning can be carried out as follows:

[0085] When the performer first enters the music, the performer starts playing from any position, the music starts from a solo part, or the free-speed playing section makes it difficult to track, the above-mentioned random positioning algorithm can be used to determine the base playing speed and the start playing time of the next measure;

[0086] When the performance repertoire contains repeated sections, the input set performance section is received, and the performance section is used as the designated range, and a random positioning algorithm is executed to determine the base playing speed and the start playing time of the next measure;

[0087] When the performance begins with the performance of the performance partner, the performance partner plays the part of the performance before the performance of the performer at the set playback speed; wherein the set playback speed may be a default playback speed;

[0088] When the performance piece changes from the solo part of the performer to the performance part of the performance partner, if the performance speed of the performer changes, the performance partner starts playing the performance piece according to the performance speed at the end of the solo part; otherwise, the performance partner starts playing the performance piece according to the set playback speed.

[0089] Step 305, determining the performance error between the performer and the performance partner according to the performance time of the current digital music score and the performance time of the matching digital music score, and adjusting the playing speed of the performance partner for the cooperative part of the performance piece according to the performance error.

[0090] This step is used to track the rhythm and adjust the actual playback speed within the measure.

[0091] First, the performance error of the performer and the partner needs to be determined. As mentioned above, the digital score includes the pitch and time of each note in the score, and this time is called the performance time of the digital score. The performance error is the difference between the performance time of the current digital score and the performance time of the matching digital score.

[0092] The method of adjusting the playing speed according to the playing error may specifically include: when the playing error is less than one beat, based on the reference playing speed determined in step 305, adjusting the playing speed in the current measure according to the playing error, so that the playing partner and the performer end the playing of the current measure of the music at the same time; thus, the playing speed can be adjusted in the current measure to catch up with the performer's playing speed in the current measure, and in coordination with the processing in step 304, the start time of the next measure can be ensured to be consistent;

[0093] When the performance error is greater than one beat, the partner pauses playing in the current measure and plays the next measure of the music according to the playing time of the next measure. Since the rhythm asynchrony is easily detected, if the performance error is too large, the partner's performance is paused in the current measure, that is, the player's playback is paused, and the cooperative part of the next measure of the score is played from the next measure, which is also coordinated with the processing of step 304.

[0094] In addition, if the current digital music score is not successfully matched within a first set time (eg, 5 seconds), the performance partner ends the playing of the performance piece.

[0095] If the performer skips a measure and pauses the performance for less than the first set time, the performance of the partner is paused in the current measure, that is, the player is paused, and the cooperative part of the next measure of the score is played again from the next measure, which is also coordinated with the processing of step 304;

[0096] If the performance ends or is interrupted (i.e., the corresponding audio frame data cannot be collected), the performance partner ends the playback of the performance piece.

[0097] At this point, the method flow in this embodiment ends.

[0098] Based on the above method, since the TV also has a display function, the following processing may be further included to enrich the user experience:

[0099] 1. Display and output the performance score and the information of the current performance position; here, according to the positioning result of step 304, the current performance position located can be displayed in real time;

[0100] 2. Allow users to select user virtual images through settings, and display the synthesized virtual performance animation in real time according to the music score positioning results; specifically, the user's virtual image can include static images and dynamic images. The static image refers to the virtual character's portrait, clothing, decorations, musical instruments, stage scenery and other fixed materials. The dynamic image refers to the animation movements synthesized in real time on the TV side when the user plays, such as character movements, camera movements, etc.; the preset animation content can be displayed according to different scenes determined by the music score positioning.

[0101] Specifically, the animation switching position can be set in advance in the performance repertoire, and when the playing progress of the performance partner reaches a certain animation switching position, the displayed virtual performance animation is changed. The content of the changed virtual performance animation can be pre-designed. For example, the animation switching position can be set as: the switching position between different instruments in the cooperative part of the performance repertoire; then accordingly, after playing to the animation switching position (that is, the switching position between different instruments), the virtual performance animation pre-set corresponding to the performance of the switched instrument is displayed. The animation switching position can be set according to the input of the performing user before the performance begins, or the animation switching position can also be included in the performance repertoire and set when the file is initially created.

[0102] Alternatively, when it is detected that the volume of the device where the playing partner is located changes, the movement amplitude of each virtual image in the virtual animation can be changed according to the change in volume.

[0103] Alternatively, when the current digital music score fails to match and / or there is a problem with the performer's rhythm (for example, the performance error corresponding to the current digital music score is greater than a set threshold), the performer's pre-set virtual image can be transformed into pre-set actions and synthesized into a corresponding virtual performance animation, so that when the digital music score fails to match and / or there is a problem with the rhythm, the performer's corresponding animated character can have a corresponding performance.

[0104] An example of corresponding virtual performance animations displayed in different scenes is given below, as shown in Table 1.

[0105] Scenario Animation (dynamic image) 1. Waiting for the show The performers took their positions and waved. 2. Band Scene 1 Camera showing all musicians 3. Band Scene 2 The camera shows the band from all angles 4. Soloist prepares to enter The soloist and the orchestra exchanged glances to indicate that they were ready. 5. Soloist plays Focus on the soloist 6. New voices enter the music The camera focuses on the person for 2-3 seconds 7. Volume changes The amplitude of the performance is adjusted according to the volume 8. Change of rhythm The person with the largest lens focus change for 2-3 seconds 9. End of the performance The band puts down their instruments in greeting, and the animation ends 10. Interruption of performance The band grabs the instrument and waits to continue, and ends the animation after 5 seconds

[0106] Table 1

[0107] In addition, the above-mentioned entire performance partner processing is performed for a specific performer. In fact, in the specific implementation, single-user and multi-user situations can also be set. This is called performance tracking, which specifically includes single-user tracking and multi-user tracking. In the single-user scenario, the user completes the basic settings and starts to play, and the performance partner always follows the set user; in the multi-user scenario, assuming that there are users A, B and C, according to conventional music cooperation, the three users should play synchronously. At this time, the performance partner follows the set user A to play the cooperation part. If the rhythm tracking part loses user A for more than a second set time (for example, 2 seconds), the tracking target will be switched to another user, and so on.

[0108] When there are multiple performing users, each user can preselect a corresponding virtual image. When displaying the virtual performance animation, only the virtual image corresponding to the current performer (that is, the user that the performance partner is following) can be displayed, and the corresponding virtual performance animation can be synthesized. When the performer switches, the virtual performance animation is switched to the virtual image corresponding to the switched performer, and the corresponding virtual performance animation can be synthesized. Alternatively, when displaying the virtual performance animation, the virtual images of all performers can also be synchronously displayed, and the corresponding virtual performance animation can be synthesized.

[0109] The above is a specific implementation of the method for realizing a virtual performance partner in this application. Through the above processing of this application, the performance partner can track and play the audio played by the performer, especially the rhythm, so that the music playback adapts to the performance progress of the performer and improves the performance experience of the performer.

[0110] The present application also provides a device for implementing a virtual performance partner, such as Figure 5 As shown, the device includes: a collection unit, a music score recognition and matching unit, a music score positioning unit and a rhythm tracking unit;

[0111] The collecting unit is used to collect the audio frame data played by the performer in units of audio frames;

[0112] The music score recognition and matching unit is used to convert each current audio frame data collected into a current digital music score, match it with a digital music score in a specified range in the performance repertoire, and determine a matching digital music score in the digital music score in the specified range that matches the current digital music score;

[0113] The music score positioning unit is used to locate the position of the matching digital music score in the performance track for each current audio frame data collected, and determine the start time of the playing of the cooperative part of the music in the next measure of the matching digital music score by the performance partner in the performance track;

[0114] The rhythm tracking unit is used to determine the performance error between the performer and the performance partner according to the performance time of the current digital music score and the performance time of the matching digital music score for each current audio frame data collected, and adjust the playing speed of the cooperative part of the performance piece by the performance partner according to the performance error.

[0115] By using the above method and device of the present application, a virtual playing partner can be realized. A small example is given below:

[0116] The system settings are shown in Table 2. Under the system settings shown in this example, the user can select the work to be performed through the settings, and set the voice part, the instrument used by the performer, and the performer's virtual animation image; the TV uses the cloud service to obtain the digital score corresponding to the user-set work from the score library according to the user settings, and obtains the neural network model of the user-set instrument from the sound library for audio to digital score conversion; the TV collects the audio data generated by the performer playing the selected instrument through the connected microphone; the TV performs score recognition, converts the collected audio data into digital score, locates the current music position after matching, and plays the cooperative part of the set music synchronously with the performer's performance; the TV synthesizes the virtual performance animation in real time according to the located position for output, and outputs the score and the located position in the score.

[0117]

[0118] Table 2

[0119] The above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. Any modifications, equivalent substitutions, improvements, etc. made within the spirit and principles of the present invention should be included in the scope of protection of the present invention.

Claims

1. A method for realizing a virtual performance partner, It is characterized in that include: Taking audio frames as units, collecting audio frame data played by the performer; For each current audio frame data collected, perform the following processing: Converting the current audio frame data into a current digital music score, and matching it with a digital music score in a specified range in the performance repertoire, and determining a matching digital music score in the digital music score in the specified range that matches the current digital music score; Locating the position of the matching digital music score in the performance piece, and determining the performance speed of the performer according to the located position of the matching digital music score and the located positions of the matching digital music score corresponding to the first N audio frames of the current audio frame data, and using the performance speed as the reference playing speed of the performance piece; Based on the reference playing speed, determining the starting time for the playing partner to play the cooperative part of the music in the next measure of the matching digital music score in the playing piece; The performance error between the performer and the performance partner is determined according to the performance time of the current digital music score and the performance time of the matching digital music score, and the playing speed of the cooperative part of the performance piece by the performance partner is adjusted according to the performance error.

2. The method according to claim 1, It is characterized in that The method further includes: determining the designated range for the next audio frame to be matched according to the position where the current digital music score is located.

3. The method according to claim 1, It is characterized in that The step of adjusting the playing speed of the performance partner for the performance piece according to the performance error comprises: When the playing error is less than one beat, the playing speed is adjusted according to the playing error in the current measure on the basis of the reference playing speed, so that the playing partner and the performer finish playing the current measure of the music at the same time; When the performance error is greater than one beat, the performance partner pauses playing in the current measure and plays the next measure of the music according to the playing time of the next measure of the music.

4. The method according to claim 1, It is characterized in that The method further comprises: When the performance music includes repeated sections, an input set performance section is received and the performance section is used as the designated range.

5. The method according to claim 1, It is characterized in that The method further comprises: When the performance piece starts with the performance of the performance partner, the performance partner plays the part of the performance piece before the performance of the performer at a set playing speed.

6. The method according to claim 1, It is characterized in that The method further comprises: When the performance piece switches from the solo part of the performer to the performance part of the performance partner, if the performance speed of the performer changes, the performance partner starts playing the performance piece at the performance speed at the end of the solo part; otherwise, the performance partner starts playing the performance piece at the set playback speed.

7. The method according to claim 1, It is characterized in that When the current digital music score fails to be matched successfully within the first set time, the performance partner ends the playing of the performance piece.

8. The method according to claim 1, It is characterized in that The converting the current audio frame data into the current digital music score comprises: The current audio frame data is processed using a pre-trained neural network model, and a current digital music score corresponding to the current audio frame data is output.

9. The method according to claim 8, It is characterized in that The digital music score is represented by a binary saliency map, and the neural network model is trained by a binary cross entropy loss function.

10. The method according to claim 1, It is characterized in that The matching operation is implemented using the NPU.

11. The method according to claim 1, It is characterized in that The method further includes: outputting the music score of the corresponding performance piece and the position determined by the positioning.

12. The method according to claim 1, It is characterized in that The method further comprises: determining a current scene according to the position determined by the positioning, and corresponding to the current scene, synthesizing a virtual performance animation corresponding to the current scene using a virtual image pre-selected by the performer.

13. The method according to claim 1, It is characterized in that When there are multiple performing users, The performer is a preset performing user; if the matching is not successful within the preset second time, the performer is switched to the next preset performing user.

14. The method according to claim 12, It is characterized in that When there are multiple performing users, saving the virtual image pre-selected by each user; When displaying a virtual performance animation, a virtual performance animation synthesized using a virtual image pre-selected by the current performer is displayed; when the performer switches, the virtual performance animation is switched to a virtual performance animation synthesized using a virtual image pre-selected by the switched performer; alternatively, virtual images pre-selected by all performing users are displayed synchronously, and the desired virtual performance animation is synthesized.

15. The method according to claim 12, It is characterized in that The step of synthesizing a virtual performance animation corresponding to the current scene using a virtual image pre-selected by a performer includes: Pre-setting an animation switching position in the performance piece, and when the playing progress of the performance piece by the performance partner reaches the animation switching position, changing the virtual performance animation; and / or, If the current digital music score is not matched successfully and / or the performance error corresponding to the current digital music score is greater than a set threshold, the performer's preset virtual image is transformed into a preset action, and the virtual performance animation is synthesized.

16. The method according to claim 15, It is characterized in that The animation switching position is set according to the input of the performing user, or the animation switching position is included in the performing music.

17. The method according to claim 15, It is characterized in that The animation switching position is: the switching position between different instruments in the cooperative part of the performance; The transforming of the virtual performance animation includes: corresponding to the switching positions between the different musical instruments, displaying the virtual performance animation corresponding to the performance settings of the musical instruments after switching.

18. A device for realizing a virtual performance partner, It is characterized in that The device comprises: a collection unit, a music score recognition and matching unit, a music score positioning unit and a rhythm tracking unit; The collecting unit is used to collect the audio frame data played by the performer in units of audio frames; The music score recognition and matching unit is used to convert each current audio frame data collected into a current digital music score, match it with a digital music score in a specified range in the performance repertoire, and determine a matching digital music score in the digital music score in the specified range that matches the current digital music score; The music score positioning unit is used to locate the position of the matching digital music score in the performance track for each current audio frame data collected, and determine the performance speed of the performer according to the located position of the matching digital music score and the located positions of the matching digital music scores corresponding to the first N audio frames of the current audio frame data, and use the performance speed as the reference playback speed of the performance track; based on the reference playback speed, determine the start playback time of the next measure of the music in the performance track of the matching digital music score by the performance partner; The rhythm tracking unit is used to determine the performance error between the performer and the performance partner according to the performance time of the current digital music score and the performance time of the matching digital music score for each current audio frame data collected, and adjust the playing speed of the cooperative part of the performance piece by the performance partner according to the performance error.

Images