A control method and device of a lute robot, the robot, and a medium

Abstract

The application discloses a control method and device of a piano playing robot, the robot and a medium. The method comprises the following steps: generating a first performance hand score and a second performance hand score according to a main melody score and an accompaniment score; determining a first target finger performance sequence and a second target finger performance sequence according to a finger index determined according to a movement cost, a first scaling factor and a second scaling factor; and controlling the robot to play the first target finger performance sequence and the second target finger performance sequence. The application constructs the first performance hand score and the second performance hand score through the main melody score and the accompaniment score, assigns fingers to each note according to the movement cost, reduces the movement time of the mechanical arm playing the notes, and improves the fluency of playing the piano piece. Meanwhile, the starting time of the notes is adjusted through the scaling factor, the time error between the arrival time and the starting time of the notes is reduced, and the fluency of playing the piano piece is further improved.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of robots, in particular to a control method and device of a piano playing robot, a robot and a medium. BACKGROUND

[0002] With the rapid development of the field of artificial intelligence, a large number of robots appear in many practical application scenarios, such as intelligent education robots, emotional communication robots, cooking robots, performance robots, etc. With the increasing progress of the precision of robots, the degree of humanization that robots can achieve is getting higher and higher, and some robots with higher humanization needs have emerged, such as piano playing robots.

[0003] The piano playing robot is mainly composed of two mechanical arms and two hand claws matched with the mechanical arms. The working principle is to control the two hand claws to perform coordinated and consistent performance of a piece of music through precise driving of the two mechanical arms. At present, when a piece of music is played by a robot, the situation of lag often occurs, thereby affecting the fluency of playing the piece of music.

[0004] Therefore, the prior art still needs to be improved and enhanced. SUMMARY

[0005] The technical problem to be solved by the present application is to provide a control method and device of a piano playing robot, a robot and a medium in view of the deficiencies of the prior art.

[0006] In order to solve the above technical problem, the present application provides a control method of a piano playing robot in the first aspect, wherein the control method of the piano playing robot specifically comprises:

[0007] obtaining a main melody score and an accompaniment score of a performance piece of music, and generating a first performance hand score and a second performance hand score according to the main melody score and the accompaniment score, wherein the notes in the first performance hand score and the second performance hand score each carry a note start time and a key value;

[0008] determining a candidate finger index corresponding to the note according to the key value of the note, and selecting a finger index for the note from the candidate finger indexes according to the movement cost of the candidate finger indexes;

[0009] converting the first performance hand score into a first finger performance sequence according to the finger index of the note, and converting the second performance hand score into a second finger performance sequence;

[0010] obtaining a first scaling factor of the first finger performance sequence, and adjusting the note start time of the finger performance action in the first finger performance sequence according to the first scaling factor to obtain a first target finger performance sequence;

[0011] obtaining a second scaling factor of the second finger playing sequence, and adjusting note start times of finger playing actions in the second finger playing sequence according to the second scaling factor to obtain a second target finger playing sequence;

[0012] controlling the piano robot according to the first target finger playing sequence and the second target finger playing sequence.

[0013] The control method of the piano robot, wherein the generating the first and second playing hand scores according to the main melody score and the accompaniment score specifically comprises:

[0014] dividing the main melody score into a left-hand main melody score and a right-hand main melody score according to movement ranges of left and right hand mechanical arms of the robot;

[0015] selecting at least one first main melody score from the left-hand main melody score and the right-hand main melody score;

[0016] generating a first playing hand score according to the first main melody score and the accompaniment score with performance time as a constraint;

[0017] generating a second playing hand score according to a second main melody score that is not selected from the left-hand main melody score and the right-hand main melody score.

[0018] The control method of the piano robot, wherein the determining the candidate finger index corresponding to the note according to the key value of the note, and selecting the finger index for the note from the candidate finger index according to movement costs of the candidate finger index specifically comprises:

[0019] allocating a candidate finger index for the note according to a preset fingering rule and the key value of the note;

[0020] for the note corresponding to one candidate finger index, taking the candidate finger index as the finger index of the note;

[0021] for the note corresponding to at least two candidate finger indexes, obtaining movement costs of the candidate finger indexes, and selecting the finger index for the note from the at least two candidate finger indexes according to the movement costs.

[0022] The control method of the piano robot, wherein the obtaining the movement cost of the candidate finger index specifically comprises:

[0023] obtaining a finger position sequence of a previous note of the note;

[0024] taking the candidate finger index as the finger index of the note, and obtaining a finger position sequence of the note;

[0025] calculate a current movement cost of the candidate finger index according to the finger position sequence of the previous note and the finger position sequence of the note;

[0026] obtain key value of several subsequent notes of the note, and calculate a future movement cost of the candidate finger index according to the key value of the subsequent notes and the finger position sequence of the note;

[0027] calculate a movement cost of the candidate finger index according to the current movement cost and the future movement cost.

[0028] The control method of the piano robot, wherein the obtaining process of the finger position sequence of the note specifically comprises:

[0029] read the key value of the note and the finger index of the note, and take the key value of the note as the finger position corresponding to the finger index of the note;

[0030] for each adjacent finger except the finger corresponding to the finger index of the note, calculate an index difference value between the finger index of the note and the adjacent finger index of the adjacent finger, and determine the adjacent finger position of the finger according to the index difference value and the key value of the note;

[0031] construct the finger position sequence according to the finger position and the adjacent finger position.

[0032] The control method of the piano robot, wherein the obtaining processes of the first scaling factor and the second scaling factor are the same, and the obtaining the first scaling factor of the first finger playing sequence specifically comprises:

[0033] obtain a maximum distance between finger playing actions in the first finger playing sequence and a number of keys corresponding to the maximum distance, and determine a unit movement time according to the maximum distance and the number of keys;

[0034] obtain a finger movement distance between two adjacent finger playing actions in the first finger playing sequence, and determine a movement time between the two adjacent finger playing actions according to the finger movement distance and the unit movement time;

[0035] determine a note interval time between two adjacent finger playing actions according to the note start time;

[0036] calculate the first scaling factor according to the movement time between the two adjacent finger playing actions and the note interval time.

[0037] The control method of the piano robot, wherein the controlling the piano robot according to the first target finger playing sequence and the second target finger playing sequence specifically comprises:

[0038] controlling the first performance hand robot arm to perform a first finger playing action in the first target finger playing sequence, and controlling the second performance hand robot arm to perform a first finger playing action in the second target finger playing sequence;

[0039] when the first finger playing action in the first performance hand robot arm is performed, controlling the first performance hand robot arm to move to a finger position corresponding to a second finger playing action in the first target finger playing sequence, and calculating a first waiting time between a time when the first performance hand robot arm reaches the finger position and the second finger playing action in the first target finger playing sequence;

[0040] when the first finger playing action in the second performance hand robot arm is performed, controlling the second performance hand robot arm to move to a finger position corresponding to a second finger playing action in the second target finger playing sequence, and calculating a second waiting time between a time when the second performance hand robot arm reaches the finger position and the second finger playing action in the second target finger playing sequence;

[0041] if the first waiting time is less than or equal to zero, controlling the first performance hand robot arm to perform the second finger playing action, and if the first waiting time is greater than zero, controlling the first performance hand robot arm to wait for the waiting time and then perform the second finger playing action;

[0042] if the second waiting time is less than or equal to zero, controlling the second performance hand robot arm to perform the second finger playing action, and if the second waiting time is greater than zero, controlling the second performance hand robot arm to wait for the waiting time and then perform the second finger playing action;

[0043] and so on until a last finger playing action in the first target finger playing sequence and a last finger playing action in the second target finger playing sequence, so as to complete the performance of the performance piece.

[0044] The second aspect of the present application provides a control device of a piano playing robot, wherein the control device of the piano playing robot specifically comprises:

[0045] a score generation module, configured to acquire a main melody score and an accompaniment score of a performance piece, and generate a first performance hand score and a second performance hand score according to the main melody score and the accompaniment score, wherein each note in the first performance hand score and the second performance hand score carries a note start time and a key value;

[0046] a selection module, configured to determine a candidate finger index corresponding to the note according to the key value of the note, and select a finger index for the note from the candidate finger index according to a movement cost of the candidate finger index;

[0047] a conversion module configured to convert the first hand score into a first finger performance sequence according to finger indexes of the notes in the first hand score, and convert the second hand score into a second finger performance sequence according to finger indexes of the notes in the second hand score;

[0048] a scaling module configured to obtain a first scaling factor of the first finger performance sequence, and adjust note start times of finger performance actions in the first finger performance sequence according to the first scaling factor to obtain a first target finger performance sequence; obtain a second scaling factor of the second finger performance sequence, and adjust note start times of finger performance actions in the second finger performance sequence according to the second scaling factor to obtain a second target finger performance sequence;

[0049] a control module configured to control the piano playing robot according to the first target finger performance sequence and the second target finger performance sequence.

[0050] A third aspect of the present application provides a computer readable storage medium storing one or more programs, which can be executed by one or more processors to implement the steps in the control method of the piano playing robot.

[0051] A fourth aspect of the present application provides a robot, comprising: a processor and a memory;

[0052] The memory stores a computer readable program executable by the processor;

[0053] The processor implements the steps in the control method of the piano playing robot when executing the computer readable program.

[0054] Beneficial effects: compared with the prior art, the present application provides a control method, device, robot and medium of a piano playing robot, the method comprises: acquiring a main melody score and an accompaniment score of a playing music piece, and generating a first playing hand score and a second playing hand score according to the main melody score and the accompaniment score; determining a candidate finger index corresponding to the note according to the key value of the note, and selecting a finger index for the note in the candidate finger index according to the movement cost of the candidate finger index; converting the first playing hand score into a first finger playing sequence and converting the second playing hand score into a second finger playing sequence according to the finger index of the note; acquiring a first scaling factor of the first finger playing sequence, and adjusting the note start time of the finger playing action in the first finger playing sequence according to the first scaling factor to obtain a first target finger playing sequence; acquiring a second scaling factor of the second finger playing sequence, and adjusting the note start time of the finger playing action in the second finger playing sequence according to the second scaling factor to obtain a second target finger playing sequence; and controlling the piano playing robot according to the first target finger playing sequence and the second target finger playing sequence. The present application constructs a first playing hand score and a second playing hand score through a main melody score and an accompaniment score, and then assigns a finger to each note according to the movement cost to reduce the movement time of the mechanical arm playing between notes and improve the playing fluency of the piano music piece. Meanwhile, the present application also adjusts the note start time through a scaling factor, reduces the time error between the arrival time and the note start time, and further improves the playing fluency of the piano music piece. BRIEF DESCRIPTION OF DRAWINGS

[0055] In order to more clearly illustrate the technical solutions in the embodiments of the present application, the drawings needed in the embodiment description will be briefly introduced. Obviously, the drawings in the following description are only some embodiments of the present application, and other drawings can also be obtained by those skilled in the art without creative labor.

[0056] Figure 1 The flow chart of the control method of the piano playing robot provided in the embodiments of the present application.

[0057] Figure 2 The flow chart of the acquisition process of the main melody score and the accompaniment score.

[0058] Figure 3 The flow chart of the acquisition process of the score.

[0059] Figure 4 The flow chart of the acquisition process of the first playing hand score and the second playing hand score.

[0060] Figure 5A flowchart of a process for selection of a finger index.

[0061] Figure 6 A flowchart of a process for obtaining a movement cost.

[0062] Figure 7 A flowchart of a process for obtaining a first scaling factor.

[0063] Figure 8 A principle block diagram of a control device of a plucking robot provided by an embodiment of the present application.

[0064] Figure 9 A principle block diagram of a robot provided by an embodiment of the present application. DETAILED DESCRIPTION

[0065] The present application provides a control method and device of a plucking robot, a robot and a medium. In order to make the purpose, technical solutions and effects of the present application clearer and more explicit, the present application is further described in detail below with reference to the drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present application and not to limit the present application.

[0066] Those skilled in the art can understand that, unless specifically stated otherwise, the singular forms "a", "an" and "the" used herein also include the plural forms. It should be further understood that the phrase "comprising" used in the specification of the present application means that the features, integers, steps, plucks, elements and / or components exist, but does not exclude the presence or addition of one or more other features, integers, steps, plucks, elements, components and / or groups thereof. It should be understood that when we say that an element is "connected" or "coupled" to another element, it can be directly connected or coupled to the other element, or there can be intermediate elements. In addition, "connected" or "coupled" used herein can include wireless connection or wireless coupling. The phrase "and / or" used herein includes all or any single unit and all combinations of the associated listed items.

[0067] Those skilled in the art can understand that, unless otherwise defined, all terms (including technical terms and scientific terms) used herein have the same meaning as that generally understood by those skilled in the art to which the present application belongs. It should also be understood that terms such as those defined in a general dictionary should be understood to have meanings consistent with those in the context of the prior art, and should not be interpreted in an idealized or overly formal sense unless specifically defined as such.

[0068] It should be understood that the sequence numbers and sizes of the steps in the embodiments do not mean the order of execution, and the execution order of the processes is determined by their functions and inherent logic, and should not constitute any limitation on the implementation process of the embodiments of the present application.

[0069] The application content will be further explained below through description of embodiments in conjunction with the accompanying drawings.

[0070] This embodiment provides a control method for a piano-playing robot. Figure 1 As shown, the method specifically includes:

[0071] S10. Obtain the main melody score and the accompaniment score of the performance piece, and generate a first performance hand score and a second performance hand score according to the main melody score and the accompaniment score, wherein the notes in the first performance hand score and the second performance hand score both carry the note start time and the key position value.

[0072] Specifically, the piano piece to be played is a piano piece to be performed by the robot. The piano piece can be pre-installed in the robot, provided to the robot by a user, or sent to the robot from an external device. The piece to be played can include the piece name, the piece name, and a music score file. The main melody score includes performance information for the main melody of the piece to be played, and the accompaniment score includes performance information for the background music of the piece to be played.

[0073] In one embodiment, Figure 2 As shown, the method of obtaining the main melody score and accompaniment score of the song to be played specifically includes:

[0074] S11, receiving a song to be played interactively by a user, and obtaining a MIDI file of the song to be played;

[0075] S12, converting the audio track in the MIDI file into a music score;

[0076] S13, classifying the converted music scores according to the main melody track and the accompaniment melody track to obtain the main melody music score and the accompaniment music score.

[0077] In step S11, the user interaction can be to interact with the robot through text or semantic means to select the song to be played. For example, the user inputs the song to be played through the keyboard, the microphone picks up the user's voice and converts the user's voice into text information to obtain the song to be played, or the user inputs the song to be played by triggering the song selection button.

[0078] The MIDI file of the to-be-performed musical piece can be obtained through the Internet or searched in a local database, where MIDI (Musical Instrument Digital Interface) is a digital music interface standard used to transmit music data between electronic musical instruments, computers and other music devices. The MIDI file is composed of two parts: a header chunk Header Chunk (MThd) and a track chunk Track Chunk (MTrk). MThd is the file header of the MIDI format file, which contains basic information of the MIDI file such as format type (Format Type), number of instrument tracks (Number of Tracks) and resolution (Time Division). MTrk is the track information in the MIDI format file, which includes the note information of each note, such as note start time, note end time, pitch value and dynamic value.

[0079] In step S12, the score is generated according to the MIDI file, which includes the performance information of each note in the to-be-performed musical piece, that is, after obtaining the MIDI file, the track information of the score is extracted according to the score storage rule of the MIDI format, the performance information of the notes is determined according to the extracted track information, and then the performance information of each note is sorted to obtain the score. Based on this, in one implementation manner, as shown in Figure 3 The conversion of the track in the MIDI file into the score specifically includes:

[0080] S121, extracting the note start time, note end time, pitch value and dynamic value of the note from the MIDI file;

[0081] S122, decomposing the pitch value of each note into a basic note and an octave number, determining the note name of the note according to the basic note, and merging the note name and the octave number to obtain the standard note symbol of the note;

[0082] S123, calculating the note duration of the note according to the note start time and the note end time;

[0083] S124, combining the standard note symbol, the note start time and the note duration into performance information;

[0084] S125, sorting the performance information according to the note start time to obtain the score.

[0085] In step S121, the to-be-played music piece can correspond to multiple audio tracks, so that the MIDI file can include multiple audio track blocks, each of which corresponds to an audio track and is used to store audio track information of the audio track. To this end, when extracting the note start time, note end time, pitch value and dynamic value of a note from the MIDI file, the note start time, note end time, pitch value and dynamic value of a note in each audio track are extracted respectively, so as to construct a corresponding score for each audio track, so that when the main melody score and the accompaniment score are determined according to the audio tracks, the corresponding score of each audio track can be read from the score, and the division speed of the main melody score and the accompaniment score is improved.

[0086] In step S122, the basic note is a pre-set basic pitch value used to determine the note name of a note, and the octave number is used to represent the offset of the note relative to the basic note, which is determined according to the pitch range of the pitch value and the basic pitch value of the note. That is, a note sequence of a pre-set basic note containing multiple consecutive pitch values is obtained, the basic note corresponding to the note is selected in the basic note sequence, and then the octave number of the note is calculated according to the pitch value and the pitch range corresponding to the note sequence. Wherein, the calculation formula of the basic note and the octave number can be:

[0087] ,

[0088] ,

[0089] Wherein, represents the basic pitch value of the basic note, represents the pitch range of the pre-set basic pitch value sequence, represents the pitch value, represents the octave number, represents the modulo operation, represents the floor function.

[0090] Further, the standard note symbol includes the note name of the basic note and the octave number, and the note in the MIDI file can be converted into the note in the score by the standard note symbol, so that each note of the converted score adopts the standard note symbol, wherein the note name of the basic note is selected according to the basic note in the pre-set corresponding relationship between the basic note and the note name.

[0091] For example, the pre-set basic note sequence is {0, 1, …, 11}, the pitch range is 12, the corresponding relationship between the basic note and the note name is , the pitch value of the note is 60, the basic note corresponding to the note = 60 mod 12 = 0, the octave number = , the note name corresponding to the basic note is , and the standard note symbol corresponding to the note is .

[0092] In step S123 and S124, the note duration refers to the duration of the note, which is the time difference between the note end time and the note start time, i.e. the note duration , represents the note end time, represents the note start time. After obtaining the note duration and the standard note symbol, the data set composed of the sharp note symbol, the note start time and the note duration is taken as the performance information.

[0093] In step S125, the note start time is used to represent the time when the note starts to be played, that is, according to the note start time, the performance order of each note can be determined, that is, the performance order of the note with earlier note start time is before the note with later note start time. Therefore, after obtaining the performance information corresponding to each note, the performance information of each note can be sorted in ascending order of the note start time to obtain the score.

[0094] The above completes the description of step S12, and the following describes step S13.

[0095] In step S13, after obtaining the score, the score can be divided into the main melody score and the accompaniment score according to the track to which each performance information corresponding note belongs. That is, after generating the score, the main melody track and the accompaniment melody track of the to-be-played score are read, and then all performance information belonging to the main melody track is selected in the score to obtain the main melody score, and all performance information belonging to the accompaniment melody track is selected in the score to obtain the accompaniment melody score. Of course, in actual application, when converting the MIDI file into the score, the conversion is performed in units of tracks, so that the score includes several scores corresponding to each track (i.e. the score is a set of track-to-score), and then the score corresponding to the main melody track can be directly read to obtain the main melody score, and the score corresponding to the accompaniment melody track can be directly read to obtain the accompaniment melody score, thereby improving the division speed of the main melody score and the accompaniment melody score.

[0096] In one embodiment, as shown in Figure 4 , the generating the first performance hand score and the second performance hand score according to the main melody score and the accompaniment score specifically includes:

[0097] S101, dividing the main melody score into a left-hand main melody score and a right-hand main melody score according to the motion range of the left and right hand mechanical arms of the robot;

[0098] S102, selecting at least one first main melody score from the left-hand main melody score and the right-hand main melody score;

[0099] S103, generating a first performance hand score according to the first main melody score and the accompaniment score as a constraint of performance time; generating a second performance hand score according to a second main melody score which is not selected from the left hand main melody score and the right hand main melody score.

[0100] In step S101, the movement range of the left and right hand mechanical arms of the robot refers to the movement range of the end of the left and right hand mechanical arms of the robot, wherein the process of obtaining the movement range of the left and right hand mechanical arms can be obtaining the coordinate transformation matrix of the first joint in the left and right mechanical arms relative to the base coordinate system, modeling the left / right mechanical arm forward kinematics by coordinate system transformation, traversing all joints in the left / right mechanical arm as a constraint of joint rotation range, obtaining the movement range of the end of the left / right mechanical arm of the robot, and taking the movement range as the movement range of the left and right hand mechanical arms. Of course, in actual application, the movement range of the left and right hand mechanical arms of the robot can also be pre-set, which is not limited here.

[0101] After obtaining the movement range of the left and right hand mechanical arms, the piano keys can be divided into a left arm performance area and a right arm performance area according to the movement range of the left and right hand mechanical arms. The left arm performance area is a piano key area that can be covered by the mechanical arm corresponding to the first main melody score, and the right arm performance area is a piano key area that can be covered by the right mechanical arm. Then, the main melody score is divided into a left hand main melody score and a right hand main melody score. The left hand main melody score includes notes in the main melody score located in the left arm performance area, and the right hand main melody score includes notes in the main melody score located in the right arm performance area.

[0102] In step S102, the first main melody score can be the left hand main melody score, the right hand main melody score, or both the left hand main melody score and the right hand main melody score. That is, the present application can use one mechanical hand to play the main melody score and the other mechanical hand to assist in playing the main melody score and the accompaniment score, or both mechanical hands can cooperate to play the main melody score and the accompaniment score.

[0103] In step S103, the first performance hand score includes the main melody notes in the first main melody score and the accompaniment melody notes in the accompaniment score, and the second performance hand score includes the main melody notes in the second main melody score. When both the left hand main melody score and the right hand main melody score are selected as the first main melody score, there is no second main melody score that is not selected, so the step of generating the second performance hand score according to the second main melody score that is not selected is not performed, and the first performance hand score is generated according to each first main melody score and the accompaniment score, that is, two first performance hand scores are generated according to two first main melody scores, and then one of the two first performance hand scores is selected as the second performance hand score, so as to control the robot to play the to-be-played piece according to the first performance hand score and the second performance hand score.

[0104] Further, the playing time is used to avoid the conflict between the accompaniment notes and the main melody notes, that is, under the condition of ensuring the integrity of the main melody, the playing time is used as a constraint to filter the accompaniment notes in the accompaniment score which do not conflict with the main melody notes in the first main melody score, and the filtered accompaniment notes are added to the first playing hand score to update the first playing hand score, so as to ensure the smoothness of playing.

[0105] Based on this, in one embodiment, the first playing hand score is generated according to the first main melody score and the accompaniment score under the constraint of the playing time, specifically comprising:

[0106] obtaining the first playing time of the main melody notes in the first main melody score and the second playing time of the accompaniment notes in the accompaniment score;

[0107] selecting the conflict accompaniment notes from the accompaniment score according to the first playing time and the second playing time;

[0108] deleting the selected conflict accompaniment notes from the accompaniment score to obtain a filtered accompaniment score;

[0109] generating the first playing hand score according to the filtered accompaniment score and the first main melody score.

[0110] Specifically, the first playing time can be the note start time of the main melody notes, or a time period determined according to the note start time and the note duration of the melody notes. The second playing time can be the note start time of the accompaniment notes, or a time period determined according to the note start time and the note duration of the accompaniment notes. When the first playing time is the note start time, the second playing time is the time period; when the first playing time is the time period, the second playing time is the note start time or the time period, so that the first playing time and the second playing time can be more accurately determined according to whether the main melody notes and the accompaniment melody notes have playing time overlap.

[0111] Further, after obtaining the first playing time and the second playing time, the second playing time of each accompaniment note can be matched with the first playing time of each main melody note in the first main melody score to determine whether there is a first playing time in the first main melody score which has overlapping time with the second playing time. When there is a first playing time which has overlapping time with the second playing time, it is determined that the accompaniment note corresponding to the second playing time is a conflict accompaniment note; otherwise, when there is no first playing time which has overlapping time with the second playing time, it is determined that the accompaniment note corresponding to the second playing time is a non-conflict accompaniment note.

[0112] The filtered accompaniment score is an accompaniment score without conflict accompaniment notes, that is, after all the conflict accompaniment notes are obtained, all the conflict accompaniment notes are deleted from the accompaniment score, and the accompaniment score after the deletion is taken as the filtered accompaniment score, and then the first performance hand score is generated according to the filtered accompaniment score and the first main melody score, for example, the filtered accompaniment score and the first main melody score are directly combined as the first performance hand score, or the filtered accompaniment score is secondarily screened according to the movement range of the mechanical arm corresponding to the first main melody score, and then the first performance hand score is generated by combining the filtered accompaniment score after the secondary screening and the first main melody score.

[0113] In one embodiment, the first performance hand score is generated according to the filtered accompaniment score and the first main melody score, and specifically includes:

[0114] obtaining the interval distance between the accompaniment notes in the filtered accompaniment score and the mechanical arm corresponding to the first main melody score;

[0115] selecting the out-of-bound accompaniment notes from the filtered accompaniment score according to the interval distance and the movement range of the mechanical arm corresponding to the first main melody score;

[0116] deleting the selected out-of-bound accompaniment notes from the filtered accompaniment score, and combining the filtered accompaniment score after the deletion with the first main melody score to obtain the first performance hand score.

[0117] Specifically, the interval distance is used to reflect the distance that the mechanical arm needs to move when playing the accompaniment note, that is, when the accompaniment note needs to be played by the mechanical hand corresponding to the mechanical arm corresponding to the first main melody score, the mechanical arm corresponding to the first main melody score needs to move the interval distance. The mechanical arm corresponding to the first main melody score has a movement range, and therefore, after the interval distance is obtained, the interval distance can be compared with the movement range of the mechanical arm corresponding to the first main melody score to detect whether the interval distance is within the movement range of the mechanical arm corresponding to the first main melody score. If yes, it is determined that the accompaniment note is not an out-of-bound accompaniment note, otherwise, if no, it is determined that the accompaniment note is an out-of-bound accompaniment note, and therefore the mechanical hand of the mechanical arm corresponding to the first main melody score cannot play the out-of-bound accompaniment note, and therefore the out-of-bound accompaniment note needs to be deleted from the filtered accompaniment score, and the filtered accompaniment score after the deletion is combined with the first main melody score to obtain the first performance hand score. When the filtered accompaniment score after the deletion is combined with the first main melody score, the accompaniment notes in the filtered accompaniment score after the deletion and the main melody notes in the first main melody score are arranged in ascending order according to the note start time.

[0118] In one embodiment, the acquiring the interval distance between the target key and the mechanical arm corresponding to the first main melody score specifically comprises:

[0119] acquiring a target key corresponding to the accompaniment note in the filtered accompaniment score;

[0120] determining the key coordinates of the target key relative to the mechanical arm corresponding to the first main melody score according to the interval rule of the keys;

[0121] calculating the interval distance between the target key and the mechanical arm corresponding to the first main melody score according to the key coordinates.

[0122] Specifically, the target key refers to the key needed to be played when playing the accompaniment note, and the key coordinates refer to the coordinates of the target key relative to the mechanical arm corresponding to the first main melody score, i.e. the coordinates of the key coordinates in the coordinate system where the mechanical arm corresponding to the first main melody score is located. Wherein, the key coordinates can be expressed as (x, y, z), x represents the x-axis coordinate, y represents the y-axis coordinate, and z represents the z-axis coordinate.

[0123] The y-axis coordinate in the key coordinates is determined according to the interval distance of the keys, and the x-axis coordinate and the z-axis coordinate are the x-axis coordinate and the z-axis coordinate of the mechanical hand when the mechanical hand is placed on the target key. Therefore, when determining the key coordinates of the target key relative to the mechanical arm corresponding to the first main melody score according to the interval rule of the keys, the mechanical hand can be controlled to move above the target key, and then the x-axis coordinate and the z-axis coordinate of the mechanical hand are acquired, and the x-axis coordinate and the z-axis coordinate are taken as the x-axis coordinate and the z-axis coordinate of the target key; then the interval key between the target key and the predicted initial key is acquired, and the y-axis coordinate of the target key is determined according to the interval rule of the keys and the interval key.

[0124] For example, taking the index finger of the mechanical hand as the reference, taking the left playing key in the left arm playing area closest to the right arm playing area as the predicted initial key of the left mechanical hand, and taking the right playing key in the right arm playing area closest to the left arm playing area as the predicted initial key of the right mechanical hand, so that when determining the key coordinates of the target key, the playing area where the target key is located can be determined first, and then the predicted initial key corresponding to the target key is selected, and then the distance between the key midline of every two adjacent keys from the predicted initial key to the target key is determined according to the interval rule of the keys, and then the y-axis coordinate of the target key is obtained by accumulating all the distances, i.e. the y-axis coordinate is the cumulative displacement along the length of the piano keyboard. Wherein, the interval rule of the keys is determined according to the playing key used by the robot (such as piano, electronic piano, etc.), for example, the interval rule of the keys is [9, 15, 15, 9, 11, 11, 9, 15, 11, 11, 15, 9, 11, 11] and the like.

[0125] It should be noted that in order to improve the response efficiency of the to-be-played music piece, the key coordinates of each key can be acquired and stored in advance, and then the key coordinates can be directly called when used. Moreover, after the key coordinates are acquired, the two mechanical arms can be controlled to move to the positions of the keys one by one according to the key coordinates, and if a collision occurs during the movement of the mechanical arms, the key coordinates can be adjusted so that the fingers of the mechanical hand can press the specified keys and the movement process will not cause a collision, further improving the fluency of playing. During the adjustment of the key coordinates, the fingers can also be controlled to press the keys to debug the pressing arc of the fingers and the like.

[0126] S20, determining a candidate finger index corresponding to the note according to the key value of the note, and selecting a finger index for the note from the candidate finger index according to the movement cost of the candidate finger index.

[0127] Specifically, the key value is the key number of the piano, and the key value can be used to determine the piano key in the playing device. For example, the correspondence between the notes and the piano keys can be and the like. The finger index is a unique identifier pre-configured for the fingers of the mechanical hand, each finger of the mechanical hand corresponds to a finger index, and the finger indexes corresponding to the fingers are different from each other. For example, the mechanical hand includes an index finger, a middle finger, a ring finger, and a little finger, the finger index of the index finger is 0, the finger index of the middle finger is 1, the finger index of the ring finger is 2, and the finger index of the little finger is 3, etc. The candidate finger index is an alternative finger index assigned to the note, and the candidate finger index can be one or more (including two or more than two), one of the candidate finger indexes is the finger index used to execute the note.

[0128] In one embodiment, as Figure 5 shown, the determining a candidate finger index corresponding to the note according to the key value of the note, and selecting a finger index for the note from the candidate finger index according to the movement cost of the candidate finger index specifically includes:

[0129] S21, assigning a candidate finger index to the note according to a preset fingering rule and the key value of the note;

[0130] S22, for the note corresponding to one candidate finger index, taking the candidate finger index as the finger index of the note; for the note corresponding to at least two candidate finger indexes, acquiring the movement cost of the candidate finger indexes, and selecting a finger index for the note from the at least two candidate finger indexes according to the movement cost.

[0131] In step S21, the preset fingering rule is a basic principle for determining the finger index corresponding to the note, and the preset fingering rule includes a fingering assignment table and a key type and finger index correspondence relationship. The fingering assignment table includes a finger index corresponding to a key value, and the finger index can be selected according to the key value in the fingering assignment table. The key type and finger index correspondence relationship is used as a supplementary condition for the fingering assignment table, and is used to assign a finger index to a note that does not match a finger index according to the preset fingering rule.

[0132] Based on this, the candidate finger index assigned to the note according to the preset fingering rule and the key value of the note specifically includes:

[0133] Matching the key value of the note with the fingering assignment table;

[0134] For the key value that matches successfully, the matched finger index is taken as the candidate finger index of the note;

[0135] For the key value that fails to match, a candidate finger index is assigned to the note according to the key type and finger index correspondence relationship.

[0136] Specifically, the preset fingering rule includes a fingering assignment table and a key type and finger index correspondence relationship. The fingering assignment table includes a finger index corresponding to a key value, and the finger index can be selected according to the key value in the fingering assignment table. The key type and finger index correspondence relationship is used as a supplementary condition for the fingering assignment table, and is used to assign a finger index to a note that does not match a finger index according to the preset fingering rule. When assigning a candidate finger index to a note according to the preset fingering rule, the finger index corresponding to the note is first matched in the fingering assignment table, and then the key type and finger index correspondence relationship is used to assign a finger index to a note that does not match a finger index according to the preset fingering rule.

[0137] For example, assume that the fingering assignment table is as shown in Table 1, and the key type and finger index correspondence relationship can be that if the key value corresponds to a white key, then the finger index of one of the index finger, middle finger, ring finger, and little finger is taken as the finger index corresponding to the note. Then, the note corresponds to a key value of 0, and the finger index corresponding to the key value 0 is found in the preset fingering rule table to be 3. Therefore, the note can only be played by the ring finger, and the note corresponds to a key value of 18, and the finger index corresponding to the key value 18 is not found in the preset fingering rule table. The key corresponding to the key value 18 is a white key, so the finger index of one of the index finger, middle finger, ring finger, and little finger can be selected as the finger index corresponding to the key value 18, i.e., the note can be played by any one of the index finger, middle finger, ring finger, and little finger.

[0138] Table 1 fingering distribution table

[0139]

[0140] In step S22, the notes in the first and second performance hand sheet music correspond to at least one candidate finger index, i.e., the notes can correspond to one, two or more candidate finger indexes. When the notes correspond to one candidate finger index, the candidate finger index can be directly used as the finger index of the notes. When the notes correspond to two or more candidate finger indexes, one candidate finger index needs to be selected from the two or more candidate finger indexes as the finger index of the notes.

[0141] The movement cost is the basis for selecting the finger index, which is used to represent the playing time cost of the mechanical arm playing the notes. The greater the movement cost, the higher the time cost from the previous note to the note. Conversely, the smaller the movement cost, the lower the time cost from the previous note to the note. For playing the piano sheet music by the robot, the lower the playing time cost of the notes, the higher the execution coherence and the higher the execution efficiency of the robot playing the sheet music. Therefore, when selecting the finger index for the notes from at least two candidate finger indexes based on the movement cost, the minimum movement cost can be used as the basis for selection, i.e., the candidate finger index with the minimum movement cost is selected as the finger index of the notes.

[0142] In one embodiment, as shown in FIG. 2, the movement cost of the candidate finger index specifically includes: Figure 6

[0143] S221, obtaining the finger position sequence of the previous note of the note;

[0144] S222, using the candidate finger index as the finger index of the note, and obtaining the finger position sequence of the note;

[0145] S223, calculating the current movement cost of the candidate finger index according to the finger position sequence of the previous note and the finger position sequence of the note;

[0146] S224, obtaining the key value of the subsequent notes of the note, and calculating the future movement cost of the candidate finger index according to the key value of the subsequent notes and the finger position sequence of the note;

[0147] S225, calculating the movement cost of the candidate finger index according to the current movement cost and the future movement cost.

[0148] ​The previous note of the note refers to a note located in front of the note in the operation sequence and closest to the note. For example, the note is the third note, and the previous note is the second note. The finger position sequence includes finger positions of fingers in the mechanical hand, such as when the mechanical hand includes an index finger, a middle finger, a ring finger, and a little finger, the finger position sequence is {index finger position, middle finger position, ring finger position, and little finger position}. The finger position indicates a key value of the finger when the mechanical hand operates the note, and the interval between adjacent two finger positions in the finger position sequence is the same number of interval keys, such as 1, 2, 3, and the like.

[0149] The finger position sequence of the previous note is calculated and stored when the finger index of the previous note is determined. When the robot stores the finger position sequence corresponding to the previous note, the finger position sequence can be directly stored as a storage object; or the finger position of a preset finger in the finger position sequence and the preset finger position relationship can be stored, and the finger position sequence is restored according to the finger position of the preset finger and the preset finger position relationship when the finger position sequence is used.

[0150] In an embodiment of the present application, the robot stores the finger position of the preset finger and the preset finger position relationship, and the preset finger position relationship includes the number of interval keys between adjacent two finger positions and the extension direction of the finger position, so as to reduce the use of the memory space of the robot. For example, the preset index finger is an index finger, the number of interval keys is 1, and the extension direction is from left to right, and then the finger position sequence of the note is stored as index finger position When the finger position sequence needs to be used, the finger position sequence can be obtained according to the number of interval keys and the extension direction, and the finger position sequence is .

[0151] In step S22, the finger position sequence of the note is determined according to the key value corresponding to the note and the finger index. That is, after the candidate finger index is used as the finger index, the finger position sequence corresponding to the note is obtained when the finger corresponding to the finger index of the mechanical hand is moved to the key value. The finger position sequence can be calculated according to the key value of the note, the finger index of the note, and the preset finger position relationship.

[0152] In one embodiment, the process of obtaining the finger position sequence of the note specifically includes:

[0153] The key value of the note is used as the finger position corresponding to the finger index of the note;

[0154] For each adjacent finger index except the finger index of the note, an index difference value between the adjacent finger index and the finger index of the note is calculated, and a neighboring finger position of the adjacent finger index is determined according to a preset finger position relationship, the index difference value and a key value of the note;

[0155] A finger position sequence is constructed according to the finger position and the neighboring finger position.

[0156] Specifically, the adjacent finger index refers to a finger index of the mechanical hand except the finger index of the note. For example, the finger indexes of the mechanical hand include a forefinger finger index, a middle finger finger index, a ring finger finger index and a little finger finger index, and the finger index of the note corresponds to the middle finger finger index. Then, the adjacent finger indexes include the forefinger finger index, the ring finger finger index and the little finger finger index.

[0157] Further, in order to avoid the finger positions exceeding the range of the piano keys of the operating device due to the too large interval of the finger indexes of the mechanical hand, the interval of the finger indexes of the mechanical hand can be set to 1, or after the index difference value between the adjacent finger index and the finger index of the note is obtained, all the index difference values are reduced by a multiple to make the smallest index difference value among all the index difference values to be 1. For example, the mechanical hand includes a forefinger, a middle finger, a ring finger and a little finger. If the finger index of the forefinger is 0, the finger index of the middle finger is 1, the finger index of the ring finger is 2 and the finger index of the little finger is 3, then the finger position sequence can be directly determined by using the calculated index difference value. If the finger index of the forefinger is 0, the finger index of the middle finger is 2, the finger index of the ring finger is 4 and the finger index of the little finger is 6, then after the index difference value is obtained, all the index difference values can be reduced by 2 times.

[0158] As known from the above, the preset finger position relationship includes the interval key number between the two adjacent finger positions and the expansion direction of the finger position. Then, after the index difference value is obtained, the key value distance value between the neighboring finger position of the adjacent finger index and the key value of the note can be calculated according to the index difference value and the interval key number, and then the positive or negative direction of the key value distance can be determined according to the expansion direction, so that the neighboring finger position of the adjacent finger index can be calculated according to the key value of the note, the key value distance value and the positive or negative direction. Finally, the sequence composed of the neighboring finger position of the adjacent finger index and the finger position of the note is taken as the finger position sequence of the note.

[0159] In one embodiment, the interval key number between the two adjacent finger positions is 1, and the expansion direction is from left to right. Then, the calculation formula of the neighboring finger position of the adjacent finger index can be:

[0160] ,

[0161] wherein, denotes the key value of the note, denotes the adjacent finger index, denotes the finger index of the note.

[0162] For example, when the finger index of the note is the index finger index of the right hand, i.e. , and the key value of the note is 18, then the finger position of the index finger index, i.e. , is 18+2*(0-2)=14, the finger position of the middle finger index, i.e. , is 18+2*(1-2)=16, and the finger position of the ring finger index, i.e. , is 18+2*(3-2)=20, so the finger position sequence of the note is .

[0163] Further, in step S23, after obtaining the finger position sequence of the previous note and the finger position sequence of the note, the moving distance of the mechanical hand from the finger position sequence of the previous note to the finger position sequence of the note can be calculated, and then the moving distance is taken as the current moving cost of the candidate finger index. The calculation formula of the moving distance can be:

[0164] ,

[0165] wherein, denotes the current moving cost, denotes the finger position of the note , denotes the finger position of the previous note .

[0166] Further, in step S24, the several subsequent notes are the several notes located after the note in the operation order and closest to the note. For example, if the several subsequent notes include 3 subsequent notes, and the note is the 3rd note, then the key values of the 4th note, the 5th note and the 6th note are obtained. Then the future moving cost of the candidate finger index is calculated according to the key values of the several subsequent notes and the finger position sequence of the note, and the future moving cost is used to predict the moving distance of the mechanical hand moving to the piano key position of the subsequent note. The calculation formula of the future moving cost is:

[0167] ,

[0168] wherein, denotes the future moving cost, denotes the finger position of the note , denotes the key value of the subsequent note.

[0169] Further, in step S25, the movement cost can be a sum of the current movement cost and the future movement cost, can be a weighted sum of the current movement cost and the future movement cost, can be a mean of the current movement cost and the future movement cost, etc. In the embodiment of the present application, the movement cost is the sum of the current movement cost and the future movement cost, so that the movement cost includes the movement cost of moving the current note from the previous note and the movement cost of the future notes, and the movement cost is comprehensively considered, which can improve the accuracy of selecting the finger index based on the movement cost subsequently.

[0170] S30, converting the first playing hand music score into a first finger playing sequence according to the finger index of the note, and converting the second playing hand music score into a second finger playing sequence.

[0171] Specifically, the first finger playing sequence includes the finger playing action of the note in the first playing hand music score, and the second finger playing sequence includes the finger playing action of the note in the second playing hand music score. The finger playing action includes the finger index, the finger position, the note start time and the note duration. The note start time represents the time when the note starts to be played, and the note duration represents the duration of the note. Wherein, the note start time and the note duration can be obtained when the first playing hand music score and the second playing hand music score are acquired.

[0172] In one embodiment, the converting the first playing hand music score into a first finger playing sequence according to the finger index of the note, and converting the second playing hand music score into a second finger playing sequence specifically includes:

[0173] acquiring a finger position sequence of the note, and selecting a finger position of a preset finger in the finger position sequence of the note;

[0174] associating the finger index of the note and the finger position of the preset finger with the note to obtain the finger playing action of the note;

[0175] converting the first playing hand music score into a first finger playing sequence according to the finger playing action of the note, and converting the second playing hand music score into a second finger playing sequence.

[0176] Specifically, the preset finger is preset, which can be an index finger, a middle finger, a ring finger, or a little finger, etc. The finger position of the preset finger is used to replace the finger position sequence to generate the finger playing action, so as to reduce the data amount included in the finger playing action. The preset finger can be any one of all fingers included in the mechanical hand, for example, the preset finger is an index finger, etc. Meanwhile, after the finger index and the finger position of the note are acquired, the finger index, the finger position, and the note can be used as the finger playing action of the note, and the finger playing action is sorted according to the operation order of the note to convert the first playing hand music score into the first finger playing sequence and convert the second playing hand music score into the second finger playing sequence.

[0177] S40, a first scaling factor of the first finger playing sequence is acquired, and the note start time of the finger playing action in the first finger playing sequence is adjusted according to the first scaling factor to obtain a first target finger playing sequence; a second scaling factor of the second finger playing sequence is acquired, and the note start time of the finger playing action in the second finger playing sequence is adjusted according to the second scaling factor to obtain a second target finger playing sequence.

[0178] Specifically, the first scaling factor is used to adjust the note start time of each note in the first finger playing sequence, and the second scaling factor is used to adjust the note start time of each note in the second finger playing sequence, so as to avoid the note execution delay problem caused by the arrival time of the mechanical arm finger reaching the finger position corresponding to the note being later than the note start time, and the execution jamming problem caused by the arrival time being earlier than the note start time (i.e. the waiting time exceeds a preset waiting time threshold).

[0179] In one embodiment, the acquisition processes of the first scaling factor and the second scaling factor are the same, as shown in Figure 7 The first scaling factor of the first finger playing sequence is acquired, and the note start time of the finger playing action in the first finger playing sequence is adjusted according to the first scaling factor to obtain a first target finger playing sequence; a second scaling factor of the second finger playing sequence is acquired, and the note start time of the finger playing action in the second finger playing sequence is adjusted according to the second scaling factor to obtain a second target finger playing sequence.

[0180] S41, the maximum distance between the finger playing actions in the first finger playing sequence and the number of keys corresponding to the maximum distance are acquired, and the unit moving time is determined according to the maximum distance and the number of keys;

[0181] S42, the finger moving distance between the adjacent two finger playing actions in the first finger playing sequence is acquired, and the moving time between the adjacent two finger playing actions is determined according to the finger moving distance and the unit moving time;

[0182] S43, the note interval time of the adjacent two finger playing actions is determined according to the note start time;

[0183] S44, calculate the first scaling factor according to the moving time between two adjacent finger playing actions and the note interval time.

[0184] In step S41, the maximum distance refers to the maximum finger position distance among all finger position distances corresponding to the first finger playing sequence. That is, for any two finger playing actions in the first finger playing sequence, the finger positions in the two finger playing actions are obtained, and then the distance between the finger positions is calculated to obtain the finger position distance, and finally the maximum finger position distance is selected from all the calculated finger position distances to obtain the maximum distance. The calculation process of the finger position distance can first obtain the key value corresponding to the finger position, obtain the piano keys spanned from one finger position to another finger position according to the key value to obtain the key information between the two finger playing actions, and then calculate the finger position distance according to the interval rule of the piano keys and the key information. The interval rule of the piano keys is pre-set, for example, the interval rule of the piano keys is [9, 15, 15, 9, 11, 11, 9, 15, 11, 11, 15, 9, 11, 11].

[0185] The number of keys corresponding to the maximum distance refers to the number of keys covered from one finger position to another finger position, which includes the piano keys corresponding to the two finger positions. For example, the number of keys covered from one finger position to another finger position is 28, etc. The total time refers to the time required for the robot arm to move the maximum distance at the maximum speed, and the unit moving time is the time required for one key, wherein the unit moving time is equal to the quotient of the total time and the maximum distance. For example, the maximum distance is the distance between 28 piano keys, and then the time required for the robot arm to move 28 keys at the maximum speed is measured as 1.8 seconds, so the total time is 1.8 seconds, and the unit moving time is 1.8 / 28.

[0186] In step S42, the finger moving distance is the finger position distance between the adjacent two finger playing actions, and the moving time between the adjacent two finger playing actions is the time required for the robot arm to move from one finger position to another finger position. Wherein, the moving time between the adjacent two finger playing actions is equal to the product of the unit moving time and the finger moving distance. In addition, it should be noted that the distance here refers to the positive distance, which is obtained by taking the absolute value.

[0187] In step S43, the note interval time is the time difference between the note start times of two notes, that is, the note interval time = the time difference between the note start time of the latter note and the note start time of the former note.

[0188] In step S44, the first scaling factor is calculated according to the moving time and the execution time interval between any two adjacent finger playing actions, or is calculated according to the moving time and the execution time interval between all adjacent finger playing actions. In the embodiment of the present application, the first scaling factor is calculated according to the moving time and the execution time interval between all adjacent finger playing actions, and the obtaining process can be as follows:

[0189] For any two adjacent finger playing actions, a candidate scaling factor corresponding to the two adjacent finger playing actions is calculated according to the moving time and the execution time interval between the two adjacent finger playing actions.

[0190] The maximum candidate scaling factor in all candidate scaling factors is taken as the first scaling factor.

[0191] Specifically, the candidate scaling factor is used to adjust the note start time of the two notes corresponding to the candidate scaling factor, and a candidate scaling factor is calculated for any two adjacent finger playing actions in the first finger playing sequence, that is, N-1 candidate scaling factors are calculated for the first finger playing sequence including N finger playing actions. The candidate scaling factor can be the quotient of the moving time and the execution time interval, that is, candidate scaling factor = moving time / execution time interval.

[0192] Further, after the candidate scaling factor corresponding to any two adjacent finger playing actions is obtained, the maximum candidate scaling factor in all obtained candidate scaling factors is taken as the first scaling factor, so as to avoid the adjustment of the note start time based on the main adjustment factor that the arrival time of the robot finger reaching the finger position corresponding to the note is later than the note start time. Based on this, the expression of the first scaling factor is as follows:

[0193] ,

[0194] wherein, denotes the first scaling factor, denotes the finger playing action in the first finger playing sequence, denotes the first adjacent finger playing action, denotes the first adjacent finger playing action, denotes the moving time of the first adjacent finger playing action, denotes the note time interval of the first adjacent finger playing action, denotes the note start time of the first finger playing action, denotes the note start time of the second finger playing action.a finger position of a finger playing action, a finger position of a finger playing action, a finger position of a finger playing action, a unit moving time.

[0195] It should be noted that in actual applications, other manners can also be adopted to select the first scaling factor, for example, the mean value of all candidate scaling factors is selected as the first scaling factor, so as to take into account the finger position corresponding to the late-arriving note or the finger position corresponding to the early-arriving note, etc.

[0196] Further, after the first scaling factor and the second scaling factor are acquired, the note start time of a finger playing action in the first finger playing sequence is adjusted according to the first scaling factor to obtain a first target finger playing sequence; the note start time of a finger playing action in the second finger playing sequence is adjusted according to the second scaling factor to obtain a second target finger playing sequence. The adjustment processes of the first finger playing sequence and the second finger playing sequence are the same, and here the first target finger playing sequence is taken as an example for description.

[0197] Specifically, adjusting the note start time of a finger playing action in the first finger playing sequence according to the first scaling factor to obtain a first target finger playing sequence specifically includes:

[0198] For a first finger playing action located at the front of the first finger playing sequence, the note start time of the first finger playing action is taken as the target note start time of the first finger playing action;

[0199] For a second finger playing action other than the first finger playing action in the first finger playing sequence, the pre-sequence note start time of a previous finger playing action of the second finger playing action is acquired, and the note start time of the second finger playing action is adjusted according to the pre-sequence note start time and the first scaling factor to obtain the target note start time of the second finger playing action;

[0200] The target note start time of each finger playing action in the first finger playing sequence is adopted to update the note start time of each finger playing action in the first finger playing sequence to obtain a target first finger playing sequence.

[0201] Specifically, since there is no early or late situation for the first finger playing action, the note start time of the first finger playing action is kept unchanged, i.e. the note start time of the first finger playing action is directly taken as the target note start time, for the first finger playing action located at the front of the first finger playing sequence. For the rest of the second finger playing actions, the finger arrival time is earlier or later than the note start time, and the finger arrival time is accumulated with the execution of the first finger playing sequence. Therefore, the application adjusts the note start time of each second finger playing action to reduce the time error between the arrival time and the note start time, and to improve the smoothness of the operation.

[0202] For each second finger playing action, in order to adjust the time error between the arrival time from the previous finger playing action to the second finger playing action and the note start time of the second finger playing action, the note start time of the previous finger playing action of the second finger playing action (i.e. the finger playing action located before the second finger playing action in time sequence and adjacent to the second finger playing action) and the first scaling factor can be used to adjust the note start time of the second finger playing action, i.e. the note time interval between the note start time of the second finger playing action and the note start time of the previous finger playing action can be adjusted by using the execution scaling to adjust the note start time of the second finger playing action.

[0203] In one embodiment, the adjusting the note start time of the second finger playing action according to the previous note start time and the first scaling factor to obtain the target note start time of the second finger playing action specifically comprises:

[0204] calculating the time difference between the note start time of the second finger playing action and the previous note start time;

[0205] calculating the adjustment time according to the first scaling factor and the time difference, and calculating the time sum of the adjustment time and the previous execution time to obtain the target note start time of the second finger playing action.

[0206] Specifically, the adjustment time is obtained by adjusting the time difference by the first scaling factor, for example, the adjustment time is the product of the first scaling factor and the time difference. The target note start time is the time sum of the adjustment time and the previous execution time, i.e. the note time interval between the note start time of the previous finger playing action and the note start time of the second finger playing action. Of course, in actual application, the note start time of the previous finger playing action obtained when adjusting the note start time of the second finger playing action is the target note start time which can be adjusted based on the first scaling factor.

[0207] S50, control the piano robot according to the first target finger playing sequence and the second target finger playing sequence.

[0208] Specifically, the mechanical arm is controlled to be located at an initial position, and then a control thread corresponding to the mechanical arm is started, and the mechanical arm is controlled to perform playing according to the first target finger playing sequence and the second target finger playing sequence through the control thread. That is, after the first target finger playing sequence and the second target finger playing sequence are obtained, the mechanical arm of the robot is controlled to move to perform a playing program according to the first target finger playing sequence and the second target finger playing sequence.

[0209] In one embodiment, the control of the piano robot according to the first target finger playing sequence and the second target finger playing sequence specifically includes:

[0210] controlling the first playing hand mechanical arm to perform a first finger playing action in the first target finger playing sequence, and the second playing hand mechanical arm to perform a first finger playing action in the second target finger playing sequence;

[0211] when the first finger playing action in the first playing hand mechanical arm is performed, controlling the first playing hand mechanical arm to move to a finger position corresponding to a second finger playing action in the first target finger playing sequence, and calculating a first waiting time between the time of reaching and the second finger playing action in the first target finger playing sequence;

[0212] when the first finger playing action in the second playing hand mechanical arm is performed, controlling the second playing hand mechanical arm to move to a finger position corresponding to a second finger playing action in the second target finger playing sequence, and calculating a second waiting time between the time of reaching and the second finger playing action in the second target finger playing sequence;

[0213] if the first waiting time is less than or equal to zero, controlling the first playing hand mechanical arm to perform the second finger playing action, and if the first waiting time is greater than zero, controlling the first playing hand mechanical arm to wait for the waiting time and then perform the second finger playing action;

[0214] if the second waiting time is less than or equal to zero, controlling the second playing hand mechanical arm to perform the second finger playing action, and if the second waiting time is greater than zero, controlling the second playing hand mechanical arm to wait for the waiting time and then perform the second finger playing action;

[0215] and so on until the last finger playing action in the first target finger playing sequence and the last finger playing action in the second target finger playing sequence, so as to complete the performance of the playing program.

[0216] Specifically, the control process of the robot arm to perform the first target finger playing sequence and the second target finger playing sequence is the same, and here the control process of the robot arm to perform the first target finger playing sequence is taken as an example for description.

[0217] The arrival time refers to the time when the robot arm reaches the finger position corresponding to the second finger playing action, which can be calculated according to the moving time of the robot arm from the first finger playing action to the second finger playing action, the execution start time of the first finger playing action, and the action duration of the first finger playing action, wherein the arrival time = the execution start time of the first finger playing action + the moving time + the action duration of the first finger playing action. The moving time can be calculated according to the operation key distance between the first finger playing action and the second finger playing action and the moving speed of the robot arm. For example, the finger position of the first finger playing action is a, the finger position of the second finger playing action is b, and the moving speed of the robot arm is c, then the moving time = (b-a) / c. c.

[0218] It should be noted that since the piano has white keys and black keys, when determining the moving time between the first finger playing action and the second finger playing action, the key types corresponding to the first finger playing action and the second finger playing action are determined. If the key types corresponding to the first finger playing action and the second finger playing action are both white keys or black keys, the moving time of the robot arm is If one of the key types corresponding to the first finger playing action and the second finger playing action is a white key and the other is a black key (i.e. black and white key conversion is required), the moving time of the robot arm is to ensure that the robot arm can move to the next key.

[0219] Further, after obtaining the arrival time, the waiting time between the arrival time and the operation start time can be obtained. If the waiting time is less than or equal to zero, the second finger playing action is executed immediately, and if the waiting time is greater than zero, the second finger playing action is executed after waiting for the waiting time.

[0220] In addition, in actual application, when the robot plays the performance piece, the positional relationship between the robot and the piano can be fixed in advance, and the key coordinates of each key in the piano relative to the robot can be determined according to the positional relationship, which can be represented as (x, y, z), x represents the x-axis coordinate, y represents the y-axis coordinate, and z represents the z-axis coordinate. The y-axis coordinate in the key coordinate is determined according to the key interval distance, and the x-axis coordinate and the z-axis coordinate are the x-axis coordinate and the z-axis coordinate of the robot when the robot is placed on the target key. Therefore, the process of obtaining the key coordinate is to control the robot to move above the target key, then obtain the x-axis coordinate and the z-axis coordinate of the robot, and take the x-axis coordinate and the z-axis coordinate as the x-axis coordinate and the z-axis coordinate of the target key; then obtain the key information of the target key and the initial key, the key information includes the keys crossed from the initial operation key to the target key, and the y-axis coordinate of the target key is determined according to the interval rule of the keys and the key information, so as to obtain the key coordinate. In addition, in order to improve the smoothness of the operation, after obtaining the key coordinate, the two robots can be controlled to move to each key position according to the key coordinate, and if a collision occurs during the movement of the robot, the key coordinate is adjusted so that the robot finger can press the specified key and the movement process will not collide.

[0221] In summary, the embodiment provides a control method of a piano robot, which includes obtaining a main melody score and an accompaniment score of a performance piece, and generating a first performance hand score and a second performance hand score according to the main melody score and the accompaniment score; determining a candidate finger index corresponding to the note according to the key value of the note, and selecting a finger index for the note from the candidate finger index according to the movement cost of the candidate finger index; converting the first performance hand score into a first finger performance sequence and the second performance hand score into a second finger performance sequence according to the finger index of the note; obtaining a first scaling factor of the first finger performance sequence, and adjusting the note start time of the finger performance action in the first finger performance sequence according to the first scaling factor to obtain a first target finger performance sequence; obtaining a second scaling factor of the second finger performance sequence, and adjusting the note start time of the finger performance action in the second finger performance sequence according to the second scaling factor to obtain a second target finger performance sequence; and controlling the piano robot according to the first target finger performance sequence and the second target finger performance sequence. The application constructs a first performance hand score and a second performance hand score through a main melody score and an accompaniment score, then assigns a finger to each note according to the movement cost to reduce the movement time of the robot between notes and improve the smoothness of the piano performance. At the same time, the application also adjusts the note start time through the scaling factor to reduce the time error between the arrival time and the note start time, and further improves the smoothness of the piano performance.

[0222] Based on the control method of the piano playing robot, the embodiment provides a control device of the piano playing robot, as shown in the figure. Figure 8 The control device of the piano playing robot specifically comprises:

[0223] The score generation module 100 is configured to acquire a main melody score and an accompaniment score of a performance piece, and generate a first performance hand score and a second performance hand score according to the main melody score and the accompaniment score, wherein each note in the first performance hand score and the second performance hand score carries a note start time and a key value.

[0224] The selection module 200 is configured to determine a candidate finger index corresponding to the note according to the key value of the note, and select a finger index for the note from the candidate finger indexes according to the movement cost of the candidate finger indexes.

[0225] The conversion module 300 is configured to convert the first performance hand score into a first finger performance sequence according to the finger index of the note, and convert the second performance hand score into a second finger performance sequence.

[0226] The adjustment module 400 is configured to acquire a first scaling factor of the first finger performance sequence, and adjust the note start time of a finger performance action in the first finger performance sequence according to the first scaling factor to obtain a first target finger performance sequence; acquire a second scaling factor of the second finger performance sequence, and adjust the note start time of a finger performance action in the second finger performance sequence according to the second scaling factor to obtain a second target finger performance sequence.

[0227] The control module 500 is configured to control the piano playing robot according to the first target finger performance sequence and the second target finger performance sequence.

[0228] Based on the control method of the piano playing robot, the embodiment provides a computer readable storage medium, which stores one or more programs, and the one or more programs can be executed by one or more processors to implement the steps in the control method of the piano playing robot.

[0229] Based on the control method of the piano playing robot, the embodiment provides a control device of the piano playing robot, as shown in the figure. Figure 9As shown, it comprises at least one processor 21 and a memory 22, and can further comprise a communications interface 23 and a bus 24. Among them, the processor 2, the memory 22 and the communications interface 23 can complete the communication between each other through the bus 24. The communications interface 23 can transmit information. The processor 21 can call the logic instructions in the memory 22 to execute the method in the above-mentioned embodiments.

[0230] In addition, the logic instructions in the memory 22 described above can be realized in the form of a software function unit and sold or used as an independent product, which can be stored in a computer readable storage medium.

[0231] The memory 22 as a kind of computer readable storage medium can be configured to store software programs, computer executable programs, such as program instructions or modules corresponding to the method in the embodiments of the present disclosure. The processor 21 executes the software programs, instructions or modules stored in the memory 22, thereby executing function applications and data processing, that is, realizing the method in the above-mentioned embodiments.

[0232] The memory 22 can include a program storage area and a data storage area, wherein the program storage area can store the application programs required by the plucking system and at least one function; the data storage area can store the data created according to the use of the robot and the like. In addition, the memory 22 can include a high-speed random access memory, and can further include a non-volatile memory. For example, a variety of media that can store program codes, such as U disk, mobile hard disk, read-only memory (Read-Only Memory, ROM), random access memory (Random Access Memory, RAM), magnetic disk or optical disk, etc. can also be a transient storage medium.

[0233] In addition, the specific process of loading and executing the instructions in the above-mentioned storage medium and the robot by the processor has been described in detail in the above-mentioned method, which will not be repeated here.

[0234] Finally, it should be pointed out that: the above embodiments are only used to illustrate the technical solutions of the present application, and not to limit them; although the present application has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that: it can still modify the technical solutions recorded in the foregoing embodiments, or make equivalent replacement to part of the technical features; and these modifications or replacements do not make the essence of the corresponding technical solutions deviate from the spirit and scope of the technical solutions of the embodiments of the present application.

Claims

1. A control method of a plectrum robot characterized by comprising: The control method of the piano playing robot specifically comprises the following steps: Obtaining a main melody score and an accompaniment score of a piece of music to be played, and generating a first playing hand score and a second playing hand score according to the main melody score and the accompaniment score, wherein each note in the first playing hand score and the second playing hand score carries a note start time and a key value; Determining a candidate finger index corresponding to the note according to the key value of the note, and selecting a finger index for the note from the candidate finger index according to a movement cost of the candidate finger index; Converting the first playing hand score into a first finger playing sequence and converting the second playing hand score into a second finger playing sequence according to the finger index of the note; Obtaining a first scaling factor of the first finger playing sequence, and adjusting the note start time of a finger playing action in the first finger playing sequence according to the first scaling factor to obtain a first target finger playing sequence; Obtaining a second scaling factor of the second finger playing sequence, and adjusting the note start time of a finger playing action in the second finger playing sequence according to the second scaling factor to obtain a second target finger playing sequence; Controlling the piano playing robot according to the first target finger playing sequence and the second target finger playing sequence; The first scaling factor is used to adjust the note start time of each note in the first finger playing sequence, and the second scaling factor is used to adjust the note start time of each note in the second finger playing sequence, so as to avoid note execution delay caused by the arrival time of the mechanical arm finger reaching the finger position corresponding to the note being later than the note start time, and execution jam caused by the arrival time being earlier than the note start time.

2. The control method of the plectrum robot according to claim 1, characterized by, The method specifically comprises the following steps: Dividing the main melody score into a left-hand main melody score and a right-hand main melody score according to the movement range of the left and right hand mechanical arms of the robot; Selecting at least one first main melody score from the left-hand main melody score and the right-hand main melody score; Generating a first playing hand score according to the first main melody score and the accompaniment score, with playing time as a constraint; Generating a second playing hand score according to a second main melody score that is not selected from the left-hand main melody score and the right-hand main melody score.

3. The control method of the plectrum robot according to claim 1, characterized by, The method specifically comprises the following steps: Assigning a candidate finger index to the note according to a preset fingering rule and the key value of the note; For a note corresponding to one candidate finger index, taking the candidate finger index as the finger index of the note; For a note corresponding to at least two candidate finger indexes, obtaining a movement cost of the candidate finger indexes, and selecting a finger index for the note from the at least two candidate finger indexes according to the movement cost.

4. The control method of the plectrum robot according to claim 3, characterized by, The method specifically comprises the following steps: obtaining a finger position sequence of a previous note of the note; obtaining a finger position sequence of the note; calculating a current movement cost of the candidate finger index according to the finger position sequence of the previous note and the finger position sequence of the note; obtaining key value of several subsequent notes of the note, and calculating a future movement cost of the candidate finger index according to the key value of the subsequent notes and the finger position sequence of the note; calculating a movement cost of the candidate finger index according to the current movement cost and the future movement cost.

5. The control method of the plectrum robot according to claim 4, characterized by, The obtaining process of the finger position sequence of the note specifically comprises: reading the key value of the note and the finger index of the note, and taking the key value of the note as the finger position corresponding to the finger index of the note; for each adjacent finger index except the finger index of the note, calculating an index difference between the adjacent finger index and the finger index of the note, and determining an adjacent finger position of the adjacent finger index according to a preset finger position relationship, the index difference and the key value of the note; constructing a finger position sequence according to the finger position and the adjacent finger position.

6. The control method of the lute robot according to claim 1, characterized in that, The obtaining processes of the first scaling factor and the second scaling factor are the same, and the obtaining of the first scaling factor of the first finger playing sequence specifically comprises: obtaining a maximum distance between finger playing actions in the first finger playing sequence and a number of keys corresponding to the maximum distance, and determining a unit movement time according to the maximum distance and the number of keys; obtaining a finger movement distance between two adjacent finger playing actions in the first finger playing sequence, and determining a movement time between the two adjacent finger playing actions according to the finger movement distance and the unit movement time; determining a note interval time between two adjacent finger playing actions according to the note start time; calculating a first scaling factor according to the movement time between the two adjacent finger playing actions and the note interval time.

7. The control method of the plectrum robot according to claim 1, characterized by, The control of the piano playing robot according to the first target finger playing sequence and the second target finger playing sequence specifically comprises: controlling the first playing hand robot arm to execute a first finger playing action in the first target finger playing sequence, and controlling the second playing hand robot arm to execute a first finger playing action in the second target finger playing sequence; after the execution of the first finger playing action in the first playing hand robot arm is completed, controlling the first playing hand robot arm to move to a finger position corresponding to a second finger playing action in the first target finger playing sequence, and calculating a first waiting time between the reaching time and the second finger playing action in the first target finger playing sequence; after the execution of the first finger playing action in the second playing hand robot arm is completed, controlling the second playing hand robot arm to move to a finger position corresponding to a second finger playing action in the second target finger playing sequence, and calculating a second waiting time between the reaching time and the second finger playing action in the second target finger playing sequence; If the first waiting time is less than or equal to zero, the first playing hand robot is controlled to perform the second finger playing action; if the first waiting time is greater than zero, the first playing hand robot is controlled to wait for the waiting time and then perform the second finger playing action; If the second waiting time is less than or equal to zero, the second playing hand robot is controlled to perform the second finger playing action; if the second waiting time is greater than zero, the second playing hand robot is controlled to wait for the waiting time and then perform the second finger playing action; And so on until the last finger playing action in the first target finger playing sequence and the last finger playing action in the second target finger playing sequence, so as to complete the performance of the playing piece.

8. A control device of a piano-playing robot characterized by comprising: The control device of the piano playing robot specifically comprises: A score generation module configured to acquire a main melody score and an accompaniment score of a playing piece, and generate a first playing hand score and a second playing hand score according to the main melody score and the accompaniment score, wherein each note in the first playing hand score and the second playing hand score carries a note start time and a key value; A selection module configured to determine a candidate finger index corresponding to the note according to the key value of the note, and select a finger index for the note from the candidate finger index according to a movement cost of the candidate finger index; A conversion module configured to convert the first playing hand score into a first finger playing sequence according to the finger index of the note, and convert the second playing hand score into a second finger playing sequence; An adjustment module configured to acquire a first scaling factor of the first finger playing sequence, and adjust the note start time of a finger playing action in the first finger playing sequence according to the first scaling factor to obtain a first target finger playing sequence; and acquire a second scaling factor of the second finger playing sequence, and adjust the note start time of a finger playing action in the second finger playing sequence according to the second scaling factor to obtain a second target finger playing sequence; A control module configured to control the piano playing robot according to the first target finger playing sequence and the second target finger playing sequence. The first scaling factor is used to adjust the note start time of each note in the first finger playing sequence, and the second scaling factor is used to adjust the note start time of each note in the second finger playing sequence, so as to avoid note execution delay caused by the arrival time of the mechanical arm finger reaching the finger position corresponding to the note being later than the note start time, and execution jam caused by the arrival time being earlier than the note start time.

9. A computer-readable storage medium, characterized in that, The computer readable storage medium stores one or more programs, which can be executed by one or more processors to implement the steps in the control method of the piano playing robot according to any one of claims 1-7.

10. A robot, characterized in that Comprise: A processor and a memory; The memory stores a computer readable program that can be executed by the processor; The processor executes the computer readable program to implement the steps in the control method of the piano playing robot according to any one of claims 1-7.

Images