Humanoid piano-playing robot

A humanoid piano playing robot with advanced control systems and sensory units enables accurate keyboard positioning and intelligent musical performance, addressing the integration of robots in piano playing.

JP7595192B2Active Publication Date: 2024-12-05ZHEJIANG LAB
View PDF 5 Cites 0 Cited by

Patent Information

Application Number
JP2023568142
Authority / Receiving Office
JP · JP
Patent Type
Patents
Current Assignee / Owner
Priority Date
2023-06-08
Filing Date
2023-09-21
Publication Date
2024-12-05
Estimated Expiration
2043-09-21

AI Technical Summary

Technical Problem

The challenge of effectively applying robots to piano performance has not been adequately addressed, making it difficult to realize their integration into musical activities.

Method used

A humanoid piano playing robot is designed with a left hand, right hand, left arm, right arm, waist, neck, visual sensing unit, left leg, right leg, and control system, featuring two degrees of freedom in the neck and waist for rotational motion, enabling it to acquire musical score images, recognize the piano keyboard, plan fingerings, and control hand movements to play the piano.

Benefits of technology

The robot can accurately position the keyboard, intelligently recognize musical scores, and play the piano with dexterity, mimicking human-like movements through controlled waist and neck rotations and arm positions.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure 0007595192000006
    Figure 0007595192000006
  • Figure 0007595192000007
    Figure 0007595192000007
  • Figure 0007595192000008
    Figure 0007595192000008
Patent Text Reader

Abstract

The present invention discloses a humanoid piano-playing robot, which includes hands, arms, a waist, a neck, a visual sensing unit, legs, a control system, and a base, and the neck is configured with two degrees of freedom to drive the yaw and pitch rotations of the head, and the waist is configured with two degrees of freedom to drive the yaw and pitch rotations of the upper body. The humanoid piano-playing robot can accurately position the keyboard by using the visual sensing unit, intelligently recognize the contents of the musical score, and automatically play the piano according to the musical score. By controlling the two degrees of freedom of the waist and neck, and the positions and orientations of the two robot arms, the piano-playing robot of the present invention can play the piano skillfully and intelligently.
Need to check novelty before this filing date? Find Prior Art

Description

[Technical field]

[0001] The present invention relates to the field of robotics, and in particular to a humanoid piano-playing robot. [Background technology]

[0002] With the advancement of science and technology, robots such as robot vacuum cleaners and medical robots are gradually being applied to people's lives. However, in practical applications, it is difficult to realize the application of robots to piano playing. Therefore, how to effectively apply robots to piano playing has become an urgent issue. Summary of the Invention [Problem to be solved by the invention]

[0003] The present invention provides a humanoid piano-playing robot to solve the above problems of the prior art. [Means for solving the problem]

[0004] The technical solutions used in the present invention are as follows:

[0005] The present invention provides a humanoid piano-playing robot, the humanoid piano-playing robot including a left hand, a right hand, a left arm, a right arm, a waist, a neck, a visual sensing unit, a left leg, a right leg, a control system, and a base, the neck being configured with two degrees of freedom for driving a yaw rotation motion and a pitch rotation motion of a head, the waist being configured with two degrees of freedom for driving a yaw rotation motion and a pitch rotation motion of an upper body, The humanoid piano-playing robot is Using the visual sensing unit, obtain an image of a musical score and an image of a piano keyboard facing the humanoid piano playing robot, recognize the image of the musical score to obtain musical score information, and position the piano keyboard based on the image of the keyboard to obtain positioning data; a control system for determining a fingering pattern to be used by the humanoid piano playing robot when the humanoid piano playing robot plays a piece of music corresponding to the score information based on the positioning data and the score information, and determining a position and posture change sequence corresponding to body parts of the humanoid piano playing robot when the humanoid piano playing robot plays the piece of music according to the fingering; The control system controls the left hand and the right hand according to the fingering, and is used to control body parts other than the left hand and the right hand based on the position and posture change sequence.

[0006] Optionally, the humanoid piano playing robot further comprises: used by the visual sensing unit to encode the score information to obtain encoded information in a predetermined format, and transmit the encoded information to the control system; The musical score information includes at least beats, notes, and ornament symbols.

[0007] Optionally, the humanoid piano playing robot further comprises a sound unit; The humanoid piano playing robot further comprises: The audio unit receives voice information from a user, and based on the voice information, determines whether the user's purpose is to request the humanoid piano playing robot to play the piano.If it is determined that the user's purpose is to request the humanoid piano playing robot to play the piano, the control system uses it to control the humanoid piano playing robot to play the piano.

[0008] Optionally, the humanoid piano playing robot is If the audio unit determines, based on the voice information, that the user's purpose is to request the humanoid piano playing robot to play the piano, it recognizes the voice information and determines song information that the user requests the humanoid piano playing robot to play, and this is used by the control system to control the humanoid piano playing robot to play the piano based on the song information.

[0009] Optionally, the humanoid piano playing robot is The audio unit is used to filter noise in the audio information, and based on the audio information after noise filtering, to determine whether the user's purpose is to request the humanoid piano-playing robot to play the piano.

[0010] Optionally, the humanoid piano playing robot is Based on the positioning data, the control system determines the position and posture of the tip of the arm at each time required for the left arm and / or the right arm to play the piece of music corresponding to the musical score information according to the fingering, and based on the position and posture of the tip of the arm, the control system determines a position and posture change sequence corresponding to the body parts of the humanoid piano playing robot when the humanoid piano playing robot plays the piece of music according to the fingering.

[0011] Optionally, the humanoid piano playing robot is The control system performs inverse kinematics calculations for the position and posture of the tip of each arm in the position and posture change sequence to determine the joint angles required for the left arm or the right arm to reach the position and posture of the tip of that arm, and the joint angles are used to control the movement of the left arm or the right arm based on the joint angles corresponding to the position and posture of the tip of each arm.

[0012] Optionally, the humanoid piano playing robot is The control system determines, for the position and posture of the tip of each arm in the position and posture change sequence, the arm to which the position and posture of the tip of that arm belongs, and transforms the position and posture of the tip of that arm into the shoulder coordinate system of the arm to which the position and posture of the tip of that arm belongs based on the fixed yaw rotation angle between the shoulder and waist corresponding to the arm to which the position and posture of the tip of that arm belongs, the yaw rotation angle and pitch rotation angle of the waist at that time, and the coordinates of the shoulder corresponding to the arm to which the position and posture of the tip of that arm belongs, in the local coordinate system of the pitch joint of the waist, to obtain a transformed position and posture, which is used to control the movement of the left arm and / or the right arm based on the transformed position and posture.

[0013] The present invention provides a control method for a humanoid piano-playing robot, the control method for the humanoid piano-playing robot being applied to a humanoid piano-playing robot, the humanoid piano-playing robot including a left hand, a right hand, a left arm, a right arm, a waist, a neck, a visual sensing unit, a left leg, a right leg, a control system, and a base, the neck being configured with two degrees of freedom for driving the yaw rotational motion and the pitch rotational motion of the head, the waist being configured with two degrees of freedom for driving the yaw rotational motion and the pitch rotational motion of the upper body, the control method for the humanoid piano-playing robot comprising: acquiring an image of a musical score and an image of a piano keyboard facing the humanoid piano-playing robot by the visual sensing unit; recognizing the image of the score by the visual sensing unit to obtain score information, and locating the piano keys according to the image of the keys to obtain positioning data; a step of planning, by the control system, fingering to be used when the humanoid piano playing robot plays a piece of music corresponding to the score information, based on the positioning data and the score information, and determining a position and posture change sequence corresponding to body parts of the humanoid piano playing robot when the humanoid piano playing robot plays the piece of music according to the fingering; The method includes a step of controlling the left hand and the right hand by the control system in accordance with the fingering, and controlling a body part other than the left hand and the right hand based on the position and posture change sequence.

[0014] Optionally, the method for controlling the humanoid piano playing robot further comprises: encoding the score information by the visual sensing unit to obtain encoded information in a predetermined format, and transmitting the encoded information to the control system; The musical score information includes at least beats, notes, and ornament symbols.

[0015] Optionally, the humanoid piano playing robot further comprises a sound unit; The method for controlling the humanoid piano playing robot further comprises: receiving, by said audio unit, audio information from a user; The method includes a step of determining, by the audio unit, based on the audio information, whether or not the user's purpose is to request the humanoid piano playing robot to play the piano, and if it is determined that the user's purpose is to request the humanoid piano playing robot to play the piano, controlling the humanoid piano playing robot by the control system to play the piano.

[0016] Optionally, the step of controlling the humanoid piano-playing robot to play the piano by the control system includes: when it is determined by the audio unit based on the audio information that the user's purpose is to request the humanoid piano playing robot to play the piano, recognizing the audio information and determining song information that the user requests the humanoid piano playing robot to play; and a step of controlling the humanoid piano playing robot to play the piano based on the music information by the control system.

[0017] Optionally, the step of determining a position / posture change sequence corresponding to body parts of the humanoid piano playing robot when the humanoid piano playing robot plays the piece of music according to the fingering includes: determining, based on the positioning data, a position and posture of the tip of the left arm and / or the right arm at each time required for the left arm and / or the right arm to play the piece of music corresponding to the musical score information in accordance with the fingering; and determining, based on the position and posture of the tip of the arm, a position and posture change sequence corresponding to the body parts of the humanoid piano playing robot when the humanoid piano playing robot plays the piece of music according to the fingering.

[0018] The present invention further provides a computer-readable storage medium storing a computer program, which, when executed by a processor, performs the above-mentioned method for controlling the humanoid piano-playing robot.

[0019] The present invention further provides an electronic device including a memory, a processor, and a computer program stored in the memory and executable on the processor, the electronic device implementing the above-mentioned method for controlling the humanoid piano-playing robot when the processor executes the computer program. Effect of the Invention

[0020] At least one of the above technical solutions used in the present invention can achieve the following beneficial effects:

[0021] As can be seen from the above-mentioned humanoid piano-playing robot, the humanoid piano-playing robot may include a left hand, a right hand, a left arm, a right arm, a waist, a neck, a visual sensing unit, a left leg, a right leg, a control system, and a base, wherein the neck is configured with two degrees of freedom to drive the yaw rotational movement and pitch rotational movement of the head, and the waist is configured with two degrees of freedom to drive the yaw rotational movement and pitch rotational movement of the upper body. The humanoid piano playing robot uses a visual sensing unit to obtain an image of musical score and an image of the piano keys that the humanoid piano playing robot faces, recognizes the image of the musical score to obtain musical score information, positions the piano keys based on the image of the keys to obtain positioning data, plans fingering to be used by the humanoid piano playing robot when playing a piece of music corresponding to the musical score information based on the positioning data and the musical score information, determines a position and posture change sequence corresponding to the body parts of the humanoid piano playing robot when the humanoid piano playing robot plays the piece of music according to the fingering, and uses a control system to control the left and right hands in accordance with the fingering and to control body parts other than the left and right hands based on the position and posture change sequence.

[0022] As can be seen from the above, the waist and neck of the humanoid piano-playing robot provided by the present invention are configured with two degrees of freedom, which can flexibly achieve the goal of playing the piano in the same way as a human being, and the visual sensing unit can accurately position the position of the keys, intelligently recognize the contents of the musical score, and automatically play the piano according to the musical score. By controlling the two degrees of freedom of the waist and neck and the positions and orientations of the two robot arms, the piano-playing robot of the present invention can play the piano skillfully and intelligently. [Brief description of the drawings]

[0023] The accompanying drawings described herein are used to enhance understanding of the present invention and constitute a part of the present invention, and the exemplary embodiments of the present invention and their descriptions are used to explain the present invention and do not constitute undue limitations of the present invention.

[0024] [Figure 1]FIG. 1 is a schematic diagram showing a playing system that combines a humanoid piano-playing robot and a piano provided according to an embodiment of the present invention. [Diagram 2] FIG. 1 is a schematic diagram showing the structure of a humanoid piano-playing robot provided according to an embodiment of the present invention. [Diagram 3] FIG. 2 is a schematic diagram showing the coordinated movement of the waist and neck of the humanoid piano-playing robot provided by an embodiment of the present invention. [Figure 4] FIG. 2 is a system block diagram illustrating a visual sensing unit provided by an embodiment of the present invention. [Diagram 5] FIG. 2 is a system block diagram illustrating a speech unit provided by an embodiment of the present invention. [Figure 6] FIG. 2 is a schematic diagram illustrating a world coordinate system and a local coordinate system corresponding to the left arm provided by an embodiment of the present invention; [Figure 7] FIG. 1 is a schematic diagram showing a flow of a control method for a humanoid piano-playing robot provided by an embodiment of the present invention. [Figure 8] 1 is a schematic diagram illustrating a structure of an electronic device provided by an embodiment of the present invention. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0025] In order to make the objectives, technical solutions and advantages of the present invention clearer, the technical solutions of the present invention will be described below clearly and completely in conjunction with specific embodiments of the present invention and corresponding accompanying drawings. Obviously, the described embodiments are only a part of the embodiments of the present invention, but not all of them. All other embodiments that can be obtained by those skilled in the art based on the embodiments of the present invention without creative efforts belong to the protection scope of the present invention.

[0026] DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS Below, in conjunction with the accompanying drawings, technical solutions provided by the embodiments of the present invention are described in detail.

[0027] The present invention provides a humanoid piano-playing robot. Figure 1 shows a performance system that combines a humanoid piano-playing robot with a piano.

[0028] FIG. 1 is a schematic diagram showing a performance system that combines a humanoid piano-playing robot and a piano provided according to an embodiment of the present invention.

[0029] Here, 1 is a humanoid piano-playing robot and 2 is a piano. The left and right arms and hands are controlled to move in space, the fingers are positioned above the corresponding keys, and the fingers are actuated to press the keys, thereby enabling the humanoid piano-playing robot to play the piano. The specific configuration of the humanoid piano-playing robot and the method for realizing piano playing are described below.

[0030] FIG. 2 is a schematic diagram showing the structure of a humanoid piano-playing robot provided according to an embodiment of the present invention.

[0031] Here, the humanoid piano-playing robot may include a left hand 11, a right hand 12, a left arm 13, a right arm 14, a waist 15, a neck 16, a visual sensing unit 17, a sound unit 18, a left leg 19, a right leg 110, a control system 111, and a base 112.

[0032] The neck unit 16 has two degrees of freedom including yaw and pitch to drive the yaw and pitch rotation of the head to realize a head shaking effect similar to that of a human being. The neck unit 16 is driven by a motor such as a servo motor.

[0033] The waist section 15 has two degrees of freedom including yaw and pitch to drive the yaw and pitch rotational movements of the upper body to realize a swaying effect of the waist similar to that of a human being. The waist section 15 is driven by a motor such as a servo motor.

[0034] As shown in FIG. 3, the neck 16 and waist 15 are each configured with two degrees of freedom, so that when playing the piano, the waist 15 and neck 16 work together to achieve the same "head-shaking and body-shaking" movements as a human being.

[0035] FIG. 3 is a schematic diagram showing the coordinated movement of the waist 15 and neck 16 of the humanoid piano-playing robot provided according to an embodiment of the present invention.

[0036] As can be seen from Fig. 3, when the humanoid piano-playing robot plays the piano, both the waist 15 and the neck 16 can swing downward. Of course, with the two degrees of freedom configured, it is also possible to swing the torso and head to the left, to the right, or forward and downward. How to control the movements of the waist 15 and neck 16 will be explained later.

[0037] The two-degree-of-freedom configuration allows the waist to rotate in yaw and pitch, which allows the hands of the humanoid piano-playing robot to more flexibly reach the positions of the keys that need to be pressed while playing the piano.

[0038] The left arm 13 and the right arm 14 are humanoid robot arms with multiple degrees of freedom. Specifically, the robot arms can be configured with seven degrees of freedom, i.e., seven joints. The arms perform position and posture calculation and motion planning according to the needs of the hands' piano playing movements, and realize accurate positioning of the hands in space.

[0039] Both the left leg 19 and the right leg 110 are designed to mimic the appearance of a human leg, with the lower part of the right leg 110 being configured with one degree of pitch freedom and capable of being used to press a piano pedal.

[0040] The control system 111 can plan the movement trajectory of the robot, send control commands to a drive system (not shown), and control the movement of each joint.

[0041] The base 112 is for mounting the robot body and has height adjustment, multi-directional movement, and locking capabilities.

[0042] When it is necessary to play the piano, the humanoid piano playing robot may use a visual sensing unit to obtain an image of musical score and an image of the piano keys facing the humanoid piano playing robot, recognize the image of the musical score to obtain musical score information, and position the piano keys based on the image of the keys to obtain positioning data.

[0043] The visual sensing unit 17 may be connected to a depth camera (RGB-D camera) and may collect images of the musical score and the keyboard through the camera. A system block diagram of the visual sensing unit 17 is shown in FIG.

[0044] FIG. 4 is a system block diagram illustrating a visual sensing unit provided by an embodiment of the present invention.

[0045] The visual sensing unit 17 may include an input module and an image processing module, which includes a keyboard positioning sub-module and a music score recognition sub-module (not shown).

[0046] In the visual sensing unit 17, when the input module obtains a score reading command, it obtains an image as input based on the RGB-D camera, and the score recognition sub-module of the image processing module detects the score and extracts score information such as the semantic information of the score, encodes the semantic information of the score into an information stream in MusicXML format, and sends it to the control system 111. Here, the semantic information of the score may include the song title, tempo, beat, key signature, notes, ornament symbols, etc.

[0047] When the input module acquires a play command, the RGB-D camera may collect an image of the keyboard, and the keyboard positioning submodule may perform calculations on the image of the keyboard to obtain the relative position and orientation of the piano with respect to the humanoid piano playing robot, determine the positioning data of the piano keys, and transmit the positioning data to the control system 111. In this way, the control system 111 may plan fingering and position and orientation change sequences based on the positioning data and the score information, and control the humanoid piano playing robot to play the piano. After acquiring the image of the keyboard, the keyboard positioning submodule may compare the acquired image with a standard keyboard template to determine whether the keyboard is missing, whether it is blocked by an object, or whether the positional relationship between the robot and the piano has changed. For example, if the keyboard positioning submodule determines that the relative position and orientation of the keyboard is abnormal, it transmits the determination result to the input module, causing the input module to reacquire the image of the keyboard and update the relative position and orientation of the keyboard with respect to the robot in real time. Alternatively, the keyboard positioning sub-module can send the judgment result to the control system 111, so as to cause the control system 111 to issue a keyboard abnormality warning and terminate the keyboard playing process in a timely manner, thereby realizing the protection of the piano and the left and right hands of the robot.

[0048] When determining the positioning data, the position of the keyboard corresponding to each key may be determined, or the position of the keyboard corresponding to the first note to be played may be determined, and the position of the keyboard corresponding to each key may be determined based on the relative positional relationship between the keys of each piano that has been preset.

[0049] Then, based on the positioning data (including the position of the keys) and the musical score information, a fingering to be used by the humanoid piano playing robot when playing the piece of music corresponding to the musical score information is planned, and a position and posture change sequence corresponding to the body parts of the humanoid piano playing robot when the humanoid piano playing robot plays the piece of music in accordance with the fingering is determined.

[0050] The fingering used when playing a piece of music corresponding to the musical score information may be a rule planned to play each note included in the musical score information with one or more fingers of the left hand 11 and / or right hand 12. The fingering may be determined by planning fingering using a score-based planning method, and planning an optimal arm and hand movement sequence (i.e., fingering) for a piece of music to play the piece of music. That is, multiple fingerings may be planned for one piece of music, and the optimal fingering may be selected. That is, multiple arm and hand movement sequences for playing the piece of music may be planned, each arm and hand movement sequence may be scored, and the arm and hand movement sequence with the highest score may be obtained as the fingering to be used. The scoring for the arm and hand movement sequences may be determined according to the cost of movement required for the arm and hand movement sequence, and the higher the cost, the lower the score.

[0051] Then, a position / posture change sequence corresponding to the body parts of the humanoid piano playing robot when playing the piece of music according to the fingering may be determined. Here, the position / posture change sequence may include positions / postures at a plurality of points in time. The positions / postures corresponding to each point in time may include positions / postures corresponding to each part, such as positions / postures corresponding to the left arm 13, the right arm 14, the waist 15, the neck 16, and the right leg 110.

[0052] Finally, the control system 111 may control the left hand 11 and the right hand 12 according to the planned fingering, and may control body parts other than the left hand 11 and the right hand 12 based on the position and posture change sequence.

[0053] Here, the hand (left hand 11 and right hand 12) may have five fingers, with the thumb and index finger each having two degrees of freedom, pitch and yaw, and the other three fingers each having one degree of freedom, pitch. All the fingers are driven by a motor such as a micro planetary reducer, and the rotation of the motor is converted into a pressing motion of the fingertips using a link transmission system.

[0054] The pressure detection module may be arranged on each finger, and detects the pressure of the fingertip on the keyboard in real time to improve the sensing and detection ability when playing the piano. The role of the pressure detection module may be various, and for example, when the pressure of the fingertip on the keyboard is too large (e.g., larger than a predetermined threshold value) by the pressure detection module, a power cut-off process of the robot may be performed to protect the keyboard. Also, for example, the pressure applied by the robot's fingertip on the keyboard may be determined based on the sound intensity information of the musical score, and when controlling the pressing of the finger, the determined pressure applied by the robot's fingertip on the keyboard may be reached by cooperation between the pressure detection module and a motor (micro planetary reducer) arranged on the finger.

[0055] The motion planning may automatically generate the above fingering, i.e., the motion sequence in which the fingers corresponding to each note press the keys, based on the contents of the musical score. Sound is generated by controlling the fingers to press the corresponding keys.

[0056] The arms (left arm 13 and right arm 14) are humanoid with multiple degrees of freedom, and can mimic the dexterous movements of the human arm to achieve the same performance effect as a human. All the joints of the arms are driven by motors such as servo motors, enabling highly accurate motion control.

[0057] The position and posture change sequence includes the position and posture of the arm tip, and the position of the arm tip is used to guide a specific finger of the hand to stay on the key to be played. Therefore, based on the positioning data, the position and posture of the arm tip at each time required for the left arm 13 and / or the right arm 14 to play the music corresponding to the musical score information according to the fingering may be determined, and based on the position and posture of the arm tip, a position and posture change sequence corresponding to the body parts of the humanoid piano playing robot when the humanoid piano playing robot plays the music according to the fingering may be determined.

[0058] Of course, the position and posture change sequence may include positions and postures corresponding to other parts, such as positions and postures corresponding to the waist and neck, and may be determined by the rhythm of the musical score information. By controlling two degrees of freedom corresponding to the waist and neck, respectively, according to the determined positions and postures, a swaying effect of the waist and neck when playing the piano similar to that of a human can be obtained. Of course, the positions and postures corresponding to the waist and neck may be determined according to a preset waist and neck swaying rule to control the movements of the waist and neck, and the specific method is not limited.

[0059] As can be seen from the above, the positioning of the hand in space is realized by controlling the movement of the arm. In the process of playing the piano, the position and orientation of the tip of the arm are calculated based on the positions of the keys and the fingers, the movement angles of each arm joint are obtained based on inverse kinematics calculation, and the arm movement is controlled to make the fingers reach the position of the key to be pressed.

[0060] The audio unit 18 is mainly used to realize voice recognition and voice interaction. Voice information from a user can be received via the audio unit 18, and the humanoid piano playing robot can determine based on the voice information whether the user's purpose is to request the humanoid piano playing robot to play the piano, and if it is determined that the user's purpose is to request the humanoid piano playing robot to play the piano, the control system 111 can control the humanoid piano playing robot to play the piano.

[0061] Here, the timing of receiving the performance command may be the timing when it is determined that the user's purpose is to request the humanoid piano playing robot to play the piano, or it may be the timing when the user initiates a performance command through a button located on the humanoid piano playing robot, and the specific timing of receiving the performance command is not limited.

[0062] If it is determined based on the voice information that the user's purpose is to request the humanoid piano playing robot to play the piano, the voice information is recognized, and information on the piece of music that the user requests the humanoid piano playing robot to play is determined, and based on the piece of music information, the control system 111 controls the humanoid piano playing robot to play the piano. Here, the piece of music information may include the name of the piece, the type of piece, etc.

[0063] FIG. 5 is a system block diagram illustrating a speech unit provided by an embodiment of the present invention.

[0064] As shown in Figure 5, first, the microphone inputs the original voice information into a pre-processing module, which performs voice noise removal and echo cancellation on the original voice information, filters out the noise and the voice data played by the device, obtains the voice information after the noise has been filtered, and transmits the voice information after the noise has been filtered to a natural language processing module.

[0065] The natural language processing module introduces speech recognition algorithms, speech classification algorithms, and semantic analysis algorithms. The speech recognition algorithm converts speech into text. The speech classification algorithm classifies speech types into task type, chat type, and question and answer (Q&A) type. The semantic analysis algorithm extracts the purpose and important parameters of the speech (e.g. song information).

[0066] After the voice is classified, if it is a chat type or a question and answer type, an intelligent answer is generated by the intelligent answer generation module, synthesized by the voice synthesis module, and finally output to the speaker to emit voice. If it is a task type and the purpose of the voice is to play the piano, the piano playing module extracts music information for playing the piano, and sends the music information to the control system 111, and the control system 111 may obtain corresponding music score information from the music information (the specific method is not limited, for example, it may be obtained via a public network, or music score information corresponding to fixed music information may be stored inside the robot so that the control information can directly obtain the music score information). Furthermore, the control system 111 drives the robot to play the piano.

[0067] As can be seen from the above, the control system 111 can be responsible for motion planning and execution control, which is carried out in two steps. In the first step, it plans the motion of playing the piano based on the song, and realizes the mapping from the beat of the music to the position and orientation of the fingers in space (this can be understood as the correspondence of which finger plays which note). In the second step, it plans the trajectory of the tip of the robot arm based on the spatial position and orientation of the fingers determined in the first step, performs inverse kinematic calculations, and drives the movement of all the joints of the arm.

[0068] Here, the second step will be specifically described. In the second step, a sequence of the position and orientation of the tip of the arm is determined, and the angles of each joint of the arm are determined based on the sequence of the position and orientation of the tip of the arm, thereby controlling the movement of the arm.

[0069] That is, inverse kinematics calculation is performed for the position and posture of the tip of each arm in the position and posture change sequence, the joint angles required for the left arm 13 or right arm 14 to reach the position and posture of the tip of that arm are determined, and the movement of the left arm 13 or right arm 14 is controlled based on the joint angles corresponding to the position and posture of the tip of each arm. For the position and posture of the tip of one arm, if the position and posture of the tip of that arm belongs to the left arm, it is necessary to determine the joint angle of the left arm, and if the position and posture of the tip of that arm belongs to the right arm, it is necessary to determine the joint angle of the right arm.

[0070] Also, since there is a difference between the left arm and the right arm, and the determined position and posture of the tip of the arm is usually in one coordinate system (for example, a world coordinate system centered on the center of the waist of the robot), for the position and posture of the tip of each arm in the position and posture change sequence, the arm to which the position and posture of the tip of the arm belongs may be determined, and the position and posture of the tip of the arm may be transformed into the shoulder coordinate system of the arm to which the position and posture of the tip of the arm belongs based on the fixed yaw rotation angle between the shoulder corresponding to the arm to which the position and posture of the tip of the arm belongs and the waist 15, the yaw rotation angle and pitch rotation angle of the waist 15 at that time, and the coordinates of the shoulder corresponding to the arm to which the position and posture of the tip of the arm belongs in the local coordinate system of the pitch joint of the waist 15, to obtain the transformed position and posture. In this way, the control system 111 may control the movement of the left arm 13 and / or the right arm 14 based on the transformed position and posture.

[0071] In addition, the position and orientation that each joint of the arm should reach may be determined based on the position and orientation of the tip of the arm, and the joint angle of each joint may be determined. These positions and orientations may be transformed into the shoulder coordinate system in the same manner as described above.

[0072] The world coordinate system and the shoulder coordinate system corresponding to the left arm are specifically shown in FIG.

[0073] FIG. 6 is a schematic diagram showing a world coordinate system and a local coordinate system corresponding to the left arm provided by an embodiment of the present invention.

[0074] The dual robot arm is controlled by position and orientation control in Cartesian space. The origin of the world coordinate system in Cartesian space is located at the center of the waist. The positive direction of the X-axis points to the front of the robot, the positive direction of the Y-axis points to the left side of the robot, and the positive direction of the Z-axis points upward of the robot. The local coordinate system of the pitch joint of the waist 15 can refer to the world coordinate system.

[0075] The origins of the two shoulder coordinate systems of the dual robot arm are located at the centers of the left and right shoulders respectively (the above description is based on the shoulder coordinate system), and the directions of the coordinate axes of the shoulder coordinate systems are parallel to the world coordinate system. Taking the right arm as an example, the control system 111 can receive the position and orientation of the tip of the right arm in the world coordinate system transformed by the musical score, plan the path of the tip of the robot arm from the current position to the target position in an inverse parabolic manner, and complete the inverse kinematic calculation for all points on the path.

[0076] At a certain point in time, the position and orientation of the tip of the robot arm can be transformed into the shoulder coordinate system. The position and orientation of the tip of the robot arm ( w p7, w R7) is the position and orientation of the right shoulder in the local coordinate system ( s p7, s R7), where α yaw and α pitch are the yaw and pitch rotation angles of the waist at that time, and β yaw is the fixed yaw rotation angle between the shoulders and hips, waist_pitch p s are the coordinates of the robot arm's shoulder in the local coordinate system of the waist pitch joint.

number

number

[0077] Each rotation matrix is ​​expressed by the following equations (3) to (5).

number

number

number

[0078] Finally, the seven joint angles of the right arm in the joint space can be obtained by the inverse kinematics calculation and transmitted to the drive system to control the robot arm to move to the target position and posture. The inverse kinematics calculation method may adopt the inverse kinematics calculation of the redundant robot arm based on the arm angle, which is currently widely used.

[0079] 7 is a schematic diagram showing a flow of a control method for a humanoid piano playing robot provided by an embodiment of the present invention. The control method for the humanoid piano playing robot is applied to a humanoid piano playing robot, and specifically includes the following steps S701 to S704.

[0080] In S701, the visual sensing unit obtains an image of a musical score and an image of the piano keyboard that the humanoid piano playing robot faces.

[0081] In S702, the visual sensing unit recognizes the image of the musical score to obtain musical score information, and positions the piano keys according to the image of the keys to obtain positioning data.

[0082] In S703, the control system plans fingering to be used by the humanoid piano playing robot when playing the piece of music corresponding to the music score information based on the positioning data and the music score information, and determines a position and posture change sequence corresponding to the body parts of the humanoid piano playing robot when the humanoid piano playing robot plays the piece of music according to the fingering.

[0083] In S704, the control system controls the left hand and the right hand in accordance with the fingering, and controls body parts other than the left hand and the right hand based on the position and posture change sequence.

[0084] The control method for the humanoid piano-playing robot provided by the embodiment of the present invention has been described in the description of the humanoid piano-playing robot above, so detailed description will not be repeated here.

[0085] As can be seen from the above, the waist and neck of the humanoid piano-playing robot provided by the embodiment of the present invention are configured with two degrees of freedom, which can flexibly achieve the goal of playing the piano in the same way as a human being. The visual sensing unit can accurately position the position of the keys, intelligently recognize the content of the musical score, and automatically play the piano according to the musical score.

[0086] In addition, the voice unit enables the humanoid piano-playing robot to intelligently recognize the song that the user requests to play on the piano and automatically play the piano. By controlling the two degrees of freedom of the waist and neck, and the positions and orientations of the two robot arms, the piano-playing robot of the embodiment of the present invention can play the piano skillfully and intelligently.

[0087] The present invention further provides a computer-readable storage medium storing a computer program, the computer program being used to execute the control method for the humanoid piano-playing robot.

[0088] The present invention further provides an electronic device. As shown in Fig. 8, at the hardware level, the electronic device includes a processor, an internal bus, a network interface, an internal memory, and a non-volatile memory, and may also include other hardware required for operation. The processor loads the corresponding computer program from the non-volatile memory into the internal memory and executes it to implement the control method for the humanoid piano playing robot.

[0089] Of course, in addition to realization by software, the present invention does not exclude other realization methods, such as logical devices or a combination of hardware and software. In other words, the entity that executes the following processing steps is not limited to each logical unit, but may be hardware or a logical device.

[0090] In the 1990s, improvements in a technology could be clearly divided into hardware improvements (such as improvements in circuit structures such as diodes, transistors, and switches) and software improvements (improvements in method flows). However, with the development of technology, many of the current method flow improvements can be considered as direct improvements to hardware circuit structures. Designers can mostly obtain the corresponding hardware circuit structures by programming the improved method flows into the hardware circuits. Therefore, it cannot be said that method flow improvements cannot be realized by hardware physical modules. For example, Programmable Logic Devices (PLDs) (e.g. Field Programmable Gate Arrays (FPGAs)) are such integrated circuits, whose logical functions are determined by the user's programming of the device. Instead of chip manufacturers designing and manufacturing dedicated integrated circuit chips, designers program and "integrate" digital systems onto a single PLD.Nowadays, instead of handcrafting integrated circuit chips, this programming is mostly realized using software called a "logic compiler", which is similar to a software compiler used to write a program, and in order to compile the original code, it needs to be written in a specific programming language, which is called a Hardware Description Language (HDL). There is not just one type of HDL, but many types such as ABEL (Advanced Boolean Expression Language), AHDL (Altera Hardware Description Language), Confluence, CUPL (Cornell University Programming Language), HDCal, JHDL (Java Hardware Description Language), Lava, Lola, MyHDL, PALASM, RHDL (Ruby Hardware Description Language), etc., and the most commonly used at present are VHDL (Very-High-Speed ​​Integrated Circuit Hardware Description Language) and Verilog. It will be clear to those skilled in the art that a hardware circuit that realizes a logical method flow can be easily obtained by simply programming a method flow logically in some of the above hardware description languages ​​and programming it into an integrated circuit.

[0091] The controller may be realized in any suitable manner, for example, the controller may take the form of a microprocessor or processor, a computer readable storage medium storing computer readable program code (e.g., software or firmware) executable by the (micro)processor, logic gates, switches, application specific integrated circuits (ASICs), programmable logic controllers, and embedded microcontrollers, examples of which include, but are not limited to, microcontrollers such as ARC 625D, Atmel AT91SAM, Microchip PIC18F26K20, Silicone Labs C8051F320, etc., and the memory controller may also be realized as part of the control logic of the memory. It will be clear to those skilled in the art that in addition to implementing the controller purely with computer readable program code, it is entirely possible to logically program method steps to cause the controller to perform the same functions in the form of logic gates, switches, application specific integrated circuits, programmable logic controllers, embedded microcontrollers, etc. Thus, such a controller may be considered as a hardware component, and the devices included therein for implementing various functions may also be considered as structures within the hardware component. Alternatively, the apparatus for implementing various functions may be considered to be software modules implementing a method, or structures within hardware components.

[0092] The systems, devices, modules or units described in the above embodiments may be specifically implemented by computer chips, entities, or products having some functions. A typical implementation device is a computer. Specifically, the computer may be, for example, a personal computer, a laptop computer, a mobile phone, a camera phone, a smartphone, a personal digital assistant, a media player, a navigation device, an email device, a game console, a tablet, a wearable device, or any combination of these devices.

[0093] For the sake of convenience, the above-mentioned device will be described by dividing it into various units according to their functions. Of course, when implementing the present invention, the functions of each unit can be realized by the same or multiple pieces of software and / or hardware.

[0094] As will be appreciated by those skilled in the art, the embodiments of the present invention may be provided as a method, a system, or a computer program product. Thus, the present invention may employ an entirely hardware embodiment, an entirely software embodiment, or an embodiment that combines software and hardware. Furthermore, the present invention may take the form of a computer program product embodied in one or more computer usable storage media (including, but not limited to, magnetic disk memory, CD-ROM, optical memory, etc.) that contain computer usable program code.

[0095] The present invention will be described with reference to flowcharts and / or block diagrams of methods, devices (systems), and computer program products according to embodiments of the present invention. It should be understood that each flow and / or block in the flowcharts and / or block diagrams, and combinations of flows and / or blocks in the flowcharts and / or block diagrams, may be realized by computer program instructions. These computer program instructions may be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing device to generate a machine, whereby the instructions executed by the processor of the computer or other programmable data processing device generate an apparatus for implementing the functions specified in one or more flows of the flowcharts and / or one or more blocks of the block diagrams.

[0096] These computer program instructions may be stored in a computer-readable memory that can direct a computer or other programmable data processing device to operate in a particular manner, such that the instructions stored in the computer-readable memory result in an article of manufacture that includes an instruction apparatus that implements the functions specified in one or more flows of the flowcharts and / or one or more blocks of the block diagrams.

[0097] These computer program instructions may be loaded into a computer or other programmable data processing device such that a series of operational steps are executed on the computer or other programmable device to generate a computer-implemented process, whereby the instructions executed on the computer or other programmable device provide steps for implementing the functions specified in one or more flows of the flowcharts and / or one or more blocks of the block diagrams.

[0098] In a typical configuration, a computing device includes one or more processors (CPUs), input / output interfaces, network interfaces, and memory.

[0099] The memory may include volatile, random access memory (RAM) and / or non-volatile memory forms of computer readable storage media, such as read only memory (ROM) or flash memory (flash RAM). The memory is one example of a computer readable storage medium.

[0100] Computer readable storage media include non-volatile and volatile media, removable and non-removable media, and may implement information storage by any method or technology. Information may be computer readable instructions, data structures, program modules, or other data. Computer storage media include, but are not limited to, Phase Change Memory (PRAM), Static Random-Access Memory (SRAM), Dynamic Random Access Memory (DRAM), other types of Random Access Memory (RAM), Read Only Memory (ROM), Electrically Erasable Programmable Read Only Memory (EEPROM), flash memory or other memory technology, Compact Disc Read Only Memory (CD-ROM), Digital Versatile Disc (DVD) or other optical storage, magnetic cassette tape, magnetic tape magnetic disk storage or other magnetic storage devices, or any other non-transmission medium that may be used to store information accessible by a computing device. As defined herein, computer readable storage media does not include transitory computer readable storage media, such as modulated data signals and carriers.

[0101] Additionally, the terms "comprise," "contain," or any other variations thereof are intended to include a non-exclusive inclusion, whereby a process, method, article, or device that comprises a set of elements not only includes those elements, but also includes other elements not expressly listed, or includes the inherent elements of such process, method, article, or device. Absent more limitations, an element qualified by the phrase "comprises a..." does not exclude the presence of further identical elements in a process, method, article, or device that includes said element.

[0102] The invention may be described in the general context of computer-executable instructions being executed by a computer, such as program modules. Generally, program modules include routines, programs, objects, components, data structures, etc. that perform particular tasks or implement particular abstract data types. The invention may also be practiced in distributed computing environments where tasks are performed by remote processing devices that are linked through a communications network. In a distributed computing environment, program modules may be located in both local and remote computer storage media, including storage devices.

[0103] Each embodiment of the present invention will be described in a stepwise manner, and the same or similar parts between each embodiment may be referred to each other, and the emphasis of each embodiment is on the differences from other embodiments. In particular, the system embodiment is basically similar to the method embodiment, so it will be briefly described, and the relevant parts may be referred to the description of part of the method embodiment.

[0104] The above is merely an embodiment of the present invention, and is not used to limit the present invention. For those skilled in the art, the present invention may have various modifications and changes. Any modifications, equivalent replacements, improvements, etc. made without departing from the spirit and principle of the present invention should be included in the scope of the claims of the present invention. [Explanation of symbols]

[0105] 1. Humanoid piano-playing robot 2. Piano 11 left hand 12 Right hand 13 Left Arm 14 Right Arm 15 Lower back 16 neck 17 Visual Sensing Unit 18 Audio Units 19 Left leg 110 Right leg 111 Control System 112 Base

Claims

1. A humanoid piano-playing robot, comprising: a left hand, a right hand, a left arm, a right arm, a waist, a neck, a visual sensing unit, a left leg, a right leg, a control system, and a base, wherein the neck is configured with two degrees of freedom for driving a yaw rotational motion and a pitch rotational motion of a head, and the waist is configured with two degrees of freedom for driving a yaw rotational motion and a pitch rotational motion of an upper body, The humanoid piano-playing robot is Using the visual sensing unit, obtain an image of a musical score and an image of a piano keyboard facing the humanoid piano playing robot, recognize the image of the musical score to obtain musical score information, and position the piano keyboard based on the image of the keyboard to obtain positioning data; a control system for determining a fingering pattern to be used by the humanoid piano playing robot when the humanoid piano playing robot plays a piece of music corresponding to the score information based on the positioning data and the score information, and determining a position and posture change sequence corresponding to body parts of the humanoid piano playing robot when the humanoid piano playing robot plays the piece of music according to the fingering; The control system is used to control the left hand and the right hand in accordance with the fingering, and to control a body part other than the left hand and the right hand based on the position / posture change sequence, Here, the humanoid piano playing robot is a control system that determines, based on the positioning data, positions and postures of the tips of the arms at each time required for the left arm and / or the right arm to play the piece of music corresponding to the score information in accordance with the fingering, and determines, based on the positions and postures of the tips of the arms, a position and posture change sequence that corresponds to body parts of the humanoid piano playing robot when the humanoid piano playing robot plays the piece of music in accordance with the fingering; the control system determines, for the position and posture of the tip of each arm in the position and posture change sequence, the arm to which the position and posture of the tip of the arm belongs, and transforms the position and posture of the tip of the arm into the shoulder coordinate system of the arm to which the position and posture of the tip of the arm belongs based on a fixed yaw rotation angle between the shoulder and waist corresponding to the arm to which the position and posture of the tip of the arm belongs, the yaw rotation angle and pitch rotation angle of the waist at that time, and the coordinates of the shoulder corresponding to the arm to which the position and posture of the tip of the arm belongs, in a local coordinate system of a pitch joint of the waist, to obtain a transformed position and posture, which is used to control movement of the left arm and / or the right arm based on the transformed position and posture. This is a humanoid piano-playing robot.

2. The humanoid piano playing robot further comprises: used by the visual sensing unit to encode the score information to obtain encoded information in a predetermined format, and transmit the encoded information to the control system; The musical score information includes at least beats, notes, and ornament symbols.

2. The humanoid piano-playing robot according to claim 1 .

3. The humanoid piano playing robot further includes a voice unit, The humanoid piano playing robot further comprises: The audio unit receives audio information from a user, and based on the audio information, determines whether the user's purpose is to request the humanoid piano playing robot to play the piano. If it is determined that the user's purpose is to request the humanoid piano playing robot to play the piano, the control system uses the audio information to control the humanoid piano playing robot to play the piano.

2. The humanoid piano-playing robot according to claim 1 .

4. The humanoid piano-playing robot is When the audio unit determines based on the audio information that the user's purpose is to request the humanoid piano playing robot to play the piano, the audio unit recognizes the audio information and determines music information that the user requests the humanoid piano playing robot to play, which is used by the control system to control the humanoid piano playing robot to play the piano based on the music information.

4. The humanoid piano-playing robot according to claim 3.

5. The humanoid piano-playing robot is The audio unit is used to filter noise in the audio information, and to determine whether the user's purpose is to request the humanoid piano playing robot to play the piano based on the audio information after the noise filtering.

4. The humanoid piano-playing robot according to claim 3.

6. The humanoid piano-playing robot is The control system performs inverse kinematics calculation for the position and posture of the tip of each arm in the position and posture change sequence to determine joint angles required for the left arm or the right arm to reach the position and posture of the tip of the arm, and the joint angles are used to control the movement of the left arm or the right arm based on the joint angles corresponding to the position and posture of the tip of each arm.

2. The humanoid piano-playing robot according to claim 1 .

7. A control method for a humanoid piano-playing robot, the control method being applied to the humanoid piano-playing robot, the humanoid piano-playing robot including a left hand, a right hand, a left arm, a right arm, a waist, a neck, a visual sensing unit, a left leg, a right leg, a control system, and a base, the neck being configured with two degrees of freedom for driving the yaw rotational motion and the pitch rotational motion of the head, the waist being configured with two degrees of freedom for driving the yaw rotational motion and the pitch rotational motion of the upper body, the control method for the humanoid piano-playing robot comprising: acquiring an image of a musical score and an image of a piano keyboard facing the humanoid piano-playing robot by the visual sensing unit; recognizing the image of the score by the visual sensing unit to obtain score information, and locating the piano keys according to the image of the keys to obtain positioning data; a step of planning, by the control system, fingering to be used when the humanoid piano playing robot plays a piece of music corresponding to the score information, based on the positioning data and the score information, and determining a position and posture change sequence corresponding to body parts of the humanoid piano playing robot when the humanoid piano playing robot plays the piece of music according to the fingering; controlling the left hand and the right hand by the control system in accordance with the fingering, and controlling a body part other than the left hand and the right hand based on the position and posture change sequence; determining a position and posture change sequence corresponding to body parts of the humanoid piano playing robot when the humanoid piano playing robot plays the piece of music according to the fingering, determining, based on the positioning data, a position and posture of the tip of the left arm and / or the right arm at each time required for the left arm and / or the right arm to play the piece of music corresponding to the musical score information in accordance with the fingering; determining a position and posture change sequence corresponding to body parts of the humanoid piano playing robot when the humanoid piano playing robot plays the piece of music according to the fingering, based on the position and posture of the tip of the arm; controlling the left hand and the right hand in accordance with the fingering and controlling a body part other than the left hand and the right hand based on the position and posture change sequence, the step of determining, for the position and posture of the tip of each arm in the position and posture change sequence, the arm to which the position and posture of the tip of the arm belongs, and transforming the position and posture of the tip of the arm into the shoulder coordinate system of the arm to which the position and posture of the tip of the arm belongs based on a fixed yaw rotation angle between the shoulder and waist corresponding to the arm to which the position and posture of the tip of the arm belongs, the yaw rotation angle and pitch rotation angle of the waist at that time, and coordinates of the shoulder corresponding to the arm to which the position and posture of the tip of the arm belongs, in a local coordinate system of a pitch joint of the waist, to obtain a transformed position and posture, and controlling a movement of the left arm and / or the right arm based on the transformed position and posture. A method for controlling a humanoid piano-playing robot.

8. The method for controlling the humanoid piano playing robot further comprises: encoding the score information by the visual sensing unit to obtain encoded information in a predetermined format, and transmitting the encoded information to the control system; The musical score information includes at least beats, notes, and ornament symbols.

8. The method for controlling the humanoid piano-playing robot according to claim 7.

9. The humanoid piano playing robot further includes a voice unit, The method for controlling the humanoid piano playing robot further comprises: receiving, by said audio unit, audio information from a user; determining, by the voice unit, based on the voice information, whether or not the user's purpose is to request the humanoid piano playing robot to play the piano, and when it is determined that the user's purpose is to request the humanoid piano playing robot to play the piano, controlling, by the control system, the humanoid piano playing robot to play the piano.

8. The method for controlling the humanoid piano-playing robot according to claim 7.

10. The step of controlling the humanoid piano playing robot to play the piano by the control system includes: when it is determined by the audio unit based on the audio information that the user's purpose is to request the humanoid piano playing robot to play the piano, recognizing the audio information and determining song information that the user requests the humanoid piano playing robot to play; and controlling the humanoid piano playing robot to play the piano based on the music information by the control system.

10. The method for controlling a humanoid piano-playing robot according to claim 9.

11. A computer-readable storage medium storing a computer program, the computer program being executed by a processor to implement the method for controlling a humanoid piano-playing robot according to any one of claims 7 to 10. A computer-readable storage medium comprising:

12. An electronic device including a memory, a processor, and a computer program stored in the memory and executable on the processor, the electronic device implementing the method for controlling a humanoid piano-playing robot according to any one of claims 7 to 10 when the processor executes the computer program.

1. An electronic device comprising:

Citation Information

Patent Citations

  • Novel piano playing robot system

    CN106903704A

  • Control method and device for piano playing by robot, storage medium and robot

    CN110861086A

  • Piano playing robot

    CN1481977A

  • Robot finger-hit playing device for electronic piano having small number of keys

    JP1996001562A

  • Reproduction log analysis program, terminal device, and autonomously moving type robot

    WO2008093841A1