Robot hand positioning and musical instrument playing control method and system based on multiple cameras

Abstract

The invention relates to the technical field of robot hand control, and discloses a multi-camera-based robot hand positioning and musical instrument playing control method and system, and the method comprises the steps: obtaining a global image through a head camera, and providing a hand rough position; the fingertip camera obtains a local image and provides fine positioning of a target; fusing coarse and fine positioning, calculating a relative position, and planning a motion track; a robot execution mechanism is controlled to execute the track to complete playing; monitoring an actual position, calculating an error, and correcting a joint when the error exceeds a threshold value; the system comprises a head camera module, a fingertip camera module, a visual fusion module, a control module, a robot execution mechanism and a feedback and correction module. According to the invention, by acquiring the high-resolution local image and combining the global rough position information provided by the head camera module, effective combination of rough positioning and fine positioning is realized, and the problem that single global vision is easily blocked by hands or the precision is insufficient due to long distance is effectively overcome.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of robot hand control, in particular to a multi-camera robot hand positioning and musical instrument playing control method and system. BACKGROUND

[0002] Robot systems are increasingly being applied in fields requiring high precision and high flexibility operations, such as musical instrument playing. Such tasks place extremely high demands on the perception and control capabilities of the robot. The system not only needs to accurately perceive the coordinate position of fine targets on the musical instrument (such as keys, buttons or strings), but also must be able to dynamically and accurately control its end effector (such as a dexterous hand) to complete complex playing actions. Therefore, high-precision visual positioning capability and high-dynamic response control capability are the key technical foundations for achieving high-level robot playing.

[0003] In existing robot applications, visual positioning schemes are the main technical path to achieve environmental perception. A common technical solution is to use global vision, i.e. setting up a camera outside the robot workspace or on the head of the robot to obtain the overall scene containing the robot and the musical instrument. Another solution is to integrate a miniature camera on the end of the robot, such as the fingers of the dexterous hand, to obtain a close-range local high-resolution image. At the control strategy level, many robot systems rely on open-loop control. This approach usually plans a complete motion trajectory based on the initial positioning results provided by the vision system, and then drives the robot effector to complete the trajectory.

[0004] However, the existing robot playing control technology using the global vision scheme has limited positioning accuracy due to the resolution of the camera, and when the robot hand approaches and operates the musical instrument, the hand itself is extremely likely to block the key interaction target, resulting in positioning failure. The fingertip camera scheme, although has high resolution, its narrow field of view results in the loss of global coordinate information, making it difficult to independently support complete motion planning. The images collected by the fingertip camera during high-speed motion of the robot finger are extremely prone to severe motion blur, resulting in reduced positioning stability. And at the control level, the open-loop execution method is extremely sensitive to calibration errors or small environmental disturbances. Any small model deviation will be accumulated during execution, ultimately leading to operation failure. Therefore, the present application provides a multi-camera robot hand positioning and musical instrument playing control method and system to solve the deficiencies in the prior art. SUMMARY

[0005] In view of the deficiencies of the prior art, the application provides a multi-camera robot hand positioning and musical instrument playing control method and system, which solves the problem that the existing robot musical instrument playing control method mostly relies on a single visual source, and when dealing with a musical instrument with a complex structure or requiring multi-finger cooperation, the positioning accuracy is not high and the closed-loop correction response is slow due to factors such as hand self-shielding and motion blur.

[0006] To achieve the above object, the application is implemented by the following technical solutions: The application provides a multi-camera robot hand positioning and musical instrument playing control method in the first aspect, which comprises the following steps: S1, acquiring a global image of a working scene by using a head camera module, and providing rough position information of a robot hand based on the global image of the working scene; S2, acquiring a local image of a target to be interacted by using a fingertip camera module arranged at the end of a robot finger, and providing a target fine positioning result based on the local image of the target to be interacted; S3, fusing the rough position information and the target fine positioning result, calculating a relative position after fusion of the end of the finger and the target to be interacted, and planning a motion trajectory based on the relative position after fusion; S4, controlling a robot execution mechanism to execute the motion trajectory to complete a playing action; S5, in the process of executing the playing action, continuously acquiring an actual position of the robot execution mechanism by a visual system, comparing the actual position with the motion trajectory to calculate a Cartesian space position error vector, and when the module length of the Cartesian space position error vector exceeds a trigger threshold, calculating a robot joint angle correction amount to update a control instruction of the robot execution mechanism.

[0007] Preferably, in step S1, the rough position information of the robot hand based on the global image of the working scene further comprises: A hand detection model based on a convolutional neural network is used to analyze the global image of the working scene to determine two-dimensional pixel coordinates of a hand region; Based on a pinhole imaging model of a camera, and in combination with camera internal and external parameters and depth information, the two-dimensional pixel coordinates are solved into three-dimensional space coordinates of the robot hand as rough position information.

[0008] Preferably, in step S2, the step of providing a target fine positioning result based on a local image of a target to be interacted further comprises: A lightweight visual Transformer model is used to process the local image of the target to be interacted to identify a specific target control and output accurate pixel coordinates of the target control on a two-dimensional image. According to the accurate pixel coordinates, and in combination with the internal and external parameters of the fingertip camera and depth information, three-dimensional space coordinates of the target control member in a world coordinate system are calculated as a target fine positioning result.

[0009] Preferably, in step S3, the step of planning a motion trajectory based on the fused relative position further comprises: a transformation matrix of a head camera module coordinate system and a fingertip camera module coordinate system is obtained through pre-calibration; a dynamic weight fusion algorithm is used to calculate the fused relative position in combination with the coarse position information, the target fine positioning result, and the transformation matrix; a Kalman filter is used to smooth the time series of the fused relative position, and a motion trajectory is generated based on the smoothed trajectory point sequence.

[0010] Preferably, in step S4, the step of controlling the robot execution mechanism to execute the motion trajectory to complete a performance action further comprises: inverse kinematics is solved for a target pose point on the motion trajectory to calculate target angles and target angular velocities of joints of the robot; a feedforward compensation torque is calculated based on a robot dynamics model; the target angles, the target angular velocities, and the feedforward compensation torque are packaged into a digital control package, and sent to the robot execution mechanism through an industrial communication bus.

[0011] Preferably, in step S5, the computer calculates a joint angle correction amount to update a control instruction for the robot execution mechanism, which further comprises: a proportional-derivative controller is used to calculate a Cartesian space position correction vector according to the Cartesian space position error vector and the rate of change; the Cartesian space position correction vector is converted to a joint space through an inverse or pseudo-inverse of a Jacobian matrix of the robot to obtain the joint angle correction amount; the joint angle correction amount is combined with the original target angle to generate a new target joint angle and update the control instruction.

[0012] Preferably, the dynamic weight fusion algorithm specifically comprises: a dynamic weight parameter is set, and the dynamic weight parameter is dynamically adjusted according to image quality or confidence of a target detection result collected by the fingertip camera module in real time; when the image quality of the fingertip camera module is reduced due to motion blur or occlusion, the value of the dynamic weight parameter is increased to rely on the coarse position information.

[0013] Preferably, the method further comprises: controlling unified triggering and timestamp management of each camera in the fingertip camera module by using a multi-camera synchronization module to ensure time synchronization of the coarse position information and the target fine positioning result.

[0014] The second aspect of the application provides a multi-camera robot hand positioning and musical instrument performance control system, comprising: a head camera module for acquiring a global image of a working scene and providing coarse position information of a robot hand; a fingertip camera module arranged at the end of a robot finger for acquiring a local image of a target to be interacted with and providing a target fine positioning result; a visual fusion module for fusing the coarse position information and the target fine positioning result to calculate a fused relative position between the end of the finger and the target to be interacted with; a control module for planning a motion trajectory according to the fused relative position and generating a control instruction; a robot execution mechanism for receiving and executing the control instruction to complete a performance action; a feedback and correction module for continuously acquiring an actual position of the robot execution mechanism through a visual system during execution of the performance action, comparing the actual position with the motion trajectory to calculate a Cartesian space position error vector, and calculating a joint angle correction amount to update the control instruction when a module length of the Cartesian space position error vector exceeds a triggering threshold.

[0015] Preferably, the robot execution mechanism comprises a multi-degree-of-freedom mechanical arm and an end effector dexterous hand.

[0016] The application provides a multi-camera robot hand positioning and musical instrument performance control method and system. The application has the following beneficial effects: 1. The fingertip camera module arranged at the end of the robot finger acquires a high-resolution local image, and the global coarse position information provided by the head camera module is combined to effectively combine coarse positioning and fine positioning. Thus, the technical defects of single global vision, such as easy to be blocked by the hand itself or insufficient precision due to long distance, are effectively overcome, and the positioning precision of the end of the robot finger relative to the target to be interacted with is improved.

[0017] 2、The application adopts a dynamic weight fusion algorithm, dynamically adjusts the fusion weight of the rough position information and the target fine positioning result according to the image quality or the confidence of target detection collected by the fingertip camera module in real time. When the image quality of the fingertip camera is reduced due to motion blur or obstruction, the system can increase the dependence on the rough positioning information of the head camera, thereby avoiding the positioning interruption or error caused by the local visual temporary failure, and improving the robustness of the system in a complex dynamic environment.

[0018] 3、The application introduces a visual-based real-time closed-loop feedback in the performance action execution process, continuously acquires the actual position of the robot execution mechanism, compares it with the planned motion trajectory to calculate the deviation, and immediately starts the correction calculation based on the proportional differential control and Jacobian matrix when the deviation exceeds the threshold. This high-frequency closed-loop correction mechanism can quickly compensate for the motion error caused by model inaccuracy or external small disturbances, ensuring the real-time accuracy of the performance action and the precision of the trajectory tracking. BRIEF DESCRIPTION OF DRAWINGS

[0019] Figure 1 The system architecture diagram of the application; Figure 2 The method step flow chart of the application; Figure 3 The closed-loop correction control flow chart of the application.

[0020] Among them, 10, head camera module; 20, fingertip camera module; 30, multi-camera synchronization module; 40, visual fusion module; 50, control module; 60, robot execution mechanism; 70, feedback and correction module. DETAILED DESCRIPTION

[0021] The technical solutions in the embodiments of the application will be described clearly and completely below with reference to the drawings in the specification of the application. Obviously, the described embodiments are only part of the embodiments of the application, not all the embodiments. Based on the embodiments in the application, all other embodiments obtained by those skilled in the art without creative labor are within the protection scope of the application.

[0022] Referring to the drawings Figure 1 , Figure 1 The system architecture diagram of one embodiment of the application. The application provides a multi-camera robot hand positioning and musical instrument performance control system, which includes a head camera module 10, a fingertip camera module 20, a multi-camera synchronization module 30, a visual fusion module 40, a control module 50, a robot execution mechanism 60, and a feedback and correction module 70.

[0023] A head camera module 10 is used to acquire global images of the working scene including the robot hand and the musical instrument, and to provide coarse position information of the robot hand.

[0024] A fingertip camera module 20 is used to acquire high-resolution local images of the target to be interacted, including multiple cameras arranged at the end of the robot fingers.

[0025] A multi-camera synchronization module 30 is used to control the unified triggering and time stamp management of the cameras in the fingertip camera module 20. In a specific embodiment, the multi-camera synchronization module 30 ensures that the timing error of each camera image acquisition does not exceed ±80ms, providing a time synchronization reference for subsequent data fusion.

[0026] A vision fusion module 40 is used to receive global image information from the head camera module 10 and synchronized local image information from the fingertip camera module 20, and to fuse multi-source information based on a unified time stamp to calculate the relative coordinates between the end of the finger and the target.

[0027] A control module 50 is used to plan the motion path based on the coordinate information output by the vision fusion module 40, and to generate control instructions that can be executed by the robot execution mechanism 60.

[0028] A robot execution mechanism 60, such as a multi-degree-of-freedom robotic arm and an end-effector dexterous hand, is used to receive and execute control instructions from the control module 50 to complete physical operations.

[0029] A feedback and correction module 70 is used to continuously acquire the actual motion state of the robot execution mechanism 60 through the vision system during the motion of the robot execution mechanism 60, compare it with the target trajectory planned by the control module 50, and start correction calculation when there is a deviation.

[0030] During system operation, the head camera module 10 and the fingertip camera module 20 collect data in parallel, the collection process of the fingertip camera module 20 is controlled by the multi-camera synchronization module 30, the collected data stream is transmitted to the vision fusion module 40 for processing, the calculation result of the vision fusion module 40 is output to the control module 50 for trajectory planning and instruction generation, and the generated instruction is sent to the robot execution mechanism 60. During the motion of the execution mechanism, the feedback and correction module 70 continuously works, and the correction signal is fed back to the control module 50 to dynamically adjust the motion of the robot execution mechanism 60.

[0031] Referring to the accompanying drawings Figure 2 , Figure 2 is a method flowchart according to an embodiment of the present application. The present application provides a multi-camera robot hand positioning and musical instrument performance control method, including the following steps: S1, coarse positioning of the robot hand by using the head camera; S2, fine positioning of the target key by using the fingertip camera; S3, fusion of the recognition information of the head camera and the fingertip camera to generate a motion trajectory of the finger to the target position; S4, control of the robot finger to perform a performance action; S5, real-time monitoring during performance, and correction if deviation occurs.

[0032] To further illustrate the present application, the specific technical implementation of each step in the above method process will be described in detail below.

[0033] The specific implementation of step S1 is described as follows: the three-dimensional position of the robot hand in the preset world coordinate system is obtained through global vision, providing an initial position reference for subsequent fine positioning and motion planning. In a specific embodiment, the following steps are included: A head camera module 10 installed on the head of the robot acquires real-time working scene space images containing the robot hand and the entire musical instrument. In a specific embodiment, the head camera module 10 can be a high-precision camera with a 90° wide-angle lens, with an image resolution of 1920x1080 and a frame rate of 60fps. The installation angle is about 45° to the ground, and the distance to the robot hand is kept at about 0.4 to 0.6 meters to ensure a stable global view.

[0034] The acquired image frames are processed to identify the robot hand and obtain its position on the two-dimensional image plane by using a pre-trained hand detection model based on a convolutional neural network (CNN). The pre-trained hand detection model analyzes the image frames and outputs a two-dimensional bounding box of the hand region, from which the pixel coordinates of the center point are determined The specific network structure and training method of the detection model belong to the known technology in the art, and will not be described here.

[0035] According to the pixel coordinates , the three-dimensional space coordinates of the robot hand in the world coordinate system are calculated . The calculation process is based on the pinhole imaging model of the camera, combined with the calibrated camera parameters and scene depth information. The projection relationship of a point in the world coordinate system to its pixel coordinates on the camera image plane can be represented by the following formula: ; wherein, is the depth value of the hand target point in the camera coordinate system; represents the pixel coordinate of the hand target point in the head camera image plane; is the intrinsic matrix of the head camera module 10, in the form of wherein is the normalized focal length of the camera in the axis direction, is the principal point coordinate; and are respectively the rotation matrix and the translation vector from the world coordinate system to the camera coordinate system, which together constitute the extrinsic parameters of the camera.

[0036] The intrinsic matrix of the camera and the extrinsic parameters can be obtained through a pre-camera calibration process. For specific methods of camera calibration, they belong to the known technology in the art, and will not be described here.

[0037] From the pixel coordinates the process of calculating the three-dimensional world coordinates is the inverse solution of the above projection relationship, and the specific calculation formula is: ; wherein, and are respectively the inverse matrix of the rotation matrix and the intrinsic matrix ; represents the hand rough three-dimensional coordinate vector.

[0038] In the calculation, the acquisition of the depth value can be realized by various methods. If the head camera module 10 itself is a depth camera (such as a time-of-flight camera ToF or a structured light camera), the depth value can be directly measured. If the head camera module 10 is a binocular camera system, the depth value can be calculated by a stereo matching algorithm. If it is a monocular camera, the depth value can be obtained by a monocular depth estimation algorithm based on deep learning, or estimated according to the kinematic model of the robot.

[0039] After obtaining the depth value , the three-dimensional coordinates of the hand in the world coordinate system can be calculated by the inverse solution formula. The three-dimensional coordinates are the output hand rough positioning results.

[0040] The specific implementation of step S2 can be executed in parallel with step S1, using close-range, multi-view local vision to accurately identify and locate the target object that the robot fingers are about to interact with, such as a specific key on a piano or a specific string on a guitar. In one specific embodiment, the following steps are included: A fingertip camera module 20, installed on the tip of the robot's finger, acquires local images of the target object. This module includes multiple miniature cameras with an image resolution of 1280×720 and a frame rate of 120fps, and is equipped with a lens with a 30° field of view. Each camera is approximately 10mm away from the corresponding fingertip, enabling close-range, detailed observation of the target.

[0041] For the acquired local images The process involves identifying target controls and obtaining their positions on a two-dimensional image plane. This process utilizes a lightweight visual Transformer model to identify specific target controls (e.g., the outline of piano keys, the center line of a string, or the geometric center of a function key) and outputs the precise pixel coordinates of the target on the two-dimensional image. The specific network structure and training methods for this type of visual Transformer model are well-known technologies in this field and will not be elaborated upon here.

[0042] The three-dimensional spatial coordinates of the target control component in the world coordinate system are calculated based on the pixel coordinates. This solution process is based on the calibrated intrinsic and extrinsic parameters and depth information of each fingertip camera. It is the inverse solution of the camera projection model, and its specific calculation formula is as follows: ; in, The depth value of the target control component in the fingertip camera coordinate system; Represents the target's fine-grained three-dimensional coordinate vector; This indicates the precise pixel coordinates of the target control element on the image plane of the fingertip camera; This is the intrinsic parameter matrix corresponding to the fingertip camera; and These are the rotation matrix and translation vector from the world coordinate system to the coordinate system of the fingertip camera, which together constitute the extrinsic parameters of the fingertip camera.

[0043] The three-dimensional coordinates of the target control component in the world coordinate system can be calculated using the inverse solution formula described above. The three-dimensional coordinates This refers to the output of the fine-grained target localization result, which provides input for the fusion of multiple visual information.

[0044] The specific implementation of step S3 involves fusing the acquired coarse positioning information and the acquired fine positioning information to calculate the precise position of the fingertip relative to the target button, and then planning the movement trajectory accordingly. In one specific implementation, this includes the following steps: External parameters need to be calibrated to establish a unified coordinate system. Before performing fusion calculations, the head camera coordinate system and the fingertip camera coordinate system need to be aligned to the same Cartesian coordinate system (e.g., the world coordinate system). The transformation matrix between the two coordinate systems needs to be obtained through pre-calibration. .

[0045] The calibration process specifically involves the following steps: In the offline calibration phase, a calibration object with a known three-dimensional geometry (such as a checkerboard pattern or a 3D calibration board of a specific shape) is simultaneously placed within the field of view of both the head-mounted camera and the fingertip camera. Feature point matching is then used to identify and match features within the images from both cameras. For each camera with the same name, since the 3D structure of the calibration object is known, the camera pose relative to the calibration object can be calculated based on its projection in the 2D image. Based on this pose, the 3D coordinates of each feature point in the current camera coordinate system can be calculated to obtain the 3D coordinates of the 3D feature points. The three-dimensional coordinates of each matching point pair in the head-mounted camera coordinate system and three-dimensional coordinates in the fingertip camera coordinate system .

[0046] Geometric constraints are established based on the coordinates of these 3D matching points, and these point pairs satisfy the following relationship. Due to the existence of measurement errors, the problem shifts to minimizing the Euclidean distance error between all 3D point pairs, which is the least squares optimization problem. Its objective function can be defined as: in, It is the transformation matrix to be solved; For the first The three-dimensional matching points are in the target coordinate system ( Three-dimensional coordinates in a coordinate system; For the first Three-dimensional matching points in the source coordinate system ( Three-dimensional coordinates in a coordinate system; This indicates that the points in the source coordinate system By transforming the matrix Transform (or project) to the target coordinate system middle; Let be the total number of 3D matching point pairs used for computation. This least-squares optimization problem is solved using standard methods such as Singular Value Decomposition (SVD) to obtain the optimal local pose estimate, i.e., the transformation matrix. The specific algorithms for solving this type of point cloud registration problem are well-known technologies in this field and will not be elaborated here.

[0047] Local two-dimensional target coordinates Convert to three-dimensional coordinates in the world coordinate system The calculation is performed by utilizing the intrinsic and extrinsic parameters of each fingertip camera as well as depth information.

[0048] Perform visual fusion calculations to obtain the fused relative position of the robot's finger with respect to the target button. By employing a dual-camera spatial coordinate fusion algorithm, a dynamic weight fusion algorithm is used to combine coarse and fine positioning information. The specific fusion calculation can be expressed by the following formula: ; in, This is the relative position vector between the finger and the target obtained after fusion; This is the approximate position vector of the hand in the world coordinate system. For the target's refined three-dimensional coordinate vector; The transformation matrix obtained from the calibration, connecting the head camera coordinate system and the fingertip camera coordinate system; These are dynamic weight parameters.

[0049] Dynamic weight parameters The value range of this parameter can be 0.3-0.7 in one embodiment. This dynamic weight parameter is dynamically adjusted according to the quality of the real-time acquired images. By evaluating the sharpness of the fingertip camera image or the confidence level of the target detection result in real time, the system will increase the weight when the image quality of the fingertip camera deteriorates due to motion blur or occlusion. The value of relies more heavily on the coarse positioning results from the head camera. Conversely, when the image from the fingertip camera is clear and the detection is stable, the system reduces . The value is used to make greater use of the high-precision positioning results provided by the fingertip camera.

[0050] The fused location information is filtered and smoothed to achieve the desired fusion location. The sequence contains noise or slight jitter. To output a smooth and stable trajectory for the moving target, a Kalman filter is used. The time series data is processed. The Kalman filter effectively filters out measurement noise and makes the optimal estimate of the target's motion state by predicting the update iterative process, ultimately outputting a smooth trajectory point sequence.

[0051] To generate the final motion trajectory, the control module 50 uses the target position point sequence smoothed by Kalman filtering as a basis to plan the motion trajectory from the current actual position of the finger to the target playing position in Cartesian space. The trajectory planning uses algorithms such as fifth-order polynomial interpolation to ensure that the generated trajectory is continuously changing in position, velocity and acceleration.

[0052] The specific implementation of step S4 involves converting the planned motion trajectory in Cartesian space into low-level control commands that the robot actuator 60 can execute, thereby driving the robot to complete the physical performance action. In one specific implementation, this includes the following steps: Inverse kinematics calculations are performed on the trajectory points in Cartesian space. The control module 50 receives a sequence of smooth trajectory points. For each target pose point on the trajectory, the inverse kinematics solver calculates the set of target angles required for all joints of the robot arm and dexterous hand to rotate so that the robot's fingertips reach the target pose point. ... Simultaneously, based on the time interval between trajectory points (e.g., based on a 5ms update cycle), the target angular velocity that each joint needs to achieve is calculated. The specific algorithms for solving the inverse kinematics of robots, such as iterative or analytical methods based on the Jacobian matrix, are well-known techniques in this field and will not be elaborated upon here.

[0053] A digital control package (CMD) is generated and packaged. The calculated joint target angle, angular velocity, and desired torque information are packaged into a predetermined format. In one embodiment, the specific format of the CMD may be: ; in, For digital control packages; A unique identifier for the target joint or its corresponding servo motor; The target angle for this joint; The target angular velocity of this joint; The desired torque to be applied to the joint.

[0054] The desired torque In a specific embodiment of the present invention, the determination method can be a feedforward compensation torque based on a robot dynamics model, which can be calculated using the following formula: ; in, This indicates the desired torque applied to the joint; Represents the angle vectors of all joints of the robot; This represents the angular acceleration vector of all joints of the robot; Here is the robot's inertia matrix; For Coriolis force and centrifugal force terms; As a gravity term, by calculating this feedforward torque, the robot's internal dynamic effects can be compensated, improving trajectory tracking accuracy. In another embodiment, It can also be the output of a force controller (such as a PID controller) used to control the contact force at the fingertips during contact tasks.

[0055] Control commands are sent to drive the robot actuator 60. The encapsulated digital control package (CMD) is transmitted by the control module 50 to the corresponding joint driver in the robot actuator 60 via an industrial communication bus. In a preferred embodiment, the communication bus can be a CAN bus, and the control command transmission cycle is once every 5 milliseconds.

[0056] The execution unit in the robot actuator 60, such as a servo motor, receives and parses the control command. In a specific embodiment, a servo motor with a response time of less than 80ms (e.g., a servo motor of model MX-106) can be used as the joint execution unit. The motor driver controls the rotation of the motor according to the target parameters in the command, thereby driving the movement of each joint of the robot to complete the planned performance action.

[0057] See attached document Figure 3 The specific implementation of step S5 is described below. During the execution of the playing action, a closed-loop correction mechanism based on visual feedback is provided to ensure the real-time accuracy of the robot's finger movements and to compensate for external disturbances or model errors. In a specific implementation, the following steps are included: During the performance, the vision system (including the head camera module 10 and the fingertip camera module 20) continuously acquires images at a high frequency and calculates the actual position of the robot's fingertips in real time through a fusion algorithm. The control module 50 compares the actual position with the desired target position at the current moment. By comparing the planned trajectories, the Cartesian spatial position error vector between the two can be calculated. .

[0058] Determine the magnitude of the actual position error vector; when the magnitude of the error... If the preset trigger threshold is exceeded, the error correction process will be initiated.

[0059] The proportional-derivative (PD) controller is used to calculate the correction in Cartesian space. This correction process generates a position correction vector to correct the deviation through the PD controller. The specific calculation formula is as follows: ; in, The calculated Cartesian space position correction vector; This is the calculated position error vector at the current moment; The rate of change of position error is approximated by dividing the difference between the error at the current time and the error at the previous time by the time interval. and These are the proportional gain coefficient and the differential gain coefficient, respectively. These two parameters can be tuned experimentally to ensure the dynamic stability of the robot under different motion speeds and force conditions.

[0060] Convert the Cartesian space correction to joint space correction and update the control commands. The calculated Cartesian space position correction... The Jacobian matrix of the robot needs to be used. Convert to joint space to obtain the joint angle correction amount. The transformation relationship can be calculated using the following formula: ; in, Indicates the amount of joint angle correction; This represents the Cartesian spatial position correction. For the robot at the current joint angle The inverse or pseudo-inverse of the Jacobian matrix.

[0061] The obtained joint angle correction amount Update the control commands sent to actuator 160. Specifically, the new target joint angle. From the original target angle With correction amount Synthesis, for example The control package (CMD) containing the new target angle will replace the original instructions and be sent to the corresponding servo motor via the CAN bus.

[0062] From the moment the vision system detects the positional deviation to the control module 50 updating and issuing the correction command, the typical system response time is less than 180ms, thus ensuring the speed and effectiveness of the correction action.

[0063] In one specific embodiment, the Enya CyberG electric guitar is used as the playing object. The playing object has a stringless structure. Its left hand part is equipped with several function buttons or touch panels (such as for triggering chords, rhythms, accompaniment, switching modes, etc.), and the right hand part is an illuminated keyboard for playing melodies or main vocal parts. The triggering of the keyboard and buttons is achieved by outputting digital electrical signals or touch signals.

[0064] The indoor light intensity is approximately 500 lux, the background noise is approximately 40 dB, there is no significant mechanical interference, and the temperature and humidity are maintained at a normal room temperature.

[0065] The robot adopts the same multi-finger camera arrangement as in this embodiment. Each fingertip is equipped with a miniature camera (e.g., OV9281 model, 720p@120fps) for local precise positioning; a wide-angle high-definition camera (e.g., Sony IMX219 model, 1080p@60fps, 90° field of view) is installed on the head for global coarse positioning. The head camera is about 0.5 meters away from the robot's hand, and the installation tilt angle is about 45°.

[0066] The multi-camera synchronization module 30 is used for hardware triggering and software timestamp management to ensure that the image acquisition timing error does not exceed ±80ms.

[0067] Implementation steps: The head-mounted camera roughly defines the overall boundary between the hands and the keyboard; Each fingertip camera identifies the fine features of the target button or key; The algorithm performs synchronous alignment and multi-view information fusion to calculate the precise relative coordinates of the fingertip with respect to the target key. The trajectory is planned according to the target type. For buttons, a vertical pressing trajectory is used, and for keyboards, a continuous pressing or smooth sliding trajectory is used. Drive the robot's actuator to move along a planned trajectory; The deviation between the actual position of the finger and the target trajectory is detected in real time, and the PD controller is triggered to perform real-time correction.

[0068] The experimental data are shown in Table 1: Table 1: Performance Comparison Table for Multi-Camera Solutions in conclusion: As shown in Table 1, in terms of high-precision positioning, the keyboard and button positioning error (MAE) of the proposed solution is consistently within 3mm, and the P95 error (i.e., 95% of the measured value) is also controlled within 6mm. This represents an improvement in positioning accuracy compared to the comparative single-camera solution (error reaching ±10mm). This is thanks to the close-up, unobstructed, and detailed local view provided by the fingertip camera, which compensates for the shortcomings of the global camera, which is easily obstructed by hands and lacks accuracy.

[0069] Regarding real-time error correction: the median error correction response time of 150ms in this invention is significantly faster than the 500ms of the control scheme, achieving faster closed-loop feedback. This response speed (less than the theoretical response time of 180ms) ensures that the system can quickly and dynamically correct deviations caused by model errors or minor disturbances during performance, effectively guaranteeing the accuracy of performance actions.

[0070] Although embodiments of the invention have been shown and described, it will be understood by those skilled in the art that various changes, modifications, substitutions and alterations can be made to these embodiments without departing from the principles and spirit of the invention, the scope of which is defined by the appended claims and their equivalents.

Claims

1. A method for robot hand localization and musical instrument playing control based on multi-camera cameras, characterized in that, Includes the following steps: S1. Use the head camera module to acquire a global image of the working scene, and provide rough position information of the robot's hand based on the global image of the working scene; S2. Use the fingertip camera module set at the end of the robot's finger to acquire a local image of the target to be interacted with, and provide a fine positioning result of the target based on the local image of the target to be interacted with; S3. By fusing the coarse position information with the fine target positioning result, the relative position of the fingertip and the target to be interacted with is calculated, and a motion trajectory is planned based on the fused relative position. S4. Control the robot actuator to execute the motion trajectory to complete the performance action; S5. During the performance of the playing action, the actual position of the robot actuator is continuously acquired through the vision system. The actual position is compared with the motion trajectory to calculate the Cartesian space position error vector. When the magnitude of the Cartesian space position error vector exceeds the trigger threshold, the robot joint angle correction amount is calculated to update the control command to the robot actuator.

2. The method for robot hand localization and musical instrument playing control based on multi-camera according to claim 1, characterized in that, In step S1, the global image based on the working scene provides rough position information of the robot's hand, further including: A hand detection model based on a convolutional neural network is used to analyze the global image of the work scene to determine the two-dimensional pixel coordinates of the hand region; Based on the pinhole imaging model of the camera, and combined with the camera's intrinsic and extrinsic parameters and depth information, the two-dimensional pixel coordinates are calculated into the three-dimensional spatial coordinates of the robot's hand, serving as rough position information.

3. The method for robot hand localization and musical instrument playing control based on multi-camera according to claim 1, characterized in that, In step S2, the step of providing fine target localization results based on the local image of the target to be interacted with further includes: A lightweight visual Transformer model is used to process the local image of the target to be interacted with in order to identify specific target controls and output the precise pixel coordinates of the target controls on the two-dimensional image. Based on the precise pixel coordinates, and combined with the intrinsic and extrinsic parameters and depth information of the fingertip camera, the three-dimensional spatial coordinates of the target control component in the world coordinate system are calculated as the result of fine target positioning.

4. The method for robot hand localization and musical instrument playing control based on multi-camera according to claim 1, characterized in that, In step S3, the step of planning the motion trajectory based on the fused relative position further includes: The transformation matrix connecting the coordinate system of the head camera module and the coordinate system of the fingertip camera module is pre-calibrated; The fused relative position is calculated by using a dynamic weight fusion algorithm, combining the coarse position information, the fine target positioning result, and the transformation matrix. The time series of the fused relative positions are smoothed using a Kalman filter, and a motion trajectory is generated based on the smoothed trajectory point sequence.

5. The method for robot hand localization and musical instrument playing control based on multi-camera according to claim 1, characterized in that, In step S4, the step of controlling the robot actuator to execute the motion trajectory to complete the performance action further includes: Inverse kinematics calculations are performed on the target pose points on the motion trajectory to calculate the target angles and target angular velocities of each joint of the robot. Calculation of feedforward compensation torque based on robot dynamics model; The target angle, the target angular velocity, and the feedforward compensation torque are encapsulated into a digital control package and sent to the robot actuator via an industrial communication bus.

6. The method for robot hand localization and musical instrument playing control based on multi-camera according to claim 1, characterized in that, In step S5, calculating the robot joint angle correction amount to update the control commands to the robot actuator further includes: A proportional-derivative controller is used to calculate the Cartesian spatial position correction vector based on the Cartesian spatial position error vector and the rate of change. The Cartesian space position correction vector is transformed into joint space by using the inverse or pseudo-inverse of the robot's Jacobian matrix to obtain the joint angle correction amount. The joint angle correction amount is combined with the original target angle to generate a new target joint angle and update the control command.

7. The method for robot hand localization and musical instrument playing control based on multi-camera according to claim 4, characterized in that, The dynamic weight fusion algorithm specifically includes: Set dynamic weight parameters, and dynamically adjust the dynamic weight parameters according to the image quality or the confidence level of the target detection results acquired in real time by the fingertip camera module; When the image quality of the fingertip camera module deteriorates due to motion blur or occlusion, the value of the dynamic weight parameter is increased to rely on the coarse position information.

8. The method for robot hand localization and musical instrument playing control based on multi-camera according to claim 1, characterized in that, The method further includes: The multi-camera synchronization module is used to control the unified triggering and timestamp management of each camera in the fingertip camera module to ensure that the coarse location information is synchronized with the fine target positioning result in time.

9. A multi-camera robot hand positioning and musical instrument playing control system, applied to the multi-camera robot hand positioning and musical instrument playing control method according to any one of claims 1-8, characterized in that, include: The head camera module is used to acquire a global image of the working scene and provide rough position information of the robot's hand; The fingertip camera module, located at the tip of the robot's finger, is used to acquire local images of the target to be interacted with and provide fine-grained target localization results; The visual fusion module is used to fuse the coarse position information with the fine target positioning result and calculate the fused relative position between the fingertip and the target to be interacted with. The control module is used to plan the motion trajectory based on the fused relative positions and generate control commands. A robotic actuator is used to receive and execute the control commands to complete the performance actions; The feedback and correction module is used to continuously acquire the actual position of the robot actuator through the vision system during the execution of the performance action, compare the actual position with the motion trajectory to calculate the Cartesian space position error vector, and calculate the joint angle correction amount to update the control command when the magnitude of the Cartesian space position error vector exceeds the trigger threshold.

10. The multi-camera robot hand positioning and musical instrument playing control system according to claim 9, characterized in that, The robot actuator includes a multi-degree-of-freedom robotic arm and an end effector.

Images