Multi-camera robot hand positioning and musical instrument playing control method and system

Abstract

The application relates to the technical field of robot hand control, and discloses a robot hand positioning and musical instrument playing control method and system based on multiple cameras, the method comprising the following steps: a head camera acquires a global image to provide a rough hand position; a fingertip camera acquires a local image to provide target fine positioning; coarse and fine positioning are fused to calculate a relative position and plan a motion trajectory; a robot actuator is controlled to execute the trajectory to complete playing; an actual position is monitored to calculate an error, and if the error exceeds a threshold, a joint is corrected; the system comprises a head camera module, a fingertip camera module, a visual fusion module, a control module, a robot actuator and a feedback and correction module. The application realizes effective combination of coarse positioning and fine positioning by acquiring a high-resolution local image and combining global rough position information provided by the head camera module, and effectively overcomes the problem that single global vision is easily blocked by the hand itself or has insufficient precision due to a long distance.

Description

Technical Field

[0001] This invention relates to the field of robot hand control technology, specifically to a method and system for robot hand positioning and musical instrument playing control based on multi-camera cameras. Background Technology

[0002] Robotic systems are increasingly being applied in fields requiring high precision and flexibility, such as musical instrument performance. These tasks place extremely high demands on the robot's perception and control capabilities. The system not only needs to accurately perceive the coordinates of fine targets on the instrument (such as keys, buttons, or strings), but must also be able to dynamically and accurately control its end effector (such as a dexterous hand) to complete complex playing actions. Therefore, high-precision visual positioning capabilities and high-dynamic-response control capabilities are the key technological foundations for achieving high-level robotic performance.

[0003] In existing robotic applications, visual localization is the primary technological approach for achieving environmental perception. One common approach is global vision, which involves placing cameras outside the robot's workspace or on its head to capture the overall scene, including the robot and the musical instrument. Another approach is to integrate miniature cameras on the robot's end effector, such as the fingers of a dexterous hand, to obtain high-resolution, close-up images. At the control strategy level, many robotic systems rely on open-loop control. This approach typically plans a complete motion trajectory based on the initial localization results provided by the vision system, and then drives the robot's actuators to complete that trajectory.

[0004] However, existing robot performance control technologies, which employ global vision solutions, are limited in positioning accuracy by the camera's resolution. Furthermore, when the robot's hand approaches and manipulates the instrument, the hand itself easily obstructs key interactive targets, leading to positioning failure. While using a single fingertip camera offers high resolution, its narrow field of view results in the loss of global coordinate information, making it difficult to independently support complete motion planning. During high-speed movements of the robot's fingers, images captured by the fingertip camera are prone to severe motion blur, reducing positioning stability. Moreover, at the control level, the reliance on open-loop execution is extremely sensitive to calibration errors or minor environmental disturbances; any small model deviation accumulates during execution, ultimately leading to operational failure. Therefore, this invention provides a multi-camera-based robot hand positioning and instrument performance control method and system to address the shortcomings of existing technologies. Summary of the Invention

[0005] To address the shortcomings of existing technologies, this invention provides a method and system for robot hand localization and musical instrument playing control based on multi-camera cameras. This solves the problems of low positioning accuracy and slow closed-loop correction response caused by factors such as hand occlusion and motion blur when dealing with instruments with complex structures or requiring multi-finger coordination, which mostly rely on a single visual source.

[0006] To achieve the above objectives, the present invention provides the following technical solution: The first aspect of this invention provides a method for robot hand localization and musical instrument playing control based on multi-camera cameras, comprising the following steps: S1. Use the head camera module to acquire a global image of the working scene, and provide rough position information of the robot's hand based on the global image of the working scene; S2. Use the fingertip camera module set at the end of the robot's finger to acquire a local image of the target to be interacted with, and provide a fine positioning result of the target based on the local image of the target to be interacted with; S3. By fusing the coarse position information with the fine target positioning result, the relative position of the fingertip and the target to be interacted with is calculated, and a motion trajectory is planned based on the fused relative position. S4. Control the robot actuator to execute the motion trajectory to complete the performance action; S5. During the performance of the playing action, the actual position of the robot actuator is continuously acquired through the vision system. The actual position is compared with the motion trajectory to calculate the Cartesian space position error vector. When the magnitude of the Cartesian space position error vector exceeds the trigger threshold, the robot joint angle correction amount is calculated to update the control command to the robot actuator.

[0007] Preferably, in step S1, the global image based on the working scene provides coarse positional information of the robot's hand, further including: A hand detection model based on a convolutional neural network is used to analyze the global image of the work scene to determine the two-dimensional pixel coordinates of the hand region; Based on the pinhole imaging model of the camera, and combined with the camera's intrinsic and extrinsic parameters and depth information, the two-dimensional pixel coordinates are calculated into the three-dimensional spatial coordinates of the robot's hand, serving as rough position information.

[0008] Preferably, in step S2, the step of providing fine target localization results based on the local image of the target to be interacted with further includes: A lightweight visual Transformer model is used to process the local image of the target to be interacted with in order to identify specific target controls and output the precise pixel coordinates of the target controls on the two-dimensional image. Based on the precise pixel coordinates, and combined with the intrinsic and extrinsic parameters and depth information of the fingertip camera, the three-dimensional spatial coordinates of the target control component in the world coordinate system are calculated as the result of fine target positioning.

[0009] Preferably, in step S3, the step of planning the motion trajectory based on the fused relative position further includes: The transformation matrix connecting the coordinate system of the head camera module and the coordinate system of the fingertip camera module is obtained through pre-calibration; The fused relative position is calculated by using a dynamic weight fusion algorithm, combining the coarse position information, the fine target positioning result, and the transformation matrix. The time series of the fused relative positions are smoothed using a Kalman filter, and a motion trajectory is generated based on the smoothed trajectory point sequence.

[0010] Preferably, in step S4, the step of controlling the robot actuator to execute the motion trajectory to complete the performance action further includes: Inverse kinematics calculations are performed on the target pose points on the motion trajectory to calculate the target angles and target angular velocities of each joint of the robot. Calculation of feedforward compensation torque based on robot dynamics model; The target angle, the target angular velocity, and the feedforward compensation torque are encapsulated into a digital control package and sent to the robot actuator via an industrial communication bus.

[0011] Preferably, in step S5, calculating the robot joint angle correction amount to update the control commands to the robot actuator further includes: A proportional-derivative controller is used to calculate the Cartesian spatial position correction vector based on the Cartesian spatial position error vector and the rate of change. The Cartesian space position correction vector is transformed into joint space by using the inverse or pseudo-inverse of the robot's Jacobian matrix to obtain the joint angle correction amount. The joint angle correction amount is combined with the original target angle to generate a new target joint angle and update the control command.

[0012] Preferably, the dynamic weight fusion algorithm specifically includes: Set dynamic weight parameters, and dynamically adjust the dynamic weight parameters according to the image quality or the confidence level of the target detection results acquired in real time by the fingertip camera module; When the image quality of the fingertip camera module deteriorates due to motion blur or occlusion, the value of the dynamic weight parameter is increased to rely on the coarse position information.

[0013] Preferably, the method further includes: The multi-camera synchronization module is used to control the unified triggering and timestamp management of each camera in the fingertip camera module to ensure that the coarse location information is synchronized with the fine target positioning result in time.

[0014] A second aspect of the present invention provides a multi-camera robot hand positioning and musical instrument playing control system, comprising: The head camera module is used to acquire a global image of the working scene and provide rough position information of the robot's hand; The fingertip camera module, located at the tip of the robot's finger, is used to acquire local images of the target to be interacted with and provide fine-grained target localization results; The visual fusion module is used to fuse the coarse position information with the fine target positioning result and calculate the fused relative position between the fingertip and the target to be interacted with. The control module is used to plan the motion trajectory based on the fused relative positions and generate control commands. A robotic actuator is used to receive and execute the control commands to complete the performance actions; The feedback and correction module is used to continuously acquire the actual position of the robot actuator through the vision system during the execution of the performance action, compare the actual position with the motion trajectory to calculate the Cartesian space position error vector, and calculate the joint angle correction amount to update the control command when the magnitude of the Cartesian space position error vector exceeds the trigger threshold.

[0015] Preferably, the robot actuator includes a multi-degree-of-freedom robotic arm and an end effector.

[0016] This invention provides a method and system for robot hand localization and musical instrument playing control based on multi-camera cameras. It has the following beneficial effects: 1. This invention achieves an effective combination of coarse and fine positioning by acquiring high-resolution local images through a fingertip camera module located at the end of the robot's finger, and combining this with global coarse position information provided by a head camera module. This effectively overcomes the technical shortcomings of relying solely on global vision, which is easily obstructed by the hand itself or lacks accuracy due to distance, thus improving the positioning accuracy of the robot's fingertip relative to the target to be interacted with.

[0017] 2. This invention employs a dynamic weighted fusion algorithm, dynamically adjusting the fusion weights of coarse location information and fine target localization results based on the image quality or target detection confidence level acquired in real-time by the fingertip camera module. When the image quality of the fingertip camera deteriorates due to motion blur or occlusion, the system can increase its reliance on the coarse localization information from the head camera, thereby avoiding localization interruptions or errors caused by temporary local visual failures and improving the system's robustness in complex dynamic environments.

[0018] 3. This invention introduces vision-based real-time closed-loop feedback during the performance process. It continuously acquires the actual position of the robot's actuator and compares it with the planned motion trajectory to calculate the deviation. When the deviation exceeds a threshold, a correction calculation based on proportional-derivative control and the Jacobian matrix is ​​immediately initiated. This high-frequency closed-loop correction mechanism can quickly compensate for motion errors caused by model inaccuracies or minor external disturbances, ensuring the real-time accuracy of the performance and the precision of trajectory tracking. Attached Figure Description

[0019] Figure 1 This is a system architecture diagram of the present invention; Figure 2 This is a flowchart of the method steps of the present invention; Figure 3 This is a control flowchart for the closed-loop correction of the present invention.

[0020] Among them, 10 is the head camera module; 20 is the fingertip camera module; 30 is the multi-camera synchronization module; 40 is the visual fusion module; 50 is the control module; 60 is the robot actuator; and 70 is the feedback and correction module. Detailed Implementation

[0021] The technical solutions in the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0022] See attached document Figure 1 , Figure 1 This is a system architecture diagram of one embodiment of the present invention. The present invention provides a multi-camera robot hand positioning and musical instrument playing control system, including a head camera module 10, a fingertip camera module 20, a multi-camera synchronization module 30, a visual fusion module 40, a control module 50, a robot actuator 60, and a feedback and correction module 70.

[0023] The head camera module 10 is used to acquire a global image of the working scene, including the robot's hand and musical instruments, and to provide rough position information of the robot's hand accordingly.

[0024] The fingertip camera module 20 includes multiple cameras set at the tip of the robot's finger for acquiring high-resolution local images of the target to be interacted with.

[0025] The multi-camera synchronization module 30 is used to control the unified triggering and timestamp management of each camera in the fingertip camera module 20. In one specific embodiment, the multi-camera synchronization module 30 ensures that the timing error of image acquisition by each camera does not exceed ±80ms, providing a time synchronization benchmark for subsequent data fusion.

[0026] The visual fusion module 40 is used to receive global image information from the head camera module 10 and local image information from the fingertip camera module 20 after synchronous processing, and to fuse multi-source information based on a unified timestamp to calculate the relative coordinates between the fingertip and the target.

[0027] The control module 50 is used to plan the motion path based on the coordinate information output by the vision fusion module 40 and generate control commands that the robot actuator 60 can execute.

[0028] Robotic actuators 60, such as multi-degree-of-freedom robotic arms and end effectors, are used to receive and execute control commands from control module 50 to complete physical operations.

[0029] The feedback and correction module 70 is used to continuously acquire the actual motion state of the robot actuator 60 through the vision system during the movement of the robot actuator 60, compare it with the target trajectory planned by the control module 50, and start correction calculation when deviation occurs.

[0030] During system operation, the head camera module 10 and the fingertip camera module 20 collect data in parallel. The data acquisition process of the fingertip camera module 20 is controlled by the multi-camera synchronization module 30. The acquired data stream is transmitted to the visual fusion module 40 for processing. The calculation results of the visual fusion module 40 are output to the control module 50 for trajectory planning and instruction generation. The generated instructions are sent to the robot actuator 60. During the movement of the actuator, the feedback and correction module 70 continuously operates, feeding back correction signals to the control module 50 to dynamically adjust the movement of the robot actuator 60.

[0031] See attached document Figure 2 , Figure 2 This is a flowchart of a method according to an embodiment of the present invention. The present invention provides a method for robot hand localization and musical instrument playing control based on a multi-camera system, comprising the following steps: S1. Use the head camera to perform coarse positioning of the robot's hand; S2. Use the fingertip camera to precisely locate the target button; S3. It integrates the recognition information from the head camera and the fingertip camera to generate the movement trajectory of the finger from the target position; S4. Control the robot's fingers to perform playing actions; S5. Real-time monitoring during the performance, and corrections if deviations occur.

[0032] To further clarify the present invention, the specific technical implementation of each step in the above method flow will be described in detail below.

[0033] The specific implementation of step S1 involves acquiring the three-dimensional position of the robot's hand in a preset world coordinate system through global vision, providing an initial position reference for subsequent fine positioning and motion planning. In one specific implementation, this includes the following steps: A head-mounted camera module 10, installed on the robot's head, captures real-time images of the workspace, including the robot's hand and the musical instrument. In one specific embodiment, the head-mounted camera module 10 can be a high-precision camera with a 90° wide-angle lens, an image resolution of 1920×1080, a frame rate of 60fps, an installation angle of approximately 45° to the ground, and a distance of approximately 0.4 to 0.6 meters from the robot's hand to ensure a stable global field of view.

[0034] The acquired image frames are processed to identify the robot's hand and obtain its position on a two-dimensional image plane by utilizing a pre-trained hand detection model based on a convolutional neural network (CNN). This pre-trained hand detection model analyzes the image frames, outputs a two-dimensional bounding box of the hand region, and thereby determines the pixel coordinates of its center point. The specific network structure and training methods of the detection model are well-known technologies in this field and will not be elaborated upon here.

[0035] Based on pixel coordinates Solve the three-dimensional spatial coordinates of the robot's hand in the world coordinate system. This solution process is based on the camera's pinhole imaging model, and combines calibrated camera parameters with scene depth information to determine the three-dimensional coordinates of a point in the world coordinate system. To its pixel coordinates on the camera image plane The projection relationship can be expressed by the following formula: ; in, This represents the depth value of the hand target point in the camera coordinate system. This represents the pixel coordinates of the hand target point on the image plane of the head-mounted camera; The intrinsic parameter matrix of the head-mounted camera module 10 is in the form of: ,in For the camera Normalized focal length along the axial direction, Principal point coordinates; and These are the rotation matrix and translation vector from the world coordinate system to the camera coordinate system, which together constitute the camera's extrinsic parameters.

[0036] Camera intrinsic parameter matrix With external references This can be achieved through a pre-calibrated camera process. The specific methods for camera calibration are well-known in the field and will not be elaborated upon here.

[0037] From pixel coordinates The process of calculating 3D world coordinates is the inverse solution of the above projection relationship, and its specific calculation formula is as follows: ; in, and Rotation matrices and intrinsic parameter matrix The inverse matrix; This represents a rough three-dimensional coordinate vector of the hand.

[0038] Depth value during calculation The acquisition can be achieved through various methods. If the head camera module 10 itself is a depth camera (e.g., a Time-of-Flight (ToF) camera or a structured light camera), then It can be directly measured. If the head camera module 10 is a binocular camera system, then It can be calculated using a stereo matching algorithm. If it's a monocular camera, It can be obtained through a monocular depth estimation algorithm based on deep learning, or estimated based on the robot's kinematics model.

[0039] Obtaining depth values Then, the three-dimensional coordinates of the hand in the world coordinate system can be calculated using the inverse kinematics formula. The three-dimensional coordinates This is the output coarse positioning result of the hand.

[0040] The specific implementation of step S2 can be executed in parallel with step S1. It utilizes close-range, multi-view local vision to accurately identify and locate the target object that the robot's finger is about to interact with, such as a specific key on a piano or a specific string on a guitar. In one specific implementation, it includes the following steps: A fingertip camera module 20, installed on the tip of the robot's finger, acquires local images of the target object. This module includes multiple miniature cameras with an image resolution of 1280×720 and a frame rate of 120fps, and is equipped with a lens with a 30° field of view. Each camera is approximately 10mm away from the corresponding fingertip, enabling close-range, detailed observation of the target.

[0041] For the acquired local images The process involves identifying target controls and obtaining their positions on a two-dimensional image plane. This process utilizes a lightweight visual Transformer model to identify specific target controls (e.g., the outline of piano keys, the center line of a string, or the geometric center of a function key) and outputs the precise pixel coordinates of the target on the two-dimensional image. The specific network structure and training methods for this type of visual Transformer model are well-known technologies in this field and will not be elaborated upon here.

[0042] The three-dimensional spatial coordinates of the target control component in the world coordinate system are calculated based on the pixel coordinates. This solution process is based on the calibrated intrinsic and extrinsic parameters and depth information of each fingertip camera. It is the inverse solution of the camera projection model, and its specific calculation formula is as follows: ; in, The depth value of the target control component in the fingertip camera coordinate system; Represents the target's fine-grained three-dimensional coordinate vector; This indicates the precise pixel coordinates of the target control element on the image plane of the fingertip camera; This is the intrinsic parameter matrix corresponding to the fingertip camera; and These are the rotation matrix and translation vector from the world coordinate system to the coordinate system of the fingertip camera, respectively, which together constitute the extrinsic parameters of the fingertip camera.

[0043] The three-dimensional coordinates of the target control component in the world coordinate system can be calculated using the inverse solution formula described above. The three-dimensional coordinates This refers to the output of the fine-grained target localization result, which provides input for the fusion of multiple visual information.

[0044] The specific implementation of step S3 involves fusing the acquired coarse positioning information and the acquired fine positioning information to calculate the precise position of the fingertip relative to the target button, and then planning the movement trajectory accordingly. In one specific implementation, this includes the following steps: External parameters need to be calibrated to establish a unified coordinate system. Before performing fusion calculations, the head camera coordinate system and the fingertip camera coordinate system need to be aligned to the same Cartesian coordinate system (e.g., the world coordinate system). The transformation matrix between the two coordinate systems needs to be obtained through pre-calibration. .

[0045] The calibration process specifically involves the following steps: In the offline calibration phase, a calibration object with a known three-dimensional geometry (such as a checkerboard pattern or a 3D calibration board of a specific shape) is simultaneously placed within the field of view of both the head-mounted camera and the fingertip camera. Feature point matching is then used to identify and match features within the images from both cameras. For each camera with the same name, since the 3D structure of the calibration object is known, the camera pose relative to the calibration object can be calculated based on its projection in the 2D image. Based on this pose, the 3D coordinates of each feature point in the current camera coordinate system can be calculated to obtain the 3D coordinates of the 3D feature points. The three-dimensional coordinates of each matching point pair in the head-mounted camera coordinate system and three-dimensional coordinates in the fingertip camera coordinate system .

[0046] Geometric constraints are established based on the coordinates of these 3D matching points, and these point pairs satisfy the following relationship. Due to the existence of measurement errors, the problem shifts to minimizing the Euclidean distance error between all 3D point pairs, which is the least squares optimization problem. Its objective function can be defined as: ; in, It is the transformation matrix to be solved; For the first The three-dimensional matching points are in the target coordinate system ( Three-dimensional coordinates in a coordinate system; For the first Three-dimensional matching points in the source coordinate system ( Three-dimensional coordinates in a coordinate system; This indicates that the points in the source coordinate system By transforming the matrix Transform (or project) to the target coordinate system middle; Let be the total number of 3D matching point pairs used for computation. This least-squares optimization problem is solved using standard methods such as Singular Value Decomposition (SVD) to obtain the optimal local pose estimate, i.e., the transformation matrix. The specific algorithms for solving this type of point cloud registration problem are well-known technologies in this field and will not be elaborated here.

[0047] Local two-dimensional target coordinates Convert to three-dimensional coordinates in the world coordinate system The calculation is performed by utilizing the intrinsic and extrinsic parameters of each fingertip camera as well as depth information.

[0048] Perform visual fusion calculations to obtain the fused relative position of the robot's finger with respect to the target button. By employing a dual-camera spatial coordinate fusion algorithm, a dynamic weight fusion algorithm is used to combine coarse and fine positioning information. The specific fusion calculation can be expressed by the following formula: ; in, This is the relative position vector between the finger and the target obtained after fusion; This is the approximate position vector of the hand in the world coordinate system. For the target's refined three-dimensional coordinate vector; The transformation matrix obtained from the calibration, connecting the head camera coordinate system and the fingertip camera coordinate system; These are dynamic weight parameters.

[0049] Dynamic weight parameters The value range of this parameter can be 0.3-0.7 in one embodiment. This dynamic weight parameter is dynamically adjusted according to the quality of the real-time acquired images. By evaluating the sharpness of the fingertip camera image or the confidence level of the target detection result in real time, the system will increase the weight when the image quality of the fingertip camera deteriorates due to motion blur or occlusion. The value of relies more heavily on the coarse positioning results from the head camera. Conversely, when the image from the fingertip camera is clear and the detection is stable, the system reduces . The value is used to make greater use of the high-precision positioning results provided by the fingertip camera.

[0050] The fused location information is filtered and smoothed to achieve the desired fusion location. The sequence contains noise or slight jitter. To output a smooth and stable trajectory for the moving target, a Kalman filter is used. The time series data is processed. The Kalman filter effectively filters out measurement noise and makes the optimal estimate of the target's motion state by predicting the update iterative process, ultimately outputting a smooth trajectory point sequence.

[0051] To generate the final motion trajectory, the control module 50 uses the target position point sequence smoothed by Kalman filtering as a basis to plan the motion trajectory from the current actual position of the finger to the target playing position in Cartesian space. The trajectory planning uses algorithms such as fifth-order polynomial interpolation to ensure that the generated trajectory is continuously changing in position, velocity and acceleration.

[0052] The specific implementation of step S4 involves converting the planned motion trajectory in Cartesian space into low-level control commands that the robot actuator 60 can execute, thereby driving the robot to complete the physical performance action. In one specific implementation, this includes the following steps: Inverse kinematics calculations are performed on the trajectory points in Cartesian space. The control module 50 receives a sequence of smooth trajectory points. For each target pose point on the trajectory, the inverse kinematics solver calculates the set of target angles required for all joints of the robot arm and dexterous hand to rotate so that the robot's fingertips reach the target pose point. ... Simultaneously, based on the time interval between trajectory points (e.g., based on a 5ms update cycle), the target angular velocity that each joint needs to achieve is calculated. The specific algorithms for solving the inverse kinematics of robots, such as iterative or analytical methods based on the Jacobian matrix, are well-known techniques in this field and will not be elaborated upon here.

[0053] A digital control package (CMD) is generated and packaged. The calculated joint target angle, angular velocity, and desired torque information are packaged into a predetermined format. In one embodiment, the specific format of the CMD may be: ; in, For digital control packages; A unique identifier for the target joint or its corresponding servo motor; The target angle for this joint; The target angular velocity of this joint; The desired torque to be applied to the joint.

[0054] The desired torque In a specific embodiment of the present invention, the determination method can be a feedforward compensation torque based on a robot dynamics model, which can be calculated using the following formula: ; in, This indicates the desired torque applied to the joint; Represents the angle vectors of all joints of the robot; This represents the angular acceleration vector of all joints of the robot; Here is the robot's inertia matrix; For Coriolis force and centrifugal force terms; As a gravity term, by calculating this feedforward torque, the robot's internal dynamic effects can be compensated, improving trajectory tracking accuracy. In another embodiment, It can also be the output of a force controller (such as a PID controller) used to control the contact force at the fingertips during contact tasks.

[0055] Control commands are sent to drive the robot actuator 60. The encapsulated digital control package (CMD) is transmitted by the control module 50 to the corresponding joint driver in the robot actuator 60 via an industrial communication bus. In a preferred embodiment, the communication bus can be a CAN bus, and the control command transmission cycle is once every 5 milliseconds.

[0056] The execution unit in the robot actuator 60, such as a servo motor, receives and parses the control command. In a specific embodiment, a servo motor with a response time of less than 80ms (e.g., a servo motor of model MX-106) can be used as the joint execution unit. The motor driver controls the rotation of the motor according to the target parameters in the command, thereby driving the movement of each joint of the robot to complete the planned performance action.

[0057] See attached document Figure 3 The specific implementation of step S5 is described below. During the execution of the playing action, a closed-loop correction mechanism based on visual feedback is provided to ensure the real-time accuracy of the robot's finger movements and to compensate for external disturbances or model errors. In a specific implementation, the following steps are included: During the performance, the vision system (including the head camera module 10 and the fingertip camera module 20) continuously acquires images at a high frequency and calculates the actual position of the robot's fingertips in real time through a fusion algorithm. The control module 50 compares the actual position with the desired target position at the current moment. By comparing the planned trajectories, the Cartesian spatial position error vector between the two can be calculated. .

[0058] Determine the magnitude of the actual position error vector; when the magnitude of the error... If the preset trigger threshold is exceeded, the error correction process will be initiated.

[0059] The proportional-derivative (PD) controller is used to calculate the correction in Cartesian space. This correction process generates a position correction vector to correct the deviation through the PD controller. The specific calculation formula is as follows: ; in, The calculated Cartesian space position correction vector; This is the calculated position error vector at the current moment; The rate of change of position error is approximated by dividing the difference between the error at the current time and the error at the previous time by the time interval. and These are the proportional gain coefficient and the differential gain coefficient, respectively. These two parameters can be tuned experimentally to ensure the dynamic stability of the robot under different motion speeds and force conditions.

[0060] Convert the Cartesian space correction to joint space correction and update the control commands. The calculated Cartesian space position correction... The Jacobian matrix of the robot needs to be used. Convert to joint space to obtain the joint angle correction amount. The transformation relationship can be calculated using the following formula: ; in, Indicates the amount of joint angle correction; This represents the Cartesian spatial position correction. For the robot at the current joint angle The inverse or pseudo-inverse of the Jacobian matrix.

[0061] The obtained joint angle correction amount Update the control commands sent to actuator 160. Specifically, the new target joint angle. From the original target angle With correction amount Synthesis, for example The control package (CMD) containing the new target angle will replace the original instructions and be sent to the corresponding servo motor via the CAN bus.

[0062] From the moment the vision system detects the positional deviation to the control module 50 updating and issuing the correction command, the typical system response time is less than 180ms, thus ensuring the speed and effectiveness of the correction action.

[0063] In one specific embodiment, the Enya CyberG electric guitar is used as the playing object. The playing object has a stringless structure. Its left hand part is equipped with several function buttons or touch panels (such as for triggering chords, rhythms, accompaniment, switching modes, etc.), and the right hand part is an illuminated keyboard for playing melodies or main vocal parts. The triggering of the keyboard and buttons is achieved by outputting digital electrical signals or touch signals.

[0064] The indoor light intensity is approximately 500 lux, the background noise is approximately 40 dB, there is no significant mechanical interference, and the temperature and humidity are maintained at a normal room temperature.

[0065] The robot adopts the same multi-finger camera arrangement as in this embodiment. Each fingertip is equipped with a miniature camera (e.g., OV9281 model, 720p@120fps) for local precise positioning; a wide-angle high-definition camera (e.g., Sony IMX219 model, 1080p@60fps, 90° field of view) is installed on the head for global coarse positioning. The head camera is about 0.5 meters away from the robot's hand, and the installation tilt angle is about 45°.

[0066] The multi-camera synchronization module 30 is used for hardware triggering and software timestamp management to ensure that the image acquisition timing error does not exceed ±80ms.

[0067] Implementation steps: The head-mounted camera roughly defines the overall boundary between the hands and the keyboard; Each fingertip camera identifies the fine features of the target button or key; The algorithm performs synchronous alignment and multi-view information fusion to calculate the precise relative coordinates of the fingertip with respect to the target key. The trajectory is planned according to the target type. For buttons, a vertical pressing trajectory is used, and for keyboards, a continuous pressing or smooth sliding trajectory is used. Drive the robot's actuator to move along a planned trajectory; The deviation between the actual position of the finger and the target trajectory is detected in real time, and the PD controller is triggered to perform real-time correction.

[0068] The experimental data are shown in Table 1: Table 1: Performance Comparison Table for Multi-Camera Solutions in conclusion: As shown in Table 1, in terms of high-precision positioning, the keyboard and button positioning error (MAE) of the proposed solution is consistently within 3mm, and the P95 error (i.e., 95% of the measured value) is also controlled within 6mm. This represents an improvement in positioning accuracy compared to the comparative single-camera solution (error reaching ±10mm). This is thanks to the close-up, unobstructed, and detailed local view provided by the fingertip camera, which compensates for the shortcomings of the global camera, which is easily obstructed by hands and lacks accuracy.

[0069] Regarding real-time error correction: the median error correction response time of 150ms in this invention is significantly faster than the 500ms of the control scheme, achieving faster closed-loop feedback. This response speed (less than the theoretical response time of 180ms) ensures that the system can quickly and dynamically correct deviations caused by model errors or minor disturbances during performance, effectively guaranteeing the accuracy of performance actions.

[0070] Although embodiments of the invention have been shown and described, it will be understood by those skilled in the art that various changes, modifications, substitutions and alterations can be made to these embodiments without departing from the principles and spirit of the invention, the scope of which is defined by the appended claims and their equivalents.

Claims

1. A method for robot hand localization and musical instrument playing control based on multi-camera cameras, characterized in that, Includes the following steps: S1. Use the head camera module to acquire a global image of the working scene, and provide rough position information of the robot's hand based on the global image of the working scene; S2. Use the fingertip camera module set at the end of the robot's finger to acquire a local image of the target control component to be interacted with, and provide a fine positioning result of the target based on the local image of the target control component to be interacted with; S3. By fusing the obtained coarse position information with the obtained fine target positioning results, the precise position of the fingertip relative to the target control component is calculated, and the motion trajectory is planned accordingly, specifically including: By unifying the head camera coordinate system and the fingertip camera coordinate system into the same Cartesian coordinate system, the transformation matrix between the two coordinate systems is obtained through pre-calibration. Perform visual fusion calculations to obtain the fused relative position of the robot's fingertip with respect to the target control element. The dynamic weighted fusion algorithm is used for fusion calculation, and the formula is as follows: ； In the formula, This is the approximate position vector of the hand in the world coordinate system; For the target control component, a fine three-dimensional coordinate vector is provided. The transformation matrix is... This is a dynamic weighting parameter, with a value range of 0.3-0.7; The image sharpness of the fingertip camera module is evaluated, and the dynamic weight parameter is increased when motion blur or occlusion occurs in the fingertip camera module. The value of the dynamic weight parameter is reduced when the image from the fingertip camera module is clear and the detection is stable. The value; The fused location information is then filtered and smoothed using a Kalman filter. The time series is processed, and the motion state of the target control component is optimally estimated, and finally a smooth trajectory point sequence is output. Based on the sequence of trajectory points, the motion trajectory from the current actual position of the fingertip to the target playing position is planned in Cartesian space; S4. Control the robot actuator to execute the motion trajectory to complete the performance action. The robot actuator includes a multi-degree-of-freedom robotic arm and an end effector dexterous hand. S5. During the performance, images are continuously captured by the head camera module and the fingertip camera module, and the actual position of the fingertip is calculated in real time using a fusion algorithm. The actual position is compared with the desired target position at the current moment to calculate the Cartesian spatial position error vector. When the magnitude of the Cartesian spatial position error vector exceeds a preset trigger threshold, a proportional-derivative controller is used to calculate the Cartesian spatial position correction vector based on the Cartesian spatial position error vector and the rate of change. The Cartesian spatial position correction vector is converted to joint space using the inverse or pseudo-inverse of the robot's Jacobian matrix to obtain the joint angle correction amount. The joint angle correction amount is combined with the original target angle to generate a new target joint angle and update the control command. In step S4, the step of controlling the robot actuator to execute the motion trajectory to complete the performance action further includes: Inverse kinematics calculations are performed on the target pose points on the motion trajectory to calculate the target angles and target angular velocities of each joint of the robot. Calculation of feedforward compensation torque based on robot dynamics model; The target angle, the target angular velocity, and the feedforward compensation torque are encapsulated into a digital control package and sent to the robot actuator via an industrial communication bus.

2. The method for robot hand localization and musical instrument playing control based on multi-camera according to claim 1, characterized in that, In step S1, the global image based on the working scene provides rough position information of the robot's hand, further including: A hand detection model based on a convolutional neural network is used to analyze the global image of the work scene to determine the two-dimensional pixel coordinates of the hand region; Based on the pinhole imaging model of the camera, and combined with the camera's intrinsic and extrinsic parameters and depth information, the two-dimensional pixel coordinates are calculated into the three-dimensional spatial coordinates of the robot's hand, serving as rough position information.

3. The method for robot hand localization and musical instrument playing control based on multi-camera according to claim 1, characterized in that, In step S2, the step of providing fine target localization results based on the local image of the target control to be interacted with further includes: A lightweight visual Transformer model is used to process the local image of the target control to be interacted with, so as to identify the corresponding target control and output the precise pixel coordinates of the target control on the two-dimensional image; Based on the precise pixel coordinates, and combined with the internal and external parameters and depth information of the fingertip camera module, the three-dimensional spatial coordinates of the target control component in the world coordinate system are calculated as the result of fine target positioning.

4. The method for robot hand localization and musical instrument playing control based on multi-camera according to claim 1, characterized in that, The method further includes: The multi-camera synchronization module is used to control the unified triggering and timestamp management of each camera in the fingertip camera module to ensure that the coarse location information is synchronized with the fine target positioning result in time.

5. A multi-camera robot hand positioning and musical instrument playing control system, applied to the multi-camera robot hand positioning and musical instrument playing control method according to any one of claims 1-4, characterized in that, include: The head camera module is used to acquire a global image of the working scene and provide rough position information of the robot's hand; The fingertip camera module, located at the end of the robot's finger, is used to acquire local images of the target control component to be interacted with and to provide fine-grained target localization results; The visual fusion module is used to fuse the coarse position information with the fine target positioning result and calculate the fused relative position between the fingertip and the target control to be interacted with. The control module is used to plan the motion trajectory based on the fused relative positions and generate control commands. A robotic actuator is used to receive and execute the control commands to complete the performance actions; The feedback and correction module is used to continuously acquire images through the head camera module and the fingertip camera module during the performance of the playing action, and calculate the actual position of the fingertip through a fusion algorithm; compare the actual position with the expected target position at the current moment, and calculate the Cartesian space position error vector. When the magnitude of the Cartesian spatial position error vector exceeds a preset trigger threshold, a proportional-derivative controller is used to calculate the Cartesian spatial position correction vector based on the Cartesian spatial position error vector and the rate of change. The Cartesian spatial position correction vector is then converted to joint space using the inverse or pseudo-inverse of the robot's Jacobian matrix to obtain the joint angle correction amount. The joint angle correction amount is combined with the original target angle to generate a new target joint angle and update the control command.

Images