Writing brush motion trail reproduction robot system and method based on binocular vision

Abstract

The invention discloses a writing brush motion trail reproduction robot system and method based on binocular vision. The writing brush motion trail reproduction robot system based on binocular vision comprises a binocular vision module; a key point identification and tracking module; a three-dimensional attitude calculation module; a motion trail generation module; a robot pen holding module; and a control module. The method for reproducing the writing brush motion trail through the robot based on binocular vision uses the system and comprises the following steps that system calibration is conducted; recording a writing process image; key point tracking and three-dimensional reconstruction; generating a motion track; and robot motion reproduction. The motion of the writing brush can be captured in real time in a non-contact and high-precision mode, and posture changes of the writing brush can be reproduced.

Description

Technical Field

[0001] This invention belongs to the field of robot motion trajectory reproduction technology, and particularly relates to a system and method for robot reproduction of brush motion trajectory based on binocular vision. Background Technology

[0002] As a treasure of traditional Chinese culture, calligraphy's digital replication and automated preservation are of great value for cultural heritage protection and art dissemination. However, existing technologies in the field of calligraphy replication have significant limitations, making it difficult to meet the demands for high-precision and high-fidelity replication. For example, pre-programmed path robots rely on offline programming to generate fixed motion instructions for the robotic arm, which cannot adapt to dynamic adjustments during the writing process, nor can they reproduce key brushstrokes such as "lifting and pressing," resulting in stiff and lifeless replicas lacking artistic spirit. Two-dimensional image tracing robots generate filling paths based solely on static ink marks, completely losing dynamic visual information such as writing speed, brush tip height, and brush tilt angle, only replicating the "form" but not the "spirit." Contact-based motion capture solutions require sensors installed on the brush, which not only alters the brush's physical properties and interferes with normal writing but also suffers from complex deployment, poor adaptability, and the inability to replicate remotely.

[0003] Furthermore, none of the above solutions establishes a logical connection between "visual information from writing video - three-dimensional movement of the brush - reproduction by robotic arm," making it impossible to extract dynamic information of the brush from sensorless writing videos to achieve "what you see is what you get."

[0004] Therefore, there is an urgent need for a non-contact, video-based technical solution that can fully capture the three-dimensional movement of a calligraphy brush and is highly adaptable, in order to overcome the shortcomings of existing technologies. Summary of the Invention

[0005] A primary objective of this invention is to provide a robot system and method for reproducing the trajectory of a calligraphy brush, based on binocular vision, capable of capturing the movement of a calligraphy brush in real time with high precision and without contact, and reproducing its posture changes. Furthermore, the system and its modules work together to achieve a complete process of video information acquisition, dynamic data processing, and robot reproduction.

[0006] To achieve the above-mentioned objectives, the present invention adopts the following technical solution:

[0007] As in one aspect of the present invention, a robot system for reproducing the motion trajectory of a calligraphy brush based on binocular vision is provided for reproducing calligraphy works, comprising:

[0008] A binocular vision module is used to simultaneously acquire images of brush movement with key points.

[0009] The key point recognition and tracking module is used to process the brush motion image, identify and track the key points therein;

[0010] The three-dimensional posture calculation module is used to calculate the three-dimensional spatial coordinates and brush posture based on the two-dimensional image coordinates of the key points through stereo matching and triangulation.

[0011] The motion trajectory generation module establishes a corresponding time series based on the data of three-dimensional spatial coordinates and brush posture, and generates the six-degree-of-freedom motion trajectory of the brush through Kalman filtering and B-spline smoothing algorithms.

[0012] The robot pen-holding module has an end effector that adopts an adjustable clamping structure adapted to the size of the brush, which is used to clamp and drive the brush to move.

[0013] The control module pre-establishes the conversion relationship between the binocular vision coordinate system and the robot base coordinate system through hand-eye calibration. This is used to convert the six-degree-of-freedom motion trajectory into angular displacement and angular velocity commands for each joint of the robot pen-holding module, thereby driving the robot pen-holding module to drive the brush to move for writing.

[0014] In one specific embodiment of the present invention, the key point is divided into at least two parts, which are respectively set at both ends of the brush handle as artificially added visual markers or natural feature points. The two points determine a straight line in space, which is used to calculate the three-dimensional position of the brush.

[0015] As in one specific embodiment of the present invention, the key point recognition and tracking module employs a feature extraction method based on color segmentation, contour extraction, or template matching, and combines it with the KLT optical flow method for continuous inter-frame tracking.

[0016] As in one specific embodiment of the present invention, the three-dimensional posture calculation module calculates the spatial posture angle of the brush by defining the vector direction of the key points of the two parts.

[0017] As in a specific embodiment of the present invention, the system derives the instantaneous acceleration of the pen stroke through speed data, establishes a mapping relationship between acceleration characteristics and pen stroke type, and outputs position-posture-speed-acceleration-pen stroke type.

[0018] As in one specific embodiment of the present invention, the motion trajectory generation module includes a filtering and smoothing algorithm unit for denoising and smoothing the original three-dimensional coordinate data.

[0019] As in one specific embodiment of the present invention, the robot pen-holding module is a six-axis robotic arm.

[0020] As in one specific embodiment of the present invention, the control module obtains the transformation relationship between the binocular vision coordinate system and the robot base coordinate system in advance through hand-eye calibration.

[0021] As in another aspect of the present invention, a method for reproducing the motion trajectory of a calligraphy brush based on binocular vision is provided. Using the aforementioned binocular vision-based robot system for reproducing the motion trajectory of a calligraphy brush, the method includes the following steps:

[0022] System calibration involves precisely calibrating the binocular vision module, obtaining the intrinsic parameters of the binocular camera, calibrating the relative position between the two cameras using a checkerboard target to ensure the stereo matching accuracy of the binocular vision, and establishing a rigid transformation relationship between the binocular camera coordinate system and the mechanical output coordinate system using a calibration board.

[0023] The writing process is recorded in images. The key point scheme of the brush is selected according to the actual application scenario. The writer completes the writing action according to normal writing habits. The binocular vision module starts synchronous acquisition and captures the complete image stream from the beginning of the stroke to the end of the stroke in real time.

[0024] Key point tracking and 3D reconstruction: The binocular images are processed, and then the 2D coordinates of the tracked key points are matched in a stereo model. Then, the 3D position of the brush with the brush tip as a reference and the pitch and yaw attitude of the brush handle defined by the two-point vector are calculated to realize the real-time 3D reconstruction of the brush movement.

[0025] The motion trajectory generation process receives the continuous key point 3D coordinates and posture data output by the 3D posture calculation module and integrates them into the original data sequence according to the timestamp order. First, the Kalman filter algorithm is used to eliminate data fluctuations caused by image noise, and then the data is interpolated by the B-spline smoothing algorithm to generate a spatiotemporally continuous six-degree-of-freedom motion trajectory file of the brush. This trajectory file not only completely preserves the original writing position and posture information, but also accurately reproduces the speed and acceleration changes during the brush stroke process, ensuring that the subsequent robot reproduction can match the original writing force and rhythm.

[0026] The robot's motion is reproduced by the control module reading the trajectory file and using the inverse kinematics algorithm to calculate the angular displacement and angular velocity commands of each joint. At the same time, a real-time monitoring mechanism is activated to collect the pose feedback data of the robot's end effector in real time and compare it with the theoretical values ​​in the trajectory file. Then, the robot's pen-holding module is driven to perform the copying action.

[0027] As in a specific embodiment of the present invention, the system calibration step includes calibrating the internal and external parameters of the binocular vision module, and performing hand-eye calibration of the coordinate system transformation relationship between the vision system and the robot system. The robot motion reproduction step controls the robot to not only reproduce the path of the pen tip, but also to reproduce the lifting, tilting and twisting movements of the brush.

[0028] As can be seen from the above technical solution, the advantages and positive effects of the robot system and method for reproducing the motion trajectory of a calligraphy brush based on binocular vision of the present invention are as follows:

[0029] In this invention, the movement of a brush can be captured in real time with high precision and without contact, and its posture changes can be reproduced. Moreover, the modules work together to realize the complete process of video information acquisition, dynamic data calculation and robot reproduction.

[0030] The present invention also includes the following beneficial effects:

[0031] 1. Non-contact measurement: Compared with the existing technology that requires the installation of sensors on the brush for "contact motion capture", this invention relies on pure visual capture, without any physical modification to the brush (whether it is artificial marking or natural feature scheme, it does not change the weight or grip of the brush), and does not interfere with the writer's pen grip posture and pen movement habits. It can capture the most realistic writing state that is closest to daily life, avoid pen distortion caused by equipment interference, and lay the foundation for subsequent reproduction of delicate pen techniques such as "lifting and pressing" and "central and side strokes".

[0032] 2. Complete Information: Unlike two-dimensional image copying robots that only record static ink marks, this invention, through the collaboration of a key point recognition and tracking module and a three-dimensional posture calculation module, can not only record the three-dimensional position path of the brush tip, but also completely capture the three-dimensional posture information of the brush—including the pitch of the brush handle around the X-axis and the yaw around the Z-axis. This dynamic posture information is the core of the "spirit" of calligraphy, solving the key problem of existing technologies that "only replicate the form, but fail to reproduce the spirit."

[0033] 3. High Precision and Real-Time Performance: This invention leverages the hardware advantages of a binocular vision module, employing stereo matching and triangulation to calculate 3D coordinates, and the KLT optical flow algorithm to achieve sub-pixel-level tracking. This enables sub-millimeter-level key point positioning accuracy, ensuring precise capture of various brushstrokes. Simultaneously, binocular synchronization matches the writer's penmanship rhythm in real time, avoiding trajectory deviations caused by dynamic capture lag, thus meeting the "timeliness" requirement for calligraphy replication. In calligraphy, the essence of the brushstroke is determined by the instantaneous acceleration change. This project establishes an "acceleration-brushstroke type" mapping model by precisely calculating instantaneous acceleration, driving the robot to reproduce the authentic soft-brush calligraphy writing effect by changing the brush tilt angle and acceleration. Sudden acceleration magnitudes result in "pauses," sudden acceleration directions result in "folds," and gradual acceleration results in "stroke movement" or "stroke ending."

[0034] 4. Wide adaptability: From the perspective of application scenarios, this invention supports both "offline replication," which involves recording writing videos and subsequently extracting the trajectory for digital archiving of cultural heritage, such as dynamic replication of ancient calligraphy masterpieces; and "online instant conversion," where the binocular vision module collects data in real time and the system synchronously calculates the trajectory during on-site creation by artists, enabling the robot to instantly copy the work for "real-time reproduction and display" in art exhibitions. From the perspective of adaptability, the key point solution supports both manual marking and natural features, and the robot pen-holding module can adapt to the pen-holding needs of different brush types and calligraphy styles, flexibly serving multiple fields such as calligraphy teaching (reproducing teachers' brushstrokes for students to observe), art dissemination (reproducing works for display in multiple locations), and re-creation (generating new works by adjusting parameters based on digital trajectories).

[0035] 5. Automation and Intelligence: This invention achieves full-process automation of "video acquisition - data processing - trajectory generation - robot reproduction" through the collaborative operation of various modules: the system calibration process adopts the mature Zhang Zhengyou calibration method and hand-eye calibration logic, which does not require complex debugging by professional personnel; the key point tracking integrates the "occlusion prediction - recapture" mechanism, which can avoid tracking interruption without manual intervention; the control module can automatically convert trajectory data into robot joint commands and monitor and adjust the running status in real time, without the need for manual intervention in trajectory correction or motion control throughout the process, which greatly reduces the operational threshold of the technology application and facilitates its promotion and use in non-professional technical fields such as education and cultural heritage. Attached Figure Description

[0036] To more clearly illustrate the technical solutions in the embodiments of this application or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments recorded in this application. For those skilled in the art, other drawings can be obtained from these drawings without creative effort.

[0037] Figure 1 This is a schematic diagram of the basic process of the key point recognition and tracking module of the robot system for reproducing the motion trajectory of a calligraphy brush based on binocular vision according to the present invention.

[0038] Figure 2 This is a schematic diagram of edge classification in a robot system and method for reproducing the motion trajectory of a calligraphy brush based on binocular vision.

[0039] Figure 3 This is a schematic diagram illustrating an example of gradient magnitude in the binocular vision-based robot system and method for reproducing the motion trajectory of a calligraphy brush, as described in this invention. Detailed Implementation

[0040] Exemplary embodiments will now be described more fully with reference to the accompanying drawings. However, these exemplary embodiments can be implemented in many forms and should not be construed as limited to the embodiments set forth herein; rather, they are provided so that the invention will be thorough and complete, and the concept of the exemplary embodiments will be fully conveyed to those skilled in the art. The same reference numerals in the drawings denote the same or similar structures, and therefore their detailed description will be omitted.

[0041] like Figures 1 to 3 As shown, the robot system for reproducing the motion trajectory of a calligraphy brush based on binocular vision includes a binocular vision module, a key point recognition and tracking module, a 3D pose calculation module, a motion trajectory generation module, a robot pen-holding module, and a control module. The binocular vision module is used to synchronously acquire calligraphy brush motion images with key points. The key point recognition and tracking module processes the calligraphy brush motion images, identifying and tracking the key points. The 3D pose calculation module calculates the 3D spatial coordinates and calligraphy brush pose based on the 2D image coordinates of the key points using stereo matching and triangulation. The motion trajectory generation module establishes a corresponding time series based on the 3D spatial coordinates and calligraphy brush pose data, and generates the six-degree-of-freedom motion trajectory of the calligraphy brush using Kalman filtering and B-spline smoothing algorithms. The end effector of the robot pen-holding module adopts an adjustable clamping structure adapted to the size of the calligraphy brush, used to clamp and drive the calligraphy brush movement. The control module pre-establishes the conversion relationship between the binocular vision coordinate system and the robot base coordinate system through hand-eye calibration. This is used to convert the six-degree-of-freedom motion trajectory into angular displacement and angular velocity commands for each joint of the robot pen-holding module, thereby driving the robot pen-holding module to drive the brush to move for writing.

[0042] In one specific embodiment of the present invention, the key point is divided into at least two parts, which are respectively set at both ends of the brush handle as artificially added visual markers or natural feature points for calculating the three-dimensional position of the brush.

[0043] As in a specific embodiment of the present invention, the key point recognition and tracking module employs a feature extraction method based on color segmentation, contour extraction or template matching, and combines it with KLT optical flow method or correlation filtering algorithm for continuous inter-frame tracking.

[0044] As in one specific embodiment of the present invention, the three-dimensional posture calculation module calculates the spatial posture angle of the brush by defining the vector direction of the key points of the two parts.

[0045] As in one specific embodiment of the present invention, the motion trajectory generation module includes a filtering and smoothing algorithm unit for denoising and smoothing the original three-dimensional coordinate data.

[0046] As in one specific embodiment of the present invention, the robot pen-holding module is a six-axis industrial robot.

[0047] As in one specific embodiment of the present invention, the control module obtains the transformation relationship between the binocular vision coordinate system and the robot base coordinate system in advance through hand-eye calibration.

[0048] The method for reproducing the motion trajectory of a calligraphy brush based on binocular vision, using the aforementioned binocular vision-based calligraphy brush motion trajectory reproduction robot system, includes the following steps:

[0049] System calibration involves precisely calibrating the binocular vision module, obtaining the intrinsic parameters of the binocular camera, calibrating the relative position between the two cameras using a checkerboard target to ensure the stereo matching accuracy of the binocular vision, and establishing a rigid transformation relationship between the binocular camera coordinate system and the mechanical output coordinate system using a calibration board.

[0050] The writing process is recorded in images. The key point scheme of the brush is selected according to the actual application scenario. The writer completes the writing action according to normal writing habits. The binocular vision module starts synchronous acquisition and captures the complete image stream from the beginning of the stroke to the end of the stroke in real time.

[0051] Key point tracking and 3D reconstruction: The binocular images are processed, and then the 2D coordinates of the tracked key points are matched in a stereo model. Then, the 3D position of the brush with the brush tip as a reference and the pitch and yaw attitude of the brush handle defined by the two-point vector are calculated to realize the real-time 3D reconstruction of the brush movement.

[0052] The motion trajectory generation process receives the continuous key point 3D coordinates and posture data output by the 3D posture calculation module and integrates them into the original data sequence according to the timestamp order. First, the Kalman filter algorithm is used to eliminate data fluctuations caused by image noise, and then the data is interpolated by the B-spline smoothing algorithm to generate a spatiotemporally continuous six-degree-of-freedom motion trajectory file of the brush. This trajectory file not only completely preserves the original writing position and posture information, but also accurately reproduces the speed and acceleration changes during the brush stroke process, ensuring that the subsequent robot reproduction can match the original writing force and rhythm.

[0053] The robot's motion is reproduced by the control module reading the trajectory file and using the inverse kinematics algorithm to calculate the angular displacement and angular velocity commands of each joint. At the same time, a real-time monitoring mechanism is activated to collect the pose feedback data of the robot's end effector in real time and compare it with the theoretical values ​​in the trajectory file. Then, the robot's pen-holding module is driven to perform the copying action.

[0054] As in a specific embodiment of the present invention, the system calibration step includes calibrating the internal and external parameters of the binocular vision module, and performing hand-eye calibration of the coordinate system transformation relationship between the vision system and the robot system. The robot motion reproduction step controls the robot to not only reproduce the path of the pen tip, but also to reproduce the lifting, tilting and twisting movements of the brush.

[0055] As in a specific embodiment of the present invention, the system derives the instantaneous acceleration of the pen stroke through speed data, establishes a mapping relationship between acceleration characteristics and pen stroke type, and outputs position-posture-speed-acceleration-pen stroke type.

[0056] The "Pen Movement Speed ​​Control Submodule" is supplemented with an "Instantaneous Acceleration Calculation and Pen Stroke Generation Unit," which is used to deduce the instantaneous acceleration of pen movement from speed data, establish a mapping relationship between acceleration characteristics and pen stroke type, and output "Position—Posture—Speed—Acceleration—Pen Stroke Type."

[0057] The motion trajectory generation module includes a filtering and smoothing algorithm unit, a pen stroke speed control submodule, and an instantaneous acceleration calculation and pen stroke generation unit.

[0058] The instantaneous acceleration calculation and pen stroke generation unit is used for:

[0059] (1) Calculate the instantaneous acceleration in three-dimensional space using the instantaneous velocity difference (Δv / Δt) between consecutive frames. Combined with the rate of change of acceleration direction (Δθ) a Extract acceleration features using / Δt);

[0060] (2) Pre-set a library of pen stroke types (6 core pen stroke types: concealed stroke, exposed stroke, central stroke, side stroke, folded stroke, and turning stroke) and establish a mapping model of "acceleration characteristics - pen stroke type".

[0061] Based on the 3D pose calculation module, the following formulas are added:

[0062] Let the system frame rate be f ps The frame time interval is:

[0063]

[0064] Let the current frame time be t, and the three-dimensional coordinates of keypoint P in the three frames be as follows:

[0065] -(P(t-1)=(X t-1 ,Y t-1 Z t-1 ))

[0066] -(P(t=(X) t ,Y t Z t ))

[0067] -(P(t+1)=(X t+1 ,Y t+1 Z t+1 ))

[0068] 1. Instantaneous velocity calculation (using the central difference method for higher accuracy):

[0069]

[0070] 2. Instantaneous acceleration calculation

[0071]

[0072] 3. Acceleration scalar

[0073]

[0074] Analysis of acceleration direction and direction change

[0075] 1. Acceleration unit direction vector

[0076]

[0077] 2. Acceleration direction angle (in camera coordinate system):

[0078] Yaw angle: The direction in the XY plane.

[0079] φ yaw =arctan2(a y ,a x )

[0080] Pitch: The angle between the pitch and the XY plane.

[0081]

[0082] 3. Rate of change of direction (based on direction angle):

[0083] Δφ yaw =φ yaw (t)-φ yaw (t-1)

[0084] Δφ pitch =φ pitch (t)-φ pitch (t-1)

[0085] 4. Angle of change of direction (based on the dot product of unit vectors):

[0086] This is the most robust indicator for measuring sudden changes in direction. Calculate the dot product of the unit direction vectors in two consecutive frames and determine the angle θ between them.

[0087]

[0088] -(θ≈0): Directionally stable.

[0089] -(θ≈90°): The direction changes drastically.

[0090] -(θ≈180°): The direction is completely reversed.

[0091] In actual operation, the binocular vision camera faces the writer, maintaining a similar height and angle to the writing plane. This ensures the brush remains fully within the camera's visual range throughout the writing process, with clear imaging of the brush handle, tip, and key markers. The binocular vision module connects to a host computer via USB. The host computer runs an OpenCV image acquisition program, triggering the camera to synchronously acquire image sequences of the writing process in real time. The image resolution is set to 1280×720 pixels, and the image format is RGB24. During acquisition, the program automatically adds a timestamp (1ms accuracy) to each frame and stores it on the local hard drive in a "left eye -- right eye -- timestamp" format. This completes the entire process of brush movement from the start of the stroke to the end, providing a high-quality, highly synchronized image source for subsequent key point recognition and 3D pose calculation modules.

[0092] Specific implementation of the key point identification and tracking module

[0093] refer to Figure 1 As shown, the key point recognition and tracking module runs on the host computer and processes the image sequence output by the binocular vision module to realize the recognition and continuous tracking of key points at both ends of the brush. The specific implementation scheme is as follows:

[0094] 1. Predefine two visual key points and set them at both ends of the brush:

[0095] Key point 1 (pen tip point): Located 1cm away from the pen tip, the pen barrel position is used to calculate the three-dimensional position of the pen tip;

[0096] Key point 2 (pen tail point): Located 1cm from the pen tail, on the pen shaft, it is distributed along the axis of the pen shaft with key point 1. The distance between the two points is adjusted according to the length of the pen, ensuring that the line connecting the two points coincides with the axis of the pen shaft ≥95%.

[0097] After the two points are calculated to have three-dimensional coordinates, the spatial posture of the brush handle is defined by the principle of "two points determine a straight line". It can completely identify the five degrees of freedom of the brush (X / Y / Z axis translation, pitch around the X axis, and yaw around the Z axis), which meets the dynamic capture requirements of brush techniques such as "lifting and pressing" (Z axis translation) and "central / side stroke" (pitch around the X axis) in calligraphy replication.

[0098] The module supports two key point schemes: manual marking and natural features, which are suitable for different implementation scenarios. The manual marking scheme is used for high-precision scenarios (such as cultural heritage replication), while the natural feature scheme is used for unmodified scenarios (such as general handwriting instruction).

[0099] Implementation of manual marker scheme

[0100] 2. When the implementation scenario allows for manual modification of the brush, a high-precision manual marking point scheme should be adopted. The specific implementation steps are as follows:

[0101] a) A high-precision manual marker solution under conditional deployment:

[0102] Marker dot creation: Use circular patches with a diameter of 3-5mm. Two types of materials are available: high-contrast (red or yellow), with a color contrast of >70% with the black / brown pen barrel.

[0103] Marker point pasting: Paste the prepared marker points at the predefined positions at both ends of the brush. When pasting, ensure that the deviation between the center of the marker point and the key point is ≤0.5mm, and that the marker points are free of wrinkles or tilts to avoid affecting the image quality.

[0104] Recognition Parameter Settings: The host computer program sets recognition parameters for manually marked points—for high-contrast marked points, HSV color space threshold segmentation is used (red marked points H=0-10 or 160-179, S=40-255, V=40-255; yellow marked points H=20-40, S=40-255, V=40-255); for highly reflective marked points, grayscale threshold segmentation is used (grayscale value ≥230). Both methods can achieve stable extraction of marked points.

[0105] b) Natural feature adaptation scheme without additional conditions:

[0106] When the implementation scenario does not allow for modification of the brush, the natural feature point solution is adopted, and the specific implementation steps are as follows:

[0107] Feature point selection: Stable physical features inherent in the brush handle are manually selected as natural feature points. The selection criteria are: feature size (e.g., scratch width, seam length) ≥ 3mm, ensuring an image size ≥ 5 pixels at 1280×720 pixel resolution; grayscale or color contrast between the feature and the surrounding brush handle area ≥ 40%, meeting the signal-to-noise ratio requirements of the feature extraction algorithm. Common optional features include: obvious color difference bands in the wood grain of the brush handle, and the edges of engraved characters on the brush handle.

[0108] Feature point registration: The host computer program runs the ORB feature extraction algorithm to register the natural feature points in the initial frame image, generate feature point descriptors, and store the two-dimensional coordinates of the feature points in the initial frame as a reference for subsequent tracking.

[0109] Recognition and Matching: During subsequent frame processing, the program uses the ORB feature matching algorithm to match the current frame feature points with the registered baseline feature points. After a successful match, the RANSAC algorithm is used to remove mismatched points, thereby achieving stable recognition of natural feature points.

[0110] 3. Image Processing and Tracking: Whether using manually marked points or natural feature points, the KLT optical flow algorithm is employed for continuous inter-frame tracking. Specific implementation parameters are as follows:

[0111] The tracking window size was set to 15×15 pixels, using a 3-layer image pyramid structure (pyramid scaling factor 0.5), with 20 optical flow iterations and an error threshold of 0.01. During tracking, the program calculated the optical flow response value in real time. When the response value was ≥0.5, tracking was considered normal; when the response value was <0.5 (e.g., a hand occluding a brush), the Kalman filter algorithm was automatically activated for motion prediction. The prediction window was set to 3 frames, and re-capture was continuously attempted within the predicted coordinate neighborhood (10×10 pixels). If tracking failed to recover after more than 3 frames, a pop-up prompt was triggered on the host computer, requiring the operator to manually reinitialize the key points to ensure the continuity of the tracking process (tracking interruption rate ≤0.5%).

[0112] For each frame of the image, the computer first performs a color space conversion (RGB to HSV) and then uses the red channel threshold to initially extract possible marker regions.

[0113] Morphological operations and contour analysis are performed on the extracted region to accurately calculate the pixel coordinates (uL, vL) and (uR, vR) of the two key points in the left and right camera images.

[0114] These key points are continuously tracked in the video sequence using the KLT optical flow algorithm or correlation filtering algorithm.

[0115] 4. Three-dimensional solution:

[0116] Based on the pre-calibrated camera parameters, triangulation is performed on the same key point matched in the left and right images (such as P1 left and P1 right) to calculate its three-dimensional coordinates (X1, Y1, Z1) in the binocular vision system coordinate system.

[0117] Similarly, the three-dimensional coordinates (X2, Y2, Z2) of point P2 are obtained.

[0118] Connecting P1 and P2 forms vector V. This vector defines the principal direction (i.e., posture) of the brush handle. The position of the brush tip can be estimated based on the position of P1 and the physical dimensions of the brush (e.g., assuming the brush tip is a few centimeters below the extension of P1). The rotation angle of the brush (such as the torsion about its own axis) can be further determined by tracing other features on the brush handle or assuming its relationship with the normal vector of the writing plane.

[0119] 5. Trajectory Generation:

[0120] The pen tip position (X_tip, Y_tip, Z_tip) and pose vector V calculated for each frame are stored in an array in chronological order.

[0121] The trajectory data is smoothed using a Kalman filter or a low-pass filter to eliminate jitter noise and generate a high-quality trajectory with continuous time and pose.

[0122] 6. Robot reproduction:

[0123] The computer sends the trajectory data to the robot controller through the robot communication interface.

[0124] The controller performs trajectory planning, inverse kinematics solution, and drives the joint motors to move, so that the writing brush held at the end moves strictly according to the recorded trajectory, including precise lifting, writing, and retracting strokes and tilting actions, thereby reproducing a "yong" character on the paper that is highly similar to the demonstration writing.

[0125] Key point recognition and tracking module

[0126] The key point recognition and tracking module runs on the upper computer and is responsible for real-time processing of the image sequence collected by the binocular vision module to achieve the recognition, tracking, and position update of the key points at both ends of the writing brush. This module mainly includes an image preprocessing unit, a key point detection unit, a key point tracking unit, and a state prediction and anomaly recovery unit.

[0127] (1) Image acquisition and preprocessing

[0128] Here, the system uses the Yabo Smart Gemini335 binocular depth camera as the vision sensor. The RGB image and depth image streams of the left and right cameras are synchronously collected through the SDK interface to ensure that the timestamps of each frame of the image are consistent. After the upper computer reads the image stream using the OpenCV interface, the following preprocessing is performed: RGB→HSV color space conversion, 3×3 Gaussian filtering for denoising, morphological opening and closing operations (kernel 5×5) to fill holes, and the Canny algorithm is used to extract edges.

[0129] (2) Key point selection and calibration

[0130] To accurately determine the pose of the writing brush in three-dimensional space, at least two key points are set on the pen shaft of the writing brush, which are located near the upper end of the pen shaft and the tip of the pen respectively, to define the pen shaft direction and the spatial position of the tip of the pen. The key points can be marked with high-saturation colors (such as red or yellow patches), with a diameter of 2-~4 mm to enhance the recognition contrast in the RGB image.

[0131] Before system startup, the Gemini335 camera's intrinsic and extrinsic parameters are calibrated using the ArUco checkerboard calibration board to obtain the focal length, principal point coordinates, and distortion coefficients of the left and right cameras. Furthermore, the depth calibration function provided by the SDK is used to establish a mapping relationship between pixel coordinates and actual depth values. After calibration, the system can convert the identified keypoint pixel coordinates into three-dimensional spatial coordinates (triangulation algorithm), achieving precise positioning of the brush in the camera coordinate system.

[0132] (3) Key point detection algorithm

[0133] After image acquisition, the system first performs color space conversion (BGR→HSV) on the RGB image and segments candidate regions for keypoints by setting a color threshold range. To suppress background noise, morphological opening and closing operations are used to remove isolated points, and target regions are retained by filtering based on contour area and roundness. The centroid coordinates of the detected candidate regions are calculated as the initial positions of the keypoints. If natural features (such as pen cap edges or pen tip shadows) are used as keypoints, edge detection combined with Hough circle transform is used for automatic extraction.

[0134] (4) Keypoint tracking algorithm

[0135] Between consecutive frames, the system employs the KLT optical flow algorithm for sub-pixel-level tracking of key points. By minimizing the brightness difference in the neighborhood, the optical flow vector is obtained, achieving stable tracking. When the detection confidence level falls below a set threshold, the system automatically switches to a template-matching-based re-localization mode to correct drift caused by occlusion or changes in illumination.

[0136] Simultaneously, the system utilizes the depth information from the Gemini335 to perform disparity matching on the left and right images, calculates pixel disparity, and obtains the spatial depth value of each keypoint. Thus, the coordinate sequence of the keypoints in three-dimensional space can be obtained in real time.

[0137] (5) Attitude and robustness optimization

[0138] To improve tracking stability, the system incorporates a Kalman filter during tracking to dynamically predict and correct key point positions, smoothing out displacement fluctuations caused by image noise. When a key point is detected as missing, position data from the interrupted frame is recovered through trajectory interpolation.

[0139] Simultaneously, by calculating the vector direction between two key points, the system solves for the pitch and yaw angles of the brush handle, and defines the overall posture of the brush by combining this with the position of the brush tip key point. The system can update the three-dimensional position and posture information of the brush in real time at a sampling rate of 30 frames per second, with a spatial positioning accuracy better than ±0.5mm.

[0140] (6) Module output interface

[0141] The data output by the key point recognition and tracking module includes:

[0142] Key point 3D coordinates (x, y, z)

[0143] Pen handle direction vector (defining the brush's posture)

[0144] Timestamp per frame (used for trajectory time series)

[0145] Tracking confidence and filter state variables

[0146] This data will be transmitted to the 3D pose calculation module for subsequent trajectory generation and robot motion control.

[0147] 2. 3D Attitude Calculation Module (Detailed Explanation of Stereo Matching and Triangulation Methods)

[0148] The 3D pose calculation module is used to accurately calculate the 3D spatial coordinates of key points of the brushstrokes based on the 2D pixel coordinates of corresponding key points in the left and right images acquired by the binocular vision system, combined with camera calibration parameters, through stereo matching and triangulation. This module includes a feature matching subunit and a 3D reconstruction subunit.

[0149] (1) Camera Imaging and Binocular Geometric Model

[0150] After completing the camera calibration, the intrinsic parameter matrices of the left and right cameras are known:

[0151]

[0152] And the extrinsic parameters (rotation matrix R and translation vector T = [B, 0, 0]). T (where B is the baseline length).

[0153] For a point P(X,Y,Z) in three-dimensional space, its projection relationship on the left and right camera image planes is as follows:

[0154]

[0155] The lateral parallax of the left and right image points is defined as follows:

[0156] d=uL-uR

[0157] 3D matching algorithm flow

[0158] To obtain the pixel correspondence of key points in the left and right images, a block-matching-based stereo matching algorithm is used. The specific steps are as follows:

[0159] Parallax search range determination: Based on the camera baseline B, focal length f, and working distances Zmin and Zmax, the parallax search range is determined as follows:

[0160]

[0161] 1. Matching cost calculation: Define a window size w×w centered on pixel (u,v) in the left image.

[0162]

[0163] 2. Calculate the sum of squares of the absolute differences between the corresponding regions in the right image and the right image:

[0164] 3. Parallax Selection: For each pixel, select the parallax value with the lowest cost.

[0165]

[0166] 4. Subpixel optimization: Parabolic interpolation or cubic spline is used to smooth the disparity curve to obtain disparity estimation with subpixel accuracy.

[0167] 5. Post-processing: Apply medium filtering (3×3 window) to remove isolated noise points, and use left-right consistency detection to eliminate false matches.

[0168] (2) Triangulation calculation formula

[0169] After obtaining the disparity map d(u,v), the 3D coordinates of each keypoint are calculated based on the binocular geometry:

[0170]

[0171] in:

[0172] fx, fy: Focal lengths of the left and right cameras;

[0173] cx, cy: coordinates of the principal point;

[0174] B: Camera baseline;

[0175] d: pixel parallax.

[0176] The result is the three-dimensional spatial position P(X,Y,Z) of the key point in the camera coordinate system.

[0177] (4) Calculation of brush posture

[0178] For the key points P1(X1,Y1,Z1) and P2(X2,Y2,Z2) at both ends of the brush,

[0179] Its pen shaft direction vector is defined as:

[0180] The pitch and yaw angles of the pen barrel are calculated as follows:

[0181]

[0182] (5) Error and Real-time Performance Optimization

[0183] 1. By using the native stereo synchronization mechanism of the Gemini335, the time difference between the left and right image frames is controlled within ≤1ms, ensuring stereo matching accuracy.

[0184] 2. Subpixel parallax interpolation and median filtering are used in post-processing to control the depth error within ±0.3mm.

[0185] 3. The algorithm runs at a frame rate of approximately 25-30fps, enabling real-time attitude calculation.

[0186] (6) Module Output

[0187] The output data includes:

[0188] The three-dimensional coordinates (Xi, Yi, Zi) of each key point in each frame;

[0189] Pen handle direction vector and posture angle;

[0190] Disparity confidence and filter state variables.

[0191] This module provides high-precision, continuous three-dimensional spatiotemporal input for the motion trajectory generation module.

[0192] By combining Kalman filtering and B-spline smoothing algorithms, generating a six-degree-of-freedom motion trajectory of a calligraphy brush involves extracting a continuous and smooth trajectory from the brush's posture and position data.

[0193] 1. Data Collection

[0194] First, sensors are used to acquire the brush's motion data, typically including position (x, y, z coordinates) and orientation (such as rotation angles around the x, y, and z axes). Position data (P... t =(x t ,y t ,z t )) and attitude data (R t =(roll) t pitch t ,yaw t As time changes, noise or data loss may occur between sampling points.

[0195] 2. Kalman Filtering: Denoising and Prediction

[0196] The role of Kalman filtering is to reduce noise through prediction and update steps, resulting in a more accurate state estimate.

[0197] The main steps of Kalman filtering:

[0198] Prediction: Predicting the state at the next moment based on the current state.

[0199]

[0200] Where A is the state transition matrix, B is the control input matrix, (u t ) is the control input (such as speed or acceleration).

[0201] Update: The forecast has been revised based on the new observations.

[0202] K t =P t|t-1 H T HP t|t-1 H T +R) -1

[0203]

[0204] Among them, K t is the Kalman gain, Pt|t-1 is the prediction error covariance, R is the observation noise covariance matrix, and H is the observation matrix.

[0205] In each iteration, the Kalman filter updates the six-degree-of-freedom position and orientation of the brush based on new measurement data, thereby reducing noise interference.

[0206] 3. B-spline smoothing: smoothing trajectory

[0207] B-spline algorithms can be used to smooth trajectories, making them more continuous and smoother. They construct a smooth curve using a series of control points, ensuring continuity across all points.

[0208] We have the time series positions {P1, P2, ..., P} after Kalman filtering. n B-splines can be used to smooth data through the following steps:

[0209] B-spline curves are composed of a series of weighted combinations of basis functions, and the formula is:

[0210]

[0211] Where S(t) is the interpolation function, representing the smooth curve value at parameter t, and N i,k (t) is the i-th basis function, and k is the order of the B-spline; P i These are control points, representing discrete data points in space.

[0212] The basis functions N of the B-spline i,k (t) is defined by a recursive relation and has the characteristic of local support, meaning that each basis function only affects adjacent control points. The order k of the B-spline controls the smoothness of the curve; higher-order B-splines can generate smoother curves.

[0213] Motion trajectory smoothing process:

[0214] 1. Input data preparation:

[0215] The three-dimensional coordinate data output by the Kalman filter ((x i ,y i ,z i This is continuous time series data, representing the position of the brush in space. Each data point corresponds to a timestamp and contains information about the position and posture of the brush in three-dimensional space.

[0216] 2. Construction of B-spline curves:

[0217] To ensure the smoothness of the curves, B-splines in three-dimensional space are used for interpolation. First, independent B-spline curves are constructed for each dimension (X, Y, Z). These curves work together to maintain the smoothness of the brushstrokes across all dimensions. The curves in each dimension are interpolated through the corresponding set of control points ((x...). i ),(y i ),(z i It is represented by )) and smoothed using the basis functions of B-splines.

[0218] 3. Node and order selection:

[0219] The nodes of a B-spline determine the position of data points on the curve. In this invention, a uniform distribution or adaptive node selection method is used to ensure a reasonable distribution of the curve along the time axis. The order k is typically chosen as 3, i.e., cubic B-spline, to ensure that the interpolated curve achieves good smoothness in terms of position, velocity, and acceleration.

[0220] 4. Interpolation calculation and curve generation:

[0221] By using B-spline interpolation, a smooth 3D trajectory is generated. The interpolated curve contains more points, thus obtaining subtle trajectory changes that can accurately reproduce the minute variations in calligraphic movements. For example, the acceleration and velocity changes of the brush are smoothed out on the interpolated curve, avoiding abrupt changes in the original data.

[0222] 4. Six-DOF trajectory generation

[0223] Ultimately, in order to generate the six-degree-of-freedom trajectory of the brush, including position data (P... t =(x t ,y t ,z t )) and attitude data (R t =(roll) t pitch t ,yaw tB-splines can be applied not only to position data but also to attitude data.

[0224] For location data, separate B-spline curves can be used to smooth the x, y, and z coordinates.

[0225] For the attitude data, we smoothed the three components of the Euler angles (roll, pitch, and yaw) using B-splines.

[0226] Advantages include:

[0227] Smoothness: B-splines can generate very smooth and continuous trajectories, avoiding unnatural motion changes.

[0228] Local control: The local control characteristics of B-splines mean that changing a control point will not affect the entire curve, increasing flexibility.

[0229] Flexibility: Different curve shapes can be achieved by adjusting the number and position of control points to meet the needs of different calligraphic styles.

[0230] Feature extraction methods based on color segmentation and contour extraction, combined with KLT optical flow method for continuous inter-frame tracking

[0231] 1. Color segmentation

[0232] By converting the input RGB image to the HSV color space, filtering pixel values ​​using a set color threshold, and then performing binarization, the brush area is segmented to obtain a clear brush edge.

[0233] 2. Contour Extraction

[0234] refer to Figure 2 and Figure 3 As shown, Canny edge detection is used to extract the contours of the image, and the formula is as follows:

[0235]

[0236] Among them, (I) x ) and (I y ) is the gradient of the image in the x and y directions, and G is the intensity of the edge image.

[0237] 3. Template matching

[0238] The extracted contour is matched with a preset template to confirm the position and direction of the brushstroke. Then, a smoothing process is performed to reduce noise interference and improve detection accuracy.

[0239] The precise positioning of the calligraphy brush is achieved using a template matching algorithm, as shown in the following formula:

[0240]

[0241] Where T(i,j) is the template image and I(x+i,y+j) is the image region within the sliding window. The best matching position is found by calculating the difference between this region and the template.

[0242] 4. Combination of continuous inter-frame tracking methods

[0243] (1) KLT optical flow method

[0244] The KLT optical flow method calculates the trajectory of a brushstroke by tracking feature points. The basic formula used in calculating optical flow is:

[0245] I x u+I y v+I t =0

[0246] (2) Correlation filtering algorithm

[0247] Correlation filtering tracks data by calculating the similarity between a template and an image region. The formula is:

[0248]

[0249] Where W(i,j) is the template, I(x+i,y+j) is the pixel value of a certain region in the image, and R(x,y) is the correlation between the region and the template.

[0250] The combination of KLT optical flow and correlation filtering: By tracking feature points using KLT optical flow, smooth tracking of brush motion is achieved, and the combination with correlation filtering algorithm reduces noise, ensuring stable tracking between consecutive frames.

[0251] In addition, the experiment used Six-axis robot The Mirobot six-axis robotic arm has six degrees of freedom, enabling it to perform relatively complex spatial movements and better simulate the actions of a writer.

[0252] Those skilled in the art should understand that the specific structures and processes shown in the above detailed embodiments are merely exemplary and not restrictive. Furthermore, those skilled in the art can combine the various technical features described above in various possible ways to form new technical solutions or make other modifications, all of which fall within the scope of this invention.

Claims

1. A robot system for reproducing the motion trajectory of a calligraphy brush based on binocular vision, used to reproduce calligraphy works, characterized in that, include: A binocular vision module is used to simultaneously acquire images of brush movement with key points. The key point recognition and tracking module is used to process the brush motion image, identify and track the key points therein; The three-dimensional posture calculation module is used to calculate the three-dimensional spatial coordinates and brush posture based on the two-dimensional image coordinates of the key points through stereo matching and triangulation. The motion trajectory generation module establishes a corresponding time series based on the data of three-dimensional spatial coordinates and brush posture, and generates the six-degree-of-freedom motion trajectory of the brush through Kalman filtering and B-spline smoothing algorithms. The robot pen-holding module has an end effector that adopts an adjustable clamping structure adapted to the size of the brush, which is used to clamp and drive the brush to move. The control module pre-establishes the conversion relationship between the binocular vision coordinate system and the robot base coordinate system through hand-eye calibration. This is used to convert the six-degree-of-freedom motion trajectory into angular displacement and angular velocity commands for each joint of the robot pen-holding module, thereby driving the robot pen-holding module to drive the brush to move for writing.

2. The robot system for reproducing the motion trajectory of a calligraphy brush based on binocular vision according to claim 1, characterized in that: The key point is divided into at least two parts, which are set at both ends of the brush handle as artificially added visual markers or natural feature points. The two points determine a straight line in space, which is used to calculate the three-dimensional position of the brush.

3. The robot system for reproducing the motion trajectory of a calligraphy brush based on binocular vision according to claim 2, characterized in that: The key point recognition and tracking module adopts feature extraction methods based on color segmentation, contour extraction or template matching, and combines KLT optical flow method for continuous inter-frame tracking.

4. The robot system for reproducing the motion trajectory of a calligraphy brush based on binocular vision according to claim 2, characterized in that: The three-dimensional posture calculation module calculates the spatial posture angle of the brush by defining the vector direction of the key points in the two parts.

5. The robot system for reproducing the motion trajectory of a calligraphy brush based on binocular vision according to claim 1, characterized in that: The system derives the instantaneous acceleration of the pen stroke from the velocity data, establishes a mapping relationship between acceleration characteristics and pen stroke type, and outputs position—posture—velocity—acceleration—pen stroke type.

6. The robot system for reproducing the motion trajectory of a calligraphy brush based on binocular vision according to claim 1, characterized in that: The motion trajectory generation module includes a filtering and smoothing algorithm unit, which is used to denoise and smooth the original three-dimensional coordinate data.

7. The robot system for reproducing the motion trajectory of a calligraphy brush based on binocular vision according to claim 1, characterized in that: The robot pen-holding module is a six-axis robotic arm.

8. The robot system for reproducing the motion trajectory of a calligraphy brush based on binocular vision according to claim 7, characterized in that: The control module pre-observes the conversion relationship between the binocular vision coordinate system and the robot base coordinate system through hand-eye calibration.

9. A robot method for reproducing the motion trajectory of a calligraphy brush based on binocular vision, characterized in that: The robot system for reproducing the motion trajectory of a calligraphy brush based on binocular vision, as described in any one of claims 1-8, includes the following steps: System calibration involves precisely calibrating the binocular vision module, obtaining the intrinsic parameters of the binocular camera, calibrating the relative position between the two cameras using a checkerboard target to ensure the stereo matching accuracy of the binocular vision, and establishing a rigid transformation relationship between the binocular camera coordinate system and the mechanical output coordinate system using a calibration board. The writing process is recorded in images. The key point scheme of the brush is selected according to the actual application scenario. The writer completes the writing action according to normal writing habits. The binocular vision module starts synchronous acquisition and captures the complete image stream from the beginning of the stroke to the end of the stroke in real time. Key point tracking and 3D reconstruction: The binocular images are processed, and then the 2D coordinates of the tracked key points are matched in a stereo model. Then, the 3D position of the brush with the brush tip as a reference and the pitch and yaw attitude of the brush handle defined by the two-point vector are calculated to realize the real-time 3D reconstruction of the brush movement. The motion trajectory generation process receives the continuous key point 3D coordinates and posture data output by the 3D posture calculation module and integrates them into the original data sequence according to the timestamp order. First, the Kalman filter algorithm is used to reduce the data fluctuation caused by image noise due to hand occlusion. Then, the data is interpolated using the B-spline smoothing algorithm to generate a spatiotemporally continuous six-degree-of-freedom motion trajectory file of the brush. This trajectory file not only completely preserves the original writing position and posture information, but also accurately reproduces the speed and acceleration changes during the brush stroke process, ensuring that the subsequent robot reproduction can match the original writing force and rhythm. The robot's motion is reproduced by the control module reading the trajectory file and using the inverse kinematics algorithm to calculate the angular displacement and angular velocity commands of each joint. At the same time, a real-time monitoring mechanism is activated to collect the pose feedback data of the robot's end effector in real time and compare it with the theoretical values ​​in the trajectory file. Then, the robot's pen-holding module is driven to perform the copying action.

10. The robot method for reproducing the motion trajectory of a calligraphy brush based on binocular vision according to claim 9, characterized in that: The system calibration steps include calibrating the intrinsic and extrinsic parameters of the binocular vision module, and performing hand-eye calibration of the coordinate system transformation relationship between the vision system and the robot system; the robot motion reproduction steps control the robot to not only reproduce the path of the pen tip, but also reproduce the lifting, tilting and twisting movements of the brush.

Images