Mirror image motion implementation method and system for upper limb mirror image rehabilitation robot

Abstract

The invention belongs to the technical field of upper limb rehabilitation training, and particularly relates to a mirror image motion implementation method and system for an upper limb mirror image rehabilitation robot. The mirror image motion implementation method for the upper limb mirror image rehabilitation robot comprises the steps that S1, an image of a patient is obtained through a depth camera, a preset key point of a healthy limb and a preset key point of an affected limb are recognized, pixel coordinates of the preset key points on the image are obtained, and the pixel coordinates are converted into three-dimensional coordinates; s2, mirroring the target key point of the healthy limb to one side of the affected limb through a preset mirroring algorithm to obtain a three-dimensional coordinate of the target key point of the affected limb, and converting the three-dimensional coordinate of the target key point of the affected limb into a mechanical arm base coordinate system through a conversion matrix to obtain a target three-dimensional coordinate; and S3, the target three-dimensional coordinates are input into a mechanical arm control system so as to drive the mechanical arm to achieve synchronous movement. The upper limb rehabilitation device has the advantages of reliability, convenience and the like when being used for upper limb rehabilitation of a stroke patient.

Description

Technical Field

[0001] The invention belongs to the technical field of upper limb rehabilitation training, and in particular relates to a method and system for realizing mirror motion of an upper limb mirror rehabilitation robot. Background Art

[0002] With the increasing aging of the global population and the rising incidence of neurological diseases such as stroke, the number of patients with upper limb motor dysfunction has increased significantly. Stroke is a common cardiovascular disease in the elderly, with a high disability rate, resulting in irreversible motor function damage. Such motor function damage may cause many difficulties for patients in their daily lives and seriously affect their quality of life. If patients receive professional rehabilitation training treatment earlier, it will help patients recover their motor function.

[0003] In the process of upper limb rehabilitation, compared with conventional motor rehabilitation, robot mirror rehabilitation is a new form of robotic rehabilitation and has received widespread attention. In mirror therapy, the uninjured hemisphere and the injured hemisphere interact by mirroring the synchronous and coordinated movement patterns of the bilateral limbs, thereby inducing the reorganization of the motor cortex network and promoting cortical neuroplasticity. Based on the theory of mirror therapy, the use of upper limb rehabilitation robots for auxiliary training can help promote the brain's re-cognition and control of the affected limbs, and can effectively improve the upper limb motor function of stroke patients.

[0004] In the face of the growing aging population, the development of an upper limb rehabilitation training robot system combined with mirror therapy has become an urgent need and is worthy of promotion. Therefore, the research on upper limb mirror rehabilitation has certain theoretical and practical value. Summary of the invention

[0005] The present invention aims to solve the technical problem that the prior art lacks a convenient and reliable upper limb mirror rehabilitation robot for improving upper limb rehabilitation training for stroke patients, and aims to provide a method and system for realizing mirror motion of the upper limb mirror rehabilitation robot.

[0006] In order to solve the aforementioned technical problems, a first aspect of the present invention provides a method for implementing mirror motion of an upper limb mirror rehabilitation robot, the method for implementing mirror motion of an upper limb mirror rehabilitation robot comprising:

[0007] S1, acquiring an image of a patient through a depth camera, identifying preset key points of a healthy limb and preset key points of an affected limb in the image, obtaining pixel coordinates of a number of the preset key points on the image, and converting the pixel coordinates into corresponding three-dimensional coordinates through the depth camera;

[0008] S2, mirroring the target key point of the healthy limb to the side of the affected limb through a preset mirroring algorithm to obtain the three-dimensional coordinates of the target key point of the affected limb, and converting the three-dimensional coordinates of the target key point of the affected limb to the base coordinate system of the robotic arm through the pre-calculated transformation matrix between the depth camera and the robotic arm to obtain the target three-dimensional coordinates, wherein the target key point is one of the preset key points;

[0009] S3, inputting the target three-dimensional coordinates into the robot control system to drive the robot to achieve synchronous movement.

[0010] Optionally, in the aforementioned method for implementing mirror motion of an upper limb mirror rehabilitation robot, identifying preset key points of a healthy limb and preset key points of an affected limb in the image, and obtaining pixel coordinates of several preset key points on the image includes:

[0011] The preset key points are identified through the BlazePose human posture key point detection model of the Mediapipe human posture recognition framework, and the pixel coordinates of several of the preset key points on the image are obtained.

[0012] Optionally, in the mirror motion implementation method for the upper limb mirror rehabilitation robot as described above, after the pixel coordinates are converted into corresponding three-dimensional coordinates by the depth camera, invalid data and duplicate data are removed, and low-pass filtering and median filtering are used to obtain valid three-dimensional coordinate data.

[0013] Optionally, in the aforementioned method for implementing mirror motion of an upper limb mirror rehabilitation robot, removing invalid data includes:

[0014] Determine whether the currently received three-dimensional coordinate data is empty. If it is empty, it is invalid data and is removed. If it is not empty, it is valid data and is retained.

[0015] Optionally, in the aforementioned method for implementing mirror motion of an upper limb mirror rehabilitation robot, removing duplicate data preferably includes:

[0016] Determine whether the currently received three-dimensional coordinate data is the same as the three-dimensional coordinate data received last time. If they are the same, they are duplicate data and are removed. If they are different, they are valid data and are retained.

[0017] Optionally, in the method for implementing mirror motion of an upper limb mirror rehabilitation robot as described above, the transformation matrix is ​​obtained by an eye-outside-hand calibration method.

[0018] Optionally, in the aforementioned method for implementing mirror motion of an upper limb mirror rehabilitation robot, the process of obtaining the transformation matrix is ​​as follows:

[0019] S01, fixing the depth camera, determining a target area, the target area is at a preset distance from the depth camera, the end of the robotic arm is located at one side of the target area, and a calibration plate is placed at the end of the robotic arm;

[0020] S02, changing the posture of the end of the robotic arm, collecting a number of calibration plate images of different postures through the depth camera, obtaining the position and posture of the end of the robotic arm in the robotic arm base coordinate system and the transformation of the end coordinate system to the robotic arm base coordinate system at different postures;

[0021] S03, detecting the corner points of the chessboard in each calibration plate image, obtaining the coordinates of each corner point after the corner points are correct, and further optimizing the accuracy of the corner points;

[0022] S04, calculate the intrinsic parameter matrix and distortion coefficient of the depth camera, estimate the posture of each calibration plate image, obtain the rotation matrix and translation matrix, and finally obtain several groups of rotation matrices and translation matrices, and finally obtain the transformation matrix from the camera coordinate system to the robot base coordinate system.

[0023] Optionally, in the mirror motion implementation method for the upper limb mirror rehabilitation robot as described above, the preset key points of the healthy limb are the healthy limb shoulder joint key points and the healthy limb wrist joint key points, the preset key points of the affected limb are the affected limb shoulder joint key points, the target key points of the healthy limb are the healthy limb wrist joint key points, and the target key points of the affected limb are the affected limb wrist joint target key points.

[0024] Optionally, in the aforementioned method for implementing mirror motion of an upper limb mirror rehabilitation robot, step S2 comprises:

[0025] Assume that the three-dimensional coordinates of the key points of the healthy limb shoulder joint are (x1, y1, z1), the three-dimensional coordinates of the key points of the affected limb shoulder joint are (x2, y2, z2), the three-dimensional coordinates of the key points of the healthy limb wrist joint are (x4, y4, z4), the three-dimensional coordinates of the target key points of the affected limb wrist joint are (x5, y5, z5), and the target three-dimensional coordinates in the robot arm base coordinate system are (x6, y6, z6);

[0026] By formula y5=y4 and z5=z4, to obtain the three-dimensional coordinates of the target key point of the wrist joint of the affected limb after mirroring;

[0027] Suppose matrix T1 = [x5 y5 z5 1], matrix T2 = [x6 y6 z6 1], according to the formula Obtaining the three-dimensional coordinates of the target;

[0028] in, is the transformation matrix.

[0029] Optionally, in the aforementioned mirror motion implementation method for the upper limb mirror rehabilitation robot, the length of the grip at the end of the robotic arm is assumed to be L meters, and the three-dimensional coordinates of the target after the final adjustment are assumed to be (x7, y7, z7);

[0030] The final adjusted three-dimensional coordinates of the target are obtained through the formulas x7=x6, y7=y6 and z7=z6+L.

[0031] Optionally, in the aforementioned method for implementing mirror motion of an upper limb mirror rehabilitation robot, step S3 comprises:

[0032] The target three-dimensional coordinates are transmitted in real time to the robotic arm control system in the industrial computer for controlling the movement of the robotic arm through the UDP communication protocol. The robotic arm and the healthy limb are synchronized in real time through the robotic arm real-time control interface to complete the mirror rehabilitation.

[0033] Optionally, in the aforementioned method for implementing mirror motion of an upper limb mirror rehabilitation robot, before inputting the target three-dimensional coordinates into a robotic arm control system, the method further includes:

[0034] By performing interpolation processing between the target three-dimensional coordinates and the actual three-dimensional coordinates, one or several intermediate three-dimensional coordinates are obtained, and the intermediate three-dimensional coordinates and the target three-dimensional coordinates are input into the robot control system in sequence, so that the robot control system controls the robot arm to gradually move to the target position where the target three-dimensional coordinates are located.

[0035] Optionally, in the aforementioned mirror motion implementation method for the upper limb mirror rehabilitation robot, one or more intermediate three-dimensional coordinates are obtained by interpolating the target three-dimensional coordinates with the actual three-dimensional coordinates, including:

[0036] Calculate the interpolation between the current position and the target position:

[0037] ΔP=DP current

[0038] Among them, P current is the current position of the end of the robot arm, that is, the actual three-dimensional coordinate, and D is the target position of the end of the robot arm, that is, the target three-dimensional coordinate;

[0039] Decompose the interpolation into N coordinate points:

[0040]

[0041] Among them, P i is the three-dimensional coordinate of the end of the robot arm at the i-th coordinate point, N is a natural number, and N-1 is the number of intermediate points;

[0042] Get N-1 intermediate points (P1, P2, ..., P N-1 ) is one or more intermediate three-dimensional coordinates, and the target three-dimensional coordinate is P N .

[0043] In order to solve the aforementioned technical problem, a second aspect of the present invention provides a mirror motion implementation system, the mirror motion implementation system comprising:

[0044] An upper limb mirror image rehabilitation robot, wherein the upper limb mirror image rehabilitation robot has a mechanical arm;

[0045] An industrial computer, the industrial computer is connected to the upper limb mirror rehabilitation robot, and the industrial computer has a mechanical arm control system for driving the mechanical arm;

[0046] A depth camera, for acquiring an image of the patient and converting the pixel coordinates into corresponding three-dimensional coordinates;

[0047] A key point recognition module, used to recognize preset key points of a healthy limb and preset key points of an affected limb in the image, obtain pixel coordinates of a number of the preset key points on the image, and transmit the pixel coordinates to the depth camera;

[0048] A calculation module is used to mirror the target key points of the healthy limb to the side of the affected limb to obtain the three-dimensional coordinates of the target key points of the affected limb, and transform the three-dimensional coordinates of the target key points of the affected limb into the robotic arm base coordinate system through the pre-calculated transformation matrix between the depth camera and the robotic arm to obtain the target three-dimensional coordinates, and input the target three-dimensional coordinates into the robotic arm control system, wherein the target key point is one of the preset key points.

[0049] The positive progressive effect of the present invention is that the present invention identifies several preset key points in the image and converts them into three-dimensional coordinates, converts the three-dimensional coordinates into the robotic arm base coordinate system through a mirror algorithm and a transformation matrix, inputs the three-dimensional coordinate position of the target mapped to the affected limb into the robotic arm control system, drives the robotic arm to achieve synchronous movement with the healthy limb, and finally achieves the effect of mirror rehabilitation therapy. When used in the upper limb rehabilitation of stroke patients, it has the advantages of reliability and convenience. BRIEF DESCRIPTION OF THE DRAWINGS

[0050] The disclosure of the present invention will become more apparent with reference to the accompanying drawings. It should be understood that these drawings are only for illustrative purposes and are not intended to limit the scope of protection of the present invention. In the drawings:

[0051] Figure 1 A flow chart of the present invention;

[0052] Figure 2 A schematic diagram of the structure of the calibration plate of the present invention;

[0053] Figure 3 It is a schematic diagram of an application of the present invention during mirror image rehabilitation. DETAILED DESCRIPTION

[0054] The following describes the embodiments of the present invention through specific examples, and those skilled in the art can easily understand other advantages and effects of the present invention from the contents disclosed in this specification. The present invention can also be implemented or applied through other different specific embodiments, and the details in this specification can also be modified or changed in various ways based on different viewpoints and applications without departing from the spirit of the present invention.

[0055] It should be noted that the following embodiments and features therein may be combined with each other if there is no conflict.

[0056] In the description of the present invention, it should be noted that directional words such as the terms "outside", "middle", "inside", "outside", etc., which indicate directions and positional relationships, are based on the directions or positional relationships shown in the drawings. They are only for the convenience of describing the present invention and simplifying the description, and do not indicate or imply that the device or element referred to must have a specific direction, be constructed and operated in a specific direction, and cannot be understood as limiting the specific protection scope of the present invention.

[0057] In addition, the terms "first" and "second" are used for descriptive purposes only and cannot be understood as indicating or implying relative importance or implicitly indicating the number of technical features. Therefore, the definition of "first" and "second" features can explicitly or implicitly include one or more of the features. In the description of the present invention, "several" and "a number" mean two or more, unless otherwise clearly and specifically defined.

[0058] The embodiment of the present invention provides a mirror motion implementation method for an upper limb mirror rehabilitation robot, and the mirror motion implementation method can be used in upper limb rehabilitation scenarios for stroke patients. Figure 1 , the mirror motion implementation method specifically includes the following steps:

[0059] S1, key point recognition and conversion: obtain the patient's image through a depth camera, identify the preset key points of the healthy limb and the preset key points of the affected limb in the image, obtain the pixel coordinates of several preset key points on the image, and convert the pixel coordinates into corresponding three-dimensional coordinates through the depth camera.

[0060] In this step, the depth camera can be a depth camera that is readily available in the prior art, such as the RealSense D455 depth camera.

[0061] In this step, when acquiring the patient's image, the patient's whole body image or upper body image can be acquired, and the image at least includes the patient's left upper limb and right upper limb. The preset key points are some key points that are predetermined to be identified, and these key points are key points related to upper limb rehabilitation training, such as shoulder joint key points and wrist joint key points.

[0062] In some embodiments, the preset key points of the healthy limb and the preset key points of the affected limb are identified in the image, and the pixel coordinates of the preset key points on the image are obtained, including:

[0063] The preset key points are identified through the BlazePose human posture key point detection model of the Mediapipe human posture recognition framework, and the pixel coordinates of several preset key points on the image are obtained.

[0064] Mediapipe is an open source human posture recognition framework developed by Google that can be used for a variety of solutions, covering common computer vision and machine learning tasks, such as hand key point detection, face detection and facial key point recognition, human posture estimation, object detection and tracking, etc.

[0065] BlazePose is a lightweight convolutional neural network architecture and an efficient human pose detection method proposed in Google's MediaPipe framework, focusing on real-time, high-precision 2D and 3D human pose estimation. It can recognize 33 key points of the human body during reasoning and can run at 20-30 frames per second. The design goal of BlazePose is to achieve high-precision, low-latency human pose detection, especially in application scenarios that require real-time feedback (such as motion capture, augmented reality, etc.).

[0066] This embodiment uses Mediapipe to obtain the pixel coordinates of each preset key point in the image, passes the pixel coordinates to the depth camera, and finally obtains the three-dimensional coordinates of the key point by calling the relevant built-in functions of the depth camera itself for subsequent processing.

[0067] Usually, the depth camera collects two images, one is an RGB image and the other is a depth image. The RGB image and the depth image are aligned to the same frame (same time). The pixel coordinates refer to the depth pixel coordinates in the depth image.

[0068] Taking the RealSense D455 depth camera as an example, the camera's intrinsics are obtained through the RealSense SDK, and the get_distance() function is used to obtain the depth value corresponding to the depth pixel coordinate. The depth camera's intrinsics, depth pixel coordinates, and depth value are input into the rs2_deproject_pixel_to_point() function to convert the depth pixel coordinates into 3D coordinate values ​​in the camera coordinate system.

[0069] In some embodiments, after the pixel coordinates are converted into corresponding three-dimensional coordinates by the depth camera, invalid data and duplicate data are removed, and low-pass filtering is first used, and then median filtering is used to obtain valid three-dimensional coordinate data to filter the data from noise and jitter.

[0070] In some embodiments, removing invalid data includes:

[0071] Determine whether the currently received three-dimensional coordinate data is empty. If it is empty, it is invalid data and is removed. If it is not empty, it is valid data and is retained.

[0072] In some embodiments, removing duplicate data includes:

[0073] Determine whether the currently received three-dimensional coordinate data is the same as the three-dimensional coordinate data received last time. If they are the same, they are duplicate data and are removed. If they are different, they are valid data and are retained.

[0074] S2, target three-dimensional coordinate calculation: through the preset mirror algorithm, the target key point of the healthy limb is mirrored to the side of the affected limb to obtain the three-dimensional coordinates of the target key point of the affected limb; through the pre-calculated transformation matrix between the depth camera and the robotic arm, the three-dimensional coordinates of the target key point of the affected limb are converted to the robotic arm base coordinate system to obtain the target three-dimensional coordinates, where the target key point is one of the preset key points.

[0075] In this step, the target key points of the healthy limb (healthy side limb) can be mirrored to the side of the affected limb (affected side limb) through the mirror algorithm to generate the three-dimensional coordinates of the affected limb, which is the core step to achieve mirror rehabilitation therapy.

[0076] In some embodiments, the preset key points of the healthy limb are the healthy limb shoulder joint key points and the healthy limb wrist joint key points, the preset key points of the affected limb are the affected limb shoulder joint key points, the target key points of the healthy limb are the healthy limb wrist joint key points, and the target key points of the affected limb are the affected limb wrist joint target key points. Based on this, step S2 of the present invention includes the following specific processes:

[0077] Assume that the three-dimensional coordinates of the key points of the healthy limb shoulder joint are (x1, y1, z1), the three-dimensional coordinates of the key points of the affected limb shoulder joint are (x2, y2, z2), the three-dimensional coordinates of the key points of the healthy limb wrist joint are (x4, y4, z4), the three-dimensional coordinates of the target key points of the affected limb wrist joint are (x5, y5, z5), and the target three-dimensional coordinates in the robotic arm base coordinate system are (x6, y6, z6).

[0078] Among them, the three-dimensional coordinates of the key points of the healthy limb shoulder joint, the three-dimensional coordinates of the key points of the affected limb shoulder joint, and the three-dimensional coordinates of the key points of the healthy limb wrist joint can all be obtained through step S1.

[0079] First, calculate the midline position of the healthy limb shoulder joint key points and the affected limb shoulder joint key points. At the same time, since the healthy limb wrist joint key points and the affected limb wrist joint target key points are also on the same midline, the equation holds

[0080] After mirroring, the x-axis coordinates of the target key points of the healthy limb wrist joint and the affected limb wrist joint are different, but the y-axis coordinates and z-axis coordinates are the same.

[0081] Therefore, the three-dimensional coordinates (x5, y5, z5) of the target key points of the wrist joint of the affected limb after mirroring are obtained through the formulas x5=x1+x2-x4, y5=y4 and z5=z4.

[0082] Then, the obtained three-dimensional coordinates of the target key points of the affected limb wrist joint after mirroring are transformed into the robot arm base coordinate system: let matrix T1 = [x5 y5 z5 1], matrix T2 = [x6 y6 z6 1], according to the conversion relationship formula between the two Get the target three-dimensional coordinates (x6, y6, z6). Among them, is the transformation matrix.

[0083] In some embodiments, when actually conducting upper limb rehabilitation training, a handle is usually installed at the end of the robotic arm for ease of operation. The hand on the affected limb side holds the handle at the end of the robotic arm, and the handle is at a certain distance from the end of the robotic arm. After the conversion is completed, the target three-dimensional coordinates of the target key point of the wrist joint of the affected limb in the robotic arm base coordinate system are actually the position of the end of the robotic arm. To obtain the final target three-dimensional coordinates, the distance between the handle and the end of the robotic arm needs to be added. Therefore, the following optimization is also performed:

[0084] Assume that the length of the grip at the end of the robotic arm is L meters, and the final adjusted target three-dimensional coordinates are (x7, y7, z7). At this time, the positions of the x-axis and y-axis remain unchanged, and the position in the z-axis direction needs to be moved upward by L meters.

[0085] Therefore, through the formulas x7=x6, y7=y6 and z7=z6+L, the final adjusted target three-dimensional coordinates (x7, y7, z7) are obtained.

[0086] S3, synchronous drive: input the target three-dimensional coordinates into the robotic arm control system to drive the robotic arm to drive the affected limb to achieve synchronous movement.

[0087] Through the movement of the robotic arm, the patient's affected limb can be synchronized with the healthy limb in real time, ultimately achieving the effect of mirror rehabilitation therapy.

[0088] In some embodiments, this step specifically includes:

[0089] The target three-dimensional coordinates are transmitted in real time to the robotic arm control system in the industrial computer for controlling the movement of the robotic arm through the UDP communication protocol. Through the robotic arm real-time control interface, the robotic arm and the healthy limb can move synchronously in real time to complete the mirror rehabilitation.

[0090] Specifically, the data consisting of the target three-dimensional coordinates is transmitted to the code corresponding to the robotic arm control system, and the robotic arm is driven to achieve synchronous movement through the execution of the code.

[0091] During transmission, the three parameters of the target three-dimensional coordinates can be separated by commas. After the robotic arm control system receives the data, the data before the first, second, and third commas are x-axis data, y-axis data, and z-axis data, respectively. Finally, the data is separated by commas. At the same time, a new thread can be opened to transfer the processed coordinate data to the thread, and processed through the robotic arm real-time control interface, such as the servoL() function, so that the robotic arm and the healthy limb can move synchronously in real time to complete the mirror rehabilitation.

[0092] In some embodiments, in order to avoid jitter in the real-time mirroring process, before inputting the target three-dimensional coordinates into the robot control system, the data needs to be interpolated to make the data smoother. The specific process is as follows:

[0093] By interpolating the target three-dimensional coordinates with the actual three-dimensional coordinates, one or several intermediate three-dimensional coordinates are obtained, and the intermediate three-dimensional coordinates and the target three-dimensional coordinates are input into the robotic arm control system in sequence, so that the robotic arm is controlled by the robotic arm control system to gradually move to the target position where the target three-dimensional coordinates are located.

[0094] In this embodiment, not only are the target three-dimensional coordinates input into the robotic arm control system, but before the robotic arm control system uses the target three-dimensional coordinates to control the motion of the robotic arm, it also first uses one or more intermediate three-dimensional coordinates to control the motion of the robotic arm, and uses an interpolation algorithm to decompose the target position into multiple intermediate points to achieve smooth movement of the robotic arm, avoid sudden jumps or jitters of the robotic arm, and avoid discomfort to the patient due to large-scale movement of the robotic arm.

[0095] The interpolation processing of this embodiment can be implemented by using an interpolation algorithm in the prior art, preferably by using the following method:

[0096] (1) Target position calculation

[0097] Calculate the interpolation between the current position and the target position:

[0098] ΔP=DP current

[0099] Among them, P current is the current position of the end of the robot arm, that is, the actual three-dimensional coordinate, and D is the target position of the end of the robot arm, that is, the target three-dimensional coordinate;

[0100] (2) Interpolation motion

[0101] Decompose the interpolation into N coordinate points:

[0102]

[0103] Among them, P i is the three-dimensional coordinate of the end of the robot arm at the i-th coordinate point, N is a natural number, and N-1 is the number of intermediate points;

[0104] Get N-1 intermediate points (P1, P2, ..., P N-1 ) is one or more intermediate three-dimensional coordinates. N That is, the target three-dimensional coordinates corresponding to the target position D.

[0105] For example, when N=10, 9 intermediate points (P1, P2, ..., P9) are obtained, that is, 9 intermediate three-dimensional coordinates are obtained, P 10 is the target 3D coordinate.

[0106] The robot control system controls the robot to gradually move 9 coordinate points (P1, P2, ..., P9) until it moves to coordinate point P 10 .

[0107] Therefore, the actual three-dimensional coordinates input into the robot control system are:

[0108]

[0109] Among them, (P1, P2, ..., P N-1 ) is the intermediate three-dimensional coordinate, P N are the target 3D coordinates.

[0110] Reference Figure 1 , the mirror motion implementation method of the present invention further includes:

[0111] Step S0, calibrate and obtain the transformation matrix: establish an eye-outside-hand calibration method to obtain the transformation matrix between the depth camera and the robotic arm.

[0112] Based on this, the transformation matrix of step S2 is obtained by an eye-outside-hand calibration method.

[0113] Eye-in-hand calibration technology is a robot vision calibration method used to determine the transformation matrix between the camera and the robot arm. This method is to place the depth camera in a fixed position outside the robot, which will not move with the robot. The depth camera has a fixed field of view and can cover the entire working area. It does not require frequent calibration and is easier to install and maintain. Through calibration, the three-dimensional coordinates in the camera coordinate system can be converted to the base coordinate system of the robot arm.

[0114] In some embodiments, the conversion matrix is ​​obtained as follows:

[0115] S01, fix the depth camera, determine the target area, the target area is at a preset distance from the depth camera, the end of the robotic arm is located on one side of the target area, and a calibration plate is placed at the end of the robotic arm.

[0116] The target area is the area where the patient performs upper limb rehabilitation training, and the target area is the area that can be covered when the depth camera captures an image.

[0117] The calibration board needs to have various parameters determined in advance, including the number of rows and columns of the checkerboard and the actual area of ​​each small square of the checkerboard. Figure 2 It is a calibration plate structure. The intersection of two straight lines in the chessboard is the corner point. Therefore, there are 9 rows and 14 columns of corner points. The actual area of ​​each small square of the chessboard is 0.018m 2 .

[0118] S02, change the posture of the end of the robot arm, collect several calibration plate images with different postures through the depth camera, obtain the position and posture (TcpPos) of the end of the robot arm in the robot arm base coordinate system at different postures and the transformation from the end coordinate system to the robot arm base coordinate system (the inverse of TcpPos).

[0119] S03, detecting the corner points of the chessboard in each calibration plate image, obtaining the coordinates of each corner point after the corner points are correct, and further optimizing the accuracy of the corner points.

[0120] S04, calculate the intrinsic parameter matrix and distortion coefficient of the depth camera, estimate the posture of each calibration plate image, that is, the relative position of the depth camera and the chessboard, obtain the rotation matrix and translation matrix, and finally obtain several sets of rotation matrices and translation matrices, and finally obtain the transformation matrix from the camera coordinate system to the robot arm base coordinate system.

[0121] The conversion matrix is ​​a 4*4 matrix.

[0122] The embodiment of the present invention further provides a mirror motion implementation system, the mirror motion implementation system comprising:

[0123] The upper limb mirror image rehabilitation robot has a mechanical arm;

[0124] An industrial computer, the industrial computer is connected to the upper limb mirror rehabilitation robot, and the industrial computer has a robotic arm control system for driving the robotic arm;

[0125] A depth camera is used to acquire images of the patient and convert pixel coordinates into corresponding three-dimensional coordinates;

[0126] A key point recognition module is used to recognize preset key points of a healthy limb and preset key points of an affected limb in an image, obtain pixel coordinates of a number of preset key points on the image, and transmit the pixel coordinates to a depth camera;

[0127] The calculation module is used to mirror the target key points of the healthy limb to the side of the affected limb to obtain the three-dimensional coordinates of the target key points of the affected limb, and transform the three-dimensional coordinates of the target key points of the affected limb into the base coordinate system of the robotic arm through the pre-calculated transformation matrix between the depth camera and the robotic arm to obtain the target three-dimensional coordinates, and input the target three-dimensional coordinates into the robotic arm control system, where the target key point is one of the preset key points.

[0128] In some embodiments, the mirror motion implementation system further includes:

[0129] The calibration module is used to establish an eye-outside-hand calibration method to obtain the transformation matrix between the depth camera and the robotic arm.

[0130] Embodiment 1:

[0131] Reference Figure 3 The depth camera adopts RealSense D455 depth camera, which is fixed by a tripod 10. A chair 30 for patients to sit on during upper limb rehabilitation training is placed next to the end of the robotic arm 20. The RealSense D455 depth camera is 1.1 meters away from the chair, and a calibration plate is placed at the end of the robotic arm.

[0132] Changing the posture of the end of the robotic arm, the RealSense D455 depth camera collects 16 calibration plate images with different postures, and obtains the position and posture (TcpPos) of the end of the robotic arm in the robotic arm base coordinate system and the transformation from the end coordinate system to the robotic arm base coordinate system (the inverse of TcpPos) at different postures.

[0133] The calibration plate image is processed. During processing, the findChessboardCorners() function is used to detect the corner points of the chessboard in each calibration plate image. After the corner points are correct, the coordinates of each corner point are obtained, and the accuracy of the corner points is further optimized through the cornerSubPix() function.

[0134] The camera's intrinsic matrix (cameraMatrix) and distortion coefficient (distCoeffs) are calculated by the calibrateCamera() function, and the solvePnP() function is used to estimate the pose of each image, that is, the relative position of the RealSense D455 depth camera and the chessboard, to obtain the rotation matrix and translation matrix, and finally obtain 16 sets of rotation matrix and translation matrix. Finally, the calibrateHandEye() function is used to obtain the transformation matrix from the camera coordinate system to the robot base coordinate system. The matrix is ​​a 4*4 matrix.

[0135] After the above calibration is completed and the conversion matrix is ​​obtained, upper limb rehabilitation training begins.

[0136] The healthy limb of the patient is the right limb, and the affected limb is the left limb. The patient sits on a chair, and the hand of the affected limb is gripped on the handle 40 at the end of the robotic arm, and the distance between the handle 20 and the end of the robotic arm is 15 cm.

[0137] While the patient is performing upper limb rehabilitation movements, the RealSense D455 depth camera acquires images in real time and uses the BlazePose human posture key point detection model of the Mediapipe human posture recognition framework to identify three key points of the right limb (healthy limb) in the image, namely the right shoulder joint key point, the right wrist joint key point, and the left shoulder joint key point of the left limb (affected limb), and obtains the pixel coordinates of these four key points on the image.

[0138] The above real-time method can be implemented by acquiring 25 images in 1 second. Of course, different time periods can also be used to acquire different numbers of images according to different scene requirements.

[0139] When the RealSense D455 depth camera acquires images in real time, the images can be displayed in real time through the display terminal 50 connected to the RealSense D455 depth camera.

[0140] The pixel coordinates are transmitted to the get_3d_camera_coordinate() function of the RealSense D455 depth camera for processing, and the three-dimensional coordinates of the four key points are obtained in real time. After obtaining the three-dimensional coordinates of the four points, invalid data and duplicate data are removed, and then low-pass filtering is used first, and then median filtering is used to obtain the final valid three-dimensional coordinate data.

[0141] The three-dimensional coordinates of the key points of the right shoulder joint are obtained as (x1, y1, z1), the three-dimensional coordinates of the key points of the left shoulder joint are obtained as (x2, y2, z2), and the three-dimensional coordinates of the key points of the right wrist joint are obtained as (x4, y4, z4).

[0142] The three-dimensional coordinates (x5, y5, z5) of the target key point of the left wrist joint after mirroring are obtained through the formulas x5=x1+x2-x4, y5=y4 and z5=z4.

[0143] Suppose matrix T1 = [x5 y5 z5 1], matrix T2 = [x6 y6 z6 1], according to the conversion relationship formula between the two Get the target three-dimensional coordinates (x6, y6, z6).

[0144] The final adjusted target three-dimensional coordinates (x7, y7, z7) are obtained through the formulas x7=x6, y7=y6 and z7=z6+0.15.

[0145] By interpolating the target 3D coordinates (x7, y7, z7) with the actual 3D coordinates of the current end of the robotic arm, one or several intermediate 3D coordinates are obtained, and the intermediate 3D coordinates and the target 3D coordinates are input into the robotic arm control system in real time through UDP communication. The robotic arm control system receives and recognizes the x-axis data, y-axis data, and z-axis data in real time. At the same time, a new thread is opened, and the processed coordinate points are transferred to the thread, and processed through the robotic arm real-time control interface servoL() function, so that the robotic arm gradually moves to the target position where the target 3D coordinates are located, and finally the robotic arm and the right limb move synchronously in real time to complete the mirror rehabilitation.

[0146] The present invention is described in detail above in conjunction with the embodiments of the accompanying drawings. A person skilled in the art can make various variations of the present invention according to the above description. Therefore, some details in the embodiments should not be construed as limiting the present invention, and the present invention shall be protected by the scope defined by the attached claims.

Claims

1. A method for implementing mirror motion of an upper limb mirror rehabilitation robot, characterized in that: The mirror motion implementation method for the upper limb mirror rehabilitation robot comprises: S1, acquiring an image of a patient through a depth camera, identifying preset key points of a healthy limb and preset key points of an affected limb in the image, obtaining pixel coordinates of a number of the preset key points on the image, and converting the pixel coordinates into corresponding three-dimensional coordinates through the depth camera; S2, mirroring the target key point of the healthy limb to the side of the affected limb through a preset mirroring algorithm to obtain the three-dimensional coordinates of the target key point of the affected limb, and converting the three-dimensional coordinates of the target key point of the affected limb to the base coordinate system of the robotic arm through the pre-calculated transformation matrix between the depth camera and the robotic arm to obtain the target three-dimensional coordinates, wherein the target key point is one of the preset key points; S3, inputting the target three-dimensional coordinates into the robot control system to drive the robot to achieve synchronous movement.

2. The method for realizing mirror motion of an upper limb mirror rehabilitation robot according to claim 1, characterized in that: Identifying preset key points of a healthy limb and preset key points of an affected limb in the image, and obtaining pixel coordinates of several preset key points on the image, including: The preset key points are identified through the BlazePose human posture key point detection model of the Mediapipe human posture recognition framework, and the pixel coordinates of several of the preset key points on the image are obtained.

3. The method for realizing mirror motion of an upper limb mirror rehabilitation robot according to claim 1, characterized in that: After the pixel coordinates are converted into corresponding three-dimensional coordinates by the depth camera, invalid data and duplicate data are removed, and low-pass filtering and median filtering are used to obtain valid three-dimensional coordinate data; Removing invalid data preferably includes: determining whether the currently received three-dimensional coordinate data is empty, if it is empty, it is invalid data and is removed, if it is not empty, it is valid data and is retained; Removing duplicate data preferably includes: determining whether the currently received three-dimensional coordinate data is the same as the last received three-dimensional coordinate data, if the data is the same, it is duplicate data and is removed, if it is different, it is valid data and is retained.

4. The method for realizing mirror motion of an upper limb mirror rehabilitation robot according to claim 1, characterized in that: The transformation matrix is ​​obtained by an eye-outside-hand calibration method; The process of obtaining the conversion matrix is ​​preferably as follows: S01, fixing the depth camera, determining a target area, the target area is at a preset distance from the depth camera, the end of the robotic arm is located at one side of the target area, and a calibration plate is placed at the end of the robotic arm; S02, changing the posture of the end of the robotic arm, collecting a number of calibration plate images of different postures through the depth camera, obtaining the position and posture of the end of the robotic arm in the robotic arm base coordinate system and the transformation of the end coordinate system to the robotic arm base coordinate system at different postures; S03, detecting the corner points of the chessboard in each calibration plate image, obtaining the coordinates of each corner point after the corner points are correct, and further optimizing the accuracy of the corner points; S04, calculate the intrinsic parameter matrix and distortion coefficient of the depth camera, estimate the posture of each calibration plate image, obtain the rotation matrix and translation matrix, and finally obtain several groups of rotation matrices and translation matrices, and finally obtain the transformation matrix from the camera coordinate system to the robot base coordinate system.

5. The method for realizing mirror motion of an upper limb mirror rehabilitation robot according to claim 1, characterized in that: The preset key points of the healthy limb are the healthy limb shoulder joint key points and the healthy limb wrist joint key points, the preset key points of the affected limb are the affected limb shoulder joint key points, the target key points of the healthy limb are the healthy limb wrist joint key points, and the target key points of the affected limb are the affected limb wrist joint target key points.

6. The method for realizing mirror motion of an upper limb mirror rehabilitation robot according to claim 5, characterized in that: Step S2 comprises: Assume that the three-dimensional coordinates of the key points of the healthy limb shoulder joint are (x1, y1, z1), the three-dimensional coordinates of the key points of the affected limb shoulder joint are (x2, y2, z2), the three-dimensional coordinates of the key points of the healthy limb wrist joint are (x4, y4, z4), the three-dimensional coordinates of the target key points of the affected limb wrist joint are (x5, y5, z5), and the target three-dimensional coordinates in the robot arm base coordinate system are (x6, y6, z6); By formula y5=y4 and z5=z4, to obtain the three-dimensional coordinates of the target key point of the wrist joint of the affected limb after mirroring; Suppose matrix T1 = [x5 y5 z5 1], matrix T2 = [x6 y6 z6 1], according to the formula Obtaining the three-dimensional coordinates of the target; in, is the transformation matrix.

7. The method for realizing mirror motion of an upper limb mirror rehabilitation robot according to claim 6, characterized in that: The length of the grip at the end of the robotic arm is L meters, and the three-dimensional coordinates of the target after final adjustment are assumed to be (x7, y7, z7); The final adjusted three-dimensional coordinates of the target are obtained through the formulas x7=x6, y7=y6 and z7=z6+L.

8. The method for realizing mirror motion of an upper limb mirror rehabilitation robot according to claim 1, characterized in that: Step S3 includes: The target three-dimensional coordinates are transmitted in real time to the robotic arm control system in the industrial computer for controlling the movement of the robotic arm through the UDP communication protocol. The robotic arm and the healthy limb are synchronized in real time through the robotic arm real-time control interface to complete the mirror rehabilitation.

9. The method for realizing mirror motion of an upper limb mirror rehabilitation robot according to claim 1, characterized in that: Before inputting the target three-dimensional coordinates into the robot control system, the method further includes: By performing interpolation processing between the target three-dimensional coordinates and the actual three-dimensional coordinates, one or more intermediate three-dimensional coordinates are obtained, and the intermediate three-dimensional coordinates and the target three-dimensional coordinates are sequentially input into the robot control system, so that the robot control system controls the robot to gradually move to the target position where the target three-dimensional coordinates are located; By performing interpolation processing between the target three-dimensional coordinates and the actual three-dimensional coordinates, one or more intermediate three-dimensional coordinates are obtained, preferably including: Calculate the interpolation between the current position and the target position: ΔP=D-P current Among them, P current is the current position of the end of the robot arm, that is, the actual three-dimensional coordinate, and D is the target position of the end of the robot arm, that is, the target three-dimensional coordinate; Decompose the interpolation into N coordinate points: Among them, P i is the three-dimensional coordinate of the end of the robot arm at the i-th coordinate point, N is a natural number, and N-1 is the number of intermediate points; Get N-1 intermediate points (P1, P2, ..., P N-1 ) is one or more intermediate three-dimensional coordinates, and the target three-dimensional coordinate is P N .

10. A mirror motion realization system, characterized in that: The mirror motion realization system comprises: An upper limb mirror image rehabilitation robot, wherein the upper limb mirror image rehabilitation robot has a mechanical arm; An industrial computer, the industrial computer is connected to the upper limb mirror rehabilitation robot, and the industrial computer has a mechanical arm control system for driving the mechanical arm; A depth camera, for acquiring an image of the patient and converting the pixel coordinates into corresponding three-dimensional coordinates; A key point recognition module, used to recognize preset key points of a healthy limb and preset key points of an affected limb in the image, obtain pixel coordinates of a number of the preset key points on the image, and transmit the pixel coordinates to the depth camera; A calculation module is used to mirror the target key points of the healthy limb to the side of the affected limb to obtain the three-dimensional coordinates of the target key points of the affected limb, and transform the three-dimensional coordinates of the target key points of the affected limb into the robotic arm base coordinate system through the pre-calculated transformation matrix between the depth camera and the robotic arm to obtain the target three-dimensional coordinates, and input the target three-dimensional coordinates into the robotic arm control system, wherein the target key point is one of the preset key points.

Images