Intelligent linkage control and safety protection system based on electromechanical practical training equipment

Abstract

The application discloses an intelligent linkage control and safety protection system based on electromechanical practical training equipment, and relates to the technical field of industrial safety education, and specifically comprises: a motion capture module: real-time motion capture data of an operator in a practical training operation process is acquired, and the real-time motion capture data contains a three-dimensional space coordinate sequence of a hand joint of the operator; a tactile feedback acquisition module: a real-time tactile feedback signal sequence generated by an operation end of the electromechanical practical training equipment when performing an operation is acquired, and the real-time tactile feedback signal sequence contains a pressure value sequence of the operation end; by simultaneously acquiring the real-time motion capture data of the operator and the real-time tactile feedback signal sequence of the operation end of the equipment, the operation intention of the operator and the actual response state of the equipment are coupled and analyzed, and safety protection is triggered only when the timing corresponding relationship between the two is abnormal, so that false alarms when the equipment response is abnormal but the posture of the operator is normal are avoided.

Description

Technical Field

[0001] This invention relates to the field of industrial safety education technology, specifically to an intelligent linkage control and safety protection system based on electromechanical training equipment. Background Technology

[0002] When conducting electromechanical training, operators need to practice repeatedly on real equipment to master standardized operating skills. Electromechanical equipment usually involves high-speed rotation, high-power drive, or precision motion control. Improper operation may lead to equipment damage, workpiece scrapping, or even personal injury to the operator. Therefore, how to provide operators with a realistic operating experience while ensuring safety is a technical problem that urgently needs to be solved in the field of electromechanical training.

[0003] In the existing technology, there are two methods for safety protection of electromechanical training equipment: physical isolation protection and active emergency stop protection. Physical isolation protection involves setting up protective fences, safety light curtains or mechanical barriers around the dangerous area. When the operator enters the dangerous area, the sensor is triggered and the power of the equipment is immediately cut off. This method can only respond passively after the operator has entered the dangerous area. It cannot intervene in the early stage of misoperation. Moreover, frequent shutdowns will affect the continuity of training and teaching.

[0004] Active emergency stop protection involves setting up an emergency stop button on the control panel or around the equipment. When the operator or supervisor discovers a danger, they can manually press the button to stop the equipment. This method relies entirely on the subjective judgment and reaction speed of the operator or supervisor. A delayed reaction may result in an accident already occurring, and the emergency stop operation itself may be prone to errors due to tension. Summary of the Invention

[0005] To address the shortcomings of existing technologies, this invention provides an intelligent linkage control and safety protection system based on electromechanical training equipment.

[0006] To achieve the above objectives, the present invention provides the following technical solution: an intelligent linkage control and safety protection system based on electromechanical training equipment, specifically comprising: Motion capture module: acquires real-time motion capture data of the operator during the training process, the real-time motion capture data including the three-dimensional spatial coordinate sequence of the operator's hand joints; Tactile feedback acquisition module: acquires the real-time tactile feedback signal sequence generated by the operating end of the electromechanical training equipment when performing operations, the real-time tactile feedback signal sequence including the pressure value sequence of the operating end; Coupling judgment module: Based on the temporal correspondence between the real-time motion capture data and the real-time tactile feedback signal sequence, determine the degree of matching between the operator's intention and the device's response state; Permission adjustment module: When the matching degree is lower than the preset linkage safety threshold, the operation permission limit operation is executed to reduce the operator's control permission on the electromechanical training equipment.

[0007] Preferably, the motion capture module uses image acquisition sensors set around the training site to synchronously capture image sequences containing the operator's hands. For each frame, the foreground region in the image is extracted by background subtraction. The background subtraction method uses a pre-acquired empty scene image coordinate background template that does not contain the operator to compare the pixel grayscale values ​​of the current frame image with the background template. Pixels with differences greater than a preset threshold are marked as foreground pixels. The connected regions of all foreground pixels constitute the body contour of the operator. A space of interest is preset in the hand operation area, and only the foreground region within the space is retained, thereby segmenting the hand area from the body contour and obtaining a binary mask image of the hand area. Within the binarized mask image of the hand region, the boundary pixels of the hand contour are extracted by the edge detection operator. The edge detection operator calculates the gradient change of the pixel gray value in the image and marks the pixels with a gradient magnitude greater than a preset edge threshold as edge points. All edge points are connected to form a closed curve of the hand contour. Along the hand contour curve, local curvature maxima are identified on the contour through curvature analysis. These curvature maxima correspond to candidate positions of joints such as fingertips and finger roots. The curvature maxima are filtered and classified to determine the accurate pixel position of each hand joint. Based on the geometric constraints of the hand skeletal structure, the curvature maxima are classified into different joint categories, including wrist joint, thumb joint, index finger joint, middle finger joint, ring finger joint, and little finger joint. For each type of joint, the point with the highest confidence is selected from the candidate points as the final pixel position of the joint. The above steps are repeated for each frame of the image to obtain the two-dimensional pixel coordinate set of all hand joints in that frame of the image. For image frames captured at least two perspectives at the same instant, the pixel coordinates of the same physical joint point in the images from different perspectives are matched. The matching is based on epipolar geometric constraints, that is, the projection point of the same spatial point in different perspective images must be located on the corresponding epipolar line. For a joint point pixel position in the first perspective image, its epipolar equation in the second perspective image is calculated according to the pre-calibrated fundamental matrix. The pixel point that best matches the joint point feature is searched along the epipolar line in the second perspective image. This pixel point is the corresponding point of the same joint point. After completing the cross-perspective matching of all joint points, the pixel coordinate pairs of each joint point in at least two perspectives are obtained. Using pre-calibrated internal and external parameters of the image acquisition sensor, triangulation is performed on the pixel coordinate pairs of each joint point. The internal parameters include focal length, principal point coordinates, and distortion coefficients, while the external parameters include the rotation matrix and translation vector between the two image acquisition sensors. The pixel coordinate pairs are converted into normalized image coordinates. The coordinate values ​​of the joint point in three-dimensional space are obtained by solving a system of linear equations. The above triangulation calculation is repeated for all joint points at the current instant to obtain the three-dimensional spatial coordinates of all hand joint points at that instant. The three-dimensional spatial coordinates of all hand joints acquired at each instant are stored in chronological order. The time interval between adjacent instants is determined by the frame rate of the image acquisition sensor. Each set of three-dimensional spatial coordinates corresponds to a timestamp. The coordinate sets under all timestamps are arranged in ascending order of time to form a real-time three-dimensional spatial coordinate sequence of the operator's hand joints.

[0008] Preferably, the tactile feedback acquisition module is provided with at least one tactile sensor on the surface of the operating end of the electromechanical training equipment, and the tactile sensor is used to collect the pressure value generated when the operating end comes into contact with the operating object; At each sampling moment, the pressure value output by the tactile sensor is written to the next storage location of a data buffer. At the same time, a timestamp of the time when the pressure value was collected is added to each pressure value. The data buffer stores the pressure values ​​in the order they were written, so that the buffer always maintains a sequence of pressure values ​​arranged in ascending order of timestamp. The time interval between two adjacent pressure values ​​in the pressure value sequence is equal to the fixed sampling period of the tactile sensor. The sequence of pressure values ​​arranged in ascending order of timestamp is the real-time tactile feedback signal sequence. At least one marker point that can be identified by an image acquisition sensor is set on the surface of the operating end. At least two image acquisition sensors synchronously capture an image sequence containing the marker point at the same sampling frequency as the tactile sensor. The pixel position of the marker point is identified from each frame of the image. The coordinate value of the marker point in three-dimensional space is calculated by triangulation using the pixel positions of the marker point from at least two viewpoints. The coordinate value is used as the real-time position coordinate of the operating end at the sampling time. Using a unified master clock time as a reference, the master clock time of the acquisition moment is recorded as the timestamp of the pressure value for each pressure value, and the master clock time of the corresponding image exposure moment is recorded as the timestamp of the position coordinate for each operation end position. Based on the timestamp of the pressure value sequence, each pressure value in the pressure value sequence is traversed, and the position coordinate with the same timestamp is searched in the operation end position coordinate data according to the timestamp of the pressure value. If the position coordinate with the same timestamp is found, the position coordinate is associated with the pressure value and stored. If there is a missing timestamp, the interpolated position coordinates for that time are generated by linear interpolation using the position coordinates of adjacent times. The interpolated position coordinates are then associated with and stored with the pressure value. After all pressure values ​​are searched and interpolated, a set of triplet data records with a one-to-one correspondence between the timestamp, pressure value, and operation end position coordinates are generated.

[0009] Preferably, the coupling judgment module extracts the real-time three-dimensional spatial coordinate sequence of the operator's hand joints from the real-time motion capture data to generate a first hand motion trajectory curve; The real-time position coordinates of the operating end are extracted from the real-time haptic feedback signal sequence, and the position coordinates of each sampling time are connected in chronological order to generate the motion trajectory curve of the second device end. Using the same time reference, the position of the first hand movement trajectory curve and the second device end movement trajectory curve are compared frame by frame, and the position deviation value of each frame is calculated to obtain the position deviation sequence. The specific steps for comparing the positional position of the first hand movement trajectory curve with the second device end-effector movement trajectory curve frame by frame using the same time reference, and calculating the positional deviation value for each frame to obtain the positional deviation sequence are as follows: The timestamp of the first valid sampling moment in the real-time motion capture data is obtained as the comparison start time, and the timestamp of the last valid sampling moment in the real-time motion capture data is obtained as the comparison end time. Set a time pointer variable, assign the initial value of the time pointer variable to the timestamp of the start time, and read the three-dimensional spatial coordinates of the hand joints with the same timestamp from the first hand movement trajectory curve based on the timestamp value of the time pointer variable, and use them as the current hand position point. Based on the timestamp value of the time pointer variable, the three-dimensional spatial coordinates of the operation end with the same timestamp are read from the motion trajectory curve of the second device end and used as the current position point of the device end. The spatial straight-line distance between the current hand position point and the current device end position point is calculated and used as the position deviation value at the current moment. The current position deviation value is associated with the current timestamp and stored, and written to the end of the position deviation sequence. Determine if the timestamp value of the time pointer variable is equal to the timestamp of the comparison termination time. If they are equal, complete the comparison processing of all sampling times and output the position deviation sequence. If they are not equal, increase the timestamp value of the time pointer variable by the time length corresponding to one sampling period, generate a new timestamp value and assign it to the time pointer variable, and return to the step of reading the hand position point at the current time to continue execution until the timestamp value of the time pointer variable is equal to the timestamp of the comparison termination time. The maximum position deviation value is extracted from the position deviation sequence and determined as the degree of matching between the operator's intention and the equipment response status.

[0010] Preferably, the permission adjustment module extracts the maximum position deviation value from the position deviation sequence, stores the maximum position deviation value in a first register, reads a preset linkage safety threshold from a pre-stored parameter storage area, and stores the preset linkage safety threshold in a second register. A numerical comparator compares the maximum position deviation value stored in the first register with the preset linkage safety threshold stored in the second register bit by bit. The numerical comparator first compares the most significant bits of the two values. If the most significant bits are different, the value with the most significant bit set to 1 is determined to be greater than the value with the most significant bit set to 0, and the comparison result is output. If the most significant bits are the same, the next most significant bit is compared, and so on, until all significant bits have been compared. Based on the result of the bit-by-bit comparison, the numerical comparator outputs a Boolean state signal. When the maximum position deviation value is greater than the preset linkage safety threshold, the Boolean state signal is at the first logic level, and when the maximum position deviation value is not greater than the preset linkage safety threshold, the Boolean state signal is at the second logic level. When the Boolean state signal output by the numerical comparator is at the first logic level, it is determined that the maximum position deviation value is greater than the preset linkage safety threshold, and a pulse generation circuit is triggered to generate an access restriction trigger signal. A permission restriction trigger signal is sent to an operation permission regulator. In response to the permission restriction trigger signal, the operation permission regulator switches the operator's current control permission from the first control permission to the second control permission. The number of operable parameters corresponding to the second control permission is less than the number of operable parameters corresponding to the first control permission.

[0011] This invention provides an intelligent linkage control and safety protection system based on electromechanical training equipment, which has the following beneficial effects: This invention simultaneously acquires real-time motion capture data of the operator and real-time tactile feedback signal sequences from the device's operating end, coupling and analyzing the operator's intentions with the actual response state of the device. Safety protection is only triggered when the temporal correspondence between the two is abnormal, avoiding missed alarms when the device response is abnormal but the operator's posture is normal. The invention synchronously captures depth image sequences containing the operator's entire body through image acquisition sensors around the training site, extracts three-dimensional point cloud data of the operator's body contour from the depth image sequences, identifies the operator's skeletal joints, and generates the three-dimensional spatial coordinates of each skeletal joint in each frame of the image. The acquired motion capture data can more realistically reflect the operator's actual intentions. Attached Figure Description

[0012] Figure 1 This is a schematic diagram of the system structure of the present invention. Detailed Implementation

[0013] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0014] Please see Figure 1 This invention provides an intelligent linkage control and safety protection system based on electromechanical training equipment, comprising: Motion capture module: acquires real-time motion capture data of the operator during the training process, the real-time motion capture data including the three-dimensional spatial coordinate sequence of the operator's hand joints; In this embodiment of the invention, the motion capture module needs to be specifically described. The motion capture module uses image acquisition sensors set around the training site to synchronously capture image sequences containing the operator's hands. For each frame of image, the foreground region in the image is extracted by background subtraction. The background subtraction method uses a pre-acquired empty scene image coordinate background template that does not contain the operator to compare the pixel grayscale values ​​of the current frame image with the background template. Pixels with differences greater than a preset threshold are marked as foreground pixels. The connected regions of all foreground pixels constitute the body contour of the operator. A space of interest is preset in the hand operation area, and only the foreground region located within the space is retained, thereby segmenting the hand area from the body contour and obtaining a binarized mask image of the hand area. Within the binarized mask image of the hand region, the boundary pixels of the hand contour are extracted by the edge detection operator. The edge detection operator calculates the gradient change of the pixel gray value in the image and marks the pixels with a gradient magnitude greater than a preset edge threshold as edge points. All edge points are connected to form a closed curve of the hand contour. Along the hand contour curve, local curvature maxima are identified on the contour through curvature analysis. These curvature maxima correspond to candidate positions of joints such as fingertips and finger roots. The curvature maxima are filtered and classified to determine the accurate pixel position of each hand joint. Based on the geometric constraints of the hand skeletal structure, such as the relative distance between fingertips and the direction of the line connecting the fingertips and wrist, the curvature maxima are classified into different joint categories, including wrist joint, thumb joint, index finger joint, middle finger joint, ring finger joint, and little finger joint. For each type of joint, the point with the highest confidence is selected from the candidate points as the final pixel position of the joint. The above steps are repeated for each frame of the image to obtain the two-dimensional pixel coordinate set of all hand joints in that frame of the image. For image frames captured at least two perspectives at the same instant, the pixel coordinates of the same physical joint point in the images from different perspectives are matched. The matching is based on epipolar geometric constraints, that is, the projection point of the same spatial point in different perspective images must be located on the corresponding epipolar line. For a joint point pixel position in the first perspective image, its epipolar equation in the second perspective image is calculated according to the pre-calibrated fundamental matrix. The pixel point that best matches the joint point feature is searched along the epipolar line in the second perspective image. This pixel point is the corresponding point of the same joint point. After completing the cross-perspective matching of all joint points, the pixel coordinate pairs of each joint point in at least two perspectives are obtained. Using pre-calibrated internal and external parameters of the image acquisition sensor, triangulation is performed on the pixel coordinate pairs of each joint point. The internal parameters include focal length, principal point coordinates, and distortion coefficients, while the external parameters include the rotation matrix and translation vector between the two image acquisition sensors. The pixel coordinate pairs are converted into normalized image coordinates. The coordinate values ​​of the joint point in three-dimensional space are obtained by solving a system of linear equations. The above triangulation calculation is repeated for all joint points at the current instant to obtain the three-dimensional spatial coordinates of all hand joint points at that instant. The three-dimensional spatial coordinates of all hand joints acquired at each instant are stored in chronological order. The time interval between adjacent instants is determined by the frame rate of the image acquisition sensor. Each set of three-dimensional spatial coordinates corresponds to a timestamp. The coordinate sets under all timestamps are arranged in ascending order of time to form a real-time three-dimensional spatial coordinate sequence of the operator's hand joints.

[0015] It should be noted that calculating the epipolar equation in the second-view image based on the pre-calibrated fundamental matrix specifically includes the following steps: Obtain the pixel position of the hand joint to be processed in the first-view image. The pixel position consists of the horizontal and vertical coordinates. Convert the pixel position into homogeneous coordinate form, that is, construct a vector containing three values, where the first two values ​​are the horizontal and vertical coordinates of the hand joint to be processed, respectively, and the third value is set to a fixed constant of one. Read the pre-calibrated and stored fundamental matrix. The fundamental matrix is ​​a three-row, three-column numerical matrix. Each value in the numerical matrix represents the calibration result of the geometric constraint relationship between the two image acquisition sensors. The fundamental matrix is ​​obtained by calculating the multi-view images of the known spatial point array during the initialization phase. The basic matrix is ​​combined with the homogeneous coordinates of the first-view key points. The specific calculation process is as follows: multiply the three values ​​in the first row of the basic matrix by the three values ​​of the homogeneous coordinates respectively and then sum them to obtain the first result value. Multiply the three values ​​in the second row of the basic matrix by the three values ​​of the homogeneous coordinates respectively and then sum them to obtain the second result value. Multiply the three values ​​in the third row of the basic matrix by the three values ​​of the homogeneous coordinates respectively and then sum them to obtain the third result value. These three result values ​​constitute a new three-dimensional vector. The newly generated three-dimensional vector is the coefficient of the straight line equation corresponding to the epipolar line in the second-view image. The first value of the three-dimensional vector represents the coefficient of the horizontal coordinate in the straight line equation, the second value represents the coefficient of the vertical coordinate in the straight line equation, and the third value represents the constant term. Therefore, all points in the second-view image that satisfy the straight line equation, that is, all pixel positions that make the first value multiplied by the horizontal coordinate plus the second value multiplied by the vertical coordinate plus the third value equal to zero, are the possible corresponding points of the first-view key points. Based on the coefficients of the above-mentioned linear equation, a unique straight line is determined in the second-view image. This straight line runs through the entire image plane. When searching for corresponding points in the second-view image that match the key points of the first view, it is only necessary to perform feature comparison pixel by pixel on this straight line, thereby reducing the two-dimensional search range to a one-dimensional straight line.

[0016] To acquire the operator's hand movement information in three-dimensional space, at least two image acquisition sensors need to be set up around the training site. These image acquisition sensors are simultaneously aimed at the operating area where the operator's hand is located from different perspectives to ensure that the hand is always within the common field of view of at least two devices during the movement. The at least two image acquisition sensors are connected by a synchronous trigger signal to ensure that image frames of the operator's hand are captured at the same moment, thereby forming a multi-view image sequence that is strictly aligned in time. The image sequence output by each image acquisition sensor consists of a series of consecutive images, and each image frame records the two-dimensional projection of the operator's hand at a certain moment from that perspective. After obtaining a synchronized multi-view image sequence, each frame of the image needs to be processed to identify the position of the operator's hand joints. Hand joints refer to the key points that make up the skeletal structure of the hand, such as the wrist joint, the root joints of each finger, the middle joints of the fingers, and the fingertip joints. For each frame of the image, the hand contour is extracted by image processing methods, and the pixel coordinates of each hand joint are located within the contour. The recognition operation is performed frame by frame, that is, each frame of the image outputs a set of pixel positions of the hand joints. Each set of pixel positions corresponds to the hand posture observed from a certain viewpoint at a certain moment. For the same moment, at least two viewpoints of the image output the pixel positions of the hand joints under their respective viewpoints. These pixel positions under different viewpoints together describe the two-dimensional projection distribution of the hand in space at that moment. After obtaining the pixel positions of the hand joints from at least two perspectives at the same instant, 3D reconstruction is performed using binocular or multi-view vision principles. For each hand joint, the pixel position in the first perspective image and the pixel position in the second perspective image form a corresponding point pair. Based on the pre-calibrated internal parameters (such as focal length and principal point coordinates) and external parameters (such as relative position and relative posture) of the two image acquisition sensors, the actual position coordinates of the joint in 3D space are calculated using triangulation. The above calculation is repeated for each frame of each instant to obtain the 3D spatial coordinates of all hand joints at that instant. The 3D spatial coordinates obtained from all instants are arranged in chronological order to form a real-time 3D spatial coordinate sequence of the operator's hand joints.

[0017] Tactile feedback acquisition module: acquires the real-time tactile feedback signal sequence generated by the operating end of the electromechanical training equipment when performing operations, the real-time tactile feedback signal sequence including the pressure value sequence of the operating end; In this embodiment of the invention, the tactile feedback acquisition module is specifically described. The tactile feedback acquisition module has at least one tactile sensor on the surface of the operating end of the electromechanical training equipment. The tactile sensor is used to collect the pressure value generated when the operating end comes into contact with the operating object. At each sampling moment, the pressure value output by the tactile sensor is written to the next storage location of a data buffer. At the same time, a timestamp of the time when the pressure value was collected is added to each pressure value. The data buffer stores the pressure values ​​in the order they were written, so that the buffer always maintains a sequence of pressure values ​​arranged in ascending order of timestamp. The time interval between two adjacent pressure values ​​in the pressure value sequence is equal to the fixed sampling period of the tactile sensor. The sequence of pressure values ​​arranged in ascending order of timestamp is the real-time tactile feedback signal sequence. At least one marker point that can be identified by an image acquisition sensor is set on the surface of the operating end. At least two image acquisition sensors synchronously capture an image sequence containing the marker point at the same sampling frequency as the tactile sensor. The pixel position of the marker point is identified from each frame of the image. The coordinate value of the marker point in three-dimensional space is calculated by triangulation using the pixel positions of the marker point from at least two viewpoints. The coordinate value is used as the real-time position coordinate of the operating end at the sampling time. Using a unified master clock time as a reference, the master clock time of the acquisition moment is recorded as the timestamp of the pressure value for each pressure value, and the master clock time of the corresponding image exposure moment is recorded as the timestamp of the position coordinate for each operation end position. Based on the timestamp of the pressure value sequence, each pressure value in the pressure value sequence is traversed, and the position coordinate with the same timestamp is searched in the operation end position coordinate data according to the timestamp of the pressure value. If the position coordinate with the same timestamp is found, the position coordinate is associated with the pressure value and stored. If there is a missing timestamp, the interpolated position coordinates for that time are generated by linear interpolation using the position coordinates of adjacent times. The interpolated position coordinates are then associated with and stored with the pressure value. After all pressure values ​​are searched and interpolated, a set of triplet data records with a one-to-one correspondence between the timestamp, pressure value, and operation end position coordinates are generated.

[0018] It should be noted that the tactile sensor outputs a pressure value at each sampling moment. The pressure value is a numerical value that represents the current contact force. The system has an internal data buffer area, which is used to temporarily store all pressure values ​​collected in a recent period of time. At the first sampling moment, the first pressure value collected is written to the first storage location of the buffer area, and the sampling time corresponding to the pressure value is recorded as a timestamp. At the second sampling moment, the system writes the second pressure value acquired into the second storage location of the buffer, and records the acquisition time corresponding to the pressure value as a timestamp. At each subsequent sampling moment, the currently acquired pressure value is written into the next storage location of the buffer, and a corresponding acquisition time timestamp is appended to each pressure value. When the buffer is full, the earliest pressure value and its timestamp are removed according to the first-in-first-out principle, and the latest pressure value and its timestamp are written to the end of the buffer. Through continuous writing operations, the buffer always maintains a pressure value sequence arranged in ascending order of timestamps. The time interval between two adjacent pressure values ​​in the pressure value sequence is equal to the fixed sampling period of the tactile sensor, thus fully presenting the dynamic trajectory of the pressure value changing continuously over time. This sequence is the real-time tactile feedback signal sequence. The real-time position coordinates of the end effector are obtained in the same way as the motion capture unit. At least one marker point that can be recognized by the image acquisition sensor is set on the surface of the end effector. The marker point has significant visual features that distinguish it from the background. The image acquisition sensor synchronously captures an image sequence containing the marker point of the end effector at the same sampling frequency as the tactile sensor. For each frame of the image, the pixel position of the marker point in the image is identified by image processing. The identification process includes binarizing the image to extract the marker point region, calculating the center pixel coordinates of the marker point region as the pixel position of the marker point, using the pixel positions of the marker point captured synchronously by the image acquisition sensors from at least two perspectives, and calculating the coordinate value of the marker point in three-dimensional space by triangulation. The coordinate value is the real-time position coordinate of the end effector at that sampling time. The above identification and calculation are repeated for each frame of the image to obtain the real-time position coordinates of the end effector corresponding to each sampling time. The system has a unified master clock, which provides a time reference for synchronizing the tactile sensor and the image acquisition sensor. Each sampling moment of the tactile sensor and each exposure moment of the image acquisition sensor are recorded with reference to the master clock. For each pressure value collected by the tactile sensor, the master clock time at the moment of pressure value collection is recorded as the timestamp of the pressure value. For each frame of image captured by the image acquisition sensor, the master clock time at the moment of exposure of the frame is recorded as the timestamp of the image, which is then used as the timestamp of the operation end position coordinates calculated from the frame. While generating a real-time haptic feedback signal sequence, the timestamp of each pressure value is associated with and stored with the pressure value itself. After calculating the position coordinates of the operation end at each sampling moment, the position coordinates are associated with and stored with their corresponding timestamps. Then, based on the timestamps of the pressure value sequence, each pressure value in the pressure value sequence is traversed. The position coordinates with the same timestamps are searched in the position coordinate data according to the timestamps of the pressure values. If the position coordinates with the same timestamps are found, the position coordinates are associated with the pressure value and stored as a complete data record. If there is a missing timestamp in the location coordinate data, linear interpolation is performed using the location coordinates of adjacent times to generate the interpolated location coordinates for that time. The interpolated location coordinates are then associated with and stored with the pressure value. After searching and interpolation, a set of triplet data records containing timestamp, pressure value, and operation end location coordinates is finally generated. The pressure value and location coordinates in each record are strictly aligned in time, thus achieving complete time synchronization between the real-time location coordinates and the pressure value sequence.

[0019] Coupling judgment module: Based on the temporal correspondence between the real-time motion capture data and the real-time haptic feedback signal sequence, determine the degree of matching between the operator's operational intention and the device's response state; In this embodiment of the invention, the coupling judgment module needs to be specifically described. The coupling judgment module extracts the real-time three-dimensional spatial coordinate sequence of the operator's hand joints from the real-time motion capture data and generates the first hand motion trajectory curve. The real-time position coordinates of the operating end are extracted from the real-time haptic feedback signal sequence, and the position coordinates of each sampling time are connected in chronological order to generate the motion trajectory curve of the second device end. Using the same time reference, the position of the first hand movement trajectory curve and the second device end movement trajectory curve are compared frame by frame, and the position deviation value of each frame is calculated to obtain the position deviation sequence. The specific steps for comparing the positional position of the first hand movement trajectory curve with the second device end-effector movement trajectory curve frame by frame using the same time reference, and calculating the positional deviation value for each frame to obtain the positional deviation sequence are as follows: The timestamp of the first valid sampling moment in the real-time motion capture data is obtained as the comparison start time, and the timestamp of the last valid sampling moment in the real-time motion capture data is obtained as the comparison end time. Set a time pointer variable, assign the initial value of the time pointer variable to the timestamp of the start time, and read the three-dimensional spatial coordinates of the hand joints with the same timestamp from the first hand movement trajectory curve based on the timestamp value of the time pointer variable, and use them as the current hand position point. Based on the timestamp value of the time pointer variable, the three-dimensional spatial coordinates of the operation end with the same timestamp are read from the motion trajectory curve of the second device end and used as the current position point of the device end. The spatial straight-line distance between the current hand position point and the current device end position point is calculated and used as the position deviation value at the current moment. The current position deviation value is associated with the current timestamp and stored, and written to the end of the position deviation sequence. Determine if the timestamp value of the time pointer variable is equal to the timestamp of the comparison termination time. If they are equal, complete the comparison processing of all sampling times and output the position deviation sequence. If they are not equal, increase the timestamp value of the time pointer variable by the time length corresponding to one sampling period, generate a new timestamp value and assign it to the time pointer variable, and return to the step of reading the hand position point at the current time to continue execution until the timestamp value of the time pointer variable is equal to the timestamp of the comparison termination time. The maximum position deviation value is extracted from the position deviation sequence and determined as the degree of matching between the operator's intention and the equipment response status.

[0020] It should be noted that the real-time motion capture data contains the three-dimensional spatial coordinates of the operator's hand joints at each sampling moment. These coordinate values ​​are arranged in chronological order, forming a set of discrete position points of the hand joints changing over time. In order to obtain a continuous motion trajectory, these discrete position points are connected sequentially in chronological order. The system reads the three-dimensional spatial coordinates of the hand joints at the first sampling moment from the real-time motion capture data and uses it as the starting point of the trajectory curve. Then, it reads the three-dimensional spatial coordinates of the hand joints at the second sampling moment and uses it as the second point of the trajectory curve. The system reads the three-dimensional spatial coordinates of the hand joints at all sampling moments in sequence and connects these coordinate points in chronological order to form a continuous curve describing the movement path of the hand joints in three-dimensional space over time. This curve is the first hand motion trajectory curve. This curve completely records the spatial position change process of the operator's hand throughout the entire training operation, reflecting the operator's operational intention and motion execution trajectory. The real-time tactile feedback signal sequence contains not only pressure values, but also the real-time position coordinates of the operating end synchronized with each pressure value. These position coordinates are also a set of discrete points arranged in chronological order, describing the spatial position of the operating end at each sampling moment. The position coordinates of the operating end at each sampling moment are read sequentially from the real-time tactile feedback signal sequence. Starting from the position coordinates of the first sampling moment, the position coordinates of the second, third, and so on until the last sampling moment are read sequentially. Connecting these position coordinates in chronological order forms a continuous curve describing the movement path of the operating end in three-dimensional space over time. The continuous curve is the motion trajectory curve of the second device end. The motion trajectory curve fully records the spatial position change process of the device operating end during the entire training operation, reflecting the actual response state of the device to the operator's actions. The first hand movement trajectory curve and the second device end effector movement trajectory curve are both generated based on the same time reference. That is, each point on the two curves corresponds to the same sampling time. The comparison is performed frame by frame with the sampling time as the unit. That is, at the same time, the three-dimensional spatial coordinates of the hand joints on the first hand movement trajectory curve at that time are read, and the three-dimensional spatial coordinates of the operating end on the second device end effector movement trajectory curve at that time are read. The straight-line distance between these two three-dimensional spatial coordinate points is calculated. The straight-line distance is the position deviation value at that time. The above calculation is repeated for the first sampling time to obtain the first position deviation value. The above calculation is repeated for the second sampling time to obtain the second position deviation value. The calculation is performed for all sampling times one by one to obtain a series of position deviation values ​​arranged in chronological order. These position deviation values ​​are arranged in chronological order according to the corresponding sampling times to form a position deviation sequence. The position deviation sequence reflects the degree of spatial deviation between the operator's hand and the device end effector at every instant during the entire operation process. The positional deviation sequence contains a set of values, each representing the spatial distance between the hand and the device's end effector at a sampling moment. The process iterates through all values ​​in the positional deviation sequence, comparing each value sequentially and recording the maximum value encountered during the current iteration. First, the first positional deviation value is set as the current maximum value. Then, the second positional deviation value is read and compared with the current maximum value. If the second positional deviation value is greater than the current maximum value, the current maximum value is updated to the second positional deviation value; otherwise, the current maximum value remains unchanged, and the process continues until all values ​​in the positional deviation sequence have been traversed, resulting in the final positional deviation sequence. The current maximum value is the maximum positional deviation value in the sequence. The maximum positional deviation value represents the maximum spatial deviation between the operator's hand and the end effector of the device during the entire operation. Since the operator's intention is reflected through the movement trajectory of the hand and the device's response state is reflected through the movement trajectory of the end effector of the device, the maximum deviation between the two directly reflects the maximum inconsistency between the operator's intention and the device's response state. The maximum positional deviation value is used as a quantitative representation of the degree of matching. That is, when the maximum positional deviation value is small, it indicates that the operator's intention and the device's response state are well matched. When the maximum positional deviation value is large, it indicates that there is a significant deviation between the two, and safety intervention is required.

[0021] Permission adjustment module: When the matching degree is lower than the preset linkage safety threshold, the operation permission limit operation is executed to reduce the operator's control permission on the electromechanical training equipment.

[0022] In this embodiment of the invention, the permission adjustment module needs to be specifically described. The permission adjustment module extracts the maximum position deviation value from the position deviation sequence, stores the maximum position deviation value in a first register, reads a preset linkage safety threshold from a pre-stored parameter storage area, and stores the preset linkage safety threshold in a second register. A numerical comparator compares the maximum position deviation value stored in the first register with the preset linkage safety threshold stored in the second register bit by bit. The numerical comparator first compares the most significant bits of the two values. If the most significant bits are different, the value with the most significant bit set to 1 is determined to be greater than the value with the most significant bit set to 0, and the comparison result is output. If the most significant bits are the same, the next most significant bit is compared, and so on, until all significant bits have been compared. Based on the result of the bit-by-bit comparison, the numerical comparator outputs a Boolean state signal. When the maximum position deviation value is greater than the preset linkage safety threshold, the Boolean state signal is at the first logic level, and when the maximum position deviation value is not greater than the preset linkage safety threshold, the Boolean state signal is at the second logic level. When the Boolean state signal output by the numerical comparator is at the first logic level, it is determined that the maximum position deviation value is greater than the preset linkage safety threshold, and a pulse generation circuit is triggered to generate an access restriction trigger signal. A permission restriction trigger signal is sent to an operation permission regulator. In response to the permission restriction trigger signal, the operation permission regulator switches the operator's current control permission from the first control permission to the second control permission. The number of operable parameters corresponding to the second control permission is less than the number of operable parameters corresponding to the first control permission.

[0023] It should be noted that after obtaining the maximum positional deviation value, this value needs to be compared with a pre-set threshold value to determine whether the current operating state is within a safe range. The preset linkage safety threshold is a fixed value. The preset linkage safety threshold is pre-calibrated and stored in the system based on factors such as the safety operation specifications of the training equipment, the operator's skill level, and the allowable deviation tolerance of the equipment. The preset linkage safety threshold represents the maximum allowable spatial deviation between the operator's hand and the end of the equipment. When the actual deviation exceeds this value, it means that there is an unacceptable inconsistency between the operator's operating intention and the actual response of the equipment, which may indicate misoperation or an impending danger. When the system performs the comparison operation, the maximum positional deviation value is used as the comparison number, and the preset linkage safety threshold is used as the reference number. The relationship between the two is determined by the numerical value judgment logic. The output of the comparison step is a Boolean value, that is, it is true when the maximum positional deviation value is greater than the preset threshold, and false when the maximum positional deviation value is not greater than the preset threshold. When the output of the comparison step is true, indicating that the maximum position deviation has exceeded the allowable safety range, the system needs to generate a control command to initiate a safety protection action. This control command is presented as an electrical signal, called the access restriction trigger signal. The generation of the access restriction trigger signal is executed by the system's logic judgment circuit or control unit. Upon receiving a true comparison result, it automatically generates an electrical pulse with a specific amplitude, pulse width, or encoding format according to preset trigger conditions. The trigger signal carries two key pieces of information: first, it identifies the access restriction operation that needs to be performed; second, it identifies the access restriction level that needs to be performed. To ensure reliable signal transmission, the trigger signal is amplified by a signal driving circuit after generation, giving it sufficient driving capability to overcome losses and loads on subsequent transmission paths. The generated permission restriction trigger signal needs to be transmitted to the actuator, i.e., the operation permission regulator, through a signal transmission line. The operation permission regulator is a key node set between the operator's control end and the equipment control end. Its physical location is usually in the signal path of the control system. It can intercept, forward, or modify the control commands issued by the operator. The permission restriction trigger signal is transmitted to the signal receiving port of the operation permission regulator through wired or wireless means. To ensure the accuracy and real-time performance of the signal transmission, the transmission line is shielded to prevent external electromagnetic interference. At the same time, the signal receiving port is equipped with a signal identification circuit to verify whether the received signal is a valid trigger signal issued by this system and to avoid false triggering. When the operation permission regulator successfully receives and identifies the trigger signal, the trigger signal becomes the input command to drive the regulator to execute subsequent actions. The operation permission regulator has an internal permission status storage unit and a permission switching execution unit. The permission status storage unit predefines at least two different control permission levels, namely the first control permission and the second control permission. The first control permission corresponds to the full range of parameters that the operator can operate, including all adjustable parameters such as the speed, force, and stroke of the equipment. The second control permission corresponds to a restricted range of parameters, which usually only retains the necessary safety operation functions, such as only allowing low-speed operation or only allowing emergency stop operations. The number of operable parameters in the second control permission is significantly less than that in the first control permission. When the operation permission regulator receives a permission restriction trigger signal, the permission switching execution unit immediately reads the parameter configuration corresponding to the second control permission in the permission status storage unit and switches the currently effective permission configuration from the first control permission to the second control permission.

[0024] Those skilled in the art will recognize that the units and algorithm steps of the various examples described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are implemented in hardware or software depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of this application.

[0025] Those skilled in the art will understand that, for the sake of convenience and brevity, the specific working processes of the systems, devices, and units described above can be referred to the corresponding processes in the foregoing method embodiments, and will not be repeated here.

[0026] In the several embodiments provided in this application, it should be understood that the disclosed systems, apparatuses, and methods can be implemented in other ways. For example, the apparatus embodiments described above are merely illustrative; for instance, the division of units is only a logical functional division, and in actual implementation, there may be other division methods. For example, multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed. Furthermore, the coupling or direct coupling or communication connection shown or discussed may be through some interfaces; the indirect coupling or communication connection between apparatuses or units may be electrical, mechanical, or other forms.

[0027] The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the units can be selected to achieve the purpose of this embodiment according to actual needs.

[0028] The above description is merely a specific embodiment of this application, but the scope of protection of this application is not limited thereto. Any variations or substitutions that can be easily conceived by those skilled in the art within the scope of the technology disclosed in this application should be included within the scope of protection of this application. Therefore, the scope of protection of this application should be determined by the scope of protection of the described technical solution.

Claims

1. A smart linkage control and safety protection system based on electromechanical training equipment, characterized in that, include: Motion capture module: acquires real-time motion capture data of the operator during the training process, the real-time motion capture data including the three-dimensional spatial coordinate sequence of the operator's hand joints; Tactile feedback acquisition module: acquires the real-time tactile feedback signal sequence generated by the operating end of the electromechanical training equipment when performing operations, the real-time tactile feedback signal sequence including the pressure value sequence of the operating end; Coupling judgment module: Based on the temporal correspondence between the real-time motion capture data and the real-time tactile feedback signal sequence, determine the degree of matching between the operator's operational intention and the device's response state; Permission adjustment module: When the matching degree is lower than the preset linkage safety threshold, the operation permission limit operation is executed to reduce the operator's control permission on the electromechanical training equipment.

2. The intelligent linkage control and safety protection system based on electromechanical training equipment according to claim 1, characterized in that, The motion capture module uses image acquisition sensors set around the training site to synchronously capture image sequences containing the operator's hands. For each frame, the foreground region in the image is extracted using a background subtraction method. The background subtraction method uses a pre-acquired empty scene image coordinate background template that does not contain the operator to compare the pixel grayscale values ​​of the current frame image with the background template. Pixels with differences greater than a preset threshold are marked as foreground pixels. The connected regions of all foreground pixels constitute the body contour of the operator. A space of interest is preset in the hand operation area, and only the foreground region within the space is retained, thereby segmenting the hand area from the body contour and obtaining a binary mask image of the hand area. Within the binarized mask image of the hand region, the boundary pixels of the hand contour are extracted by the edge detection operator. The edge detection operator calculates the gradient change of the pixel gray value in the image and marks the pixels with a gradient magnitude greater than a preset edge threshold as edge points. All edge points are connected to form a closed curve of the hand contour. Along the hand contour curve, local curvature maxima are identified on the contour through curvature analysis. These curvature maxima correspond to candidate positions of joints such as fingertips and finger roots. The curvature maxima are filtered and classified to determine the accurate pixel position of each hand joint. Based on the geometric constraints of the hand skeletal structure, the curvature maxima are classified into different joint categories, including wrist joint, thumb joint, index finger joint, middle finger joint, ring finger joint, and little finger joint. For each type of joint, the point with the highest confidence is selected from the candidate points as the final pixel position of the joint. The above steps are repeated for each frame of the image to obtain the two-dimensional pixel coordinate set of all hand joints in that frame of the image. For image frames captured at least two perspectives at the same instant, the pixel coordinates of the same physical joint point in the images from different perspectives are matched. The matching is based on epipolar geometric constraints, that is, the projection point of the same spatial point in different perspective images must be located on the corresponding epipolar line. For a joint point pixel position in the first perspective image, its epipolar equation in the second perspective image is calculated according to the pre-calibrated fundamental matrix. In the second perspective image, the pixel point that best matches the joint point feature is searched along the epipolar line. This pixel point is the corresponding point of the same joint point. After completing the cross-perspective matching of all joint points, the pixel coordinate pairs of each joint point in at least two perspectives are obtained.

3. The intelligent linkage control and safety protection system based on electromechanical training equipment according to claim 2, characterized in that, Using pre-calibrated internal and external parameters of the image acquisition sensor, triangulation is performed on the pixel coordinate pairs of each joint point. The internal parameters include focal length, principal point coordinates, and distortion coefficients, while the external parameters include the rotation matrix and translation vector between the two image acquisition sensors. The pixel coordinate pairs are converted into normalized image coordinates. The coordinate values ​​of the joint point in three-dimensional space are obtained by solving a system of linear equations. The above triangulation calculation is repeated for all joint points at the current instant to obtain the three-dimensional spatial coordinates of all hand joint points at that instant. The three-dimensional spatial coordinates of all hand joints acquired at each instant are stored in chronological order. The time interval between adjacent instants is determined by the frame rate of the image acquisition sensor. Each set of three-dimensional spatial coordinates corresponds to a timestamp. The coordinate sets under all timestamps are arranged in ascending order of time to form a real-time three-dimensional spatial coordinate sequence of the operator's hand joints.

4. The intelligent linkage control and safety protection system based on electromechanical training equipment according to claim 1, characterized in that, The tactile feedback acquisition module is equipped with at least one tactile sensor on the surface of the operating end of the electromechanical training equipment, and uses the tactile sensor to collect the pressure value generated when the operating end comes into contact with the operating object; At each sampling moment, the pressure value output by the tactile sensor is written to the next storage location of a data buffer. At the same time, a timestamp of the time when the pressure value was collected is added to each pressure value. The data buffer stores the pressure values ​​in the order they were written, so that the buffer always maintains a sequence of pressure values ​​arranged in ascending order of timestamps. The time interval between two adjacent pressure values ​​in the pressure value sequence is equal to the fixed sampling period of the tactile sensor. The sequence of pressure values ​​arranged in ascending order of timestamps is the real-time tactile feedback signal sequence.

5. The intelligent linkage control and safety protection system based on electromechanical training equipment according to claim 4, characterized in that, At least one marker point that can be identified by an image acquisition sensor is set on the surface of the operating end. At least two image acquisition sensors synchronously capture an image sequence containing the marker point at the same sampling frequency as the tactile sensor. The pixel position of the marker point is identified from each frame of the image. The coordinate value of the marker point in three-dimensional space is calculated by triangulation using the pixel positions of the marker point from at least two viewpoints. The coordinate value is used as the real-time position coordinate of the operating end at the sampling time. Using a unified master clock time as a reference, the master clock time of the acquisition moment is recorded as the timestamp of the pressure value for each pressure value, and the master clock time of the corresponding image exposure moment is recorded as the timestamp of the position coordinate for each operation end position. Based on the timestamp of the pressure value sequence, each pressure value in the pressure value sequence is traversed, and the position coordinate with the same timestamp is searched in the operation end position coordinate data according to the timestamp of the pressure value. If the position coordinate with the same timestamp is found, the position coordinate is associated with the pressure value and stored. If there is a missing timestamp, the interpolated position coordinates for that time are generated by linear interpolation using the position coordinates of adjacent times. The interpolated position coordinates are then associated with and stored with the pressure value. After all pressure values ​​are searched and interpolated, a set of triplet data records with a one-to-one correspondence between the timestamp, pressure value, and operation end position coordinates are generated.

6. The intelligent linkage control and safety protection system based on electromechanical training equipment according to claim 1, characterized in that, The coupling judgment module extracts the real-time three-dimensional spatial coordinate sequence of the operator's hand joints from the real-time motion capture data and generates the first hand motion trajectory curve. The real-time position coordinates of the operating end are extracted from the real-time haptic feedback signal sequence, and the position coordinates of each sampling time are connected in chronological order to generate the motion trajectory curve of the second device end. Using the same time reference, the position of the first hand movement trajectory curve and the second device end movement trajectory curve are compared frame by frame, and the position deviation value of each frame is calculated to obtain the position deviation sequence.

7. The intelligent linkage control and safety protection system based on electromechanical training equipment according to claim 6, characterized in that, The specific steps for comparing the positional position of the first hand movement trajectory curve with the second device end-effector movement trajectory curve frame by frame using the same time reference, and calculating the positional deviation value for each frame to obtain the positional deviation sequence are as follows: The timestamp of the first valid sampling moment in the real-time motion capture data is obtained as the comparison start time, and the timestamp of the last valid sampling moment in the real-time motion capture data is obtained as the comparison end time. Set a time pointer variable, assign the initial value of the time pointer variable to the timestamp of the start time, and read the three-dimensional spatial coordinates of the hand joints with the same timestamp from the first hand movement trajectory curve based on the timestamp value of the time pointer variable, and use them as the current hand position point. Based on the timestamp value of the time pointer variable, the three-dimensional spatial coordinates of the operation end with the same timestamp are read from the motion trajectory curve of the second device end and used as the current position point of the device end. The spatial straight-line distance between the current hand position point and the current device end position point is calculated and used as the position deviation value at the current moment. The current position deviation value is associated with the current timestamp and stored, and written to the end of the position deviation sequence. Determine if the timestamp value of the time pointer variable is equal to the timestamp of the comparison termination time. If they are equal, complete the comparison processing of all sampling times and output the position deviation sequence. If they are not equal, increase the timestamp value of the time pointer variable by the time length corresponding to one sampling period, generate a new timestamp value and assign it to the time pointer variable, and return to the step of reading the hand position point at the current time to continue execution until the timestamp value of the time pointer variable is equal to the timestamp of the comparison termination time. The maximum position deviation value is extracted from the position deviation sequence and determined as the degree of matching between the operator's intention and the equipment response status.

8. The intelligent linkage control and safety protection system based on electromechanical training equipment according to claim 1, characterized in that, The permission adjustment module extracts the maximum position deviation value from the position deviation sequence, stores the maximum position deviation value in a first register, reads a preset linkage safety threshold from a pre-stored parameter storage area, and stores the preset linkage safety threshold in a second register. A numerical comparator compares the maximum position deviation value stored in the first register with the preset linkage safety threshold stored in the second register bit by bit. The numerical comparator first compares the most significant bit of the two values. If the most significant bits are different, the value with the most significant bit set to 1 is determined to be greater than the value with the most significant bit set to 0, and the comparison result is output. If the most significant bits are the same, the next most significant bit is compared until all significant bits have been compared.

9. The intelligent linkage control and safety protection system based on electromechanical training equipment according to claim 8, characterized in that, Based on the result of the bit-by-bit comparison, the numerical comparator outputs a Boolean state signal. When the maximum position deviation value is greater than the preset linkage safety threshold, the Boolean state signal is at the first logic level, and when the maximum position deviation value is not greater than the preset linkage safety threshold, the Boolean state signal is at the second logic level. When the Boolean state signal output by the numerical comparator is at the first logic level, it is determined that the maximum position deviation value is greater than the preset linkage safety threshold, and a pulse generation circuit is triggered to generate an access restriction trigger signal. A permission restriction trigger signal is sent to an operation permission regulator. In response to the permission restriction trigger signal, the operation permission regulator switches the operator's current control permission from the first control permission to the second control permission. The number of operable parameters corresponding to the second control permission is less than the number of operable parameters corresponding to the first control permission.

Images