Control Method and System for Collecting Soft Tissue Secretions Based on a Robot Arm

Abstract

The present invention provides a control method and system for collecting soft tissue secretions based on a robotic arm. The method includes: obtaining a region to be extracted by using a stereo vision matching algorithm, and obtaining the specific position of the target detection region by using a threshold segmentation algorithm; using flexible tactile sensors distributed in different orientations at the end of the robotic arm to collect contact force vectors; combining a fuzzy control algorithm with a PID force-displacement control to achieve stable contact control of the soft tissue during the process of collecting secretions by moving the end of the robotic arm, and maintaining the stability of the contact pressure during this period. Implementing the embodiments of the present invention combines stereo vision with multi-dimensional touch to control the robotic arm to extract oral soft tissue secretions, improving the safety, reliability, automation, and intelligence of oral soft tissue secretion extraction.

Description

Technical Field

[0001] The present invention relates to the technical field of computer vision and tactile control. Specifically, it relates to a control method and system for collecting soft tissue secretions based on a robotic arm, and particularly to a device and control method for collecting soft tissue secretions by a robotic arm based on visual touch. Background Art

[0002] With the development of robot technology and sensing technology, more and more tasks can be completed by intelligent robots instead of humans. Visual sensors and tactile sensors have important applications in the field of robotics. There is a potential risk of infection in extracting oral soft tissue secretions in medical treatment, and using an intelligent robotic arm instead of humans can improve accuracy and ensure safety.

[0003] Currently, monocular cameras, binocular cameras, depth cameras, etc. are widely used in visual perception systems. Depth cameras measure objects based on infrared structured light to obtain depth values, or calculate depth based on the time difference of reflection. Depth cameras have been widely used in fields such as 3D modeling, driverless, robot navigation, and face recognition. Object detection is one of the most important and challenging branches in the field of computer vision. It has been widely applied in people's lives, such as security monitoring, autonomous driving, etc. The task of object detection is to locate instances of a certain type of semantic object. With the rapid development of deep learning networks for detection tasks, the performance of object detectors has been greatly improved. The research fields of object detection include multi-class detection, edge detection, salient object detection, pose detection, scene text detection, face detection, pedestrian detection, etc. As an important part of scene understanding, object detection is widely used in many fields of modern life, such as the security field, military field, transportation field, medical field, and daily life field.

[0004] In the patent document with the publication number CN113171132A, an intelligent oral extract collection device with a disinfection function is disclosed. Compared with the existing intelligent collection devices, the intelligent collection device includes a fixing mechanism that fits with the lips and is fixed by the collector's biting and further fixed during the collection of oral secretions, a collection module for collecting the liquid secretions of the collector, a testing module for synchronously testing different target diseases of the oral secretions, a scraping module for scraping the corresponding secretions from the throat of the collector, a disinfection module for disinfecting the collection device after the oral extract is collected to cut off the virus transmission route, and a control module that is electrically connected and controls the operation of each electrical component in the collection device.

[0005] Currently, tactile control methods are mainly applied to the static contact control of objects, such as grasping and identifying objects. However, continuous control methods for moving on soft tissues while maintaining stable contact pressure are currently less applied. The basic requirement of modern control theory is an accurate mathematical model of the controlled object. For the touch control of soft tissues, each part of its surface is different and it is impossible to establish an accurate mathematical model, resulting in difficulties in realizing the tactile control of soft tissues by general controls such as PID. The emergence of fuzzy mathematics makes it possible to control the contact of soft tissues. Its fuzzy nature makes the control highly adaptable and has strong anti-interference ability. However, relying solely on fuzzy control cannot achieve stable force-displacement control in the robotic arm. By combining fuzzy control with PID force-displacement control and actively adjusting the PID control parameters using the self-tuning nature of fuzzy control, it can be applied to the contact control of complex objects such as soft tissues and achieve continuous dynamic control of intelligent contact.

[0006] Therefore, a technical solution needs to be proposed to improve the above technical problems. Summary of the Invention

[0007] Aiming at the defects in the prior art, the purpose of the present invention is to provide a control method and system for collecting soft tissue secretions based on a robotic arm.

[0008] According to a control method for collecting soft tissue secretions based on a robotic arm provided by the present invention, the method includes the following steps:

[0009] Step S1: Collect images, set up a light source, connect a computer to a depth camera, and obtain an RGB real-time video stream and a depth video stream.

[0010] Step S2: Extract the face region 1, load the RGB image taken by the camera, convert it into a binary image, use the EigenFace eigenface recognition classifier to transform the face from the pixel space to the feature space for similarity calculation, and use PCA to obtain the principal components of the face distribution.

[0011] Step S3: Apply a template matching algorithm to region 1 to obtain a disposable bite segmentation region 2, and perform threshold segmentation on region 2 to obtain a region 3 for collecting soft tissue secretions.

[0012] Step S4: According to the coordinate transformation matrix between the camera coordinate system and the base coordinate system obtained in hand-eye calibration, calculate the coordinates of the target point to be detected in the base coordinate system and transmit them to the robotic arm.

[0013] Step S5: Perform inverse kinematics solution of the robotic arm according to the coordinates of the target point to be detected, solve the target joint angles of the first three joints, and send the target position data to each servo actuator.

[0014] Step S6: Drive the end effector to move forward along the rotation axis until the value of the pressure sensor changes;

[0015] Step S7: Quantify the differences between the pressures of the three-direction sensors and the experimentally set pressure and the change rates of the differences, fuzzify the input quantities using the triangular membership function, calculate the fuzzy subsets according to the fuzzy rules, and then use the centroid method to solve the three parameters required for PID and input them into the PID controller; Under the action of the PID controller, the system calculates the required displacement adjustment vector;

[0016] Step S8: Decompose the displacement adjustment vector into the rotational motion of the end effector around the rotation axis and the forward and backward motion along the rotation axis, and control the end effector of the robotic arm to move according to the displacement adjustment vector.

[0017] Preferably, the visual recognition algorithm for the acquisition area includes using a face recognition algorithm to obtain the lower half of the human face area 1 that includes the oral cavity, and using template matching within this area to extract the segmentation area 2 of the disposable bite block; Using the threshold segmentation algorithm, obtain the area 3 of the soft tissue secretion to be collected;

[0018] The tactile control algorithm includes using a fuzzy PID control algorithm to intelligently self-adjust the PID control parameters and calculate a primary fine-tuning displacement amount.

[0019] Preferably, the specific process of the face recognition algorithm in step S2 for extracting area 1 includes the following steps:

[0020] Step S2.1: Load the RGB image obtained by the camera shooting;

[0021] Step S2.2: Convert the RGB image into a binary image;

[0022] Step S2.3: Use the EigenFace eigenface recognition classifier to extract the lower half of the face area as area 1.

[0023] Preferably, the specific process of the template matching algorithm in step S3 for extracting area 2 includes the following steps:

[0024] Step S3.1: Pre-take the cropped disposable bite block template image at the same angular pose;

[0025] Step S3.2: Compare the disposable bite block template image with the image to be recognized;

[0026] Step S3.3: Compare the two images pixel by pixel, and select a corresponding window as the matching result according to the difference or similarity to obtain area 2.

[0027] Preferably, the specific process of the threshold segmentation algorithm in step S3 for extracting area 3 includes:

[0028] Perform threshold segmentation on Region 2. According to the depth information and RGB information, extract the oral mucosa region inside the disposable bite, and output the coordinates of its center point to obtain Region 3.

[0029] Preferably, after obtaining the coordinates of the center point of the target detection region by the visual recognition algorithm of the acquisition region, use the robot motion control method to calculate the joint angles required for the secretion collection head clamped by the end effector of the robotic arm to reach this coordinate, and drive the joints of the robotic arm to move to the target angles; the motion control method includes obtaining the inverse kinematics solution of the robotic arm and the joint position control closed loop.

[0030] Preferably, the tactile control algorithm starts to execute after the secretion collection head is guided by visual control to contact the soft tissue. The tactile control algorithm keeps the pressure sensor value within a certain range. After the secretion collection head clamped by the end effector of the robotic arm reciprocally contacts the soft tissue to scrape the secretion, it stops contacting and returns.

[0031] Preferably, the tactile control algorithm cyclically executes the steps of optimizing the pressure vector, calculating the displacement adjustment vector, and fine-tuning the position of the end effector of the robotic arm during the entire process of collecting the secretion;

[0032] The step of optimizing the pressure vector is to real-time input the values of the acquisition pressure sensors in three directions, perform weighted fusion on the pressure values in the three directions to obtain the final real-time pressure vector;

[0033] The step of calculating the displacement adjustment vector inputs the difference between the desired pressure and the actual pressure and its change rate into the fuzzy controller; uses the triangular membership function to fuzzify the input quantity, calculates the fuzzy subset according to the fuzzy rules, and then uses the centroid method to solve the three parameters required by PID and input them into the PID controller; under the action of the PID controller of the system, calculate the required displacement adjustment vector;

[0034] The step of fine-tuning the position of the end effector of the robotic arm decomposes the displacement adjustment vector into the rotational motion of the end effector around the rotation axis and the forward and backward motion along the rotation axis, and controls the end effector of the robotic arm to move according to the displacement adjustment vector.

[0035] The present invention also provides a control system for collecting the secretion of soft tissue based on a robotic arm. The system includes the following modules:

[0036] Module M1: Collect images, set up a light source, connect a computer to a depth camera, and obtain the RGB real-time video stream and the depth video stream;

[0037] Module M2: Extract the face region 1, load the RGB image captured by the camera, convert it into a binary image, use the EigenFace eigenface recognition classifier to transform the face from the pixel space to the feature space for similarity calculation, and use PCA to obtain the principal components of the face distribution;

[0038] Module M3: Apply the template matching algorithm on region 1 to obtain the disposable bite block segmentation region 2, and perform threshold segmentation on region 2 to obtain the region 3 for collecting soft tissue secretions;

[0039] Module M4: According to the coordinate transformation matrix between the camera coordinate system and the base coordinate system obtained in the hand-eye calibration, calculate the coordinates of the target point to be detected in the base coordinate system and transmit them to the robotic arm;

[0040] Module M5: Perform inverse kinematics solution of the robotic arm according to the coordinates of the target point to be detected, solve the target joint angles of the first three joints, and send the target position data to each servo actuator;

[0041] Module M6: Drive the end effector to move forward along the rotation axis direction until the value of the pressure sensor changes;

[0042] Module M7: Quantify the pressure difference between the three-direction sensor pressure and the experimentally set pressure and the rate of change of the difference, use the triangular membership function to fuzzify the input quantity, calculate the fuzzy subset according to the fuzzy rules, and then use the centroid method to solve the three parameters required by PID and input them into the PID controller; Under the action of the PID controller, the system calculates the required displacement adjustment vector;

[0043] Module M8: Decompose the displacement adjustment vector into the rotational motion of the end effector around the rotation axis and the forward and backward motion along the rotation axis direction, and control the end effector of the robotic arm to move according to the displacement adjustment vector.

[0044] Preferably, the visual recognition algorithm for the acquisition area includes using the face recognition algorithm to obtain the lower half of the human face including the oral cavity as region 1, and using template matching within this region to extract the segmentation region 2 of the disposable bite block; Using the threshold segmentation algorithm, obtain the region 3 for collecting soft tissue secretions;

[0045] The tactile control algorithm includes using the fuzzy PID control algorithm to intelligently self-adjust the PID control parameters and calculate a primary fine-tuning displacement amount;

[0046] The specific process of the face recognition algorithm in module M2 for extracting region 1 includes the following modules:

[0047] Module M2.1: Load the RGB image captured by the camera;

[0048] Module M2.2: Convert the RGB image into a binary image;

[0049] Module M2.3: Use the EigenFace eigenface recognition classifier to extract the lower half of the face area as Region 1;

[0050] The specific process of extracting Region 2 by the template matching algorithm in Module M3 includes the following modules:

[0051] Module M3.1: Pre-take a cut one-time bite template image at the same angular pose in advance;

[0052] Module M3.2: Compare the one-time bite template image with the image to be recognized;

[0053] Module M3.3: Compare the two images pixel by pixel, and select a corresponding window as the matching result according to the difference or similarity to obtain Region 2;

[0054] The specific process of extracting Region 3 by the threshold segmentation algorithm in Module M3 includes:

[0055] Perform threshold segmentation on Region 2, and according to the depth information and RGB information, extract the oral mucosa area inside the one-time bite, and output its center point coordinates to obtain Region 3.

[0056] Compared with the prior art, the present invention has the following beneficial effects:

[0057] 1. The present invention adopts a control method combining visual recognition and tactile perception. First, determine the coordinates of the area to be sampled through visual recognition, so that the forward direction of the end of the robotic arm is aligned with the area to be sampled. Then, during the process of the end of the robotic arm extending forward, rely on the end force sensing function to control the contact force during the soft body contact process, making the automatic sampling process standard and gentle;

[0058] 2. The present invention scores and compares various template matching methods, and adopts the method of non-maximum suppression to select the best matching effect, accurately identify the one-time bite area, and guide the robotic arm to adjust the angle to adapt to the collection of oral secretion samples of people with different heights;

[0059] 3. The present invention uses a threshold segmentation algorithm to further extract the tissue area to be sampled from the matching area, provides an accurate sampling position, and guides the robotic arm to reach the target position, so that the sampling system can adjust itself according to the tissue differences of the subjects, with flexible adaptability;

[0060] 4. The present invention adopts a four-degree-of-freedom robotic arm, which can decompose the sampling process into two actions: advancing along the axis and rotating along the axis, which can reduce the control difficulty and improve the solution speed of the contact force control algorithm;

[0061] 5. The present invention can quickly achieve stable control of contact by combining the fuzzy control algorithm with PID, reduce the number of adjustments, has strong anti-interference ability, and can achieve intelligent self-adjustment, being applicable to the complex variable contact of soft bodies. BRIEF DESCRIPTION OF THE DRAWINGS

[0062] Other features, objects, and advantages of the present invention will become more apparent by reading the following detailed description of non-limiting embodiments with reference to the accompanying drawings:

[0063] Figure 1 Schematic diagram of the robotic arm and sensor configuration of the present invention;

[0064] Figure 2 Flowchart of the algorithm of the present invention;

[0065] Figure 3 Schematic diagram of the vision recognition area of the present invention;

[0066] Figure 4 Flowchart of the fuzzy PID parameter calculation of the present invention;

[0067] Figure 5 Block diagram of the fuzzy PID force-displacement control system of the present invention.

[0068] Wherein,

[0069] Four-degree-of-freedom robotic arm 1 Region 1: Face region End effector 2 Region 2: Single-use bite region Depth camera 3 Region 3: Oral posterior wall region Light source 4 DETAILED DESCRIPTION OF THE EMBODIMENTS

[0070] The present invention will be described in detail below in conjunction with specific embodiments. The following embodiments will help those skilled in the art to further understand the present invention, but do not limit the present invention in any form. It should be noted that those of ordinary skill in the art can make several changes and improvements without departing from the concept of the present invention. These all belong to the protection scope of the present invention.

[0071] A device for collecting soft tissue secretions based on a robotic arm includes a four-degree-of-freedom robotic arm 1; the four-degree-of-freedom robotic arm 1 includes the first three degrees of freedom for adjusting the center coordinates of the end effector 2 and the orientation of the rotation axis of the end effector 2, and the last degree of freedom for the end effector 2 to rotate around the rotation axis; three-direction pressure sensors are arranged inside the end effector 2; a depth camera 3 and a light source 4 are mounted on a link of the robotic arm and always face the rotation axis of the end effector 2. The pressure sensor is a flexible film pressure sensor; the sampling frequency of the pressure sensor is 10Hz - 100Hz.

[0072] The embodiments of the present invention provide a device and a control method for collecting soft tissue secretions based on a robotic arm. The robotic arm and the sensor of the method are configured as follows Figure 1 shown, including a depth camera, flexible tactile pressure sensors in three directions, and a detected target. The specific implementation steps are as follows Figure 2 shown and include:

[0073] Step 1: Image acquisition. Set up a light source, connect a computer to the depth camera, and obtain an RGB real-time video stream and a depth video stream for subsequent target detection.

[0074] Step 2: As shown in Figure 3 shown, extract the face area 1. Load the RGB image captured by the camera, convert it into a binary image, use the EigenFace eigenface recognition classifier to transform the face from the pixel space to the feature space for similarity calculation, and use PCA to obtain the principal components of the face distribution. Specifically, it is implemented by performing eigenvalue decomposition on the covariance matrix of all face images in the training set to obtain eigenvectors representing the features of the face. Project the image to be detected onto the eigenvectors for classification, and the one closest to the training data vector will be recognized as a face. Extract the lower half of the face area as area 1, and obtain the pixel coordinates of the upper left corner point and the lower right corner point of this area.

[0075] Step 3: As shown in Figure 3 shown, extract the disposable bite area 2. Pre-take the actual detection images in the same angular pose and crop out the disposable bite as a template. Use six template matching algorithms in the image of area 1, namely sum of squared differences matching, normalized sum of squared differences matching, correlation matching, normalized correlation matching, correlation coefficient matching, and normalized correlation coefficient matching. The symbols included in the formula of template matching are explained as follows: T(x, y), I(x, y), and R(x, y) are all functions, representing the pixel value at (x, y) in the template image, the pixel value at (x, y) in the image to be detected, and the element value at (x, y) in the result matrix respectively; w is the width of the template image, and h is the height of the template image, both represented by the number of pixels. For each pixel (x, y) in the image to be detected except for the edge area, it corresponds to the element (x, y) in the result matrix, and the value of this element represents the matching degree between a small area in the lower right of the pixel (x, y) in the image to be detected and the template image. The element values in the result matrix can be calculated according to the following six template matching formulas:

[0076] Sum of squared differences matching:

[0077]

[0078] Normalized sum of squared differences matching:

[0079]

[0080] Related matching:

[0081]

[0082] Standard related matching:

[0083]

[0084] Correlation coefficient matching:

[0085]

[0086] Wherein:

[0087]

[0088]

[0089] Standard correlation coefficient matching:

[0090]

[0091] Wherein:

[0092]

[0093]

[0094] Each method will obtain a score, and the method of maximum value suppression is used to obtain the best matching result with the best final effect. The template matching obtains the coordinates of the upper left corner point and the lower right corner point of the smallest circumscribed rectangle area 2 of the one-time bite relative to area 1.

[0095] Step 4: As Figure 3 shown, perform threshold segmentation on area 2 to obtain area 3 where the soft tissue secretion to be collected is located. The one-time bite is milky white with a pixel value close to 255, and the area to be detected inside the oral cavity appears dark red under the illumination of the light source. The average gray difference between the two is relatively obvious. At the same time, according to the depth information and RGB information, the oral mucosa area inside the one-time bite can be extracted, and all the pixels outside the area to be detected are set to 0. Finally, after transformation, the pixel coordinates of the points to be detected in the original image coordinate system are obtained.

[0096] Step 5: According to the coordinate transformation matrix between the camera coordinate system and the base coordinate system obtained in the hand-eye calibration, calculate the coordinates of the target point to be detected in the base coordinate system and transmit them to the robotic arm.

[0097] After obtaining the coordinates of the center point of the target detection area by the collection area visual recognition algorithm, the robot motion control method needs to be used to calculate the joint angles required for the secretion collection head held by the end effector of the robot arm to reach the coordinates, and drive the joints of the robot arm to move to the target angles. The motion control method includes obtaining the inverse solution of the robot arm kinematics and the joint position control closed loop.

[0098] The steps of obtaining the inverse solution of the robot kinematics include: first, locking the last degree of freedom of the robot, that is, making the end effector of the robot rotate to a fixed angle around its rotation axis, at this time, the 4*4 pose matrix of the secretion collection head held by the end effector in the robot base coordinate system can be obtained, and the pose matrix only contains the angle variables of the first three degrees of freedom. The first three rows of the fourth column of the pose matrix are the center coordinate positions of the collection head, which are equal to the center coordinates of the target detection area, and the solution equations for determining the joint angles of the first three joints of the robot can be obtained. Then, combined with the angle range limit inequality, a unique set of target angles of each joint of the robot is finally obtained.

[0099] The joint position control closed loop is realized by a servo actuator with an integrated controller. After the target angle data is input into the servo actuator controller through the communication line, the position feedback closed loop inside the controller controls the output current so that the actual angle of the servo actuator is consistent with the target angle.

[0100] Step 6: Perform inverse kinematics solution of the robot arm according to the coordinates of the target point to be detected, solve the target joint angles of the first three joints, and send the target position data to each servo motor to make the center of the acquisition head clamped by the end effector move to the specified coordinate point.

[0101] Step 7: Drive the end effector to move forward slowly along its rotation axis until the pressure sensor value changes, indicating that the acquisition head has come into contact with the soft tissue. At this time, the tactile control cycle begins to work.

[0102] Step 8: Figure 4 As shown, the difference between the sensor pressure in three directions and the experimental set pressure and the rate of change of the difference are quantified, the input quantity is fuzzified using the triangular membership function, the fuzzy subset is calculated according to the fuzzy rule, and then the three parameters required for PID are solved using the center of gravity method and input into the PID controller; under the action of the PID controller, the system calculates the required displacement adjustment vector.

[0103] The tactile control algorithm includes the use of fuzzy PID control algorithm to intelligently self-adjust PID control parameters and calculate a fine-tuning displacement. After feedback iteration, the pressure at the contact position can be quickly stabilized, and the pressure can also be quickly stabilized at the discrete points reached. That is, the discrete stability of the contact pressure is maintained from the starting point to the end point of scraping secretions.

[0104] The tactile control algorithm starts to execute after the secretion collection head is guided by visual control to contact the soft tissue. The tactile control algorithm keeps the pressure sensor value within a certain range. After the secretion collection head clamped by the end effector of the robotic arm reciprocally contacts the soft tissue to scrape the secretion, it stops contacting and returns.

[0105] Among them, the triangular membership function of the fuzzification formula is a function used to evaluate the membership degree of the actual input quantity x and the input quantity level in the fuzzy control rule table, and the value range of the function is [0,1]. The fuzzification process is the process of attributing the actual input quantity to the input quantity level. The triangular membership function can be designed in the following form:

[0106]

[0107] Among them, a and c are respectively the lower limit and upper limit of the actual input quantity belonging to this input quantity level, and b is the intermediate value of the actual input quantity belonging to this input quantity level, which is selected according to experimental experience.

[0108] Among them, to determine the fuzzy control rules, it is necessary to determine them separately according to the different functions of Kp, Ki, and Kd in PID control. Taking the 7*7 matrix control rule as an example, among them, the first row is the error level, the first column is the error change rate level, and the values in the table are the output quantity levels. The larger the value, the stronger the effect: Kp controls the system response speed. In the early stage, Kp should be appropriately large, and in the later stage, for small-range correction, Kp should be reduced to ensure a small overshoot and a certain response speed; Ki is integral control, with a weak effect in the initial stage to prevent overshoot, moderate in the middle stage, and strong in the later stage to reduce the static error; Kd is differential control, large in the initial stage to avoid overshoot, appropriately smaller in the middle stage, and reduced in the later stage to reduce the adjustment time.

[0109]

[0110]

[0111]

[0112] Among them, the defuzzification process is the process of converting the output quantity membership degree curve obtained from the fuzzy control rule table into the actual output quantity. The defuzzification formula of the centroid method is as follows:

[0113]

[0114] Among them, n is the number of output quantization levels, M i represents the membership degree corresponding to the i-th output quantity level, and F i represents the value of the i-th output quantity level.

[0115] Step 9: Decompose the displacement adjustment vector into the rotational motion of the end effector around the rotation axis and the forward and backward motion along the rotation axis, and then control the end effector of the robotic arm to move according to the displacement adjustment vector.

[0116] As Figure 5 shown, Steps 7 and 8 are cyclically executed during the entire process of collecting secretions to achieve the effect of keeping the actual pressure within the desired range.

[0117] The present invention also provides a control system for collecting soft tissue secretions based on a robotic arm. The system includes the following modules: Module M1: Collect images, set up a light source, connect a computer to a depth camera, and obtain an RGB real-time video stream and a depth video stream; Module M2: Extract face region 1, load the RGB image captured by the camera, convert it into a binary image, use the EigenFace eigenface recognition classifier to transform the face from the pixel space to the feature space for similarity calculation, and use PCA to obtain the principal components of the face distribution; Module M2.1: Load the RGB image captured by the camera; Module M2.2: Convert the RGB image into a binary image; Module M2.3: Use the EigenFace eigenface recognition classifier to extract the lower half of the face region as region 1.

[0118] Module M3: Apply a template matching algorithm to region 1 to obtain a disposable bite block segmentation region 2, perform threshold segmentation on region 2 to obtain a region 3 for collecting soft tissue secretions; Module M3.1: Pre-capture a trimmed disposable bite block template image at the same angular pose; Module M3.2: Compare the disposable bite block template image with the image to be recognized; Module M3.3: Compare the two images pixel by pixel, and select a corresponding window as the matching result according to the difference or similarity to obtain region 2; perform threshold segmentation on region 2, extract the oral mucosa region inside the disposable bite block according to the depth information and RGB information, and output its center point coordinates to obtain region 3.

[0119] Module M4: Calculate the coordinates of the target point to be detected in the base coordinate system according to the coordinate transformation matrix between the camera coordinate system and the base coordinate system obtained in hand-eye calibration, and transmit them to the robotic arm; Module M5: Perform inverse kinematics of the robotic arm based on the coordinates of the target point to be detected, solve the target joint angles of the first three joints, and send the target position data to each servo; Module M6: Drive the end effector to move forward along the rotation axis direction until the value of the pressure sensor changes; Module M7: Quantify the pressure difference between the three-direction sensor pressure and the experimentally set pressure and the rate of change of the difference, fuzzify the input quantity using the triangular membership function, calculate the fuzzy subset according to the fuzzy rules, and then use the centroid method to solve the three parameters required for PID and input them into the PID controller; Under the action of the PID controller, the system calculates the required displacement adjustment vector; Module M8: Decompose the displacement adjustment vector into the rotational motion of the end effector around the rotation axis and the forward and backward motion along the rotation axis direction, and control the end effector of the robotic arm to move according to the displacement adjustment vector.

[0120] The visual recognition algorithm for the acquisition area includes using the face recognition algorithm to obtain the lower half of the human face including the oral cavity area 1, and using template matching to extract the segmented area 2 of the disposable bite block within this area; Using the threshold segmentation algorithm to obtain the area 3 of the soft tissue secretion to be collected; The tactile control algorithm includes using the fuzzy PID control algorithm to intelligently self-adjust the PID control parameters and calculate a primary fine-tuning displacement amount.

[0121] The present invention scores and compares various template matching methods, and uses the method of non-maximum suppression to select the best matching effect, accurately identify the disposable bite block area, and guide the robotic arm to adjust the angle to adapt to the collection of oral secretion samples of people of different heights; The combination of the fuzzy control algorithm and PID can quickly achieve stable contact control, reduce the number of adjustments, has strong anti-interference ability, and can achieve intelligent self-adjustment, and is suitable for complex variable contact of soft bodies.

[0122] Those skilled in the art know that in addition to implementing the system and its various devices, modules, and units provided by the present invention in the form of pure computer-readable program code, the method steps can be logically programmed to enable the system and its various devices, modules, and units provided by the present invention to be implemented in the form of logic gates, switches, application-specific integrated circuits, programmable logic controllers, and embedded microcontrollers, etc. to achieve the same function. Therefore, the system and its various devices, modules, and units provided by the present invention can be considered as a hardware component, and the devices, modules, and units included therein for implementing various functions can also be regarded as the structure within the hardware component; The devices, modules, and units for implementing various functions can also be regarded as either software modules for implementing the method or the structure within the hardware component.

[0123] The specific embodiments of the present invention have been described above. It should be understood that the present invention is not limited to the above specific embodiments, and those skilled in the art can make various changes or modifications within the scope of the claims, which do not affect the essence of the present invention. Without conflict, the embodiments of the present application and the features in the embodiments can be arbitrarily combined with each other.

Claims

1. A control method for collecting soft tissue secretions based on a robotic arm, characterized in that, the method comprises the following steps: Step S1: Collect images, set up a light source, connect a computer to a depth camera, and obtain an RGB real-time video stream and a depth video stream; Step S2: Extract face region 1, load the RGB image captured by the camera, convert it into a binary image, use the EigenFace eigenface recognition classifier to transform the face from the pixel space to the feature space for similarity calculation, and use PCA to obtain the principal components of the face distribution; Step S3: Apply a template matching algorithm to region 1 to obtain a disposable bite segmentation region 2, perform threshold segmentation on region 2 to obtain a region 3 for collecting soft tissue secretions to be collected; Step S4: According to the coordinate transformation matrix between the camera coordinate system and the base coordinate system obtained in hand-eye calibration, calculate the coordinates of the target point to be detected in the base coordinate system and transmit them to the robotic arm; Step S5: Perform inverse kinematics solution of the robotic arm according to the coordinates of the target point to be detected, solve the target joint angles of the first three joints, and send the target position data to each servo actuator; Step S6: Drive the end effector to advance along the rotation axis direction until the value of the pressure sensor changes; Step S7: Quantify the pressure difference between the three-direction sensor pressure and the experimentally set pressure and the rate of change of the difference, use the triangular membership function to fuzzify the input quantity, calculate the fuzzy subset according to the fuzzy rules, and then use the centroid method to solve the three parameters required by PID and input them into the PID controller; Under the action of the PID controller, the system calculates the required displacement adjustment vector; Step S8: Decompose the displacement adjustment vector into the rotational motion of the end effector around the rotation axis and the forward and backward motion along the rotation axis direction, and control the end effector of the robotic arm to move according to the displacement adjustment vector; The tactile control algorithm loops through the steps of optimizing the pressure vector, calculating the displacement adjustment vector, and fine-tuning the position of the end effector of the robotic arm during the entire process of collecting secretions; The step of optimizing the pressure vector is to continuously input the numerical values of the three-direction collection pressure sensors in real time, perform weighted fusion on the three-direction pressure values to obtain the final real-time pressure vector; The step of calculating the displacement adjustment vector inputs the difference between the desired pressure and the actual pressure and its rate of change into the fuzzy controller; Use the triangular membership function to fuzzify the input quantity, calculate the fuzzy subset according to the fuzzy rules, and then use the centroid method to solve the three parameters required by PID and input them into the PID controller; Under the action of the PID controller, the system calculates the required displacement adjustment vector; The step of fine-tuning the position of the end effector of the robotic arm decomposes the displacement adjustment vector into the rotational motion of the end effector around the rotation axis and the forward and backward motion along the rotation axis direction, and controls the end effector of the robotic arm to move according to the displacement adjustment vector.

2. The control method for collecting soft tissue secretions based on a robotic arm according to claim 1, characterized in that, The visual recognition algorithm for the acquisition area includes using a face recognition algorithm to obtain the lower half of the human face area 1 that includes the oral cavity, and using template matching to extract the segmentation area 2 of the disposable bite block within this area; using a threshold segmentation algorithm to obtain the area 3 of the soft tissue secretion to be collected; The tactile control algorithm includes using a fuzzy PID control algorithm to intelligently self-adjust the PID control parameters and calculate a primary fine-tuning displacement.

3. The control method for collecting soft tissue secretions based on a robotic arm according to claim 1, wherein, The specific process of the face recognition algorithm for extracting area 1 in step S2 includes the following steps: Step S2.1: Load the RGB image captured by the camera; Step S2.2: Convert the RGB image into a binary image; Step S2.3: Use the EigenFace eigenface recognition classifier to extract the lower half of the face area as area 1.

4. The control method for collecting soft tissue secretions based on a robotic arm according to claim 1, wherein, The specific process of the template matching algorithm for extracting area 2 in step S3 includes the following steps: Step S3.1: Pre-capture a cropped disposable bite block template image at the same angular pose in advance; Step S3.2: Compare the disposable bite block template image with the image to be recognized; Step S3.3: Compare the two images pixel by pixel, and select a corresponding window as the matching result according to the difference or similarity to obtain area 2.

5. The control method for collecting soft tissue secretions based on a robotic arm according to claim 1, wherein, The specific process of the threshold segmentation algorithm for extracting area 3 in step S3 includes: Perform threshold segmentation on area 2, and according to the depth information and RGB information, extract the oral mucosa area inside the disposable bite block, and output its center point coordinates to obtain area 3.

6. The control method for collecting soft tissue secretions based on a robotic arm according to claim 1, wherein, After obtaining the center point coordinates of the target detection area by the visual recognition algorithm for the acquisition area, use the robot motion control method to calculate the joint angles required for the secretion collection head held by the end effector of the robotic arm to reach this coordinate, and drive the joints of the robotic arm to move to the target angles; the motion control method includes obtaining the inverse kinematics of the robotic arm and the joint position control closed loop.

7. The control method for collecting soft tissue secretions based on a robotic arm according to claim 1, wherein, The tactile control algorithm starts to execute after the secretion collection head is guided by visual control to contact the soft tissue. The tactile control algorithm keeps the pressure sensor value within a certain range. After the secretion collection head held by the end effector of the robotic arm reciprocally contacts the soft tissue to scrape the secretion, it stops contacting and returns.

8. A control system for collecting soft tissue secretions based on a robotic arm, wherein, The system includes the following modules: Module M1: Collect images, build a light source, connect a computer to a depth camera, and obtain an RGB real-time video stream and a depth video stream; Module M2: Extract the face region 1, load the RGB image captured by the camera, convert it into a binary image, use the EigenFace eigenface recognition classifier to transform the face from the pixel space to the feature space for similarity calculation, and use PCA to obtain the principal components of the face distribution; Module M3: Apply the template matching algorithm on region 1 to obtain the disposable bite block segmentation region 2, and perform threshold segmentation on region 2 to obtain the region 3 for collecting soft tissue secretions; Module M4: According to the coordinate transformation matrix between the camera coordinate system and the base coordinate system obtained in the hand-eye calibration, calculate the coordinates of the target point to be detected in the base coordinate system and transmit them to the robotic arm; Module M5: Perform inverse kinematics solution of the robotic arm according to the coordinates of the target point to be detected, solve the target joint angles of the first three joints, and send the target position data to each servo; Module M6: Drive the end effector to move forward along the axis of rotation until the value of the pressure sensor changes; Module M7: Quantify the pressure difference between the three-direction sensor pressure and the experimentally set pressure and the rate of change of the difference, use the triangular membership function to fuzzify the input quantity, calculate the fuzzy subset according to the fuzzy rules, and then use the centroid method to calculate the three parameters required for PID and input them into the PID controller; Under the action of the PID controller, the system calculates the required displacement adjustment vector; Module M8: Decompose the displacement adjustment vector into the rotational motion of the end effector around the axis of rotation and the forward and backward motion along the axis of rotation, and control the end effector of the robotic arm to move according to the displacement adjustment vector; The tactile control algorithm cyclically executes the steps of optimizing the pressure vector, calculating the displacement adjustment vector, and fine-tuning the position of the end effector of the robotic arm during the entire process of collecting secretions; The step of optimizing the pressure vector is to real-time input the numerical values of the three-direction acquisition pressure sensors, perform weighted fusion on the three-direction pressure values, and obtain the final real-time pressure vector; The step of calculating the displacement adjustment vector inputs the difference between the desired pressure and the actual pressure and its rate of change into the fuzzy controller; Use the triangular membership function to fuzzify the input quantity, calculate the fuzzy subset according to the fuzzy rules, and then use the centroid method to calculate the three parameters required for PID and input them into the PID controller; Under the action of the PID controller, the system calculates the required displacement adjustment vector; The step of fine-tuning the position of the end effector of the robotic arm decomposes the displacement adjustment vector into the rotational motion of the end effector around the axis of rotation and the forward and backward motion along the axis of rotation, and controls the end effector of the robotic arm to move according to the displacement adjustment vector.

9. The control system for collecting soft tissue secretions based on a robotic arm according to claim 8, characterized in that The visual recognition algorithm for the acquisition area includes using a face recognition algorithm to obtain the lower half of the human face containing the oral cavity as region 1, and using template matching to extract the segmentation region 2 of the disposable bite block within this region; using a threshold segmentation algorithm to obtain the region 3 for collecting soft tissue secretions; The tactile control algorithm includes using a fuzzy PID control algorithm to intelligently self-adjust the PID control parameters and calculate a primary fine-tuning displacement amount; The specific process of extracting Region 1 by the face recognition algorithm in Module M2 includes the following modules: Module M2.1: Load the RGB image captured by the camera; Module M2.2: Convert the RGB image into a binary image; Module M2.3: Use the EigenFace eigenface recognition classifier to extract the lower half of the face region as Region 1; The specific process of extracting Region 2 by the template matching algorithm in Module M3 includes the following modules: Module M3.1: Pre-capture the cropped disposable bite template image at the same angular pose; Module M3.2: Compare the disposable bite template image with the image to be recognized; Module M3.3: Compare the two images pixel by pixel, and select a corresponding window as the matching result according to the difference or similarity to obtain Region 2; The specific process of extracting Region 3 by the threshold segmentation algorithm in Module M3 includes: Perform threshold segmentation on Region 2, extract the oral mucosa region inside the disposable bite according to the depth information and RGB information, and output its center point coordinates to obtain Region 3.

Images