Puncture needle force sense perception system and method based on multi-source information

Abstract

The invention relates to the technical field of puncture needle force perception, in particular to a puncture needle force perception system and method based on multi-source information, and the system comprises a miniature camera which is used for collecting and detecting visual information of contact force between a puncture needle and tissue; the F / T sensor is used for collecting and detecting tactile information of contact force between the puncture needle and tissue; the visual perception force sense unit is used for detecting and extracting the contact force between the puncture needle and the tissue from the visual information; the tactile sensing force sense unit is used for detecting and extracting the contact force between the detection puncture needle and the tissue from the tactile information; and the multi-source sensing fusion unit is used for determining the optimal solution of the contact force between the puncture needle and the tissue. According to the invention, multi-source fusion of visual perception and tactile perception is realized to perform multi-modal perception of the contact force, the force perception performance of the puncture needle is improved, the optimal solution of the contact force between the puncture needle and the tissue is determined, and high-precision perception optimization of the contact force between the puncture needle and the tissue is realized.

Description

Technical Field

[0001] The present invention relates to the technical field of puncture needle force perception, and in particular to a puncture needle force perception system and method based on multi-source information. Background Art

[0002] Master-slave laparoscopic minimally invasive surgical robots are widely used. The master operator, serving as the human-machine interface for the laparoscopic minimally invasive surgical robot system, must record the surgeon's hand movements and transmit them to the slave surgical arm. Furthermore, during the surgical procedure, the operator must also be able to sense the contact force between the instrument (puncture needle) and tissue. Currently, this contact force between the needle and tissue is typically detected using sensors.

[0003] In the existing technology, the contact force between the puncture needle and the tissue is only detected by sensors. The force perception is single and the accuracy is limited. It is difficult to improve the perception ability of the contact force between the puncture needle and the tissue by using multi-source information. Correspondingly, it is difficult to achieve high-precision perception optimization of the contact force between the puncture needle and the tissue. Summary of the Invention

[0004] The purpose of the present invention is to provide a puncture needle force perception system and method based on multi-source information to solve the technical problems in the prior art of detecting the contact force between the puncture needle and tissue by sensors, with single force perception and limited accuracy.

[0005] In order to solve the above technical problems, the present invention specifically provides the following technical solutions: A puncture needle force perception system based on multi-source information, comprising: A miniature camera, mounted on the front end of the laparoscope, is used to collect visual information to detect the contact force between the puncture needle and the tissue; The F / T sensor is mounted on the slave arm of the laparoscopic surgical robot and is used to collect tactile information to detect the contact force between the puncture needle and the tissue; a visual perception force unit, configured to detect and extract the contact force between the puncture needle and the tissue from the visual information as a first perceived contact force; a tactile sensing unit, configured to detect and extract the contact force between the puncture needle and the tissue from the tactile information as a second sensed contact force; The multi-source sensing fusion unit is used to perform multi-source fusion of the first sensed contact force and the second sensed contact force to determine the optimal solution of the contact force between the puncture needle and the tissue.

[0006] As a preferred solution of the present invention, the micro camera, visual perception force unit and multi-source perception fusion unit are communicatively connected in sequence, and the F / T sensor, tactile perception force unit and multi-source perception fusion unit are communicatively connected in sequence.

[0007] As a preferred embodiment of the present invention, the tactile information obtained by the F / T sensor includes dynamic information of the puncture needle and contact force information between the puncture needle and the tissue. The expression of the tactile information is: ; Where, is the sensor information, is the dynamic information of the puncture needle, The contact force information between the puncture needle and tissue in the 6-dimensional coordinate system of the F / T sensor; in, , where They are the 6-dimensional components of the dynamic information of the puncture needle in the 6-dimensional coordinate system of the F / T sensor; , where are the 6-dimensional components of the contact force between the puncture needle and the tissue in the 6-dimensional coordinate system of the F / T sensor, and T is the transpose operator.

[0008] As a preferred embodiment of the present invention, the method for detecting and extracting the contact force between the puncture needle and the tissue by the visual perception force unit includes: The coordinate information of the puncture needle is marked in each image of the visual information obtained by the micro camera, and the coordinate position information of the puncture needle is matched with the image time sequence one by one to obtain the motion information of the puncture needle, wherein the motion information of the puncture needle is , where For time series The coordinates of the puncture needle at , N is the total number of images obtained by the micro camera up to the current moment, , Sequential The coordinates of the puncture needle at the 3D component in the 3D coordinate system, T is the transposition operator; The instantaneous speed of the puncture needle is approximately described by the average speed of the puncture needle passing through two adjacent coordinates. According to the motion information of the puncture needle, , the instantaneous speed information of the puncture needle , where , For time series The instantaneous velocity of the puncture needle, For time series The coordinates of the puncture needle at According to the instantaneous speed information of the puncture needle , use the average acceleration method to approximate the instantaneous acceleration information of the puncture needle , where , For time series The instantaneous acceleration of the puncture needle at For time series The instantaneous velocity of the puncture needle, For time series The instantaneous velocity of the puncture needle; The timing The instantaneous acceleration of the puncture needle at is converted into a three-dimensional coordinate system and obtained , where They are 3D components in a 3D coordinate system; The force analysis of the tissue in contact with the puncture needle is performed, and the force on the tissue includes: the contact force between the puncture needle and the tissue , They are The 3D component in the 3D coordinate system, the gravity on the tissue , according to the laws of mechanics, 、 The relationship between them is: , where is the quality of the organization, is the gravitational acceleration of the earth, and its direction is along the 3D coordinate system. Axis negative direction; use as well as Find the tissue in contact with the puncture needle Contact force at any moment ; in, ; Will This is labeled as the first sensed contact force.

[0009] As a preferred embodiment of the present invention, the method for the tactile sensing unit to detect and extract the contact force between the puncture needle and the tissue includes: According to the rigid body dynamics model of the puncture needle established based on the Lagrangian method, the dynamic information of the puncture needle is determined , where is the inertia matrix of the puncture needle, is the Coriolis force matrix of the puncture needle, is the gravity vector of the puncture needle, is the friction force vector of the puncture needle, is the joint angle of the slave robot arm, is the joint angular velocity of the slave manipulator, is the joint angular acceleration of the slave manipulator; use and Solve , and then Converted to a 3D coordinate system, we get ; Among them, the The coordinate system transformation formula is: ; Where, is the contact force between the puncture needle and the tissue in the 3D coordinate system, is the 3D component of the contact force between the puncture needle and the tissue in the 3D coordinate system, l is the length of the puncture needle, d is the distance between the puncture needle and the F / T sensor, T is the transposition operator, For tactile information, is the dynamic information of the puncture needle, is the contact force between the puncture needle and tissue in the 6-dimensional coordinate system of the F / T sensor; Will Labeled as the second sense contact force.

[0010] As a preferred solution of the present invention, the multi-source fusion method of the multi-source perception fusion unit includes: The first sensed contact force and the second sensed contact force are weighted averaged to obtain the optimal solution of the contact force between the puncture needle and the tissue. ; Where, is the optimal solution for the contact force between the puncture needle and the tissue, is the first sense contact force, is the second sensory contact force, for The weight of for The weight of in, and Methods for determining include: The stability of the visual information is quantified by using variance statistics of the N images obtained by the micro camera up to the current moment. , where For the stability of visual information, for and The structural similarity between and are the i-th image and the i+1-th image acquired by the micro camera respectively; The stability of the N tactile information obtained by the F / T sensor up to the current moment is quantified using variance statistics. , where For the stability of tactile information, for and The Euclidean distance between and are the i-th tactile information and the i+1-th tactile information obtained by the F / T sensor respectively; The stability of visual information and the stability of tactile information Perform normalization and obtain and ,in, ; .

[0011] As a preferred solution of the present invention, the N images obtained by the micro camera up to the current moment are normalized, and the N tactile information obtained by the F / T sensor up to the current moment are normalized.

[0012] As a preferred embodiment of the present invention, the present invention provides a puncture needle force perception method based on multi-source information, which is applied to a puncture needle force perception system based on multi-source information, comprising the following steps: A micro camera is used to collect visual information about the contact force between the puncture needle and the tissue; The F / T sensor is used to collect tactile information of the contact force between the puncture needle and the tissue; Detecting and extracting a contact force between the puncture needle and the tissue from the visual information as a first sensed contact force, and detecting and extracting a contact force between the puncture needle and the tissue from the tactile information as a second sensed contact force; The first sensed contact force and the second sensed contact force are fused from multiple sources to determine the optimal solution of the contact force between the puncture needle and the tissue.

[0013] As a preferred solution of the present invention, the multi-source fusion method includes: The first sensed contact force and the second sensed contact force are weighted averaged to obtain the optimal solution of the contact force between the puncture needle and the tissue. ; Where, is the optimal solution for the contact force between the puncture needle and the tissue, is the first sense contact force, is the second sensory contact force, for The weight of for The weight of .

[0014] As a preferred embodiment of the present invention, and Methods for determining include: The stability of the visual information is quantified by using variance statistics of the N images obtained by the micro camera up to the current moment. , where For the stability of visual information, for and The structural similarity between and are the i-th image and the i+1-th image acquired by the micro camera respectively; The stability of the N tactile information obtained by the F / T sensor up to the current moment is quantified using variance statistics. , where For the stability of tactile information, for and The Euclidean distance between and are the i-th tactile information and the i+1-th tactile information obtained by the F / T sensor respectively; The stability of visual information and the stability of tactile information Perform normalization and obtain and ,in, ; .

[0015] Compared with the prior art, the present invention has the following beneficial effects: While using an F / T sensor to detect the contact force between the puncture needle and tissue, the present invention combines the use of a micro camera to detect and extract the contact force between the puncture needle and tissue, thereby realizing multi-source fusion of visual perception and tactile perception for multimodal perception of contact force, improving the performance of the puncture needle's force perception, determining the optimal solution for the contact force between the puncture needle and tissue, and achieving high-precision perception optimization of the contact force between the puncture needle and tissue. BRIEF DESCRIPTION OF THE DRAWINGS

[0016] To more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the following briefly introduces the drawings required for the embodiments or the description of the prior art. Obviously, the drawings described below are merely exemplary, and those skilled in the art can derive other implementation drawings based on the provided drawings without inventive effort.

[0017] Figure 1 A block diagram of a puncture needle force sensing system provided by an embodiment of the present invention; Figure 2 A flow chart of the puncture needle force perception method provided by an embodiment of the present invention; Figure 3 A physical diagram of the laparoscopic surgical robot tactile interaction system provided by an embodiment of the present invention; Figure 4 This is a schematic diagram of the installation of the F / T sensor provided by an embodiment of the present invention. DETAILED DESCRIPTION

[0018] The following will clearly and completely describe the technical solutions in the embodiments of the present invention in conjunction with the accompanying drawings. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without making creative efforts are within the scope of protection of the present invention.

[0019] like Figure 1 、 Figure 3 and Figure 4 As shown, the present invention provides a puncture needle force perception system based on multi-source information, comprising: A miniature camera, mounted on the front end of the laparoscope, is used to collect visual information to detect the contact force between the puncture needle and the tissue; The F / T sensor is mounted on the slave arm of the laparoscopic surgical robot and is used to collect tactile information to detect the contact force between the puncture needle and the tissue; The data processor is divided into three units: visual perception unit, tactile perception unit and multi-source perception fusion unit.

[0020] A visual perception force unit, configured to detect and extract the contact force between the puncture needle and the tissue from the visual information as a first perceived contact force; A tactile sensing unit is used to detect and extract the contact force between the puncture needle and the tissue from the tactile information as a second sensed contact force; The multi-source sensing fusion unit is used to perform multi-source fusion of the first sensed contact force and the second sensed contact force to determine the optimal solution of the contact force between the puncture needle and the tissue.

[0021] The present invention first uses the F / T sensor mounted on the slave robotic arm of the laparoscopic surgical robot to perform tactile detection of the contact force between the puncture needle and the patient's tissue. The tactile information or sensor information obtained by the F / T sensor consists of two parts: the first part is the dynamic information of the puncture needle itself, and the second part is the contact force information between the puncture needle and the patient's tissue. Therefore, the present invention can realize the detection of the contact force between the puncture needle and the patient's tissue through the F / T sensor.

[0022] After the present invention obtains the sensor information by using the F / T sensor, it is necessary to separate the contact force between the puncture needle and the patient's tissue from the sensor information. The data processor determines the dynamic information of the puncture needle according to the rigid body dynamic model of the puncture needle established based on the Lagrangian method. , the dynamic information of the puncture needle was obtained , in sensor information Subtract Correspondingly separated contact force between the puncture needle and the patient's tissue in the 6-dimensional coordinate system of the F / T sensor , and then convert it into a 3D coordinate system to get , that is, the contact force between the puncture needle and the patient's tissue that has practical significance , that is, the force perception of the puncture needle is achieved with the help of the F / T sensor.

[0023] After the force perception of the puncture needle is realized by the F / T sensor, the present invention uses the micro camera on the laparoscope to visually detect the contact force between the puncture needle and the patient's tissue, extracts the trajectory information of the puncture needle from the images continuously captured by the micro camera, and performs kinematic and mechanical analysis on the trajectory information of the puncture needle to solve the contact force between the puncture needle and the tissue. , with the help of a micro camera, force perception of the puncture needle is achieved.

[0024] The present invention obtains the contact force between the puncture needle and the patient's tissue through tactile perception and visual perception to obtain the contact force between the puncture needle and the patient's tissue Afterwards, the contact forces obtained from the two are weighted averaged to determine the optimal solution of the contact force. The weights are quantified by the stability of the tactile information and visual information. The more stable the tactile information or visual information is, the better the contact force is. or The higher the reliability, the higher the weight should be given. Correspondingly, the more unstable the tactile information or visual information is, the more or The lower the reliability, the lower the weight should be given.

[0025] The present invention will and After weighted averaging processing, the fusion of tactile perception and visual perception is achieved in contact force perception, so that contact force perception utilizes multimodal data to make up for the shortcomings of single modality perception, and the perception ability is improved, thereby achieving high-precision perception optimization of the contact force between the puncture needle and the tissue.

[0026] The micro camera, the visual perception force unit and the multi-source perception fusion unit are communicatively connected in sequence, and the F / T sensor, the tactile perception force unit and the multi-source perception fusion unit are communicatively connected in sequence.

[0027] The present invention first uses an F / T sensor mounted on the slave robotic arm of a laparoscopic surgical robot to perform tactile detection of the contact force between the puncture needle and the patient's tissue. The tactile information, or sensor information, obtained by the F / T sensor consists of two parts: the first part is the dynamic information of the puncture needle itself, and the second part is the contact force information between the puncture needle and the patient's tissue. Therefore, the present invention can detect the contact force between the puncture needle and the patient's tissue through the F / T sensor, as follows: The tactile information obtained by the F / T sensor includes the dynamic information of the puncture needle and the contact force information between the puncture needle and the tissue. The expression of the tactile information is: ; Where, is the sensor information, is the dynamic information of the puncture needle, The contact force information between the puncture needle and tissue in the 6-dimensional coordinate system of the F / T sensor; in, , where They are the 6-dimensional components of the dynamic information of the puncture needle in the 6-dimensional coordinate system of the F / T sensor; , where are the 6-dimensional components of the contact force between the puncture needle and the tissue in the 6-dimensional coordinate system of the F / T sensor, and T is the transpose operator.

[0028] The method for detecting and extracting the contact force between the puncture needle and the tissue by the tactile sensing unit includes: According to the rigid body dynamics model of the puncture needle established based on the Lagrangian method, the dynamic information of the puncture needle is determined , where is the inertia matrix of the puncture needle, is the Coriolis force matrix of the puncture needle, is the gravity vector of the puncture needle, is the friction force vector of the puncture needle, is the joint angle of the slave robot arm, is the joint angular velocity of the slave manipulator, is the joint angular acceleration of the slave manipulator; use and Solve , and then Converted to a 3D coordinate system, we get ; in, The coordinate system transformation formula is: ; Where, is the contact force between the puncture needle and the tissue in the 3D coordinate system, is the 3D component of the contact force between the puncture needle and the tissue in the 3D coordinate system, l is the length of the puncture needle, d is the distance between the puncture needle and the F / T sensor, T is the transposition operator, For tactile information, is the dynamic information of the puncture needle, is the contact force between the puncture needle and tissue in the 6-dimensional coordinate system of the F / T sensor; Will Labeled as the second sense contact force.

[0029] The present invention uses a micro camera on a laparoscope to visually detect the contact force between the puncture needle and the patient's tissue, extracts the trajectory information of the puncture needle from the images continuously captured by the micro camera, and performs kinematic and mechanical analysis on the trajectory information of the puncture needle to solve the contact force between the puncture needle and the tissue. , the force perception of the puncture needle is realized with the help of a micro camera, as follows: The method for detecting and extracting the contact force between the puncture needle and the tissue by the visual perception force unit includes: The coordinate information of the puncture needle is marked in each image of the visual information obtained by the micro camera, and the coordinate position information of the puncture needle is matched with the image time sequence one by one to obtain the motion information of the puncture needle, wherein the motion information of the puncture needle is , where For time series The coordinates of the puncture needle at , N is the total number of images obtained by the micro camera up to the current moment, , Sequential The coordinates of the puncture needle at the 3D component in the 3D coordinate system, T is the transposition operator; The instantaneous speed of the puncture needle is approximately described by the average speed of the puncture needle passing through two adjacent coordinates. According to the motion information of the puncture needle, , the instantaneous velocity information of the puncture needle , where , For time series The instantaneous velocity of the puncture needle, For time series The coordinates of the puncture needle at According to the instantaneous speed information of the puncture needle , use the average acceleration method to approximate the instantaneous acceleration information of the puncture needle , where , For time series The instantaneous acceleration of the puncture needle at For time series The instantaneous velocity of the puncture needle, For time series The instantaneous velocity of the puncture needle; ; The timing The instantaneous acceleration of the puncture needle at is converted into a three-dimensional coordinate system and obtained , where They are 3D components in a 3D coordinate system; The force analysis of the tissue in contact with the puncture needle is performed, and the force on the tissue includes: the contact force between the puncture needle and the tissue , They are The 3D component in the 3D coordinate system, the gravity on the tissue , according to the laws of mechanics, 、 The relationship between them is: , where is the quality of the organization, is the gravitational acceleration of the earth, and its direction is along the 3D coordinate system. Axis negative direction; use as well as Find the tissue in contact with the puncture needle Contact force at any moment ; in, ; Will Labeled as the first perceived contact force.

[0030] The multi-source fusion method of the multi-source perception fusion unit includes: The first sensed contact force and the second sensed contact force are weighted averaged to obtain the optimal solution of the contact force between the puncture needle and the tissue. ; Where, is the optimal solution for the contact force between the puncture needle and the tissue, is the first sense contact force, is the second sensory contact force, for The weight of for The weight of in, and Methods for determining include: The stability of visual information is quantified by using variance statistics of the N images obtained by the micro camera up to the current moment. , where For the stability of visual information, for and The structural similarity between and are the i-th image and the i+1-th image acquired by the micro camera respectively; The stability of the tactile information is quantified by using variance statistics of the N tactile information obtained by the F / T sensor up to the current moment. , where For the stability of tactile information, for and The Euclidean distance between and are the i-th tactile information and the i+1-th tactile information obtained by the F / T sensor respectively; The stability of visual information and the stability of tactile information Perform normalization and obtain and ,in, ; .

[0031] The present invention quantifies stability by means of variance statistics. The larger the variance, the more unstable the data and the lower the reliability, so it should be given a lower weight. and By weight distribution, the more reliable the tactile information is, the more stable the correspondence between the visual information is. and The higher the weight.

[0032] Among them, the stability of visual information is quantified by the structural similarity between each image in the visual information, while the stability of tactile information is quantified by the Euclidean distance between each sensor information in the tactile information.

[0033] The present invention obtains the contact force between the puncture needle and the patient's tissue through tactile perception and visual perception to obtain the contact force between the puncture needle and the patient's tissue Afterwards, the contact forces obtained from the two are weighted averaged to determine the optimal solution of the contact force. The weights are quantified by the stability of the tactile information and visual information. The more stable the tactile information or visual information is, the better the contact force is. or The higher the reliability, the higher the weight should be given. Correspondingly, the more unstable the tactile information or visual information is, the more or The lower the reliability, the lower the weight should be given.

[0034] The present invention will and After weighted averaging processing, the fusion of tactile perception and visual perception is achieved in contact force perception, so that contact force perception utilizes multimodal data to make up for the shortcomings of single modality perception, and the perception ability is improved, thereby achieving high-precision perception optimization of the contact force between the puncture needle and the tissue.

[0035] Normalization processing is performed on the N images obtained by the micro camera up to the current moment, and normalization processing is performed on the N tactile information obtained by the F / T sensor up to the current moment.

[0036] like Figure 2 As shown, the present invention provides a puncture needle force perception method based on multi-source information, which is applied to a puncture needle force perception system based on multi-source information, including the following steps: The visual information of the contact force between the puncture needle and the tissue is collected by a micro camera, and the tactile information of the contact force between the puncture needle and the tissue is collected by an F / T sensor; Detecting and extracting the contact force between the puncture needle and the tissue from the visual information as a first sensed contact force, and detecting and extracting the contact force between the puncture needle and the tissue from the tactile information as a second sensed contact force; The first sensed contact force and the second sensed contact force are fused from multiple sources to determine the optimal solution of the contact force between the puncture needle and the tissue.

[0037] Multi-source fusion methods include: The first sensed contact force and the second sensed contact force are weighted averaged to obtain the optimal solution of the contact force between the puncture needle and the tissue. ; Where, is the optimal solution for the contact force between the puncture needle and the tissue, is the first sense contact force, is the second sensory contact force, for The weight of for The weight of .

[0038] and Methods for determining include: The stability of visual information is quantified by using variance statistics of the N images obtained by the micro camera up to the current moment. , where For the stability of visual information, for and The structural similarity between and are the i-th image and the i+1-th image acquired by the micro camera respectively; The stability of the tactile information is quantified by using variance statistics of the N tactile information obtained by the F / T sensor up to the current moment. , where For the stability of tactile information, for and The Euclidean distance between and are the i-th tactile information and the i+1-th tactile information obtained by the F / T sensor respectively; The stability of visual information and the stability of tactile information Perform normalization and obtain and ,in, ; .

[0039] While using an F / T sensor to detect the contact force between the puncture needle and tissue, the present invention combines the use of a micro camera to detect and extract the contact force between the puncture needle and tissue, thereby realizing multi-source fusion of visual perception and tactile perception for multimodal perception of contact force, improving the performance of the puncture needle's force perception, determining the optimal solution for the contact force between the puncture needle and tissue, and achieving high-precision perception optimization of the contact force between the puncture needle and tissue.

[0040] The above embodiments are merely exemplary embodiments of the present application and are not intended to limit the scope of the present application. The scope of protection of the present application is defined by the claims. Those skilled in the art may make various modifications or equivalent substitutions to the present application within the essence and scope of protection of the present application, and such modifications or equivalent substitutions shall also be deemed to fall within the scope of protection of the present application.

Claims

1. A puncture needle force perception system based on multi-source information, characterized in that: include: A miniature camera, mounted on the front end of the laparoscope, is used to collect visual information to detect the contact force between the puncture needle and the tissue; The F / T sensor is mounted on the slave arm of the laparoscopic surgical robot and is used to collect tactile information to detect the contact force between the puncture needle and the tissue; a visual perception force unit, configured to detect and extract the contact force between the puncture needle and the tissue from the visual information as a first perceived contact force; a tactile sensing unit, configured to detect and extract the contact force between the puncture needle and the tissue from the tactile information as a second sensed contact force; The multi-source sensing fusion unit is used to perform multi-source fusion of the first sensed contact force and the second sensed contact force to determine the optimal solution of the contact force between the puncture needle and the tissue.

2. The puncture needle force perception system based on multi-source information according to claim 1, characterized in that: The micro camera, the visual perception force unit and the multi-source perception fusion unit are communicatively connected in sequence, and the F / T sensor, the tactile perception force unit and the multi-source perception fusion unit are communicatively connected in sequence.

3. The puncture needle force perception system based on multi-source information according to claim 1, characterized in that: The tactile information obtained by the F / T sensor includes the dynamic information of the puncture needle and the contact force information between the puncture needle and the tissue. The expression of the tactile information is: t s =t dev +t con ; Where, τ s is the sensor information, τ dev is the dynamic information of the puncture needle, τ con The contact force information between the puncture needle and tissue in the 6-dimensional coordinate system of the F / T sensor; in, Where, They are the 6-dimensional components of the dynamic information of the puncture needle in the 6-dimensional coordinate system of the F / T sensor; Where, are the 6-dimensional components of the contact force between the puncture needle and the tissue in the 6-dimensional coordinate system of the F / T sensor, and T is the transpose operator.

4. The puncture needle force perception system based on multi-source information according to claim 3, characterized in that: The method for detecting and extracting the contact force between the puncture needle and the tissue by the visual perception force unit includes: The coordinate information of the puncture needle is marked in each image of the visual information obtained by the micro camera, and the coordinate position information of the puncture needle is matched with the image time sequence one by one to obtain the motion information of the puncture needle, wherein the motion information of the puncture needle is Where, is the time series t i The coordinates of the puncture needle at , N is the total number of images obtained by the micro camera up to the current moment, The time series t i The coordinates of the puncture needle at the 3D component in the 3D coordinate system, T is the transposition operator; The instantaneous speed of the puncture needle is approximately described by the average speed of the puncture needle passing through two adjacent coordinates. According to the motion information of the puncture needle, The instantaneous speed information of the puncture needle Where, V s (tv) is the instantaneous velocity of the puncture needle at time sequence tv, is the time series t i―1 The coordinates of the puncture needle at According to the instantaneous speed information of the puncture needle The average acceleration method is used to approximate the instantaneous acceleration information of the puncture needle Where, A s (ta) is the instantaneous acceleration of the puncture needle at time ta, For time series The instantaneous velocity of the puncture needle, For time series The instantaneous velocity of the puncture needle; The timing The instantaneous acceleration of the puncture needle at is converted into a three-dimensional coordinate system and is obtained Where a Nx ,a Ny ,a Nz A s 3D components in a 3D coordinate system; The force analysis of the tissue in contact with the puncture needle is performed, and the force on the tissue includes: the contact force between the puncture needle and the tissue F con1 The three-dimensional component in the three-dimensional coordinate system is the gravity G on the tissue. According to the laws of mechanics, the relationship between F and G is: Where m is the mass of the tissue, g is the gravitational acceleration of the earth, and the direction is along the negative Z-axis in the 3D coordinate system; use as well as Calculate the tissue contacting the puncture needle at t N Contact force at any moment in, Will This is labeled as the first sensed contact force.

5. The puncture needle force perception system based on multi-source information according to claim 4, characterized in that: The method for detecting and extracting the contact force between the puncture needle and the tissue by the tactile sensing unit includes: According to the rigid body dynamics model of the puncture needle established based on the Lagrangian method, the dynamic information of the puncture needle is determined Where M s (q) is the inertia matrix of the puncture needle, is the Coriolis force matrix of the puncture needle, G s (q) is the gravity vector of the puncture needle, f s is the friction force vector of the puncture needle, q is the joint angle of the slave end robot arm, is the joint angular velocity of the slave manipulator, is the joint angular acceleration of the slave manipulator; Using τ s =τ dev +τ con and Solve for τ con , and then τ con Converted to a 3D coordinate system, we get Among them, the τ con The coordinate system transformation formula is: Where, F con2 is the contact force between the puncture needle and the tissue in the 3D coordinate system, is the 3D component of the contact force between the puncture needle and the tissue in the 3D coordinate system, l is the length of the puncture needle, d is the distance between the puncture needle and the F / T sensor, T is the transposition operator, τ s is the tactile information, τ dev is the dynamic information of the puncture needle, τ con is the contact force between the puncture needle and tissue in the 6-dimensional coordinate system of the F / T sensor; Will Labeled as the second sense contact force.

6. The puncture needle force perception system based on multi-source information according to claim 5, characterized in that: The multi-source fusion method of the multi-source perception fusion unit includes: The first sensed contact force and the second sensed contact force are weighted averaged to obtain the optimal solution of the contact force between the puncture needle and the tissue. Where, is the optimal solution of the contact force between the puncture needle and the tissue, F con1 is the first sensed contact force, F con2 is the second sensory contact force, w con1 F con1 The weight, w con2 F con2 The weight of Among them, w con1 and w con2 Methods for determining include: The stability of the visual information is quantified by using variance statistics of the N images obtained by the micro camera up to the current moment. Where r con1 is the stability of visual information, ssim(S i ,S i+1 ) is S i and S i+1 The structural similarity between i and S i+1 are the i-th image and the i+1-th image acquired by the micro camera respectively; The stability of the N tactile information obtained by the F / T sensor up to the current moment is quantified using variance statistics. Where r con2 is the stability of tactile information, dis(τ si ,τ si+1 ) is τ si and τ si+1 The Euclidean distance between si and τ si+1 are the i-th tactile information and the i+1-th tactile information obtained by the F / T sensor respectively; The stability of visual information r con1 and the stability of tactile information con2 Perform normalization and get w con1 and w con2 ,in, 7. The puncture needle force perception system based on multi-source information according to claim 6, characterized in that: Normalization processing is performed on the N images obtained by the micro camera up to the current moment, and normalization processing is performed on the N tactile information obtained by the F / T sensor up to the current moment.

8. A puncture needle force perception method based on multi-source information, characterized in that: A puncture needle force perception system based on multi-source information as described in any one of claims 1 to 7 comprises the following steps: A micro camera is used to collect visual information about the contact force between the puncture needle and the tissue; The F / T sensor is used to collect tactile information of the contact force between the puncture needle and the tissue; Detecting and extracting a contact force between the puncture needle and the tissue from the visual information as a first sensed contact force, and detecting and extracting a contact force between the puncture needle and the tissue from the tactile information as a second sensed contact force; The first sensed contact force and the second sensed contact force are fused from multiple sources to determine the optimal solution of the contact force between the puncture needle and the tissue.

9. The method for force perception of a puncture needle based on multi-source information according to claim 8, characterized in that: The multi-source fusion method includes: The first sensed contact force and the second sensed contact force are weighted averaged to obtain the optimal solution of the contact force between the puncture needle and the tissue. Where, is the optimal solution of the contact force between the puncture needle and the tissue, F con1 is the first sensed contact force, F con2 is the second sensory contact force, w con1 F con1 The weight, w con2 F con2 The weight of .

10. The method for force perception of a puncture needle based on multi-source information according to claim 9, characterized in that: The w con1 and w con2 Methods for determining include: The stability of the visual information is quantified by using variance statistics of the N images obtained by the micro camera up to the current moment. Where r con1 is the stability of visual information, ssim(S i ,S i+1 ) is S i and S i+1 The structural similarity between i and S i+1 are the i-th image and the i+1-th image acquired by the micro camera respectively; The stability of the N tactile information obtained by the F / T sensor up to the current moment is quantified using variance statistics. Where r con2 is the stability of tactile information, dis(τ si ,τ si+1 ) is τ si and τ si+1 The Euclidean distance between si and τ si+1 are the i-th tactile information and the i+1-th tactile information obtained by the F / T sensor respectively; The stability of visual information r con1 and the stability of tactile information con2 Perform normalization and get w con1 and w con2 ,in,

Images