A fully automatic venipuncture recognition robot

Abstract

The present invention relates to a fully automatic venous puncture identification integrated robot, comprising a puncture module, an image acquisition module and a positioning platform; the puncture module comprises a puncture module housing, a y-axis rotation unit, an x-axis rotation unit, a z-axis rotation unit and a puncture head component; the y-axis rotation unit comprises a y-axis motor and a y-axis bearing; the x-axis rotation unit comprises an x-axis rotating shaft and an x-axis housing; the z-axis rotation unit comprises a z-axis motor, a z-axis rotating shaft, a z-axis bearing, a worm and a worm gear; the image acquisition module comprises an ultrasonic probe, an image linear motor and a near-infrared camera, and the ultrasonic probe is fixed on the front active end of the image linear motor through a probe bracket to perform up and down linear motion. Compared with the prior art, the present invention greatly reduces the overall volume of the device through the three-dimensional staggered design of each module; realizes precise control of the puncture head component; and realizes fully automated precise control.

Description

Technical Field

[0001] The invention relates to the field of venipuncture robots, and in particular to a fully automatic venipuncture recognition integrated robot. Background Art

[0002] Manual venipuncture by medical staff has low accuracy and is prone to cross-infection between doctors and patients. Automatic venipuncture robots are an emerging direction in smart healthcare due to their high degree of automation and accurate puncture. However, existing venipuncture robots have problems such as low control accuracy, large size, and single mode.

[0003] Chinese patent CN 109960285 A discloses an automatic blood sampling robot, which collects blood through the cooperation of different modules. However, the overall size of the device is large and the mode is single, which makes it difficult to adapt to small application places such as hospital beds, and it is also unable to automatically identify and collect blood vessels, and has poor safety.

[0004] Chinese patent CN 110812577 A discloses an automatic venous puncture device to avoid various problems caused by manual puncture. However, the device has a single mode and cannot identify the internal structure of the blood vessel, such as the blood vessel wall, blood vessel cavity, venous valve, etc., which can easily cause medical accidents such as venous valve scratches and blood vessel wall penetration.

[0005] Alvin I. Chen et al. disclosed a venous puncture robot in the paper "deep learning robotic guidance for autonomous vascular access", which uses binocular near-infrared combined with ultrasound to identify the patient's forearm blood vessels and perform puncture control. Binocular near-infrared identification of the patient's forearm venous network and modeling consumes too much computing power, takes a long time to calculate, is difficult to achieve real-time, and has a low modeling utilization rate. Only the three-dimensional information of the target blood vessel is required, and the target blood vessel morphological information will change due to the compression of the ultrasound probe, resulting in large errors in the three-dimensional information recognition. Summary of the invention

[0006] The purpose of the present invention is to overcome the defects of the above-mentioned prior art and provide a fully automatic venipuncture identification integrated robot.

[0007] The purpose of the present invention can be achieved by the following technical solutions:

[0008] A fully automatic venipuncture recognition integrated robot, comprising a puncture module, an image acquisition module and a positioning platform;

[0009] The puncture module comprises a puncture module housing, a y-axis rotation unit, an x-axis rotation unit, a z-axis rotation unit and a puncture head component, wherein the y-axis rotation unit and the z-axis rotation unit are installed side by side in the puncture module housing, the x-axis rotation unit is installed below the y-axis rotation unit and connected to the z-axis rotation unit, and the puncture head component is connected to the x-axis rotation unit;

[0010] The y-axis rotation unit includes a y-axis motor and a y-axis bearing; the puncture module housing includes a main body and a connecting part, the connecting part is arched, the two ends of the y-axis motor are first fixedly connected and pass through the main body, and then on the outside of the main body, the two ends of the y-axis motor are connected through the y-axis bearing and the two ends of the arch of the connecting part, the output shaft of the y-axis motor passes through a y-axis bearing and is fixed to the connecting part 112, the center line of the two y-axis bearings is the y-axis line, and one side of the connecting part is connected to the positioning platform;

[0011] The x-axis rotation unit comprises an x-axis rotating shaft and an x-axis housing, the x-axis rotating shaft passes through the rear end of the x-axis housing transversely, the puncture head component is connected to the front end of the x-axis housing, and the axis of the x-axis rotating shaft is the x-axis line;

[0012] The z-axis rotation unit comprises a z-axis motor, a z-axis rotating shaft, a z-axis bearing, a worm and a worm wheel, wherein the z-axis motor is mounted on the main body, the lower end of the z-axis motor is connected to the z-axis rotating shaft, the z-axis bearing and the worm are nested on the z-axis rotating shaft, the outer ring of the z-axis bearing is connected to the main body, the worm is vertically connected to the worm wheel, the worm wheel is connected to the x-axis rotating shaft, and the axis of the z-axis rotating shaft is the z-axis line;

[0013] The image acquisition module includes an ultrasonic probe, an image linear motor and a near-infrared camera. The ultrasonic probe is fixed on the front movable end of the image linear motor through a probe bracket to perform up and down linear motion. The near-infrared camera is connected to the back static end of the image linear motor through a camera bracket, and the camera bracket is also installed on a positioning platform.

[0014] Furthermore, the piercing head component includes a stepper motor, a lead screw, a first bearing, a straight rod, a lead screw seat, a thrust plate and an injection unit, the rear end of the lead screw is connected to the stepper motor, the stepper motor is fixed in the x-axis housing, the front end of the lead screw passes through the lead screw seat and is connected to the tubular channel inside the x-axis housing through the first bearing, the thrust plate is connected to the inner wall of the front end surface of the x-axis housing, and is connected to the lead screw seat through two parallel straight rods, and the lower ends of the thrust plate and the lead screw seat are connected to the injection unit.

[0015] Furthermore, the injection unit includes a tablet feeder seat, a puncture motor, a drug feed push seat and a syringe clamp, the top of the tablet seat is connected to the screw seat thrust plate and the screw seat, the syringe clamp is installed under one end of the tablet seat, and the puncture motor is installed under the other end of the tablet seat. The output shaft of the puncture motor passes through the drug feed push seat and is connected to the syringe clamp through a bearing. The drug feed push seat and the output shaft of the puncture motor are connected with the screw, and the drug feed push seat is slidably connected to the tablet seat.

[0016] Furthermore, a lateral side plate is provided on one side of the connecting portion for connecting with the positioning platform.

[0017] Furthermore, the positioning platform includes a beam unit and a base unit, the beam unit is fixed on the base unit, and includes a first beam linear motor and a second beam linear motor parallel to each other, the puncture module housing is connected to the active end of the first beam linear motor, and the camera bracket is connected to the active end of the second beam linear motor.

[0018] Furthermore, the base unit includes a base, a positioning horizontal linear motor, a positioning mounting frame, a vertical rotary motor, a driving gear, a driven gear and a vertical rotating shaft. The horizontal linear motor is mounted on the base, the lower end of the vertical rotating shaft is fixedly connected to the movable end of the horizontal linear motor, the upper end of the vertical rotating shaft passes through the positioning mounting frame for rotational connection, the driven gear is mounted on the vertical rotating shaft, the vertical rotary motor is fixed in the positioning mounting frame, the output end of the vertical rotary motor is connected to the driven gear through the driving gear, and the top end of the positioning mounting frame is connected to the beam unit.

[0019] Furthermore, the base unit also includes a lifting platform, and the lifting platform is arranged between the beam unit and the top end of the positioning mounting frame.

[0020] Furthermore, a buffer pad for placing the arms is provided on the bottom plate.

[0021] Furthermore, the diameter of the driven gear is greater than the diameter of the driving gear.

[0022] Furthermore, the bottom end of the ultrasonic probe is an arc-shaped contact head.

[0023] Furthermore, the near infrared camera in the robot is a monocular near infrared camera, and its control method specifically includes the following steps:

[0024] S1. A monocular near-infrared camera continuously acquires two-dimensional near-infrared images of the target, and scores the blood vessels identified in the two-dimensional infrared images through an image neural network, and the vein with the highest score is taken as the target blood vessel;

[0025] S2, the puncture module moves the puncture head component to the field of view of the near-infrared camera and performs a posture change. The near-infrared camera obtains the postures of the puncture needle of the puncture head component before and after the change, and refreshes and calibrates the current posture of the puncture needle;

[0026] S3, the image acquisition module moves the ultrasound probe to the top of the target blood vessel and compresses the target blood vessel;

[0027] S4, the ultrasound probe collects cross-sectional images of different layers of the compressed target blood vessel and performs three-dimensional reconstruction of the venous blood vessels;

[0028] S5. The puncture module punctures the target blood vessel according to the three-dimensional blood vessel model. During the puncture process, the near-infrared camera obtains the current posture of the puncture needle in real time, and performs posture refresh and calibration.

[0029] Furthermore, in step S1, scoring the veins identified in the two-dimensional infrared image specifically includes the following steps:

[0030] S11, using an image neural network based on an SSD framework to classify objects in an input two-dimensional near-infrared image to obtain classification labels, where the classification includes at least a puncture needle, a background, an arm, a straight vein, and a vein bifurcation;

[0031] S12, filtering out blood vessel images in the classification labels;

[0032] S13, segmenting the blood vessel image using an image segmentation algorithm based on a U-Net framework to obtain a blood vessel segmentation map;

[0033] S14, scoring the blood vessel segmentation map, and scoring a certain point p on the blood vessel in the blood vessel segmentation map using an appropriate degree Ψ p express:

[0034]

[0035] In the formula, ω1(L v ) is the score corresponding to the length of the blood vessel, L v is the length of the blood vessel in the blood vessel segmentation map, ω2(W v ) is the score corresponding to the vessel width, W v is the average width of blood vessels in the blood vessel segmentation map, ω1(V v ) is the score corresponding to the blood vessel pixel value, V v is the average pixel value of blood vessels in the blood vessel segmentation map, ɑ1 is the pixel value weight of point p, is the average pixel value of point p and its neighborhood, V i is the pixel value of point p and its eight adjacent points, α2 is the distance weight between point p and the centroid of the blood vessel, D p is the distance between point p and the centroid of the blood vessel, (u p ,vp ) is the pixel coordinate of point p, (u I ,v I ) is the pixel coordinate of the centroid of the blood vessel in the blood vessel segmentation map, ∑ tip V i u i is the sum of the product of the pixel values ​​of all pixels in the blood vessel segmentation map and the pixel horizontal coordinate of the pixel, ∑ tip V i v i is the sum of the product of the pixel values ​​of all pixels in the blood vessel segmentation map and the pixel ordinate of the pixel, ∑ tip V i It is the sum of the pixel values ​​of all pixels in the blood vessel segmentation image.

[0036] Furthermore, the step S2 specifically includes:

[0037] S21, the near-infrared camera obtains posture images of the puncture needle of the puncture head component before and after the transformation, which are respectively recorded as I' and I".

[0038] S22, dedistorting the images of the front and back posture images; the calculation expression of dedistortion is:

[0039]

[0040]

[0041] in, and is the pixel coordinates of a point in the original image, x and y are the pixel coordinates of the corresponding point in the rectified image, x1 and y1 are the pixel coordinates of the center point of the rectified image, x0 and y0 are the pixel coordinates of the center point of the original image, r 2 is the square sum of the number of pixels from the image point to the center point in the horizontal and vertical directions after correction, k1, k2 and k3 are radial distortion coefficients;

[0042] S22. The dedistorted image is subjected to an image segmentation algorithm based on the U-Net framework to obtain the position of the puncture needle in the front and back posture images. The position of the puncture needle in the robot coordinate system is determined by calibrating the position of the puncture needle and its needle tip relative to the reference object, thereby confirming the position of the puncture needle in the world coordinate system.

[0043] Furthermore, the step S3 also includes: converting the two-dimensional information of the target blood vessel selected in step S1 into positioning information of the ultrasound probe, and the image acquisition module controls the movement of the ultrasound probe according to the positioning information.

[0044] Furthermore, in step S4,

[0045] S41, the ultrasonic probe acquires an ultrasonic image of the target blood vessel to obtain a two-dimensional ultrasonic image of the blood vessel cavity;

[0046] S42, using a mean shift segmentation algorithm as a segmentation method for vascular lumen from ultrasound images, modeling vascular features as a potential probability density function, and iteratively attempting to identify parameters of the probability density;

[0047] S43, calculating a color histogram of the vascular lumen segmentation image according to a probability density function, and establishing a three-dimensional initial model of the target blood vessel;

[0048] S44, registering the two-dimensional ultrasound image obtained in step S41 with the three-dimensional initial model; fully training the individual images in a convolutional neural network;

[0049] S45, registering the individual trained in step S44 in a Markov decision process by observing the local area;

[0050] S46. Take final actions according to the suggestions in the Markov decision process and weight them according to the corresponding confidence levels to complete the registration of the two-dimensional ultrasound image with the three-dimensional initial model and generate a final three-dimensional model of the target blood vessel.

[0051] Furthermore, in the mean shift segmentation algorithm of step S42, the kernel K and the bandwidth parameter h are determined, and the kernel density estimation expression for a given d-dimensional point set is:

[0052]

[0053] Among them, x represents the sample point in the point set, x i Represents the sample points in the neighborhood of sample point x.

[0054] Furthermore, in step S43, the three-dimensional initial model of the target blood vessel is established in OpenCV software, using the grayscale image, the gradient amplitude image and the Hessian image as the first, second and third dimensions of the input three-channel image.

[0055] Furthermore, the ultrasound probe uses volume ultrasound as an ultrasound mode.

[0056] Compared with the prior art, the present invention has the following beneficial effects:

[0057] 1. The present invention can greatly reduce the overall volume of the device through the three-dimensional staggered design of the puncture module, the image acquisition module and the positioning platform; the puncture module cooperates with each other through the y-axis rotation unit, the x-axis rotation unit and the z-axis rotation unit to achieve precise control of the "nodding" and "swaying" of the puncture head component. At the same time, the present invention sets up an image acquisition module so that the robot can first use the image neural network through a near-infrared camera to perform two-dimensional recognition and segmentation of the blood vessels, and then use the ultrasonic probe to compress the selected location, and establish a three-dimensional model of the target blood vessel at the compression location, and perform puncture based on the three-dimensional model. Not only does it make the recognition accuracy of the puncture process higher, the modeling speed faster, the display of the detailed structure of the blood vessels (vascular cavity, vascular wall, venous valve, etc.) more accurate and complete, and significantly improve the safety of the puncture process, but it also achieves fully automated control.

[0058] 2. The puncture module of the present invention is provided with an injection unit for adapting a needle-tube syringe. During the intravenous puncture, the syringe is pushed by moving the medicine push seat, thereby realizing the robot's injection puncture.

[0059] 3. The near-infrared vein recognition step in the control method of the present invention adopts the U-Net segmentation algorithm. This method adopts the method of encoding first and then decoding, which can avoid noise to a large extent, while retaining image information and quickly identifying the puncture site; after identification, the best target blood vessel is determined by scoring. The steps are simple and reliable, which is convenient for the subsequent movement and operation of the ultrasound probe.

[0060] 4. The present invention adopts a mean shift segmentation algorithm in the process of three-dimensional modeling of the target blood vessel, which has high robustness, few required parameters and good reliability. BRIEF DESCRIPTION OF THE DRAWINGS

[0061] Figure 1 It is a schematic diagram of the structure of this embodiment.

[0062] Figure 2 It is a structural diagram of the puncture module.

[0063] Figure 3 This is a schematic diagram of the structure of the puncture module from another angle.

[0064] Figure 4 Schematic diagram of the structure of the z-axis rotation unit.

[0065] Figure 5 It is a schematic structural diagram of the puncture head component.

[0066] Figure 6 It is a schematic diagram of the structure of the injection unit.

[0067] Figure 7 It is a side view structural diagram of the image acquisition module.

[0068] Figure 8 Schematic diagram of the three-dimensional structure of the image acquisition module.

[0069] Fig. 9 This is a structural diagram of the positioning platform.

[0070] Fig.10 Schematic diagram of the external structure of the base unit.

[0071] Fig.11 Schematic diagram of the internal structure of the base unit.

[0072] Fig.12 Schematic diagram of the use process of this embodiment.

[0073] Fig.13 Schematic diagram of the control method.

[0074] Fig.14 Schematic diagram of the steps for scanning and selecting target vessels.

[0075] Fig.15 Schematic diagram of the near-infrared puncture needle identification steps.

[0076] Fig.16a and Fig.16b Schematic diagram of the camera internal and external parameter matrix for the near-infrared puncture needle recognition step.

[0077] Fig.17 This is the basic principle diagram of camera calibration and imaging for the near-infrared puncture needle recognition step.

[0078] Fig.18 This is a schematic diagram of the operation of obtaining and calibrating the puncture needle position information.

[0079] Fig.19 The schematic diagram is for determining the vertical plane of the puncture needle tip in the near-infrared puncture needle recognition step.

[0080] Fig. 20 Schematic diagram of puncture needle coordinate calculation in the near-infrared puncture needle recognition step.

[0081] Fig.21 Schematic diagram of the steps to create a 3D model of the target vessel for the ultrasound modality.

[0082] Reference numerals:

[0083] 1. puncture module, 11. puncture module housing, 111. main body, 112. connection part, 12. y-axis rotation unit, 121. y-axis motor, 123. y-axis bearing, 13. x-axis rotation unit, 131. x-axis rotating shaft, 132. x-axis housing, 14. z-axis rotation unit, 141. z-axis motor, 142. z-axis rotating shaft, 143. z-axis bearing, 144. worm, 145. worm gear, 15. puncture head component, 151. stepping motor, 152. lead screw, 153. first bearing, 154. straight rod, 155. lead screw seat, 156. thrust plate, 157. injection unit, 157a. tablet feeding seat, 157b. puncture motor, 157c. medicine feeding push seat, 157d. syringe clamp;

[0084] 2. Image acquisition module, 21. Ultrasonic probe, 22. Image linear motor, 23. Near infrared camera, 24. Probe bracket, 25. Camera bracket;

[0085] 3. Positioning platform, 31. Beam unit, 311. First beam linear motor, 312. Second beam linear motor, 32. Base unit, 321. Base, 322. Positioning horizontal linear motor, 323. Positioning mounting frame, 324. Vertical rotary motor, 325. Driving gear, 326. Driven gear, 327. Vertical rotating shaft, 328. Lifting platform, 329. Buffer pad. DETAILED DESCRIPTION

[0086] The present invention is described in detail below in conjunction with the accompanying drawings and specific embodiments. This embodiment is implemented based on the technical solution of the present invention, and provides a detailed implementation method and specific operation process, but the protection scope of the present invention is not limited to the following embodiments.

[0087] like Figure 1 As shown, this embodiment provides a fully automatic venipuncture identification integrated robot, including a puncture module 1, an image acquisition module 2 and a positioning platform 3.

[0088] like Figure 2 and Figure 3 As shown, the puncture module 1 includes a puncture module housing 11, a y-axis rotation unit 12, an x-axis rotation unit 13, a z-axis rotation unit 14, and a puncture head component 15. The y-axis rotation unit 12 and the z-axis rotation unit 14 are installed side by side in the puncture module housing 11, and the x-axis rotation unit 13 is installed below the y-axis rotation unit 12 and connected to the z-axis rotation unit 14. The puncture head component 15 is connected to the x-axis rotation unit 13.

[0089] The y-axis rotation unit 12 includes a y-axis motor 121 and a y-axis bearing 123; the puncture module housing 11 includes a main body 111 and a connecting part 112. The connecting part 112 is arched. The two ends of the y-axis motor 121 are first fixedly connected to the main body 111 by bolts and other parts. Then, on the outside of the main body 111, the two ends of the y-axis motor 121 are connected to the two ends of the arch of the connecting part 112 through the y-axis bearing 123. A horizontal side plate 112a is also provided on one side of the connecting part 112 for stably connecting to the positioning platform 3. The output shaft of the y-axis motor 121 passes through a y-axis bearing 123 and is fixed to the connecting part 112 through a key structure. The center line of the two y-axis bearings 123 is the y-axis line. Through this structure, it can be realized that under the drive of the y-axis motor 121, the main body 111 rotates around the connecting part 112 (y-axis line) through the y-axis bearing 123.

[0090] like Figure 4 and Figure 5 As shown, the x-axis rotation unit 13 includes an x-axis rotating shaft 131 and an x-axis housing 132, the x-axis rotating shaft 131 passes through the rear end of the x-axis housing 132 horizontally, and the puncture head component 15 of the puncture module 1 is connected to the front end of the x-axis housing 132. The axis of the x-axis rotating shaft 131 is the x-axis line. The z-axis rotation unit 14 includes a z-axis motor 141, a z-axis rotating shaft 142, a z-axis bearing 143, a worm 144, and a worm gear 145. The z-axis motor 141 is mounted on the main body 11 of the puncture module housing, the lower end of the z-axis motor 141 is connected to the z-axis rotating shaft 142, and the z-axis bearing 143 and the worm 144 are nested on the z-axis rotating shaft 142. The outer ring of the z-axis bearing 142 is connected to the main body 11 of the puncture module housing, and the worm 144 is vertically connected to the worm gear 145. The worm gear 145 is connected to the x-axis rotating shaft 131. The axis of the z-axis rotating shaft 142 is the z-axis line. In this embodiment, the x-axis, y-axis and z-axis are perpendicular to each other to form a three-dimensional rectangular coordinate. Therefore, the rotation of the z-axis motor 141 drives the worm 144 to rotate, the worm 144 drives the worm gear 145 to rotate, and the worm gear 145 drives the main body 11 connected to the x-axis shaft 131 to nod up and down.

[0091] The puncture head component 15 includes a stepper motor 151, a lead screw 152, a first bearing 153, a straight rod 154, a lead screw seat 155, a thrust plate 156 and an injection unit 157. The rear end of the lead screw 152 is connected to the stepper motor 151, and the stepper motor 151 is fixed in the x-axis housing 132; the front end of the lead screw 152 passes through the lead screw seat 155 and is connected to the tubular channel inside the x-axis housing 132 through the first bearing 153. The thrust plate 156 is connected to the inner wall of the front end surface of the x-axis housing 132, and is connected to the lead screw seat 155 through two parallel straight rods 154. The lower ends of the thrust plate 156 and the lead screw seat 155 are connected to the injection unit 157. This structure converts the rotation of the lead screw 152 into the linear motion of the lead screw seat 155 through the stepper motor 151, the lead screw 152 and the lead screw seat 155, that is, the injection unit 157 can move forward and backward along the y-axis direction.

[0092] like Figure 6 As shown, the injection unit 157 includes a tablet feed seat 157a, a puncture motor 157b, a medicine feed push seat 157c and a syringe clamp 157d. The top of the tablet seat 157a is connected to the screw seat thrust plate 156 and the screw seat 155, and the syringe clamp 157d is installed below the front end of the tablet seat 157a, and the puncture motor 157b is installed below the rear end of the tablet seat 157a. The output shaft of the puncture motor 157b passes through the medicine feed push seat 157c and is connected to the end of the syringe clamp 157d through a bearing. The medicine feed push seat 157c is connected to the output shaft screw of the puncture motor 157b, and is slidably connected to the tablet seat 157a at its top. When in use, the structure drives the medicine feed push seat 157c to move forward and backward through the puncture motor 157b so that the syringe supported by the syringe clamp 157d completes the medicine push.

[0093] like Figure 7 and Figure 8 As shown, the image acquisition module 2 includes an ultrasonic probe 21, an image linear motor 22 and a near infrared camera 23. The ultrasonic probe 21 is fixed to the front movable end of the image linear motor 22 through a probe bracket 24 to perform up and down linear motion. The near infrared camera 23 is connected to the back static end of the image linear motor 22 through a camera bracket 25, and the camera bracket 25 is also installed on the positioning platform 3.

[0094] like Fig. 9As shown, the positioning platform 3 includes a beam unit 31 and a base unit 32. The beam unit 31 is fixed on the base unit 32, and includes a first beam linear motor 311 and a second beam linear motor 312 which are parallel to each other. The puncture module housing 11 is connected to the movable end of the first beam linear motor 311 through the lateral side plate 112a. The camera bracket 25 is connected to the movable end of the second beam linear motor 312, and the first beam linear motor 311 and the second beam linear motor 312 cooperate with each other, so that the puncture head component 15 is always located within the imaging range of the near-infrared camera 23.

[0095] like Fig.10 and Fig.11 As shown, the base unit 32 includes a base 321, a positioning horizontal linear motor 322, a positioning mounting frame 323, a vertical rotary motor 324, a driving gear 325, a driven gear 326 and a vertical rotating shaft 327. The horizontal linear motor 322 is mounted on the base 321. The lower end of the vertical rotating shaft 327 is fixedly connected to the movable end of the horizontal linear motor 322, and the upper end of the vertical rotating shaft 327 passes through the positioning mounting frame 323 and is rotatably connected to the positioning mounting frame 323. The driven gear 326 is mounted on the vertical rotating shaft 327. The vertical rotary motor 324 is fixed in the positioning mounting frame 323, and the output end of the vertical rotary motor 324 is connected to the driven gear 326 through the driving gear 325. The diameter of the driven gear 326 is much larger than the diameter of the driving gear 325, which is used to better adjust the rotation speed. The top end of the positioning mounting frame 323 is connected to the beam unit 31. Driven by the vertical rotating motor 324, the driving gear 325 rotates around the driven gear 326, thereby driving the vertical rotating motor 324 and the positioning mounting frame 323 to rotate around the vertical rotating shaft 327, thereby realizing the left and right swing of the positioning mounting frame 323. A lifting platform 328 can also be provided between the top of the beam unit 31 and the positioning mounting frame 323. The beam unit 31 is connected to the top of the positioning mounting frame 323 through the lifting platform to realize the lifting of the beam unit 31. The lifting platform 328 is a common motor and slide rail structure, so it is not unfolded. A buffer pad 329 is also provided on the base 321 to facilitate the placement of the arm. In addition, in this embodiment, the near-infrared camera 23 can adopt a monocular or binocular near-infrared camera; the bottom end of the ultrasonic probe 21 is an arc-shaped contact head.

[0096] The working principle of this embodiment is:

[0097] like Fig.12As shown, a syringe is placed in the syringe fixture, and the patient places the arm on the cushion 329. The image acquisition module 2 and the positioning platform 3 cooperate to control the near-infrared camera 23 to move to the top of the arm, take continuous images, and then identify the best puncture point through an external control system (the dotted circle in the figure is the shooting angle, and the X mark is the best puncture point). The image acquisition module 2 and the positioning platform 3 cooperate to control the ultrasound probe 21 to move to the front and top of the best puncture point, and slightly compress a distance (about 1 cm) upstream of the best puncture point. At the same time, the ultrasound probe 21 collects ultrasonic information and constructs a 3D model of the intravascular structure through an external control system. The puncture module 1 obtains the calculated best puncture posture, and controls the puncture head component to adjust the puncture posture through the y-axis rotation unit, the x-axis rotation unit, and the z-axis rotation unit. Finally, the puncture is achieved through the puncture head component, and the injection unit pushes the drug in the syringe to complete the puncture.

[0098] The ultrasound mode of the ultrasound probe in this embodiment adopts volume ultrasound, which can obtain the three-dimensional subdivision structure of the blood vessel, clearly distinguish the blood vessel wall and the blood vessel cavity, and improve the safety of venous puncture. The near-infrared camera is a monocular near-infrared camera. The specific control method in the above working principle is as follows: Fig.13 As shown:

[0099] Step S1, a monocular near-infrared camera continuously acquires a two-dimensional near-infrared image of the target, and scores the blood vessels identified in the two-dimensional infrared image through an image neural network, and the vein with the highest score is taken as the target blood vessel;

[0100] Step S2: the puncture module moves the puncture head component into the field of view of the near-infrared camera and performs a posture change. The near-infrared camera obtains the postures of the puncture needle of the puncture head component before and after the change, and refreshes and calibrates the current posture of the puncture needle.

[0101] Step S3: the image acquisition module moves the ultrasound probe to the top of the target blood vessel and compresses the target blood vessel;

[0102] Step S4, the ultrasonic probe collects cross-sectional images of different layers of the compressed target blood vessel and performs three-dimensional reconstruction of the venous blood vessel;

[0103] Step S5: The puncture module punctures the target blood vessel according to the three-dimensional blood vessel model. During the puncture process, the near-infrared camera obtains the current posture of the puncture needle in real time, and performs posture refresh and calibration.

[0104] The details are as follows:

[0105] 1. If Fig.14 As shown, the scoring of the veins identified in the two-dimensional infrared image in step S1 specifically includes the following steps:

[0106] Vascular classification:

[0107] Step S11, using an image neural network based on the SSD framework to classify objects in the input two-dimensional near-infrared image to obtain classification labels, the classifications including puncture needle, background, arm, straight vein and vein bifurcation, etc.;

[0108] Step S12, filtering out blood vessel images in the classification labels;

[0109] Vessel segmentation:

[0110] Step S13, segmenting the blood vessel image using an image segmentation algorithm based on a U-Net framework to obtain a blood vessel segmentation map;

[0111] Label scoring and target vein selection:

[0112] Step S14: Score the vascular segmentation map based on the Bayesian theorem to determine the appropriate puncture point, and use the vein with the highest score as the target vessel. The score of a point p on the blood vessel in the vascular segmentation map is scored using the appropriate degree Ψ p express:

[0113]

[0114] In the formula, ω1(L v ) is the score corresponding to the length of the blood vessel, L v is the length of the blood vessel in the blood vessel segmentation map, ω2(W v ) is the score corresponding to the vessel width, W v is the average width of blood vessels in the blood vessel segmentation map, ω1(V v ) is the score corresponding to the blood vessel pixel value, V v is the average pixel value of blood vessels in the blood vessel segmentation map, α1 is the pixel value weight of point p, is the average pixel value of point p and its neighborhood, V i is the pixel value of point p and its eight adjacent points, α2 is the distance weight between point p and the centroid of the blood vessel, D p is the distance between point p and the centroid of the blood vessel, (u p ,v p ) is the pixel coordinate of point p, (u I ,v I ) is the pixel coordinate of the centroid of the blood vessel in the blood vessel segmentation map, ∑ tip V i u i is the sum of the product of the pixel values ​​of all pixels in the blood vessel segmentation map and the pixel horizontal coordinate of the pixel, ∑ tip V i v i is the sum of the product of the pixel values ​​of all pixels in the blood vessel segmentation map and the pixel ordinate of the pixel, ∑ tip V iIt is the sum of the pixel values ​​of all pixels in the blood vessel segmentation image.

[0115] 2. If Figures 15-20 As shown, step S2 is specifically expanded as follows:

[0116] First, the parameters of the monocular near-infrared camera and the telescopic transformation parameters between the image and the real object are calibrated. Then, the puncture module extends the puncture needle under the near-infrared probe, rotates along the axis of the puncture needle, and scans the puncture needle with the monocular near-infrared camera to calibrate the morphological parameters of the puncture needle. The calibration steps are as follows:

[0117] Step S21: Calibration of the monocular near-infrared camera and telescopic transformation parameters. This step is actually completed before using the puncture robot. Fig.16a As shown, the specifications of the calibration board chessboard are determined, the square is a square, and the side length is l chessboard , use a monocular near-infrared camera and a calibration board checkerboard to take multiple calibration pictures. Use the calibration program to analyze the calibration pictures to obtain the internal reference Q of the monocular near-infrared camera. intrinsic and distortion coefficients {k1, k2…k5}. The two will be used for subsequent image dedistortion calculations. After the monocular near-infrared camera is calibrated, the scaling factor m in the projection relationship can be obtained based on the side length of the calibration plate square and the corresponding dedistorted image. x , m y The two transformation factors will be used in the subsequent process of determining the puncture needle position information based on the image.

[0118] A: Calibration of internal and external parameters of monocular near-infrared camera. Fig.17 As shown, the internal parameter matrix Q intrinsic It is to correspond the pixel position on the image plane to the physical position in the focal plane. The external parameters include the center C of the monocular near-infrared camera and the rotation matrix R of the monocular near-infrared camera. It is the correspondence established between the corresponding points in the world coordinate system and the image coordinate system:

[0119]

[0120] Among them, P m is the physical position coordinate in the focal plane, (xy 1) T is the homogeneous coordinate of the pixel position in the image plane, α x , α y are the focal lengths in the x and y directions, respectively, x , p y They are the physical coordinates of the principal point in the focal plane relative to its upper left corner origin.

[0121] B: Calibration of telescopic transformation parameters. After the monocular near-infrared camera is calibrated, the dedistorted calibration plate image can be calculated based on the internal parameters and distortion parameters.chessboard And the pixel coordinates of the corresponding single grid corner point in the dedistorted image can obtain the scaling transformation factor m in the projection relationship x , m y The two transformation factors will be used in the subsequent process of determining the puncture needle position information based on the image.

[0122]

[0123]

[0124] in, is the pixel position coordinate, The horizontal projection coordinates in the world coordinate system are as follows: Fig.16b shown.

[0125] Step S22: Acquisition and calibration of puncture needle position information. This step is performed after the near-infrared camera has finished scanning the blood vessels. Fig.18 As shown, first, the puncture needle is moved to a horizontal position below the near-infrared camera to capture the current image, which is recorded as I'; the puncture needle and the robotic arm are rotated to another angle, and the image is captured again, which is recorded as I". Then, combined with the internal reference Q in S21 intrinsic The collected image is dedistorted by using the distortion coefficients {k1, k2…k5}. After that, the position of the puncture needle in the image is obtained by U-net fast segmentation, and the segmentation results are recorded as I s ' eg , I″ seg The basic structure of U-net is an encoder and a decoder. Then, the position of the puncture needle in the robot coordinate system (i.e., the world coordinate system) is determined by calibrating the position of the puncture needle and the needle tip relative to the puncture moving mechanism. These position parameters will be used for subsequent precise puncture guidance.

[0126] C: Image distortion removal. Image distortion here mainly refers to radial distortion, which means that light is more bent away from the center of the lens than near the center. The compensation correction formula for radial distortion is as follows:

[0127]

[0128]

[0129] in, and is the pixel coordinates of a point in the original image, x and y are the pixel coordinates of the corresponding point in the rectified image, x1 and y1 are the pixel coordinates of the center point of the rectified image, x0 and y0 are the pixel coordinates of the center point of the original image, r 2is the square sum of the number of pixels from the corrected image point in the horizontal and vertical directions to the center point, k1, k2, k3 are radial distortion coefficients, k4, k5 are tangential distortion coefficients, and {k1, k2...k5} are all obtained through step S21.

[0130] D: Calibration calculation of puncture needle position. Fig.19 As shown, the vertical plane of the puncture needle tip The position in the robot coordinate system can be determined based on the position of the end of the puncture robot arm and the position of the needle tip in the image. The XY plane in the figure is the coordinate plane corresponding to the robot coordinate system, that is, the world coordinate system. It can be expressed in coordinate form as

[0131] Afterwards, if Fig. 20 As shown in the figure, the position of the Y′ axis of the spatial coordinate system O′-X′Y′Z′ in the world coordinate system is determined by the robot control. Its coordinate form is:

[0132]

[0133] The coordinate x of the needle tip relative to O′-X′Y′Z′ can be determined based on the vertical relationship between the two images and the projection relationship of the needle tip. p ,z p Therefore, the world coordinates of the needle tip can be expressed as:

[0134]

[0135] During the puncture process, the deformation of the puncture needle can be ignored. Therefore, the world coordinates of the needle tip combined with the position of the end of the robot arm in the world coordinate system can obtain the puncture needle shape and position parameters required during the puncture process.

[0136] 3. The specific expansion of step S3 is as follows:

[0137] Step S31: Convert the two-dimensional information of the target blood vessel selected by the near-infrared light in S2 into the positioning information of the ultrasound probe, and input it into the mechanical control program for controlling the movement of the ultrasound probe. The positioning platform positions the ultrasound probe above the target blood vessel in the patient's forearm by translation and rotation.

[0138] Step S32: The ultrasound probe moves downward through the image acquisition module until it reaches the surface of the patient's arm skin. According to the ultrasound probe positioning information input into the program in step S31, the robotic arm of the image acquisition module controls the ultrasound probe to 1 cm above the target blood vessel positioning coordinates, and then the robotic arm controls the ultrasound probe to slowly descend. At this time, the ultrasound probe presses the target blood vessel with a certain force and sticks to the patient's skin, which is convenient for observing the patient's target blood vessel structure and performing target blood vessel segmentation and modeling.

[0139] 4. Fig.21As shown, step S4 specifically includes:

[0140] Step S41: The ultrasound probe acquires an ultrasound image of the target blood vessel.

[0141] Step S42, using the mean shift segmentation algorithm to segment the vascular cavity from the ultrasound image, modeling the vascular features as potential probability density, and iteratively trying to identify the parameters of the probability density to obtain a vascular cavity segmentation map. Based on the maximum likelihood criterion, if there is a dense cluster in the characteristic vascular space, it corresponds to the local maximum of the probability density function. Given the kernel K and the bandwidth parameter h, the kernel density of a given d-dimensional point set is estimated as:

[0142]

[0143] Determine the weights of nearby points and re-estimate the mean. The weighted average of the density within the range of the ultrasound probe determined by K is:

[0144]

[0145] Where N(x) is the neighborhood of x, x represents the sample point in the point set, and x i Represents the sample points in the neighborhood of sample point x.

[0146] After obtaining m(x), the program is iterated, and the m(x) of the previous iteration is substituted into the x of this iteration step, that is, x←m(x), and the gradient descent algorithm is used to re-estimate until m(x) converges. After convergence, the vascular lumen is successfully segmented and the vascular lumen segmentation map is obtained.

[0147] Step S43, perform ultrasound modeling preprocessing on the target blood vessel. Calculate the color histogram of the vascular lumen segmentation map, use it as a probability density function, and establish a three-dimensional initial model of the target blood vessel. Establishing the three-dimensional initial model of the target blood vessel is implemented in OpenCV software, using grayscale images, gradient amplitude images, and Hessian images as the first, second, and third dimensions of the input three-channel image.

[0148] Step S44, registering the two-dimensional ultrasound image obtained in step S41 with the three-dimensional initial model; and fully training the individual images in a convolutional neural network.

[0149] Step S45: By observing the local area, the individuals trained in step S44 are registered in the Markov decision process. The Markov decision process is determined by a 5-dimensional space {Г, A, P(·), R(·), γ}. Where Г is a set of possible states; A is a set of possible activities; P A (T) is the state obtained by taking action A in state T; R A(T) is the reward obtained by taking action A in state T; γ is the discount factor that controls the importance of future rewards. With the action space A and the reward schedule R(·) determined, the core problem of MDP is to find a solution π(·) to specify the best action π(T t ), so that state T t Maximizing long-term returns:

[0150]

[0151] Step S46: taking final actions according to the suggestions in the Markov decision process and weighting them according to the corresponding confidence levels to complete the registration of the two-dimensional ultrasound image with the three-dimensional initial model and generate a final three-dimensional model of the target blood vessel.

[0152] The preferred specific embodiments of the present invention are described in detail above. It should be understood that a person skilled in the art can make many modifications and changes based on the concept of the present invention without creative work. Therefore, any technical solution that can be obtained by a person skilled in the art through logical analysis, reasoning or limited experiments based on the concept of the present invention on the basis of the prior art should be within the scope of protection determined by the claims.

Claims

1. A fully automatic venipuncture identification integrated robot, characterized in that: It comprises a puncture module (1), an image acquisition module (2) and a positioning platform (3); The puncture module (1) comprises a puncture module housing (11), a y-axis rotation unit (12), an x-axis rotation unit (13), a z-axis rotation unit (14) and a puncture head component (15); the y-axis rotation unit (12) and the z-axis rotation unit (14) are installed side by side in the puncture module housing (11); the x-axis rotation unit (13) is installed below the y-axis rotation unit (12) and connected to the z-axis rotation unit (14); and the puncture head component (15) is connected to the x-axis rotation unit (13); The y-axis rotation unit (12) comprises a y-axis motor (121) and a y-axis bearing (123); the puncture module housing (11) comprises a main body (111) and a connecting part (112); the connecting part (112) is in an arcuate shape; the two ends of the y-axis motor (121) are first fixedly connected and pass through the main body (111); then, on the outside of the main body (111), the two ends of the y-axis motor (121) are connected to the two ends of the arcuate connecting part (112) through the y-axis bearing (123); the output shaft of the y-axis motor (121) passes through a y-axis bearing (123) and is fixed to the connecting part (112); the center line of the two y-axis bearings (123) is the y-axis line; one side of the connecting part (112) is connected to the positioning platform (3); The x-axis rotation unit (13) comprises an x-axis rotation shaft (131) and an x-axis housing (132), the x-axis rotation shaft (131) transversely passes through the rear end of the x-axis housing (132), the puncture head component (15) is connected to the front end of the x-axis housing (132), and the axis of the x-axis rotation shaft (131) is the x-axis line; The z-axis rotation unit (14) comprises a z-axis motor (141), a z-axis rotating shaft (142), a z-axis bearing (143), a worm (144) and a worm wheel (145); the z-axis motor (141) is mounted on the main body (111); the lower end of the z-axis motor (141) is connected to the z-axis rotating shaft (142); the z-axis bearing (143) and the worm (144) are nested on the z-axis rotating shaft (142); the outer ring of the z-axis bearing (143) is connected to the main body (111); the worm (144) is vertically connected to the worm wheel (145); the worm wheel (145) is connected to the x-axis rotating shaft (131); and the axis of the z-axis rotating shaft (142) is the z-axis line; The image acquisition module (2) comprises an ultrasonic probe (21), an image linear motor (22) and a near-infrared camera (23); the ultrasonic probe (21) is fixed to the front movable end of the image linear motor (22) via a probe bracket (24) to perform vertical linear motion; the near-infrared camera (23) is connected to the rear static end of the image linear motor (22) via a camera bracket (25), and the camera bracket (25) is also mounted on the positioning platform (3); The puncture head component (15) comprises a stepper motor (151), a lead screw (152), a first bearing (153), a straight rod (154), a lead screw seat (155), a thrust plate (156) and an injection unit (157); the rear end of the lead screw (152) is connected to the stepper motor (151); the stepper motor (151) is fixed in the x-axis housing (132); the front end of the lead screw (152) passes through the lead screw seat (155) and is connected to the tubular channel inside the x-axis housing (132) through the first bearing (153); the thrust plate (156) is connected to the inner wall of the front end surface of the x-axis housing (132) and is connected to the lead screw seat (155) through two parallel straight rods (154); the lower ends of the thrust plate (156) and the lead screw seat (155) are connected to the injection unit (157); The injection unit (157) comprises a tablet feed seat (157a), a puncture motor (157b), a drug feed push seat (157c) and a syringe clamp (157d); the top end of the tablet seat (157a) is connected to a thrust plate (156) and a screw seat (155); the syringe clamp (157d) is installed below one end of the tablet seat (157a); the puncture motor (157b) is installed below the other end of the tablet seat (157a); the output shaft of the puncture motor (157b) passes through the drug feed push seat (157c) and is connected to the syringe clamp (157d) via a bearing; the drug feed push seat (157c) and the output shaft screw of the puncture motor (157b) are connected, and the drug feed push seat (157c) is slidably connected to the tablet seat (157a); The positioning platform (3) comprises a crossbeam unit (31) and a base unit (32); the crossbeam unit (31) is fixed on the base unit (32), and comprises a first crossbeam linear motor (311) and a second crossbeam linear motor (312) which are parallel to each other; the puncture module housing (11) is connected to the movable end of the first crossbeam linear motor (311); and the camera bracket (25) is connected to the movable end of the second crossbeam linear motor (312); The base unit (32) comprises a base (321), a positioning horizontal linear motor (322), a positioning mounting frame (323), a vertical rotary motor (324), a driving gear (325), a driven gear (326) and a vertical rotating shaft (327), wherein the horizontal linear motor (322) is mounted on the base (321), the lower end of the vertical rotating shaft (327) is fixedly connected to the movable end of the horizontal linear motor (322), the upper end of the vertical rotating shaft (327) passes through the positioning mounting frame (323) and is rotatably connected to the positioning mounting frame (323), the driven gear (326) is mounted on the vertical rotating shaft (327), the vertical rotary motor (324) is fixed in the positioning mounting frame (323), the output end of the vertical rotary motor (324) is connected to the driven gear (326) through the driving gear (325), and the top end of the positioning mounting frame (323) is connected to the beam unit (31); The base unit (32) further comprises a lifting platform (328), and the lifting platform (328) is arranged between the crossbeam unit (31) and the top end of the positioning mounting frame (323).

2. The fully automatic venipuncture identification integrated robot according to claim 1, characterized in that: The near-infrared camera is a monocular near-infrared camera, and the control method of the fully automatic venipuncture recognition integrated robot comprises the following steps: S1. A monocular near-infrared camera continuously acquires two-dimensional near-infrared images of the target, and scores the blood vessels identified in the two-dimensional infrared images through an image neural network, and the vein with the highest score is taken as the target blood vessel; S2, the puncture module moves the puncture head component to the field of view of the near-infrared camera and performs a posture change. The near-infrared camera obtains the postures of the puncture needle of the puncture head component before and after the change, and refreshes and calibrates the current posture of the puncture needle; S3, the image acquisition module moves the ultrasound probe to the top of the target blood vessel and compresses the target blood vessel; S4, the ultrasound probe collects cross-sectional images of different layers of the compressed target blood vessel and performs three-dimensional reconstruction of the venous blood vessels; S5. The puncture module punctures the target blood vessel according to the three-dimensional blood vessel model. During the puncture process, the near-infrared camera obtains the current posture of the puncture needle in real time, and performs posture refresh and calibration.

3. The fully automatic venipuncture identification integrated robot according to claim 2, characterized in that: In step S1 of the control method, scoring the veins identified in the two-dimensional infrared image specifically includes the following steps: S11, using an image neural network based on an SSD framework to classify objects in an input two-dimensional near-infrared image to obtain classification labels, where the classification includes at least a puncture needle, a background, an arm, a straight vein, and a vein bifurcation; S12, filtering out blood vessel images in the classification labels; S13, segmenting the blood vessel image using an image segmentation algorithm based on a U-Net framework to obtain a blood vessel segmentation map; S14, scoring the blood vessel segmentation map, and scoring a certain point p on the blood vessel in the blood vessel segmentation map using an appropriate degree Ψ p express: In the formula, ω1(L v ) is the score corresponding to the length of the blood vessel, L v is the length of the blood vessel in the blood vessel segmentation map, ω2(W v ) is the score corresponding to the vessel width, W v is the average width of blood vessels in the blood vessel segmentation map, ω1(V v ) is the score corresponding to the blood vessel pixel value, V v is the average pixel value of blood vessels in the blood vessel segmentation map, α1 is the pixel value weight of point p, is the average pixel value of point p and its neighborhood, V i is the pixel value of point p and its eight adjacent points, ɑ2 is the distance weight between point p and the centroid of the blood vessel, D p is the distance between point p and the centroid of the blood vessel, (u p ,v p ) is the pixel coordinate of point p, (u I ,v I ) is the pixel coordinate of the centroid of the blood vessel in the blood vessel segmentation map, ∑ tip V i μ i is the sum of the product of the pixel values ​​of all pixels in the blood vessel segmentation map and the pixel horizontal coordinate of the pixel, ∑ tip V i v i is the sum of the product of the pixel values ​​of all pixels in the blood vessel segmentation map and the pixel ordinate of the pixel, ∑ tip V i It is the sum of the pixel values ​​of all pixels in the blood vessel segmentation image.

4. The fully automatic venipuncture identification integrated robot according to claim 2, characterized in that: Step S2 of the control method specifically includes: S21, the near-infrared camera obtains posture images of the puncture needle of the puncture head component before and after the transformation, which are respectively recorded as I' and I". S22, dedistorting the images of the front and back posture images, the calculation expression for dedistortion is: in, and is the pixel coordinates of a point in the original image, x and y are the pixel coordinates of the corresponding point in the rectified image, x1 and y1 are the pixel coordinates of the center point of the rectified image, x0 and y0 are the pixel coordinates of the center point of the original image, r 2 is the square sum of the number of pixels from the image point to the center point in the horizontal and vertical directions after correction, k1, k2 and k3 are radial distortion coefficients; S22. The dedistorted image is subjected to an image segmentation algorithm based on the U-Net framework to obtain the position of the puncture needle in the front and back posture images. The position of the puncture needle in the robot coordinate system is determined by calibrating the position of the puncture needle and its needle tip relative to the reference object, thereby confirming the position of the puncture needle in the world coordinate system.

5. The fully automatic venipuncture identification integrated robot according to claim 2, characterized in that: Step S4 of the control method specifically includes: S41, the ultrasonic probe acquires an ultrasonic image of the target blood vessel to obtain a two-dimensional ultrasonic image of the blood vessel cavity; S42, using a mean shift segmentation algorithm as a segmentation method for vascular lumen from ultrasound images, modeling vascular features as a potential probability density function, and iteratively attempting to identify parameters of the probability density; S43, calculating a color histogram of the vascular lumen segmentation image according to a probability density function, and establishing a three-dimensional initial model of the target blood vessel; S44, registering the two-dimensional ultrasound image obtained in step S41 with the three-dimensional initial model; fully training the individual images in a convolutional neural network; S45, registering the individual trained in step S44 in a Markov decision process by observing the local area; S46. Take final actions according to the suggestions in the Markov decision process and weight them according to the corresponding confidence levels to complete the registration of the two-dimensional ultrasound image with the three-dimensional initial model and generate a final three-dimensional model of the target blood vessel.

Images