Thyroid puncture auxiliary system with automatic path planning function

Abstract

The invention provides a thyroid puncture auxiliary system with image scoring and path planning functions and a working method of the thyroid puncture auxiliary system. The thyroid puncture auxiliary system can realize automatic closed-loop control from ultrasonic image acquisition, target structure detection, image scoring, path planning to puncture control; the system can automatically screen out image frames most suitable for puncture through a multi-dimensional factor image scoring function, meanwhile, key structures such as thyroid nodules, tracheas and blood vessels can be recognized in the path planning process, and then a puncture path meeting angle constraints and avoiding important tissues is planned through an obstacle avoidance path algorithm. The puncture risk caused by missing detection and false detection is fundamentally reduced; continuous scoring, path planning and action execution in a real-time ultrasonic video stream are supported, high-precision dynamic mapping from a 2D image path to a 3D mechanical arm space is achieved, limitation of traditional static calibration is overcome, it is ensured that operations such as automatic scanning, frame selection, positioning and needle inserting are accurate, manual intervention is reduced, and operation efficiency is improved.

Description

TECHNICAL FIELD

[0001] The present application relates to the field of medical image processing, robot automatic control and ultrasound-guided puncture, in particular to a thyroid puncture auxiliary system with image scoring and path planning functions, and belongs to the cross-technology application of intelligent medical equipment and image navigation control. BACKGROUND

[0002] The detection rate of thyroid nodules in the population is about 20% to 76%, of which about 6% to 18% are malignant tumors. Accurate judgment of the nature of the nodule is of great significance for the development of individualized diagnosis and treatment plan. Ultrasound imaging has wide application value in thyroid intervention operation due to its non-invasive, real-time and non-radiation advantages. Ultrasound-guided thyroid fine needle biopsy as an efficient and minimally invasive sampling method has become one of the standard methods for clinical diagnosis of thyroid diseases.

[0003] In actual clinical operation, the doctor needs to hold the ultrasound probe with one hand and perform puncture with the other hand, and the whole process is highly dependent on the experience of the operator. During the puncture process, the doctor needs to judge the image clarity in real time and determine the appropriate puncture path, which often causes problems such as long scanning time, strong subjectivity of image frame selection, inaccurate path planning, and disappearance of the needle body in the image, directly affecting the safety and operation efficiency of the puncture.

[0004] In view of the limitations of traditional manual operation, some studies attempt to combine ultrasound images with robot technology to develop an automated puncture auxiliary system. However, existing systems generally have low automation, complex operation and other problems. Most systems still require manual frame selection by doctors, and an automatic scoring mechanism based on image quality has not been established; path planning usually uses a fixed straight path, lacks obstacle avoidance ability for structures such as trachea and blood vessels, and is difficult to meet the diversified regulation and control requirements for puncture angle; in addition, existing automatic detection algorithms are easily disturbed by ultrasound artifacts in real-time video streams, resulting in unstable detection results of the target (especially blood vessels), with potential safety hazards such as missed detection or flickering; at the same time, the calibration conversion method between image coordinates and mechanical arm working space is relatively static, and it is difficult to adapt to the dynamic changes of images and poses in real-time scanning process. SUMMARY

[0005] The present application aims to solve the key problems existing in the current ultrasound-guided thyroid puncture operation, such as strong subjectivity of image frame selection, lack of obstacle avoidance ability in path planning, unstable real-time detection results prone to flickering, uncontrollable puncture angle, and difficulty in automatic mapping between image coordinates and mechanical arm control space, and proposes a thyroid puncture auxiliary system with image scoring and path planning functions and its working method. The system can realize automatic closed-loop control from ultrasound image acquisition, target structure detection, image scoring, path planning to puncture control, and improve the intelligent degree, accuracy and safety of puncture operation.

[0006] In a first aspect, the present application provides a thyroid puncture auxiliary system based on image scoring and path planning, characterized in that the system comprises the following steps:

[0007] An ultrasound image acquisition module is configured to continuously acquire a sequence of ultrasound image frames and synchronously record a six-degree-of-freedom pose of a robot arm corresponding to each image frame during scanning by the robot arm driving an ultrasound probe;

[0008] An image processing module is configured to perform optimization processing on the acquired ultrasound image frames to obtain first image frames;

[0009] A target detection module is configured to automatically identify key anatomical structures in the image by applying a pre-trained deep learning model to the preprocessed first image frames;

[0010] An image scoring module is configured to receive target anatomical structures and position box information output by the target detection module, and calculate image frame scores according to a preset scoring function;

[0011] A path planning module is configured to select a best reference image frame based on the scoring results, and plan an automatic planning of a puncture path that meets a preset puncture angle range in the best reference image frame, to obtain an optimal path, pixel coordinates p of key points of the optimal path on a two-dimensional image, and a custom path parameter C corresponding to the optimal path path ;

[0012] A coordinate calibration module is configured to use six-degree-of-freedom pose data of the robot arm end synchronously recorded when the best reference image frame is acquired, and combine a pre-calibrated coordinate system transformation relationship, to solve three-dimensional space coordinates of the optimal path pixel coordinates p in a robot control system recognizable coordinate system through the coordinate system transformation relationship;

[0013] A puncture execution module includes an angle adjustment mechanism, a puncture needle device, and a controller, and is configured to adjust a puncture posture and perform a puncture operation according to the received three-dimensional coordinates and the custom path parameter C path .

[0014] Further, specifically, the target detection module is configured to,

[0015] input the preprocessed first image frames into a deep learning model to output continuous frame original detection results;

[0016] perform time sequence consistency verification on the continuous frame original detection results according to the continuous frame original detection results to obtain stable key anatomical structures.

[0017] Preferably, the scoring function includes factors including thyroid nodule area, nodule center distance, detection confidence, obstacle penalty term and position validity penalty term;

[0018] The preset scoring function is:

[0019] Score=w A *A-w D *D+w C *C-O-I

[0020] Wherein, A is the nodule area score, D is the distance score of the nodule center and the image center, C is the nodule detection confidence score, O is the obstacle penalty term, and I is the nodule position validity penalty term; Set the weight and penalty value of each parameter as: w A =1.0, w D =0.5, w C =0.5; Obstacle penalty term O=100; Position invalidity penalty term I=500; Calculate the comprehensive score Score of the current frame.

[0021] Specifically, the preset puncture angle range is preferably 30° to 45°.

[0022] Specifically, the path planning module selects the frame with the highest score as the best reference frame based on the scores of each frame calculated by the image scoring module; and,

[0023] The path planning module is specifically used for,

[0024] The nodule center identified in the key anatomical structure is the starting point, and a plurality of candidate puncture straight lines are drawn to the edge of the skin surface on the image to obtain a plurality of candidate paths;

[0025] For each of the candidate path lines, obstacle avoidance verification is performed using the obstacle position information in the key anatomical structure;

[0026] The path with the longest distance from the nearest obstacle and the relatively shortest path length is selected as the optimal path.

[0027] Specifically, the puncture execution module specifically includes:

[0028] The system controls the six-degree-of-freedom robot arm to move to the corresponding puncture preparation position according to the target three-dimensional coordinates;

[0029] Puncture instruction conversion is performed to convert the target three-dimensional coordinates into control instructions;

[0030] The puncture execution module controls the angle adjusting mechanism to accurately adjust the incident angle of the puncture needle to the optimal path C path .

[0031] In a second aspect, the present application provides a thyroid puncture assisting method, which is applied to the thyroid puncture assisting system based on the image score and path planning as described above, and the method comprises the following steps:

[0032] S1, controlling the mechanical arm to carry out a lateral scan with the ultrasonic probe, and collecting an ultrasonic image frame and corresponding mechanical arm end six-degree-of-freedom pose information in real time;

[0033] S2, pre-processing and optimizing the collected image frame, extracting a maximum connected domain, and removing invalid information at the edge of the image;

[0034] S3, calling a pre-trained target detection network to detect a key anatomical structure in the image, and checking the detection results of continuous frames through a time sequence checking unit to confirm the stable key anatomical structure;

[0035] S4, constructing a score function based on the position box information of a target anatomical structure in the key anatomical structure, and calculating the scores of each image frame according to the score function;

[0036] S5, screening out an image frame with the highest score as an optimal reference frame for path planning;

[0037] S6, in the optimal reference frame, combining the distribution of the stable key anatomical structure optimal reference frame to plan a puncture path that meets the angle constraint range and does not overlap with an obstacle anatomical structure;

[0038] S7, using the six-degree-of-freedom pose data of the mechanical arm end recorded synchronously when the optimal reference frame is collected, combining a pre-calibrated coordinate system transformation relationship, and converting the path pixel coordinates into three-dimensional path points in the mechanical arm coordinate system through the coordinate system transformation relationship;

[0039] S8, a puncture execution module controls a puncture mechanism to adjust the posture according to the path coordinates and angle information, and drives a puncture needle to perform a puncture operation along the path;

[0040] S9, if the score of the current image frame is lower than a preset threshold, the system controls the mechanical arm end to deflect by a certain angle to reacquire an image and returns to step S1 until the score meets the standard.

[0041] In a third aspect, the present application provides a device for dynamically predicting the mechanical properties after stent implantation, which comprises a memory and one or more programs, wherein the one or more programs are stored in the memory and configured to be executed by one or more processors, and the one or more programs comprise the steps of the thyroid puncture assisting method.

[0042] In a fourth aspect, the present application provides a computer readable storage medium, wherein the computer readable storage medium stores a computer program, and the computer program, when executed by a processor, implements the steps of any of the thyroid puncture assisting methods.

[0043] Based on the present application, a thyroid puncture assisting system with image scoring and path planning functions and a working method thereof are provided, which achieve the following beneficial technical effects:

[0044] 1. The objectivity and accuracy of the puncture image frame selection are improved. By constructing an image scoring function based on multiple dimensions such as nodule area, center position, detection confidence, obstacle penalty term, and position validity, the system can automatically select the most suitable image frame for puncture, overcoming the subjectivity of manual frame selection and improving the consistency and accuracy of frame selection.

[0045] 2. The reliability of obstacle identification is improved, and the path safety is significantly enhanced. The system can identify key structures such as thyroid nodules, trachea, and blood vessels during path planning. The present application particularly introduces a time sequence verification unit, which can effectively filter out transient detection flicker (such as temporary disappearance of nodules) caused by artifacts or jitter in the ultrasound video stream, ensuring the stability and reliability of obstacle identification, and then planning a puncture path that meets the angle constraint and avoids important tissues through obstacle avoidance path algorithm, thereby fundamentally reducing the puncture risk caused by missed detection and false detection.

[0046] 3. The degree of automation and dynamic accuracy of puncture positioning are improved. The system supports continuous scoring, path planning, and action execution in real-time ultrasound video stream. By combining the six-degree-of-freedom mechanical arm pose recorded synchronously with the pre-calibrated hand-eye relationship, the present application realizes high-precision dynamic mapping from 2D image path to 3D mechanical arm space. This method overcomes the limitations of traditional static calibration, ensures accurate reach of automatic scanning, frame selection, positioning, and needle insertion operations, reduces manual intervention, and improves operation efficiency.

[0047] 4. It has good clinical adaptability and expansibility. The system structure of the present application is reasonable and has high function integration, which can adapt to different types of ultrasound equipment and puncture needle control mechanisms, has the potential for wide application in thyroid and other superficial organ puncture operations, and can also be extended to other ultrasound-guided interventional scenarios. BRIEF DESCRIPTION OF DRAWINGS

[0048] In order to more clearly illustrate the technical solutions in the embodiments of the present application or the prior art, the following will briefly introduce the drawings needed to be used in the embodiment or prior art description. Obviously, the drawings in the following description are only some embodiments described in the present application, and for those skilled in the art, other drawings can also be obtained without creative labor on the basis of these drawings.

[0049] Figure 1 A schematic diagram of the overall physical structure of the system in the embodiment of the present application;

[0050] Figure 2 A network architecture diagram of the YOLOv11 module in the embodiment of the present application;

[0051] Figure 3 A principle diagram of unstable detection filtering of the timing verification unit in the embodiment of the present application;

[0052] Figure 4 A scoring factor diagram of the image scoring module in the embodiment of the present application;

[0053] Figure 5 A schematic diagram of the obstacle avoidance path generated by the path planning module in the embodiment of the present application;

[0054] Figure 6 A principle flowchart of data fusion and transformation of the coordinate calibration module in the embodiment of the present application;

[0055] Figure 7 A coordinate system diagram of the hand-eye relationship calibration principle in the embodiment of the present application;

[0056] Figure 8 An internal structure diagram of the puncture execution module in the embodiment of the present application. DETAILED DESCRIPTION

[0057] The embodiments of the present application will be described in detail below with specific, concrete examples. Those skilled in the art can easily understand other advantages and effects of the present application from the content disclosed in the specification. The present application can also be implemented or applied in different specific embodiments, and the details in the specification can be modified or changed based on different views and applications without departing from the spirit of the present application. It should be noted that the following embodiments and features in the embodiments can be combined with each other without conflict.

[0058] The present application aims to solve the key problems existing in the existing ultrasound-guided thyroid puncture operation, such as strong subjectivity of image frame selection, no obstacle avoidance ability of path planning, unstable real-time detection results prone to flickering, uncontrollable puncture angle, and difficulty in automatic mapping of image coordinates and mechanical arm control space, and proposes a thyroid puncture auxiliary system with image scoring and path planning functions and a working method thereof. The system can realize automatic closed-loop control from ultrasound image acquisition, target structure detection, image scoring, path planning to puncture control, and improve the intelligent degree, precision and safety of puncture operation.

[0059] In the following embodiments of the present application, the specific exemplary technical solutions are examples and the implementation of the technical solutions is not limited thereto.

[0060] First embodiment

[0061] The application provides a thyroid puncture auxiliary system based on image scoring and path planning. The system integrates key function modules such as target structure identification, image scoring, path planning, coordinate calibration and puncture control, can continuously run in real-time video stream, realizes full-process automation from image acquisition to puncture control, and significantly improves the intelligent degree, safety and clinical practicability of thyroid puncture operation.

[0062] Referring to Figure 1 , which is a schematic diagram of the overall physical structure of the system in the embodiment of the application. The system mainly comprises a six-degree-of-freedom mechanical arm, the end of which is provided with a puncture execution module of an ultrasonic probe. Not shown in the figure also includes a main control computer for control and processing and an ultrasonic device, which are connected with the mechanical arm and the puncture module and work cooperatively.

[0063] To achieve the above object, the application provides a thyroid puncture auxiliary system based on image scoring and path planning, which comprises:

[0064] An ultrasonic image acquisition module is used to continuously acquire a sequence of ultrasonic image frames and synchronously record the six-degree-of-freedom pose of the mechanical arm corresponding to each image frame during the scanning process of the mechanical arm driving the ultrasonic probe.

[0065] Specifically, preferably, referring to Figure 1 the physical structure of the system, the six-degree-of-freedom mechanical arm is controlled so that the 11.4MHz phased array ultrasonic probe carried at the end thereof performs transverse scanning along the neck region of the patient. The control process of the six-degree-of-freedom mechanical arm is based on the principle of robot inverse kinematics: the system first plans a scanning trajectory of the probe in the Cartesian space, the controller calculates the required rotation angles (θ1 to θ6) of the six joints of the mechanical arm for the space coordinate points on the trajectory in real time through inverse kinematics algorithm; then, pulse instructions are sent to the joint motors through a servo drive system, the motor outputs torque, and the position and speed closed-loop PID control is realized through encoder feedback, so that the end probe moves accurately and smoothly along the preset trajectory.

[0066] In this process, through the interface connection with the ultrasonic device, the mapping relationship between the mechanical arm and the ultrasonic probe is constructed, the ultrasonic image frame sequence can be collected in real time and continuously, and the six-degree-of-freedom pose of the mechanical arm corresponding to the shooting position of the current frame can be obtained correspondingly. At the same time, by reading the state information of the mechanical arm controller, the six-degree-of-freedom pose data of the mechanical arm at the exact moment when each frame of image is collected (i.e. the three-dimensional space position coordinates X, Y, Z of the tool center point TCP at the base coordinate system, and the three Euler angles: roll angle a, pitch angle β and yaw angle γ describing the spatial pose of the tool center point TCP, which are denoted as T WE , representing the transformation relationship of the end relative to the base) are recorded synchronously. Preferably, in an embodiment, the image frame rate of the collected image is suggested to be ≥10fps to ensure the continuity of the data stream and the pose synchronization accuracy. The collected image and pose data pair are cached.

[0067] The image processing module optimizes the collected ultrasonic image frames to obtain first image frames.

[0068] Specifically, in an embodiment, the optimization processing includes pre-processing such as cropping, denoising and enhancing on the ultrasonic image frames to highlight the target region and improve the accuracy of subsequent detection and the stability of the model. The specific pre-processing method can be set according to the actual situation, and the limitation of the present application is not limited thereto.

[0069] Specifically, in an embodiment, in order to adaptively extract the effective ultrasonic imaging region and eliminate the invalid background, the image processing module takes each frame of original ultrasonic image I raw as the processing object, and executes the following detailed algorithm function steps:

[0070] S21: the ultrasonic image frame pre-processing and double-threshold segmentation. First, the original ultrasonic image (the original ultrasonic image can be understood as a color image with three channels) I raw is converted into a gray image, and a Gaussian blur (kernel size 5x5) is applied to smooth the noise in the image. Then, a double-threshold segmentation strategy is adopted to generate a binary mask: the image after noise removal is subjected to double-threshold segmentation, a high threshold (such as 200) is applied in the first path to extract strong features, a low threshold (such as 30) is applied in the second path to extract weak features, and the results of the two paths are combined by performing a bitwise OR operation to obtain a combined initial mask. Finally, the combined initial mask is sequentially subjected to morphological closing operation (kernel size 9x9, filling the internal cavities of the region) and opening operation (removing edge noise), to generate a binary mask I bin with good connectivity.

[0071] S22: multi-level target screening. The system adopts a two-level strategy to lock the target region:

[0072] The first stage (geometric constraint screening): traverse all contours, calculate the aspect ratio and center position of the circumscribed rectangle of the contour. If there is a contour with an aspect ratio within a predetermined range (such as 0.8 to 1.2) and a center within the central region of the image (such as within 1 / 3 to 2 / 3 of the image width and height), it is marked as a target contour.

[0073] The second stage (maximum connected domain bottom-up): if the first stage screening does not find a contour that meets the conditions, the system automatically triggers the maximum connected domain extraction mechanism, directly calculates the pixel area of all contours, and selects the contour C with the largest area max as the final target ultrasound region.

[0074] S23: ROI secondary refinement and cropping. Perform primary cropping according to the circumscribed rectangle of the selected contour. To further remove the edge black frame, perform secondary threshold segmentation (such as threshold 10) and contour extraction within the primary cropped region, lock the largest connected region inside again and update the cropping coordinates (x, y, w, h), and finally output the pure ultrasound image I of the region of interest. roi .

[0075] S24: image enhancement. Finally, apply median filtering to suppress speckle noise, and apply histogram equalization algorithm to improve local contrast, output the final preprocessed image. roi .

[0076] The above image preprocessing process can be performed separately or in combination with multiple preprocessing optimization methods.

[0077] A target detection module is used to apply a pre-trained deep learning model to the preprocessed first image frame to automatically identify key anatomical structures in the image.

[0078] Specifically, the key anatomical structures include thyroid nodules, trachea, blood vessels, etc. In one embodiment, the target detection module further includes a temporal verification unit for receiving continuous frame detection results generated by the deep learning model and performing temporal consistency verification on the results. Only when a target structure is detected in at least M frames of the N frames, the target structure is confirmed as a stable target.

[0079] In one embodiment, the identification and confirmation of key anatomical structures are completed by the target detection module, which is divided into the following steps:

[0080] S31: deep learning target detection. The preprocessed first image frame is input into the deep learning model. In this embodiment, the YOLOv11 convolutional neural network architecture is used, which is based on the Ultralytics framework and fine-tuned for thyroid ultrasound data through transfer learning.

[0081] The model architecture is as followsFigure 2 As shown in the figure, the model architecture contains three core cascaded modules, each with the following functions and data flow:

[0082] Backbone: As a feature extractor, it adopts the structure of C3k2 module in series with SPPF (Spatial Pyramid Pooling-Fast) module. The input is the normalized ultrasound image (640x640), which is extracted layer by layer through the C3k2 module, and the end SPPF module is used to enhance the perception of different scale space information, outputting three different scale feature maps.

[0083] Neck: As a feature fusioner, it adopts PANet structure and integrates C2PSA (Cross Stage Partial Spatial Attention) attention mechanism. The input is the multi-scale feature map output by the Backbone, which is fused through the top-down and bottom-up paths (Upsample and Concat operation) to integrate deep semantic and shallow texture information, and the C2PSA module is used to further strengthen the key feature weight, outputting the aggregated high expression capability feature map.

[0084] Head: As a predictor, it adopts a multi-scale decoupled head. The input is the aggregated features output by the Neck, which are processed in parallel through independent classification branches (recognizing normal thyroid, nodules, etc.) and regression branches (precise positioning) at three different scales, finally outputting a prediction tensor containing target class probability and bounding box coordinates.

[0085] Transfer learning and training process: This system uses transfer learning technology to adapt to the medical ultrasound field. First, load the official weight file (such as yolo11.pt) pre-trained on the COCO large-scale general dataset, so that the model inherits the general edge and shape extraction ability. Then, perform structure adaptation, reset the output layer dimension of the Head, and modify the output channel to adapt to the specific number of classes of the system task. Finally, use the constructed 5098 labeled ultrasound image dataset (covering normal thyroid, thyroid nodules, trachea, blood vessels, etc. Key structures), use the backpropagation algorithm to fine-tune the detection head and all network parameters, so that the model can accurately lock the location of nodules and other tissues through the high confidence probability output by the classification branch.

[0086] S32, the original detection results of consecutive frames output by the YOLO model are checked for temporal consistency (anti-shake processing) to filter out detection flicker caused by ultrasound artifacts or probe micro-vibration. This process is built into the temporal verification unit, and the principle process is shown in Figure 3 , the following five algorithm steps are executed:

[0087] 1. Data normalization: input is the current frame raw detection set output by the YOLO model; function is to convert the normalized coordinates of the model's detection box into image pixel coordinates, and initialize the pending identity identifier for each detection box; output is the normalized current frame detection list.

[0088] 2. IoU association matching: input is the current frame detection list and the sliding window history queue maintaining the data of the last N frames (N=5 in this embodiment); function is to traverse the current frame detection box, calculate its intersection over union (IoU) with the last frame historical target, if IoU>0.5 and the categories are consistent, then inherit the unique ID of the historical target, otherwise assign a new global ID; output is the current frame target list with unique identity ID.

[0089] 3. Sliding window update: input is the current frame list after ID assignment; function is to use the first-in-first-out (FIFO) strategy to push the current frame data into the sliding window queue, and remove the earliest frame to keep the queue length constant; output is the updated historical state queue.

[0090] 4. Stability statistics: input is the updated historical state queue; function is to traverse the queue and count the cumulative number of occurrences of each unique ID within the N-frame window; output is a state table containing each target ID and its frequency statistics value.

[0091] 5. Threshold decision and output: input is the frequency state table; function is to apply a stability threshold (M=3 in this embodiment), only when the cumulative number of occurrences of an ID is greater than or equal to M, it is determined as a stable target and retained, otherwise it is removed as transient noise; output is the final stable target position information for subsequent scoring and path planning modules.

[0092] An image scoring module is configured to receive the position box information of the target anatomical structure in the key anatomical structure output by the target detection module, and calculate an image frame score according to a preset scoring function.

[0093] Specifically, in one embodiment, the target anatomical structure is a thyroid nodule structure, and the scoring function includes factors including thyroid nodule area, nodule center distance, detection confidence, obstacle penalty term, and position validity penalty term.

[0094] Specifically, in one embodiment, the image frame containing the valid "stable target" (corresponding to the stable anatomical structure) is quantitatively scored by the image scoring module.

[0095] Specifically, as shown in Figure 4 , the score is based on a preset five-factor scoring function:

[0096] Score = wA*A - wD*D + wC*C - O - I

[0097] wherein A is the nodule area score (calculated according to the bounding box of the stable target), D is the distance score of the nodule center from the image center, C is the nodule detection confidence score, O is the obstacle penalty term, and I is the nodule position validity penalty term. In this embodiment, the weight of each parameter and the penalty value are set as follows: w A = 1.0, w D = 0.5, w C = 0.5; the obstacle penalty term O = 100 (if the stable target contains a trachea / vessel and is too close to the potential path); the position invalidity penalty term I = 500 (if the nodule bounding box is mostly outside the thyroid profile). The comprehensive score Score of the current frame is calculated.

[0098] Through the above module functions, based on the recognition of key anatomical structures and quantitative scoring, the objectivity and accuracy of the puncture image frame selection are improved. By constructing an image scoring function based on multiple dimensions such as nodule area, center position, detection confidence, obstacle penalty term, and position validity, the system can automatically select the most suitable image frame for puncture, overcoming the subjectivity of manual frame selection and improving the consistency and accuracy of frame selection.

[0099] The path planning module selects the best reference image frame based on the scoring results, and automatically plans a puncture path that meets the preset puncture angle range in the best reference image frame, to obtain the optimal path and the pixel coordinates p of the key points of the optimal path on the two-dimensional image, and the custom path parameters C path corresponding to the optimal path.

[0100] Specifically, the preferred preset puncture angle range is preferably 30° to 45°, so as to ensure that the path avoids sensitive structures such as trachea and blood vessels; the position information of the stable target anatomical structure is the position information corresponding to the thyroid nodule.

[0101] Specifically, the path planning module selects the frame with the highest score as the best reference frame based on the scores of each frame calculated by the image scoring module. At the same time, it checks whether the highest score reaches a preset threshold (which can be set by an experienced physician according to the specific situation), if all frame scores are lower than the threshold, the system can automatically control the mechanical arm to perform a small pose adjustment (for example, rotating ±10° around the current position), and then returns to the image acquisition module to re-perform image acquisition and subsequent processing.

[0102] Specifically, in one embodiment, the automatic planning of the puncture path is performed according to the target position information corresponding to the best reference image frame, and this function step is subdivided into:

[0103] S61: candidate path generation. Specifically, starting from the nodule center identified in the stable target, multiple candidate puncture straight lines are drawn to the image upper edge (skin surface). These straight lines are constrained within an angle range of 30° to 45° with the image upper edge. Preferably, one candidate path line can be generated at every 5 degrees within this angle range.

[0104] S62: constraint checking and screening. Specifically, for each candidate path line generated in S61, obstacle avoidance checking is performed using the position information of obstacles (such as trachea, blood vessels) in the stable target. It is checked whether the path line overlaps with the obstacle region or is too close (for example, the nearest distance between the path line and the obstacle bounding box or contour is calculated, and it is required to be greater than a preset safety threshold, such as greater than 3mm). The candidate path line that does not meet the safety distance requirement is excluded.

[0105] S63: optimal path selection. Specifically, from all the valid candidate path lines checked in S62, the final puncture path is selected according to the preset optimization target (the effect is shown in Figure 5 (a), Figure 5 (b) is an enlarged view of the ultrasound image during the puncture process). In a preferred embodiment, the optimization target path is to select the path that is farthest from the nearest obstacle and has a relatively short path length. Two sets of data are finally output: 1) the pixel coordinates p of the key points of the selected optimal path on the two-dimensional image (for the mechanical arm positioning of the coordinate calibration module); 2) the custom path parameters C corresponding to the path (for example, puncture angle and depth, for the puncture module control of the puncture execution module). path

[0106] Through the functions of the above path planning module, the reliability of obstacle recognition is improved, and the path safety is significantly enhanced. The system can identify key structures such as thyroid nodules, trachea, blood vessels, etc. during path planning. The present application particularly introduces a time sequence checking unit, which can effectively filter out transient detection flicker (such as temporary disappearance of nodules) caused by artifacts or jitter in the ultrasound video stream, ensuring the stability and reliability of obstacle recognition, and then planning a puncture path that meets the angle constraint and avoids important tissues through the obstacle avoidance path algorithm, thereby fundamentally reducing the puncture risk caused by missed detection and false detection.

[0107] The coordinate calibration module is used to convert the pixel coordinates p of the key points of the optimal path in the image space generated by the path planning module on the two-dimensional image, using the six-degree-of-freedom pose data of the mechanical arm end recorded synchronously when the best reference image frame is collected, and combining the coordinate system transformation relationship calibrated in advance, to calculate the optimal path coordinates in the three-dimensional space recognizable by the mechanical arm control system through the coordinate system transformation relationship.

[0108] ​Specifically, the function of the coordinate calibration module can be understood as completing coordinate transformation from a two-dimensional image space coordinate system to a three-dimensional world space coordinate system by the coordinate calibration module.

[0109] Specifically, the coordinate system transformation relationship includes a hand-eye relationship and image physical parameters. The pre-calibrated hand-eye relationship refers to a fixed rigid transformation model of the ultrasound probe coordinate system {T} relative to the mechanical arm end coordinate system {E} (which integrates the offset of the probe to the clamping mechanism and the installation relationship of the clamping mechanism to the mechanical arm end), and the image physical parameters include a scale coefficient Scale between the image pixel coordinates and the actual physical space size (i.e., the physical millimeter number represented by a unit pixel). The detailed conversion process involves the following two steps:

[0110] 1) The system first converts the pixel coordinates p of the planning point on the two-dimensional image into physical coordinates in the probe coordinate system {T} by using a linear transformation formula p' = p * Scale according to the scale coefficient Scale obtained by calibration;

[0111] 2) Based on the kinematic chain of the robot, spatial mapping is applied by using the formula P = T WE *T HandEye *p'. Wherein, T WE is the real-time pose matrix of the mechanical arm end recorded synchronously when the image is collected, and T HandEye is a constant transformation matrix (determined by structural offset ΔX and other parameters) describing the relative position relationship between the probe and the mechanical arm end obtained by pre-calibration, so as to finally solve the three-dimensional coordinates P in the world coordinate system recognizable by the mechanical arm control system.

[0112] Specifically, in one embodiment, the principle flow can be as shown in Figure 6 , which is subdivided into:

[0113] S71: Conversion of the pixel coordinates P to the millimeter coordinates of the ultrasound device probe coordinate system. The pixel distance d of the nylon line with a known physical distance D is measured in the image, the calibrated image physical parameter Scale = D / d (millimeter / pixel), and the two-dimensional path pixel coordinates p are converted into three-dimensional millimeter coordinates p' in the probe coordinate system {T} (for example, p' can be assumed to be [p x *Scale,p y *Scale,0] T ).

[0114] S72: Conversion of the ultrasound device probe coordinate system to the world coordinate system.

[0115] Specifically, this step combines the coordinate system transformation principle shown in Figure 7 and the coordinate system transformation principle shown in Figure 8The structure shown in a, the millimeter coordinates p' obtained by the function step S71, expressed in the probe coordinate system {T} (with U1, U2, U3 axes) is converted to the world coordinate system {W}. The conversion process uses the following data:

[0116] 1) The six degrees of freedom pose T WE ( representing the transformation of the end of arm coordinate system {E} relative to the world coordinate system {W} ) recorded in step S1 synchronized with the optimal reference frame;

[0117] 2) The known jig-end relationship T EG ( describing the fixed installation pose of the puncture clamping mechanism coordinate system {G} relative to the end of arm coordinate system {E} ) ;

[0118] 3) The three constant offsets (ΔX1, ΔX2, ΔX3) describing the simplified transformation relationship T GT of {T} relative to {G} are obtained in advance based on the establishment of the degrees of freedom decoupled clamping mechanism coordinate system {G} (with X1, X2, X3 axes) and the probe coordinate system {T} (with U1, U2, U3 axes) by the specific calibration method of the present application, as shown in Figure 7 .

[0119] The coordinate transformation is carried out in the following order: first, the coordinates p' in the ultrasound device probe coordinate system {T} are converted to the coordinates p" in the clamping mechanism coordinate system {G} using the constant offsets obtained by calibration. Due to the special design of the coordinate system, degrees of freedom decoupling is achieved, and this step can be simplified to directly apply the offsets. Finally, the coordinates p'" in the end of arm coordinate system {E} are converted to the world coordinate system {W} using the real-time pose T WE , obtaining the final target three-dimensional space coordinates P = [P x , P y , P z ] T .

[0120] The coordinate system conversion in the above embodiment is a relatively fine process. The present application can also directly use the key point matching in the ultrasound image frame to the three-dimensional coordinate system to establish a conversion matrix to realize the conversion of the two-dimensional coordinate system to the three-dimensional coordinate system. The coordinate conversion method of the present application is not limited to this, and other methods known in the art can also be used.

[0121] The puncture execution module includes an angle adjusting mechanism, a puncture needle device, and a controller, which is used to adjust the puncture posture and perform the puncture operation according to the received three-dimensional coordinates and the custom path parameters C path .

[0122] Specifically, in one embodiment, the process of performing the puncture operation can specifically perform the following functions.

[0123] S81: The mechanical arm is positioned. The system controls the six-degree-of-freedom mechanical arm to move to a corresponding puncture preparation position according to the target three-dimensional coordinates, and then the mechanical arm stops running.

[0124] S82: The puncture instruction conversion is performed to convert the coordinates into control instructions.

[0125] Specifically, this step performs the instruction conversion of the puncture module. The controller in the system directly maps and calculates the control instructions (such as the motion step number or the encoder target value) required by the respective motors (33, 34, 35) of the puncture module through a pre-calibrated function relationship or a lookup table (i.e., the calibration module of the puncture module) according to the received custom path parameters C path , such as the planned angle and depth value.

[0126] S83: The puncture execution module controls its angle adjustment mechanism to accurately adjust the incident angle of the puncture needle to the angle planned by the path planning module.

[0127] Specifically, as shown in FIG. 8, Figure 8 wherein Figure 8 (a) is a front view of the puncture execution module, 8(b) is a back isometric view of the puncture execution module, 8(c) is a side isometric view of the puncture execution module, and 8(d) is a top view of the puncture execution module. The motor 33 drives the gear 315 to slide on the track, drives the mounting plate 37 fixed on the slider 310, and can adjust the position of point A, which controls the puncture position. The adjustment motor 34 drives the gear 314 to slide on the track, drives the mounting plate 38 fixed on the slider 311, and forms the required angle between the puncture needle fixed on the mounting plate 38 and the puncture position. Figure 7

[0128] S84: The puncture needle is inserted into the skin.

[0129] Specifically, as shown in FIG. 8, Figure 8 (c), after the angle is adjusted in place, the puncture advance mechanism is controlled to accurately and stably advance the puncture needle along the fixed posture at a speed of 1-2 mm / s to the puncture target point E by controlling the motor 35 to drive the gear 316.

[0130] In step S9, if the score in the image scoring module is insufficient or the path planning in the path planning module fails, the system can automatically control the mechanical arm to perform a small posture adjustment (for example, rotating ±10° around the current position), and then return to the image acquisition module to reacquire images and perform subsequent processing.

[0131] In addition, the present application is provided with a system control module for coordinating the above-mentioned functional modules to form a complete closed-loop process from image analysis to puncture control. ​

[0132] Through the functions of various modules in the system, the degree of automation and dynamic accuracy of puncture positioning are improved. The system supports continuous scoring, path planning and action execution in real-time ultrasound video stream. By combining the six-degree-of-freedom mechanical arm pose recorded synchronously and the pre-calibrated hand-eye relationship, the present application realizes high-precision dynamic mapping from 2D image path to 3D mechanical arm space. This method overcomes the limitations of traditional static calibration, ensures the accurate reach of automatic scanning, frame selection, positioning and needle insertion operations, reduces manual intervention and improves operation efficiency.

[0133] Embodiment two

[0134] Based on the above system, the present application provides a thyroid puncture assisting method based on the aforementioned image scoring and path planning based thyroid puncture assisting system, which comprises the following steps:

[0135] S1, controlling the mechanical arm to carry out transverse scanning with the ultrasonic probe, and collecting ultrasonic image frames and corresponding six-degree-of-freedom pose information of the mechanical arm end in real time;

[0136] S2, pre-processing and optimizing the collected image frames, extracting the maximum connected domain and removing invalid information at the image edge;

[0137] S3, calling a pre-trained target detection network (preferably YOLOv11) to detect the key structure in the image, and checking the detection results of the continuous frames through a time sequence checking unit to confirm the stable key anatomical structure;

[0138] S4, constructing a scoring function based on the position box information of the target anatomical structure in the key anatomical structure, and calculating the scores of each image frame accordingly;

[0139] S5, screening out the image frame with the highest score as the optimal reference frame for path planning;

[0140] S6, in the reference frame, combining the distribution of the stable target, planning a puncture path that meets the angle constraint range and does not overlap with the trachea or blood vessels;

[0141] S7, using the six-degree-of-freedom pose data of the mechanical arm end recorded synchronously when collecting the optimal reference frame, combining the pre-calibrated hand-eye relationship and image physical parameters, and converting the path pixel coordinates into three-dimensional path points in the mechanical arm coordinate system through coordinate system transformation;

[0142] S8, the puncture execution module controls the puncture mechanism to adjust the posture according to the path coordinates and angle information, and drives the puncture needle to perform puncture operation along the path;

[0143] S9, if the current image frame score is lower than the preset threshold, the system controls the mechanical arm end to deflect by a certain angle to reacquire the image and returns to step S1 until the score meets the standard.

[0144] Through the above steps, the automation degree and dynamic accuracy of puncture positioning are improved. The system supports continuous scoring, path planning and action execution in real-time ultrasound video stream. By combining the six-degree-of-freedom mechanical arm pose recorded synchronously and the pre-calibrated hand-eye relationship, the application realizes high-precision dynamic mapping from 2D image path to 3D mechanical arm space. The method overcomes the limitations of traditional static calibration, ensures the accurate reach of automatic scanning, frame selection, positioning and needle insertion operations, reduces manual intervention and improves operation efficiency.

[0145] Embodiment three

[0146] The application further provides another embodiment, and the application provides a thyroid puncture auxiliary device, which comprises a processor 1 and a memory 2.

[0147] The memory 2 is used for storing a computer program.

[0148] The memory 2 comprises various media capable of storing program codes, such as ROM, RAM, a disk, a U disk, a memory card or an optical disk.

[0149] The processor 1 is connected with the memory 2 and is used for executing the computer program stored in the memory 2, so that the processor realizes the thyroid puncture auxiliary method described above when executing the computer program.

[0150] Preferably, the processor 1 can be a central processing unit (CPU) and can also be an application specific integrated circuit (ASIC).

[0151] Embodiment four

[0152] The application further provides another embodiment, that is, a computer readable storage medium is provided, the computer readable storage medium stores a computer program, and the computer program can be executed by at least one processor to enable the at least one processor to execute the steps of the thyroid puncture auxiliary method described above.

[0153] In the above embodiments, the description of each embodiment has its own focus, and the parts not described or recorded in detail in a certain embodiment can be referred to the relevant description of other embodiments.

[0154] Those skilled in the art can understand that the modules, units and / or method steps of each embodiment described in combination with the embodiments disclosed herein can be implemented by electronic hardware, or a combination of computer software and electronic hardware. Whether the functions are performed in hardware or software depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of the present application.

[0155] In several embodiments provided by the present application, it should be understood that the disclosed devices, apparatuses and methods can be implemented in other ways. For example, the above-described device embodiments are merely illustrative, for example, the division of the modules is only a logical function division, and actual implementation can have another division manner, for example, a plurality of modules or components can be combined or integrated into another device or system, or some features can be ignored or not executed.

[0156] In addition, each functional module in each embodiment of the present application can be integrated into a processing module, or each module can exist physically independently, or two or more modules can be integrated into one module. The above integrated unit can be realized in the form of hardware or in the form of a software functional unit.

[0157] The above-described embodiments are merely used to illustrate the technical solutions of the present application, rather than limit the present application; although the present application has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that the technical solutions recorded in the foregoing embodiments can be modified, or some technical features can be replaced equivalently; and such modifications or replacements do not cause the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of the present application.

Claims

1. A thyroid biopsy assistance system based on image scoring and path planning, characterized in that, The system includes the following steps: The ultrasonic image acquisition module is used to continuously acquire ultrasonic image frame sequences during the scanning process of the ultrasonic probe driven by the robotic arm, and synchronously record the six-degree-of-freedom pose of the robotic arm corresponding to each frame image. The image processing module optimizes the acquired ultrasound image frames to obtain the first image frame; The target detection module is used to automatically identify key anatomical structures in the image by applying a pre-trained deep learning model to the pre-processed first image frame. The image scoring module is used to receive the target anatomical structure and location box information in the key anatomical structure output by the target detection module, and calculate the image frame score according to the preset scoring function. The path planning module selects the best reference image frame based on the scoring results and automatically plans a puncture path within the best reference image frame that meets the preset puncture angle range. This yields the optimal path, the pixel coordinates p of the key points of the optimal path on the 2D image, and the corresponding custom path parameters C. path ; The coordinate calibration module is used to calculate the three-dimensional spatial coordinates of the optimal path pixel coordinates p in the two-dimensional image by using the six-degree-of-freedom pose data of the end effector of the robot arm that was synchronously recorded when the optimal reference image frame was acquired, and by combining the pre-calibrated coordinate system transformation relationship. The puncture execution module includes an angle adjustment mechanism, a puncture needle insertion device, and a controller, used to execute the puncture based on the received three-dimensional coordinates and the custom path parameter C. path Adjust the puncture position and perform the puncture procedure.

2. The thyroid biopsy assistance system based on image scoring and path planning according to claim 1, characterized in that, The target detection module is used for, The preprocessed first image frame is input into the deep learning model, and the output is the raw detection results of consecutive frames; Based on the original detection results of the consecutive frames, a temporal consistency check is performed on the original detection results of the consecutive frames to obtain the stable key anatomical structure.

3. The thyroid biopsy assistance system based on image scoring and path planning according to claim 2, characterized in that, The scoring function includes factors such as thyroid nodule area, distance from the center of the nodule, detection confidence, obstacle penalty, and location validity penalty. The preset scoring function is: Score=w A *A-w D *D+w C *C-O-I Where A is the nodule area score, D is the distance score between the nodule center and the image center, C is the nodule detection confidence score, O is the obstacle penalty, and I is the nodule location validity penalty; the weights and penalties for each parameter are set as follows: w A =1.0,w D =0.5,w C =0.5; Obstacle penalty O=100; Invalid position penalty I=500; Calculate the overall score for the current frame.

4. The thyroid biopsy assistance system based on image scoring and path planning according to any one of claims 1-3, characterized in that, The preset puncture angle range is preferably 30° to 45°.

5. The thyroid biopsy assistance system based on image scoring and path planning according to claim 4, characterized in that, The path planning module selects the frame with the highest score from the image scoring module as the best reference frame; and... The path planning module is specifically used for, Starting from the center of the nodule identified in the key anatomical structure, multiple candidate puncture straight lines are drawn out to the edge of the skin surface on the image to obtain several candidate paths; For each candidate path, obstacle avoidance verification is performed using the obstacle location information in the key anatomical structure. The optimal path is the one that is furthest from the nearest obstacle and has a relatively short path length.

6. The thyroid biopsy assistance system based on image scoring and path planning according to claim 5, characterized in that, The puncture execution module specifically includes: The system controls a six-degree-of-freedom robotic arm to move it to the corresponding puncture preparation position based on the target's three-dimensional coordinates. The puncture command is converted into a control command, transforming the target's three-dimensional coordinates into control commands. The puncture execution module controls its angle adjustment mechanism to precisely adjust the incident angle of the puncture needle to the optimal path C planned by the path planning module. path .

7. A thyroid biopsy assistance method, applied to the aforementioned image-based scoring and path planning-based thyroid biopsy assistance system, the method comprising the following steps: S1. Control the robotic arm to carry the ultrasonic probe to perform a lateral scan, and collect ultrasonic image frames and corresponding six-degree-of-freedom pose information of the robotic arm end effector in real time. S2. Preprocess and optimize the acquired image frames, extract the largest connected component, and remove invalid information from image edges; S3. Call the pre-trained target detection network to detect key anatomical structures in the image, and verify the detection results of consecutive frames through the temporal verification unit to confirm the stable key anatomical structures. S4. Construct a scoring function based on the location box information of the target anatomical structure in the key anatomical structure, and calculate the score of each image frame accordingly; S5. Select the image frame with the highest score as the optimal reference frame for path planning; S6. In the optimal reference frame, combined with the distribution of the optimal reference frames of the stable key anatomical structures, plan a puncture path that satisfies the angle constraint range and does not overlap with the obstacle anatomical structure. S7. Using the six-degree-of-freedom pose data of the robotic arm end effector that was synchronously recorded when the optimal reference frame was acquired, and combined with the pre-calibrated coordinate system transformation relationship, the path pixel coordinates were converted into three-dimensional path points in the robotic arm coordinate system through the coordinate system transformation relationship. S8. The puncture execution module controls the puncture mechanism to adjust its posture based on the path coordinates and angle information, and drives the puncture needle to perform the puncture operation along the path. S9. If the current image frame score is lower than the preset threshold, the system controls the end of the robotic arm to deflect at a certain angle to reacquire the image and return to step S1 until the score meets the standard.

8. A thyroid biopsy auxiliary device, characterized in that, It includes a memory and one or more programs, wherein one or more programs are stored in the memory and configured to be executed by one or more processors, wherein the one or more programs include steps for performing the thyroid biopsy-assisted method as described in claim 7.

9. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores a computer program that, when executed by a processor, implements the steps of the thyroid biopsy-assisted method as described in claim 7.

Images