Method and system for controlling a robotic arm to scan the thyroid gland based on ultrasonic image feedback

Abstract

The present invention discloses a method and system for controlling a robotic arm to scan the thyroid gland based on ultrasonic images. The method includes: S1. Use a robotic arm with ultrasound to perform a transverse parallel scan from top to bottom on one side of the patient's neck, record continuous images M and the spatial coordinate position P of each frame; S2. Use a preset segmentation deep neural network to extract the thyroid position of each frame in the continuous images M; S3. Control the robotic arm to move the probe to the spatial position Ps1 corresponding to the starting frame Ms where the thyroid appears, and use the inclination angle as the initial position towards the head, and perform a sector scan with the probe contact surface as the center to a position perpendicular to the neck; S4. With the position Ps3 and posture at the end of step S3, control the robotic arm to move horizontally along the neck by a physical distance D to the spatial position Ps2 corresponding to the frame Pe where the thyroid disappears; S5. Control the robotic arm to perform a sector scan from a position perpendicular to the neck to a position with an inclination angle towards the chest with the probe contact surface as the center. The present invention ensures the integrity of the scan.

Description

Technical Field

[0001] The present invention relates to the field of ultrasonic technology, and more specifically, to a method and system for controlling a robotic arm to scan the thyroid gland based on ultrasonic image feedback. Background Art

[0002] Currently, when using a robotic arm to ultrasonically scan a patient's thyroid gland, it is generally carried out by means of a fixed scanning trajectory or a preset scanning trajectory. However, the position of the thyroid gland varies among individuals, resulting in incomplete scanning of the preset trajectory and prone to problems such as missed scanning and too many invalid images. Therefore, it is necessary to develop a method and system for controlling a robotic arm to scan the thyroid gland based on ultrasonic image feedback. Summary of the Invention

[0003] The purpose of the present invention is to provide a method and system for controlling a robotic arm to scan the thyroid gland based on ultrasonic image feedback to overcome the defects existing in the prior art.

[0004] To achieve the above purpose, the technical solution adopted by the present invention is as follows:

[0005] A method for controlling a robotic arm to scan the thyroid gland based on ultrasonic image feedback includes the following steps:

[0006] S1. Horizontally scan the patient's neck from top to bottom in a cross-section parallel manner with a robotic arm equipped with an ultrasonic device, record continuous images M and the spatial coordinate position P of each frame;

[0007] S2. Use a preset segmentation deep neural network to extract the position of the thyroid gland in each frame of the continuous image M, calculate the area S of the thyroid gland in each frame, and count the starting frame M of the appearance of the thyroid gland s and the corresponding spatial position P s1 , the frame P when the thyroid gland disappears e and the corresponding spatial position P s2 , and the frame M with the largest area of the thyroid gland m ;

[0008] S3. Control the robotic arm to move the probe to the spatial position P s corresponding to the starting frame M of the appearance of the thyroid gland s1 , and use the inclination angle θ1 as the initial position towards the head, and perform fan scanning with the contact surface of the probe as the center until it reaches a position perpendicular to the neck;

[0009] S4. With the position P s3 and posture at the end of step S3, control the robotic arm to horizontally move along the neck a physical distance D to scan to the spatial position P e corresponding to the frame P when the thyroid gland disappears s2 ;

[0010] S5, controlling the robotic arm to sweep from the vertical position of the neck to the position with an inclination angle θ2 toward the chest with the contact surface of the ultrasound probe as the center of the circle, and then ending the scanning.

[0011] Furthermore, the step S2 further includes: calculating the maximum thyroid area frame M m Thyroid thickness max , calculate the starting frame M where the thyroid gland appears s and the thyroid gland disappears frame P e The physical distance D between the upper part of the thyroid gland and the closest distance H to the skin surface min .

[0012] Furthermore, the training steps of the segmentation deep neural network preset in step S2 include:

[0013] S20, select a standard segmentation network;

[0014] S21, adjust the weight W of the standard segmentation network training layer Set to 1 and train to overfitting;

[0015] S22, randomly select N features in each layer of the standard segmentation network, and adjust the training weights W layer Set it to 0, perform inference on the validation set to obtain the accuracy, repeat M times to select the parameter with the highest accuracy;

[0016] S23, repeat step S22 until each layer W layer 1 and the number of features is reduced to half of the original, and W is cut off layer Features that are 0;

[0017] S24, train the segmentation network again until the accuracy of the validation set no longer improves;

[0018] S25. Repeat steps S22-S24 until the number of features reaches a predetermined target.

[0019] Furthermore, the thyroid area in step S2 is set to the number of pixels of the thyroid in the image.

[0020] Furthermore, the standard segmentation network adopts Unet or Deeplab.

[0021] Furthermore, the calculation formula of the inclination angle θ1 in step S3 is:

[0022]

[0023] Where, 5°≤θ1≤45°, α1=0.5, T max The maximum frame M in the thyroid area m The thickness of the thyroid gland, Hmin It is the closest distance from the upper part of the thyroid gland to the skin surface.

[0024] Furthermore, the calculation formula for the inclination angle θ2 is:

[0025]

[0026] In the formula, 15° ≤ θ2 ≤ 45°, α2 = 0.65, T max is the inner and outer diameter thickness of the thyroid gland at the frame M with the largest thyroid gland area m , and H min is the closest distance from the upper part of the thyroid gland to the skin surface.

[0027] The present invention also provides a system according to the above method for controlling the robotic arm to scan the thyroid gland based on ultrasonic image feedback, including:

[0028] A transverse parallel scanning module that uses a robotic arm with ultrasound to perform transverse parallel scanning from top to bottom on one side of the patient's neck, records continuous images M and the spatial coordinate position P of each frame;

[0029] An extraction and calculation module that uses a preset segmentation deep neural network to extract the position of the thyroid gland in each frame of the continuous image M, calculates the area S of the thyroid gland in each frame, and counts the starting frame M s of the appearance of the thyroid gland and the corresponding spatial position P s1 , the frame P e when the thyroid gland disappears and the corresponding spatial position P s2 , as well as the frame M m with the largest thyroid gland area;

[0030] A first control module for controlling the robotic arm to move the probe to the spatial position P s corresponding to the starting frame M s1 of the appearance of the thyroid gland, and starting from the probe contact surface as the center, fanning to a position perpendicular to the neck at an inclination angle θ1;

[0031] A second control module, with the position P s3 and posture at the end of the first control module, controls the robotic arm to move horizontally along the neck by a physical distance D to scan to the spatial position P e corresponding to the frame P s2 when the thyroid gland disappears;

[0032] A third control module for controlling the robotic arm to fan from a position perpendicular to the neck to an inclination angle θ2 position towards the chest with the ultrasound probe contact surface as the center, and then end the scan.

[0033] Compared with the prior art, the advantages of the present invention are as follows: The present invention first obtains a preliminary image through a robotic arm pre-scan, analyzes the image information feedback, calculates the trajectory of a detailed scan plan, and completes the fine scanning process. Compared with the traditional method of a fixed scan trajectory, the present invention realizes a closed loop between the robotic arm scan and the image validity, ensuring the integrity of the scan. BRIEF DESCRIPTION OF THE DRAWINGS

[0034] In order to more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the following will briefly introduce the drawings required for use in the description of the embodiments or the prior art. Obviously, the drawings in the following description are only some embodiments of the present invention, and those of ordinary skill in the art can also obtain other drawings based on these drawings without creative efforts.

[0035] Figure 1 is a flowchart of a method for controlling a robotic arm to scan the thyroid based on ultrasonic image feedback according to the present invention.

[0036] Figure 2 is a training flowchart of a segmentation deep neural network in the present invention.

[0037] Figure 3 is a flowchart of a system for controlling a robotic arm to scan the thyroid based on ultrasonic image feedback according to the present invention. DETAILED DESCRIPTION OF THE EMBODIMENTS

[0038] The following will elaborate on the preferred embodiments of the present invention in conjunction with the drawings, so that the advantages and features of the present invention can be more easily understood by those skilled in the art, thereby making a clearer and more definite definition of the protection scope of the present invention.

[0039] Refer to Figure 1 As shown, this embodiment discloses a method for controlling a robotic arm to scan the thyroid based on ultrasonic image feedback, including the following steps:

[0040] Step S1: Use a robotic arm with ultrasound to perform a horizontal parallel scan from top to bottom on one side of the patient's neck, and record the continuous image M and the spatial coordinate position P of each frame.

[0041] Step S2: Use a preset segmentation deep neural network to extract the thyroid position in each frame of the continuous image M, calculate the area S of the thyroid in each frame, and count the starting frame M s of the appearance of the thyroid and the corresponding spatial position P s1 , the frame P e when the thyroid disappears and the corresponding spatial position P s2 , and the frame M m with the largest thyroid area.

[0042] Specifically, it further includes: calculating in the frame Mm Thyroid thickness max , calculate the starting frame M where the thyroid gland appears s and the thyroid gland disappears frame P e The physical distance D between the upper part of the thyroid gland and the closest distance H to the skin surface min .

[0043] Specifically, the standard segmentation network uses Unet or Deeplab.

[0044] Segment the thyroid gland and other parts. The open source network is biased towards one side between performance and efficiency, that is, the larger the model, the better the accuracy, but the lower the efficiency. In this embodiment, a segmentation network based on random parameter descent iteration is specifically proposed to balance performance and time. The characteristic of this network is that the parameters of each layer need to be multiplied by the training adjustment weight W layer By controlling W during training layer To achieve network slimming. Compared with general pruning and distillation methods to shrink the network, this method has high accuracy, stable convergence, and fast training speed.

[0045] Specifically, Figure 2 As shown, the training steps of the preset segmentation deep neural network include:

[0046] Step S20, select a standard segmentation network, taking the standard Unet as an example, the feature number per layer is [64, 128, 256, 512, 1024]. The feature number of the target segmentation network is [8, 16, 32, 64, 128] per layer.

[0047] Step S21: adjust the training weights W of the standard segmentation network layer Set to 1 and train until overfitting occurs, that is, the training set loss no longer decreases and the validation set accuracy no longer improves.

[0048] Step S22: Randomly select N features from each layer of the standard segmentation network and adjust the training weights W layer Set it to 0, perform inference on the validation set to obtain the accuracy, repeat M times and select the parameter with the highest accuracy.

[0049] Step S23, repeat step S22 until each layer W layer 1 and the number of features is reduced to half of the original, and W is cut off layer Features that are 0;

[0050] Step S24, training the segmentation network again until the accuracy of the validation set no longer improves;

[0051] Step S25, repeat steps S22-S24 until the number of features reaches a predetermined target.

[0052] Specifically, the thyroid area can be approximately counted by the number of pixels of the thyroid in the image.

[0053] Then, when planning the scanning path, scan from top to bottom, which is divided into three consecutive scans, namely steps S3 - S5.

[0054] In step S3, the first stage is the corner - entry scan. Control the robotic arm to move the probe to the starting frame M s of the corresponding spatial position P s1 where the thyroid appears, and use the inclination angle θ1 as the initial position towards the head, and perform a sector scan with the contact surface of the probe as the center until it reaches a position perpendicular to the neck.

[0055] Among them, the calculation formula for the inclination angle θ1 is:

[0056]

[0057] In the formula, 5° ≤ θ1 ≤ 45°, α1 = 0.5, T max is the thickness of the inner - outer diameter of the thyroid at the frame M m when the thyroid area is the largest, and H min is the closest distance from the upper part of the thyroid to the skin surface.

[0058] In step S4, the second stage is the flat scan. Control the robotic arm to move horizontally along the neck by a physical distance D from the position P s3 and attitude at the end of step S3 until it scans to the frame P e corresponding to the spatial position P s2 where the thyroid disappears;

[0059] In step S5, the third stage is the corner - exit scan. Control the robotic arm to perform a sector scan from the position perpendicular to the neck with the contact surface of the ultrasonic probe as the center to the position of the inclination angle θ2 towards the chest, and then end the scan.

[0060] Among them, the calculation formula for the inclination angle θ2 is:

[0061]

[0062] In the formula, 15° ≤ θ2 ≤ 45°, α2 = 0.65, T max is the thickness of the inner - outer diameter of the thyroid at the frame M m when the thyroid area is the largest, and H min is the closest distance from the upper part of the thyroid to the skin surface.

[0063] Refer to Figure 3As shown, the present invention also provides a system according to the above-mentioned method of controlling the robotic arm to scan the thyroid gland based on ultrasound image feedback, comprising: a transverse parallel scanning module 1, which uses a robotic arm with ultrasound to perform a transverse parallel scanning from top to bottom on one side of the patient's neck, and records the continuous image M and the spatial coordinate position P of each frame; an extraction calculation module 2, which uses a preset segmentation deep neural network to extract the thyroid gland position of each frame in the continuous image M, and calculates the area S of the thyroid gland in each frame, and counts the starting frame M where the thyroid gland appears. s and the corresponding spatial position P s1 , the frame where the thyroid gland disappears e and the corresponding spatial position P s2 , and the maximum frame M of the thyroid area m The first control module 3 is used to control the robot arm to move the probe to the starting frame M where the thyroid gland appears s The corresponding spatial position P s1 The first control module 4 is a control module that is located at the position P at the end of the first control module. s3 and posture, control the robotic arm to move horizontally along the neck physical distance D to scan the frame P where the thyroid gland disappears e The corresponding spatial position P s2 The third control module 5 is used to control the robotic arm to sweep from the vertical position of the neck to the position with an inclination angle θ2 toward the chest with the contact surface of the ultrasound probe as the center, and then end the scan.

[0064] The present invention obtains a preliminary image through a pre-scan by a robotic arm, analyzes the image information feedback and calculates the trajectory of a detailed scanning plan to complete the fine scanning process. Compared with the traditional method of fixing the scanning trajectory, the present invention realizes a closed loop of robotic arm scanning and image validity, ensuring the integrity of the scanning.

[0065] Although the embodiments of the present invention are described in conjunction with the accompanying drawings, the patent owner may make various variations or modifications within the scope of the appended claims. As long as they do not exceed the protection scope described in the claims of the present invention, they should be within the protection scope of the present invention.

Claims

1. A method for controlling a robotic arm to scan the thyroid gland based on ultrasonic image feedback, characterized in that, The following steps are involved: S1. Perform a transverse parallel scan from top to bottom on one side of the patient's neck using a robotic arm equipped with ultrasound, and record continuous images M and the spatial coordinate position P of each frame; S2. Use a preset segmentation deep neural network to extract the thyroid position in each frame of the continuous image M, calculate the area S of the thyroid in each frame, and count the starting frame M of the appearance of the thyroid s and the corresponding spatial position P s1 , the frame P where the thyroid disappears e and the corresponding spatial position P s2 , and the frame M with the largest thyroid area m ; S3. Control the robotic arm to move the probe to the starting frame M where the thyroid appears s The corresponding spatial position P s1 , and use the inclination angle θ1 as the initial position towards the head, and perform a sector scan with the probe contact surface as the center to the position perpendicular to the neck; S4. At the position P at the end of step S3 s3 and posture, control the robotic arm to horizontally move along the neck by a physical distance D to scan to the frame P where the thyroid gland disappears e at the corresponding spatial position P s2 ; S5, controlling the robotic arm to sweep from the vertical position of the neck to the position with an inclination angle θ2 toward the chest with the contact surface of the ultrasound probe as the center of the circle, and then ending the scanning.

2. The method for controlling a robotic arm to scan the thyroid gland based on ultrasonic image feedback according to claim 1, characterized in that, The step S2 further includes: calculating the inner-outer diameter thickness T of the thyroid gland at the frame M with the largest thyroid area m ; calculating the starting frame M max of the appearance of the thyroid gland and the physical distance D between the frame P s where the thyroid gland disappears, and calculating the closest distance H from the upper part of the thyroid gland to the skin surface e . min .

3. The method for controlling a robotic arm to scan the thyroid gland based on ultrasonic image feedback according to claim 1, characterized in that, The training steps of the segmentation deep neural network preset in step S2 include: S20, select a standard segmentation network; S21. Set the training adjustment weight W of the standard segmentation network to 1 and train it until overfitting; layer ​ S22. Randomly select N features from each layer of the standard segmentation network, set the training adjustment weight W layer to 0, perform inference on the validation set to obtain the accuracy, and repeat the selection M times to obtain the parameter with the highest accuracy; S23. Repeat step S22 until each layer W layer is 1 and the number of features is reduced to half of the original, and prune the features with W layer being 0; S24, train the segmentation network again until the accuracy of the validation set no longer improves; S25. Repeat steps S22-S24 until the number of features reaches a predetermined target.

4. The method for controlling a robotic arm to scan the thyroid gland based on ultrasonic image feedback according to claim 1, characterized in that, The thyroid area in step S2 is set to the number of pixels of the thyroid in the image.

5. The method for controlling a robotic arm to scan the thyroid gland based on ultrasonic image feedback according to claim 3, characterized in that, The standard segmentation network adopts Unet or Deeplab.

6. The method for controlling a robotic arm to scan the thyroid gland based on ultrasonic image feedback according to claim 1, characterized in that, The calculation formula of the inclination angle θ1 in step S3 is: Where 5°≤θ1≤45°, α1 = 0.5, T max is the inner and outer diameter thickness of the thyroid gland at the frame M m when the thyroid area is the largest, and H min is the closest distance from the upper part of the thyroid gland to the skin surface.

7. The method for controlling a robotic arm to scan the thyroid gland based on ultrasonic image feedback according to claim 3, characterized in that, The calculation formula of the inclination angle θ2 is: where 15°≤θ2≤45°, α2 = 0.65, T max is the inner and outer diameter thickness of the thyroid gland at the frame M with the largest thyroid area m and H min is the closest distance from the upper part of the thyroid gland to the skin surface.

8. A system for the method of controlling a robotic arm to scan the thyroid gland based on ultrasonic image feedback according to any one of claims 1-7, characterized in that, include: The transverse parallel scanning module uses a robotic arm with ultrasound to perform a transverse parallel scan from top to bottom on one side of the patient's neck, recording continuous images M and the spatial coordinate position P of each frame; An extraction calculation module uses a preset segmentation deep neural network to extract the thyroid position of each frame in the continuous image M, calculate the area S of the thyroid in each frame, and count the starting frame M when the thyroid appears s and the corresponding spatial position P s1 , the frame P when the thyroid disappears e and the corresponding spatial position P s2 , and the frame M with the largest thyroid area m ; The first control module is used to control the robotic arm to move the probe to the starting frame M where the thyroid appears s to the corresponding spatial position P s1 , and take the inclination angle θ1 as the initial position towards the head, and perform sector scanning with the probe contact surface as the center to the position perpendicular to the neck; The second control module, at the position P at the end of the first control module s3 and attitude, controls the robotic arm to horizontally move along the neck by a physical distance D to scan to the frame P where the thyroid gland disappears e corresponding spatial position P s2 ; The third control module is used to control the robot arm to sweep from the vertical position of the neck to the position with an inclination angle θ2 toward the chest with the contact surface of the ultrasound probe as the center, and then end the scanning.

Images