A Trajectory Planning Method for Ultrasonic Robots Based on Human Key Feature Localization

Abstract

This application discloses an ultrasonic robot trajectory planning method based on human body key feature localization, belonging to the field of automatic control technology. The method includes: detecting a human body in an image acquired by a global camera and outputting a detection box for the human body; predicting the position of key human body features based on the detection box; the key human body features include the human neck; controlling the ultrasonic probe in the ultrasonic robot to move to the position of the human neck, and then calculating the pose of the ultrasonic probe as the initial pose; calculating the desired pose of the ultrasonic probe at the key human body features, and then controlling the ultrasonic probe to move from the initial pose to the desired pose; and performing trajectory planning for the ultrasonic probe from the initial scanning position when it is in the desired pose according to the key human body features. This application uses human body key features to perform trajectory planning for the ultrasonic probe, which can adjust the pose of the ultrasonic probe during the scanning process and improve the scanning accuracy of the ultrasonic probe.

Description

Technical Field

[0001] This application relates to the field of automatic control technology, and in particular to an ultrasonic robot trajectory planning method based on the localization of key human body features. Background Technology

[0002] Ultrasound robots are a product of the combination of traditional medicine and artificial intelligence. They are generally composed of an ultrasound robotic arm, ultrasound equipment, a global camera, force sensors, etc. They can gradually replace ultrasound physicians in performing autonomous scanning and process the ultrasound images obtained in real time to assist ultrasound physicians in diagnosis and treatment.

[0003] In order for ultrasound robots to scan and acquire images of target skin areas in the human body, it is necessary to locate the anatomical structure of the target tissue. Some studies use robotic arm teaching for localization, while others register magnetic resonance imaging (MRI) or computed tomography (CT) images with ultrasound images to locate the target tissue with the help of other medical images. However, the first method has too little autonomy and relies too much on human operation; the second method is more expensive, as it requires MRI or CT operations while performing ultrasound scanning, which contradicts the low cost of ultrasound diagnosis and treatment. Summary of the Invention

[0004] The main objective of this application is to propose an ultrasonic robot trajectory planning method based on the localization of key human body features, so as to improve the scanning accuracy of ultrasonic robot scanning.

[0005] To achieve the above objectives, one aspect of this application proposes an ultrasonic robot trajectory planning method based on the localization of key human body features, the method comprising the following steps: Detect human bodies in images acquired by the global camera and output the detection bounding box for the human body; The location of key human features is predicted based on the detection box; wherein, the key human features include the human neck; The ultrasonic probe in the ultrasonic robot is controlled to move to the position of the human neck, and then the pose of the ultrasonic probe is calculated as the initial pose. The desired pose of the ultrasound probe at the key features of the human body is calculated, and then the ultrasound probe is controlled to move from the initial pose to the desired pose. Based on the key human body features, the ultrasound probe is trajectory planned from the initial scanning position; wherein, the position of the ultrasound probe when it is in the desired pose is taken as the initial scanning position.

[0006] In some embodiments, detecting human bodies in images acquired by a global camera and then outputting a detection bounding box for the human body includes the following steps: ResNet is used as the detection backbone network, and the feature map of the image is extracted using the detection backbone network as the detection feature map; Using FPN as the detection neck network, the detection neck network is used to perform feature fusion and enhancement on the detection feature map to obtain a multi-scale feature map; Using Faster R-CNN as the detection head network, candidate regions are generated on the multi-scale feature map using the RPN in the detection head network. Then, the features of the candidate regions are extracted through the Align operation, and the features of the candidate regions are classified and the detection box is regressed to obtain the detection box of the human body.

[0007] In some embodiments, predicting the location of key human features based on the detection box; wherein the key human features include the human neck, includes the following steps: Using ResNet as the prediction backbone network, features are extracted from the sub-images corresponding to the detection boxes in the image to obtain the prediction feature map. Using FMP as the prediction neck network, the prediction feature map is selected, stitched together, and scaled using the prediction neck network to convert it into a format suitable for the prediction head network, thus obtaining the converted feature map. The prediction head network is used to output the key human features in the predicted feature map and the transformed feature map as corresponding heatmaps; wherein, the value of each heatmap is used to reflect the confidence of the corresponding key human feature at each position on the corresponding feature map; Based on the heatmap, a decoded image including the key human body features is generated, and then the location of the key human body features is predicted based on the decoded image.

[0008] In some embodiments, the calculation of the ultrasonic probe's pose as the initial pose includes the following steps: The target normal vector at the position passing through the neck of the human body is calculated and used as the y-axis of the ultrasound probe; Based on the aforementioned key human features, the feature points of the left and right shoulders of the human body are determined. Connect the feature points of the left and right shoulders of the human body; The straight line parallel to the line connecting the two points and passing through the neck of the human body is defined as the x-axis of the ultrasound probe. The y-axis and x-axis of the ultrasonic probe are used as the initial pose.

[0009] In some embodiments, calculating the target normal vector passing through the neck of the human body as the y-axis of the ultrasound probe includes the following steps: Multiple non-collinear target points are selected within a preset range of the human neck position, and each target point is located in the region of the human neck. The depth information of each target point is obtained using the global camera; A plane passing through the neck of the human body is formed based on the depth information; The target normal vector is calculated based on the plane and the position of the human neck and used as the y-axis of the ultrasound probe.

[0010] In some embodiments, calculating the desired pose of the ultrasound probe at the key human feature includes the following steps: The quadratic programming problem is as follows: ; in, For the decision variables to be optimized, It is a symmetric positive definite quadratic matrix. A linear coefficient vector, Used to describe inequality constraints Used to describe equality constraints Used for constraints Size; The quadratic programming problem is discretized based on the boundary conditions of the robotic arm, the end-effector pose of the robotic arm, the obstacle avoidance constraints, and the joint angle constraints of the robotic arm, resulting in a trajectory optimization problem. The trajectory optimization problem is: ; in, For the joint angle variable to be optimized, Used to describe obstacle constraints. Used to describe boundary constraints. Used to describe joint angle constraints; The desired joint angles of each joint in the robotic arm are obtained by solving the trajectory optimization problem. The desired pose is determined based on each desired joint angle.

[0011] In some embodiments, the trajectory planning of the ultrasound probe from the initial scanning position based on the key human body features includes the following steps: Multiple scanning points are obtained by scanning and sampling along the same direction starting from the initial scanning position, and the scanning trajectory is obtained by using polynomial interpolation. The desired probe pose of the ultrasonic probe at each of the scanning points is calculated based on the depth information of the points acquired by the global camera. The ultrasonic robot is controlled to perform the scanning operation based on the desired probe pose. Rotate the ultrasound probe 90° and return to the step of scanning and sampling multiple scanning points along the same direction starting from the initial scanning position, and using polynomial interpolation to obtain the scanning trajectory; During the ultrasound probe scanning process, a random walk algorithm is used to solve the confidence level of pixels in the image captured by the global camera; when the confidence level of pixels in any region is lower than a set threshold, the pose of the ultrasound probe is adjusted. During the ultrasound probe scanning process, the image captured by the global camera is segmented using the nnU-Net model to obtain the thyroid region; the pose of the ultrasound probe is dynamically adjusted based on the segmented thyroid region.

[0012] To achieve the above objectives, another aspect of this application proposes an ultrasonic robot trajectory planning device based on the localization of key human body features, the device comprising: The human detection unit is used to detect human bodies in images acquired by the global camera and then output the detection bounding box of the human body; A feature prediction unit is used to predict the location of key human body features based on the detection box; wherein, the key human body features include the human neck. The pose calculation unit is used to control the ultrasonic probe in the ultrasonic robot to move to the position of the human neck, and then calculate the pose of the ultrasonic probe as the initial pose. The probe moving unit is used to calculate the desired pose of the ultrasound probe at the key feature of the human body, and then control the ultrasound probe to move from the initial pose to the desired pose. The trajectory planning unit is used to plan the trajectory of the ultrasound probe from the initial scanning position based on the key human body features; wherein the position of the ultrasound probe when it is in the desired pose is taken as the initial scanning position.

[0013] To achieve the above objectives, another aspect of this application provides an electronic device, which includes a memory and a processor. The memory stores a computer program, and the processor executes the computer program to implement the above-described method.

[0014] To achieve the above objectives, another aspect of the embodiments of this application proposes a computer-readable storage medium storing a computer program that, when executed by a processor, implements the above-described method.

[0015] The embodiments of this application include at least the following beneficial effects: This application can detect human bodies in images acquired by a global camera and output a detection bounding box for the human body; predict the positions of key human body features based on the detection bounding box; wherein the key human body features include the human neck; control the ultrasonic probe in the ultrasonic robot to move to the position of the human neck, and then calculate the pose of the ultrasonic probe as the initial pose; calculate the desired pose of the ultrasonic probe at the key human body features, and then control the ultrasonic probe to move from the initial pose to the desired pose; perform trajectory planning for the ultrasonic probe from the initial scanning position based on the key human body features; wherein the position of the ultrasonic probe when it is in the desired pose is used as the initial scanning position. This application performs trajectory planning for the ultrasonic probe using key human body features, which can adjust the pose of the ultrasonic probe during the scanning process and improve the scanning accuracy of the ultrasonic probe. Attached Figure Description

[0016] To more clearly illustrate the technical solutions in the embodiments of this application, the accompanying drawings used in the description of the embodiments will be briefly introduced below. Obviously, the accompanying drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0017] Figure 1 A flowchart illustrating an ultrasonic robot trajectory planning method based on the localization of key human body features, provided for an embodiment of this application; Figure 2 This is a schematic diagram of the MDH coordinate system of the ultrasonic robotic arm provided in the embodiments of this application; Figure 3 This is a schematic diagram of the kinematic model of the ultrasonic robot provided in the embodiments of this application; Figure 4 A structural diagram of the human key feature recognition and localization model provided in the embodiments of this application; Figure 5 A framework diagram for scanning trajectory planning provided in the embodiments of this application; Figure 6 A schematic diagram of the structure of an ultrasonic robot trajectory planning device based on the localization of key human body features, provided for an embodiment of this application; Figure 7 This is a schematic diagram of the hardware structure of an electronic device provided in an embodiment of this application. Detailed Implementation

[0018] To make the objectives, technical solutions, and advantages of this application clearer, the following detailed description is provided in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative of this application and are not intended to limit it. In the following description, when referring to the accompanying drawings, unless otherwise indicated, the same numbers in different drawings represent the same or similar elements. The embodiments described in the following exemplary embodiments do not represent all embodiments consistent with those of this application; they are merely examples of apparatuses and methods consistent with some aspects of the embodiments of this application as detailed in the appended claims.

[0019] It is understood that the terms “first,” “second,” etc., used in this application may be used herein to describe various concepts, but unless otherwise stated, these concepts are not limited by these terms. These terms are only used to distinguish one concept from another. For example, without departing from the scope of the embodiments of this application, first information may also be referred to as second information, and similarly, second information may also be referred to as first information. Depending on the context, the words “if,” “when,” or “in response to a determination” as used herein may be interpreted as “when…” or “when…” or “in response to a determination.”

[0020] As used in this application, the terms "at least one", "multiple", "each", "any", etc., "at least one" includes one, two or more, "multiple" includes two or more, "each" refers to each of the corresponding multiples, and "any" refers to any one of the multiples.

[0021] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this application belongs. The terminology used herein is for the purpose of describing embodiments of this application only and is not intended to limit this application.

[0022] Before providing a detailed description of the embodiments of this application, some related technologies involved in the embodiments of this application will be described first, as follows: After locating the target tissue in the human body, corresponding trajectory planning is equally important. Trajectory planning for ultrasound robots consists of two parts: the first part is trajectory planning from the initial position of the ultrasound robotic arm to the location of the target tissue; the second part is trajectory planning within the target tissue area to scan and obtain the complete structure of the human tissue, while ensuring patient comfort during the scanning process. For the first part of trajectory planning, the patient's posture needs to be fully considered, and the movement trajectory of the ultrasound robotic arm needs to be constrained to ensure that the robotic arm does not collide with the human body during movement. For the second part of trajectory planning, many studies have reconstructed the target tissue area using point cloud cameras, defining multiple planning points in the reconstructed area and connecting these points into a planned path. However, in actual scanning, due to the diversity of human skin surface structures and the errors inherent in point cloud cameras, scanning along a predefined path may not completely cover the target tissue. Furthermore, some ultrasound images may exhibit low image quality. Therefore, the posture of the ultrasound probe and the planned path need to be adjusted in real time.

[0023] Reference Figure 1 This application provides a method for trajectory planning of an ultrasonic robot based on the localization of key human body features. This method may include, but is not limited to, steps S100 to S140, as follows: S100: Detects human bodies in images acquired by the global camera and outputs the human body detection box; S110: Predict the location of key human features based on the detection box; wherein, the key human features include the human neck; S120: Control the ultrasonic probe in the ultrasonic robot to move to the position of the human neck, and then calculate the pose of the ultrasonic probe as the initial pose. S130: Calculate the desired pose of the ultrasound probe at the key feature of the human body, and then control the ultrasound probe to move from the initial pose to the desired pose; S140: Based on the key human body features, the ultrasound probe is trajectory planned from the initial scanning position; wherein, the position of the ultrasound probe when it is in the desired pose is taken as the initial scanning position.

[0024] Optionally, the step of detecting human bodies in images acquired by the global camera and then outputting detection bounding boxes for the human bodies includes the following steps: ResNet is used as the detection backbone network, and the feature map of the image is extracted using the detection backbone network as the detection feature map; Using FPN as the detection neck network, the detection neck network is used to perform feature fusion and enhancement on the detection feature map to obtain a multi-scale feature map; Using Faster R-CNN as the detection head network, candidate regions are generated on the multi-scale feature map using the RPN in the detection head network. Then, the features of the candidate regions are extracted through the Align operation, and the features of the candidate regions are classified and the detection box is regressed to obtain the detection box of the human body.

[0025] Optionally, predicting the location of key human features based on the detection box; wherein the key human features include the human neck, includes the following steps: Using ResNet as the prediction backbone network, features are extracted from the sub-images corresponding to the detection boxes in the image to obtain the prediction feature map. Using FMP as the prediction neck network, the prediction feature map is selected, stitched together, and scaled using the prediction neck network to convert it into a format suitable for the prediction head network, thus obtaining the converted feature map. The prediction head network is used to output the key human features in the predicted feature map and the transformed feature map as corresponding heatmaps; wherein, the value of each heatmap is used to reflect the confidence of the corresponding key human feature at each position on the corresponding feature map; Based on the heatmap, a decoded image including the key human body features is generated, and then the location of the key human body features is predicted based on the decoded image.

[0026] Optionally, the calculation of the ultrasonic probe's pose as the initial pose includes the following steps: The target normal vector at the position passing through the neck of the human body is calculated and used as the y-axis of the ultrasound probe; Based on the aforementioned key human features, the feature points of the left and right shoulders of the human body are determined. Connect the feature points of the left and right shoulders of the human body; The straight line parallel to the line connecting the two points and passing through the neck of the human body is defined as the x-axis of the ultrasound probe. The y-axis and x-axis of the ultrasonic probe are used as the initial pose.

[0027] Optionally, calculating the target normal vector passing through the neck of the human body as the y-axis of the ultrasound probe includes the following steps: Multiple non-collinear target points are selected within a preset range of the human neck position, and each target point is located in the region of the human neck. The depth information of each target point is obtained using the global camera; A plane passing through the neck of the human body is formed based on the depth information; The target normal vector is calculated based on the plane and the position of the human neck and used as the y-axis of the ultrasound probe.

[0028] Optionally, the calculation of the desired pose of the ultrasound probe at the key human body feature includes the following steps: The quadratic programming problem is as follows: ; in, For the decision variables to be optimized, It is a symmetric positive definite quadratic matrix. A linear coefficient vector, Used to describe inequality constraints Used to describe equality constraints Used for constraints Size; The quadratic programming problem is discretized based on the boundary conditions of the robotic arm, the end-effector pose of the robotic arm, the obstacle avoidance constraints, and the joint angle constraints of the robotic arm, resulting in a trajectory optimization problem. The trajectory optimization problem is: ; in, For the joint angle variable to be optimized, Used to describe obstacle constraints. Used to describe boundary constraints. Used to describe joint angle constraints; The desired joint angles of each joint in the robotic arm are obtained by solving the trajectory optimization problem. The desired pose is determined based on each desired joint angle.

[0029] Optionally, the step of planning the trajectory of the ultrasound probe from the initial scanning position based on the key human body features includes the following steps: Multiple scanning points are obtained by scanning and sampling along the same direction starting from the initial scanning position, and the scanning trajectory is obtained by using polynomial interpolation. The desired probe pose of the ultrasonic probe at each of the scanning points is calculated based on the depth information of the points acquired by the global camera. The ultrasonic robot is controlled to perform the scanning operation based on the desired probe pose. Rotate the ultrasound probe 90° and return to the step of scanning and sampling multiple scanning points along the same direction starting from the initial scanning position, and using polynomial interpolation to obtain the scanning trajectory; During the ultrasound probe scanning process, a random walk algorithm is used to solve the confidence level of pixels in the image captured by the global camera; when the confidence level of pixels in any region is lower than a set threshold, the pose of the ultrasound probe is adjusted. During the ultrasound probe scanning process, the image captured by the global camera is segmented using the nnU-Net model to obtain the thyroid region; the pose of the ultrasound probe is dynamically adjusted based on the segmented thyroid region.

[0030] The following section will provide a detailed introduction and explanation of the solutions in the embodiments of this application, using specific application examples.

[0031] This embodiment discloses an ultrasound robot trajectory planning method based on key human body features. Ultrasound robots can gradually replace ultrasound physicians in autonomously performing ultrasound scans. This autonomy is primarily reflected in the ultrasound robot's ability to autonomously locate the tissue region to be scanned and control the ultrasound probe to autonomously complete the scanning operation, acquiring high-quality ultrasound images for better subsequent diagnosis and treatment. The method in this embodiment locates the tissue region to be scanned based on key human body features. Simultaneously, a quadratic programming method is used to initially plan the scanning trajectory, and the trajectory is continuously refined by combining the confidence map of the ultrasound image and real-time segmentation results, thereby continuously improving the quality of the acquired ultrasound images.

[0032] 1. Robot kinematics model.

[0033] 1.1 Kinematic model of the robotic arm.

[0034] The kinematic model of an ultrasonic robotic arm is a mathematical model that describes the pose of the end effector and the relationship between the joints of the robotic arm. It consists of two parts: the forward kinematic model and the inverse kinematic model.

[0035] The forward kinematics model of an ultrasonic robotic arm describes how to solve the pose of the end effector in a base coordinate system given each joint angle of the robotic arm. The ultrasonic robotic arm used in this embodiment is 6-DOF (degrees of freedom). The forward kinematics model is derived using the MDH (Mean Distance Hierarchy) method, establishing 6 MDH coordinate systems at the joints, such as... Figure 2 As shown, where, The MDH coordinate system represents a 6-DOF robotic arm. It's joint offset. It is the length of the connecting rod.

[0036] Let the joint angles of the 6-DOF robotic arm be... The connecting rod torsion angle is , for coordinate system to Homogeneous transformation matrix between coordinate systems ,but Represented as: (1) According to the chain rule, the forward kinematic model of the ultrasonic robotic arm is as follows: (2) The inverse kinematics model of the ultrasonic robotic arm describes how to solve for the joint angles of the robotic arm given the target pose of the end effector in the base coordinate system. Since the ultrasonic robotic arm in this embodiment satisfies the Pieper criterion (i.e., the three adjacent joint axes of the robotic arm intersect at a single point), a closed-form solution can be calculated using inverse kinematics. In this embodiment, the last three joint axes of the ultrasonic robotic arm intersect at a single point, i.e., the coordinate system... , and If the three axes intersect at a point, then the homogeneous coordinates of that point in the robot arm's base coordinate system can be expressed as: (3) From equation (1), we can see that Substituting equation (1) into equation (3), we can obtain about The expression, known The coordinates can then be obtained accordingly. The coordinate values ​​are then listed. The equation for the sum of squares is solved first by using the method of variable substitution. Then solve Finally, solve The axes of the last three joints of the robotic arm intersect at a single point. The rotation angles of these three joints affect the end effector's posture, which can be determined from the rotation matrix representing the end effector's posture. Solve for .

[0037] 1.2 System kinematic model.

[0038] The ultrasound robot system consists of a robotic arm, an ultrasound probe, a probe connector, a six-dimensional force sensor, and a global camera. The robotic arm is a 6-DOF (degrees of freedom) robotic arm; the ultrasound probe is a linear probe used for imaging superficial tissues and blood vessels, operating in B-mode, which converts the echo intensity of the ultrasound waves emitted by the probe into a grayscale image; the probe connector is manufactured using 3D printing; the six-dimensional force sensor is used to quickly and accurately measure the force and torque on the x, y, and z axes of the sensor; the global camera is an RGB-D camera used to detect the pose of the robotic arm's end effector and reconstruct the surface of the human tissue to be scanned, providing visual input information for the robot's trajectory planning.

[0039] Perform kinematic modeling of the entire ultrasonic robot system, such as Figure 3 As shown, where, Let be the robot's base coordinate system. Let this be the robot's end-effector coordinate system. For the robot's marked coordinate system, Let be the image coordinate system of the ultrasound probe. This is the coordinate system of the global camera. A QR code is used to determine the relationship between the global camera and the ultrasonic robotic arm. A six-dimensional force sensor measures the force and torque experienced by the ultrasonic probe during scanning. The feedback force further controls the force and torque applied by the ultrasonic robot, ensuring patient comfort. It is important to note that the six-dimensional force sensor requires gravity compensation before application to remove the gravity acting on the ultrasonic probe and its mounting components, reducing the impact of this gravity and thus more accurately controlling the force and torque applied at the end effector. Based on the forward kinematics model of the robotic arm, the following can be calculated in real time: By capturing the QR code with a global camera, extracting information from the four corner points of the QR code, and using the PNP algorithm to solve it, real-time results can be obtained. Through multi-objective calibration methods, we can obtain... This ultimately constitutes the kinematic model of the entire system.

[0040] 2. Methods for locating key human body features.

[0041] 2.1 Identification and localization of key human body features.

[0042] When an ultrasound robot performs autonomous scanning, it is necessary to first determine the initial scanning pose of the ultrasound probe, that is, to move the ultrasound probe to the skin surface of the human tissue to be scanned before proceeding with subsequent scanning and image acquisition operations.

[0043] The ultrasound robot proposed in this embodiment is mainly used to scan the thyroid tissue of the human body, thereby assessing the structure, size, shape, and lesions of the thyroid gland. The thyroid tissue is located in the lower part of the neck and consists of two lobes and an isthmus. The neck is one of the key features of the human body. This embodiment utilizes this key feature to roughly locate the anatomical structure of the thyroid gland, thereby guiding the ultrasound robot to move the ultrasound probe to a designated position as the starting point for subsequent ultrasound scanning operations.

[0044] To achieve the identification and localization of key features at the human neck, this embodiment employs a deep learning approach, using a trained neural network model to locate these key features. This localization model uses a top-down method, first detecting the human body in the image acquired by the global camera and outputting a bounding box for the human body. Then, based on this bounding box, the position of the human neck is predicted, thereby achieving the identification and localization of key human features.

[0045] The structure of the localization model is as follows Figure 4 As shown, it is divided into three modules: data processing module, detection module and prediction module. The input is video frames captured by the global camera, and the output is an image containing key features, which can realize the real-time recognition and localization of key features.

[0046] The data processing module mainly processes video frames acquired by the global camera, performing operations such as normalization and channel order transformation of the video frames, and outputs preprocessed images.

[0047] In the detection module, the input is the image output from the data processing module, and the final output is an image containing detection boxes. For the detection backbone network, ResNet is used as the backbone to perform feature extraction on the input image. For the detection neck network, FPN is used as the neck network to perform feature fusion and enhancement on the feature map output from the detection backbone network. The introduction of the detection neck network enables the model to achieve better results when handling both small and large targets, while also providing more unified and semantically rich features for the subsequent head network. For the detection head network, the input is the multi-scale features output from the neck network. Faster R-CNN is used as the head network, and RPN is used to generate candidate regions on the feature map. Then, the Align operation is used to extract the features corresponding to the candidate regions, and more refined classification and detection box regression are performed. Finally, the data is decoded to obtain an image containing detection boxes, which is then input into the prediction module. The DCN enhancement operator is introduced into the detection backbone network, neck network, and head network to enhance the feature extraction capabilities of the modules.

[0048] In the prediction module, only the bounding boxes of the image are used as input to predict key features. For the prediction backbone network, ResNet is used as the backbone network to extract features from the bounding boxes. For the prediction neck network, FMP is used as the neck network, and non-parametric transformations such as selection, concatenation, and scaling are used to transform the feature maps output by the prediction backbone network into a format suitable for the prediction head network. For the prediction head network, the feature maps processed by the prediction backbone and neck networks are converted into the final pose estimation result, i.e., the location prediction of key features. It outputs a heatmap corresponding to each key feature, where the value of each heatmap reflects the confidence of the corresponding key feature at each location in the image. Finally, the image containing the key features is decoded, providing the location information of the key features for subsequent operations.

[0049] The positioning model used in this embodiment can obtain the position information of the human neck. Since the thyroid tissue is located below the human neck, the position of the human thyroid tissue can be roughly estimated using the obtained neck position information.

[0050] When the human body is positioned sideways to the global camera, with only the left side of the neck facing the camera, the key features predicted by the localization model correspond to points in the left lobe region of the thyroid gland. When only the right side of the neck faces the camera, the key features predicted by the localization model correspond to points in the right lobe region of the thyroid gland. Therefore, when the human body is positioned sideways to the global camera, the ultrasound robot can obtain the positional information of one lobe of the thyroid gland, thereby guiding the ultrasound probe to scan that lobe.

[0051] When the human body is facing the global camera, the key features predicted by the localization model deviate from the points in the isthmus region of the thyroid tissue. However, this deviation only exists on a certain axis. Therefore, it is only necessary to determine the direction of this axis and guide the ultrasound probe along this axis until the isthmus region of the thyroid tissue is identified in the ultrasound image, thus locating the thyroid tissue. After the ultrasound probe moves to the skin surface of the isthmus region of the thyroid tissue, longitudinal movement can scan the entire isthmus region, while lateral movement can move to the left and right lobe regions of the thyroid tissue, and longitudinal movement can scan the entire lobe region within the left and right lobe regions.

[0052] 2.2 Ultrasonic probe attitude calculation.

[0053] Having identified and located key human body features, the initial target position for ultrasound probe scanning can be obtained. The next step is to determine the probe's target orientation, i.e., the known target orientation. The position of the origin of the coordinate system, solution coordinate axes of the coordinate system .

[0054] This embodiment uses the normal vector method to determine the initial posture of the ultrasound probe during scanning. Given the initial target position, it is necessary to calculate the target normal vector passing through that point, i.e. This normal vector is perpendicular to the neck region of the human body, thus guiding the ultrasound probe perpendicular to the neck region and obtaining a cross-sectional ultrasound image of the neck area. To calculate... In this embodiment, several non-collinear points near the target location are selected, and these points are all located in the neck region of the human body. Using the depth information of these points acquired by a global camera, a plane passing through the target location is formed. Finally, based on the plane Given the information and target location point information, the target normal vector can be calculated, i.e. .

[0055] Solving for the results After knowing one axis of the initial scanning posture of the ultrasound probe, it is necessary to determine the other axis. Only then can the initial posture information be determined. To better align the probe posture during ultrasonic robot scanning with that during manual ultrasonic manipulation, and considering the structural characteristics of the human body, this embodiment incorporates key features of the human shoulders, using these features in conjunction with key features of the neck to determine... The identification and localization of key features of the shoulders are also achieved through deep learning methods. The basic structure of the localization model is... Figure 4 Similarly, a localization model is obtained by training on a dataset labeled with key features of the shoulders. This model identifies the key feature points of the human shoulders, and the feature points of the left and right shoulders can form a straight line. Based on the structural characteristics of the human body, Set as with A given straight line parallel to the coordinate axes passing through the target location point. With the target location information, it is possible to calculate... At the same time, located within the plane of the neck region and... The vertical axis passing through the target location is the deviation axis mentioned in the identification and localization of key human features. The ultrasound probe can locate the isthmus region of the thyroid tissue from the ultrasound image by moving along this deviation axis.

[0056] 3. Trajectory planning method.

[0057] 3.1 Initial trajectory planning method.

[0058] After calculating the desired pose of the ultrasound robot at key human features, initial trajectory planning is required to control the ultrasound probe from its initial pose to the desired pose, thus enabling subsequent ultrasound scanning operations. This embodiment employs a quadratic programming method to achieve the initial trajectory planning of the ultrasound robot.

[0059] Quadratic programming is a nonlinear programming method whose objective function is a quadratic function, and its solution is as follows: (4) in, For the decision variables to be optimized, It is a symmetric positive definite quadratic matrix. A linear coefficient vector, Used to describe inequality constraints Used to describe equality constraints Used for constraints Size.

[0060] This embodiment describes the joint angle of the ultrasonic robotic arm. As a decision variable, and given the initial values ​​of the joint angles, while the expected values ​​of the joint angles can be calculated using the inverse kinematics model of the ultrasonic robotic arm based on the positioning information of key human features, then... , ,in, The start time, If the termination time is specified, then the boundary conditions can be defined as follows: (5) Simultaneously, it is necessary to ensure that the ultrasonic robotic arm does not collide with the human body during movement. Since the ultrasonic probe needs to move to the neck area, in actual operation, if only simple trajectory planning is used, the end effector of the ultrasonic robot, i.e., the ultrasonic probe, may collide with the shoulder or face area of ​​the human body. Therefore, this embodiment again utilizes human key feature recognition and localization methods to introduce obstacle avoidance constraints for the ultrasonic robot. The trained model is used to locate the key features of the shoulder and face areas of the human body, obtaining a set of coordinate points. ,in, Simultaneously, the end-effector pose of the ultrasonic robotic arm is defined. Then it needs to satisfy: (6) in, To maintain a safe distance, a linearization method is used to approximate the obstacle avoidance constraint as follows: (7) Since each joint of the ultrasonic robotic arm has an angle limitation, it can be defined as follows: (8) Discretize problem (4) and set the time... Discretize into Given discrete time intervals and constraints (5), (7), and (8), the trajectory optimization problem can be reduced to: (9) Finally, the initial trajectory of the ultrasonic robot can be solved. A desired joint angle is determined, thereby guiding the ultrasound probe to the desired position.

[0061] 3.2 Scan trajectory planning method.

[0062] After the ultrasound robot guides the ultrasound probe along the initial trajectory to the initial scanning position, the scanning trajectory planning begins. The scanning trajectory planning method proposed in this embodiment is divided into two modules that operate simultaneously. For module 1, the human tissue region to be scanned is reconstructed based on images acquired by the global camera, and scanning points are manually set within the reconstructed region. The desired pose of the ultrasound probe corresponding to each scanning point can be obtained through ultrasound probe pose calculation. For module 2, while the ultrasound robot scans along the trajectory obtained in the first step, adjustments are made in real time based on the ultrasound confidence map and ultrasound image segmentation results to ensure that the scan obtains clear and complete thyroid tissue. The overall framework of the scanning trajectory planning method is as follows: Figure 5 As shown.

[0063] In Module 1, based on the depth information of the human tissue region obtained by the global camera, multiple pre-scanning points are sampled, and the corresponding trajectory is obtained using polynomial interpolation. The initial pose and longitudinal scanning direction of the ultrasound probe in the target tissue region are determined by the human key feature localization method. Using the key points obtained in the human key feature localization method as starting points, other scanning points are sampled along the longitudinal scanning direction, and the corresponding trajectory is obtained using polynomial interpolation. Finally, based on the depth information of the points obtained by the global camera, the desired probe pose of the sampled scanning points can be calculated, thereby enabling the ultrasound robot to perform simple scanning operations, such as longitudinal scanning of the left and right lobes and isthmus of the human thyroid tissue, obtaining the corresponding horizontal ultrasound images. If the ultrasound probe is rotated 90° and a similar operation is performed, a transverse scan can be performed to obtain the corresponding sagittal ultrasound images. However, due to the diversity of human skin structure, the scanning trajectory formed by module 1 may not be able to completely cover the entire thyroid tissue. At the same time, due to the inherent error in the depth information acquired by the global camera, the pose calculated based on the depth information may not be the optimal pose. That is, under the calculated pose, the ultrasound probe may not be able to scan a sufficiently clear ultrasound image or an ultrasound image that covers a relatively complete thyroid tissue. Therefore, module 2 is required to make real-time adjustments.

[0064] In module 2, this embodiment utilizes feedback from the ultrasound confidence map to adjust the ultrasound probe's orientation in real time, ensuring a sufficiently clear ultrasound image. Simultaneously, it uses feedback from the ultrasound image organ segmentation results to adjust the probe's orientation in real time, ensuring the obtained ultrasound image covers a relatively complete thyroid tissue. To obtain clearer ultrasound images for subsequent diagnosis and treatment, this embodiment introduces an ultrasound confidence map to evaluate image quality. The ultrasound confidence map uses a random walk algorithm to calculate the confidence score of each pixel. A higher confidence score for a pixel indicates higher image quality. Initially, the pixel closest to the ultrasound probe has a confidence score of 1, the farthest pixel has a confidence score of 0, and other pixels have confidence scores in between. When the confidence score of a certain area of ​​the ultrasound image is low, the ultrasound robot controls the ultrasound probe to adjust its orientation accordingly to improve the overall confidence score of the ultrasound image, thereby obtaining a clearer ultrasound image. In addition to acquiring clear ultrasound images, it is also necessary to ensure that the ultrasound images can cover a relatively complete thyroid tissue so that each frame of the ultrasound image contains more effective information. Therefore, this embodiment uses the nnU-Net model to segment the image and separate the thyroid region in the ultrasound image. During the ultrasound robot scanning process, the pose of the ultrasound probe is adjusted according to the real-time segmentation results to ensure that the segmented thyroid tissue is located in the central region of the ultrasound image, thereby increasing the effective information of the ultrasound image.

[0065] Reference Figure 6 This application also provides an ultrasonic robot trajectory planning device based on human key feature localization, which can realize the above-mentioned ultrasonic robot trajectory planning method based on human key feature localization. The device includes: The human detection unit is used to detect human bodies in images acquired by the global camera and then output the detection bounding box of the human body; A feature prediction unit is used to predict the location of key human body features based on the detection box; wherein, the key human body features include the human neck. The pose calculation unit is used to control the ultrasonic probe in the ultrasonic robot to move to the position of the human neck, and then calculate the pose of the ultrasonic probe as the initial pose. The probe moving unit is used to calculate the desired pose of the ultrasound probe at the key feature of the human body, and then control the ultrasound probe to move from the initial pose to the desired pose. The trajectory planning unit is used to plan the trajectory of the ultrasound probe from the initial scanning position based on the key human body features; wherein the position of the ultrasound probe when it is in the desired pose is taken as the initial scanning position.

[0066] It is understood that the content of the above method embodiments is applicable to the present device embodiments. The specific functions implemented by the present device embodiments are the same as those of the above method embodiments, and the beneficial effects achieved are also the same as those achieved by the above method embodiments.

[0067] This application also provides an electronic device, which includes a memory and a processor. The memory stores a computer program, and the processor executes the computer program to implement the method of this application. This electronic device can be any smart terminal, including tablet computers, in-vehicle computers, etc.

[0068] It is understood that the content of the above method embodiments is applicable to the device embodiments. The specific functions implemented by the device embodiments are the same as those of the methods of this application, and the beneficial effects achieved are the same as those achieved by the methods of this application.

[0069] Please see Figure 7 , Figure 7 The hardware structure of an electronic device according to another embodiment is illustrated. The electronic device includes: The processor 701 can be implemented using a general-purpose CPU (Central Processing Unit), microprocessor, application-specific integrated circuit (ASIC), or one or more integrated circuits, and is used to execute relevant programs to implement the technical solutions provided in the embodiments of this application. The memory 702 can be implemented as a read-only memory (ROM), a static storage device, a dynamic storage device, or a random access memory (RAM). The memory 702 can store the operating system and other application programs. When the technical solutions provided in the embodiments of this specification are implemented through software or firmware, the relevant program code is stored in the memory 702 and is called and executed by the processor 701. The input / output interface 703 is used to implement information input and output; The communication interface 704 is used to enable communication and interaction between this device and other devices. Communication can be achieved through wired means (such as USB, Ethernet cable, etc.) or wireless means (such as mobile network, WIFI, Bluetooth, etc.). Bus 705 transmits information between various components of the device (e.g., processor 701, memory 702, input / output interface 703, and communication interface 704); The processor 701, memory 702, input / output interface 703, and communication interface 704 are connected to each other within the device via bus 705.

[0070] This application also provides a computer-readable storage medium storing a computer program that, when executed by a processor, implements the method of this application.

[0071] It is understood that the content of the above method embodiments is applicable to this storage medium embodiment. The specific functions implemented in this storage medium embodiment are the same as those in the above method embodiments, and the beneficial effects achieved are also the same as those achieved in the above method embodiments.

[0072] Memory, as a non-transitory computer-readable storage medium, can be used to store non-transitory software programs and non-transitory computer-executable programs. Furthermore, memory may include high-speed random access memory, and may also include non-transitory memory, such as at least one disk storage device, flash memory device, or other non-transitory solid-state storage device. In some embodiments, memory may optionally include memory remotely located relative to the processor, and these remote memories can be connected to the processor via a network. Examples of such networks include, but are not limited to, the Internet, intranets, local area networks, mobile communication networks, and combinations thereof.

[0073] The embodiments described in this application are for the purpose of more clearly illustrating the technical solutions of the embodiments of this application, and do not constitute a limitation on the technical solutions provided by the embodiments of this application. As those skilled in the art will know, with the evolution of technology and the emergence of new application scenarios, the technical solutions provided by the embodiments of this application are also applicable to similar technical problems.

[0074] Those skilled in the art will understand that the technical solutions shown in the figures do not constitute a limitation on the embodiments of this application, and may include more or fewer steps than shown, or combine certain steps, or different steps.

[0075] The device embodiments described above are merely illustrative. The units described as separate components may or may not be physically separate; that is, they may be located in one place or distributed across multiple network units. Some or all of the modules can be selected to achieve the purpose of this embodiment according to actual needs.

[0076] Those skilled in the art will understand that all or some of the steps in the methods disclosed above, as well as the functional modules / units in the systems and devices, can be implemented as software, firmware, hardware, or suitable combinations thereof.

[0077] The terms “first,” “second,” “third,” “fourth,” etc. (if present) in the specification and accompanying drawings of this application are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. It should be understood that such data can be interchanged where appropriate so that the embodiments of this application described herein can be implemented in orders other than those illustrated or described herein. Furthermore, the terms “comprising” and “having,” and any variations thereof, are intended to cover non-exclusive inclusion; for example, a process, method, system, product, or apparatus that comprises a series of steps or units is not necessarily limited to those steps or units explicitly listed, but may include other steps or units not explicitly listed or inherent to such processes, methods, products, or apparatus.

[0078] It should be understood that in this application, "at least one (item)" means one or more, and "more than" means two or more. "And / or" is used to describe the relationship between related objects, indicating that three relationships can exist. For example, "A and / or B" can represent three cases: only A exists, only B exists, and both A and B exist simultaneously, where A and B can be singular or plural. The character " / " generally indicates that the preceding and following related objects are in an "or" relationship. "At least one (item) of the following" or similar expressions refer to any combination of these items, including any combination of single or plural items. For example, at least one (item) of a, b, or c can represent: a, b, c, "a and b", "a and c", "b and c", or "a and b and c", where a, b, and c can be single or multiple.

[0079] In the several embodiments provided in this application, it should be understood that the disclosed apparatus and methods can be implemented in other ways. For example, the apparatus embodiments described above are merely illustrative; for instance, the division of the units described above is only a logical functional division, and in actual implementation, there may be other division methods. For example, multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed. Furthermore, the coupling or direct coupling or communication connection shown or discussed may be through some interfaces; the indirect coupling or communication connection between apparatuses or units may be electrical, mechanical, or other forms.

[0080] The units described above as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the units can be selected to achieve the purpose of this embodiment according to actual needs.

[0081] Furthermore, the functional units in the various embodiments of this application can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The integrated unit can be implemented in hardware or as a software functional unit.

[0082] If the integrated unit is implemented as a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, or all or part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes multiple instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods of the various embodiments of this application. The aforementioned storage medium includes various media capable of storing programs, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0083] The preferred embodiments of the present application have been described above with reference to the accompanying drawings, but this does not limit the scope of the claims of the present application. Any modifications, equivalent substitutions, and improvements made by those skilled in the art without departing from the scope and substance of the embodiments of the present application shall be within the scope of the claims of the present application.

Claims

1. A trajectory planning method for an ultrasonic robot based on the localization of key human body features, characterized in that, The method includes the following steps: Detect human bodies in images acquired by the global camera and output the detection bounding box for the human body; The location of key human features is predicted based on the detection box; wherein, the key human features include the human neck; The ultrasonic probe in the ultrasonic robot is controlled to move to the position of the human neck, and then the pose of the ultrasonic probe is calculated as the initial pose. The desired pose of the ultrasound probe at the key features of the human body is calculated, and then the ultrasound probe is controlled to move from the initial pose to the desired pose. Based on the key human body features, the ultrasound probe is trajectory-planned from its initial scanning position; wherein, the position of the ultrasound probe when it is in the desired pose is taken as the initial scanning position. The calculation of the ultrasonic probe's pose as the initial pose includes the following steps: The target normal vector at the position passing through the neck of the human body is calculated and used as the y-axis of the ultrasound probe; Based on the aforementioned key human features, the feature points of the left and right shoulders of the human body are determined. Connect the feature points of the left and right shoulders of the human body; The straight line parallel to the line connecting the two points and passing through the neck of the human body is defined as the x-axis of the ultrasound probe. The y-axis and x-axis of the ultrasonic probe are used as the orientation in the initial pose. The step of calculating the target normal vector passing through the neck of the human body and using it as the y-axis of the ultrasound probe includes the following steps: Multiple non-collinear target points are selected within a preset range of the human neck position, and each target point is located in the region of the human neck. The depth information of each target point is obtained using the global camera; A plane passing through the neck of the human body is formed based on the depth information; The target normal vector is calculated based on the plane and the position of the human neck and used as the y-axis of the ultrasound probe; The process of planning the trajectory of the ultrasound probe from its initial scanning position based on the key human body features includes the following steps: Multiple scanning points are obtained by scanning and sampling along the same direction starting from the initial scanning position, and the scanning trajectory is obtained by using polynomial interpolation. The desired probe pose of the ultrasonic probe at each of the scanning points is calculated based on the depth information of the points acquired by the global camera. The ultrasonic robot is controlled to perform the scanning operation based on the desired probe pose. Rotate the ultrasound probe 90° and return to the step of scanning and sampling multiple scanning points along the same direction starting from the initial scanning position, and using polynomial interpolation to obtain the scanning trajectory; During the ultrasound probe scanning process, a random walk algorithm is used to solve the confidence level of pixels in the image captured by the global camera; when the confidence level of pixels in any region is lower than a set threshold, the pose of the ultrasound probe is adjusted. During the ultrasound probe scanning process, the image captured by the global camera is segmented using the nnU-Net model to obtain the thyroid region; the pose of the ultrasound probe is dynamically adjusted based on the segmented thyroid region.

2. The ultrasonic robot trajectory planning method based on human key feature localization according to claim 1, characterized in that, The process of detecting human bodies in images acquired by a global camera and then outputting a detection bounding box for the human body includes the following steps: ResNet is used as the detection backbone network, and the feature map of the image is extracted using the detection backbone network as the detection feature map; Using FPN as the detection neck network, the detection neck network is used to perform feature fusion and enhancement on the detection feature map to obtain a multi-scale feature map; Using Faster R-CNN as the detection head network, candidate regions are generated on the multi-scale feature map using the RPN in the detection head network. Then, the features of the candidate regions are extracted through the Align operation, and the features of the candidate regions are classified and the detection box is regressed to obtain the detection box of the human body.

3. The ultrasonic robot trajectory planning method based on human key feature localization according to claim 1, characterized in that, The method involves predicting the location of key human features based on the detection box; wherein the key human features include the human neck, and includes the following steps: Using ResNet as the prediction backbone network, features are extracted from the sub-images corresponding to the detection boxes in the image to obtain the prediction feature map. Using FMP as the prediction neck network, the prediction feature map is selected, stitched together, and scaled using the prediction neck network to convert it into a format suitable for the prediction head network, thus obtaining the converted feature map. The prediction head network is used to output the key human features in the predicted feature map and the transformed feature map as corresponding heatmaps; wherein, the value of each heatmap is used to reflect the confidence of the corresponding key human feature at each position on the corresponding feature map; Based on the heatmap, a decoded image including the key human body features is generated, and then the location of the key human body features is predicted based on the decoded image.

4. The ultrasonic robot trajectory planning method based on human key feature localization according to claim 1, characterized in that, The calculation to obtain the desired pose of the ultrasound probe at the key features of the human body includes the following steps: The quadratic programming problem is as follows: ； in, For the decision variables to be optimized, It is a symmetric positive definite quadratic matrix. A linear coefficient vector, Used to describe inequality constraints Used to describe equality constraints Used for constraints Size; The quadratic programming problem is discretized based on the boundary conditions of the robotic arm, the end-effector pose of the robotic arm, the obstacle avoidance constraints, and the joint angle constraints of the robotic arm, resulting in a trajectory optimization problem. The trajectory optimization problem is: ； in, The joint angle variable to be optimized is... Used to describe obstacle constraints. Used to describe boundary constraints. Used to describe joint angle constraints; The desired joint angles of each joint in the robotic arm are obtained by solving the trajectory optimization problem. The desired pose is determined based on each desired joint angle.

5. An ultrasonic robot trajectory planning device based on the localization of key human body features, characterized in that, The device is used to implement the ultrasonic robot trajectory planning method based on human key feature localization as described in claim 1, and the device includes: The human detection unit is used to detect human bodies in images acquired by the global camera and then output the detection bounding box of the human body; A feature prediction unit is used to predict the location of key human body features based on the detection box; wherein, the key human body features include the human neck. The pose calculation unit is used to control the ultrasonic probe in the ultrasonic robot to move to the position of the human neck, and then calculate the pose of the ultrasonic probe as the initial pose. The probe moving unit is used to calculate the desired pose of the ultrasound probe at the key feature of the human body, and then control the ultrasound probe to move from the initial pose to the desired pose. The trajectory planning unit is used to plan the trajectory of the ultrasound probe from the initial scanning position based on the key human body features; wherein the position of the ultrasound probe when it is in the desired pose is taken as the initial scanning position.

6. An electronic device, characterized in that, The electronic device includes a memory and a processor, the memory storing a computer program, and the processor executing the computer program to implement the method as described in any one of claims 1 to 4.

7. A computer-readable storage medium storing a computer program, characterized in that, When the computer program is executed by a processor, it implements the method as described in any one of claims 1 to 4.

Images