Ultrasonic robot trajectory planning method based on human body key feature positioning

Abstract

The invention discloses an ultrasonic robot trajectory planning method based on human body key feature positioning, and relates to the technical field of automatic control, and the method comprises the steps: detecting a human body in an image obtained by a global camera, and outputting a detection frame of the human body; predicting positions of human body key features based on a detection frame; the human body key features comprise a human body neck; an ultrasonic probe in the ultrasonic robot is controlled to move to the position of the neck of the human body, and then the pose of the ultrasonic probe is calculated to serve as the initial pose; the expected pose of the ultrasonic probe at the key features of the human body is calculated, and then the ultrasonic probe is controlled to move from the initial pose to the expected pose; and performing trajectory planning on the ultrasonic probe from the initial scanning position when the ultrasonic probe is in the expected pose according to the key features of the human body. Track planning is performed on the ultrasonic probe through the key features of the human body, the pose of the ultrasonic probe in the scanning process can be adjusted, and the scanning precision of the ultrasonic probe can be improved.

Description

Technical Field

[0001] This application relates to the field of automatic control technology, and particularly to an ultrasonic robot trajectory planning method based on human key feature positioning. Background Art

[0002] An ultrasonic robot is a product of the combination of traditional medicine and artificial intelligence. It generally consists of an ultrasonic robotic arm, an ultrasonic device, a global camera, a force sensor, etc. It can gradually replace ultrasonic physicians to perform autonomous scans and simultaneously process the ultrasonic images obtained by the scans in real time to assist ultrasonic physicians in diagnosis and treatment.

[0003] In order to enable an ultrasonic robot to scan and acquire images in the human target skin area, it is necessary to locate the anatomical structure of the human target tissue. Some studies have carried out positioning through the method of robotic arm teaching, and some studies have registered magnetic resonance imaging (MRI) or computed tomography (CT) images with ultrasonic images to locate the human target tissue by means of other medical images. However, the first method has too poor autonomy and relies too much on manual operation; the second method has a high cost. While performing ultrasonic scans, it is also necessary to perform MRI or CT operations, which violates the low-cost feature of ultrasonic diagnosis and treatment. Summary of the Invention

[0004] The main purpose of the embodiments of this application is to propose an ultrasonic robot trajectory planning method based on human key feature positioning to improve the scanning accuracy of the ultrasonic robot.

[0005] To achieve the above purpose, on the one hand, an embodiment of this application proposes an ultrasonic robot trajectory planning method based on human key feature positioning. The method includes the following steps:

[0006] Detect a human body in the image obtained by the global camera and output a detection frame of the human body;

[0007] Predict the positions of human key features based on the detection frame; wherein, the human key features include the human neck;

[0008] Control the ultrasonic probe in the ultrasonic robot to move to the position of the human neck, and then calculate the pose of the ultrasonic probe as the initial pose;

[0009] Calculate the desired pose of the ultrasonic probe at the human key feature, and then control the ultrasonic probe to move from the initial pose to the desired pose;

[0010] Perform trajectory planning on the ultrasonic probe starting from the initial scanning position according to the human key feature; wherein, the position where the ultrasonic probe is located when it is in the desired pose is used as the initial scanning position.

[0011] In some embodiments, detecting a human body in the image acquired by the detection global camera and then outputting a detection box of the human body includes the following steps:

[0012] Using ResNet as the detection backbone network, and extracting a feature map of the image as a detection feature map by using the detection backbone network;

[0013] Using FPN as the detection neck network, and performing feature fusion and enhancement on the detection feature map by using the detection neck network to obtain a multi-scale feature map;

[0014] Using Faster R-CNN as the detection head network, generating candidate regions on the multi-scale feature map by using the RPN in the detection head network, then extracting features of the candidate regions through the Align operation, and further classifying the features of the candidate regions and performing detection box regression to obtain the detection box of the human body.

[0015] In some embodiments, predicting the positions of the key human body features based on the detection box; wherein, the key human body features include the human neck, and the method includes the following steps:

[0016] Using ResNet as the prediction backbone network, and extracting features from the sub-image corresponding to the detection box in the image by using the prediction backbone network to obtain a prediction feature map;

[0017] Using FMP as the prediction neck network, and performing selection, splicing and scaling on the prediction feature map by using the prediction neck network and then converting it into a format suitable for the prediction head network to obtain a converted feature map;

[0018] Using the prediction head network to output the key human body features in the prediction feature map and the converted feature map as corresponding heatmaps; wherein, the value of each heatmap is used to reflect the confidence of the corresponding key human body feature at each position on the corresponding feature map;

[0019] Decoding a decoded image including the key human body features according to the heatmap, and then predicting the positions of the key human body features according to the decoded image.

[0020] In some embodiments, calculating the pose of the ultrasound probe as the initial pose includes the following steps:

[0021] Calculating a target normal vector passing through the position of the human neck as the y-axis of the pose of the ultrasound probe;

[0022] Determining feature points of the left and right shoulders of the human body according to the key human body features;

[0023] Connecting the feature points of the left and right shoulders of the human body;

[0024] Determine the straight line parallel to the connecting line and passing through the position of the human neck as the x-axis of the pose of the ultrasonic probe;

[0025] Wherein, the y-axis and the x-axis of the pose of the ultrasonic probe are used as the initial pose.

[0026] In some embodiments, calculating the target normal vector passing through the position of the human neck as the y-axis of the pose of the ultrasonic probe includes the following steps:

[0027] Select a plurality of non-collinear target points within a preset range of the position of the human neck, and each of the target points is located in the area of the human neck;

[0028] Use the global camera to obtain the depth information of each of the targets;

[0029] Form a plane passing through the position of the human neck according to the depth information;

[0030] Calculate the target normal vector as the y-axis of the pose of the ultrasonic probe according to the plane and the position of the human neck.

[0031] In some embodiments, calculating the desired pose of the ultrasonic probe at the key features of the human body includes the following steps:

[0032] Construct the following quadratic programming problem:

[0033]

[0034] Wherein, x is the decision variable to be optimized, H is a symmetric positive definite quadratic matrix, f is a linear coefficient vector, (A b) is used to describe the inequality constraint, (A eq b eq ) is used to describe the equality constraint, and (lb ub) is used to constrain the magnitude of x;

[0035] Discretize the quadratic programming problem according to the boundary conditions of the robotic arm of the ultrasonic robot, the end pose of the robotic arm, the obstacle avoidance constraint conditions, and the joint angle limit conditions of the robotic arm to obtain a trajectory optimization problem;

[0036] The trajectory optimization problem is:

[0037]

[0038] Wherein, q is the joint angle variable to be optimized, (A obs b obs ) is used to describe the obstacle constraint conditions, (A bd b bd) is used to describe boundary constraint conditions, (q min q max ) is used to describe joint angle constraint conditions;

[0039] The desired joint angles of each joint in the robotic arm are obtained by solving the trajectory optimization problem;

[0040] The desired pose is determined according to each of the desired joint angles.

[0041] In some embodiments, the trajectory planning of the ultrasound probe starting from the initial scanning position according to the human key features includes the following steps:

[0042] Starting from the initial scanning position, scan and sample in the same direction to obtain a plurality of scan points, and use the method of polynomial interpolation to obtain the scanning trajectory;

[0043] The desired probe poses of the ultrasound probe at each of the scan points are calculated according to the depth information of the points obtained by the global camera;

[0044] The ultrasound robot is controlled according to each of the desired probe poses to implement the scanning operation;

[0045] Rotate the ultrasound probe by 90°, and return to the step of starting from the initial scanning position, scanning and sampling in the same direction to obtain a plurality of scan points, and using the method of polynomial interpolation to obtain the scanning trajectory;

[0046] During the scanning of the ultrasound probe, the confidence of the pixels in the image captured by the global camera is solved using a random walk algorithm; when the confidence of the pixels in any area is lower than the set threshold, the pose of the ultrasound probe is adjusted;

[0047] During the scanning of the ultrasound probe, the nnU-Net model is used to segment the image captured by the global camera to obtain the thyroid region; the pose of the ultrasound probe is dynamically adjusted according to the segmented thyroid region.

[0048] To achieve the above object, another aspect of the embodiments of the present application proposes an ultrasound robot trajectory planning device based on human key feature positioning, and the device includes:

[0049] A human detection unit, configured to detect a human body in the image obtained by the global camera and then output a detection frame of the human body;

[0050] A feature prediction unit, configured to predict the positions of human key features based on the detection frame; wherein, the human key features include the human neck;

[0051] A pose calculation unit is used to control the ultrasonic probe in the ultrasonic robot to move to the position of the human neck, and then calculate the pose of the ultrasonic probe as the initial pose.

[0052] A probe movement unit is used to calculate the desired pose of the ultrasonic probe at the key features of the human body, and then control the ultrasonic probe to move from the initial pose to the desired pose.

[0053] A trajectory planning unit is used to perform trajectory planning on the ultrasonic probe starting from the initial scanning position according to the key features of the human body; wherein, the position where the ultrasonic probe is located when it is in the desired pose is used as the initial scanning position.

[0054] To achieve the above object, on the other hand, an embodiment of the present application proposes an electronic device, which includes a memory and a processor. The memory stores a computer program, and when the processor executes the computer program, the above method is implemented.

[0055] To achieve the above object, on the other hand, an embodiment of the present application proposes a computer-readable storage medium, which stores a computer program, and when the computer program is executed by a processor, the above method is implemented.

[0056] The embodiments of the present application at least include the following beneficial effects:

[0057] The present application can detect the human body in the image obtained by the global camera and then output the detection frame of the human body; predict the position of the key features of the human body based on the detection frame; wherein, the key features of the human body include the human neck; control the ultrasonic probe in the ultrasonic robot to move to the position of the human neck, and then calculate the pose of the ultrasonic probe as the initial pose; calculate the desired pose of the ultrasonic probe at the key features of the human body, and then control the ultrasonic probe to move from the initial pose to the desired pose; perform trajectory planning on the ultrasonic probe starting from the initial scanning position according to the key features of the human body; wherein, the position where the ultrasonic probe is located when it is in the desired pose is used as the initial scanning position. By performing trajectory planning on the ultrasonic probe according to the key features of the human body, the present application can adjust the pose of the ultrasonic probe during the scanning process, and can improve the scanning accuracy of the ultrasonic probe. Description of the Drawings

[0058] To more clearly illustrate the technical solutions in the embodiments of the present application, the following will briefly introduce the drawings required for the description of the embodiments. Obviously, the following drawings are only some embodiments of the present application. For those of ordinary skill in the art, without creative efforts, other drawings can also be obtained based on these drawings.

[0059] Figure 1Schematic flowchart of an ultrasonic robot trajectory planning method based on human key feature positioning provided by an embodiment of the present application;

[0060] Figure 2 Schematic diagram of the MDH coordinate system of the ultrasonic robotic arm provided by an embodiment of the present application;

[0061] Figure 3 Schematic diagram of the kinematic model of the ultrasonic robot provided by an embodiment of the present application;

[0062] Figure 4 Structure diagram of the human key feature recognition and positioning model provided by an embodiment of the present application;

[0063] Figure 5 Schematic diagram of the scanning trajectory planning framework provided by an embodiment of the present application;

[0064] Figure 6 Schematic diagram of the structure of an ultrasonic robot trajectory planning device based on human key feature positioning provided by an embodiment of the present application;

[0065] Figure 7 Schematic diagram of the hardware structure of an electronic device provided by an embodiment of the present application. Detailed implementation manners

[0066] In order to make the objectives, technical solutions and advantages of the present application clearer and more understandable, the present application will be further described in detail below with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present application and are not used to limit the present application. When the following description refers to the accompanying drawings, unless otherwise indicated, the same numbers in different drawings represent the same or similar elements. The embodiments described in the following exemplary embodiments do not represent all embodiments consistent with the embodiments of the present application. They are only examples of devices and methods consistent with some aspects of the embodiments of the present application as detailed in the appended claims.

[0067] It can be understood that the terms "first", "second", etc. used in the present application may be used herein to describe various concepts, but unless otherwise specified, these concepts are not limited by these terms. These terms are only used to distinguish one concept from another. For example, without departing from the scope of the embodiments of the present application, the first information may also be referred to as the second information, and similarly, the second information may also be referred to as the first information. Depending on the context, the words "if", "when" as used herein may be interpreted as "when...", "while...", or "in response to determining".

[0068] The terms "at least one", "a plurality", "each", "any one", etc. used in this application, "at least one" includes one, two or more, "a plurality" includes two or more, "each" refers to each one of the corresponding plurality, and "any one" refers to any one of the plurality.

[0069] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as commonly understood by those skilled in the technical field to which this application belongs. The terms used herein are only for the purpose of describing the embodiments of this application and are not intended to limit this application.

[0070] Before elaborating on the embodiments of this application in detail, first, some related technologies involved in the embodiments of this application are described as follows:

[0071] After positioning the human target tissue, it is equally important to perform corresponding trajectory planning. The trajectory planning of the ultrasound robot is divided into two parts. The first part is the trajectory planning from the initial position of the ultrasound robotic arm to the position of the human target tissue, and the second part is the corresponding trajectory planning in the human target tissue area to scan and obtain the complete structure of the human tissue. At the same time, the comfort of the patient needs to be ensured during the scanning process. For the trajectory planning of the first part, the patient's posture needs to be fully considered to constrain the movement trajectory of the ultrasound robotic arm to ensure that the robotic arm does not collide with the human body during the movement process; for the trajectory planning of the second part, many studies reconstruct the human target tissue area through a point cloud camera, customize multiple planning points in the reconstructed area, and connect the multiple planning points into a planning path. However, in actual scanning, due to the diversity of the human skin surface structure and the errors of the point cloud camera, scanning along the predefined path may not fully cover the human target tissue, and at the same time, some ultrasound images may have problems such as low imaging quality. Therefore, the posture of the ultrasound probe during scanning and the planned path need to be adjusted in real time.

[0072] Referring to Figure 1 , the embodiments of this application provide an ultrasound robot trajectory planning method based on human key feature positioning, and this method may include but is not limited to S100 to S140, specifically as follows:

[0073] S100: Detect the human body in the image obtained by the global camera and output the detection frame of the human body;

[0074] S110: Predict the positions of human key features based on the detection frame; wherein, the human key features include the human neck;

[0075] S120: Control the ultrasound probe in the ultrasound robot to move to the position of the human neck, and then calculate and obtain the pose of the ultrasound probe as the initial pose;

[0076] S130: Calculate the expected pose of the ultrasonic probe at the key human feature, and then control the ultrasonic probe to move from the initial pose to the expected pose;

[0077] S140: Perform trajectory planning on the ultrasonic probe starting from the initial scanning position according to the key human feature; wherein, the position where the ultrasonic probe is located when in the expected pose is used as the initial scanning position.

[0078] Optionally, detecting the human body in the image acquired by the global detection camera and then outputting the detection frame of the human body includes the following steps:

[0079] Using ResNet as the detection backbone network, and extracting the feature map of the image as the detection feature map by using the detection backbone network;

[0080] Using FPN as the detection neck network, and performing feature fusion and enhancement on the detection feature map by using the detection neck network to obtain a multi-scale feature map;

[0081] Using Faster R-CNN as the detection head network, generating candidate regions on the multi-scale feature map by using the RPN in the detection head network, then extracting the features of the candidate regions through the Align operation, and further classifying the features of the candidate regions and performing detection frame regression to obtain the detection frame of the human body.

[0082] Optionally, predicting the position of the key human feature based on the detection frame; wherein, the key human feature includes the human neck, and includes the following steps:

[0083] Using ResNet as the prediction backbone network, and extracting the features of the sub-image corresponding to the detection frame in the image by using the prediction backbone network to obtain a prediction feature map;

[0084] Using FMP as the prediction neck network, and performing selection, splicing and scaling on the prediction feature map by using the prediction neck network and then converting it into a format suitable for the prediction head network to obtain a converted feature map;

[0085] Using the prediction head network to output the key human features in the prediction feature map and the converted feature map as corresponding heatmaps; wherein, the value of each heatmap is used to reflect the confidence of the corresponding key human feature at each position on the corresponding feature map;

[0086] Decoding a decoded image including the key human feature according to the heatmap, and then predicting the position of the key human feature according to the decoded image.

[0087] Optionally, calculating the pose of the ultrasonic probe as the initial pose includes the following steps:

[0088] Calculate the target normal vector passing through the position of the human neck as the y-axis of the pose of the ultrasonic probe;

[0089] Determine the feature points of the left and right shoulders of the human body according to the key features of the human body;

[0090] Connect the feature points of the left and right shoulders of the human body;

[0091] Determine the straight line parallel to the connection line and passing through the position of the human neck as the x-axis of the pose of the ultrasonic probe;

[0092] Wherein, the y-axis and the x-axis of the pose of the ultrasonic probe are used as the initial pose.

[0093] Optionally, calculating the target normal vector passing through the position of the human neck as the y-axis of the pose of the ultrasonic probe includes the following steps:

[0094] Select a plurality of non-collinear target points within a preset range of the position of the human neck, and each of the target points is located in the area of the human neck;

[0095] Use the global camera to obtain the depth information of each of the targets;

[0096] Form a plane passing through the position of the human neck according to the depth information;

[0097] Calculate the target normal vector as the y-axis of the pose of the ultrasonic probe according to the plane and the position of the human neck.

[0098] Optionally, calculating the expected pose of the ultrasonic probe at the key features of the human body includes the following steps:

[0099] Construct the following quadratic programming problem:

[0100]

[0101] Where x is the decision variable to be optimized, H is a symmetric positive definite quadratic matrix, f is a linear coefficient vector, (A b) is used to describe the inequality constraint, (A eq b eq ) is used to describe the equality constraint, and (lb ub) is used to constrain the magnitude of x;

[0102] Discretize the quadratic programming problem according to the boundary conditions of the robotic arm of the ultrasonic robot, the end pose of the robotic arm, the obstacle avoidance constraint conditions, and the joint angle limit conditions of the robotic arm to obtain a trajectory optimization problem;

[0103] The trajectory optimization problem is:

[0104]

[0105] Among them, q is the joint angle variable to be optimized, (A obs b obs ) is used to describe the obstacle constraint condition, (A bd b bd ) is used to describe the boundary constraint condition, (q min q max ) is used to describe the joint angle constraint condition;

[0106] The desired joint angles of each joint in the robotic arm are obtained by solving the trajectory optimization problem;

[0107] The desired pose is determined according to each of the desired joint angles.

[0108] Optionally, the trajectory planning of the ultrasonic probe starting from the initial scanning position according to the human key features includes the following steps:

[0109] Starting from the initial scanning position, scan and sample in the same direction to obtain multiple scan points, and use the method of polynomial interpolation to obtain the scanning trajectory;

[0110] The desired probe poses of the ultrasonic probe at each of the scan points are calculated according to the depth information of the points obtained by the global camera;

[0111] The ultrasonic robot is controlled to perform a scanning operation according to each of the desired probe poses;

[0112] Rotate the ultrasonic probe by 90°, and return to the step of starting from the initial scanning position, scanning and sampling in the same direction to obtain multiple scan points, and using the method of polynomial interpolation to obtain the scanning trajectory;

[0113] During the scanning of the ultrasonic probe, the confidence of the pixels in the image captured by the global camera is solved by using a random walk algorithm; when the confidence of the pixels in any area is lower than the set threshold, the pose of the ultrasonic probe is adjusted;

[0114] During the scanning of the ultrasonic probe, the image of the global camera is segmented by using the nnU-Net model to obtain the thyroid region; the pose of the ultrasonic probe is dynamically adjusted according to the segmented thyroid region.

[0115] Next, specific application examples will be combined to introduce and illustrate the solutions of the embodiments of the present application in detail.

[0116] This embodiment discloses an ultrasonic robot trajectory planning method based on human key feature positioning. The ultrasonic robot can gradually replace ultrasonic physicians to autonomously complete ultrasonic scanning operations. Among them, the autonomy is mainly reflected in that the ultrasonic robot can autonomously locate the tissue area to be scanned on the human body and control the ultrasonic probe to autonomously complete the scanning operation and collect high-quality ultrasonic images for better subsequent diagnosis and treatment. The method of this embodiment locates the tissue area to be scanned based on the key features of the human body. At the same time, the quadratic programming method is used to initially plan the scanning trajectory, and combined with the confidence map of the ultrasonic image and the real-time segmentation result, the scanning trajectory is continuously improved, and the quality of the collected ultrasonic images is continuously improved.

[0117] 1. Robot kinematic model.

[0118] 1.1 Manipulator kinematic model.

[0119] The kinematic model of the ultrasonic manipulator is a mathematical model that describes the relationship between the pose of the end effector of the manipulator and each joint of the manipulator, and is divided into two parts: the forward kinematic model and the inverse kinematic model.

[0120] The forward kinematic model of the ultrasonic manipulator describes how to solve the pose of the end effector in the base coordinate system given the joint angle of each joint of the manipulator. The ultrasonic manipulator used in this embodiment has 6 degrees of freedom. The forward kinematic model is derived by the MDH method, and 6 MDH coordinate systems are established at the joints, as Figure 2 shown, where {0}:{6} represents the MDH coordinate system of the 6-degree-of-freedom manipulator, {d i}(i = 1, 4, 6) is the joint offset, and {a3} is the link length.

[0121] Let the joint angle of the 6-degree-of-freedom manipulator be θ i , the link twist angle be α i , i-1 T i be the homogeneous transformation matrix from the {i - 1} coordinate system to the {i} coordinate system, i = 1, 2, L, 6, then i-1 T i is expressed as:

[0122]

[0123] According to the chain rule, the forward kinematic model of the ultrasonic manipulator is as follows:

[0124] 0 T6(θ1, θ2,..., θ6) = 0 T1 1 T2L 5 T6 (2)

[0125] The inverse kinematic model of the ultrasonic robotic arm describes how to solve the joint angles of the robotic arm given the target pose of the end effector in the base coordinate system. Since the ultrasonic robotic arm of this embodiment satisfies the Pieper criterion, that is, the three adjacent joint axes of the robotic arm intersect at a point, a closed-form solution can be calculated according to the inverse kinematics. The last three joint axes of the ultrasonic robotic arm of this embodiment intersect at a point, that is, the three axes of coordinate systems {4}, {5}, and {6} intersect at a point. Then, the homogeneous coordinates of this point in the base coordinate system of the robotic arm can be expressed as:

[0126]

[0127] As can be seen from Equation (1) 3 p4 = [a3 - d4sinα3 d4cosα3 1] T , substituting Equation (1) into Equation (3), we can get 0 The expression of p4 with respect to (θ1, θ2, θ3), given 0 the coordinate values of p6, the coordinate values of p4 can be obtained accordingly. Then, list 0 the sum-of-squares equation of p4, and first solve for θ3, then solve for θ2, and finally solve for θ1 by the variable substitution method. The axes of the last three joints of the robotic arm intersect at a point, and the rotations of these three joints will affect the end pose of the robotic arm. (θ4, θ5, θ6) can be solved from the rotation matrix 0 R6 representing the end pose. 0 R6.

[0128] 1.2 System Kinematic Model.

[0129] The ultrasonic robot system consists of a robotic arm, an ultrasonic probe, a probe connector, a six-axis force sensor, and a global camera. Among them, the robotic arm is a 6-degree-of-freedom robotic arm; the ultrasonic probe is a linear probe used for imaging of shallow tissues and blood vessels, and the imaging mode is B-mode, which converts the echo intensity of the ultrasonic waves emitted by the ultrasonic probe into a grayscale image; the probe connector is manufactured by 3D printing; the six-axis force sensor is used to quickly and accurately measure the forces and torques on the three coordinate axes of x, y, and z of the sensor; the global camera is an RGB-D camera used to detect the pose of the end effector of the robotic arm. At the same time, it reconstructs the surface of the human tissue to be scanned and provides visual input information for the trajectory planning of the robot.

[0130] Perform kinematic modeling on the entire ultrasonic robot system, as Figure 3As shown in the figure, where {base} is the base coordinate system of the robot, {end} is the end coordinate system of the robot, {tag} is the marker coordinate system of the robot, {img} is the image coordinate system of the ultrasonic probe, and {gc} is the coordinate system of the global camera. The QR code is used to determine the relationship between the global camera and the ultrasonic robotic arm; the six-axis force sensor is used to measure the force and torque received during the scanning process of the ultrasonic probe. By feeding back the force, the force and torque applied by the ultrasonic robot are further controlled to ensure the comfort level of the patient. It should be noted that the six-axis force sensor needs to be gravity compensated before application to remove the gravity received by the ultrasonic probe and the probe fixture, reducing the influence of this part of gravity, so as to more accurately control the force and torque applied at the end. According to the forward kinematics model of the robotic arm, it can be solved in real time to obtain base T end ; By taking pictures of the QR code with the global camera, extracting the information of the four corner points of the QR code and using the PNP algorithm for solution, it can be obtained in real time gc T tag ; Through the multi-object calibration method, it can be obtained end T tag , tag T img , gc T base . Finally, the kinematics model of the entire system is formed.

[0131] 2. Method for locating key human body features.

[0132] 2.1. Identification and location of key human body features.

[0133] When the ultrasonic robot performs autonomous scanning, it is necessary to first determine the initial scanning pose of the ultrasonic probe, that is, it is necessary to move the ultrasonic probe to the skin surface of the human tissue to be scanned, and then perform subsequent scanning and image acquisition operations.

[0134] The ultrasonic robot proposed in this embodiment is mainly used to scan the thyroid tissue of the human body to evaluate the structure, size, shape and lesion conditions of the thyroid. The thyroid tissue is located below the human neck and consists of two left and right lobes and an isthmus. The human neck is one of the key features of the human body. In this embodiment, this key feature is used to roughly locate the anatomical structure of the thyroid, so as to guide the ultrasonic robot to move the ultrasonic probe to the specified position as the initial point of subsequent ultrasonic scanning operations.

[0135] In order to identify and locate the key features at the human neck, in this embodiment, a deep learning method is adopted to locate the key features through a trained neural network model. The positioning model adopts a top-down method. First, it detects the human body in the image obtained by the global camera, outputs the detection box of the human body, and then predicts the position of the human neck based on this detection box, so as to realize the identification and location of the key features of the human body.

[0136] The structure of the positioning model is as Figure 4 shown. It is divided into three modules in total, namely the data processing module, the detection module, and the prediction module. The input is the video frame collected by the global camera, and the output is the image containing the key features, which can realize the real-time identification and location of the key features.

[0137] In the data processing module, it mainly processes the video frame obtained by the global camera, and completes operations such as normalization of the video frame and transformation of the channel order, so as to output the preprocessed picture.

[0138] In the detection module, the input is the picture output by the data processing module, and the final output is the picture containing the detection box. For the detection backbone network, ResNet is used as the backbone network to perform feature extraction operations on the input picture. For the neck detection network, FPN is used as the neck network to perform feature fusion and enhancement on the feature map output by the detection backbone network. The introduction of the neck detection network enables the model to obtain good results when dealing with small targets and large targets, and at the same time provides more unified and semantically rich features for the subsequent head network. For the head detection network, the input is the multi-scale features output by the neck network. Faster R-CNN is used as the head network. The RPN is used to generate candidate regions on the feature map, and then the features corresponding to the candidate regions are extracted through the Align operation, and more refined classification and detection box regression are performed. Finally, the data is decoded to obtain the picture containing the detection box, which is input into the prediction module. In the detection backbone network, the neck network, and the head network, the DCN enhancement operator is introduced to enhance the feature extraction ability of the module.

[0139] In the prediction module, only the detection box part of the image is used as the input, and key features are predicted for this part. For the prediction backbone network, ResNet is used as the backbone network to extract features from the detection box part. For the prediction neck network, FMP is used as the neck network, and through non-parametric transformations such as selection, splicing, and scaling, the feature map output by the prediction backbone network is converted into a format suitable for the prediction head network. For the prediction head network, the feature maps processed by the prediction backbone network and the neck network are converted into the final pose estimation results, that is, the position prediction of key features. It outputs a heatmap corresponding to each key feature. Among them, the value of each heatmap reflects the confidence of the corresponding key feature at each position on the image. Finally, the image containing the key features is decoded to provide the position information of the key features for subsequent operations.

[0140] The positioning model adopted in this embodiment can obtain the position information of the human neck. Since the thyroid tissue is located below the human neck, the obtained neck position information can be used to roughly estimate the position of the human thyroid tissue.

[0141] When the human body is facing the global camera sideways and only the left side of the neck faces the global camera, the key features predicted by the positioning model correspond to the points in the left lobe region of the thyroid; when only the right side of the neck faces the global camera, the key features obtained by the positioning model correspond to the points in the right lobe region of the thyroid. Therefore, when the human body is facing the global camera sideways, the ultrasound robot can obtain the position information of one lobe of the thyroid, thereby guiding the ultrasound probe to scan one lobe. The specific scanning method will be introduced in Section 3 below.

[0142] When the human body is facing the global camera directly, there is a deviation between the key features predicted by the positioning model and the points in the isthmus region of the thyroid. However, this deviation only exists on a certain axis. Therefore, only the direction of this axis needs to be determined, and the ultrasound probe is guided to move along this axis until the isthmus region of the thyroid tissue is recognized in the ultrasound image, then the thyroid tissue can be located. The determination of the deviation axis will be introduced in Section 2.2 below, and the identification of the thyroid tissue based on the ultrasound image will be introduced in Section 3 below. After the ultrasound probe moves to the skin surface of the isthmus region of the thyroid tissue, the entire isthmus region can be scanned through longitudinal movement, and through lateral movement, it can move to the left and right lobe regions of the thyroid tissue, and the entire lobe region can be scanned through longitudinal movement in the left and right lobe regions.

[0143] 2.2. Ultrasound probe pose solution.

[0144] From Section 2.1, the initial target position for the ultrasound probe to scan can already be obtained. Next, the target pose of the probe needs to be determined, that is, given the origin position of the {img} coordinate system, solve for the coordinate axes (x img y img)。

[0145] In this embodiment, a method using the normal vector is adopted to determine the initial pose of the ultrasonic probe scanning. Given the initial target position point, it is necessary to solve the target normal vector passing through this point, that is, y img , this normal vector is perpendicular to the neck region of the human body, so as to be able to guide the ultrasonic probe perpendicular to the neck region of the human body, and then scan to obtain the cross-sectional ultrasonic image of the human neck region. In order to solve for y img , in this embodiment, several non-collinear points near the target position point are selected, and these points are all located in the neck region of the human body. With the depth information of the points obtained by the global camera, a plane α passing through the target position point is formed. Finally, according to the information of plane α and the target position point information, the target normal vector, that is, y img can be solved.

[0146] After solving for y img , given one axis of the initial pose of the ultrasonic probe scanning, another axis x img still needs to be determined to determine the initial pose information. In order to better make the probe pose of the ultrasonic robot during scanning similar to the probe pose during manual ultrasonic operation, and at the same time according to the structural characteristics of the human body, this embodiment introduces the key features of the human shoulders, and determines x img with the help of the key features of the shoulders and the key features of the neck. The recognition and positioning of the key features of the shoulders are also through deep learning methods. The basic structure of the positioning model is similar to Figure 4 , and it is trained through a data set marked with the key features of the shoulders to obtain the corresponding positioning model. Through this positioning model, the key feature points of the human shoulders are obtained. The feature points of the left and right shoulders can form a straight line l. Combining the structural characteristics of the human body, x img is set as the coordinate axis parallel to l and passing through the target position point. Given the information of the straight line l and the target position point, x img can be solved. At the same time, the axis located in the neck region plane and perpendicular to x img and passing through the target position point is the deviation axis mentioned in Section 2.1. The ultrasonic probe moving along this deviation axis can locate the isthmus region of the thyroid tissue from the ultrasonic image.

[0147] 3. Trajectory planning method.

[0148] 3.1 Initial trajectory planning method.

[0149] After solving the desired pose of the ultrasonic robot at the key features of the human body, initial trajectory planning is required to control the ultrasonic probe to move from the starting pose to the desired pose in order to achieve subsequent ultrasonic scanning operations. This embodiment adopts the method of quadratic programming to realize the initial trajectory planning of the ultrasonic robot.

[0150] The quadratic programming method is a non-linear programming method with a quadratic function as the objective function, and its solution form is as follows:

[0151]

[0152] where x is the decision variable to be optimized, H is a symmetric positive definite quadratic matrix, f is a linear coefficient vector, (A b) is used to describe the inequality constraints, and (A eq b eq ) is used to describe the equality constraints, and (lb ub) is used to constrain the magnitude of x.

[0153] In this embodiment, the joint angle q(t) of the ultrasonic robotic arm is used as the decision variable. At the same time, the starting value of the joint angle is known, and the expected value of the joint angle can be calculated through the inverse kinematics model of the ultrasonic robotic arm based on the positioning information of the human key features. Then q(t0) = q0, q(t f ) = q f , where t0 is the starting time and t f is the ending time. Then the boundary conditions can be defined as:

[0154] A bd q(t) = b bd (5)

[0155] At the same time, it is necessary to ensure that the ultrasonic robotic arm does not collide with the human body during the movement. Since the ultrasonic probe needs to move to the neck area of the human body, in actual operation, if only a simple trajectory planning is adopted, the end effector of the ultrasonic robot, that is, the ultrasonic probe, will collide with the shoulder or facial area of the human body during the movement. Therefore, in this embodiment, the human key feature recognition and positioning method adopted in Section 2 above is used again to introduce the obstacle avoidance constraint of the ultrasonic robot. The trained model is used to locate the key features of the shoulder and facial areas of the human body to obtain the coordinate point set Ω = [p1 p2... p n , where p i = [x i y i z i T , i = 1, 2,..., n. At the same time, the end pose p end (t) of the ultrasonic robotic arm is defined, and then it is necessary to satisfy:

[0156] ||p end (t) - p i || ≥ d safe (6)

[0157] where d safe is the safety distance. Then, using the linearization method, the obstacle avoidance constraint is approximated as:

[0158] A​obs q(t) ≤ b obs (7)

[0159] Since there are angular limitations for each moving joint of the ultrasonic robotic arm, it can be defined as:

[0160] q min ≤ q(t) ≤ q max (8)

[0161] Perform a discretization operation on problem (4), discretize time t into N discrete time moments. At the same time, combining constraints (5), (7), and (8), the trajectory optimization problem can be transformed into:

[0162]

[0163] Finally, N desired joint angles in the initial trajectory of the ultrasonic robot can be solved, thereby guiding the ultrasonic probe to move to the desired pose.

[0164] 3.2. Scanning trajectory planning method.

[0165] After the ultrasonic robot guides the ultrasonic probe to reach the initial scanning position along the initial trajectory, the scanning trajectory planning starts. The scanning trajectory planning method proposed in this embodiment is carried out synchronously in two modules. For module 1, the human tissue area to be scanned is reconstructed according to the image obtained by the global camera, and scanning points are artificially set in the reconstructed area. The desired poses of the ultrasonic probe corresponding to each scanning point can be obtained by the method proposed in Section 2.2 above. For module 2, when the ultrasonic robot scans along the trajectory obtained in the first step, it is adjusted in real time by combining the ultrasonic confidence map and the ultrasonic image segmentation result to ensure that clear and complete thyroid tissue is scanned. The overall framework of the scanning trajectory planning method is as Figure 5 shown.

[0166] In Module 1, based on the depth information of the human tissue area obtained by the global camera, multiple pre-scanning points are sampled, and the corresponding trajectory is obtained by using the method of polynomial interpolation. It can be known from Section 2 above about the initial pose of the ultrasonic probe scanning in the target tissue area and the direction of longitudinal scanning. Thus, taking the key points obtained in Section 2 above as the starting points, other scanning points are sampled along the direction of longitudinal scanning, and the corresponding trajectory is obtained by using the method of polynomial interpolation. Finally, according to the depth information of the points obtained by the global camera, the expected probe pose of the sampled scanning points can be calculated, so that the ultrasonic robot can be controlled to perform a simple scanning operation, longitudinally scan the left and right lobes and the isthmus area of the human thyroid tissue, and obtain the corresponding horizontal plane ultrasonic images. If the ultrasonic probe is rotated by 90° and a similar operation is performed, transverse scanning can be carried out to obtain the corresponding sagittal plane ultrasonic images. However, due to the diversity of the human skin structure, the scanning trajectory formed by Module 1 may not be able to completely cover the entire thyroid tissue. At the same time, due to the errors existing in the depth information obtained by the global camera itself, the pose calculated according to the depth information may not be the optimal pose, that is, under the calculated pose, the ultrasonic probe may not be able to scan and obtain a sufficiently clear ultrasonic image or cover a relatively complete ultrasonic image of the thyroid tissue. Therefore, Module 2 is needed for real-time adjustment.

[0167] In Module 2, in this embodiment, with the feedback of the ultrasonic confidence map, the pose of the ultrasonic probe is adjusted in real time to ensure obtaining a sufficiently clear ultrasonic image. At the same time, with the feedback of the result of ultrasonic image organ segmentation, the pose of the ultrasonic probe is adjusted in real time to ensure that the obtained ultrasonic image can cover a relatively complete thyroid tissue. In order to scan and obtain a clearer ultrasonic image for subsequent diagnosis and treatment, this embodiment introduces an ultrasonic confidence map to evaluate the quality of the ultrasonic image. The ultrasonic confidence map uses the random walk algorithm to solve the confidence of pixels. The higher the confidence of a certain pixel, the higher the imaging quality of this pixel. Initially, it is stipulated that the confidence of the pixel closest to the ultrasonic probe is 1, the confidence of the pixel farthest from the ultrasonic probe is 0, and the confidence of other pixels is between the two. When the confidence of a certain area of the ultrasonic image is low, the ultrasonic robot controls the ultrasonic probe to perform corresponding pose adjustments to improve the overall confidence of the ultrasonic image, so as to obtain a clearer ultrasonic image. In addition to obtaining a clear ultrasonic image, it is also necessary to ensure that the ultrasonic image can cover a relatively complete thyroid tissue, so that each frame of the ultrasonic image has more effective information. Therefore, this embodiment uses the nnU-Net model for image segmentation to segment the thyroid area in the ultrasonic image. During the scanning process of the ultrasonic robot, according to the real-time segmentation results, the pose of the ultrasonic probe is adjusted to ensure that the segmented thyroid tissue is located in the central area of the ultrasonic image, thereby increasing the effective information of the ultrasonic image.

[0168] Refer to Figure 6, the embodiment of the present application also provides an ultrasonic robot trajectory planning device based on human key feature positioning, which can implement the above-mentioned ultrasonic robot trajectory planning method based on human key feature positioning. The device includes:

[0169] A human body detection unit, configured to detect the human body in the image acquired by the global camera and then output a detection frame of the human body;

[0170] A feature prediction unit, configured to predict the positions of human key features based on the detection frame; wherein, the human key features include the human neck;

[0171] A pose calculation unit, configured to control the ultrasonic probe in the ultrasonic robot to move to the position of the human neck, and then calculate the pose of the ultrasonic probe as the initial pose;

[0172] A probe movement unit, configured to calculate the desired pose of the ultrasonic probe at the human key feature, and then control the ultrasonic probe to move from the initial pose to the desired pose;

[0173] A trajectory planning unit, configured to perform trajectory planning on the ultrasonic probe starting from the initial scanning position according to the human key features; wherein, the position where the ultrasonic probe is located when in the desired pose is used as the initial scanning position.

[0174] It can be understood that the content in the above method embodiments is applicable to the device embodiments of the present application. The functions specifically implemented by the device embodiments of the present application are the same as those of the above method embodiments, and the beneficial effects achieved are also the same as those of the above method embodiments.

[0175] The embodiment of the present application also provides an electronic device. The electronic device includes a memory and a processor. The memory stores a computer program, and when the processor executes the computer program, the method of the embodiment of the present application is implemented. The electronic device can be any intelligent terminal including a tablet computer, an in-vehicle computer, etc.

[0176] It can be understood that the content in the above method embodiments is applicable to the device embodiments of the present application. The functions specifically implemented by the device embodiments of the present application are the same as those of the method of the present application, and the beneficial effects achieved are also the same as those of the method of the present application.

[0177] Please refer to Figure 7 , Figure 7 , which schematically shows the hardware structure of an electronic device in another embodiment. The electronic device includes:

[0178] The processor 701 can be implemented in the form of a general-purpose CPU (Central Processing Unit), a microprocessor, an application-specific integrated circuit (ASIC), or one or more integrated circuits, etc., and is used to execute relevant programs to implement the technical solutions provided in the embodiments of the present application;

[0179] The memory 702 can be implemented in the form of a read-only memory (ROM), a static storage device, a dynamic storage device, or a random access memory (RAM), etc. The memory 702 can store an operating system and other application programs. When implementing the technical solutions provided in the embodiments of this specification through software or firmware, the relevant program codes are stored in the memory 702 and are called by the processor 701 to execute the methods in the embodiments of the present application;

[0180] The input / output interface 703 is used to implement information input and output;

[0181] The communication interface 704 is used to implement communication interaction between this device and other devices, and can communicate through wired means (such as USB, network cable, etc.) or wireless means (such as mobile network, WIFI, Bluetooth, etc.);

[0182] The bus 705 transmits information between the various components of the device (such as the processor 701, the memory 702, the input / output interface 703, and the communication interface 704);

[0183] Among them, the processor 701, the memory 702, the input / output interface 703, and the communication interface 704 achieve communication connections with each other inside the device through the bus 705.

[0184] The embodiments of the present application also provide a computer-readable storage medium, which stores a computer program, and when the computer program is executed by a processor, the method of the present application is implemented.

[0185] It can be understood that the content in the above method embodiments is applicable to the embodiments of this storage medium. The functions specifically implemented by the embodiments of this storage medium are the same as those in the above method embodiments, and the beneficial effects achieved are also the same as those in the above method embodiments.

[0186] The memory, as a non-transitory computer-readable storage medium, can be used to store non-transitory software programs and non-transitory computer-executable programs. In addition, the memory may include high-speed random access memory, and may also include non-transitory memory, such as at least one magnetic disk storage device, a flash memory device, or other non-transitory solid-state storage devices. In some embodiments, the memory may optionally include a memory remotely disposed relative to the processor, and these remote memories can be connected to the processor through a network. Examples of the above network include but are not limited to the Internet, an intranet, a local area network, a mobile communication network, and combinations thereof.

[0187] The embodiments described in the embodiments of the present application are for more clearly illustrating the technical solutions of the embodiments of the present application, and do not constitute a limitation on the technical solutions provided by the embodiments of the present application. Those skilled in the art will know that with the evolution of technology and the emergence of new application scenarios, the technical solutions provided by the embodiments of the present application are equally applicable to similar technical problems.

[0188] Those skilled in the art can understand that the technical solutions shown in the figures do not constitute a limitation on the embodiments of the present application, and may include more or fewer steps than those shown in the figures, or combine certain steps, or different steps.

[0189] The device embodiments described above are merely illustrative. The units described as separate components may or may not be physically separated, that is, they may be located in one place, or may be distributed to multiple network units. Some or all of the modules can be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0190] Those of ordinary skill in the art can understand that all or some of the steps in the methods disclosed above, and the functional modules / units in the systems and devices, can be implemented as software, firmware, hardware, and appropriate combinations thereof.

[0191] The terms "first", "second", "third", "fourth", etc. (if any) in the specification of the present application and the above drawings are used to distinguish similar objects, and do not have to be used to describe a specific order or sequence. It should be understood that such data can be interchanged under appropriate circumstances so that the embodiments of the present application described here can be implemented in an order other than those illustrated or described here. In addition, the terms "comprising" and "having" and any variations thereof are intended to cover non-exclusive inclusion. For example, a process, method, system, product, or device that includes a series of steps or units does not have to be limited to those steps or units clearly listed, but may include other steps or units not clearly listed or inherent to these processes, methods, products, or devices.

[0192] It should be understood that in this application, "at least one (item)" means one or more, and "a plurality" means two or more. "And / or" is used to describe the association relationship of associated objects and indicates that there can be three relationships. For example, "A and / or B" can mean: only A exists, only B exists, and both A and B exist at the same time. Among them, A and B can be singular or plural. The character " / " generally represents an "or" relationship between the associated objects before and after. "At least one (one)" or a similar expression below refers to any combination of these items, including any combination of single items (one) or plural items (ones). For example, at least one (one) of a, b, or c can mean: a, b, c, "a and b", "a and c", "b and c", or "a and b and c", where a, b, and c can be single or multiple.

[0193] In several embodiments provided in this application, it should be understood that the disclosed devices and methods can be implemented in other ways. For example, the device embodiments described above are merely illustrative. For example, the above division of units is only a logical function division. In actual implementation, there can be other division methods. For example, multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the displayed or discussed coupling or direct coupling or communication connection to each other can be through some interfaces. The indirect coupling or communication connection of devices or units can be in electrical, mechanical or other forms.

[0194] The units described above as separate components may or may not be physically separated. The components displayed as units may or may not be physical units, that is, they may be located in one place, or they may be distributed to multiple network units. Some or all of the units can be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0195] In addition, the functional units in each embodiment of this application can be integrated into one processing unit, or each unit can exist physically alone, or two or more units can be integrated into one unit. The above integrated unit can be implemented in the form of hardware or in the form of a software functional unit.

[0196] When the integrated unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on such an understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, or all or part of this technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes multiple instructions for causing a computer device (which may be a personal computer, a server, or a network device, etc.) to execute all or part of the steps of the methods of various embodiments of this application. The foregoing storage medium includes: various media that can store programs, such as USB flash drives, mobile hard disks, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical discs.

[0197] The preferred embodiments of the embodiments of this application have been described above with reference to the accompanying drawings, and thus do not limit the scope of the rights of the embodiments of this application. Any modifications, equivalent replacements, and improvements made by those skilled in the art without departing from the scope and essence of the embodiments of this application shall be within the scope of the rights of the embodiments of this application.

Claims

1. An ultrasonic robot trajectory planning method based on human key feature positioning, characterized in that: The method comprises the following steps: Detect the human body in the image acquired by the global camera and then output the detection frame of the human body; Predicting the position of key features of the human body based on the detection frame; wherein the key features of the human body include the human neck; Controlling the ultrasonic probe in the ultrasonic robot to move to the position of the human neck, and then calculating the position and posture of the ultrasonic probe as the initial position and posture; Calculating and obtaining the desired position and posture of the ultrasound probe at the key feature of the human body, and then controlling the ultrasound probe to move from the initial position and posture to the desired position and posture; The trajectory of the ultrasound probe is planned starting from an initial scanning position according to the key features of the human body; wherein the position of the ultrasound probe when it is in the desired posture is used as the initial scanning position.

2. The ultrasonic robot trajectory planning method based on human body key feature positioning according to claim 1 is characterized in that: The method of detecting a human body in an image acquired by a global camera and then outputting a detection frame of the human body comprises the following steps: ResNet is used as a detection backbone network, and a feature map of the image is extracted using the detection backbone network as a detection feature map; Using FPN as a detection neck network, and utilizing the detection neck network to perform feature fusion and enhancement on the detection feature map to obtain a multi-scale feature map; Faster R-CNN is used as the head detection network, and the RPN in the head detection network is used to generate a candidate region on the multi-scale feature map. The features of the candidate region are extracted through an Align operation, and then the features of the candidate region are classified and the detection frame is regressed to obtain the detection frame of the human body.

3. The ultrasonic robot trajectory planning method based on human body key feature positioning according to claim 1 is characterized in that: The method of predicting the position of a key feature of a human body based on the detection frame, wherein the key feature of the human body includes a human neck, comprises the following steps: ResNet is used as a prediction backbone network, and the prediction backbone network is used to extract features of a sub-image corresponding to the detection box in the image to obtain a prediction feature map; Using FMP as a prediction neck network, the prediction feature map is selected, spliced ​​and scaled by the prediction neck network to be converted into a format suitable for the prediction head network to obtain a conversion feature map; Outputting the predicted feature map and the key features of the human body in the converted feature map as corresponding heat maps using the predicted head network; wherein the value of each heat map is used to reflect the confidence of the corresponding key features of the human body at each position on the corresponding feature map; A decoded image including the key features of the human body is decoded according to the heat map, and then the position of the key features of the human body is predicted according to the decoded image.

4. The ultrasonic robot trajectory planning method based on human body key feature positioning according to claim 1 is characterized in that: The step of calculating the position and posture of the ultrasound probe as an initial position and posture comprises the following steps: Calculate the target normal vector passing through the position of the human body neck as the y-axis of the position of the ultrasound probe; Determine the feature points of the left and right shoulders of the human body according to the key features of the human body; Connect the feature points of the left and right shoulders of the human body; Determine a straight line that is parallel to the connecting line and passes through the position of the human body neck as the x-axis of the position of the ultrasound probe; The y-axis and x-axis of the ultrasound probe's posture are used as the initial posture.

5. The ultrasonic robot trajectory planning method based on human body key feature positioning according to claim 4 is characterized in that: The method of calculating the target normal vector passing through the position of the human body neck as the y-axis of the position and posture of the ultrasound probe comprises the following steps: Selecting a plurality of non-collinear target points within a preset range of the position of the human neck, and each of the target points is located in the region of the human neck; Acquiring depth information of each of the targets using the global camera; Forming a plane passing through the position of the human body neck according to the depth information; The target normal vector is calculated according to the position of the plane and the human neck as the y-axis of the position of the ultrasound probe.

6. The ultrasonic robot trajectory planning method based on human key feature positioning according to claim 1 is characterized in that: The method of solving and obtaining the desired position and posture of the ultrasound probe at the key feature of the human body includes the following steps: The quadratic programming problem is constructed as follows: Where x is the decision variable to be optimized, H is a symmetric positive definite quadratic matrix, f is a linear coefficient vector, (A b) is used to describe the inequality constraint, (A eq b eq ) is used to describe equality constraints, and (lb ub) is used to constrain the size of x; The quadratic programming problem is discretized according to the boundary conditions of the robotic arm of the ultrasonic robot, the end position of the robotic arm, the obstacle avoidance constraint conditions and the joint angle constraint conditions of the robotic arm to obtain a trajectory optimization problem; The trajectory optimization problem is: Among them, q is the joint angle variable to be optimized, (A obs b obs ) is used to describe obstacle constraints, (A bd b bd ) is used to describe boundary constraints, (q min q max ) is used to describe joint angle constraints; Obtaining the desired joint angles of each joint in the robotic arm according to the solution of the trajectory optimization problem; The expected posture is determined according to each of the expected joint angles.

7. The ultrasonic robot trajectory planning method based on human body key feature positioning according to any one of claims 1 to 6, characterized in that: The step of planning the trajectory of the ultrasound probe from an initial scanning position according to the key features of the human body comprises the following steps: Starting from the initial scanning position, scanning and sampling are performed in the same direction to obtain a plurality of scanning points, and a scanning trajectory is obtained by using a polynomial interpolation method; Calculating the desired probe posture of the ultrasound probe at each of the scanning points according to the depth information of the points acquired by the global camera; Controlling the ultrasonic robot to perform scanning operations according to each of the desired probe postures; The ultrasonic probe is rotated 90 degrees, and the scanning sampling is returned to the initial scanning position along the same direction to obtain a plurality of scanning points, and a scanning trajectory is obtained by using a polynomial interpolation method; During the scanning process of the ultrasonic probe, a random walk algorithm is used to solve the confidence of pixels in the image taken by the global camera; when the confidence of pixels in any area is lower than a set threshold, the position and posture of the ultrasonic probe is adjusted; During the scanning process of the ultrasound probe, the image taken by the global camera is segmented using the nnU-Net model to obtain the thyroid region; and the position of the ultrasound probe is dynamically adjusted according to the segmented thyroid region.

8. An ultrasonic robot trajectory planning device based on human body key feature positioning, characterized in that: The device comprises: A human body detection unit, used to detect a human body in an image acquired by a global camera and then output a detection frame of the human body; A feature prediction unit, used to predict the position of a key feature of a human body based on the detection frame; wherein the key feature of a human body includes a human neck; A posture solving unit is used to control the ultrasonic probe in the ultrasonic robot to move to the position of the human neck, and then solve the posture of the ultrasonic probe as the initial posture; A probe moving unit, used for calculating the desired position and posture of the ultrasound probe at the key feature of the human body, and then controlling the ultrasound probe to move from the initial position and posture to the desired position and posture; A trajectory planning unit is used to plan the trajectory of the ultrasound probe starting from an initial scanning position according to the key features of the human body; wherein the position of the ultrasound probe when it is in the desired posture is used as the initial scanning position.

9. An electronic device, characterized in that: The electronic device comprises a memory and a processor, the memory stores a computer program, and the processor implements the method according to any one of claims 1 to 7 when executing the computer program.

10. A computer-readable storage medium storing a computer program, characterized in that: When the computer program is executed by a processor, the method according to any one of claims 1 to 7 is implemented.

Images