Mechanical arm pose adjusting method and device based on ultrasonic image feedback

Abstract

The invention discloses a mechanical arm pose adjusting method and device based on ultrasonic image feedback, and the method comprises the steps: reading an ultrasonic image, obtaining an ultrasonic detection original point and an ultrasonic image mass center of the ultrasonic image, and calculating a deviation angle of a mechanical arm based on the ultrasonic detection original point and the ultrasonic image mass center; and the deviation angle is fed back to the mechanical arm so that the mechanical arm can conduct pose adjustment. According to the method, the deviation angle of the current image is estimated through the collected ultrasonic image, and the deviation angle is fed back to the mechanical arm in real time, so that the pose can be dynamically adjusted during scanning of the mechanical arm, the organ scanned by the ultrasonic image can be placed at the most ideal position, and a better ultrasonic image is obtained.

Description

Technical Field

[0001] The present application relates to the technical field of automatic ultrasonic scanning, and in particular to a method and device for adjusting the posture of a robotic arm based on ultrasonic image feedback. Background Art

[0002] When a target uses a robotic arm for automatic ultrasound scanning, it is common to plan an automatic scanning path in advance before the automatic scanning, and then the robotic arm drives the ultrasound probe to perform automatic ultrasound scanning along the automatic scanning path. However, when using a robotic arm for automatic ultrasound scanning, slight movements of the body may cause deviations from the scanning path, thereby affecting the image quality of the acquired ultrasound image.

[0003] Therefore the prior art still needs to be improved and enhanced. Summary of the invention

[0004] The technical problem to be solved by the present application is to provide a method and device for adjusting the posture of a robotic arm based on ultrasonic image feedback in view of the deficiencies in the prior art.

[0005] In order to solve the above technical problems, the first aspect of the present application provides a method for adjusting the posture of a robotic arm based on ultrasonic image feedback, wherein the method for adjusting the posture of a robotic arm based on ultrasonic image feedback specifically includes:

[0006] Reading an ultrasonic image, and obtaining an ultrasonic detection origin and a centroid of the ultrasonic image, wherein the ultrasonic image is collected by automatic scanning according to a pre-planned scanning path;

[0007] Determine a centroid line of the ultrasonic image based on the centroid of the ultrasonic image and the ultrasonic detection origin, and determine a center line of the ultrasonic image based on the ultrasonic detection origin;

[0008] The positional relationship between the centroid line of the ultrasound image and the center line of the ultrasound image is calculated, and the scanning path is adjusted based on the positional relationship.

[0009] In the method for adjusting the posture of a robotic arm based on ultrasonic image feedback, the process of obtaining the ultrasonic detection origin specifically includes:

[0010] Performing filtering processing on the ultrasonic image to obtain a filtered ultrasonic image;

[0011] Performing edge detection on the filtered ultrasonic image to obtain image edges;

[0012] An ultrasound detection origin of the ultrasound image is determined based on the image edge.

[0013] The method for adjusting the position and posture of a robotic arm based on ultrasonic image feedback, wherein, before performing edge detection on the filtered ultrasonic image to obtain the image edge, the method further comprises:

[0014] Extracting target image points whose grayscale values ​​are within a first grayscale range in the filtered ultrasonic image;

[0015] Redistributing the grayscale value of the extracted target image point to a second grayscale range, wherein an upper grayscale value of the second grayscale range is greater than an upper grayscale value of the first grayscale range;

[0016] The grayscale value of each image point except the target image point in the filtered ultrasonic image is reallocated as the upper limit grayscale value of the second grayscale range.

[0017] In the method for adjusting the posture of a robotic arm based on ultrasonic image feedback, the step of performing edge detection on the filtered ultrasonic image to obtain the image edge specifically comprises:

[0018] Performing Canny edge detection on the filtered ultrasonic image to obtain an image edge set, and selecting an upper left corner straight line and an upper right corner straight line from the image edge set;

[0019] If the upper left straight line and the upper right straight line are selected, the upper left straight line and the upper right straight line are used as the image edge of the ultrasound image;

[0020] If the upper left straight line is selected, the upper right straight line is drawn according to the selected upper left straight line, and the upper left straight line and the drawn upper right straight line are used as the image edge of the ultrasound image;

[0021] If the upper right straight line is selected, the upper left straight line is drawn according to the selected upper right straight line, and the drawn upper left straight line and upper right straight line are used as the image edge of the ultrasound image;

[0022] If the upper left straight line and the upper right straight line are not selected, it will prompt that the image edge detection is wrong.

[0023] The method for adjusting the position and posture of a robotic arm based on ultrasonic image feedback, wherein the process of obtaining the centroid of the ultrasonic image specifically includes:

[0024] Acquire the total image brightness, the first grayscale value weighted distribution in the x-direction, and the second grayscale value weighted distribution in the y-direction of the ultrasound image;

[0025] Calculating the ratio of the total brightness of the image to the weighted distribution of the first grayscale value to obtain the centroid coordinates in the x direction;

[0026] The ratio of the total brightness of the image to the weighted distribution of the second grayscale value is calculated to obtain the centroid coordinates in the y direction.

[0027] In the method for adjusting the posture of a robotic arm based on ultrasonic image feedback, the step of calculating the positional relationship between the centroid line of the ultrasonic image and the center line of the ultrasonic image specifically comprises:

[0028] Calculating the angle between the centroid line of the ultrasonic image and the center line of the ultrasonic image;

[0029] Based on the angle, the offset direction and angle of the centroid relative to the center line of the ultrasound image are determined to obtain the positional relationship between the centroid of the ultrasound image and the center line of the ultrasound image, wherein the offset direction is that the centroid is located on the left side of the center line or the centroid is located on the right side of the center line.

[0030] The method for adjusting the position and posture of a robotic arm based on ultrasonic image feedback, wherein the process of adjusting the scanning path based on the position relationship specifically includes:

[0031] Reading the next path point of the path point corresponding to the ultrasound image;

[0032] The rotation angle in the x direction of the next path point is adjusted based on the positional relationship to adjust the robot arm posture of the next path point.

[0033] The second aspect of the present application provides a robotic arm posture adjustment device based on ultrasonic image feedback, wherein the robotic arm posture adjustment device based on ultrasonic image feedback specifically includes:

[0034] A reading module, used to read an ultrasonic image and obtain an ultrasonic detection origin and a centroid of the ultrasonic image, wherein the ultrasonic image is collected by automatically scanning according to a pre-planned scanning path;

[0035] a determination module, configured to determine a centroid line of the ultrasonic image based on the centroid of the ultrasonic image and the ultrasonic detection origin, and to determine a center line of the ultrasonic image based on the ultrasonic detection origin;

[0036] An adjustment module is used to calculate the positional relationship between the centroid line of the ultrasound image and the center line of the ultrasound image, and adjust the scanning path based on the positional relationship.

[0037] A third aspect of the present application provides a computer-readable storage medium, which stores one or more programs, and the one or more programs can be executed by one or more processors to implement the steps in any of the above-described methods for adjusting the posture of a robotic arm based on ultrasonic image feedback.

[0038] A fourth aspect of the present application provides a terminal device, comprising: a processor and a memory;

[0039] The memory stores a computer-readable program executable by the processor;

[0040] When the processor executes the computer-readable program, the steps in any of the above-mentioned methods for adjusting the posture of a robotic arm based on ultrasonic image feedback are implemented.

[0041] Beneficial effects: Compared with the prior art, the present application provides a method and device for adjusting the posture of a robotic arm based on ultrasonic image feedback, the method comprising reading an ultrasonic image, and obtaining the ultrasonic detection origin and the ultrasonic image centroid of the ultrasonic image; determining the centroid line of the ultrasonic image based on the ultrasonic image centroid and the ultrasonic detection origin, and determining the center line of the ultrasonic image based on the ultrasonic detection origin; calculating the positional relationship between the centroid line of the ultrasonic image and the center line of the ultrasonic image, and adjusting the scanning path based on the positional relationship. In the process of robotic arm scanning, the present application obtains the ultrasonic detection origin and the ultrasonic image centroid of the ultrasonic image after the ultrasonic image is acquired, and determines the deviation angle of the ultrasonic image based on the ultrasonic detection origin and the ultrasonic image centroid, and feeds back the deviation angle to the robotic arm in real time, so that the probe posture can be dynamically adjusted during the robotic arm scanning, so that the organ scanned by the ultrasonic image can be placed in the most ideal position, thereby improving the image quality of the acquired ultrasonic image. BRIEF DESCRIPTION OF THE DRAWINGS

[0042] In order to more clearly illustrate the technical solutions in the embodiments of the present application, the drawings required for use in the description of the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without creative work.

[0043] Figure 1 A flowchart of a method for adjusting the posture of a robotic arm based on ultrasound image feedback provided in an embodiment of the present application.

[0044] Figure 2 Schematic diagram of obtaining the target straight line from edge detection.

[0045] Figure 3 Schematic diagram of an ultrasound image after grayscale redistribution.

[0046] Figure 4 Schematic diagram of the centroid and center line.

[0047] Figure 5 Schematic diagram of the probe coordinate system.

[0048] Figure 6This is a principle block diagram of a robotic arm posture adjustment device based on ultrasonic image feedback provided in an embodiment of the present application.

[0049] Figure 7 A functional block diagram of a terminal device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0050] The embodiment of the present application provides a method and device for adjusting the posture of a robotic arm based on ultrasonic image feedback. In order to make the purpose, technical solution and effect of the present application clearer and more specific, the present application is further described in detail with reference to the accompanying drawings and examples. It should be understood that the specific embodiments described herein are only used to explain the present application and are not used to limit the present application.

[0051] It will be understood by those skilled in the art that, unless expressly stated, the singular forms "one", "said", and "the" used herein may also include plural forms. It should be further understood that the term "comprising" used in the specification of the present application refers to the presence of the features, integers, steps, operations, elements and / or components, but does not exclude the presence or addition of one or more other features, integers, steps, operations, elements, components and / or groups thereof. It should be understood that when we refer to an element as being "connected" or "coupled" to another element, it may be directly connected or coupled to the other element, or there may be an intermediate element. In addition, the "connection" or "coupling" used herein may include wireless connection or wireless coupling. The term "and / or" used herein includes all or any unit and all combinations of one or more associated listed items.

[0052] It will be understood by those skilled in the art that, unless otherwise defined, all terms (including technical and scientific terms) used herein have the same meaning as those generally understood by those skilled in the art to which this application belongs. It should also be understood that terms such as those defined in general dictionaries should be understood to have meanings consistent with those in the context of the prior art, and will not be interpreted with idealized or overly formal meanings unless specifically defined as here.

[0053] It should be understood that the sequence numbers and sizes of the steps in this embodiment do not mean the order of execution. The execution order of each process is determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiment of the present application.

[0054] After research, it was found that when the target uses a robotic arm for automatic ultrasound scanning, the automatic scanning path is generally planned in advance before the automatic scanning, and then the robotic arm drives the ultrasound probe to perform automatic ultrasound scanning along the automatic scanning path. However, when using a robotic arm for automatic ultrasound scanning, slight movements of the body will cause deviations from the scanning path, thereby affecting the image quality of the acquired ultrasound image.

[0055] In order to solve the above problems, in an embodiment of the present application, an ultrasonic image is read, and the ultrasonic detection origin and the ultrasonic image centroid of the ultrasonic image are obtained; the centroid line of the ultrasonic image is determined based on the ultrasonic image centroid and the ultrasonic detection origin, and the center line of the ultrasonic image is determined based on the ultrasonic detection origin; the positional relationship between the centroid line of the ultrasonic image and the center line of the ultrasonic image is calculated, and the scanning path is adjusted based on the positional relationship. In the present application, during the robotic arm scanning process, after the ultrasonic image is acquired, the ultrasonic detection origin and the ultrasonic image centroid of the ultrasonic image are acquired, and the deviation angle of the ultrasonic image is determined based on the ultrasonic detection origin and the ultrasonic image centroid, and the deviation angle is fed back to the robotic arm in real time, so that the probe posture can be dynamically adjusted during the robotic arm scanning, so that the organ scanned by the ultrasonic image can be placed in the most ideal position, thereby improving the image quality of the acquired ultrasonic image.

[0056] The application content is further explained below through the description of embodiments in conjunction with the accompanying drawings.

[0057] This embodiment provides a method for adjusting the position and posture of a robotic arm based on ultrasound image feedback. Figure 1 As shown, the method includes:

[0058] S10, reading an ultrasonic image, and obtaining an ultrasonic detection origin and a centroid of the ultrasonic image.

[0059] Specifically, the ultrasound image is acquired by automatically scanning according to a pre-planned scanning path, for example, the ultrasound image is acquired by automatically scanning the abdomen with the ultrasound probe driven by a mechanical arm, or the liver with the ultrasound probe driven by a mechanical arm, etc. The ultrasound probe will perform ultrasound acquisition at each path point in the pre-planned scanning path, that is, the ultrasound probe is automatically scanned according to the pre-planned scanning path driven by the mechanical arm, and each time an ultrasound image is acquired, the ultrasound image will be read so as to adjust the ultrasound probe posture of the next path point based on the ultrasound image, wherein the ultrasound probe may be a convex array probe, the imaging area of ​​the convex array probe is a fan-shaped structure, the upper and lower edges are arc-shaped, and the left and right edges are straight lines, extending from the upper and lower arc-shaped edges of the ultrasound image to the left and right to form the side of the fan.

[0060] The scanning path is pre-planned. Here, the scanning path planning process is explained by taking the automatic scanning of the liver as an example. The path planning process of the automatic scanning of the liver can be:

[0061] H10, obtaining the human body point cloud data and human body image of the object to be scanned;

[0062] H20. Determine a human body surface based on the human body point cloud data, and obtain a normal vector of each surface point in the human body surface;

[0063] H30, determining the liver region of the object to be scanned based on the human body image, and constructing an initial scanning path for the object to be scanned based on the liver region;

[0064] H40. Based on the normal vector of each surface point in the human body surface, determine the probe posture corresponding to each path point in the initial scanning path, and add the probe posture to the path point to obtain the target scanning path of the object to be scanned.

[0065] Specifically, the human body point cloud data includes point cloud data of the human body surface corresponding to the liver area, and the human body image includes the liver area. The human body point cloud data is obtained by scanning the object to be scanned, for example, scanning the object to be scanned by a kinect camera to obtain the human body point cloud data of the object to be scanned. The human body image is the RGB-D image data of the object to be scanned, for example, the image of the human body is obtained by taking an image of the object to be scanned by a depth camera. The human body surface is obtained by reconstructing the human body through the human body point cloud data, wherein the human body surface is a 3D model of the human body. For example, the human body surface is processed by the PCL library to obtain the normal vector of the surface position of each point cloud data in the human body point cloud data of the object to be scanned.

[0066] Further, the liver region can be determined based on the correspondence between the rib position and the liver, wherein the rib position can be obtained through a trained human segmentation model. That is, the human image will first be input into the trained human segmentation model, and the rib position of the object to be scanned will be output through the human segmentation model; then, according to the correspondence between the rib position and the liver, the liver region of the object to be scanned is determined based on the rib position of the object to be scanned. Wherein, the human segmentation model is a trained deep learning model (e.g., CNN, DNN, etc.) for identifying the rib position in the human image. The correspondence between the rib position and the liver is that the upper boundary is located in the gap between the 5th and 6th ribs on the right, the lower boundary of the liver is located at the 12th rib at the level of the right midaxillary line, the right lower boundary liver width extends from the rib edge to the junction of the 7th and 8th ribs on the right, and extends obliquely from the junction to the upper left to the 6th rib on the left, the left boundary is close to the liver at the edge of the left rib, and the right boundary is below the nipple. Therefore, after the rib positions are acquired, the liver region of the object to be scanned can be outlined based on the correspondence between the rib positions and the liver, wherein the liver region is distributed in a right-angle trapezoid.

[0067] Further, the initial scanning path includes a transverse scanning path, a longitudinal scanning path, and a lateral scanning path, and each path point in the transverse scanning path, the longitudinal scanning path, and the lateral scanning path includes position information, wherein the position information includes an x-direction coordinate, a y-direction coordinate, and a z-direction coordinate. The transverse scanning path and the longitudinal scanning path are used to observe liver lesions, and the lateral scanning path is used to evaluate the liver vascular condition. The starting position of the lateral scanning path is the xiphoid process position, and the lateral scanning path extends from the xiphoid process position along the lower edge of the right rib to the lowest end of the right rib. Specifically, the starting point of the transverse scanning path is a regional vertex of the liver region, and the starting point of the longitudinal scanning path is also a regional vertex of the liver region, and the starting point of the transverse scanning path may be different from the starting point of the longitudinal scanning path. For example, the starting point of the transverse scanning path is the upper left vertex of the liver region, and the starting point of the longitudinal scanning path is the upper right vertex of the liver region. Of course, the starting point of the transverse scanning path and the starting point of the longitudinal scanning path can be the same. For example, the starting point of the transverse scanning path and the starting point of the longitudinal scanning path are both the upper left vertices of the right-angled trapezoidal area.

[0068] Furthermore, in order to minimize the distance that the robot moves during actual operation, when constructing the initial scanning path, the width of the ultrasound probe can also be determined according to the liver area. Based on this, when constructing the initial scanning path, the transverse width and longitudinal width of the liver area can be first obtained; then based on the transverse width and longitudinal width, the width of the ultrasound probe used for ultrasound examination of the object to be scanned is determined. Among them, the transverse width is the maximum distance of the liver area in the transverse direction, and the longitudinal width is the maximum distance of the liver area in the longitudinal direction. The ultrasound probe width can be based on the greatest common divisor of the transverse width and the longitudinal width, for example, the greatest common divisor is used as the ultrasound probe width, or 1 / n of the greatest common divisor is used as the ultrasound probe width, or n times the greatest common divisor is used as the ultrasound probe width, and n is a positive integer greater than 1. In addition, the first distance between two adjacent rows of transverse scanning paths in the transverse scanning path is equal to the ultrasound probe width; the second distance between two adjacent rows of longitudinal scanning paths in the longitudinal scanning path is equal to the ultrasound probe width. Of course, in actual applications, the first distance between two adjacent rows of the transverse scanning path in the transverse scanning path and the second distance between two adjacent rows of the longitudinal scanning path in the longitudinal scanning path are both smaller than the width of the ultrasound probe, and the difference between the width of the ultrasound probe and the first distance and the difference between the width of the ultrasound probe and the second distance both meet the preset conditions (for example, smaller than the preset difference threshold, etc.).

[0069] Furthermore, when determining the probe posture corresponding to each path point in the initial scanning path based on the normal vector of each surface point in the human body surface, the surface point corresponding to the path point can be obtained, and the normal vector direction of the surface point is used as the Z-axis direction of the path point; the moving direction of the path point is read, and the moving direction is used as the X-axis direction of the path point; the Y-axis direction of the path point is determined based on the X-axis direction and the Z-axis direction to obtain the probe coordinate system of the ultrasound probe at the path point, and then the rotation matrix between the probe coordinate system and the global coordinate system of the liver region is determined, and the probe posture corresponding to the path point is determined based on the rotation matrix.

[0070] It should be noted that, in actual applications, when different parts are automatically scanned, the scanning path may be constructed in different ways. Of course, the scanning path for automatic scanning may also be manually set by the operator based on his or her own experience.

[0071] The ultrasonic detection origin is a specific point on the ultrasonic detection surface, which is used as the starting reference position for measurement, and helps to improve the accuracy and consistency of measurement. Exemplarily, the process of obtaining the ultrasonic detection origin specifically includes:

[0072] S11, performing filtering processing on the ultrasonic image to obtain a filtered ultrasonic image;

[0073] S12, performing edge detection on the filtered ultrasonic image to obtain image edges;

[0074] S13. Determine an ultrasound detection origin of the ultrasound image based on the image edge.

[0075] Specifically, in step S11, the filtering process may filter the ultrasound image with a Gaussian filter, thereby reducing noise or interference to the internal structure of certain organs, and better maintaining the structural integrity of the image, thereby improving the accuracy and reliability of subsequent analysis.

[0076] Further, in step S12, edge detection is used to detect edge data in the filtered ultrasound image, and the area where the brightness changes sharply in the ultrasound image is captured by edge detection to identify the edge straight lines in the ultrasound image and / or the side edges of the fan-shaped non-image, for example, the edge between some gas and substance in the ultrasound detection organ.

[0077] For example, Figure 2 As shown, the edge detection of the filtered ultrasonic image to obtain the image edge specifically includes:

[0078] Performing Canny edge detection on the filtered ultrasonic image to obtain an image edge set, and selecting an upper left corner straight line and an upper right corner straight line from the image edge set;

[0079] If the upper left straight line and the upper right straight line are selected, the upper left straight line and the upper right straight line are used as the image edge of the ultrasound image;

[0080] If the upper left straight line is selected, the upper right straight line is drawn according to the selected upper left straight line, and the upper left straight line and the drawn upper right straight line are used as the image edge of the ultrasound image;

[0081] If the upper right straight line is selected, the upper left straight line is drawn according to the selected upper right straight line, and the drawn upper left straight line and upper right straight line are used as the image edge of the ultrasound image;

[0082] If the upper left straight line and the upper right straight line are not selected, it will prompt that the image edge detection is wrong.

[0083] Specifically, the Canny edge detection can be performed using the Canny function in OpenCV, and the Canny edge detection is used to capture the area where the brightness in the filtered ultrasound image changes sharply, so as to obtain the image edge set in the filtered ultrasound image. After the image edge set is obtained, the image edge set is detected to select the upper left corner straight line and the upper right corner straight line from the image edge set, wherein the process of selecting the upper left corner straight line and the upper right corner straight line from the image edge set can be: firstly detect the candidate straight lines in the image edge set, and then select the straight lines located in the upper left corner and the upper right corner of the image from the detected candidate straight lines, so as to obtain the upper left corner straight line and the upper right corner straight line.

[0084] Furthermore, when detecting candidate straight lines in the image edge set, the HoughLinesP function in OpenCV can be used to perform cumulative probability Hough transform to detect candidate straight lines according to preset straight line conditions, where the preset straight line conditions are that the straight line length is greater than a first threshold, the straight line gap is less than or equal to a second threshold, and the first threshold is greater than the second threshold. The embodiment of the present application pre-sets the range of straight line length and straight line gap to filter the straight lines inside the organs in the ultrasound image, so that the side area straight lines can be better detected, thereby better detecting the regional edge straight lines of the fan-shaped inspection area of ​​the ultrasound probe.

[0085] When a convex array ultrasound probe is used to collect ultrasound images, the ultrasound detection origin of the ultrasound image can be determined by the upper left straight line or the upper right straight line of the fan-shaped inspection area of ​​the convex array ultrasound probe. For this reason, after obtaining the candidate straight lines, the upper left straight line and the upper right straight line in the candidate straight lines can be screened, and then the ultrasound detection origin is determined based on the upper left straight line and the upper right straight line. In addition, for some ultrasound images with edge gray values, only the upper left straight line or the upper right straight line may be detected. For this reason, when only the upper left straight line or the upper right straight line is detected, the upper left straight line or the upper right straight line is symmetrical with the image center line to draw the upper right straight line corresponding to the upper left straight line and the upper left straight line corresponding to the upper right straight line. Of course, in some cases, the upper left straight line and the upper right straight line may not be identified, and an image edge detection error may be prompted at this time.

[0086] Furthermore, during edge detection, for portions with low grayscale values, especially portions with grayscale values ​​close to 0, it is easy to miss detection, which in turn affects the accuracy of the ultrasonic detection origin. To this end, in one implementation, before performing edge detection on the filtered ultrasonic image to obtain the image edge, the method further includes:

[0087] Extracting target image points whose grayscale values ​​are within a first grayscale range in the filtered ultrasonic image;

[0088] Redistributing the extracted grayscale value of the target image point to a second grayscale range;

[0089] The grayscale value of each image point except the target image point in the filtered ultrasonic image is reallocated as the upper limit grayscale value of the second grayscale range.

[0090] Specifically, the first grayscale range and the second grayscale range are both pre-set, the upper grayscale value of the second grayscale range is greater than the upper grayscale value of the first grayscale range, and the lower grayscale value of the second grayscale range and the lower grayscale value of the first grayscale range are both 0. In this way, the grayscale values ​​in the first grayscale range can be redistributed to the second grayscale range, the edge detection effect of the low grayscale value portion can be improved, and the subtle changes of the ultrasound image can be made more obvious, which can further improve the edge detection effect of the low grayscale value.

[0091] In a specific embodiment, since the gray value is above 100, it is easy to detect the edge through the edge. Figure 3As shown, the first grayscale range can be set to 0-100, the second grayscale range can be set to 0-255, and for image points with grayscale values ​​greater than 100, the grayscale value of the image point is reallocated to 255, and then for image points with grayscale values ​​between 0-100, the grayscale value of the image point is reallocated to 0-255. In this specific embodiment, by reallocating image points with grayscale values ​​between 0-100 to 0-255, the contrast of these image points can be effectively enhanced, and subtle changes can be made more obvious, thereby improving the edge detection effect of low grayscale values. In addition, since image points with grayscale values ​​higher than 100 have no effect on edge detection, these points can be directly reallocated to 255.

[0092] Further, in step S13, after acquiring the image edge, the upper left straight line and the upper right straight line in the image edge can be extended to determine the focus of the upper left straight line and the upper right straight line, and then the focus is used as the ultrasound detection origin of the ultrasound image.

[0093] The above is a description of the detection process of the ultrasonic detection origin. The following is a description of the process of obtaining the centroid of the ultrasonic image. Specifically, the process of obtaining the centroid of the ultrasonic image includes:

[0094] Acquire the total image brightness, the first grayscale value weighted distribution in the x-direction, and the second grayscale value weighted distribution in the y-direction of the ultrasound image;

[0095] Calculating the ratio of the total brightness of the image to the weighted distribution of the first grayscale value to obtain the centroid coordinates in the x direction;

[0096] The ratio of the total brightness of the image to the weighted distribution of the second grayscale value is calculated to obtain the centroid coordinates in the y direction.

[0097] Specifically, the sum of all pixel values ​​in the ultrasound image is first calculated to obtain the total brightness of the image, wherein the calculation formula for the total brightness of the image is:

[0098]

[0099] Among them, M 00 represents the total brightness of the image, W represents the width of the ultrasound image, H represents the height of the ultrasound image, and G(x,y) represents the grayscale value of the image point (x,y).

[0100] Secondly, the first grayscale value weighted distribution in the x direction and the second grayscale value weighted distribution in the y direction are calculated, wherein the calculation formulas of the first grayscale value weighted distribution and the second grayscale value weighted distribution are respectively:

[0101]

[0102] Among them, M10 represents the first gray value weighted distribution, M 01 represents the second gray value weighted distribution.

[0103] Finally, the ratios of the total brightness of the image to the first grayscale value weighted distribution and the second grayscale value weighted distribution are calculated to obtain the centroid coordinates in the x direction and the centroid coordinates in the y direction, wherein the calculation formulas of the centroid coordinates in the x direction and the centroid coordinates in the y direction are respectively:

[0104]

[0105] Among them, C x represents the coordinates of the center of mass in the x direction, C y Represents the centroid coordinate in the y direction.

[0106] S20. Determine a centroid line of the ultrasonic image based on the centroid of the ultrasonic image and the ultrasonic detection origin, and determine a center line of the ultrasonic image based on the ultrasonic detection origin.

[0107] Specifically, the centroid line of the ultrasonic image is the line connecting the ultrasonic detection origin and the ultrasonic image centroid, and the center line of the ultrasonic image is the line connecting the ultrasonic detection origin and the image center point of the ultrasonic image. That is to say, after obtaining the ultrasonic image centroid and the ultrasonic detection origin, the image centerline point of the ultrasonic image can be read, and then the ultrasonic detection origin is connected with the ultrasonic image centroid to form the centroid line of the ultrasonic image, and the ultrasonic detection origin is connected with the image center point to form the center line of the ultrasonic image.

[0108] S30, calculating the positional relationship between the centroid line of the ultrasound image and the center line of the ultrasound image, and adjusting the scanning path based on the positional relationship.

[0109] Specifically, the positional relationship is used to reflect the offset information of the centroid line of the ultrasound image relative to the center line of the ultrasound image, wherein the positional relationship includes an offset direction and an offset angle, and the offset direction is used to reflect that the centroid line is located on the left or right side of the center line, that is, the offset direction is that the centroid line is located on the left side of the center line or the centroid line is located on the right side of the center line.

[0110] Exemplarily, the calculating the positional relationship between the centroid line of the ultrasound image and the center line of the ultrasound image specifically includes:

[0111] Calculating the angle between the centroid line of the ultrasonic image and the center line of the ultrasonic image;

[0112] Based on the angle, the offset direction and angle of the centroid relative to the center line of the ultrasound image are determined to obtain the positional relationship between the centroid of the ultrasound image and the center line of the ultrasound image, wherein the offset direction is that the centroid is located on the left side of the center line or the centroid is located on the right side of the center line.

[0113] Specifically, the angle between the centroid of the ultrasound image and the center line of the ultrasound image can be calculated according to trigonometric functions. After the angle is calculated, the angle can be compared with 180°. If it is less than 180°, it means that the centroid is located on the left side of the center line. If it is greater than 180°, it means that the centroid is located on the right side of the center line. Of course, if the angle is equal to 180°, it can be determined that the centroid is located on the left side of the center line, or it can be determined that the centroid is located on the right side of the center line, which is not limited here.

[0114] Further, after determining the offset direction, if the offset direction is such that the centroid is located on the left side of the center line, the included angle is used as the offset angle; if the offset direction is such that the centroid is located on the right side of the center line, the difference between 360 and the included angle is used as the offset angle to obtain the positional relationship between the centroid of the ultrasound image and the center line of the ultrasound image. For example, Figure 4 As shown, the calculated included angle is 353.27°, and the positional relationship between the centroid line of the ultrasound image and the center line of the ultrasound image is that the centroid line is located 6.73° to the right of the center line.

[0115] Exemplarily, the process of adjusting the scanning path based on the position relationship specifically includes:

[0116] Reading the next path point of the path point corresponding to the ultrasound image;

[0117] The rotation angle in the x direction of the next path point is adjusted based on the positional relationship to adjust the robot arm posture of the next path point.

[0118] Specifically, after obtaining the positional relationship between the centroid of the ultrasound image and the centerline of the ultrasound image, the positional relationship is fed back to the robot arm posture, so that the robot arm is adjusted in real time, that is, the rotation angle in the x direction in the next path point is adjusted. Figure 5 As shown in the figure, the Z axis must be perpendicular to the human body surface during scanning. To this end, it is only necessary to adjust the rx in the robot arm posture (x, y, z, rx, ry, rz) based on the partial position relationship to make the center of mass line of the ultrasound image close to the midline, thereby making the ultrasound image clearer.

[0119] In summary, this embodiment provides a method and device for adjusting the posture of a robotic arm based on ultrasonic image feedback, the method comprising reading an ultrasonic image, obtaining the ultrasonic detection origin and the ultrasonic image centroid of the ultrasonic image, calculating the offset angle of the robotic arm based on the ultrasonic detection origin and the ultrasonic image centroid, and feeding back the offset angle to the robotic arm so that the robotic arm can adjust its posture. The present application estimates the offset angle of the current image through the collected ultrasonic image, and feeds back the offset angle to the robotic arm in real time, so that the posture of the robotic arm can be dynamically adjusted during scanning, so that the organ scanned by the ultrasonic image can be placed in the most ideal position to obtain a better ultrasonic image.

[0120] Based on the above-mentioned method for adjusting the position and posture of a robotic arm based on ultrasonic image feedback, this embodiment provides a device for adjusting the position and posture of a robotic arm based on ultrasonic image feedback, such as Figure 6 As shown, the robotic arm posture adjustment device based on ultrasonic image feedback specifically includes:

[0121] The reading module 100 is used to read the ultrasonic image and obtain the ultrasonic detection origin and the ultrasonic image centroid of the ultrasonic image, wherein the ultrasonic image is collected by automatic scanning according to a pre-planned scanning path;

[0122] A determination module 200, configured to determine a centroid line of the ultrasound image based on the ultrasound image centroid and the ultrasound detection origin, and to determine a center line of the ultrasound image based on the ultrasound detection origin;

[0123] The adjustment module 300 is used to calculate the positional relationship between the centroid line of the ultrasound image and the center line of the ultrasound image, and adjust the scanning path based on the positional relationship.

[0124] Based on the above-mentioned method for adjusting the posture of a robotic arm based on ultrasonic image feedback, this embodiment provides a computer-readable storage medium, which stores one or more programs. The one or more programs can be executed by one or more processors to implement the steps in the method for adjusting the posture of a robotic arm based on ultrasonic image feedback as described in the above-mentioned embodiment.

[0125] Based on the above-mentioned method for adjusting the position and posture of a robotic arm based on ultrasonic image feedback, the present application also provides a terminal device, such as Figure 7As shown, it includes at least one processor (processor) 20; display screen 21; and memory (memory) 22, and may also include a communications interface (Communications Interface) 23 and a bus 24. Among them, the processor 20, the display screen 21, the memory 22 and the communication interface 23 can communicate with each other through the bus 24. The display screen 21 is configured to display a preset user guide interface in the initial setting mode. The communication interface 23 can transmit information. The processor 20 can call the logic instructions in the memory 22 to execute the method in the above embodiment.

[0126] In addition, the logic instructions in the memory 22 can be implemented in the form of software functional units and can be stored in a computer-readable storage medium when sold or used as an independent product.

[0127] The memory 22 is a computer-readable storage medium that can be configured to store software programs, computer executable programs, such as program instructions or modules corresponding to the methods in the embodiments of the present disclosure. The processor 20 executes functional applications and data processing by running the software programs, instructions or modules stored in the memory 22, that is, implementing the methods in the above embodiments.

[0128] The memory 22 may include a program storage area and a data storage area, wherein the program storage area may store an operating system and at least one application required for a function; the data storage area may store data created according to the use of the terminal device, etc. In addition, the memory 22 may include a high-speed random access memory and may also include a non-volatile memory. For example, a variety of media that can store program codes, such as a USB flash drive, a mobile hard disk, a read-only memory (ROM), a random access memory (RAM), a disk or an optical disk, may also be a transient storage medium.

[0129] In addition, the specific process of loading and executing the multiple instruction processors in the above-mentioned storage medium and the terminal device has been described in detail in the above-mentioned method, and will not be described one by one here.

[0130] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present application, rather than to limit it. Although the present application has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or make equivalent replacements for some of the technical features therein. However, these modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the embodiments of the present application.

Claims

1. A method for adjusting the posture of a robotic arm based on ultrasonic image feedback, characterized in that: The method for adjusting the position and posture of a robotic arm based on ultrasonic image feedback specifically includes: Reading an ultrasonic image, and obtaining an ultrasonic detection origin and a centroid of the ultrasonic image, wherein the ultrasonic image is collected by automatic scanning according to a pre-planned scanning path; Determine a centroid line of the ultrasonic image based on the centroid of the ultrasonic image and the ultrasonic detection origin, and determine a center line of the ultrasonic image based on the ultrasonic detection origin; The positional relationship between the centroid line of the ultrasound image and the center line of the ultrasound image is calculated, and the scanning path is adjusted based on the positional relationship.

2. The method for adjusting the position and posture of a robotic arm based on ultrasonic image feedback according to claim 1, characterized in that: The process of obtaining the ultrasonic detection origin specifically includes: Performing filtering processing on the ultrasonic image to obtain a filtered ultrasonic image; Performing edge detection on the filtered ultrasonic image to obtain image edges; An ultrasound detection origin of the ultrasound image is determined based on the image edge.

3. The method for adjusting the position and posture of a robotic arm based on ultrasonic image feedback according to claim 2, characterized in that: Before performing edge detection on the filtered ultrasonic image to obtain the image edge, the method further includes: Extracting target image points whose grayscale values ​​are within a first grayscale range in the filtered ultrasonic image; Redistributing the grayscale value of the extracted target image point to a second grayscale range, wherein an upper grayscale value of the second grayscale range is greater than an upper grayscale value of the first grayscale range; The grayscale value of each image point except the target image point in the filtered ultrasonic image is reallocated as the upper limit grayscale value of the second grayscale range.

4. The method for adjusting the position and posture of a robotic arm based on ultrasonic image feedback according to claim 2, characterized in that: The performing edge detection on the filtered ultrasonic image to obtain the image edge specifically includes: Performing Canny edge detection on the filtered ultrasonic image to obtain an image edge set, and selecting an upper left corner straight line and an upper right corner straight line from the image edge set; If the upper left straight line and the upper right straight line are selected, the upper left straight line and the upper right straight line are used as the image edge of the ultrasound image; If the upper left straight line is selected, the upper right straight line is drawn according to the selected upper left straight line, and the upper left straight line and the drawn upper right straight line are used as the image edge of the ultrasound image; If the upper right straight line is selected, the upper left straight line is drawn according to the selected upper right straight line, and the drawn upper left straight line and upper right straight line are used as the image edge of the ultrasound image; If the upper left straight line and the upper right straight line are not selected, it will prompt that the image edge detection is wrong.

5. The method for adjusting the position and posture of a robotic arm based on ultrasonic image feedback according to claim 1, characterized in that: The process of obtaining the centroid of the ultrasound image specifically includes: Acquire the total image brightness, the first grayscale value weighted distribution in the x-direction, and the second grayscale value weighted distribution in the y-direction of the ultrasound image; Calculating the ratio of the total brightness of the image to the weighted distribution of the first grayscale value to obtain the centroid coordinates in the x direction; The ratio of the total brightness of the image to the weighted distribution of the second grayscale value is calculated to obtain the centroid coordinates in the y direction.

6. The method for adjusting the position and posture of a robotic arm based on ultrasonic image feedback according to claim 1, characterized in that: The calculating the positional relationship between the centroid line of the ultrasonic image and the center line of the ultrasonic image specifically includes: Calculating the angle between the centroid line of the ultrasonic image and the center line of the ultrasonic image; Based on the angle, the offset direction and angle of the centroid relative to the center line of the ultrasound image are determined to obtain the positional relationship between the centroid of the ultrasound image and the center line of the ultrasound image, wherein the offset direction is that the centroid is located on the left side of the center line or the centroid is located on the right side of the center line.

7. The method for adjusting the position and posture of a robotic arm based on ultrasonic image feedback according to claim 1, characterized in that: The process of adjusting the scanning path based on the position relationship specifically includes: Reading the next path point of the path point corresponding to the ultrasound image; The rotation angle in the x direction of the next path point is adjusted based on the positional relationship to adjust the robot arm posture of the next path point.

8. A robotic arm posture adjustment device based on ultrasonic image feedback, characterized in that: The robotic arm posture adjustment device based on ultrasonic image feedback specifically includes: A reading module, used to read an ultrasonic image and obtain an ultrasonic detection origin and a centroid of the ultrasonic image, wherein the ultrasonic image is collected by automatically scanning according to a pre-planned scanning path; a determination module, configured to determine a centroid line of the ultrasonic image based on the centroid of the ultrasonic image and the ultrasonic detection origin, and to determine a center line of the ultrasonic image based on the ultrasonic detection origin; An adjustment module is used to calculate the positional relationship between the centroid line of the ultrasound image and the center line of the ultrasound image, and adjust the scanning path based on the positional relationship.

9. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores one or more programs, and the one or more programs can be executed by one or more processors to implement the steps in the method for adjusting the posture of a robotic arm based on ultrasonic image feedback as described in any one of claims 1-7.

10. A terminal device, characterized in that: include: Processor and memory; The memory stores a computer-readable program executable by the processor; When the processor executes the computer-readable program, the steps in the method for adjusting the posture of a robotic arm based on ultrasonic image feedback are implemented as described in any one of claims 1 to 7.

Images