A mechanical arm pose adjustment method and device based on ultrasound image feedback

Abstract

The application discloses a mechanical arm pose adjustment method and device based on ultrasonic image feedback. The method comprises reading an ultrasonic image, obtaining an ultrasonic detection origin and an ultrasonic image centroid of the ultrasonic image, calculating an offset angle of a mechanical arm based on the ultrasonic detection origin and the ultrasonic image centroid, and feeding back the offset angle to the mechanical arm to make the mechanical arm adjust the pose. The application estimates the offset angle of the current image through the collected ultrasonic image, and feeds back the offset angle to the mechanical arm in real time, so that the pose can be dynamically adjusted in the mechanical arm scanning, and the organ scanned by the ultrasonic image can be placed in the most ideal position to obtain better ultrasonic images.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of automatic ultrasound scanning, in particular to a mechanical arm pose adjustment method and device based on ultrasound image feedback. BACKGROUND

[0002] When automatic ultrasound scanning is performed by a mechanical arm, the automatic scanning path is generally planned in advance before automatic scanning, and then the mechanical arm drives the ultrasound probe to perform automatic ultrasound scanning according to the automatic scanning path. However, in the automatic ultrasound scanning by the mechanical arm, small movements of the body can cause deviation of the scanning path, thereby affecting the image quality of the collected ultrasound image.

[0003] Therefore, the prior art still needs to be improved and enhanced. SUMMARY

[0004] The technical problem to be solved by the present application is to provide a mechanical arm pose adjustment method and device based on ultrasound image feedback in view of the deficiencies of the prior art.

[0005] To solve the above technical problems, the first aspect of the present application provides a mechanical arm pose adjustment method based on ultrasound image feedback, wherein the mechanical arm pose adjustment method based on ultrasound image feedback specifically comprises:

[0006] reading an ultrasound image and obtaining an ultrasound detection origin and an ultrasound image centroid of the ultrasound image, wherein the ultrasound image is collected by automatic scanning according to a pre-planned scanning path;

[0007] determining a centroid line of the ultrasound image based on the ultrasound image centroid and the ultrasound detection origin, and determining a center line of the ultrasound image based on the ultrasound detection origin;

[0008] calculating the positional relationship between the centroid line of the ultrasound image and the center line of the ultrasound image, and adjusting the scanning path based on the positional relationship.

[0009] The mechanical arm pose adjustment method based on ultrasound image feedback, wherein the obtaining process of the ultrasound detection origin specifically comprises:

[0010] filtering the ultrasound image to obtain a filtered ultrasound image;

[0011] performing edge detection on the filtered ultrasound image to obtain an image edge;

[0012] determining the ultrasound detection origin of the ultrasound image based on the image edge.

[0013] The mechanical arm pose adjustment method based on ultrasound image feedback, wherein, before the edge detection on the filtered ultrasound image to obtain image edges, the method further comprises:

[0014] extracting target image points in the filtered ultrasound image with gray values in a first gray value range;

[0015] redistributing the gray values of the extracted target image points to a second gray value range, wherein the upper limit gray value of the second gray value range is greater than the upper limit gray value of the first gray value range;

[0016] redistributing the gray values of each image point in the filtered ultrasound image except the target image points to the upper limit gray value of the second gray value range.

[0017] The mechanical arm pose adjustment method based on ultrasound image feedback, wherein the edge detection on the filtered ultrasound image to obtain image edges specifically comprises:

[0018] performing Canny edge detection on the filtered ultrasound image to obtain an image edge set, and selecting a left upper corner straight line and a right upper corner straight line in the image edge set;

[0019] if the left upper corner straight line and the right upper corner straight line are selected, taking the left upper corner straight line and the right upper corner straight line as the image edges of the ultrasound image;

[0020] if the left upper corner straight line is selected, drawing a right upper corner straight line according to the selected left upper corner straight line, and taking the left upper corner straight line and the drawn right upper corner straight line as the image edges of the ultrasound image;

[0021] if the right upper corner straight line is selected, drawing a left upper corner straight line according to the selected right upper corner straight line, and taking the drawn left upper corner straight line and the right upper corner straight line as the image edges of the ultrasound image;

[0022] if the left upper corner straight line and the right upper corner straight line are not selected, prompting an image edge detection error.

[0023] The mechanical arm pose adjustment method based on ultrasound image feedback, wherein the process of obtaining the ultrasound image centroid specifically comprises:

[0024] obtaining the total image brightness of the ultrasound image, a first gray value weighted distribution in the x direction, and a second gray value weighted distribution in the y direction;

[0025] calculating the ratio of the total image brightness to the first gray value weighted distribution to obtain the centroid coordinate in the x direction;

[0026] calculating the ratio of the total image brightness to the second gray value weighted distribution to obtain the centroid coordinate in the y direction.

[0027] The mechanical arm pose adjustment method based on ultrasound image feedback, wherein the calculation of the position relationship between the center of mass line of the ultrasound image and the center line of the ultrasound image specifically comprises:

[0028] calculating an included angle between the center of mass line of the ultrasound image and the center line of the ultrasound image;

[0029] determining an offset direction and an offset angle of the center of mass line relative to the center line of the ultrasound image based on the included angle, so as to obtain the position relationship between the center of mass line of the ultrasound image and the center line of the ultrasound image, wherein the offset direction is that the center of mass line is on the left side of the center line or the center of mass line is on the right side of the center line.

[0030] The mechanical arm pose adjustment method based on ultrasound image feedback, wherein the adjustment of the scanning path based on the position relationship specifically comprises:

[0031] reading a next path point of a path point corresponding to the ultrasound image;

[0032] adjusting a rotation angle in an x direction in the next path point based on the position relationship, so as to adjust a mechanical arm pose of the next path point.

[0033] The second aspect of the present application provides a mechanical arm pose adjustment device based on ultrasound image feedback, wherein the mechanical arm pose adjustment device based on ultrasound image feedback specifically comprises:

[0034] a reading module, configured to read an ultrasound image and acquire an ultrasound detection origin and a center of mass of the ultrasound image, wherein the ultrasound image is collected by automatic scanning according to a pre-planned scanning path;

[0035] a determination module, configured to determine a center of mass line of the ultrasound image based on the center of mass of the ultrasound image and the ultrasound detection origin, and determine a center line of the ultrasound image based on the ultrasound detection origin;

[0036] an adjustment module, configured to calculate a position relationship between the center of mass line of the ultrasound image and the center line of the ultrasound image, and adjust the scanning path based on the position relationship.

[0037] The third aspect of the present application provides a computer readable storage medium, the computer readable storage medium stores one or more programs, the one or more programs can be executed by one or more processors to implement the steps in the mechanical arm pose adjustment method based on ultrasound image feedback as any of the above.

[0038] The fourth aspect of the present application provides a terminal device, comprising: a processor and a memory;

[0039] The memory has stored thereon a computer readable program executable by the processor;

[0040] The processor implements the steps in the method for adjusting the pose of a mechanical arm based on ultrasound image feedback according to any of the above when executing the computer readable program.

[0041] Advantages: Compared with the prior art, the present application provides a method and device for adjusting the pose of a mechanical arm based on ultrasound image feedback, which comprises reading an ultrasound image, and obtaining an ultrasound detection origin and an ultrasound image centroid of the ultrasound image; determining a centroid line of the ultrasound image based on the ultrasound image centroid and the ultrasound detection origin, and determining a center line of the ultrasound image based on the ultrasound detection origin; calculating the positional relationship between the centroid line of the ultrasound image and the center line of the ultrasound image, and adjusting the scanning path based on the positional relationship. In the present application, in the scanning process of the mechanical arm, after an ultrasound image is collected, the ultrasound detection origin and the ultrasound image centroid of the ultrasound image are obtained, and the offset angle of the ultrasound image is determined based on the ultrasound detection origin and the ultrasound image centroid, which is fed back to the mechanical arm in real time, so that the probe pose can be dynamically adjusted in the scanning of the mechanical arm, so that the organ scanned by the ultrasound image can be placed in the most ideal position, thereby improving the image quality of the collected ultrasound image. BRIEF DESCRIPTION OF DRAWINGS

[0042] In order to more clearly illustrate the technical solutions in the embodiments of the present application, the drawings needed in the embodiment description will be briefly introduced. Obviously, the drawings in the following description are only some embodiments of the present application, and other drawings can be obtained by those skilled in the art without creative labor.

[0043] Figure 1 The flowchart of the method for adjusting the pose of a mechanical arm based on ultrasound image feedback provided by the embodiments of the present application.

[0044] Figure 2 The schematic diagram for obtaining a target straight line from edge detection.

[0045] Figure 3 The schematic diagram of the ultrasound image after gray scale reassignment.

[0046] Figure 4 The schematic diagram of the centroid line and the center line.

[0047] Figure 5 The schematic diagram of the probe coordinate system.

[0048] Figure 6A principle block diagram of a mechanical arm pose adjustment device based on ultrasound image feedback is provided in the embodiments of the present application.

[0049] Figure 7 A principle block diagram of a terminal device is provided in the embodiments of the present application. DETAILED DESCRIPTION

[0050] The embodiments of the present application provide a mechanical arm pose adjustment method and device based on ultrasound image feedback. In order to make the purpose, technical solutions and effects of the present application more clear and explicit, the present application is further described in detail below with reference to the drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present application and not to limit the present application.

[0051] Those skilled in the art can understand that, unless specifically stated otherwise, the singular forms "a", "an", "said" and "the" used herein also include the plural forms. It should be further understood that the phrase "comprising" used in the specification of the present application means that the features, integers, steps, operations, elements and / or components exist, but does not exclude the presence or addition of one or more other features, integers, steps, operations, elements, components and / or groups thereof. It should be understood that when we say that an element is "connected" or "coupled" to another element, it can be directly connected or coupled to the other element, or there can be intermediate elements. In addition, "connected" or "coupled" used herein can include wireless connection or wireless coupling. The phrase "and / or" used herein includes all or any single unit and all combinations of the associated listed items.

[0052] Those skilled in the art can understand that, unless otherwise defined, all terms (including technical terms and scientific terms) used herein have the same meaning as that generally understood by those skilled in the art to which the present application belongs. It should also be understood that terms such as those defined in a general dictionary should be understood to have meanings consistent with those in the context of the prior art, and should not be interpreted in an idealized or overly formal sense unless specifically defined as such.

[0053] It should be understood that the sequence numbers and sizes of the steps in the embodiments do not mean the order of execution, and the execution order of the processes is determined by their functions and inherent logic, and should not constitute any limitation on the implementation process of the embodiments of the present application.

[0054] It is found through research that when a target is automatically scanned by a mechanical arm, the automatic scanning path is generally planned in advance before automatic scanning, and then the mechanical arm drives the ultrasound probe to automatically scan according to the automatic scanning path. However, in the automatic ultrasound scanning by the mechanical arm, small movements of the body can cause deviation of the scanning path, thereby affecting the image quality of the collected ultrasound images.

[0055] To solve the above problems, in the embodiment of the application, an ultrasound image is read, and an ultrasound detection origin and an ultrasound image centroid of the ultrasound image are obtained; a centroid line of the ultrasound image is determined based on the ultrasound image centroid and the ultrasound detection origin, and a center line of the ultrasound image is determined based on the ultrasound detection origin; a positional relationship between the centroid line of the ultrasound image and the center line of the ultrasound image is calculated, and the scan path is adjusted based on the positional relationship. In the mechanical arm scanning process, after an ultrasound image is collected, the ultrasound detection origin and the ultrasound image centroid of the ultrasound image are obtained, and the offset angle of the ultrasound image is determined based on the ultrasound detection origin and the ultrasound image centroid, which is fed back to the mechanical arm in real time, so that the probe pose can be dynamically adjusted in the mechanical arm scanning, and the organ scanned by the ultrasound image can be placed in the most ideal position, thereby improving the image quality of the collected ultrasound image.

[0056] The application content will be further described by the description of the embodiments in combination with the drawings.

[0057] The embodiment provides a mechanical arm pose adjustment method based on ultrasound image feedback, as shown in Figure 1 The method comprises the following steps of:

[0058] S10, an ultrasound image is read, and an ultrasound detection origin and an ultrasound image centroid of the ultrasound image are obtained.

[0059] Specifically, the ultrasound image is collected by automatic scanning according to a preplanned scan path, for example, the ultrasound image is obtained by automatic scanning of an abdomen by a mechanical arm driving an ultrasound probe, or the ultrasound image is obtained by automatic scanning of a liver by the mechanical arm driving the ultrasound probe, etc. The ultrasound probe performs ultrasound collection at each path point in the preplanned scan path, that is, the ultrasound probe is driven by the mechanical arm to automatically scan according to the preplanned scan path, and each time an ultrasound image is collected, the ultrasound image is read to adjust the ultrasound probe pose of the next path point based on the ultrasound image, wherein the ultrasound probe can be a convex array probe, the imaging area of the convex array probe is in a fan-shaped structure, the upper and lower edges are arc-shaped, and the left and right edges are straight lines.

[0060] The scan path is preplanned, and the planning process of the scan path is described by taking automatic scanning of a liver as an example. The path planning process of the liver automatic scanning can be as follows:

[0061] H10, obtaining human point cloud data and a human image of an object to be scanned;

[0062] H20, determining a human body surface based on the human body point cloud data, and obtaining a normal vector of each surface point in the human body surface;

[0063] H30, determining a liver region of the to-be-scanned object based on the human body image, and constructing an initial scanning path for the to-be-scanned object based on the liver region;

[0064] H40, determining a probe posture corresponding to each path point in the initial scanning path based on the normal vector of each surface point in the human body surface, and adding the probe posture to the path point to obtain a target scanning path of the to-be-scanned object.

[0065] Specifically, the human body point cloud data includes point cloud data of a human body surface corresponding to the liver region, and the human body image includes the liver region. The human body point cloud data is obtained by scanning the to-be-scanned object, for example, by scanning the to-be-scanned object by a kinect camera to obtain the human body point cloud data of the to-be-scanned object. The human body image is the RGB-D image data of the to-be-scanned object, for example, by taking an image of the to-be-scanned object by a depth camera to obtain the human body image. The human body surface is obtained by human body reconstruction based on the human body point cloud data, and the human body surface is a human body 3D model. For example, the normal vector of the surface position of each point cloud data in the human body point cloud data of the to-be-scanned object is obtained by processing the human body surface by a PCL library.

[0066] Further, the liver region can be determined based on the corresponding relationship between the rib position and the liver, wherein the rib position can be obtained by a trained human body segmentation model. That is, the human body image is first input into the trained human body segmentation model, and the rib position of the to-be-scanned object is output by the human body segmentation model; then the liver region of the to-be-scanned object is determined based on the rib position of the to-be-scanned object according to the corresponding relationship between the rib position and the liver. The human body segmentation model is a trained deep learning model (for example, CNN, DNN, etc.) for identifying the rib position in the human body image. The corresponding relationship between the rib position and the liver is that the upper boundary of the liver is located between the right 5th and 6th ribs, the lower boundary of the liver is located at the 12th rib of the right midaxillary line, the right lower boundary of the liver extends from the rib edge to the junction of the right 7th and 8th ribs, and then extends obliquely to the left upper side to the left 6th rib, the left side boundary is close to the liver of the left rib edge, and the right side boundary is below the nipple. Therefore, after obtaining the rib position, the liver region of the to-be-scanned object can be outlined according to the corresponding relationship between the rib position and the liver, and the liver region is in a right-angle trapezoidal distribution.

[0067] Further, the initial scanning path includes a transverse scanning path, a longitudinal scanning path, and a lateral scanning path, each of the transverse scanning path, the longitudinal scanning path, and the lateral scanning path includes position information, where the position information includes an x-direction coordinate, a y-direction coordinate, and a z-direction coordinate. The transverse scanning path and the longitudinal scanning path are used for observing liver lesions, and the lateral scanning path is used for evaluating liver blood vessel conditions. The lateral scanning path starts at a xiphoid position and extends along a right lower rib edge to a right lowest rib end. Specifically, a starting point of the transverse scanning path is a region vertex of the liver region, a starting point of the longitudinal scanning path is also a region vertex of the liver region, and the starting point of the transverse scanning path and the starting point of the longitudinal scanning path can be different. For example, the starting point of the transverse scanning path is a left upper vertex of the liver region, and the starting point of the longitudinal scanning path is a right upper vertex of the liver region. Of course, the starting point of the transverse scanning path and the starting point of the longitudinal scanning path can be the same. For example, the starting point of the transverse scanning path and the starting point of the longitudinal scanning path are both left upper vertices of a right angle trapezoidal region.

[0068] Further, in order to minimize the moving distance of the mechanical arm in actual operation, when constructing the initial scanning path, the width of the ultrasound probe can also be determined according to the liver region. Based on this, when constructing the initial scanning path, the transverse width and the longitudinal width of the liver region can be obtained first; then based on the transverse width and the longitudinal width, the width of the ultrasound probe used for the ultrasound examination of the to-be-scanned object is determined. Wherein, the transverse width is the maximum distance of the liver region in the transverse direction, and the longitudinal width is the maximum distance of the liver region in the longitudinal direction. The width of the ultrasound probe can be based on the greatest common divisor of the transverse width and the longitudinal width, for example, the greatest common divisor is taken as the width of the ultrasound probe, or the greatest common divisor is 1 / n as the width of the ultrasound probe, or the greatest common divisor is n times as the width of the ultrasound probe, n is a positive integer greater than 1. In addition, the first distance between adjacent two rows of the transverse scanning path in the transverse scanning path is equal to the width of the ultrasound probe; the second distance between adjacent two rows of the longitudinal scanning path in the longitudinal scanning path is equal to the width of the ultrasound probe. Of course, in actual application, the first distance between adjacent two rows of the transverse scanning path and the second distance between adjacent two rows of the longitudinal scanning path are both less than the width of the ultrasound probe, and the difference between the width of the ultrasound probe and the first distance and the difference between the width of the ultrasound probe and the second distance both satisfy a preset condition (for example, less than a preset difference threshold).

[0069] Further, in determining the probe posture corresponding to each path point in the initial scan path based on the normal vector of each surface point in the human body surface, the surface point corresponding to the path point can be obtained, and the normal vector direction of the surface point is taken as the Z-axis direction of the path point; the moving direction of the path point is read, and the moving direction is taken as the X-axis direction of the path point; the Y-axis direction of the path point is determined based on the X-axis direction and the Z-axis direction to obtain the probe coordinate system of the ultrasound probe at the path point, and then the rotation matrix between the probe coordinate system and the global coordinate system of the liver region is determined, and the probe posture corresponding to the path point is determined based on the rotation matrix.

[0070] It should be noted that in actual application, the construction mode of the scan path can also be different when different parts are automatically scanned. Of course, the scan path of automatic scanning can also be manually set by the operator according to his own experience.

[0071] The ultrasound detection origin is a specific point of the ultrasound detection surface, which is used as the starting reference position of measurement, and helps to improve the accuracy and consistency of measurement. For example, the acquisition process of the ultrasound detection origin specifically includes:

[0072] S11, performing filtering processing on the ultrasound image to obtain a filtered ultrasound image;

[0073] S12, performing edge detection on the filtered ultrasound image to obtain an image edge;

[0074] S13, determining an ultrasound detection origin of the ultrasound image based on the image edge.

[0075] Specifically, in step S11, the filtering processing can filter the ultrasound image by a Gaussian filter, reduce noise or interference of some internal substantial structures of organs through Gaussian filtering, and better maintain the structural integrity of the image, improve the accuracy and reliability of subsequent analysis.

[0076] Further, in step S12, edge detection is used to detect edge data in the filtered ultrasound image, and the region with sharp change in brightness in the ultrasound image is captured through edge detection to identify edge straight lines and / or side edges of sectors in the ultrasound image, for example, the edge between some gas and substance in the ultrasound detection organ.

[0077] For example, as shown in Figure 2 The edge detection on the filtered ultrasound image to obtain an image edge specifically includes:

[0078] Canny edge detection is performed on the filtered ultrasound image to obtain an image edge set, and the left upper corner straight line and the right upper corner straight line are selected in the image edge set.

[0079] If the left-top corner line and the right-top corner line are selected, the left-top corner line and the right-top corner line are taken as the image edges of the ultrasound image;

[0080] If the left-top corner line is selected, a right-top corner line is drawn according to the selected left-top corner line, and the left-top corner line and the drawn right-top corner line are taken as the image edges of the ultrasound image;

[0081] If the right-top corner line is selected, a left-top corner line is drawn according to the selected right-top corner line, and the drawn left-top corner line and the right-top corner line are taken as the image edges of the ultrasound image;

[0082] If neither the left-top corner line nor the right-top corner line is selected, an image edge detection error is prompted.

[0083] Specifically, the Canny edge detection can use the Canny function in OpenCV to perform Canny edge detection, and capture the region with sharp changes in brightness in the filtered ultrasound image by the Canny edge detection to obtain an image edge set in the filtered ultrasound image. After obtaining the image edge set, the image edge set is detected to select a left-top corner line and a right-top corner line in the image edge set, wherein the process of selecting the left-top corner line and the right-top corner line in the image edge set can be: first detecting candidate lines in the image edge set, and then screening the detected candidate lines to obtain the left-top corner line and the right-top corner line located at the left-top corner and the right-top corner of the image.

[0084] Further, when detecting the candidate lines in the image edge set, the HoughLinesP function in OpenCV can be used to detect the candidate lines according to a preset line condition by cumulative probability Hough transformation, the preset line condition being that the line length is greater than a first threshold value and the line gap is less than or equal to a second threshold value, the first threshold value being greater than the second threshold value. The embodiment of the present application filters the internal lines of the organ in the ultrasound image by pre-setting the range of the line length and the line gap, so that the side region lines can be better detected, and thus the region edge lines of the fan-shaped examination region of the ultrasound probe can be better detected.

[0085] Since in the process of acquiring an ultrasound image by using a convex array ultrasound probe, the ultrasound detection origin of the ultrasound image can be determined by the left upper corner line or the right upper corner line of the fan-shaped examination region of the convex array ultrasound probe. Therefore, after the candidate lines are obtained, the left upper corner line and the right upper corner line in the candidate lines can be screened, and then the ultrasound detection origin is determined based on the left upper corner line and the right upper corner line. In addition, for some ultrasound images with edge gray values, only the left upper corner line or the right upper corner line can be detected. Therefore, when only the left upper corner line or the right upper corner line is detected, the left upper corner line or the right upper corner line is symmetrical to the image center line to draw the right upper corner line corresponding to the left upper corner line and the left upper corner line corresponding to the right upper corner line. Of course, in some cases, neither the left upper corner line nor the right upper corner line can be identified, at which time the image edge detection error can be prompted.

[0086] Further, in the process of edge detection, for the part with low gray value, especially the part with gray value close to 0, it is easy to miss detection, which will affect the accuracy of the ultrasound detection origin. Therefore, in one implementation, before the edge detection of the filtered ultrasound image is performed to obtain the image edge, the method further comprises:

[0087] extracting target image points in the filtered ultrasound image with gray values in a first gray value range;

[0088] redistributing the gray values of the extracted target image points in a second gray value range;

[0089] redistributing the gray values of each image point in the filtered ultrasound image except the target image points as the upper limit gray value of the second gray value range.

[0090] Specifically, the first gray value range and the second gray value range are both pre-set, the upper limit gray value of the second gray value range is greater than the upper limit gray value of the first gray value range, and the lower limit gray value of the second gray value range and the lower limit gray value of the first gray value range are both 0. In this way, the gray values in the first gray value range can be redistributed to the second gray value range, which can improve the edge detection effect of the part with low gray value, and can make the subtle changes of the ultrasound image more obvious, which can further improve the edge detection effect of the low gray value.

[0091] In one specific embodiment, since the gray value above 100 can be easily detected by edge detection, therefore, as shown in FIG. 6, the gray value of the target image points in the first gray value range is redistributed to the second gray value range, and the gray value of each image point in the filtered ultrasound image except the target image points is redistributed as the upper limit gray value of the second gray value range. Figure 3As shown, the first gray scale range can be set as 0-100, the second gray scale range can be set as 0-255, for the image point with the gray scale value greater than 100, the gray scale value of the image point is re-allocated as 255, and then for the image point with the gray scale value between 0-100, the gray scale value of the image point is re-allocated as 0-255. By re-allocating the image point with the gray scale value between 0-100 as 0-255, the embodiment can effectively enhance the contrast of the image point, can make the subtle change more obvious, and thus improve the edge detection effect of the low gray scale value. In addition, since the image point with the gray scale value higher than 100 has no influence on the edge detection, the part can be directly re-allocated as 255.

[0092] Further, in step S13, after the image edge is obtained, the upper left straight line and the upper right straight line in the image edge can be extended to determine the focal points of the upper left straight line and the upper right straight line, and then the focal points are taken as the ultrasonic detection origin of the ultrasonic image.

[0093] The above completes the description of the detection process of the ultrasonic detection origin, and the acquisition process of the ultrasonic image centroid is described below. Specifically, the acquisition process of the ultrasonic image centroid specifically includes:

[0094] acquiring the total brightness of the ultrasonic image, the first gray scale value weighted distribution in the x direction, and the second gray scale value weighted distribution in the y direction;

[0095] calculating the ratio of the total brightness to the first gray scale value weighted distribution to obtain the centroid coordinate in the x direction;

[0096] calculating the ratio of the total brightness to the second gray scale value weighted distribution to obtain the centroid coordinate in the y direction.

[0097] Specifically, first, the sum of all pixel values in the ultrasonic image is calculated to obtain the total brightness of the image, and the calculation formula of the total brightness of the image is:

[0098]

[0099] wherein M 00 represents the total brightness of the image, W represents the width of the ultrasonic image, H represents the height of the ultrasonic image, and G(x, y) represents the gray scale value of the image point (x, y).

[0100] Secondly, the first gray scale value weighted distribution in the x direction and the second gray scale value weighted distribution in the y direction are calculated, and the calculation formulas of the first gray scale value weighted distribution and the second gray scale value weighted distribution are respectively:

[0101]

[0102] wherein M10 denotes the first gray value weighted distribution, M 01 denotes the second gray value weighted distribution.

[0103] Finally, the ratio of the total brightness of the image to the first gray value weighted distribution and the second gray value weighted distribution is calculated respectively to obtain the centroid coordinate in the x direction and the centroid coordinate in the y direction, wherein the calculation formula of the centroid coordinate in the x direction and the centroid coordinate in the y direction is respectively:

[0104]

[0105] wherein C x denotes the centroid coordinate in the x direction, C y denotes the centroid coordinate in the y direction.

[0106] S20, determining the centroid line of the ultrasound image based on the ultrasound image centroid and the ultrasound detection origin, and determining the center line of the ultrasound image based on the ultrasound detection origin.

[0107] Specifically, the centroid line of the ultrasound image is the line connecting the ultrasound detection origin and the ultrasound image centroid, and the center line of the ultrasound image is the line connecting the ultrasound detection origin and the image center point of the ultrasound image. That is, after obtaining the ultrasound image centroid and the ultrasound detection origin, the image center line point of the ultrasound image can be read, then the ultrasound detection origin and the ultrasound image centroid are connected to form the centroid line of the ultrasound image, and the ultrasound detection origin and the image center point are connected to form the center line of the ultrasound image.

[0108] S30, calculating the positional relationship between the centroid line of the ultrasound image and the center line of the ultrasound image, and adjusting the scan path based on the positional relationship.

[0109] Specifically, the positional relationship is used to reflect the offset information of the centroid line of the ultrasound image relative to the center line of the ultrasound image, wherein the positional relationship includes the offset direction and the offset angle, and the offset direction is used to reflect that the centroid line is located on the left side or the right side of the center line, that is, the offset direction is that the centroid line is located on the left side of the center line or the centroid line is located on the right side of the center line.

[0110] Specifically, the calculation of the positional relationship between the centroid line of the ultrasound image and the center line of the ultrasound image includes:

[0111] calculating the included angle between the centroid line of the ultrasound image and the center line of the ultrasound image;

[0112] Based on the included angle, the offset direction and offset angle of the centroid line relative to the center line of the ultrasound image are determined to obtain the positional relationship between the centroid line of the ultrasound image and the center line of the ultrasound image, wherein the offset direction is that the centroid line is located to the left of the center line or the centroid line is located to the right of the center line.

[0113] Specifically, the angle between the centroid line and the center line of the ultrasound image can be calculated using trigonometric functions. After calculating the angle, it can be compared with 180°. If it is less than 180°, it means the centroid line is to the left of the center line; if it is greater than 180°, it means the centroid line is to the right of the center line. Of course, as shown in the figure, if the angle is equal to 180°, it can be determined that the centroid line is to the left or to the right of the center line; no limitation is made here.

[0114] Furthermore, after determining the offset direction, if the offset direction is such that the centroid line is to the left of the center line, the included angle is used as the offset angle; if the offset direction is such that the centroid line is to the right of the center line, the difference between 360° and the included angle is used as the offset angle, thus obtaining the positional relationship between the centroid line of the ultrasound image and the center line of the ultrasound image. For example, Figure 4 As shown, the calculated included angle is 353.27°, and the positional relationship between the centroid line of the ultrasound image and the center line of the ultrasound image is that the centroid line is located 6.73° to the right of the center line.

[0115] For example, the process of adjusting the scanning path based on the positional relationship specifically includes:

[0116] Read the next path point corresponding to the path point in the ultrasound image;

[0117] Based on the positional relationship, the rotation angle in the x-direction of the next path point is adjusted to adjust the posture of the robotic arm at the next path point.

[0118] Specifically, after obtaining the positional relationship between the centroid line and the center line of the ultrasound image, this positional relationship is fed back to the robotic arm's pose, thereby enabling real-time adjustments to the robotic arm, specifically adjusting the rotation angle in the x-direction at the next path point. For example, as... Figure 5 As shown, the Z-axis must be perpendicular to the human body surface during scanning. Therefore, it is only necessary to adjust the rx in the robot arm pose (x,y,z,rx,ry,rz) based on the off-position relationship to make the centroid line of the ultrasound image closer to the midline, thereby making the ultrasound image clearer.

[0119] In summary, the embodiment provides an ultrasonic image feedback-based robot pose adjustment method and device, the method comprising reading an ultrasonic image, obtaining an ultrasonic detection origin and an ultrasonic image centroid of the ultrasonic image, calculating an offset angle of a robot based on the ultrasonic detection origin and the ultrasonic image centroid, and feeding back the offset angle to the robot to make the robot adjust its pose. The application estimates the offset angle of the current image through the collected ultrasonic image, and feeds back the offset angle to the robot in real time, so that the robot can dynamically adjust its pose during scanning, and the organ scanned by the ultrasonic image can be placed in the most ideal position to obtain a better ultrasonic image.

[0120] Based on the above ultrasonic image feedback-based robot pose adjustment method, the embodiment provides an ultrasonic image feedback-based robot pose adjustment device, as shown in Figure 6 The ultrasonic image feedback-based robot pose adjustment device specifically comprises:

[0121] A reading module 100 is configured to read an ultrasonic image and obtain an ultrasonic detection origin and an ultrasonic image centroid of the ultrasonic image, wherein the ultrasonic image is collected through automatic scanning according to a pre-planned scanning path.

[0122] A determining module 200 is configured to determine a centroid line of the ultrasonic image based on the ultrasonic image centroid and the ultrasonic detection origin, and determine a center line of the ultrasonic image based on the ultrasonic detection origin.

[0123] An adjusting module 300 is configured to calculate the positional relationship between the centroid line of the ultrasonic image and the center line of the ultrasonic image, and adjust the scanning path based on the positional relationship.

[0124] Based on the above ultrasonic image feedback-based robot pose adjustment method, the embodiment provides a computer-readable storage medium, which stores one or more programs, and the one or more programs can be executed by one or more processors to implement the steps of the ultrasonic image feedback-based robot pose adjustment method.

[0125] Based on the above ultrasonic image feedback-based robot pose adjustment method, the application further provides a terminal device, as shown in Figure 7As shown, it includes at least one processor 20, a display screen 21, and a memory 22, and can also include a communications interface 23 and a bus 24. Among them, the processor 20, the display screen 21, the memory 22 and the communications interface 23 can complete the communication among each other through the bus 24. The display screen 21 is configured to display the user guide interface preset in the initial setting mode. The communications interface 23 can transmit information. The processor 20 can call the logical instructions in the memory 22 to execute the method in the above-mentioned embodiments.

[0126] In addition, the logical instructions in the memory 22 described above can be realized in the form of a software function unit and sold or used as an independent product, which can be stored in a computer readable storage medium.

[0127] The memory 22 as a computer readable storage medium can be configured to store software programs, computer executable programs, such as program instructions or modules corresponding to the method in the embodiments of the present disclosure. The processor 20 executes the functions of the application and data processing by running the software programs, instructions or modules stored in the memory 22, that is, realizes the method in the above-mentioned embodiments.

[0128] The memory 22 can include a program storage area and a data storage area, wherein the program storage area can store an operating system and at least one application required by a function; the data storage area can store data created according to the use of the terminal device, etc. In addition, the memory 22 can include a high-speed random access memory, and can also include a non-volatile memory. For example, a variety of media that can store program codes, such as a U disk, a mobile hard disk, a read-only memory (Read-Only Memory, ROM), a random access memory (Random Access Memory, RAM), a magnetic disk or an optical disk, etc., can also be a transient storage medium.

[0129] In addition, the specific process of the above-mentioned storage medium and the plurality of instructions in the terminal device loaded and executed by the processor has been described in detail in the above-mentioned method, and here it will not be stated one by one.

[0130] Finally, it should be pointed out that: the above embodiments are only used to illustrate the technical solutions of the present application, and not to limit them; although the present application has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that: it can still modify the technical solutions recorded in the foregoing embodiments, or make equivalent replacement for part of the technical features; and these modifications or replacements do not make the essence of the corresponding technical solutions deviate from the spirit and scope of the technical solutions of the embodiments of the present application.

Claims

1. A method for adjusting a pose of a robot arm based on ultrasound image feedback, the method comprising: The mechanical arm pose adjustment method based on ultrasound image feedback specifically comprises the following steps. An ultrasound image is read, and an ultrasound detection origin and an ultrasound image centroid of the ultrasound image are obtained, wherein the ultrasound image is collected by automatic scanning according to a pre-planned scanning path; A centroid line of the ultrasound image is determined based on the ultrasound image centroid and the ultrasound detection origin, and a center line of the ultrasound image is determined based on the ultrasound detection origin; A positional relationship between the centroid line of the ultrasound image and the center line of the ultrasound image is calculated, and the scanning path is adjusted based on the positional relationship; The process of obtaining the ultrasound detection origin specifically comprises the following steps. The ultrasound image is filtered to obtain a filtered ultrasound image; Edge detection is performed on the filtered ultrasound image to obtain an image edge; The ultrasound detection origin of the ultrasound image is determined based on the image edge; The process of obtaining the ultrasound image centroid specifically comprises the following steps. The total brightness of the ultrasound image, a first gray value weighted distribution in an x direction and a second gray value weighted distribution in a y direction are obtained; A ratio of the total brightness of the ultrasound image to the first gray value weighted distribution is calculated to obtain a centroid coordinate in the x direction; A ratio of the total brightness of the ultrasound image to the second gray value weighted distribution is calculated to obtain a centroid coordinate in the y direction; The process of calculating the positional relationship between the centroid line of the ultrasound image and the center line of the ultrasound image specifically comprises the following steps. An included angle between the centroid line of the ultrasound image and the center line of the ultrasound image is calculated; Based on the included angle, an offset direction and an offset angle of the centroid line relative to the center line of the ultrasound image are determined to obtain the positional relationship between the centroid line of the ultrasound image and the center line of the ultrasound image, wherein the offset direction is that the centroid line is located on the left side of the center line or the centroid line is located on the right side of the center line. 2.The method of claim 1, wherein, Before the process of performing edge detection on the filtered ultrasound image to obtain an image edge, the method further comprises the following steps. Target image points in the filtered ultrasound image with gray values in a first gray value range are extracted; The gray values of the extracted target image points are reassigned to a second gray value range, wherein an upper limit gray value of the second gray value range is greater than an upper limit gray value of the first gray value range; The gray values of image points other than the target image points in the filtered ultrasound image are reassigned to the upper limit gray value of the second gray value range. 3.The method of claim 1, wherein, The process of performing edge detection on the filtered ultrasound image to obtain an image edge specifically comprises the following steps. Canny edge detection is performed on the filtered ultrasound image to obtain an image edge set, and a left upper corner straight line and a right upper corner straight line are selected from the image edge set; If the left upper corner straight line and the right upper corner straight line are selected, the left upper corner straight line and the right upper corner straight line are taken as the image edge of the ultrasound image; If the left upper corner straight line is selected, a right upper corner straight line is drawn according to the selected left upper corner straight line, and the left upper corner straight line and the drawn right upper corner straight line are taken as the image edge of the ultrasound image; If the right upper corner straight line is selected, a left upper corner straight line is drawn according to the selected right upper corner straight line, and the drawn left upper corner straight line and the right upper corner straight line are taken as image edges of the ultrasound image; If the left upper corner straight line and the right upper corner straight line are not selected, an image edge detection error is prompted. 4.The method of claim 1, wherein, The adjusting the scan path based on the position relationship specifically includes: reading a next path point of the path point corresponding to the ultrasound image; adjusting an x-direction rotation angle in the next path point based on the position relationship to adjust a mechanical arm posture of the next path point.

5. A robotic arm pose adjustment device based on ultrasonic image feedback, characterized in that, The mechanical arm posture adjusting device based on ultrasound image feedback specifically includes: a reading module, configured to read an ultrasound image, and acquire an ultrasound detection origin and an ultrasound image centroid of the ultrasound image, wherein the ultrasound image is collected by automatic scanning according to a pre-planned scan path; a determining module, configured to determine a centroid line of the ultrasound image based on the ultrasound image centroid and the ultrasound detection origin, and determine a center line of the ultrasound image based on the ultrasound detection origin; an adjusting module, configured to calculate a position relationship between the centroid line of the ultrasound image and the center line of the ultrasound image, and adjust the scan path based on the position relationship; wherein the acquiring process of the ultrasound detection origin specifically includes: performing filtering processing on the ultrasound image to obtain a filtered ultrasound image; performing edge detection on the filtered ultrasound image to obtain an image edge; determining the ultrasound detection origin of the ultrasound image based on the image edge; the acquiring process of the ultrasound image centroid specifically includes: acquiring an image total brightness, a first gray value weighted distribution in an x direction and a second gray value weighted distribution in a y direction of the ultrasound image; calculating a ratio of the image total brightness to the first gray value weighted distribution to obtain a centroid coordinate in the x direction; calculating a ratio of the image total brightness to the second gray value weighted distribution to obtain a centroid coordinate in the y direction; the calculating the position relationship between the centroid line of the ultrasound image and the center line of the ultrasound image specifically includes: calculating an included angle between the centroid line of the ultrasound image and the center line of the ultrasound image; determining an offset direction and an offset angle of the centroid line relative to the center line of the ultrasound image based on the included angle to obtain the position relationship between the centroid line of the ultrasound image and the center line of the ultrasound image, wherein the offset direction is that the centroid line is located on a left side of the center line or the centroid line is located on a right side of the center line.

6. A computer-readable storage medium, characterized in that, The computer readable storage medium stores one or more programs which can be executed by one or more processors to implement the steps in the mechanical arm posture adjusting method based on ultrasound image feedback as claimed in any one of claims 1-4.

7. A terminal device, characterized by, including: a processor and a memory; the memory stores a computer readable program which can be executed by the processor; the processor executes the computer readable program to implement the steps in the mechanical arm posture adjusting method based on ultrasound image feedback as claimed in any one of claims 1-4.

Images