Method, device, storage medium and program product for determining position and posture of a robotic arm

Abstract

The present invention discloses a method, device, storage medium and program product for determining the posture of a robot arm, including: obtaining a global image and a local image of a human body through an imaging sensor, both of which contain a target detection area, and the imaging sensor is fixed to the end of the robot arm; segmenting the global image of the human body into a target detection area to determine the contour of the target detection area; mapping the contour of the target detection area in the global image of the human body to the local image of the human body to obtain a two-dimensional path in the local image of the human body; projecting the pixels of the two-dimensional path to a three-dimensional point cloud to obtain a three-dimensional path; based on the three-dimensional path, determining the posture of the target robot arm, the posture of the target robot arm refers to the corresponding actuator posture of the end of the robot arm when the target detection area is detected based on the three-dimensional path. The scheme provided by the present application can automatically identify the area to be detected, and automatically determine the posture of the actuator at the end of the robot arm according to the area to be detected.

Description

Technical Field

[0001] The present invention relates to the field of robotics technology, and in particular to a method, device, storage medium and program product for determining the position and posture of a robotic arm. Background Art

[0002] Nowadays, ultrasound scanning is widely used in the field of clinical medicine, especially in the detection and diagnosis of internal organs. The human body is scanned with ultrasound beams, and images of internal organs are obtained by receiving and processing reflected signals.

[0003] In the related art, when performing ultrasound scanning, doctors usually need to manually operate the ultrasound equipment to perform ultrasound scanning on the area to be detected. However, when the doctor manually operates, it often depends on the doctor's experience and professional skills. For inexperienced medical staff, it is usually impossible to perform ultrasound scanning independently. Therefore, there is an urgent need for a solution for automated ultrasound scanning. After accurately locating the area to be detected, the position of the end effector of the robotic arm fixed with the ultrasound equipment is determined, so as to realize automated ultrasound scanning of the area to be detected. Summary of the invention

[0004] The purpose of the present invention is to provide a method, device, storage medium and program product for determining the posture of a robotic arm, so as to realize automatic identification of the human body area to be detected, and accurately determine the posture of the robotic arm end effector for detecting the area to be detected, which is conducive to realizing automatic detection of the area to be detected.

[0005] To achieve the above objectives, on the one hand, the present application provides a method for determining a robot arm posture, the method comprising:

[0006] Acquire a global image of a human body and a local image of a human body through an imaging sensor, wherein both the global image of a human body and the local image of a human body include a target detection area, and the imaging sensor is fixed at the end of the robotic arm;

[0007] Segmenting the global image of the human body into a target detection region to determine a contour of the target detection region;

[0008] Mapping the target detection area contour in the human body global image to the human body local image to obtain a two-dimensional path in the human body local image;

[0009] Projecting the pixels of the two-dimensional path onto a three-dimensional point cloud to obtain a three-dimensional path;

[0010] Based on the three-dimensional path, a target robotic arm posture is determined, where the target robotic arm posture refers to an actuator posture at the end of the robotic arm corresponding to when the target detection area is detected based on the three-dimensional path.

[0011] As a further improvement of the present application, mapping the target detection area contour in the global human body image to the local human body image to obtain a two-dimensional path in the local human body image includes:

[0012] Based on the first robotic arm posture when shooting the global image of the human body, the second robotic arm posture when shooting the local image of the human body, and the hand-eye calibration result, determine the sensor posture transformation matrix, the robotic arm posture is used to describe the relative position and posture relationship between the robotic arm and the robotic arm base, and the root of the robotic arm is fixed to the robotic arm base;

[0013] Based on the sensor posture transformation matrix, determining the conversion of the target detection area contour to intermediate contour data of the human body partial image, wherein the intermediate contour data represents the target detection area contour in a camera coordinate system;

[0014] Based on the internal parameters of the imaging sensor, the two-dimensional path corresponding to the intermediate contour data in the partial human body image is determined, and the two-dimensional path represents the contour of the target detection area in the uv coordinate system.

[0015] As a further improvement of the present application, the sensor posture transformation matrix is ​​determined based on the first robotic arm posture when shooting the global image of the human body, the second robotic arm posture when shooting the partial image of the human body, and the hand-eye calibration result, including:

[0016] Based on the hand-eye calibration result and the first robotic arm posture, determine a first sensor posture matrix, wherein the sensor posture matrix is ​​used to describe the relative position and posture relationship between the imaging sensor and the robotic arm base;

[0017] Determine a second sensor posture matrix based on the hand-eye calibration result and the second robotic arm posture;

[0018] The sensor posture transformation matrix for transforming the imaging sensor from a first sensor posture to a second sensor posture is determined based on the first sensor posture matrix and the second sensor posture matrix.

[0019] As a further improvement of the present application, determining a first sensor posture matrix based on the hand-eye calibration result and the first robotic arm posture includes:

[0020] Multiplying the hand-eye calibration result by the first robotic arm posture to obtain the first sensor posture matrix;

[0021] The determining of a second sensor posture matrix based on the hand-eye calibration result and the second robotic arm posture includes:

[0022] Multiplying the hand-eye calibration result by the second robotic arm posture to obtain the second sensor posture matrix;

[0023] The determining the sensor posture transformation matrix for transforming the imaging sensor from a first sensor posture to a second sensor posture comprises:

[0024] The first sensor posture matrix is ​​divided by the second sensor posture matrix to obtain the sensor posture transformation matrix.

[0025] As a further improvement of the present application, the step of determining, based on the sensor posture transformation matrix, the conversion of the target detection area contour to the intermediate contour data of the human body partial image includes:

[0026] The intermediate contour data is obtained by multiplying the data of each pixel point in the target detection contour with the sensor posture transformation matrix.

[0027] As a further improvement of the present application, projecting the pixels of the two-dimensional path onto a three-dimensional point cloud to obtain a three-dimensional path includes:

[0028] Traversing the pixels in the two-dimensional path, and determining a target point at a corresponding position in the global three-dimensional point cloud of the human body based on the coordinates of each pixel in the two-dimensional path;

[0029] When the target point is not included in the three-dimensional path and each component of the three-dimensional coordinates of the target point is not 0, the target point is added to the three-dimensional path.

[0030] As a further improvement of the present application, before determining the target robot arm position and posture based on the three-dimensional path, the method further includes:

[0031] Smoothing the three-dimensional path to obtain a smooth three-dimensional path;

[0032] Determining the target robot arm position and posture based on the three-dimensional path includes:

[0033] Based on the smooth three-dimensional path, the target robotic arm position and posture is determined.

[0034] As a further improvement of the present application, the step of smoothing the three-dimensional path to obtain a smooth three-dimensional path includes:

[0035] Performing linear interpolation on the three-dimensional path to obtain an expanded dense three-dimensional path;

[0036] Determine an intermediate three-dimensional path of the three-dimensional path within the cube occupied by the global three-dimensional point cloud of the human body;

[0037] Determine an intersecting three-dimensional path of the intermediate three-dimensional path and the dense three-dimensional path;

[0038] The three-dimensional point coordinates in the intersecting three-dimensional paths are traversed, and the intersecting three-dimensional paths are subjected to mean filtering to obtain the smooth three-dimensional path.

[0039] As a further improvement of the present application, determining the target robot arm position and posture based on the three-dimensional path includes:

[0040] Determine the normal vector corresponding to each point in the three-dimensional path;

[0041] Perform trajectory planning based on the normal vector, and convert the three-dimensional path into a motion point sequence, where the motion point sequence refers to a sequence of points to be detected in a camera coordinate system when performing detection based on the three-dimensional path;

[0042] Based on the hand-eye calibration result and the motion point sequence, the target robot arm posture in the base coordinate system is determined. The target robot arm posture in the base coordinate system is used to characterize the relative position relationship between the robot arm end effector and the robot arm base when detecting the point to be detected.

[0043] As a further improvement of the present application, the step of segmenting the global image of the human body into a target detection region and determining a contour of the target detection region includes:

[0044] Detecting key points of a human body based on the global image of the human body;

[0045] Based on the human body key points, the target detection area is segmented to determine the outline of the target detection area.

[0046] On the other hand, the present application provides an electronic device comprising: at least one processor; and a memory communicatively connected to the at least one processor; wherein the memory stores instructions executable by the at least one processor, and the instructions are executed by the at least one processor so that the at least one processor can execute the steps in the method for determining the robot arm posture as described in any of the above aspects.

[0047] On the other hand, the present application provides a non-transitory computer-readable storage medium storing computer instructions, wherein the computer instructions are used to enable the computer to execute the steps in the method for determining the position and posture of a robotic arm as described in any of the above aspects.

[0048] On the other hand, the present application provides a computer program product, including a computer program, which, when executed by a processor, implements the steps in the method for determining the position and posture of a robotic arm as described in any of the above aspects.

[0049] The solution provided in the embodiment of the present application has at least the following beneficial effects:

[0050] In the embodiment of the present application, the global image of the human body is first segmented into target detection areas, so that the area to be detected can be accurately located. And based on the contour of the target detection area, the two-dimensional path for detection in the local image after the segmentation of the target detection area is determined, and the two-dimensional path is projected onto the three-dimensional point cloud, so that the corresponding target robot arm posture when detecting the target detection area is determined based on the three-dimensional path. In this process, accurate robot arm posture planning can be achieved, which is conducive to controlling the robot arm to carry the ultrasonic equipment to perform automated ultrasonic detection of the target detection area. BRIEF DESCRIPTION OF THE DRAWINGS

[0051] Figure 1 A schematic diagram of an implementation environment of a method for determining a robot arm posture provided by an exemplary embodiment of the present application is shown;

[0052] Figure 2 A flowchart of a method for determining a robot arm posture provided by an exemplary embodiment of the present application is shown;

[0053] Figure 3 A schematic diagram of a global image of a human body and a global three-dimensional point cloud of a human body provided by an exemplary embodiment of the present application is shown;

[0054] Figure 4 A schematic diagram showing a target detection area outline and a two-dimensional path provided by an exemplary embodiment of the present application is shown;

[0055] Figure 5 A flowchart of a process for determining the position and posture of a robotic arm provided by an exemplary embodiment of the present application is shown;

[0056] Figure 6 A flowchart of a process for determining a sensor posture transformation matrix provided by an exemplary embodiment of the present application is shown;

[0057] Figure 7 A flowchart of a three-dimensional path smoothing process provided by an exemplary embodiment of the present application is shown;

[0058] Figure 8 A schematic diagram of a smooth three-dimensional path provided by an exemplary embodiment of the present application is shown;

[0059] Fig. 9 A structural block diagram of an electronic device provided by an exemplary embodiment of the present application is shown. DETAILED DESCRIPTION

[0060] The present invention will be described in detail below in conjunction with the specific embodiments shown in the accompanying drawings. However, these embodiments do not limit the present invention, and any structural, methodological, or functional changes made by a person skilled in the art based on these embodiments are all within the scope of protection of the present invention.

[0061] It should be noted that the term "includes" or any other variation thereof is intended to cover non-exclusive inclusion, so that a process, method, article or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes 105 elements inherent to such process, method, article or device. In addition, the terms "first", "second", etc. are used for descriptive purposes only and cannot be understood as indicating or implying relative importance.

[0062] Please refer to Figure 1 , which shows a schematic diagram of the implementation environment of a method for determining a robot arm posture provided by an exemplary embodiment of the present application, including a robot arm base 101, a robot arm 102, an imaging sensor 103, an electronic device 104 and a robot arm end effector 105.

[0063] The root of the robot arm 102 is fixed to the robot arm base 101, and the end of the robot arm is fixed with an imaging sensor 103. The robot arm base is used to support the robot arm and connect the robot arm to the installation plane. There are multiple joints and connecting rods connecting the joints in the robot arm, and there is an end effector 105 at the end of the robot arm, and the imaging sensor 103 is also fixed on the end effector. The imaging sensor 103 can be a device such as a camera or a webcam for converting an optical image into an electrical signal or a digital signal.

[0064] Optionally, the robot arm end effector 105 can be different tools such as a clamp, a vacuum suction cup, a welding head, a tool, and an ultrasonic probe. In the implementation scenario of the embodiment of the present application, the robot arm end effector 105 is a clamp for fixing ultrasonic scanning equipment and other equipment for detecting the target detection area.

[0065] In the embodiment of the present application, the robot base 101 and the imaging sensor 103 have a communication connection with the electronic device 104, and the connection mode of the communication connection can be a wired connection, such as a serial communication interface connection, an Ethernet interface connection, etc., or the communication mode is a wireless connection, such as Wi-Fi (Wireless Fidelity) and Bluetooth, which is not limited in this embodiment. The electronic device 104 can be a tablet computer, a computer device, a terminal, etc. In the embodiment of the present application, the robot base is used to send the position of the robot arm in the base coordinate system to the electronic device based on the communication connection, and the imaging sensor 104 is used to send the captured image data to the electronic device based on the connection.

[0066] Please refer to Figure 1 In a possible implementation, the imaging sensor captures the human body image to obtain the global image of the human body, and after the electronic device controls the mechanical arm to move a certain distance, the partial image of the human body is acquired, and the captured partial image of the human body and the global image of the human body are sent to the electronic device through a communication connection. In addition, the mechanical arm base determines the posture of the end of the mechanical arm at different positions and sends it to the electronic device. When the electronic device acquires the global image of the human body and the local image of the human body, it performs target detection area segmentation, determines the contour of the target detection area, and maps the contour of the target detection area in the global image of the human body to the local image of the human body, thereby obtaining a two-dimensional path in the local image of the human body, and then obtains a three-dimensional path. Finally, based on the three-dimensional path, the target mechanical arm posture is determined. After determining the target mechanical arm posture, the electronic device sends the control program of the mechanical arm to the controller in the mechanical arm base through the internal connection established with the mechanical arm base, thereby controlling the mechanical arm end effector to move to the target mechanical arm posture according to the set program, and when an ultrasonic device is fixed in the mechanical arm end effector 105, the human target detection area is ultrasonically scanned.

[0067] It should be noted that in the following embodiments, the electronic device 104 is taken as a computer device as an example to illustrate the solution provided in this application.

[0068] Please refer to Figure 2 , which shows a flow chart of a method for determining the position and posture of a robotic arm provided by an exemplary embodiment of the present application, the process includes the following steps:

[0069] Step 201 : acquiring a global image and a local image of a human body through an imaging sensor.

[0070] Among them, the imaging sensor is fixed at the end of the robotic arm.

[0071] Please refer to Figure 3 , which shows a global image of a human body and a schematic diagram of a global three-dimensional point cloud of a human body provided by an exemplary embodiment of the present application.

[0072] After capturing the global image of the human body, the imaging sensor sends the captured global image of the human body to the computer device. After the imaging sensor is displaced to a certain extent, the imaging sensor captures a local image of the human body and sends the local image of the human body to the computer device.

[0073] Since it is necessary to determine the posture of the robotic arm during detection based on the area to be detected, when taking local images and global images of the human body, it is necessary to ensure that both the global image and the local image contain the target detection area. The target detection area may be the human surface area corresponding to the human chest organ to be detected. For example, if the human chest organ to be detected is the liver, the corresponding area to be detected is the human epidermal chest area corresponding to the liver.

[0074] Step 202, segmenting the global image of the human body into target detection regions to determine the contours of the target detection regions.

[0075] Since the process of global human body image segmentation needs to rely on the positions of some key points of the human body, most of the key point positions are included in the entire human body image, while some key points are missing in the local image, which will cause the result of target detection area segmentation based on the local human body image to be inaccurate. Therefore, the embodiment of the present application performs target detection area segmentation based on the global human body image, which is conducive to obtaining accurate segmentation results.

[0076] In a possible implementation, when determining the contour of the target detection area, human key points are first detected based on the global image of the human body, and then the target detection area is segmented based on the human key points, thereby determining the contour of the target detection area.

[0077] Optionally, the computer device uses a YOLOv5 (You Only Look Once version 5) model to identify key points of human joints, and then obtains the position and area occupied by the human body in the global image of the human body, and then segments the human target detection area through PP-LiteSeg (PaddlePaddle-Lite Semantic Segmentation, lightweight real-time semantic segmentation network) to obtain the outline of the target detection area.

[0078] Please refer to Figure 4 , which shows a schematic diagram of a target detection area outline and a two-dimensional path provided by an exemplary embodiment of the present application.

[0079] Optionally, before using YOLOv5 and PP-LiteSeg to perform target detection region segmentation on the global image of the human body, it is necessary to fine-tune and train them so that they can be applied to the environment required by the embodiment of the present application for target detection region segmentation on the global image of the human body. Therefore, during the training process, a depth camera can be used to collect samples, and H RGB images and their corresponding point cloud data in different postures of Z different human bodies can be scanned respectively, and the sample contains Z×H data. The point cloud data corresponding to the RGB image is referenced Figure 3Point cloud data corresponding to the global image of the human body shown in .

[0080] Step 203, mapping the target detection area contour in the global human body image to the local human body image to obtain a two-dimensional path in the local human body image.

[0081] By segmenting the global image of the human body to obtain the contour of the target detection area, it is possible to avoid large errors in target area segmentation due to incomplete information in the local image of the human body.

[0082] Please refer to Figure 4 , where the right side is the two-dimensional path in the human body local image.

[0083] Step 204 , projecting the pixels of the two-dimensional path onto the three-dimensional point cloud to obtain a three-dimensional path.

[0084] Since the robot arm posture, that is, the position and posture of the robot arm in three-dimensional space, needs to be determined in the end, after obtaining the two-dimensional path pixels, it is necessary to project the two-dimensional path pixels to the three-dimensional point cloud to obtain the point cloud path in three-dimensional space, so as to further determine the target robot arm posture.

[0085] Step 205: determine the target robot arm position and posture based on the three-dimensional path.

[0086] Among them, the target robot arm posture refers to the actuator posture at the end of the robot arm corresponding to the detection of the target detection area based on the three-dimensional path.

[0087] The three-dimensional path can be regarded as the scanning point path that needs to be scanned when the detection device detects the target detection area. The detection device is fixed at the end of the robotic arm. Therefore, based on the three-dimensional path, the process of determining the target robotic arm posture can be regarded as, when detecting the target detection area, determining the posture of the end effector of the robotic arm used to fix the detection device based on the detection point position detected by the detection device.

[0088] In the embodiment of the present application, the global image of the human body is first segmented into target detection areas, so that the area to be detected can be accurately located. And based on the contour of the target detection area, the two-dimensional path for detection in the local image after the segmentation of the target detection area is determined, and the two-dimensional path is projected onto the three-dimensional point cloud, so that the corresponding target robot arm posture when detecting the target detection area is determined based on the three-dimensional path. In this process, accurate robot arm posture planning can be achieved, which is conducive to controlling the robot arm to carry the ultrasonic equipment to perform automated ultrasonic detection of the target detection area.

[0089] Please refer to Figure 5 , which shows a flowchart of a process for determining the position and posture of a robotic arm provided by an exemplary embodiment of the present application, the process comprising the following steps:

[0090] Step 501 , obtaining a global image of a human body and a local image of a human body through an imaging sensor, segmenting the global image of the human body into a target detection area, and determining a contour of the target detection area.

[0091] The specific implementation of this step can refer to the above-mentioned step 201 and step 202, and this embodiment will not be described in detail here.

[0092] Step 502 , determining a sensor posture transformation matrix based on the first robotic arm posture when capturing a global image of the human body, the second robotic arm posture when capturing a partial image of the human body, and a hand-eye calibration result.

[0093] Among them, the robot arm posture is used to describe the relative position and posture relationship between the robot arm and the robot arm base, and the root of the robot arm is fixed to the robot arm base.

[0094] In the embodiment of the present application, there may be conversions of the same point in different coordinate systems, and the coordinate systems that may be involved are as follows:

[0095] The manipulator base coordinate system is the reference coordinate system of the entire manipulator system, which is usually fixed on the manipulator base and is a Cartesian coordinate system used to determine the reference of the position and posture of the manipulator in the workspace. In the embodiment of the present application, the first manipulator posture and the second manipulator posture are both in the base coordinate system.

[0096] Furthermore, the robot arm end coordinate system is usually fixed on the robot arm actuator (clamp), and the origin of the coordinate system is usually located at a specific point at the end of the robot arm, such as the center of the clamp. Optionally, in the embodiment of the present application, the robot arm end coordinate system is a flange coordinate system.

[0097] The robot end effector coordinate system is the coordinate system of the robot end effector (gripper) itself, which is used to accurately describe the movements and operations of the gripper. For example, for a gripper-type gripper, the axis of its coordinate system can be defined as the opening and closing direction of the gripper, and the coordinate axis can be defined within the plane of the gripper to describe the position of the gripper object.

[0098] The camera coordinate system is a coordinate system established with the optical center of the camera (imaging sensor) as the origin. The X and Y axes are usually parallel to the image plane of the camera, and the axis points to the shooting direction of the camera. When the camera is installed at the end of the robotic arm, the position and posture of the camera are fixed relative to the robotic arm, and the position and posture of the camera coordinate system in the robotic arm coordinate system can be determined by calculation; when the camera is fixed at a certain position in the workspace, that is, the camera is independent of the robotic arm, the relationship between the camera coordinate system and the robotic arm base can be calibrated.

[0099] Please refer to Figure 6, which shows a flowchart of a process of determining a sensor posture transformation matrix provided by an exemplary embodiment of the present application, the process comprising the following steps:

[0100] Step 502a, determining a first sensor posture matrix based on the hand-eye calibration result and the first robotic arm posture.

[0101] Among them, the sensor posture matrix is ​​used to describe the relative position and posture relationship between the imaging sensor and the robotic arm base.

[0102] The hand-eye calibration result is a transformation matrix, which describes the relative position and relationship between the camera coordinate system and the robotic arm coordinate system. That is, the hand-eye calibration result can express the position and direction of the imaging sensor relative to the end of the robotic arm.

[0103] Optionally, since the sensor posture matrix is ​​used to describe the relative posture between the imaging sensor and the robotic arm base, and the hand-eye calibration result is used to describe the posture of the camera relative to the end of the robotic arm, and the robotic arm posture is used to describe the posture of the end of the robotic arm relative to the robotic arm base, then when determining the first sensor posture matrix, the product of the hand-eye calibration result and the first robotic arm posture is determined as the first sensor posture matrix.

[0104] For example, the first robot arm pose is flangeGlobal2Base, and the hand-eye calibration result is cam2Flange, then the first sensor pose matrix T 1 for:

[0105] camGloal2Base=flangeGlobal2Base*cam2Flange

[0106] Step 502b: Determine a second sensor posture matrix based on the hand-eye calibration result and the second robotic arm posture.

[0107] Since the position of the imaging sensor in space is changed by controlling the robotic arm in the process of shooting the global image of the human body and the local image of the human body, the posture of the imaging sensor relative to the end of the robotic arm does not change, that is, the hand-eye calibration result remains unchanged.

[0108] Similar to the first sensor posture matrix, when determining the relative posture between the sensor and the robot arm base, that is, determining the second sensor posture matrix, the product of the hand-eye calibration result and the second robot arm posture is determined as the second sensor posture matrix.

[0109] For example, the second robot arm posture is flangeLocal2Base, and the hand-eye calibration result is cam2Flange, then the second sensor posture matrix T 2 for:

[0110] camLocal2Base=flangeLocal2Base*cam2Flange

[0111] Step 502c: Determine a sensor posture transformation matrix for transforming the imaging sensor from the first sensor posture to the second sensor posture based on the first sensor posture matrix and the second sensor posture matrix.

[0112] The imaging sensor changes from the first sensor posture to the second sensor posture, that is, from T 1 Move to T 2 The posture is T 1 =T 2 *H, where H is the sensor attitude transformation matrix. Then the product of the first sensor attitude matrix and the inverse matrix of the second sensor attitude matrix is ​​determined as the sensor attitude transformation matrix.

[0113] That is, H = T 2 -1 *T 1 .

[0114] Step 503: Based on the sensor posture transformation matrix, determine the intermediate contour data of the target detection area contour converted to the human body partial image.

[0115] Among them, the intermediate contour data represents the contour of the target detection area in the camera coordinate system.

[0116] Optionally, after determining the sensor attitude transformation matrix, the data of each pixel point in the target detection contour is multiplied by the sensor attitude transformation matrix to obtain intermediate contour data.

[0117] That is, after determining the posture sensor matrix, the target detection area contour P in the human body global image 1 The intermediate contour data P obtained after conversion to the human body local image 2 =H*P 1 .

[0118] For example, if the middle contour data is ptsInCamGlobal2Local, then

[0119] ptsInCamGlobal2Local=camGlobal2Local*ptsInCamGlobal

[0120] Among them, camGlobal2Local is the sensor posture transformation matrix, and ptsInCamGlobal is the target detection area outline.

[0121] Step 504 : determining a two-dimensional path corresponding to the intermediate contour data in the human body partial image based on internal parameters of the imaging sensor.

[0122] Among them, the two-dimensional path represents the contour of the target detection area in the uv coordinate system.

[0123] Usually, the internal parameter K of the imaging sensor is the internal parameter matrix, and the internal parameter matrix is:

[0124] K=[f x ,0,c x ; 0,f y ,c y ; 0,0,1]

[0125] Among them, f x and f y are the focal lengths of the imaging sensor in the x and y directions, respectively, which determine the image scaling, and c x and c y It is the coordinates of the principal point of the image plane, that is, the coordinates of the intersection of the optical axis and the image plane in the image coordinate system.

[0126] Based on the camera internal parameters, the pixel coordinates in the UV coordinate system corresponding to each pixel in the intermediate contour data are determined as follows:

[0127] u=f x *(X cam / Z cam )+c x

[0128] v=f y *(Y cam / Z cam )+c y

[0129] Among them, X cam , Y cam and Z cam It is the value of each component of the three-dimensional coordinate corresponding to each pixel corresponding to the intermediate contour data.

[0130] Step 505, traverse the pixels in the two-dimensional path, and determine the target point at the corresponding position in the global three-dimensional point cloud of the human body based on the coordinates of each pixel in the two-dimensional path.

[0131] Optionally, each pixel point in the global image of the human body has a pixel identifier, and there is a point cloud and a pixel point with a corresponding pixel identifier in the corresponding position of the global three-dimensional point cloud of the human body. After mapping the contour of the target detection area in the global image of the human body into a two-dimensional path, each pixel in the two-dimensional path still has a corresponding pixel identifier, so that the target point can be determined in the global three-dimensional point cloud of the human body based on the pixel identifier.

[0132] Step 506: If the three-dimensional path does not contain the target point and all components of the three-dimensional coordinates of the target point are not zero, add the target point to the three-dimensional path.

[0133] In the process of traversing the pixels of the two-dimensional path, after each three-dimensional point corresponding to a pixel is determined, it is necessary to determine whether the three-dimensional point is meaningful. Points with zero coordinates in the three-dimensional path may represent some invalid information. For example, in the coordinates of three-dimensional space, if a coordinate component is 0, it means that the point is on a special plane in space, but in actual path planning, such a point may not be in the effective working space of the robot. Therefore, after determining the target point, it is necessary to first determine whether all coordinate components in the three-dimensional coordinates are not 0.

[0134] Step 507, determining the normal vector corresponding to each point in the three-dimensional path.

[0135] Optionally, in the process of determining the normal vector corresponding to each point in the three-dimensional path, the computer device traverses each point in the three-dimensional path, and then determines the normal vector corresponding to each point through the NormalEstimationOMP (Orthogonal Matching Pursuit for NormalEstimation) algorithm.

[0136] Step 508: Perform trajectory planning based on the normal vector and convert the three-dimensional path into a motion point sequence.

[0137] The moving point sequence refers to a sequence of points to be detected in the camera coordinate system when detecting based on a three-dimensional path.

[0138] Generally, when using a robotic arm and detection equipment to detect the human body surface, the three-dimensional path (i.e. point cloud data) can describe the information of the points to be detected in the form of discrete points, and the normal vector can further indicate the directional characteristics of the three-dimensional path. This is conducive to more accurate trajectory planning by computer equipment.

[0139] Optionally, when converting the three-dimensional path into a sequence of motion points that the robot can actually execute, the three-dimensional path can be discretized, that is, the continuous three-dimensional path is decomposed into a series of discrete points according to a certain step size or interval, and for each discretized motion point, the posture of the end fixture of the robot arm at that point is further adjusted according to the direction quantity corresponding to the point and the posture constraint condition. Among them, the posture constraint condition is the appropriate posture of the end effector (clamp) of the mechanical table set for the automatic detection task to be performed in the embodiment of the present application. After the posture adjustment, the discrete points are arranged in order, so as to obtain a sequence of motion points.

[0140] Among them, the motion point sequence includes the position that the robot arm should reach at each moment during the task execution and the posture of the corresponding end effector (gripper).

[0141] Step 509, based on the hand-eye calibration result and the motion point sequence, determine the target robot arm position and posture in the base coordinate system.

[0142] The target robot arm posture in the base coordinate system is used to characterize the relative position relationship between the robot arm end effector and the robot arm base when detecting the detection point.

[0143] Optionally, according to the state of the robot, the computer device can obtain the pose of the robot end effector in the base coordinate system as tool2Base. It is known that the pose of the robot end coordinate system (flange coordinate system) in the robot end fixture coordinate system is flage2Tool, then the pose of the robot end coordinate system in the base coordinate system can be determined as flage2Base = tool2Base * flage2Tool. Since the hand-eye calibration result is used to characterize the pose of the imaging sensor in the base coordinate system, the hand-eye calibration result is cam2Flange, then based on the pose of the robot end coordinate system in the base coordinate system, the pose of the motion point sequence in the camera coordinate system in the base coordinate system can be determined as follows:

[0144] trajInBase=flange2Base*cam2Flange*trajInCam.

[0145] Among them, trajInBase is the pose corresponding to the motion point sequence in the base coordinate system, that is, the pose of the end effector of the robot arm in the base coordinate system during detection, and trajInCam is the motion point sequence in the camera coordinate system.

[0146] In the embodiment of the present application, based on the change of the robot arm's posture and the hand-eye calibration results, the outline of the target detection area on the global image of the human body is projected onto the local image, that is, the position information in the global image is converted to the local image, so as to avoid subsequent calculation errors caused by incomplete information in the local image. In addition, determining the motion point sequence based on the normal vector corresponding to each point in the three-dimensional path is conducive to optimizing the smoothness of the trajectory of the robot arm's movement, and enables the robot arm's movement to adapt to complex scenes in complex scenes, thereby improving the accuracy of the robot arm's posture. In addition, based on the three-dimensional path, after determining the motion point sequence, the motion point sequence in the camera coordinate system is converted into the posture of the robot arm's end effector in the base coordinate system, which can realize the automatic control of the robot arm when detecting the target detection area and reduce labor costs.

[0147] In one possible implementation, since the point cloud data generated by the depth camera when shooting the human body will generate noise due to external influences, that is, the global three-dimensional point cloud of the human body is not smooth and relatively discrete, before determining the target robotic arm position based on the three-dimensional path, the three-dimensional path needs to be smoothed first, so that after the target robotic arm position is determined, the robotic arm movement is smoother and faster.

[0148] That is, the three-dimensional path is first smoothed to obtain a smooth three-dimensional path, and then the target robot arm position and posture are determined based on the smooth three-dimensional path.

[0149] Please refer to Figure 7 , which shows a flowchart of a process of smoothing a three-dimensional path provided by an exemplary embodiment of the present application, the process comprising the following steps:

[0150] Step 701, perform linear interpolation on the three-dimensional path to obtain an expanded dense three-dimensional path.

[0151] When performing linear interpolation, there may be a set linear interpolation interval thresd, and a new point data is inserted into the three-dimensional path according to the linear interpolation interval to obtain an expanded dense three-dimensional path.

[0152] The specific process of linear interpolation is not described in detail in this embodiment.

[0153] Step 702, determining the middle three-dimensional path of the three-dimensional path in the cube occupied by the global three-dimensional point cloud of the human body.

[0154] First, determine the maximum and minimum point coordinates in the three-dimensional path, which are minPt and minPt respectively. For example, minPt and minPt can be determined by the pcl::compute3DMinMax function in PCL (Point Cloud Library).

[0155] Then, traverse the points on the global three-dimensional point cloud of the human body to find all points within the range of minPt and minPt. These points are the intermediate three-dimensional paths of the three-dimensional path in the cube occupied by the global three-dimensional point cloud of the human body.

[0156] Step 703: determine the intersecting three-dimensional path of the intermediate three-dimensional path and the dense three-dimensional path.

[0157] In a possible implementation, each point in the dense 3D path is traversed, for example, each point in the dense 3D path is eachPt, and all points in the intermediate 3D path with eachPt as the center and a radius of k_distances are searched. If there is only one point in the range, it is added to the intersecting 3D path; if there are multiple points in the range, two candidate points closest to eachPt and on both sides of eachPt are taken, the midpoint value of the two candidate points is calculated, and the midpoint value is added to the intersecting 3D path, thereby obtaining the intersecting 3D path.

[0158] Among them, all points within the radius k_distances of the point eachPt in the three-dimensional path can be found based on the pcl::radiusSearch function in PCL.

[0159] Step 704, traverse the three-dimensional point coordinates in the intersecting three-dimensional paths, perform mean filtering on the intersecting three-dimensional paths, and obtain a smooth three-dimensional path.

[0160] Optionally, traverse the 3D point eachPt in the intersecting 3D path, take the average value of all points within the radius k_indices around eachPt to replace the 3D point eachPt. In addition, duplicate point coordinates can be filtered at the same time to obtain a smooth 3D path after smoothing.

[0161] Please refer to Figure 8 , which shows a schematic diagram of a smooth three-dimensional path provided by an exemplary embodiment of the present application, wherein the smooth three-dimensional path 801 is consistent with the above Figure 4 corresponds to the two-dimensional path in .

[0162] In an embodiment of the present application, a three-dimensional path smooth fitting is adopted, including a smoothing algorithm for the three-dimensional point cloud path, so as to solve the problem that the points in the three-dimensional path do not fit the actual human body surface due to poor camera accuracy. This is beneficial for better detection of the target detection area on the human body surface when detecting based on the determined target robot arm posture, and the movement path of the robot arm is smoother, which is conducive to smoother detection of the target detection area.

[0163] Please refer to Fig. 9 , which shows a structural block diagram of an electronic device provided by an exemplary embodiment of the present application. The electronic device in the present application may include one or more of the following components: a processor 910 and a memory 920.

[0164] Optionally, the processor 910 executes the steps in the method for determining the robot arm posture provided in any of the above embodiments by running or executing instructions, programs, code sets or instruction sets stored in the memory 920, and calling data stored in the memory 920.

[0165] In addition, the processor can also perform various functions of the device and process data. Optionally, the processor 910 can be implemented in at least one hardware form of digital signal processing (DSP), field-programmable gate array (FPGA), and programmable logic array (PLA). The processor 910 can integrate one or a combination of a central processing unit (CPU), a graphics processing unit (GPU), a neural network processor (NPU), and a modem. Among them, the CPU mainly processes the operating system, user interface, and application programs; the GPU is responsible for rendering and drawing the content to be displayed on the touch display; the NPU is used to implement artificial intelligence (AI) functions; and the modem is used to process wireless communications. It is understandable that the above-mentioned modem may not be integrated into the processor 910, but may be implemented separately through a chip.

[0166] The memory 920 may include a random access memory (RAM) or a read-only memory (ROM). Optionally, the memory 920 includes a non-transitory computer-readable storage medium. The memory 920 may be used to store instructions, programs, codes, code sets, or instruction sets. The memory 920 may include a program storage area and a data storage area, wherein the program storage area may store instructions for implementing an operating system, instructions for at least one function (such as a touch function, a sound playback function, an image playback function, etc.), instructions for implementing the following various method embodiments, etc.; the data storage area may store data created according to the use of the device (such as audio data, a phone book), etc.

[0167] The device in the embodiment of the present application further includes a communication component 930 and a display component 940. The communication component 930 may be a Bluetooth component, a WiFi component, an NFC (Near Field Communication) component, etc., for communicating with an external device (server or other device) via a wired or wireless network; the display component 940 is used to display a graphical user interface and / or receive user interaction operations.

[0168] In addition, those skilled in the art will appreciate that the structure of the device shown in the above drawings does not constitute a limitation on the device, and the device may include more or fewer components than shown, or combine certain components, or arrange the components differently. For example, the device also includes components such as a radio frequency circuit, an input unit, a sensor, an audio circuit, a speaker, and a power supply, which will not be described in detail here.

[0169] An embodiment of the present application also provides a non-transitory computer-readable storage medium storing computer instructions, wherein the storage medium stores at least one program code, and the program code is loaded and executed by a processor to implement the method for determining the position and posture of a robotic arm as described in any of the above embodiments.

[0170] The embodiment of the present application provides a computer program product, which includes computer instructions stored in a computer-readable storage medium. When the computer program is executed by a processor, the method for determining the position and posture of a robotic arm as described in any of the above embodiments is implemented.

Claims

1. A method for determining the position and posture of a robotic arm, characterized in that: The method comprises: Acquire a global image of a human body and a local image of a human body through an imaging sensor, wherein both the global image of a human body and the local image of a human body include a target detection area, and the imaging sensor is fixed at the end of the robotic arm; Segmenting the global image of the human body into a target detection region to determine a contour of the target detection region; Mapping the target detection area contour in the human body global image to the human body local image to obtain a two-dimensional path in the human body local image; Projecting the pixels of the two-dimensional path onto a three-dimensional point cloud to obtain a three-dimensional path; Based on the three-dimensional path, a target robotic arm posture is determined, where the target robotic arm posture refers to an actuator posture at the end of the robotic arm corresponding to when the target detection area is detected based on the three-dimensional path.

2. The method according to claim 1, characterized in that The step of obtaining the two-dimensional path in the human body partial image comprises: Based on the first robotic arm posture when shooting the global image of the human body, the second robotic arm posture when shooting the local image of the human body, and the hand-eye calibration result, determine the sensor posture transformation matrix, the robotic arm posture is used to describe the relative position and posture relationship between the robotic arm and the robotic arm base, and the root of the robotic arm is fixed to the robotic arm base; Based on the sensor posture transformation matrix, determining the conversion of the target detection area contour to intermediate contour data of the human body partial image, wherein the intermediate contour data represents the target detection area contour in a camera coordinate system; Based on the internal parameters of the imaging sensor, the two-dimensional path corresponding to the intermediate contour data in the partial human body image is determined, and the two-dimensional path represents the contour of the target detection area in the uv coordinate system.

3. The method according to claim 2, characterized in that The steps of determining the sensor attitude transformation matrix include: Based on the hand-eye calibration result and the first robotic arm posture, determine a first sensor posture matrix, wherein the sensor posture matrix is ​​used to describe the relative position and posture relationship between the imaging sensor and the robotic arm base; Determine a second sensor posture matrix based on the hand-eye calibration result and the second robotic arm posture; The sensor posture transformation matrix for transforming the imaging sensor from a first sensor posture to a second sensor posture is determined based on the first sensor posture matrix and the second sensor posture matrix.

4. The method according to claim 3, characterized in that The step of determining a first sensor posture matrix comprises: The product of the hand-eye calibration result and the first robotic arm posture is determined as the first sensor posture matrix; Based on the hand-eye calibration result and the second robotic arm posture, the step of determining a second sensor posture matrix includes: Determine the product of the hand-eye calibration result and the second robotic arm posture as the second sensor posture matrix; The step of determining the sensor posture transformation matrix of the imaging sensor from a first sensor posture to a second sensor posture comprises: The sensor attitude transformation matrix is ​​determined as the product of the first sensor attitude matrix and the inverse matrix of the second sensor attitude matrix.

5. The method according to claim 4, characterized in that The step of determining the target detection area contour converted to the intermediate contour data of the human body partial image comprises: The product of each pixel point data in the target detection contour and the sensor posture transformation matrix is ​​determined as the intermediate contour data.

6. The method according to claim 1, characterized in that The step of projecting the pixels of the two-dimensional path to a three-dimensional point cloud to obtain the three-dimensional path includes: Traversing the pixels in the two-dimensional path, and determining a target point at a corresponding position in the global three-dimensional point cloud of the human body based on the coordinates of each pixel in the two-dimensional path; When the target point is not included in the three-dimensional path and each component of the three-dimensional coordinates of the target point is not 0, the target point is added to the three-dimensional path.

7. The method according to claim 1, characterized in that Before determining the target robot arm position and posture based on the three-dimensional path, the method further includes: Smoothing the three-dimensional path to obtain a smooth three-dimensional path; Determining the target robot arm position and posture based on the three-dimensional path includes: Based on the smooth three-dimensional path, the target robotic arm position and posture is determined.

8. The method according to claim 7, characterized in that The step of smoothing the three-dimensional path to obtain a smooth three-dimensional path includes: Performing linear interpolation on the three-dimensional path to obtain an expanded dense three-dimensional path; Determine an intermediate three-dimensional path of the three-dimensional path within the cube occupied by the global three-dimensional point cloud of the human body; Determine an intersecting three-dimensional path of the intermediate three-dimensional path and the dense three-dimensional path; The three-dimensional point coordinates in the intersecting three-dimensional paths are traversed, and the intersecting three-dimensional paths are subjected to mean filtering to obtain the smooth three-dimensional path.

9. The method according to claim 1, characterized in that: The steps to determine the target robot arm position include: Determine the normal vector corresponding to each point in the three-dimensional path; Perform trajectory planning based on the normal vector, and convert the three-dimensional path into a motion point sequence, where the motion point sequence refers to a sequence of points to be detected in a camera coordinate system when performing detection based on the three-dimensional path; Based on the hand-eye calibration result and the motion point sequence, the target robot arm posture in the base coordinate system is determined. The target robot arm posture in the base coordinate system is used to characterize the relative position relationship between the robot arm end effector and the robot arm base when detecting the point to be detected.

10. The method according to claim 1, characterized in that The step of determining the target detection area contour includes: Detecting key points of a human body based on the global image of the human body; Based on the human body key points, the target detection area is segmented to determine the outline of the target detection area.

11. An electronic device, comprising: at least one processor; And a memory communicatively connected to the at least one processor; wherein the memory stores instructions executable by the at least one processor, and the instructions are executed by the at least one processor so that the at least one processor can execute the robot arm posture determination method as described in any one of claims 1-10.

12. A non-transitory computer-readable storage medium storing computer instructions, wherein: The computer instructions are used to enable the computer to execute the method for determining the position and posture of a robotic arm as described in any one of claims 1-10.

13. A computer program product, comprising a computer program, wherein when the computer program is executed by a processor, the method for determining the position and posture of a robotic arm as claimed in any one of claims 1 to 10 is implemented.

Images