Control method of mechanical arm and ultrasonic equipment

Abstract

The invention is suitable for the technical field of mechanical arm control, and provides a mechanical arm control method and ultrasonic equipment. The mechanical arm is provided with a motion actuator, and the control method of the mechanical arm comprises the steps that the historical pose and the current pose of a tracked object are obtained; determining pose control data of the motion actuator according to the pose change between the historical pose and the current pose; and controlling the mechanical arm to synchronously move along with the tracked object according to the pose control data. According to the embodiment of the invention, the motion actuator can be controlled to reproduce the pose change condition of the tracked object, and remote control of the ultrasonic probe in the field of ultrasonic scanning is facilitated.

Description

Technical Field

[0001] The present application belongs to the technical field of robot arm control, and in particular, relates to a robot arm control method and ultrasonic equipment. Background Art

[0002] In the medical field, the quality of the diagnostic process often depends on the doctor's operating skills. Take ultrasound scanning as an example. For the same lesion, different doctors have different techniques and experiences, which may result in different diagnostic results. Traditional ultrasound scanning requires doctors to hold ultrasound probes on the patient's body, which means that in many cases, experts or patients need to be transported so that doctors can operate next to the patient, but this will consume a lot of manpower and material resources. Therefore, some remote-controlled robotic arms have emerged for ultrasound scanning.

[0003] The current remote control methods generally control the robotic arm based on fixed equipment such as the robotic arm main hand, remote sensing equipment, virtual reality equipment, etc., which are not universal enough for some scenarios where fixed equipment cannot be carried or used. Summary of the invention

[0004] The embodiments of the present application provide a control method of a robotic arm and an ultrasonic device, which can control a motion actuator to reproduce the posture transformation of a tracked object, thereby facilitating remote control of an ultrasonic probe in the field of ultrasonic scanning.

[0005] A first aspect of an embodiment of the present application provides a control method for a robotic arm, wherein the robotic arm is provided with a motion actuator, and the control method comprises: acquiring a historical posture and a current posture of a tracked object; determining posture control data of the motion actuator according to a posture change between the historical posture and the current posture; and controlling the robotic arm to follow the tracked object for synchronous movement according to the posture control data.

[0006] In some embodiments of the first aspect, obtaining the historical posture and current posture of the tracked object includes: obtaining an object image of the tracked object, the object image including at least a first image reflecting the historical posture and a second image reflecting the current posture; determining a marking point in the first image, the marking point being used to identify the tracked object in the first image; determining the posture of the object at the marking point in the first image to obtain the historical posture; and determining the current posture based on the historical posture and the second image.

[0007] In some embodiments of the first aspect, determining the position and posture of the object at the marking point in the first image includes: acquiring a three-dimensional model of the tracked object; and aligning the three-dimensional model of the tracked object with the object at the marking point to obtain the position and posture of the object at the marking point.

[0008] In some embodiments of the first aspect, determining the posture of the object at the marked point in the first image to obtain the historical posture includes: segmenting the object at the marked point in the first image to obtain a mask of the object at the marked point; and matching the masks of the tracked object in multiple preset postures according to the mask of the object at the marked point to determine the historical posture.

[0009] In some embodiments of the first aspect, determining the current posture based on the historical posture and the second image includes: determining the current posture based on the historical posture, a three-dimensional model of the tracked object, and the second image.

[0010] In some embodiments of the first aspect, determining the posture control data of the motion actuator based on the posture change between the historical posture and the current posture includes: aligning a first coordinate system where the historical posture and / or the current posture is located with a second coordinate system where the motion actuator is located to obtain an alignment result; and converting the posture change between the current posture and the historical posture based on the alignment result to obtain the posture control data.

[0011] In some implementations of the first aspect, after acquiring the historical posture and current posture of the tracked object, the method further includes: filtering the historical posture and current posture.

[0012] In some embodiments of the first aspect, determining the posture control data of the motion actuator based on the posture change between the historical posture and the current posture includes: scaling the posture change and determining the posture control data based on the scaled posture change.

[0013] In some embodiments of the first aspect, the control method also includes performing admittance control on the movement of the robotic arm, including: obtaining the actual force of the motion actuator when performing synchronous movement, and the damping coefficient of the moving surface; and performing admittance control on the synchronous movement of the robotic arm based on the actual force and the damping coefficient.

[0014] In some embodiments of the first aspect, obtaining the actual force of the motion actuator when performing synchronous motion includes: obtaining the current posture of the force sensor and the original force of the motion actuator collected by the force sensor; determining the mass of the motion actuator, the center of mass position of the motion actuator, the zero point of the force sensor, and the base installation inclination angle of the robotic arm; based on the current posture of the force sensor, the mass of the motion actuator, the center of mass position of the motion actuator, the zero point of the force sensor, and the base installation inclination angle, correcting the original force to determine the actual force.

[0015] In some embodiments of the first aspect, the admittance control includes: obtaining the motion state of the motion actuator; determining the change in the damping coefficient based on the change of the motion surface over time; determining the error based on the actual force and the preset force; and adjusting the motion state of the motion actuator in real time according to the relationship between the error, the change in the damping coefficient and the motion state.

[0016] A second aspect of an embodiment of the present application provides a control device for a robotic arm, wherein the robotic arm is provided with a motion actuator, and the control device includes: a posture acquisition unit, used to acquire a historical posture and a current posture of a tracked object; a posture mapping unit, used to determine posture control data of the motion actuator according to a posture change between the historical posture and the current posture; and a robotic arm control unit, used to control the robotic arm to follow the tracked object for synchronous movement according to the posture control data.

[0017] A third aspect of an embodiment of the present application provides an ultrasound device, comprising: a control module, a robotic arm, and an ultrasound probe arranged on the robotic arm; the control module is used to include a memory, a processor, and a computer program stored in the memory and executable on the processor, and when the processor executes the computer program, the steps of the control method of the robotic arm as described in any one of the first aspects are implemented; the robotic arm is used to control the movement of the ultrasound probe.

[0018] In some embodiments of the third aspect, the ultrasound device further includes a force sensor disposed between the mechanical arm and the ultrasound probe; the force sensor is used to collect an actual force between the motion actuator and the object it contacts.

[0019] A fourth aspect of an embodiment of the present application provides a computer-readable storage medium, wherein the computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the steps of the above-mentioned method for controlling the robotic arm are implemented.

[0020] A fifth aspect of an embodiment of the present application provides a computer program product. When the computer program product is run on an electronic device, the electronic device executes the steps of the above-mentioned method for controlling the robotic arm.

[0021] In the implementation of the present application, the posture control data of the motion actuator is determined based on the posture change between the historical posture and the current posture of the tracked object, and the robotic arm is controlled to follow the tracked object for synchronous movement according to the posture control data, so that the motion actuator can be controlled to reproduce the posture change of the tracked object. This method can use any object to remotely control the motion actuator, and has high universality. In the field of ultrasonic scanning, the ultrasonic probe on the robotic arm can be remotely controlled through any object in scenarios where it is impossible to carry or use fixed equipment, and the imaging quality of the ultrasonic image can be maintained. BRIEF DESCRIPTION OF THE DRAWINGS

[0022] In order to more clearly illustrate the technical solutions in the embodiments of the present application, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative labor.

[0023] Figure 1 It is a schematic diagram of the implementation flow of the control method of the robotic arm provided in an embodiment of the present application;

[0024] Figure 2 It is a schematic diagram of a specific implementation process of determining historical posture and current posture provided in an embodiment of the present application;

[0025] Figure 3 is a schematic diagram of a first image provided in an embodiment of the present application;

[0026] Figure 4 This is a schematic diagram of a specific implementation process for determining the current posture provided in an embodiment of the present application;

[0027] Figure 5 is a schematic diagram of a robotic arm provided in an embodiment of the present application;

[0028] Figure 6 is a flow chart of the admittance control provided by the embodiment of the present application;

[0029] Figure 7 is a structural schematic diagram of a control device for a robotic arm provided in an embodiment of the present application;

[0030] Figure 8 is a schematic diagram of the structure of an electronic device provided in an embodiment of the present application;

[0031] Fig. 9It is a schematic diagram of the structure of the ultrasonic device provided in the embodiment of the present application. DETAILED DESCRIPTION

[0032] In order to make the purpose, technical solutions and advantages of the present application more clearly understood, the present application is further described in detail below in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present application and are not intended to limit the present application. Based on the embodiments of the present application, all other embodiments obtained by those skilled in the art without making creative work are protected by the present application.

[0033] It should be understood that when used in the present specification and the appended claims, the term "comprising" indicates the presence of described features, wholes, steps, operations, elements and / or components, but does not exclude the presence or addition of one or more other features, wholes, steps, operations, elements, components and / or combinations thereof.

[0034] In the description of the present application specification and the appended claims, the terms "first", "second", "third", etc. are only used to distinguish the descriptions and cannot be understood as indicating or implying relative importance.

[0035] References to "one embodiment" or "some embodiments" etc. described in the specification of this application mean that one or more embodiments of the present application include specific features, structures or characteristics described in conjunction with the embodiment. Therefore, the statements "in one embodiment", "in some embodiments", "in some other embodiments", "in some other embodiments", etc. that appear in different places in this specification do not necessarily refer to the same embodiment, but mean "one or more but not all embodiments", unless otherwise specifically emphasized in other ways. The terms "including", "comprising", "having" and their variations all mean "including but not limited to", unless otherwise specifically emphasized in other ways.

[0036] The current remote control methods generally control the robotic arm based on fixed equipment such as the robotic arm main hand, remote sensing equipment, virtual reality equipment, etc., which are not universal enough for some scenarios where fixed equipment cannot be carried or used.

[0037] In view of this, the present application proposes a control method for a robotic arm, which can enable a motion actuator to reproduce the posture changes of a tracked object, thereby allowing the motion actuator to be remotely controlled using any object.

[0038] In order to illustrate the technical solution of the present application, a specific embodiment is provided below for illustration.

[0039] Please refer to Figure 1 , Figure 1A schematic diagram of the implementation flow of a control method for a robotic arm provided in an embodiment of the present application is shown, and the method can be applied to electronic equipment.

[0040] The electronic device may refer to a remote control terminal of the robot arm, such as a computer, a mobile phone or other smart device with remote communication capabilities. The electronic device may also refer to a medical device equipped with a robot arm, which can perform medical operations through the robot arm.

[0041] In some embodiments of the present application, the electronic device may be an ultrasound device. The ultrasound device may include a control module and a scanning module, and the control module and the scanning module communicate with each other via wired or wireless means, wherein the control module may be used to monitor the position of the tracked object to determine the control instructions of the robotic arm, and the scanning module may be used to control the robotic arm to scan the patient based on the control instructions.

[0042] In an embodiment of the present application, the above-mentioned robot arm may be provided with a motion actuator. The motion actuator may be located at any position of the robot arm, for example, at the end of the robot arm. The motion actuator refers to a tool with a certain function, for example, an ultrasound probe, a surgical instrument, etc. The present application does not limit the model of the robot arm, for example, it may be a UR 5e robot arm.

[0043] Specifically, the control method of the above-mentioned robot arm may include the following steps S101 to S103.

[0044] Step S101, obtaining the historical position and current position of the tracked object.

[0045] The tracked object is an object that needs a motion actuator to reproduce a posture change, and may refer to an object held by a user. This application does not limit the type of the tracked object, and any object that is easy to access may be used as the tracked object.

[0046] The current pose is the pose of the tracked object at the current moment, and the historical pose is the pose of the tracked object at any moment before the current moment, for example, it may include the initial pose and the pose at the previous moment.

[0047] This application does not limit the method of obtaining the historical posture and current posture. Specifically, the electronic device can obtain the historical posture and current posture by photographing the tracked object and using the object image to detect the posture. In other implementations, if the tracked object itself has a posture sensor, the historical posture and current posture detected by the posture sensor can also be obtained.

[0048] In some embodiments of the present application, in order to ensure the reliability of the posture data, after obtaining the historical posture and the current posture, the historical posture and the current posture can be filtered to ensure the smoothness of the movement of the robot arm when reproducing the posture change. The present application does not limit the filtering algorithm, for example, the Kalman filter algorithm can be used.

[0049] Step S102, determining the posture control data of the motion actuator according to the posture change between the historical posture and the current posture.

[0050] In the implementation manner of the present application, the posture change characterizes the change in position and posture between the historical posture and the current posture. It is understandable that when determining the posture control data of the motion actuator, it can be determined based on the change between the initial posture and the current posture, or it can be determined based on the change between the posture at the previous moment and the current posture. The specific posture at which moment is selected as the historical posture can be selected according to the time period of the posture transformation that needs to be reproduced by the motion actuator.

[0051] The posture control data can be used to control the motion actuator to change its posture according to the posture change of the tracked object. Specifically, the posture control data can represent the amount of posture change that the motion actuator needs to adjust. In this case, the amount of posture change of the motion actuator corresponds to the posture change between the historical posture and the current posture of the tracked object, and is used to control the motion actuator to make an equal amount of posture change. The posture control data can also represent the target posture that the motion actuator needs to reach in the end. In this case, the amount of posture change between the target posture and the current posture of the motion actuator corresponds to the posture change between the historical posture and the current posture of the tracked object.

[0052] It should be noted that in some embodiments of the present application, the posture change can be scaled, and the posture control data of the motion actuator can be determined according to the scaled posture change. For example, when performing a fine ultrasound scan, the posture change can be scaled down to prevent the ultrasound probe (motion actuator) from moving too fast.

[0053] Step S103, controlling the robotic arm to follow the tracked object for synchronous movement according to the posture control data.

[0054] In an embodiment of the present application, the robotic arm is controlled according to the posture control data, so that the robotic arm can move synchronously with the tracked object, so that the motion actuator carried by the robotic arm reproduces the posture change between the current posture and the historical posture of the tracked object. Specifically, the historical posture of the robotic arm can be obtained, and the historical posture of the robotic arm can be calculated by the mechanical structure of the robotic arm, which can usually be read out using the function in the control cabinet. Based on the historical posture and posture control data of the robotic arm, the target posture of each joint of the robotic arm can be determined. By adjusting each joint to the target posture, the joint will drive the motion actuator of the robotic arm to adjust the posture, so that the motion actuator changes its posture according to the posture change between the current posture and the historical posture of the tracked object, and reaches the target posture that the motion actuator needs to reach in the end.

[0055] In the implementation of the present application, based on the posture change between the historical posture and the current posture of the tracked object, the posture control data of the motion actuator is determined, and the mechanical arm is controlled to follow the tracked object for synchronous movement according to the posture control data, so that the motion actuator can be controlled to reproduce the posture change of the tracked object. This method can use any object to remotely control the motion actuator, and has high universality.

[0056] In the field of ultrasound scanning, the ultrasound probe on the robotic arm can be remotely controlled through any object while maintaining the imaging quality of the ultrasound image in scenarios where it is impossible to carry or use fixed equipment.

[0057] In some embodiments of the present application, Figure 2 As shown, obtaining the historical position and current position of the tracked object may include: step S201 to step S204.

[0058] Step S201, obtaining an object image of the tracked object.

[0059] The object image of the tracked object includes at least a first image reflecting the historical position and a second image reflecting the current position. It should be noted that the above object image can be an RGB image or an RGB-D image, which is not limited in this application.

[0060] In some embodiments of the present application, a camera may be set at a position convenient for capturing the movement of the tracked object, for example, the camera may be at the same horizontal position or above the tracked object. The camera may shoot the tracked object to obtain an object image. When the object image is two frames, it may include a first image of the tracked object in a historical position (i.e., the first frame image), and a second image in a current position (i.e., the last frame image). When the object image is n (n>2) frames, n-2 frames of images may be included between the first image and the second image.

[0061] It should be noted that the above-mentioned camera can be an RGB camera, an RGB-D camera, or a combination of RGB and infrared cameras. This application does not limit the model of the camera. For example, the above-mentioned RGB-D camera can be a Realsense D455 camera.

[0062] Step S202: determining a marking point on the first image.

[0063] The marking point may be used to identify the tracked object in the first image, and may be any point of the tracked object in the first image.

[0064] In some embodiments of the present application, a user's selection operation on a pixel point on the first image may be received to use the selected pixel point as a marking point. Figure 3 When the user holds the tracked object and moves it, the electronic device can obtain an object image obtained by photographing the tracked object. In the first image, the user can select any point of the tracked object as a marking point (for example, Figure 3 In some other embodiments of the present application, object detection may be performed on the first image to identify the image region where the tracked object is located, and any pixel point may be selected as a marking point in the image region where the tracked object is located. The present application does not limit the method for determining the marking point.

[0065] Step S203, determining the position and posture of the object at the marked point in the first image to obtain a historical position and posture.

[0066] In some embodiments of the present application, step S203 may include: acquiring a three-dimensional model of the tracked object, aligning the three-dimensional model of the tracked object with the object at the marking point, and obtaining the position and posture of the object at the marking point.

[0067] Specifically, the electronic device can display a plurality of three-dimensional models of different categories and shapes, and the user can select a three-dimensional model according to the category and shape of the tracked object. In response to the user's selection operation of the three-dimensional model, the electronic device can use the three-dimensional model selected by the user as the three-dimensional model of the tracked object. The three-dimensional model here can refer to a computer-aided design (Computer-Aided Design, CAD) model, or a 3D MAX model or other types of three-dimensional models, which is not limited in this application. Based on the marking points on the first image, the electronic device can display the three-dimensional model of the tracked object in the first image, so that the three-dimensional model covers the marking points on the first image, which is a rough positioning. After the rough positioning is completed, the point cloud registration algorithm (Iterative Closest Point, ICP) can be used to align the three-dimensional model of the tracked object with the object on the marking point, so that the three-dimensional model of the tracked object is aligned with the outline of the object on the marking point, and the precise positioning is achieved, and then the posture of the object on the marking point is obtained, and the posture is used as the historical posture of the tracked object.

[0068] Of course, in addition to obtaining the three-dimensional model based on the selection operation, the three-dimensional model can also be obtained by scanning the tracked object through a laser radar or other methods, and this application does not impose any restrictions on this.

[0069] In some other embodiments of the present application, step S203 may include: segmenting the object at the marked point in the first image to obtain a mask of the object at the marked point. According to the mask of the object at the marked point, the masks of the tracked object in multiple preset postures are matched respectively to determine the historical posture.

[0070] Specifically, based on the marking points on the first image, the SAM (Segment Anything Model) algorithm can be used to process the first image and segment the mask of the object at the marking point. The mask can characterize the contour of the object at the marking point in the first image. The mask of the object at the marking point is matched with the mask of the tracked object in multiple preset postures, and a matching score can be performed to determine the matching score between the mask of the object at the marking point and the mask in each preset posture. Based on the matching score, the preset posture with the highest matching score can be selected as the historical posture, that is, the best matching preset posture is selected.

[0071] In some embodiments of the present application, the above-mentioned multiple preset poses can be obtained based on the three-dimensional model of the aforementioned tracked object. The above-mentioned scoring process can be implemented by the FoundationPose model or other existing algorithms, which is not limited by the present application. In this way, the step of manually matching historical poses can be omitted.

[0072] Step S204, determining the current posture according to the historical posture and the second image.

[0073] In some embodiments of the present application, the current posture is determined using the historical posture and the posture of the tracked object reflected by the second image, so that the current posture has temporal continuity.

[0074] In some embodiments of the present application, determining the current posture according to the historical posture and the second image may include: determining the current posture according to the historical posture, a three-dimensional model of the tracked object, and the second image.

[0075] Specifically, for each frame of object image other than the first image, the pose corresponding to the frame of object image can be determined according to the pose corresponding to the previous frame of object image, the three-dimensional model of the tracked object, and the frame of object image, until the current pose corresponding to the second image is obtained. Figure 4 , combined with the pose of the tracked object at time t-1 (the pose at time t-1 is the historical pose when t=1), the 3D model of the tracked object, and the image of the object at time t, the pose at time t can be determined. The pose is calculated from time t=1 until the moment corresponding to the second image is calculated, and the current pose corresponding to the second image can be obtained.

[0076] In some embodiments of the present application, the above process of determining the current posture can be implemented by the deep learning-based object posture algorithm se(3)-TrackNet, or by other posture estimation algorithms, which is not limited by the present application. The se(3)-TrackNet algorithm can effectively detect the tracked object even when it is tilted or tilted. The algorithm still has good tracking capabilities for partial occlusion caused by the hand when holding the object.

[0077] In some other embodiments of the present application, the historical pose and the current pose can be determined based on the object image and can also be directly implemented based on the model. For example, the aforementioned FoundationPose model is used to input the object image into the FoundationPose model, and the FoundationPose model can directly output the historical pose and the current pose.

[0078] The method of obtaining the historical posture and the current posture through the aforementioned three-dimensional model can improve the posture accuracy. In addition to the method of obtaining the historical posture and the current posture through the three-dimensional model, this application does not exclude the use of other existing methods to obtain the historical posture and the current posture.

[0079] After obtaining the historical posture and the current posture, the electronic device can determine the posture control data of the motion actuator.

[0080] In some embodiments of the present application, determining the posture control data of the motion actuator based on the posture change between the historical posture and the current posture can include: aligning a first coordinate system where the historical posture and / or the current posture is located with a second coordinate system where the motion actuator is located to obtain an alignment result; based on the alignment result, converting the posture change between the current posture and the historical posture to obtain posture control data.

[0081] Specifically, the first coordinate system is a coordinate system obtained by taking the tracked object as the origin and the camera coordinate system of multiple frames of object images as a reference. In some embodiments of the present application, a first coordinate system can be established with a reference point of the tracked object as the origin. If the tracked object is a regular object, the midpoint of each dimension of the tracked object can be used as the reference point. If the tracked object is an irregular object, the center of mass of the tracked object can be used as the reference point. The center of mass can refer to the physical center of mass or the center of mass of the three-dimensional model corresponding to the tracked object. The second coordinate system is the position coordinate system of the motion actuator, which is the coordinate system used for the motion control of the robotic arm.

[0082] The above alignment is to align the various degrees of freedom of the first coordinate system and the second coordinate system. Since the variables that control the movement of the robotic arm need to be based on the posture changes of the tracked object, alignment can make the dimension and relative direction of the robotic arm movement control consistent with the tracked object. According to the alignment results, the posture changes between the current posture and the historical posture are converted to obtain posture control data, which can avoid situations such as the tracked object moving in the positive direction of the x-axis and the robotic arm moving in the negative direction of the y-axis. At this time, the posture control data obtained by conversion can represent the posture changes that the motion actuator needs to adjust. If the posture control data represents the target posture that the motion actuator needs to reach in the end, the target posture can be determined based on the current posture of the motion actuator and the change obtained by conversion after converting the posture changes between the current posture and the historical posture according to the alignment results.

[0083] In order to avoid damaging or injuring the above-mentioned contact object, in some embodiments of the present application, the above-mentioned control method may also include conducting admittance control on the movement of the robotic arm, which may specifically include: obtaining the actual force of the motion actuator during synchronous movement, and the damping coefficient of the moving surface; and conducting admittance control on the synchronous movement of the robotic arm based on the actual force and the damping coefficient.

[0084] The actual force when the motion actuator performs synchronous motion refers to the actual force when the motion actuator contacts the contact object.

[0085] In some embodiments of the present application, a force sensor may be provided on the manipulator, and the force sensor may be a six-axis force sensor or other types of force sensors. The force sensor is provided between the manipulator and the motion actuator, and the force sensor may be used to collect the applied force when the motion actuator contacts the object in contact with the motion actuator. Taking the motion actuator as an ultrasonic probe as an example, at this time, the manipulator, the force sensor, and the ultrasonic probe are regarded as a whole. Since the clamping force of the manipulator is generally fixed, the relative position of the ultrasonic probe and the manipulator is ensured to remain unchanged, and real-time measurement can be performed.

[0086] The damping coefficient of the moving surface determines the performance of the robotic arm system in absorbing collision energy, which is related to the tissue surface that the robotic arm contacts and the human body structure under the tissue.

[0087] Based on the actual force and the damping coefficient, the synchronous motion of the robot arm can be controlled by admittance control. Admittance control can monitor and adjust the force between the motion actuator and its contact object in real time to avoid the force being greater than or equal to the preset force, thereby improving safety. Figure 5 A simple schematic diagram of admittance control is output. After the motion actuator contacts its contact object, the robotic arm will apply a corresponding force before reaching the preset force. After exceeding the preset force, the robotic arm can automatically adjust its position through admittance control to adjust the current force.

[0088] Specifically, in some embodiments of the present application, obtaining the actual force of the motion actuator when performing synchronous motion may include: obtaining the current posture of the force sensor and the original force of the motion actuator collected by the force sensor; determining the mass of the motion actuator, the center of mass position of the motion actuator, the zero point of the force sensor, and the base installation inclination angle of the robotic arm; based on the current posture of the force sensor, the mass of the motion actuator, the center of mass position of the motion actuator, the zero point of the force sensor, and the base installation inclination angle, correcting the original force to determine the actual force.

[0089] The original force refers to the original reading of the force collected by the force sensor. Due to the gravity of the motion actuator and the gripper on the robot arm, the original force cannot directly represent the actual contact force generated by the motion actuator and its contact object. By obtaining the current posture of the force sensor and the original force of the motion actuator collected by the force sensor, the mass of the motion actuator, the center of mass position, the zero point of the force sensor and the base installation inclination of the robot arm can be parameterized. Among them, the center of mass position can refer to the coordinate position of the center of mass of the motion actuator in the force sensor coordinate system. Combined with the current posture and zero point of the force sensor, the mass and center of mass position of the motion actuator, and the base installation inclination, the external force and external torque on the tool center point (TCP) can be calculated. Among them, the tool center point is the origin of the tool coordinate system (Tool Coordinate System) of the motion actuator. According to the external force and external torque on the tool center point and the original force, the actual force can be determined by force transformation. In this way, the weight of the motion actuator and the gripper on the robot arm can be subtracted from the original force to achieve gravity compensation and obtain the actual force, thereby ensuring that the actual force obtained is the force actually generated by the contact between the motion actuator and the contact object.

[0090] In some embodiments of the present application, during the process of controlling the synchronous movement of the robotic arm, the electronic device may also display the actual force between the motion actuator and the object it contacts.

[0091] In some embodiments of the present application, admittance control may include: obtaining the motion state of the motion actuator; determining the change in the damping coefficient based on the change of the motion surface over time; determining the error based on the actual force and the preset force; and adjusting the motion state of the motion actuator in real time according to the relationship between the error, the change in the damping coefficient and the motion state.

[0092] The motion state of the motion actuator can characterize the position of the motion actuator. The change in damping coefficient reflects the change in damping coefficient over time. The preset force is the target value of the force, which can be a fixed value or can be set according to different scanning parts. The motion state of the motion actuator can be adjusted in real time according to the relationship between the error between the actual force and the preset force, the change in damping coefficient and the motion state.

[0093] Figure 6The schematic diagram of the admittance control process is shown. The inner loop of the position control of the robot arm uses a position controller to control the position of the motion actuator. The motion state Xc of the motion actuator can be obtained by forward kinematics solution, and the environmental stiffness is solved by combining the position Xe generated by the environmental force. At the same time, the actual force F collected by the force sensor is obtained by combining the gravity compensation algorithm. According to the error between the actual force F and the preset force Fd, the trajectory correction e of the motion actuator can be output, and the trajectory correction e is added to the ideal motion state Xd to obtain the motion state Xc of the motion actuator. In this way, the motion state of the motion actuator can be adjusted in a timely manner through iteration.

[0094] In the implementation of the present application, the position and posture are determined by the image of the object. There is no need to install sensors in the tracked object. The posture of the tracked object can be obtained by the camera and transmitted to the robot arm for execution, which has a low cost. In addition, by reproducing the change of the position and posture of the tracked object, it can give people an intuitive feeling. For doctors performing ultrasound scanning, they do not need to control the robot arm to move forward or backward through the buttons or touch on the handle, but can directly hold any object to simulate the action of the handheld ultrasound probe, which is more realistic and more real-time. Since the tracked object can be any object, the present application is not limited to the use of fixed equipment (manipulator main hand, remote sensing equipment, virtual reality equipment, etc.). Users can quickly build an operating environment through the mobile phone camera they carry with them and objects that are easy to access around them, which is more convenient. In addition to ultrasound scanning, the above-mentioned motion actuator can also be a robot, surgical instrument and other instruments, so that the above-mentioned scheme can be applied to different scenarios, including but not limited to palletizing, object grabbing, massage therapy, etc., with high universality.

[0095] It should be noted that, for the sake of simplicity of description, the aforementioned method embodiments are all expressed as a series of action combinations, but those skilled in the art should know that the present application is not limited to the described order of actions, because according to the present application, certain steps can be performed in other orders.

[0096] like Figure 7 Shown is a schematic structural diagram of a control device 700 for a robotic arm provided in an embodiment of the present application, wherein the control device 700 for the robotic arm is configured on an electronic device.

[0097] Specifically, the control device 700 of the robot arm may include:

[0098] A posture acquisition unit 701 is used to acquire the historical posture and current posture of the tracked object;

[0099] A posture mapping unit 702, configured to determine posture control data of the motion actuator according to a posture change between the historical posture and the current posture;

[0100] The robot arm control unit 703 is used to control the robot arm to follow the tracked object for synchronous movement according to the posture control data.

[0101] In some embodiments of the present application, the posture acquisition unit 701 can be specifically used to: acquire an object image of the tracked object, the object image at least including a first image reflecting the historical posture and a second image reflecting the current posture; determine a marking point in the first image, the marking point being used to identify the tracked object in the first image; determine the posture of the object at the marking point in the first image to obtain the historical posture; and determine the current posture based on the historical posture and the second image.

[0102] In some embodiments of the present application, the posture acquisition unit 701 can be specifically used to: acquire a three-dimensional model of the tracked object; align the three-dimensional model of the tracked object with the object at the marking point to obtain the posture of the object at the marking point.

[0103] In some embodiments of the present application, the posture acquisition unit 701 can be specifically used to: segment the object at the marked point in the first image to obtain the mask of the object at the marked point; according to the mask of the object at the marked point, match the masks of the tracked object in multiple preset postures respectively to determine the historical posture.

[0104] In some embodiments of the present application, the posture acquisition unit 701 may be specifically configured to determine the current posture according to the historical posture, the three-dimensional model of the tracked object, and the second image.

[0105] In some embodiments of the present application, the posture acquisition unit 701 can be specifically used to: align the first coordinate system where the historical posture and / or the current posture is located with the second coordinate system where the motion actuator is located to obtain an alignment result; and according to the alignment result, convert the posture change between the current posture and the historical posture to obtain the posture control data.

[0106] In some implementations of the present application, the posture acquisition unit 701 may be specifically used to: perform filtering processing on the historical posture and the current posture.

[0107] In some implementations of the present application, the posture acquisition unit 701 may be specifically used to: perform scaling processing on the posture change, and determine the posture control data according to the scaled posture change.

[0108] In some embodiments of the present application, the posture mapping unit 702 can also be specifically used to: obtain the actual force of the motion actuator when performing synchronous motion, and the damping coefficient of the motion surface; and perform admittance control on the synchronous motion of the robotic arm based on the actual force and the damping coefficient.

[0109] In some embodiments of the present application, the robotic arm control unit 703 can be specifically used to: obtain the current posture of the force sensor and the original force of the motion actuator collected by the force sensor; determine the mass of the motion actuator, the center of mass position of the motion actuator, the zero point of the force sensor and the base installation inclination angle of the robotic arm; based on the current posture of the force sensor, the mass of the motion actuator, the center of mass position of the motion actuator, the zero point of the force sensor and the base installation inclination angle, correct the original force to determine the actual force.

[0110] In some embodiments of the present application, the robot arm control unit 703 can be specifically used to: obtain the motion state of the motion actuator; determine the change in the damping coefficient based on the change of the motion surface over time; determine the error based on the actual force and the preset force; and adjust the motion state of the motion actuator in real time according to the relationship between the error, the change in the damping coefficient and the motion state.

[0111] It should be noted that, for the convenience and simplicity of description, the specific working process of the control device 700 of the above-mentioned robot arm can be referred to Figures 1 to 6 The corresponding process of the method will not be repeated here.

[0112] like Figure 8 FIG. 8 is a schematic diagram of an electronic device 8 provided in an embodiment of the present application. Specifically, the electronic device 8 may include: a processor 80, a memory 81, and a computer program 82 stored in the memory 81 and executable on the processor 80, such as a control program for a robotic arm. When the processor 80 executes the computer program 82, the steps in the above-mentioned embodiments of the control method for the robotic arm are implemented, such as Figure 1 Alternatively, when the processor 80 executes the computer program 82, the functions of each module / unit in the above-mentioned device embodiments are realized, for example, Figure 7 The functions of the posture acquisition unit 701, the posture mapping unit 702 and the robot control unit 703 are shown.

[0113] The computer program may be divided into one or more modules / units, which are stored in the memory 81 and executed by the processor 80 to complete the present application. The one or more modules / units may be a series of computer program instruction segments capable of completing specific functions, which are used to describe the execution process of the computer program in the electronic device 8.

[0114] For example, the computer program can be divided into: a posture acquisition unit, a posture mapping unit and a manipulator control unit. The specific functions of each unit are as follows: a posture acquisition unit, used to acquire the historical posture and current posture of the tracked object; a posture mapping unit, used to determine the posture control data of the motion actuator according to the posture change between the historical posture and the current posture; a manipulator control unit, used to control the manipulator to follow the tracked object for synchronous movement according to the posture control data.

[0115] The electronic device 8 may include, but is not limited to, a processor 80 and a memory 81. Those skilled in the art will appreciate that Figure 8 It is only an example of the electronic device 8 and does not constitute a limitation of the electronic device 8. It may include more or fewer components than shown in the figure, or a combination of certain components, or different components. For example, the electronic device 8 may also include input and output devices, network access devices, buses, etc.

[0116] The processor 80 may be a central processing unit (CPU), or other general-purpose processors, digital signal processors (DSP), application-specific integrated circuits (ASIC), off-the-shelf programmable gate arrays or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. A general-purpose processor may be a microprocessor or any conventional processor, etc.

[0117] The memory 81 may be an internal storage unit of the electronic device 8, such as a hard disk or memory of the electronic device 8. The memory 81 may also be an external storage device of the electronic device 8, such as a plug-in hard disk, a smart media card (SMC), a secure digital (SD) card, a flash card, etc. equipped on the electronic device 8. Further, the memory 81 may also include both an internal storage unit of the electronic device 8 and an external storage device. The memory 81 is used to store the computer program and other programs and data required by the electronic device 8. The memory 81 may also be used to temporarily store data that has been output or is to be output.

[0118] In some embodiments of the present application, the electronic device 8 may further include a mechanical arm on which a motion actuator is disposed.

[0119] In some embodiments of the present application, the electronic device 8 may refer to an ultrasonic device, and the motion actuator may refer to an ultrasonic probe.

[0120] Specifically, Fig. 9 As shown, the present application also provides an ultrasound device 9 , including: a control module 90 , a mechanical arm 91 , and an ultrasound probe 92 disposed on the mechanical arm 91 .

[0121] The control module 90 may include a memory, a processor, and a computer program stored in the memory and executable on the processor. When the processor executes the computer program, the following operations are performed: Figures 1 to 6 The specific working process of the control module 90 can be referred to Fig. 9 The relevant description of the electronic device shown is not repeated in this application.

[0122] The mechanical arm 91 can be used to control the movement of the ultrasonic probe 92. Specifically, the mechanical arm 91 can include one or more joints. Based on the posture control data, each joint of the mechanical arm 91 can be adjusted to a corresponding target posture, driving the ultrasonic probe 92 on the mechanical arm to adjust the posture, so that the ultrasonic probe 92 changes its posture according to the posture change between the current posture and the historical posture of the tracked object.

[0123] In some embodiments of the present application, a force sensor 93 is provided between the mechanical arm and the ultrasonic probe. The force sensor 93 can be used to collect the actual force between the motion actuator and the object it contacts.

[0124] In addition to the control module 90, the mechanical arm 91, the ultrasonic probe 92 and the force sensor 93, in other embodiments of the present application, the ultrasonic device 9 may also include but is not limited to:

[0125] An image sensor for acquiring an object image of the tracked object;

[0126] A communication module, used to enable the control module 90 to remotely obtain the object image captured by the image sensor;

[0127] An ultrasonic transmitting / receiving circuit, used to generate pulses, cause mechanical oscillation of the ultrasonic probe 92, and amplify and filter the signals received by the ultrasonic probe 92;

[0128] A processing module, used to convert the signal received by the ultrasound probe 92 into a visual image;

[0129] A display is used to display visual images.

[0130] It should be noted that, for the convenience and brevity of description, the structures of the electronic device 8 and the ultrasonic device 9 can also refer to the specific description of the structures in the method embodiment, which will not be repeated here.

[0131] The technicians in the relevant field can clearly understand that for the convenience and simplicity of description, only the division of the above-mentioned functional units and modules is used as an example for illustration. In practical applications, the above-mentioned function allocation can be completed by different functional units and modules as needed, that is, the internal structure of the device can be divided into different functional units or modules to complete all or part of the functions described above. The functional units and modules in the embodiment can be integrated in a processing unit, or each unit can exist physically separately, or two or more units can be integrated in one unit. The above-mentioned integrated unit can be implemented in the form of hardware or in the form of software functional units. In addition, the specific names of the functional units and modules are only for the convenience of distinguishing each other, and are not used to limit the scope of protection of this application. The specific working process of the units and modules in the above-mentioned system can refer to the corresponding process in the aforementioned method embodiment, which will not be repeated here.

[0132] In the above embodiments, the description of each embodiment has its own emphasis. For parts that are not described or recorded in detail in a certain embodiment, reference can be made to the relevant descriptions of other embodiments.

[0133] Those of ordinary skill in the art will appreciate that the units and algorithm steps of each example described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are performed in hardware or software depends on the specific application and design constraints of the technical solution. Professional and technical personnel can use different methods to implement the described functions for each specific application, but such implementation should not be considered to be beyond the scope of this application.

[0134] In the embodiments provided in the present application, it should be understood that the disclosed devices / equipment and methods can be implemented in other ways. For example, the device / equipment embodiments described above are merely schematic, for example, the division of the modules or units is only a logical function division, and there may be other division methods in actual implementation, such as multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some interfaces, indirect coupling or communication connection of devices or units, which can be electrical, mechanical or other forms.

[0135] The units described as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed on multiple network units. Some or all of the units may be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0136] In addition, each functional unit in each embodiment of the present application may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit. The above-mentioned integrated unit may be implemented in the form of hardware or in the form of software functional units.

[0137] If the integrated module / unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the present application implements all or part of the processes in the above-mentioned embodiment method, and can also be completed by instructing the relevant hardware through a computer program. The computer program can be stored in a computer-readable storage medium, and the computer program can implement the steps of the above-mentioned various method embodiments when executed by the processor. Among them, the computer program includes computer program code, and the computer program code can be in source code form, object code form, executable file or some intermediate form. The computer-readable medium may include: any entity or device capable of carrying the computer program code, recording medium, U disk, mobile hard disk, disk, optical disk, computer memory, read-only memory (ROM), random access memory (RAM), electric carrier signal, telecommunication signal and software distribution medium. It should be noted that the content contained in the computer-readable medium can be appropriately increased or decreased according to the requirements of legislation and patent practice in the jurisdiction. For example, in some jurisdictions, according to legislation and patent practice, computer-readable media do not include electric carrier signals and telecommunication signals.

[0138] The embodiments described above are only used to illustrate the technical solutions of the present application, rather than to limit them. Although the present application has been described in detail with reference to the aforementioned embodiments, a person skilled in the art should understand that the technical solutions described in the aforementioned embodiments may still be modified, or some of the technical features may be replaced by equivalents. Such modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the embodiments of the present application, and should all be included in the protection scope of the present application.

Claims

1. A method for controlling a robotic arm, characterized in that: The mechanical arm is provided with a motion actuator, and the control method comprises: Get the historical and current poses of the tracked object; Determining posture control data of the motion actuator according to a posture change between the historical posture and the current posture; The robot arm is controlled to follow the tracked object for synchronous movement according to the posture control data.

2. The control method of the robot arm according to claim 1, characterized in that: The obtaining of the historical position and current position of the tracked object includes: Acquire an object image of the tracked object, the object image comprising at least a first image reflecting the historical posture and a second image reflecting the current posture; Determine a marking point in the first image, wherein the marking point is used to identify the tracked object in the first image; Determine the pose of the object at the marked point in the first image to obtain the historical pose; The current posture is determined according to the historical posture and the second image.

3. The control method of the robot arm according to claim 2, characterized in that: The determining the position and posture of the object at the marked point in the first image includes: Acquire a three-dimensional model of the tracked object; The three-dimensional model of the tracked object is registered with the object at the marking point to obtain the position and posture of the object at the marking point.

4. The control method of the robot arm according to claim 2, characterized in that: The determining the position and posture of the object at the marked point in the first image to obtain the historical position and posture includes: Segmenting the object at the marked point in the first image to obtain a mask of the object at the marked point; According to the mask of the object on the marking point, the masks of the tracked object in multiple preset postures are matched respectively to determine the historical posture.

5. The control method of the robot arm according to claim 2, characterized in that: The step of determining the current posture according to the historical posture and the second image includes: The current posture is determined according to the historical posture, the three-dimensional model of the tracked object, and the second image.

6. The control method of a robot arm according to claim 1, characterized in that: Determining the posture control data of the motion actuator according to the posture change between the historical posture and the current posture includes: Aligning a first coordinate system where the historical posture and / or the current posture are located with a second coordinate system where the motion actuator is located to obtain an alignment result; According to the alignment result, the posture change between the current posture and the historical posture is converted to obtain the posture control data.

7. The control method of a robot arm according to any one of claims 1 to 6, characterized in that: The control method further includes performing admittance control on the movement of the robotic arm, including: Obtaining the actual force of the motion actuator when performing synchronous motion, and the damping coefficient of the motion surface; Admittance control is performed on the synchronous motion of the robot arm based on the actual force and the damping coefficient.

8. The control method of the robot arm according to claim 7, characterized in that: The obtaining of the actual force of the motion actuator when performing synchronous motion includes: Acquire the current posture of the force sensor and the original force of the motion actuator collected by the force sensor; Determining the mass of the motion actuator, the center of mass position of the motion actuator, the zero point of the force sensor, and the base mounting inclination angle of the robotic arm; Based on the current posture of the force sensor, the mass of the motion actuator, the center of mass position of the motion actuator, the zero point of the force sensor and the base installation inclination, the original force is corrected to determine the actual force.

9. The control method of a robot arm according to claim 7, characterized in that: The admittance control comprises: Acquiring the motion state of the motion actuator; Determine the change in damping coefficient based on the change in the moving surface over time; Determining an error based on the actual force and the preset force; According to the relationship between the error, the damping coefficient change and the motion state, the motion state of the motion actuator is adjusted in real time.

10. An ultrasonic device, characterized in that: include: A control module, a mechanical arm, and an ultrasonic probe arranged on the mechanical arm; The control module comprises a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor implements the steps of the control method of the robot arm according to any one of claims 1 to 9 when executing the computer program; The mechanical arm is used to control the movement of the ultrasound probe.

Images