Control method of supporting robot, supporting robot and electronic equipment
By integrating a robotic arm and tactile sensors into the assistive robot and using a control module for compliant control, the problem of insufficient intelligence in existing assistive devices has been solved, achieving automated assistance and highly comfortable user interaction.
Patent Information
- Application Number
- CN202410163810.2
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-02-02
- Publication Date
- 2024-11-22
- Estimated Expiration
- 2044-02-02
AI Technical Summary
Existing assistive devices such as canes and simple walking aids have low levels of intelligence, cannot automatically perform assistance tasks, and lack compliant control when interacting with users, resulting in insufficient comfort.
Design a support robot equipped with a robotic arm and tactile sensors. The robot identifies the user's intention to provide support through a target sensor and uses a control module for compliant control, enabling the robotic arm to adjust its movements and provide tactile feedback to adapt to changes in the user's posture.
It enables automated assistance tasks, improves the comfort and intelligence of user interaction, and can adjust actions in real time according to the user's status, providing diverse interaction methods.
Smart Images

Figure CN118990469A_ABST
Abstract
Description
Technical Field
[0001] The present application relates to the field of robotics technology, and in particular to a control method for an assisting robot, an assisting robot and electronic equipment. Background Art
[0002] In daily life, in order to enhance balance, provide support and improve stability when standing or walking, related auxiliary activity devices are widely used in various scenarios such as daily rehabilitation. At present, commonly used auxiliary activity devices are generally crutches or other simple walkers, and the intelligence level of such auxiliary activity devices needs to be improved. Summary of the invention
[0003] The following is a summary of the subject matter described in detail in this application. This summary is not intended to limit the scope of the claims.
[0004] The embodiments of the present application provide a control method for an assisting robot, an assisting robot and an electronic device, which can automatically perform assisting tasks and have a high degree of intelligence.
[0005] On the one hand, an embodiment of the present application provides a control method of an assisting robot, wherein the assisting robot is provided with a mechanical arm and a target sensor for object perception, and the mechanical arm is provided with a tactile sensor, and the control method comprises:
[0006] In response to identifying that the target object has an intention to require assistance, the assistance robot is controlled to move toward the target object, and when the assistance robot moves to a target moving position, the mechanical arm is controlled to move to a target joint position, wherein both the target moving position and the target joint position follow the relative position change between the target object and the target sensor;
[0007] During the process of the mechanical arm moving to the target joint position or after the mechanical arm moves to the target joint position, when the tactile sensor is exerted with external force, the supporting robot is compliantly controlled.
[0008] On the other hand, an embodiment of the present application provides an assisting robot, wherein the assisting robot is provided with a control module, a mechanical arm, and a target sensor for object perception, wherein the mechanical arm is provided with a tactile sensor:
[0009] The control module is used for controlling the assisting robot to move toward the target object in response to recognizing that the target object has an intention to assist, controlling the mechanical arm to move to a target joint position when the assisting robot moves to a target moving position, and performing compliant control on the assisting robot when an external force is applied to the tactile sensor during or after the mechanical arm moves to the target joint position;
[0010] The target moving position and the target joint position both follow the relative position change between the target object and the target sensor.
[0011] Furthermore, the control module is also used for:
[0012] Acquire current target posture data of the posture sensor, and transform the target posture data based on a preset first transformation matrix to obtain the target moving position;
[0013] The assisting robot is controlled to move toward the target object based on the target moving position.
[0014] Furthermore, the control module is also used for:
[0015] Acquire the first pose data and the first covariance matrix predicted by the pose sensor at the previous moment, predict the current second pose data based on the first pose data, and predict the current second covariance matrix based on the first covariance matrix;
[0016] Acquire the measured posture data currently collected by the posture sensor, and determine the target gain based on the measured posture data and the second covariance matrix;
[0017] The second posture data is corrected based on the target gain to obtain current target posture data of the posture sensor.
[0018] Furthermore, the control module is also used for:
[0019] Acquire current image data of the visual sensor, and determine key point positions of multiple key points of the target object in the image data;
[0020] Convert the coordinate system where the key point position is located to the coordinate system where the assisting robot is located, and determine the center point positions of the multiple key points based on the multiple converted key point positions;
[0021] The center point position is transformed based on a preset second transformation matrix to obtain the target joint position, and the robot arm movement is controlled based on the target joint position.
[0022] Furthermore, the control module is also used for:
[0023] Calibrate the parameters of the visual sensor to obtain an internal parameter matrix of the visual sensor;
[0024] Determine the installation position of the visual sensor in the assisting robot, and determine the external parameter matrix of the visual sensor according to the installation position;
[0025] Based on the intrinsic parameter matrix and the extrinsic parameter matrix, the coordinate system where the key point position is located is converted to the coordinate system where the target moving position is located.
[0026] Furthermore, the control module is also used for:
[0027] The target object is subjected to posture recognition according to each of the key points, wherein a result of the posture recognition includes whether the target object has an intention to require assistance or whether the target object has no intention to require assistance.
[0028] Furthermore, the control module is also used for:
[0029] Performing kinematic modeling on the supporting robot to obtain a dynamic model of the supporting robot, wherein the dynamic model is used to indicate the relationship between the joint torque of the supporting robot and the joint acceleration of the supporting robot;
[0030] Acquire current external force data of the tactile sensor, convert the external force data, and obtain a target joint torque;
[0031] The target joint acceleration of the supporting robot is determined according to the target joint torque and the dynamic model, and the supporting robot is compliantly controlled based on the target joint acceleration.
[0032] Furthermore, the control module is also used for:
[0033] Determining a mass coefficient according to the mass of the supporting robot, the joint positions of the supporting robot, and the joint speeds of the supporting robot;
[0034] Determining a friction coefficient according to the friction force when the supporting robot moves and the joint position of the supporting robot;
[0035] Determining a gravity coefficient according to the gravity acceleration of the supporting robot and the joint position of the supporting robot;
[0036] A first product between the mass coefficient and the joint acceleration of the supporting robot is determined, and kinematic modeling of the supporting robot is performed according to the first product, the friction coefficient and the sum of the gravity coefficient to obtain a dynamic model of the supporting robot.
[0037] Furthermore, the control module is also used for:
[0038] Determine a third transformation matrix for converting the coordinate system of the assisting robot to the coordinate system of the joints of the assisting robot, and determine a fourth transformation matrix for converting the coordinate system of the joints of the assisting robot to the coordinate system of the tactile unit;
[0039] Determine, according to the third transformation matrix and the fourth transformation matrix, a conversion function between a coordinate system where each of the tactile units is located and a coordinate system where the assisting robot is located, determine the differential of the conversion function with respect to a joint position of the assisting robot, and obtain a Jacobian matrix corresponding to each of the tactile units;
[0040] A second product between the external force data collected by each of the tactile units and the corresponding Jacobian matrix is determined, and a target joint torque is obtained based on a sum of a plurality of the second products.
[0041] Furthermore, the plurality of tactile units are distributed on the robotic arm in a cylindrical shape, and the first distance between any two adjacent tactile units is equal, and the control module is further used for:
[0042] Based on the first distance corresponding to the tactile unit, a fourth transformation matrix for converting a coordinate system where the joints of the assisting robot are located to a coordinate system where the tactile unit is located is determined.
[0043] Furthermore, the control module is also used for:
[0044] In the process of the assisting robot following the target object, when an obstacle is identified within a preset angle range, detecting a target size of the obstacle;
[0045] When the target size is outside a preset size range, the assisting robot is controlled to avoid the obstacle.
[0046] Furthermore, the control module is also used for:
[0047] Detecting a second distance between the assisting robot and the obstacle, and when the second distance is less than or equal to a preset distance threshold, constructing a virtual space where the assisting robot is located;
[0048] Determining the obstacle position of the obstacle and the set end position of the assisting robot in the virtual space;
[0049] Determine a first force vector of the obstacle according to the obstacle position and the obstacle mass of the obstacle, and determine a second force vector of the end point position according to the end point position and the preset mass of the end point position;
[0050] The first force vector and the second force vector are synthesized to obtain a target force vector, and the assisting robot is controlled to avoid the obstacle according to the target force vector.
[0051] Furthermore, the tactile sensor is provided with a plurality of tactile units, and the control module is further used for:
[0052] When an external force is applied to the tactile sensor, obtaining current external force data of the tactile sensor and determining the activation number of the tactile unit;
[0053] When the external force data indicates that the external force applied to the tactile sensor is greater than or equal to an external force threshold, and the activation quantity is greater than or equal to a preset quantity threshold, the assisting robot is compliantly controlled.
[0054] On the other hand, an embodiment of the present application further provides an electronic device, including a memory and a processor, wherein the memory stores a computer program, and the processor implements the above-mentioned control method of the supporting robot when executing the computer program.
[0055] On the other hand, an embodiment of the present application further provides a computer-readable storage medium, wherein the storage medium stores a computer program, and the computer program is executed by a processor to implement the above-mentioned control method of the supporting robot.
[0056] On the other hand, the embodiment of the present application further provides a computer program product, which includes a computer program, and the computer program is stored in a computer-readable storage medium. The processor of the computer device reads the computer program from the computer-readable storage medium, and the processor executes the computer program, so that the computer device executes the control method for implementing the above-mentioned assisting robot.
[0057] The embodiments of the present application include at least the following beneficial effects: in response to identifying that the target object has the intention of requiring assistance, the assistance robot is controlled to move toward the target object; when the assistance robot moves to the target moving position, the robotic arm is controlled to move to the target joint position, wherein both the target moving position and the target joint position follow the relative position change between the target object and the target sensor, thereby being able to sense the state of the target object and automatically perform the assistance task according to the state of the target object; on this basis, during the process of the robotic arm moving to the target joint position or after the robotic arm moves to the target joint position, when an external force is applied to the tactile sensor, the robotic arm is compliantly controlled so that the robotic arm can follow the posture of the target object, thereby improving the comfort of assistance; it can be seen that the control method provided by the present application is capable of performing diversified interactions with the target object and has a high degree of intelligence.
[0058] Other features and advantages of the present application will be set forth in the following description, and in part will become apparent from the description, or may be understood by practicing the present application. BRIEF DESCRIPTION OF THE DRAWINGS
[0059] The accompanying drawings are used to provide further understanding of the technical solution of the present application and constitute a part of the specification. Together with the embodiments of the present application, they are used to explain the technical solution of the present application and do not constitute a limitation on the technical solution of the present application.
[0060] Figure 1 A schematic diagram of an optional implementation environment provided for an embodiment of the present application;
[0061] Figure 2 is a flow chart of a control method of an assisting robot provided in an embodiment of the present application;
[0062] Figure 3 A schematic diagram of a target object to be supported provided in an embodiment of the present application;
[0063] Figure 4 A schematic diagram of identifying a target object's intention to require assistance provided by an embodiment of the present application;
[0064] Figure 5 A schematic diagram of identifying a target object's intention to require assistance provided by an embodiment of the present application;
[0065] Figure 6 A schematic diagram of identifying a target object's intention to require assistance provided by an embodiment of the present application;
[0066] Figure 7 A schematic diagram of identifying a target object's intention to require assistance provided by an embodiment of the present application;
[0067] Figure 8A schematic diagram of the position transformation relationship provided in the embodiment of the present application;
[0068] Fig. 9 A schematic diagram of a process for processing posture data measured by a posture sensor provided in an embodiment of the present application;
[0069] Fig.10 A schematic diagram of the center point position provided in an embodiment of the present application;
[0070] Fig.11 A schematic diagram of converting the coordinate system where the joints of the assisting robot provided in an embodiment of the present application are located to the coordinate system where the tactile unit is located;
[0071] Fig.12 A schematic diagram of a flow chart of compliance control provided in an embodiment of the present application;
[0072] Fig.13 A schematic diagram of an obstacle avoidance assisting robot provided in an embodiment of the present application;
[0073] Fig.14 An optional overall flow chart of the control method provided in the embodiment of the present application;
[0074] Fig.15 An optional overall flow chart of the control method provided in the embodiment of the present application;
[0075] Fig.16 An optional overall flow chart of the control method provided in the embodiment of the present application;
[0076] Fig.17 A schematic diagram of an optional structure of the assisting robot provided in an embodiment of the present application;
[0077] Fig.18 A schematic diagram of an optional structure of the assisting robot provided in an embodiment of the present application;
[0078] Fig.19 A schematic diagram of the structure of the assisting robot provided in an embodiment of the present application from another perspective;
[0079] Fig. 20 A schematic diagram of the internal structure of a control cabinet of the assisting robot provided in an embodiment of the present application;
[0080] Fig.21 A schematic diagram of the structural connection of the assistance robot provided in an embodiment of the present application. DETAILED DESCRIPTION
[0081] In order to make the purpose, technical solution and advantages of the present application more clearly understood, the present application is further described in detail below in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present application and are not used to limit the present application.
[0082] It should be noted that in each specific embodiment of the present application, when it comes to the need to perform relevant processing based on data related to the characteristics of the target object such as target object attribute information or attribute information set, the permission or consent of the target object will be obtained first, and the collection, use and processing of these data will comply with relevant laws, regulations and standards. Among them, the target object can be a user. In addition, when the embodiment of the present application needs to obtain the target object attribute information, the target object's separate permission or separate consent will be obtained by means of a pop-up window or jumping to a confirmation page, etc., and after the separate permission or separate consent of the target object is clearly obtained, the necessary target object-related data for enabling the normal operation of the embodiment of the present application will be obtained.
[0083] In the embodiments of the present application, the term "module" or "unit" refers to a computer program or a part of a computer program with a predetermined function, and works together with other related parts to achieve a predetermined goal, and can be implemented in whole or in part by using software, hardware (such as processing circuits or memories) or a combination thereof. Similarly, a processor (or multiple processors or memories) can be used to implement one or more modules or units. In addition, each module or unit can be part of an overall module or unit that includes the function of the module or unit.
[0084] To facilitate understanding of the technical solutions provided in the embodiments of the present application, some key terms used in the embodiments of the present application are explained here:
[0085] Computer Vision (CV) is a science that studies how to make machines "see". To put it more specifically, it refers to machine vision that uses cameras and computers to replace human eyes to identify and measure targets, and further performs graphic processing to make computer processing into images that are more suitable for human eye observation or transmission to instruments for detection. As a scientific discipline, computer vision studies related theories and technologies, and attempts to establish an artificial intelligence system that can obtain information from images or multi-dimensional data. Computer vision technology usually includes image processing, image recognition, image semantic understanding, image retrieval, OCR, video processing, video semantic understanding, video content / behavior recognition, three-dimensional object reconstruction, 3D technology, virtual reality, augmented reality, simultaneous positioning and map construction, and other technologies, as well as common biometric recognition technologies such as face recognition and fingerprint recognition.
[0086] Artificial Intelligence (AI) is the theory, method, technology and application system that uses digital computers or machines controlled by digital computers to simulate, extend and expand human intelligence, perceive the environment, acquire knowledge and use knowledge to obtain the best results. In other words, artificial intelligence is a comprehensive technology in computer science that attempts to understand the essence of intelligence and produce a new intelligent machine that can respond in a similar way to human intelligence. Artificial intelligence is to study the design principles and implementation methods of various intelligent machines so that machines have the functions of perception, reasoning and decision-making. Artificial intelligence technology is a comprehensive discipline that covers a wide range of fields, including both hardware-level technology and software-level technology. The basic technologies of artificial intelligence generally include sensors, dedicated artificial intelligence chips, cloud computing, distributed storage, big data processing technology, pre-trained model technology, operation / interaction system, mechatronics, etc. Among them, the pre-trained model is also called the large model or basic model. After fine-tuning, it can be widely used in downstream tasks in various major directions of artificial intelligence. Artificial intelligence software technology mainly includes computer vision technology, speech processing technology, natural language processing technology, and machine learning / deep learning.
[0087] Posture describes the position and posture of an object (such as coordinates) in a specified coordinate system. For example, posture is often used to describe the position and posture of a robot in a spatial coordinate system. Among them, position refers to the positioning of a rigid body in space. The position of a rigid body can be represented by a 3×1 matrix, that is, the position of the center of the rigid body coordinate system in the base coordinate system; and posture refers to the orientation of a rigid body in space. The posture of a rigid body can be represented by a 3×3 matrix, that is, the posture of the rigid body coordinate system in the base coordinate system.
[0088] In daily life, in order to enhance balance, provide support and improve stability when standing or walking, related auxiliary activity devices are widely used in various scenarios such as daily rehabilitation. At present, the commonly used auxiliary activity devices are generally crutches or other simple walkers. The intelligence level of such auxiliary activity devices needs to be improved.
[0089] In order to solve the above problems, the embodiments of the present application provide a control method for an assisting robot, an assisting robot and an electronic device, which can automatically perform assisting tasks and have a high degree of intelligence.
[0090] Reference Figure 1 , Figure 1 A schematic diagram of an optional implementation environment provided for an embodiment of the present application, the implementation environment includes a control module of an assisting robot, the control module can communicate with a sensor provided on the assisting robot, and after obtaining sensor data from the sensor, the control module can control the operation of the assisting robot based on the obtained sensor data.
[0091] Exemplarily, if the control module identifies through the obtained sensor data that the target object has the intention of requiring assistance, the control module can control the assistance robot to move toward the target object based on the obtained sensor data. When the assistance robot is controlled to move to the target moving position, the robotic arm can be controlled to move to the target joint position, wherein both the target moving position and the target joint position follow the relative position change between the target object and the target sensor, thereby being able to sense the state of the target object and automatically perform the assistance task according to the state of the target object. On this basis, during the process of the robotic arm moving to the target joint position or after the robotic arm moves to the target joint position, when the control module senses that external force is applied to the tactile sensor, it can obtain the interaction force data detected by the tactile sensor, and based on the interaction force data, perform compliant control of the assistance robot, so that the assistance robot can follow the posture of the target object, thereby improving the comfort of assistance. It can be seen that the control method provided in the present application can interact with the target object in a variety of ways and has a high degree of intelligence.
[0092] In addition, the control module can communicate with the server, and the server updates the algorithm package corresponding to the control method to the control module in real time.
[0093] Figure 2 This is a flow chart of a control method for an assisting robot provided in an embodiment of the present application. The control method for the assisting robot may be executed by a control module of the assisting robot, or may be executed by the control module of the assisting robot and a server in cooperation. In an embodiment of the present application, the control method for the assisting robot is executed by the control module of the assisting robot as an example for explanation. The control method for the assisting robot includes but is not limited to the following steps 201 to 202.
[0094] Step 201, in response to identifying that the target object has the intention of requiring assistance, controlling the assistance robot to move toward the target object, and when the assistance robot moves to the target moving position, controlling the robot arm to move to the target joint position.
[0095] In one possible implementation, the target object may refer to a user object that can be sensed by the support robot and has the intention to be supported, and the support intention may refer to the purpose for which the target object expects to be assisted by the support robot, such as an object within the computer vision range of the support robot, or an object associated and bound to the terminal through biometric information. The support robot can sense and identify the target object through the installed target sensor, and evaluate whether the sensed target object needs support. For the support robot, the support intention of the target user can be expressed in a variety of forms, for example, the support intention can be expressed through voice language, body posture, physical interaction, signal instructions, etc. The specific method can be based on the perception method provided by the target sensor of the support robot.
[0096] In a possible implementation, the target moving position may refer to the location that the assisting robot needs to reach to complete the task of assisting the target object, such as the location close to the target object that needs to be assisted, or the target moving position may refer to the position of the assisting robot relative to the target object; and the target joint position may refer to the specific posture or joint angle that the mechanical arm of the assisting robot needs to achieve to complete the task of assisting the target object, or the target joint position may refer to the position of the mechanical arm relative to the target object. Among them, both the target moving position and the target joint position change with the relative position between the target sensor and the target object, which is equivalent to that, in the process of the assisting robot moving toward the target object, the state of the target object can be sensed in real time by the target sensor, and the relative position between the target sensor and the target object can be corrected, thereby adjusting the target moving position and the target joint position in real time, and then being able to automatically adjust the assisting task to better fit the current state of the target object and improve the comfort of the assisting.
[0097] In one possible implementation, during the process of the supporting robot performing the supporting task of the target object, since the state of the target object changes in real time, the target moving position and the target joint position can also be adjusted accordingly according to the different supporting intentions of the target object. For example, the two different supporting needs of supporting the target object to walk and supporting the target object to stand require different supporting postures (i.e., target moving position and target joint position) of the supporting robot. By automatically sensing the state of the target object and adjusting the target moving position and target joint position of the supporting robot in real time, different interaction methods, movement characteristics and assistance methods can be provided, and diversified interactions can be carried out with the target object to improve the degree of intelligence.
[0098] In one possible implementation, referring to Figure 3 , Figure 3A schematic diagram of a target object for assistance provided by an embodiment of the present application. When the assisting robot needs to assist the target object in walking, the target moving position of the assisting robot can be located at the side and rear of the target object, and the state of the target object can be sensed in real time during the assisting process. The target moving position is updated synchronously with the movement of the target object, so that the assisting robot keeps moving toward the target object, and the relative position of the target sensor and the target object is kept stable, so as to achieve the effect that the assisting robot follows the target object for assistance. In addition, the target joint position can also be adjusted in real time so that the assisting point of the robot arm is maintained at the waist, hip and wrist of the target object, imitating the human assisting posture to guide the walking direction of the target object and maintain the walking rhythm, thereby improving the comfort of assistance.
[0099] In a possible implementation, when the assisting robot needs to assist the target object to stand, the target moving position of the assisting robot can be located in front of the target object, and the target moving position is fixed relative to the target object to help the target object maintain balance, that is, the assisting robot moves toward the target object and can remain stable after moving to the target moving position. During the assisting process, the state of the target object is sensed in real time, and the target joint position is adjusted so that the assisting point of the robot arm is maintained at the upper body of the target object, such as the waist and arms, to support the weight of the target object, so that the target object changes from a sitting position or a squatting position to a standing position.
[0100] In one possible implementation, the target object's intention to require assistance can be determined by sensing the state of the target object, that is, the relative position of the target sensor and the target object. The relative position of the target object and the target sensor may include the relative positions of the target object's center of gravity, joint positions, torso positions, etc., and the target sensor. For example, if the target object's center of gravity is at a lower position relative to the target sensor and does not move significantly in consecutive frames, it can be considered that the target object is trying to stand up, and thus it can be determined that the target object has the intention to require assistance to stand; if the target object's center of gravity position changes continuously relative to the target sensor in consecutive frames, and the leg joint positions have a coherent displacement relative to the target sensor in a certain direction, it can be considered that the object is walking, and thus it can be determined that the target object has the intention to require assistance to walk.
[0101] Step 202, during the process of the robot arm moving to the target joint position or after the robot arm moves to the target joint position, when an external force is applied to the tactile sensor, the supporting robot is compliantly controlled.
[0102] In a possible implementation, when the tactile sensor senses that an external force is applied to the target object during or after the mechanical arm moves to the target joint position, it can be considered that the target object is in contact with the supporting robot. Therefore, by compliantly controlling the supporting robot, the compliant control can make the supporting robot respond appropriately according to the movement posture of the target object. For example, the supporting robot can be controlled to maintain a fixed output force without being affected by external pressure, or the supporting robot can be controlled to change position and speed accordingly with the external force according to the preset impedance characteristics to maintain a soft interaction with the target object, or feedback can be generated on touch or contact while satisfying a specific posture. Specifically, the compliant control of the supporting robot can be to adjust the parameters of the whole body joints (including upper limbs and lower limbs) of the supporting robot, such as by adjusting the target joint position, target moving position, movement speed and acceleration of each joint, etc., to reduce the occurrence of collision or excessive pressure on the target object during the supporting process, and improve the comfort of the support.
[0103] In a possible implementation, during the assisting process, the mechanical arm of the assisting robot can be compliantly controlled. Specifically, the torque, position, speed, acceleration and other parameters of each joint of the mechanical arm can be adjusted to change the motion trajectory and position of the mechanical arm of the assisting robot. For example, when the tactile sensor of the mechanical arm senses that an external force is applied, the torque output of the corresponding joint can be adjusted in real time according to the position and direction of the applied external force to maintain the posture of the target object or interact with the target object softly, while increasing the dynamic impedance parameters of the corresponding joints to make the movement of the mechanical arm smoother and slower, conforming to the posture of the target object. In addition to compliant control of the mechanical arm, the entire body of the assisting robot can also be compliant. By adjusting the entire body joints of the assisting robot to conform to the posture of the target object, the assisting robot can better imitate the limb movement characteristics of humans and improve the comfort during the assisting process. For example, when the assisting robot is a wheeled robot with a robotic arm, when it senses that an external force is applied to the tactile sensor of the robotic arm, it can adjust the degrees of freedom of the omnidirectional wheels (such as the forward distance, backward distance and rotation angle of the omnidirectional wheels) and the moving acceleration of the omnidirectional wheels to cooperate with the smooth movement of the robotic arm; when the assisting robot is a footed robot with a robotic arm, when it senses that an external force is applied to the tactile sensor of the robotic arm, it can change the height between the waist and the feet by adjusting the joint controllers between the feet and the waist to cooperate with the smooth movement of the robotic arm.
[0104] In a possible implementation, the types of target sensors include multiple types, specifically visual sensors and posture sensors, wherein the posture sensor can detect the spatial position and direction of the target object, and can also detect the spatial position and direction of the assisting robot. When the posture sensor detects that the position of the assisting robot relative to the target object has changed, the target moving position of the assisting robot can be adjusted in real time to achieve or maintain the desired relative posture. Therefore, the target moving position can follow the relative position change between the target object and the posture sensor. The visual sensor can identify the target object and its posture. When the relative posture, i.e., the relative position, between the target object and the visual sensor changes, the action position of the mechanical arm of the assisting robot can be adjusted in real time to respond to the posture change of the target object. Therefore, the target joint position follows the relative position change between the target object and the visual sensor.
[0105] In a possible implementation, at least one of the visual sensor and the posture sensor is provided, and the measurement error of a single sensor can be compensated by integrating the data of multiple sensors, thereby improving the accuracy and reliability of object perception. For example, the posture sensor may include an inertial measurement unit, a laser radar, an ultrasonic sensor, etc., so that the relative position between the target object and the posture sensor can be determined based on the relative position between each posture sensor and the target object. Specifically, by collecting the sensor data of each posture sensor at the same time, filtering out noise from all sensor data, and then fusing and analyzing all sensor data through a pre-trained neural network model, the relative position between the target object and the posture sensor is determined. Accordingly, the visual sensor may include a depth camera, a stereo camera, an infrared camera, etc., and by integrating multiple visual sensors, the three-dimensional posture of the target object can be determined more accurately, and then the relative position between the visual sensor and the target object can be determined.
[0106] In a possible implementation, the assisting robot may be provided with a visual sensor, such as a camera, and the terminal may sense and identify people within the computer vision range through the visual sensor provided by the assisting robot, and identify the posture of the perceived target object. The visual sensor may be used to capture the image data of the current assisting robot's environment in real time, and the image data may be analyzed to determine whether the main body of the target object exists in the image data. Specifically, the image data may be subjected to human body recognition through a human body posture recognition algorithm, such as a Mediapipe recognition framework, to determine the position of human body skeleton points, so that the image data may be subjected to key point detection of the target object. After determining the key points of the target object, the target object may be subjected to posture recognition to determine the human body posture of the target object, and then it may be determined whether the target object has the intention of requiring assistance to perform the assisting task. Among them, human body posture recognition algorithms with different response rates and complexities may be selected based on the sampling frequency of the visual sensor. For example, the video stream frame rate of the visual sensor is 30 frames per second, so that a model with medium model complexity may be selected to recognize human body posture, so that the frequency of human body posture recognition by each video node is 28Hz, and real-time human body recognition is realized for the current image data of the visual sensor. When it is detected that the target object has no intention of requiring assistance, the assistance robot can continue to perform the current assistance task, or continue to perform posture recognition on the target object to determine whether there is an intention to require assistance.
[0107] Reference Figure 4 , Figure 4 The schematic diagram of identifying the target object's intention to need assistance provided in the embodiment of the present application, the terminal can be deployed with a pre-trained posture estimation network model, after obtaining the image data captured by the visual sensor, the image data can be feature extracted to obtain the first image data, and then the first image data can be input into the posture estimation network model to obtain the human body key points, and then the current posture of the target object can be determined based on the human body key points, and then the current posture of the target object can be matched with the object to be helped that has the intention to need assistance, if the current posture matches the object to be helped, then the terminal can believe that the target object has the intention to need assistance. Figure 4 As shown, the posture to be assisted may be a body posture of the user half squatting, sitting, walking or making a specific gesture. It should be noted that if there are multiple user objects in the image data, the posture of each user object can be recognized, and the user object whose current posture matches the posture to be assisted is determined as the target object.
[0108] In addition, it is also possible to obtain historical image data taken by the visual sensor at the previous moment, perform feature extraction on the historical image data at the previous moment, obtain second image data, and then splice the first image data and the second image data to obtain third image data. Then, the third image data is input into the posture estimation network model to obtain the target key points, and then the estimated posture of the target object is determined based on the target key points. By combining the motion changes in continuous frames, a more accurate object posture can be obtained. If the estimated posture matches the object to be assisted with the intention of requiring assistance, the terminal can assume that the target object has the intention of requiring assistance.
[0109] In a possible implementation, the assisting robot may also be provided with an acoustic sensor, such as a microphone. The terminal may sense the voice information emitted by the target object through the acoustic sensor provided by the assisting robot, and perform voice recognition on the voice information to determine whether the voice information contains the intention of requesting assistance. For example, referring to Figure 5 , Figure 5 A schematic diagram of identifying the target object's intention to require assistance provided in an embodiment of the present application. When the user says sentences with the semantics of requiring assistance, such as "Please help me walk to the room" or "I need assistance", it can be considered that the target object has the intention to require assistance, so as to control the assistance robot to perform the assistance task.
[0110] In one possible implementation, referring to Figure 6 , Figure 6 The schematic diagram of identifying the target object's intention to assist provided by the embodiment of the present application, the terminal can sense the interaction force applied by the outside world through the tactile sensor provided on the mechanical arm of the assisting robot. Figure 6 As shown, if the terminal senses that external force is applied to the tactile sensor without controlling the assisting robot to perform the assisting task, the object with the smallest relative distance to the target sensor can be taken as the target object, and it is considered that the target object has the intention of requiring assistance. Alternatively, the terminal can call the visual sensor (if the assisting robot is also equipped with a visual sensor such as a camera) to sense and identify the posture of each object within the computer vision range, and take the object that meets the posture to be assisted as the target object, and it is considered that the target object has the intention of requiring assistance.
[0111] In one possible implementation, referring to Figure 7 , Figure 7A schematic diagram for identifying whether a target object has the intention to require assistance is provided in an embodiment of the present application. A help button may be provided on the assisting robot, or a remote controller that can be separated from the assisting robot. The terminal can sense the signal generated when the help button or the remote controller is triggered, and then it can be considered that the target object that triggered the help button or the remote controller has the intention to require assistance. Alternatively, the terminal can call on a visual sensor (if the assisting robot is also provided with a visual sensor such as a camera) to sense and identify the posture of each object within the computer vision range, and take the object that meets the posture to be assisted as the target object, and consider the target object to have the intention to require assistance.
[0112] In a possible implementation, sensor data related to the object is captured by a target sensor, for example, image data captured by a visual sensor and posture data detected by a posture sensor are used for feature extraction and splicing to obtain multi-source sensor data, and the multi-source sensor data is imported into a pre-trained deep learning model such as a convolutional neural network model, or a real-time object detection algorithm such as a YOLO (You Only Look Once) target detection algorithm, an SSD (Single Shot MultiBo10 Detector) object detection algorithm, etc., so that the human objects in the sensor data (such as the screen image) can be quickly detected and their positions can be estimated. Then, a posture estimation model such as an OpenPose human posture estimation algorithm model or an AlphaPose posture estimation algorithm based on human key point detection can be used to analyze the posture of the human objects in the sensor data, analyze the behavior pattern and action intention of the human objects, and determine whether there is an intention to need assistance. When it is recognized that the target object has the intention to need assistance, the optimal path for the assistance robot to reach the target moving position can be calculated through a dynamic path planning algorithm, and the assistance robot can be controlled to move toward the target object to be assisted. Among them, in addition to being able to sense human objects, the target sensor can also sense non-human objects, i.e., obstacle objects, such as laser radars, visual sensors, etc., to sense the surrounding environment of the assisting robot. Thus, during the movement process, the target sensor can also be used to sense obstacles on the moving path to achieve obstacle avoidance. At the same time, during the movement process, the target sensor is used to sense the target object in real time, correct the relative distance between the target object and the target sensor, and adjust the target moving position of the assisting target object. After the assisting robot reaches the target moving position, the robot arm can be controlled to move to the human support point, i.e., the target joint position, while sensing the triggering of the tactile sensor on the robot arm and the relative distance between the target sensor and the target object, judging the state of the target object, and adjusting the position and strength of the robot arm in real time to adapt to the movement and balance state of the target object.
[0113] Specifically, the assisting robot and the control method thereof provided in the embodiments of the present application can be applied to various human-computer interaction scenarios such as medical rehabilitation scenarios, daily care scenarios, public assistance scenarios, rescue and disaster relief scenarios, etc., with a high degree of intelligence. For example, in a hospital or rehabilitation center, the assisting robot can be controlled to help patients or elderly people walk, stand, sit down and perform other posture changes. Since the assisting robot can sense the state of the target object, the assisting scheme can be adjusted in real time according to the state and needs of the target object, thereby improving the assisting comfort while assisting the object in rehabilitation, and reducing the workload of the physical rehabilitation therapist. For example, by controlling the assisting robot to detect the activity state of the elderly in real time, the elderly can be helped to perform various exercises indoors or outdoors (such as going up and down stairs, standing, walking) and provide timely assistance in emergency situations (such as helping to get up after falling down); for example, in public places such as stations or shopping malls, passengers or customers with limited mobility can be helped to reach their destination. Specifically, in the station scenario, passengers with limited mobility can be helped to move from the waiting station to the side of the car door, and passengers can be helped to board the door pedal area of the bus, or passengers can be helped to sit in the car; and in the shopping mall scenario, customers can be helped to climb the steps of the escalator.
[0114] In one possible implementation, in the process of controlling the assisting robot to move toward the target object, the current target posture data of the posture sensor can be obtained, and the target posture data can be transformed based on a preset first transformation matrix to obtain the target moving position; then, the assisting robot is controlled to move toward the target object based on the target moving position.
[0115] In a possible implementation, the posture sensor can detect the human posture data of the target object, that is, the target posture data can be the human posture data of the target object measured by the posture sensor at the current moment, or can be calculated based on the measured human posture data. The preset first transformation matrix can refer to the posture transformation matrix from the human body space coordinates to the supporting robot space coordinates, which is used to represent the posture transformation relationship between the target moving position of the supporting robot and the position of the target object, so as to obtain the expected position of the supporting robot relative to the target object through the spatial coordinate conversion of the target object, that is, the target moving position, and then control the supporting robot to move to the target moving position near the target object to perform the supporting task of the target object.
[0116] In a possible implementation, multiple first candidate transformation matrices may be pre-set to accommodate different postures of the target object or different support requirements. For example, different transformation matrices may be selected according to the motion mode and posture of the target object to determine the corresponding target moving position of the support robot, thereby ensuring the safety and comfort of the support. After determining the target moving position, in the process of controlling the support robot to move toward the target object based on the target moving position, the position and posture data of the target object may be detected in real time, and the target moving position of the support robot may be updated in real time based on the first transformation matrix. Alternatively, the position and posture data of the target object may be detected in real time, the motion mode and posture of the target object may be determined, and the first transformation matrix may be updated in real time to adjust the target moving position of the support robot.
[0117] Reference Figure 8 , Figure 8 The position transformation relationship diagram provided in the embodiment of the present application. The target position data of the target object at the current moment can be detected by the position sensor, wherein the state data of the target object, i.e., the position and direction, can be extracted from the target position data, such as the position coordinates (x h ,y h , 1), and the angle θ of the target object relative to the supporting robot h Assuming that the target object's assistance requirement is to assist walking, a first transformation matrix corresponding to the assist walking can be determined from a plurality of first candidate transformation matrices, so that based on the preset first transformation matrix [0.92, 0, 1] T The position of the target object is transformed to obtain the target moving position of the assisting robot. Therefore, the assisting robot can be controlled to move from the original position to the target object and to the target moving position, thereby performing the task of assisting the target object. Specifically, the target moving position (x base ,y base , 1) The calculation process is shown in the following formula:
[0118]
[0119] Therefore, it can be seen that the target moving position of the assisting robot is related to the position of the target object relative to the assisting robot. As the assisting robot continues to move, the target posture data detected by the posture sensor continues to change, and the determined target moving position also changes accordingly, that is, the target moving position of the assisting robot changes with the relative position between the target object and the posture sensor. It should be noted that the target moving position can be determined based on the spatial coordinate system of the assisting robot itself. Therefore, although the target moving position is constantly changing, the target moving position at each moment can be relatively fixed relative to the world coordinate system or the coordinate system where the target object is located.
[0120] In one possible implementation, the target posture data can be obtained by integrating the posture data measured by the posture sensor at multiple times. Specifically, the first posture data and the first covariance matrix predicted by the posture sensor at the previous moment can be obtained, and the current second posture data can be predicted based on the first posture data, and the current second covariance matrix can be predicted based on the first covariance matrix; then, the measured posture data currently collected by the posture sensor is obtained, and the target gain is determined based on the measured posture data and the second covariance matrix; then, the second posture data is corrected based on the target gain to obtain the current target posture data of the posture sensor.
[0121] In a possible implementation, due to the measurement error of the posture sensor, it is easy to confuse the target object (such as a person) with the non-target object (such as an obstacle), resulting in the inability to perform the assistance task. Therefore, it is necessary to correct the human posture data directly measured by the posture sensor to obtain highly accurate target posture data. Fig. 9 , Fig. 9 A schematic diagram of a process for processing the posture data measured by the posture sensor provided in an embodiment of the present application, such as Fig. 9 As shown, the posture data is corrected and processed. First, the posture data of the previous moment is used for prediction, including state prediction and estimation of the posture data, and then the covariance matrix corresponding to the predicted state quantity is predicted and estimated. Then, the predicted state value is corrected using the measurement value at the current moment, including calculating the target gain, and then correcting the predicted state value based on the target gain, and finally the covariance matrix of the previous prediction and estimation is updated. Human body posture data may include the angles, directions, three-dimensional coordinate positions of key points of the body, etc. Specifically, the human body posture data detected by the posture sensor can be understood as the motion state of the target object, including the position and speed of the target object, that is, The first pose data is the pose data predicted by the pose data measured by the pose sensor at the previous moment. Since the motion state of the target object is not affected by the external control input, the external control vector and the external control input matrix can be ignored based on the state equation of the correction system. Thus, the second pose data predicted at the current moment can be obtained by multiplying the first pose data predicted at the previous moment with the state transfer matrix, that is, the state of the target object at the current moment is predicted. Among them, the state transfer matrix is used to describe the relationship between the correction system from the previous moment to the current moment. Specifically, the state transfer matrix is:
[0122]
[0123] Wherein, ΔT represents the time interval for correcting the system, indicating the time difference between the previous moment and the current moment, and is used to correct the change of the state variable caused by the time interval.
[0124] According to the state equation of the corrected system after ignoring the influence of external control input, the calculation formula of the second posture data can be obtained, which can be shown as follows:
[0125]
[0126] in, Represents the second pose data predicted at the current moment, x k-1 Represents the first position data predicted at the previous moment, ω k It is expressed as the process noise of the modified system satisfying Gaussian distribution.
[0127] In one possible implementation, the first covariance matrix refers to a matrix that measures the uncertainty of the predicted first pose data when the first pose data is predicted by the pose sensor at the previous moment, and is used to represent the uncertainty of the state estimation at the previous moment in the correction system. Among them, the first covariance matrix is an expression of the error distribution of the first pose data, and the diagonal elements in the first covariance matrix represent the variance of each state variable of the first pose data (i.e., the elements of the first pose data), while the non-diagonal elements represent the covariance between different state variables, which are used to reflect the correlation between different state variables. Based on the correction system, the second covariance matrix corresponding to the second pose data at the current moment can be obtained by multiplying the state transfer matrix with the first covariance matrix. Specifically, the calculation formula of the second covariance matrix can be shown as follows:
[0128]
[0129] in, Represented as the second pose data The corresponding second covariance matrix, P k-1 Expressed as the first pose data x k-1 The corresponding first covariance matrix, Q, is represented by the process noise ω of the modified system k The corresponding noise covariance matrix.
[0130] In one possible implementation, the target gain is used to optimize the prediction accuracy by combining the predicted state (i.e., the second pose data) predicted by the pose sensor with the observed value (i.e., the measured pose data) measured by the pose sensor, thereby making a trade-off between the second pose prediction pose and the measured pose data, correcting the second pose prediction data, and obtaining the target pose data. Specifically, the feedback gain matrix is calculated by the second covariance matrix and the observation matrix preset in the correction system, and then the target gain is calculated by the measured pose data and the feedback gain matrix. The calculation formula of the feedback gain matrix can be shown as follows:
[0131]
[0132] Among them, K k It is represented as the feedback gain matrix, H is represented as the measurement matrix, and R is represented as the observation noise matrix. Specifically, the measurement matrix can be expressed as:
[0133]
[0134] After obtaining the feedback gain matrix, the measured posture data and the second posture data can be weighed by the feedback gain matrix to obtain the target gain. The specific calculation formula can be shown as follows:
[0135]
[0136] Where B represents the target gain, z k It is represented as measured posture data; it should be noted that the measured posture data can be obtained after low-pass correction based on the initial posture data directly measured by the posture sensor. Specifically, the correction calculation formula of the measured posture data can be shown as follows:
[0137]
[0138] Among them, α is the correction coefficient, and the correction coefficient is the low-pass correction strength. It is represented by the second initial posture data directly measured by the posture sensor at the current moment, z k-1 Represents the first initial posture data directly measured by the posture sensor at the previous moment The historical measured pose data is obtained by low-pass correction.
[0139] Then, the second posture data is corrected using the target gain to obtain the current target posture data of the posture sensor. The target posture data x k The specific correction formula can be shown as follows:
[0140]
[0141] After the second posture data of the predicted state is corrected by using the target gain, the second covariance matrix corresponding to the second posture data can be updated. The specific matrix update formula can be shown as follows:
[0142]
[0143] Among them, P k It is expressed as the updated second covariance matrix, which is obtained by modifying the feedback gain matrix K of the system. k The second covariance matrix predicted by the observation matrix H Update, so that the updated second covariance matrix P can be used k Predict the target pose data at the next moment.
[0144] In a possible implementation, the relationship between the predicted state and the observed quantity in the correction system can be expressed by an observation equation. Specifically, the observation equation can be shown as follows:
[0145] z k =Hx k +v k (9)
[0146] Among them, v k It is expressed as the observation noise in the modified system that satisfies the Gaussian distribution.
[0147] In one possible implementation, in the process of controlling the movement of the robotic arm to the target joint position, the current image data of the visual sensor can be obtained first to determine the key point positions of multiple key points of the target object in the image data; then, the coordinate system where the key point positions are located is converted to the coordinate system where the supporting robot is located, and based on the multiple converted key point positions, the center point positions of the multiple key points are determined; the center point position is transformed based on a preset second transformation matrix to obtain the target joint position, and the movement of the robotic arm is controlled based on the target joint position.
[0148] In a possible implementation, there may be multiple key points of the target object. Specifically, the key points of the target object may include significant feature points on the human body such as eyes, ears, shoulders, waist, elbows, wrists, hips, knees, ankles and other joints. When performing key point detection on the image data, the key points may be determined from multiple joints based on the main part of the target object displayed in the image data. For example, if the main part of the target object displayed in the image data is the upper limbs of the target object, joints such as shoulders and waist may be selected as key points; if the main part of the target object displayed in the image data is the lower limbs of the target object, joints such as hips and knees may be selected as key points. Alternatively, when performing key point detection on the image data, the key points may be determined from multiple joints based on the support needs of the target object. For example, if the support needs of the target object are support to stand up, joints such as shoulders, elbows, wrists may be selected as key points; if the support needs of the target object are support to walk, joints such as waist, hips, elbows may be selected as key points.
[0149] In a possible implementation, the key points of the target object may be fixed joints on the human body. When the multiple fixed joints of the target object cannot be detected from the image data, the target movement is re-determined based on the relative position between the posture sensor and the target object, and after the assisting robot is re-controlled to reach the new target movement position, the image data of the visual sensor is re-acquired for analysis. For example, the key points of the target object may be the four key points of the left shoulder, right shoulder, left hip and right hip on the human body. When the above four key points of the target object cannot be completely detected in the image data obtained by the visual sensor, it can be considered that the assisting robot cannot accurately perceive the target object at present, and perception errors are prone to occur. At the same time, it can also be considered that the relative position between the assisting robot and the target object is too close or too far, and it is difficult to ensure the safe execution of the assisting task. Therefore, the target movement position can be re-determined by the relative position between the posture sensor and the target object, and the assisting robot is controlled to move to the updated target movement position and then the current image data is captured by the visual sensor until the four key points of the left shoulder, right shoulder, left hip and right hip of the target object can be completely detected in the new image data.
[0150] In a possible implementation, the key point position may refer to the coordinate point of each key point in the image data in the pixel coordinate system, wherein the pixel coordinate system may be the upper left corner of the image as the origin of the pixel coordinate system, the horizontal right direction is the positive direction of the horizontal coordinate axis, and the vertical downward direction is the positive direction of the vertical coordinate axis. After the image data is subjected to image recognition to determine a plurality of key points, the key point position of each key point in the image data, i.e., the two-dimensional coordinate point in the pixel coordinate system, may be determined by image registration. Next, the coordinate system where the key point position is located, i.e., the pixel coordinate system, is converted to the coordinate system where the assisting robot is located. At this time, the converted key point position is expressed as the three-dimensional coordinate point of each key point in the coordinate system where the assisting robot is located. Then, the arithmetic mean of the multiple converted key point positions is calculated to obtain the center point position of the multiple key points, i.e., the coordinates of the center point of the multiple key points. Then, the center point position is transformed by a preset second transformation matrix to obtain the target joint position, wherein the second transformation matrix may refer to the transformation matrix from the center point position to the required supporting position of the robotic arm, that is, the center point position is used as a reference point for the expected supporting position of the supporting robot to determine the target joint position of the robotic arm.
[0151] In one possible implementation, referring to Fig.10 , Fig.10 The schematic diagram of the center point position provided in the embodiment of the present application. The image data of the current environment of the assisting robot is captured by the visual sensor, such as Fig.10 As shown, due to the limitation of the shooting angle of the visual sensor, if the image data taken by the visual sensor needs to capture the main body of the target object, the relative distance between the visual sensor and the target object needs to be greater than the minimum shooting distance of the visual sensor. For example, if the visual sensor is a depth camera and the minimum shooting distance of the depth camera is 50 cm, then when the relative distance between the depth camera and the target object reaches 50 cm, the main body of the target object will be displayed in the image data taken by the visual sensor. Otherwise, the relative position between the support robot and the target object needs to be adjusted and the image data needs to be retaken. If the main body of the target object is displayed in the image data, the key points of the target object can be detected for the image data. Fig.10 The marked point A shown in Fig.10 The marker points B shown in Fig.10 The right hip (as shown in the Fig.10 The four key points (marked point D) shown in the figure can determine the relative position between the target object and the visual sensor, and at the same time determine the distance between the target object and the visual sensor. Then, by calculating the center point position of the four key points (such as the left shoulder, right shoulder, left hip and right hip) Fig.10 The marked point E) shown in FIG. Fig.10 For the target object shown in , the center point position is located at the waist of the target object. Therefore, the waist of the target object can be used as the reference point of the desired supporting position of the mechanical arm of the supporting robot, that is, the reference point of the target joint position. By transforming the center point position through the second transformation matrix, the supporting positions on both sides of the waist can be obtained as the target joint positions (such as Fig.10 The marking point F) is shown in FIG.
[0152] In one possible implementation, the visual sensor is calibrated to obtain an intrinsic parameter matrix of the visual sensor. Next, the installation position of the visual sensor in the assisting robot is determined, and the extrinsic parameter matrix of the visual sensor is determined according to the installation position. Then, based on the intrinsic parameter matrix and the extrinsic parameter matrix, the coordinate system of the key point position is converted to the coordinate system of the target moving position.
[0153] In a possible implementation, the intrinsic parameter matrix of the visual sensor describes the internal optical characteristics of the visual sensor, including the focal length and the imaging center (optical center). The focal length of the visual sensor can be represented by the x-axis focal length and the y-axis focal length in pixels, respectively, and the optical center can be represented by the projection position on the image plane. For example, after calibrating the parameters of the visual sensor, the intrinsic parameter matrix of the visual sensor can be obtained, and the intrinsic parameter matrix can be expressed as:
[0154]
[0155] Among them, f x Expressed as 10-axis focal length, f y Expressed as the y-axis focal length, (c x , c y ) represents the optical center position.
[0156] In a possible implementation, the external parameter matrix of the visual sensor defines the spatial relationship between the coordinate system of the visual sensor and the coordinate system of the supporting robot (base). The external parameter matrix includes a rotation matrix and a translation matrix. The rotation matrix is usually a 3X3 matrix, which represents the rotation of the visual sensor in the coordinate system of the supporting robot (base), and the translation matrix is usually a 3X1 vector, which is used to represent the position of the visual sensor relative to the supporting robot (base). For example, according to the installation position of the visual sensor in the supporting robot, the external parameter matrix can be obtained, which can be specifically expressed as:
[0157]
[0158] Among them, R represents the rotation matrix and T represents the translation matrix.
[0159] In a possible implementation, the coordinate system of the key point position is converted to the coordinate system of the target moving position. The coordinate system of the key point position, i.e., the pixel coordinate system, can be first converted to the coordinate system of the visual sensor, and then converted to the coordinate system of the assisting robot (base) through the coordinate system of the visual sensor. The specific calculation process can be shown as follows:
[0160]
[0161] Among them, (u, v) represents the pixel coordinate system, that is, the coordinate system where the key point is located; (X c , Y c , Z c ) is the coordinate system where the vision sensor is located. The origin of the coordinate system where the vision sensor is located is the optical center of the vision sensor. The Z axis is parallel to the optical axis of the vision sensor. c That is the shooting direction of the visual sensor; (X B , Y B , Z B ) represents the coordinate system where the supporting robot (base) is located.
[0162] In a possible implementation, the kinematic modeling of the supporting robot is performed based on the joint variables (such as displacement, velocity, acceleration, and position) of the supporting robot, and the dynamic model of the supporting robot can be obtained, which is used to describe the relationship between the joint torque of the supporting robot and the joint acceleration of the supporting robot. Then, the external force data is obtained through the tactile sensor set on the mechanical arm of the supporting robot, and the external force information of the force point is converted to the coordinate system of the corresponding joint of the supporting robot through the transformation matrix, so as to obtain the torque required for the joint of the supporting robot to comply with the external force information, that is, the target joint torque, so that the supporting robot can generate enough force to resist the external force while ensuring the flexibility of the movement and the stability of the support. Then, the target joint acceleration of the supporting robot is determined according to the obtained target joint torque and the established dynamic model, and then the supporting robot is compliantly controlled based on the target joint acceleration.
[0163] In one possible implementation, the mass coefficient can be first determined according to the mass of the supporting robot, the joint position of the supporting robot, and the joint speed of the supporting robot; then the friction coefficient can be determined according to the friction force when the supporting robot moves and the joint position of the supporting robot; then, the gravity coefficient can be determined according to the gravitational acceleration of the supporting robot and the joint position of the supporting robot; the first product between the mass coefficient and the joint acceleration of the supporting robot is determined, and the kinematic modeling of the supporting robot is performed according to the first product, the friction coefficient and the sum of the gravity coefficient to obtain the dynamic model of the supporting robot.
[0164] In a possible implementation, the mass coefficient can represent the inertial properties of each joint of the supporting robot. The mass coefficient can be understood as the mass matrix in the robot dynamics model, which is used to describe the mass distribution of each joint of the supporting robot. Specifically, the mass coefficient of each joint in the supporting robot can be determined by the mass, joint position and joint speed of the corresponding joint of the supporting robot, wherein the joint position of the supporting robot can be determined by the position encoder of each joint, and the mass coefficient can change with the change of the joint position. The friction coefficient can be understood as the friction matrix in the dynamics model, which represents the influence of friction on each joint when the supporting robot moves. The friction direction and friction magnitude corresponding to the joints at different joint positions are different. Therefore, the friction coefficients corresponding to the joints at different joint positions are different. The gravity coefficient can be understood as the gravity matrix in the dynamics model, which is used to represent the influence of gravity on each joint of the supporting robot. The gravity directions of the joints at different joint positions are different, which affects the posture transformation of each joint. The first product is obtained by multiplying the mass coefficient with the joint acceleration of the corresponding joint, which can represent the inertial force caused by the joint acceleration, that is, the inertial force that each joint needs to resist when it moves, to ensure timely response when subjected to external forces. The kinematic modeling of the assisting robot is carried out by the sum of the first product, the friction coefficient and the gravity coefficient, and the specific formula of the obtained dynamic model can be shown as follows:
[0165]
[0166] Where M(q) represents the mass coefficient of the joint at joint position q, Expressed as the first product, The joint velocities of the joints expressed as joint positions, The friction coefficient of the joint expressed as the joint position, is represented as the joint velocity of the joint at the joint position, g(q) is represented as the gravity coefficient of the joint at the joint position q, and τ is represented as the joint torque of the joint at the joint position q.
[0167] Next, based on the dynamic model, the target joint acceleration generated by the external force can be calculated. The specific calculation formula of the target joint acceleration can be shown as follows:
[0168]
[0169] After obtaining the target joint acceleration, an adaptive control strategy can be generated through a proportional-integral-differential controller, namely a PID controller, to adjust the torque output of the corresponding joint in response to external force, thereby controlling the movement of the assisting robot based on the target joint acceleration.
[0170] In one possible implementation, the tactile sensor includes multiple tactile units. Therefore, the third transformation matrix for converting the coordinate system of the supporting robot to the coordinate system of the joints of the supporting robot can be determined first, and the fourth transformation matrix for converting the coordinate system of the joints of the supporting robot to the coordinate system of the tactile unit can be determined; then, based on the third transformation matrix and the fourth transformation matrix, the conversion function between the coordinate system of each tactile unit and the coordinate system of the supporting robot is determined, and the differential of the conversion function with respect to the joint position of the supporting robot is determined to obtain the Jacobian matrix corresponding to each tactile unit; then, the second product between the external force data collected by each tactile unit and the corresponding Jacobian matrix is determined, and the target joint torque is obtained based on the sum of multiple second products.
[0171] In a possible implementation, it is assumed that the assisting robot needs to perform an assisting task. At this time, the task space of the assisting task is nonlinearly related to the joint space, that is, the coordinate point of the target joint position required to perform the assisting task is located in the coordinate system where the assisting robot is located, while the coordinate point of the external force data received during the assisting task is located in the coordinate system where the tactile unit is located. It is impossible to directly use the external force data to adjust the target joint position to achieve compliant control. Therefore, it is necessary to obtain a conversion function determined by the third transformation matrix and the fourth transformation matrix. The conversion function can describe how to convert the coordinate system where each tactile unit is located to the coordinate system where the assisting robot is located. Then, the joint position of the assisting robot is differentiated by the conversion function, and the two different coordinate systems can be associated, and the Jacobian matrix of the corresponding joint can be obtained. The Jacobian matrix of the corresponding joint can be expressed as follows:
[0172]
[0173] Among them, f represents the transfer function, J(q) represents the Jacobian matrix of the joint at the joint position q, and the Jacobian matrix of the corresponding joint can be obtained by taking the partial derivative of the transfer function f with respect to the joint position q.
[0174] The differential of the transfer function to the joint position of the assisting robot is used as the Jacobian matrix of each tactile unit to represent the influence of the magnitude and direction of the external force on each tactile unit on the different joint positions. Specifically, it is assumed that the direction of the rotation axis of each joint i is represented by the unit vector z r i Indicates that the position of the origin of each joint coordinate system can be represented by the vector p i Represents that the Jacobian matrix J of joint i i (q) can be expressed as:
[0175]
[0176] Among them, (x b,y b ) represents the coordinate system where the supporting robot is located, q k represents the kth joint, 1<k≤n, and n represents the total number of joints of the supporting robot, that is, the sum of the number of joints of the robotic arm and the number of joints of the base.
[0177] Since each joint of the assisting robot rotates around the z-axis, the pose transformation matrix T of each joint i is i Can be:
[0178]
[0179] Next, the posture transformation matrix of the joint corresponding to each tactile unit is substituted into the Jacobian matrix of the corresponding joint to obtain the Jacobian matrix corresponding to each tactile unit. Then, the external force data collected by each tactile unit is multiplied by the corresponding Jacobian matrix to obtain the second product, and the Jacobian matrix of each tactile unit is used to convert the external force data received by the corresponding tactile unit to the overall joint, that is, the second product can represent the force transmitted by each tactile unit to the overall joint, and then by calculating the sum of each second product, the force applied by the outside world (i.e., the target object) when interacting with the robotic arm during the support process can be represented, thereby mapping to obtain the target joint torque, wherein the calculation formula of the target joint torque τ can be shown as follows:
[0180]
[0181] in, It is represented as the Jacobian matrix of the tactile unit j of the robot joint i, It is represented as the external force data of the tactile unit j of the robot joint i. Specifically, due to the external force f z Applied in the z-axis direction, the external force data It can be expressed in the following form:
[0182]
[0183] In one possible implementation, the joints of the supporting robot include multiple sub-arms and joint components of a robotic arm, and also include a base joint of the supporting robot. Therefore, the joint torque and joint acceleration of each sub-arm and joint component of the robotic arm can be respectively controlled to achieve compliant control; when the supporting robot is a wheeled robot, that is, the base joint of the supporting robot may include an omnidirectional wheel mounted on the base, wherein the joint torque of the omnidirectional wheel can be represented by the rotation angle of the omnidirectional wheel, and the joint acceleration of the omnidirectional wheel can be represented by the angular velocity, angular acceleration, and torque of the motor driving the omnidirectional wheel to rotate, thereby achieving compliant control by adjusting the rotation direction, rotation speed, and rotation acceleration of the omnidirectional wheel to adapt to the movement trend of the assisted object; when the supporting robot is a footed robot, the base joints of the supporting robot may include ankle joints, knee joints, hip joints, etc. Similarly, compliant control can be achieved by controlling the joint torque and joint acceleration of the ankle joints, knee joints, and hip joints, respectively.
[0184] In a possible implementation, to determine the fourth transformation matrix for converting the coordinate system where the joints of the assisting robot are located to the coordinate system where the tactile unit is located, it is necessary to perform kinematic modeling for each tactile unit of the tactile sensor according to its position on the mechanical arm, and establish the relationship between the tactile unit and the kinematic chain on the mechanical arm, wherein each joint of the mechanical arm can be approximately regarded as a cylinder, therefore, it can be considered that the multiple tactile units of the tactile sensor are cylindrically distributed on the surface of the mechanical arm, and the first distance between any two adjacent tactile units is equal, wherein the first distance is represented as the distance between the two adjacent tactile units along the length direction of the mechanical arm connecting rod. At the same time, the posture of the coordinate system where the tactile unit is located relative to the coordinate system where the assisting robot is located is the same as the posture of the coordinate system where the joint corresponding to the tactile unit is located relative to the coordinate system where the assisting robot is located, so that the spacing between each tactile unit and the starting end of the joint can be determined based on the first distance corresponding to the tactile unit, and then the fourth transformation matrix for converting the coordinate system where the joints of the assisting robot are located to the coordinate system where the tactile unit is located can be determined.
[0185] Reference Fig.11 , Fig.11 Schematic diagram of the conversion of the coordinate system of the joints of the assisting robot provided in the embodiment of the present application to the coordinate system of the tactile unit. Assuming that the first distance is 0.015 meters, Fig.11As shown, the robot arm can be regarded as a cylinder. Since the first distance between any two adjacent tactile units is equal, it is equivalent to that the multiple tactile units distributed in a cylindrical shape on the surface of the robot arm can be divided along the height of the cylinder to obtain multiple rings. The spacing between two adjacent rings is fixed at 0.015 meters, and the tactile units on the same ring can be modeled as the same unit. Therefore, based on the first distance of the tactile unit and the position of the tactile unit (the ring) on the robot arm (cylinder), the conversion relationship between the tactile unit and the joint can be obtained, and then the fourth transformation matrix can be obtained. The fourth change matrix can be specifically expressed as:
[0186]
[0187] like Fig.11 As shown, t i It can be understood as the origin of the coordinate system of the joint i where the tactile unit is located, and can be expressed as the origin of the coordinate system of the tactile unit j in the joint i, (0.015×j) can be expressed as the distance between the origin of the coordinate system of the tactile unit j and the origin of the coordinate system of the joint i. It can be expressed as a fourth transformation matrix used to transform the coordinate system where the joints of the assisting robot are located to the coordinate system where the tactile unit is located.
[0188] In one possible implementation, referring to Fig.12 , Fig.12 A flow chart of compliance control provided in an embodiment of the present application. The whole-body compliance control based on the tactile sensor can be realized through the following steps: first, kinematic modeling of each tactile unit of the tactile sensor is required for its position on the robotic arm, and then, dynamic modeling of the supporting robot is performed to construct a dynamic model; then, based on the modeling of the tactile unit and the dynamic model of the supporting robot, the Jacobian matrix can be calculated for each tactile unit, so that the external force data received by the tactile sensor can be mapped to the force conditions of each joint of the robotic arm and the supporting robot through Jacobian matrix conversion, and then the joint accelerations corresponding to the joints of the supporting robot can be obtained for compliance control.
[0189] In one possible implementation, based on sensing the target object's need for assistance, the supporting robot can be controlled to follow the target object to perform the assistance task. For example, if the target object's intention to assist in walking is sensed, the supporting robot can be controlled to update the target moving position in real time based on the relative distance between the posture sensor and the target object, and the supporting robot can be controlled to move toward the target object based on the updated target moving position, thereby enabling the supporting robot to follow the target object to perform the assistance task. In the process of the assisting robot following the target object, the obstacle can be detected by the target sensor. When the obstacle is identified within the preset angle range, the obstacle outline can be determined by extracting features from the data obtained by the target sensor and then calculating the target size of the obstacle. Specifically, the environmental data of the surrounding environment can be obtained by using a posture sensor (such as a laser radar), and the environmental data can be segmented into point clouds to identify key features such as obstacles and ground. Then, the segmented point cloud data is clustered to form a "cluster" of obstacles, so that the size of the obstacle can be further analyzed; or the depth image of the current environment can be obtained by a visual sensor (such as a depth camera), and the distance data of the obstacle can be determined by calculating the depth of field of the depth image. Then, the distance data is identified and the size of the obstacle is estimated by a pre-trained neural network model. More accurate environmental data can be obtained by combining data from multiple sensors. For example, the sensor data obtained by both the posture sensor and the visual sensor can be imported into a corrector or a particle corrector for multi-source data fusion analysis to determine the size of the obstacle. Then, the target size of the obstacle is compared with the preset size range. If the target size of the obstacle exceeds the preset size range, it means that the obstacle may interfere with the moving path of the assisting robot and affect the safety of the assisting. Therefore, it is necessary to control the assisting robot to avoid the obstacle.
[0190] In one possible implementation, the target sensors such as posture sensors and / or visual sensors provided by the supporting robot can detect the second distance between the supporting robot and the obstacle. When the second distance is less than or equal to a preset distance threshold, it can be considered that the obstacle may affect the moving path of the supporting robot and the target object. Therefore, the supporting robot can be controlled to avoid the obstacle. At the same time, the supporting robot's mechanical arm can be flexibly controlled to provide information feedback to the target object (such as applying external force, issuing sound prompts, etc.) to guide the target object to avoid the obstacle.
[0191] In one possible implementation, referring to Fig.13 , Fig.13A schematic diagram of obstacle avoidance for an assisting robot provided in an embodiment of the present application. In the case where it is believed that an obstacle affects the moving path of the assisting robot, a virtual space can be created, and the sensor data sensed by the target sensor of the assisting robot can be mapped into the virtual space, so that the obstacle position of the obstacle and the terminal position of the moving path of the assisting robot can be represented in the virtual space, wherein the obstacle and the key position can be modeled to form a sphere. Based on the obstacle position and the obstacle mass corresponding to the obstacle, the first force vector of the obstacle, i.e., the repulsive force of the obstacle, can be determined. The first force vector is used to guide the assisting robot to move in a direction away from the obstacle position, and then the second force vector of the terminal position is determined based on the terminal position and the corresponding preset mass. The second force vector is used to guide the assisting robot to move toward the terminal position, wherein the first force vector decreases as the second distance increases, and the second force vector decreases as the distance between the current position of the assisting robot and the terminal position increases. Then, the first force vector and the second force vector are synthesized to obtain a target force vector. Specifically, the two vectors can be added or weighted added to obtain the target force vector. The speed and direction of the assisting robot can be controlled according to the target force vector, so that the assisting robot can avoid obstacles while moving towards the set end position.
[0192] It should be noted that when there are multiple obstacles and the second distance between them and the assisting robot is less than or equal to the preset distance threshold, multiple obstacles and the corresponding obstacle positions can be determined in the virtual space, and the first force vector corresponding to each obstacle can be obtained respectively. Then, all the first force vectors can be synthesized with the second force vector to obtain the target force vector.
[0193] In one possible implementation, during the process of assisting the target object, the target sensor can also be used to detect changes in the environment around the assisting robot in real time, so as to respond to emergencies such as environmental changes and ensure the safety of the user. For example, during the process of assisting the target object by the assisting robot across a zebra crossing, the assisting robot can perceive and identify dynamic obstacles such as vehicles and pedestrians in the surrounding environment, as well as static obstacles such as shoulders and flower beds, and guide the target object to adjust walking speed, pause walking, or change walking path to avoid potential collisions.
[0194] In a possible implementation, the tactile sensor includes a plurality of tactile units, and the plurality of tactile units can be evenly distributed on the surface of the robot arm. The tactile sensor can collect external force data in real time. Based on the detection principles of different tactile sensors, it can be determined whether the tactile sensor is triggered by resistance change, capacitance change, piezoelectric effect, etc. When an external force is applied to the tactile sensor, the electrical signal of the tactile unit will change. These signals can be converted into recognizable external force data. At the same time, it can be analyzed which tactile units are activated, thereby determining the number of activated tactile units. Specifically, it can be determined whether a tactile unit is in an activated state by setting a sensitivity or activation threshold. When the applied external force exceeds the sensitivity or activation threshold of the tactile unit, the tactile unit can be in an activated state. Among them, when external force is applied to the tactile sensor, external force data can be continuously obtained through the tactile sensor. When the detected external force data indicates that the current external force is greater than or equal to the external force threshold, and the activation quantity is greater than the preset quantity threshold, it can be considered that the current target object and the robotic arm are performing large-area force interaction behavior. Therefore, it can be considered that the target object is currently being supported by the support robot, or that the target object is currently experiencing an emergency such as unstable center of gravity. Therefore, the support robot can be compliantly controlled to adjust the target joint position and target joint acceleration of the robotic arm to adapt to the posture of the target object.
[0195] In one possible implementation, when the external force data indicates that the external force applied to the tactile sensor is greater than or equal to the external force threshold, and the activation quantity is less than the preset quantity threshold, it can be considered that the target object is interacting with the robotic arm, but the stability of the support is low at this time. If the current posture of the supporting robot is changed, it is easy for the target object to lose support balance. Therefore, the supporting robot can be controlled to move to the target moving position or the robotic arm can be controlled to move to the target joint position according to a predetermined control strategy, without the need for compliant control of the supporting robot.
[0196] In one possible implementation, when the external force data indicates that the external force change value received by the tactile sensor is greater than or equal to a first external force change threshold value within a first preset time period, and the internal and external force change value in a second preset time period after the first preset time period is less than a second external force change threshold value, it can be considered that the target object is currently performing force interaction with the robotic arm, and the current supporting posture of the supporting robot is stable with the supported posture of the target object, and the supporting robot can be compliantly controlled.
[0197] The control method of the assisting robot provided in the embodiment of the present application is described in detail below.
[0198] Reference Fig.14 , Fig.14An optional overall flow chart of a control method provided in an embodiment of the present application, wherein the control method can be executed by a terminal, and the control method includes but is not limited to the following steps 1401 to 1410:
[0199] Step 1401: In response to recognizing that the target object has an intention to require assistance, obtaining the first position data and the first covariance matrix obtained by predicting the position sensor at the previous moment.
[0200] Step 1402: predict the current second pose data based on the first pose data, and predict the current second covariance matrix based on the first covariance matrix.
[0201] Step 1403: Acquire the measured posture data currently collected by the posture sensor, and determine the target gain based on the measured posture data and the second covariance matrix.
[0202] Step 1404: Correct the second posture data based on the target gain to obtain the current target posture data of the posture sensor.
[0203] Step 1405: transform the target posture data based on a preset first transformation matrix to obtain the target moving position.
[0204] Step 1406: Control the assisting robot to move toward the target object based on the target moving position.
[0205] Step 1407: When the assisting robot moves to the target moving position, the current image data of the visual sensor is acquired to determine the key point positions of multiple key points of the target object in the image data.
[0206] Step 1408: Convert the coordinate system where the key point positions are located to the coordinate system where the assisting robot is located, and determine the center point positions of the multiple key points based on the multiple converted key point positions.
[0207] Step 1409: transform the center point position based on a preset second transformation matrix to obtain a target joint position, and control the robot arm movement based on the target joint position.
[0208] In this step, both the target moving position and the target joint position follow the relative position change between the target object and the target sensor;
[0209] Step 1410: During the process of the robotic arm moving to the target joint position or after the robotic arm moves to the target joint position, when an external force is applied to the tactile sensor, the supporting robot is compliantly controlled.
[0210] Reference Fig.15 , Fig.15An optional overall flow chart of a control method provided in an embodiment of the present application, wherein the control method can be executed by a terminal, and the control method includes but is not limited to the following steps 1501 to 1512:
[0211] Step 1501: In response to identifying that the target object has the intention of requiring assistance, controlling the assistance robot to move toward the target object.
[0212] Step 1502: When the assisting robot moves to the target moving position, the robotic arm is controlled to move to the target joint position.
[0213] In this step, both the target moving position and the target joint position follow the relative position change between the target object and the target sensor.
[0214] Step 1503: During or after the robotic arm moves to the target joint position, when an external force is applied to the tactile sensor, the mass coefficient is determined according to the mass of the supporting robot, the joint position of the supporting robot, and the joint speed of the supporting robot.
[0215] Step 1504: Determine the friction coefficient according to the friction force when the supporting robot moves and the joint position of the supporting robot.
[0216] Step 1505: Determine the gravity coefficient according to the gravity acceleration of the supporting robot and the joint position of the supporting robot.
[0217] Step 1506: Determine the first product between the mass coefficient and the joint acceleration of the supporting robot, perform kinematic modeling on the supporting robot according to the first product, the friction coefficient and the sum of the gravity coefficient, and obtain a dynamic model of the supporting robot.
[0218] In this step, the dynamic model is used to indicate the relationship between the joint torque of the supporting robot and the joint acceleration of the supporting robot.
[0219] Step 1507: Obtain current external force data of the tactile sensor.
[0220] Step 1508: Determine a third transformation matrix for converting the coordinate system of the supporting robot to the coordinate system of the joints of the supporting robot.
[0221] Step 1509: Based on the first distance corresponding to the tactile unit, determine a fourth transformation matrix for converting the coordinate system where the joints of the assisting robot are located to the coordinate system where the tactile unit is located.
[0222] In this step, the plurality of tactile units are distributed in a cylindrical shape on the robot arm, and the first distance between any two adjacent tactile units is equal.
[0223] Step 1510: Based on the third transformation matrix and the fourth transformation matrix, determine the conversion function between the coordinate system of each tactile unit and the coordinate system of the supporting robot, determine the differential of the conversion function with respect to the joint position of the supporting robot, and obtain the Jacobian matrix corresponding to each tactile unit.
[0224] Step 1511: Determine the second product between the external force data collected by each tactile unit and the corresponding Jacobian matrix, and obtain the target joint torque based on the sum of multiple second products.
[0225] Step 1512: Determine the target joint acceleration of the supporting robot according to the target joint torque and the dynamic model, and perform compliant control on the supporting robot based on the target joint acceleration.
[0226] Reference Fig.16 , Fig.16 An optional overall flow chart of a control method provided in an embodiment of the present application, wherein the control method can be executed by a terminal, and the control method includes but is not limited to the following steps 1601 to 1609:
[0227] Step 1601: In response to identifying that the target object has the intention of requiring assistance, controlling the assistance robot to move toward the target object.
[0228] Step 1602: When the assisting robot is following the target object and an obstacle is identified within a preset angle range, the target size of the obstacle is detected.
[0229] Step 1603: When the target size is outside the preset size range, detect the second distance between the assisting robot and the obstacle.
[0230] Step 1604: When the second distance is less than or equal to the preset distance threshold, construct a virtual space where the assisting robot is located.
[0231] Step 1605: Determine the obstacle position of the obstacle and the set end position of the assisting robot in the virtual space.
[0232] Step 1606: Determine a first force vector of the obstacle according to the obstacle position and the obstacle mass of the obstacle, and determine a second force vector of the end point position according to the end point position and the preset mass of the end point position.
[0233] Step 1607: synthesize the first force vector and the second force vector to obtain a target force vector, and control the assisting robot to avoid obstacles according to the target force vector.
[0234] Step 1608: When the assisting robot moves to the target moving position, the robotic arm is controlled to move to the target joint position.
[0235] In this step, both the target moving position and the target joint position follow the relative position change between the target object and the target sensor.
[0236] Step 1609: During the process of the robot arm moving to the target joint position or after the robot arm moves to the target joint position, when an external force is applied to the tactile sensor, the supporting robot is compliantly controlled.
[0237] It is to be understood that, although the steps in the above-mentioned various flow charts are shown in sequence according to the indication of the arrows, these steps are not necessarily performed in sequence according to the order indicated by the arrows. Unless there is a clear explanation in the present embodiment, the execution of these steps does not have a strict order restriction, and these steps can be performed in other orders. Moreover, at least a portion of the steps in the above-mentioned flow charts can include a plurality of steps or a plurality of stages, and these steps or stages are not necessarily performed at the same time, but can be performed at different times, and the execution order of these steps or stages is not necessarily performed in sequence, but can be performed in turn or alternately with at least a portion of the steps or stages in other steps or other steps.
[0238] Reference Fig.17 , Fig.17 This is an optional structural schematic diagram of the assisting robot provided in an embodiment of the present application. The assisting robot 1700 includes:
[0239] A mechanical arm 1701 is provided with a tactile sensor;
[0240] Target sensor 1702, for object perception;
[0241] The control module 1703 is used to control the assisting robot 1700 to move toward the target object in response to recognizing that the target object has the intention of requiring assistance, and when the assisting robot 1700 moves to the target moving position, control the robotic arm 1701 to move to the target joint position, and during the process of the robotic arm 1701 moving to the target joint position or after the robotic arm 1701 moves to the target joint position, when an external force is applied to the tactile sensor, perform compliant control on the assisting robot 1700; wherein, both the target moving position and the target joint position follow the relative position changes between the target object and the target sensor 1702.
[0242] In a possible implementation, the control module 1703 is further configured to:
[0243] Acquire current target posture data of the posture sensor, transform the target posture data based on a preset first transformation matrix, and obtain the target moving position;
[0244] The assisting robot 1700 is controlled to move toward the target object based on the target moving position.
[0245] In a possible implementation, the control module 1703 is further configured to:
[0246] Acquire the first pose data and the first covariance matrix obtained by predicting the pose sensor at the previous moment, predict the current second pose data based on the first pose data, and predict the current second covariance matrix based on the first covariance matrix;
[0247] Obtaining the measured posture data currently collected by the posture sensor, and determining the target gain based on the measured posture data and the second covariance matrix;
[0248] The second posture data is corrected based on the target gain to obtain the current target posture data of the posture sensor.
[0249] In a possible implementation, the control module 1703 is further configured to:
[0250] Acquire the current image data of the visual sensor and determine the key point positions of multiple key points of the target object in the image data;
[0251] Convert the coordinate system where the key point positions are located to the coordinate system where the assisting robot 1700 is located, and determine the center point positions of the multiple key points based on the multiple converted key point positions;
[0252] The center point position is transformed based on a preset second transformation matrix to obtain a target joint position, and the movement of the robotic arm 1701 is controlled based on the target joint position.
[0253] In a possible implementation, the control module 1703 is further configured to:
[0254] Calibrate the parameters of the visual sensor to obtain the internal parameter matrix of the visual sensor;
[0255] Determine the installation position of the visual sensor in the assisting robot 1700, and determine the external parameter matrix of the visual sensor according to the installation position;
[0256] Based on the intrinsic parameter matrix and the extrinsic parameter matrix, the coordinate system of the key point position is converted to the coordinate system of the target moving position.
[0257] In a possible implementation, the control module 1703 is further configured to:
[0258] The target object is subjected to posture recognition according to each key point, wherein a result of the posture recognition includes whether the target object has an intention to require assistance or whether the target object has no intention to require assistance.
[0259] In a possible implementation, the control module 1703 is further configured to:
[0260] Perform kinematic modeling on the supporting robot 1700 to obtain a dynamic model of the supporting robot 1700, wherein the dynamic model is used to indicate the relationship between the joint torque of the supporting robot 1700 and the joint acceleration of the supporting robot 1700;
[0261] Obtain the current external force data of the tactile sensor, convert the external force data, and obtain the target joint torque;
[0262] The target joint acceleration of the supporting robot 1700 is determined according to the target joint torque and the dynamics model, and the supporting robot 1700 is compliantly controlled based on the target joint acceleration.
[0263] In a possible implementation, the control module 1703 is further configured to:
[0264] Determine the mass coefficient according to the mass of the supporting robot 1700, the joint positions of the supporting robot 1700, and the joint speeds of the supporting robot 1700;
[0265] Determine the friction coefficient according to the friction force when the supporting robot 1700 moves and the joint position of the supporting robot 1700;
[0266] Determine the gravity coefficient according to the gravity acceleration of the supporting robot 1700 and the joint position of the supporting robot 1700;
[0267] The first product between the mass coefficient and the joint acceleration of the supporting robot 1700 is determined, and the kinematic model of the supporting robot 1700 is performed according to the first product, the friction coefficient and the sum of the gravity coefficient to obtain the dynamic model of the supporting robot 1700.
[0268] In a possible implementation, the control module 1703 is further configured to:
[0269] Determine a third transformation matrix for converting the coordinate system of the assisting robot 1700 to the coordinate system of the joints of the assisting robot 1700, and determine a fourth transformation matrix for converting the coordinate system of the joints of the assisting robot 1700 to the coordinate system of the tactile unit;
[0270] According to the third transformation matrix and the fourth transformation matrix, a conversion function between the coordinate system where each tactile unit is located and the coordinate system where the assisting robot 1700 is located is determined, and the differential of the conversion function with respect to the joint position of the assisting robot 1700 is determined to obtain the Jacobian matrix corresponding to each tactile unit;
[0271] The second product between the external force data collected by each tactile unit and the corresponding Jacobian matrix is determined, and the target joint torque is obtained based on the sum of multiple second products.
[0272] In a possible implementation, the plurality of tactile units are distributed on the mechanical arm 1701 in a cylindrical shape, and the first distance between any two adjacent tactile units is equal. The control module 1703 is further configured to:
[0273] Based on the first distance corresponding to the tactile unit, a fourth transformation matrix for converting a coordinate system where the joints of the assisting robot 1700 are located to a coordinate system where the tactile unit is located is determined.
[0274] In a possible implementation, the control module 1703 is further configured to:
[0275] When the assisting robot 1700 identifies an obstacle within a preset angle range during following the target object, it detects the target size of the obstacle;
[0276] When the target size is outside the preset size range, the assisting robot 1700 is controlled to avoid obstacles.
[0277] In a possible implementation, the control module 1703 is further configured to:
[0278] Detecting a second distance between the assisting robot 1700 and the obstacle, and when the second distance is less than or equal to a preset distance threshold, constructing a virtual space where the assisting robot 1700 is located;
[0279] Determine the obstacle position of the obstacle and the set end position of the assisting robot 1700 in the virtual space;
[0280] Determine a first force vector of the obstacle according to the obstacle position and the obstacle mass of the obstacle, and determine a second force vector of the end point position according to the end point position and the preset mass of the end point position;
[0281] The first force vector and the second force vector are synthesized to obtain a target force vector, and the assisting robot 1700 is controlled to avoid obstacles according to the target force vector.
[0282] In a possible implementation, the tactile sensor is provided with a plurality of tactile units, and the control module 1703 is further configured to:
[0283] When an external force is applied to the tactile sensor, current external force data of the tactile sensor is obtained, and the number of activated tactile units is determined;
[0284] When the external force data indicates that the external force applied to the tactile sensor is greater than or equal to the external force threshold, and the activation quantity is greater than or equal to the preset quantity threshold, the assisting robot 1700 is compliantly controlled.
[0285] In one possible implementation, referring to Fig.18 , Fig.18 This is an optional structural diagram of the assisting robot 1700 provided in the embodiment of the present application. Fig.18 As shown, the assisting robot 1700 includes a main trunk 1801, and the mechanical arm 1701 includes a first sub-arm 1802 and a second sub-arm 1803. One end of the first sub-arm 1802 is movably connected to the main trunk 1801, and the other end of the first sub-arm 1802 is movably connected to the second sub-arm 1803. Tactile sensors are provided on both the first sub-arm 1802 and the second sub-arm 1803, which means that the first sub-arm 1802 can move relative to the main trunk 1801, and the second sub-arm 1803 can move relative to the first sub-arm 1802. The first sub-arm 1802 can serve as the main part of the mechanical arm 1701, and can bear greater force than the second sub-arm 1803, so as to facilitate assisting and helping the user; and the second sub-arm 1803 acts as an extension of the first sub-arm 1802, and can perform more complex and delicate movements relative to the first sub-arm 1802, such as fine-tuning the assisting force or direction. In addition, the connecting rod length of the second sub-arm 1803 can be smaller than the connecting rod length of the first sub-arm 1802, thereby reducing the motion inertia of the second sub-arm 1803 to achieve more accurate motion control. Therefore, through the multi-level active connection between the first sub-arm 1802 and the second sub-arm 1803, a wider range of motion of the robot arm 1701 can be provided, better imitating the functions of human limbs, assisting the target object to move, and providing support and protection. It should be noted that the tactile sensors can be respectively arranged on the first sub-arm 1802 and the second sub-arm 1803, so that more comprehensive external force data can be obtained, and then the posture of the target object can be more accurately identified to perform compliant control on the assisting robot 1700.
[0286] In a possible implementation, the first joint component 1804 and the second joint component 1805 are connected at both ends of the first sub-arm 1802, the first joint component 1804 is connected to the third joint component 1806 via the first connecting axis, and the third joint component 1806 is connected to the main trunk 1801 via the second connecting axis; one end of the second sub-arm 1803 is connected to the second joint component 1805 via the third connecting axis, the other end of the second sub-arm 1803 is connected to the fourth joint component 1807 via the fourth connecting axis, the fourth joint component 1807 is connected to the fifth joint component 1808 via the fifth connecting axis, the fifth joint component 1808 is connected to the sixth joint component 1809 via the sixth connecting axis, and the sixth joint component 1809 is connected to the end effector 1810. Fig.18As shown, the mechanical arm 1701 of the assisting robot 1700 may include 6 joint components to achieve 6 degrees of freedom, wherein the third joint component 1806 may rotate relative to the main body 1801 around the second connection axis, and the first joint component 1804 may rotate relative to the third joint component 1806 around the first connection axis, so that the first sub-arm 1802 and the second joint component 1805 may move synchronously relative to the third joint component 1806. The second sub-arm 1803 may rotate relative to the second joint component 1805 around the third connection axis, the fourth joint component 1807 may rotate relative to the second sub-arm 1803 around the fourth connection axis, the fifth joint component 1808 may rotate relative to the fourth joint component 1807 around the fifth connection axis, and the sixth joint component 1809 may rotate relative to the fifth joint component 1808 around the sixth connection axis.
[0287] In a possible implementation, the sixth joint component 1809 is connected to an end effector 1810 , and the end effector 1810 is detachably mounted on the sixth joint component 1809 . The end effector 1810 can be a robot gripper, which can facilitate the transfer of objects by users, such as heavy objects, backpacks, crutches, etc.; the end effector 1810 can also be a safety restraint device, such as a seat belt, which can improve the stability and safety of the target object during its movement; the end effector 1810 can also be a support handle or armrest, which is convenient for the target object to grasp and provide stable support for the target object; the end effector 1810 can also be a task controller for setting assistance tasks, and the task controller can be communicated with the control module 1703. The control module 1703 can control the assistance robot 1700 based on the assistance task set by the task controller. For example, the original assistance task is to assist the target object to walk. When the target object reaches the designated position (such as next to the seat), the task controller of the end effector 1810 can be triggered to reset the assistance task to assist the target object to sit down. Then, the control module 1703 can change the assistance task and control the assistance robot 1700 to assist the target object to sit down on the seat.
[0288] In addition, the assisting robot 1700 may be provided with a plurality of mechanical arms 1701, such as Fig.18 As shown, the assisting robot 1700 may be provided with two robotic arms 1701 , and in order to reduce collisions between the multiple robotic arms 1701 and achieve adjustments in different directions, the robotic arms 1701 may be installed on both sides of the main trunk 1801 , respectively.
[0289] like Fig.18As shown, the assisting robot 1700 also includes a visual sensor 1811, a posture sensor, and a base 1812 for movement. The visual sensor 1811 can be installed in the front side of the main trunk 1801, and the visual sensor 1811 can be located between the two side mechanical arms 1701, so as to obtain a wider field of view to capture image data to identify human posture. Among them, the posture sensor can be installed on the front side of the base 1812 to sense the current environment of the assisting robot 1700, such as the position of the target object and the position of the obstacle. When there are multiple posture sensors, they can be arranged on the four side walls of the base 1812 to increase the sensing range and obtain more accurate environmental data.
[0290] like Fig.19 As shown, Fig.19 A schematic diagram of the structure of the assisting robot 1700 from another perspective provided in an embodiment of the present application. A posture sensor, a control cabinet 1901, an antenna 1902, a base display screen 1905 for interactively displaying the status of the base 1812, and a control display screen 1906 for interactively displaying the status of the robotic arm 1701 may be provided in the base 1812, wherein the base display screen 1905 and the control display screen 1906 are both installed on the outside of the control cabinet 1901 to facilitate interaction with the user. The posture sensor may include a three-dimensional laser radar 1903 for detecting obstacles, and a two-dimensional laser radar 1904 for detecting the position of a target object and the position of an obstacle, the two-dimensional laser radar 1904 may be installed on the side wall of the base 1812, the three-dimensional laser radar 1903 may be installed above the control cabinet 1901, and the antenna 1902 may be installed on one side of the base 1812 close to the control cabinet 1901. Fig.19 As shown, the base 1812 can be a Mecanum mobile platform, and the joints of the lower limbs of the assisting robot 1700 can be movable connection mechanisms between the omnidirectional wheels 1813 and the base 1812. The main trunk 1801 can be located at the front side of the control cabinet 1901. Due to the weight of the mechanical arm 1701 and the main trunk 1801, the center of gravity of the assisting robot 1700 moves forward. Therefore, the control cabinet 1901 can be placed at the rear side of the main trunk 1801 to add a counterweight to the base 1812 to balance the center of gravity of the assisting robot 1700 and improve the stability of the movement of the assisting robot 1700.
[0291] like Fig. 20 As shown, Fig. 20The schematic diagram of the internal structure of the control cabinet 1901 of the assisting robot 1700 provided in the embodiment of the present application. The assisting robot 1700 also includes a switch 2001 connected to the antenna 1902 and the control module 1703, a battery 2002 for power supply, and an inverter 2003 for converting the voltage of the battery 2002 for power supply, and the control module 1703, the switch 2001, the battery 2002 and the inverter 2003 are installed in the control cabinet 1901. The control module 1703 can exchange data with the server through the switch 2001 and the antenna 1902, for example, uploading sensor data to the server, or downloading an updated algorithm program from the server.
[0292] like Fig.21 As shown, Fig.21 A schematic diagram of the structural connection of the assisting robot 1700 provided in the embodiment of the present application. The battery 2002 is connected to the inverter 2003, so as to convert the 48V voltage of the battery 2002 into a 221V AC voltage to power the power load of the assisting robot 1700. The control module 1703 can be respectively connected to the tactile sensor, the posture sensor (including the three-dimensional laser radar 1903 and the two-dimensional laser radar 1904), the visual sensor 1811, the manipulator control cabinet 2101, the base display screen 1905 and the control display screen 1906 for communication, so that the control module 1703 can obtain the data of each sensor and the interactive data of each display screen, and the control module 1703 can generate a control instruction based on the obtained data and send it to the manipulator control cabinet 2101, and then control the manipulator 1701 and the base 1812 through the manipulator control cabinet 2101. Specifically, the control module 1703 can use different data transmission methods to establish connections with various components. For example, the control module 1703 can transmit data with the robotic arm 1701 and the base 1812 through the TCP / IP protocol, or transmit data with the tactile sensor and the visual sensor 1811 through the USB3.0 transmission protocol, or transmit data with the posture sensor, the base display 1905 and the posture display through the HDMI transmission protocol.
[0293] The control module 1703 may be an industrial computer, which is used to execute the control methods of the aforementioned embodiments, and in response to identifying that the target object has the intention of requiring assistance, the assistance robot is controlled to move toward the target object, and when the assistance robot moves to the target moving position, the robotic arm is controlled to move to the target joint position, wherein both the target moving position and the target joint position follow the relative position change between the target object and the target sensor, so that the state of the target object can be sensed, and the assistance task can be automatically performed according to the state of the target object. On this basis, during the process of the robotic arm moving to the target joint position or after the robotic arm moves to the target joint position, when an external force is applied to the tactile sensor, the robotic arm is compliantly controlled so that the robotic arm can follow the posture of the target object, thereby improving the comfort of assistance. It can be seen that the control method provided in the present application can interact with the target object in a variety of ways and has a high degree of intelligence.
[0294] An embodiment of the present application further provides an electronic device, including a memory and a processor, wherein the memory stores a computer program, and the processor implements the control methods of various embodiments when executing the computer program.
[0295] An embodiment of the present application further provides a computer-readable storage medium, which is used to store a computer program, and the computer program is used to execute the control methods of the aforementioned embodiments.
[0296] The embodiment of the present application also provides a computer program product, which includes a computer program stored in a computer-readable storage medium. The processor of the computer device reads the computer program from the computer-readable storage medium, and the processor executes the computer program, so that the computer device executes the above control method.
[0297] The terms "first", "second", "third", "fourth", etc. (if any) in the specification of the present application and the above-mentioned drawings are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence. It should be understood that the data used in this way can be interchanged where appropriate to describe the embodiments of the present application, for example, it can be implemented in an order other than those illustrated or described here. In addition, the terms "including" and "having" and any of their variations are intended to cover non-exclusive inclusions, for example, a process, method, system, product or device that includes a series of steps or units is not necessarily limited to those steps or units that are clearly listed, but may include other steps or units that are not clearly listed or inherent to these processes, methods, products or devices.
[0298] It should be understood that in the present application, "at least one (item)" means one or more, and "plurality" means two or more. "And / or" is used to describe the association relationship of associated objects, indicating that three relationships may exist. For example, "A and / or B" can mean: only A exists, only B exists, and A and B exist at the same time, where A and B can be singular or plural. The character " / " generally indicates that the objects associated before and after are in an "or" relationship. "At least one of the following" or similar expressions refers to any combination of these items, including any combination of single or plural items. For example, at least one of a, b or c can mean: a, b, c, "a and b", "a and c", "b and c", or "a and b and c", where a, b, c can be single or multiple.
[0299] It should be understood that in the description of the embodiments of the present application, the meaning of multiple (or multiple items) is more than two, greater than, less than, exceed, etc. are understood to not include the number, and above, below, within, etc. are understood to include the number.
[0300] In the several embodiments provided in the present application, it should be understood that the disclosed systems, devices and methods can be implemented in other ways. For example, the device embodiments described above are only schematic. For example, the division of units is only a logical function division. There may be other division methods in actual implementation. For example, multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be an indirect coupling or communication connection through some interfaces, devices or units, which can be electrical, mechanical or other forms.
[0301] The units described as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed on multiple network units. Some or all of the units may be selected according to actual needs to achieve the purpose of the solution of this embodiment.
[0302] In addition, each functional unit in each embodiment of the present application may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit. The above-mentioned integrated unit may be implemented in the form of hardware or in the form of software functional units.
[0303] If the integrated unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present application is essentially or the part that contributes to the prior art or all or part of the technical solution can be embodied in the form of a software product, and the computer software product is stored in a storage medium, including a number of instructions to enable a computer device (which can be a personal computer, a server, or a network device, etc.) to perform all or part of the steps of the various embodiments of the present application. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (Read-Only Memory, referred to as ROM), random access memory (Random Access Memory, referred to as RAM), disk or optical disk and other media that can store program codes.
[0304] It should also be understood that the various implementation methods provided in the embodiments of the present application can be combined arbitrarily to achieve different technical effects.
[0305] The above is a specific description of the preferred implementation of the present application, but the present application is not limited to the above implementation mode. Technical personnel familiar with the field can also make various equivalent modifications or substitutions under the shared conditions without violating the spirit of the present application. These equivalent modifications or substitutions are all included in the scope defined by the claims of the present application.
Claims
1. A control method for a supporting robot, It is characterized in that The assisting robot is provided with a mechanical arm and a target sensor for object perception, the mechanical arm is provided with a tactile sensor, and the control method comprises: In response to identifying that the target object has an intention to require assistance, the assistance robot is controlled to move toward the target object, and when the assistance robot moves to a target moving position, the mechanical arm is controlled to move to a target joint position, wherein both the target moving position and the target joint position follow the relative position change between the target object and the target sensor; During the process of the mechanical arm moving to the target joint position or after the mechanical arm moves to the target joint position, when the tactile sensor is exerted with external force, the supporting robot is compliantly controlled.
2. The control method according to claim 1, It is characterized in that The types of the target sensor include visual sensors and posture sensors. The target moving position follows the relative position change between the target object and the posture sensor, and the target joint position follows the relative position change between the target object and the visual sensor.
3. The control method according to claim 2, It is characterized in that The controlling the assisting robot to move toward the target object comprises: Acquire current target posture data of the posture sensor, and transform the target posture data based on a preset first transformation matrix to obtain the target moving position; The assisting robot is controlled to move toward the target object based on the target moving position.
4. The control method according to claim 3, It is characterized in that The obtaining of the current target posture data of the posture sensor includes: Acquire the first pose data and the first covariance matrix predicted by the pose sensor at the previous moment, predict the current second pose data based on the first pose data, and predict the current second covariance matrix based on the first covariance matrix; Acquire the measured posture data currently collected by the posture sensor, and determine the target gain based on the measured posture data and the second covariance matrix; The second posture data is corrected based on the target gain to obtain current target posture data of the posture sensor.
5. The control method according to claim 2, It is characterized in that The controlling the robot arm to move to a target joint position comprises: Acquire current image data of the visual sensor, and determine key point positions of multiple key points of the target object in the image data; Convert the coordinate system where the key point position is located to the coordinate system where the assisting robot is located, and determine the center point positions of the multiple key points based on the multiple converted key point positions; The center point position is transformed based on a preset second transformation matrix to obtain the target joint position, and the robot arm movement is controlled based on the target joint position.
6. The control method according to claim 5, It is characterized in that The step of converting the coordinate system where the key point position is located to the coordinate system where the target moving position is located includes: Calibrate the parameters of the visual sensor to obtain an internal parameter matrix of the visual sensor; Determine the installation position of the visual sensor in the assisting robot, and determine the external parameter matrix of the visual sensor according to the installation position; Based on the intrinsic parameter matrix and the extrinsic parameter matrix, the coordinate system where the key point position is located is converted to the coordinate system where the target moving position is located.
7. The control method according to claim 5, It is characterized in that In response to identifying that the target object has an intention to require assistance, before controlling the assistance robot to move toward the target object, the control method further includes: The target object is subjected to posture recognition according to each of the key points, wherein a result of the posture recognition includes whether the target object has an intention to require assistance or whether the target object has no intention to require assistance.
8. The control method according to claim 1, It is characterized in that The compliant control of the assisting robot comprises: Performing kinematic modeling on the supporting robot to obtain a dynamic model of the supporting robot, wherein the dynamic model is used to indicate the relationship between the joint torque of the supporting robot and the joint acceleration of the supporting robot; Acquire current external force data of the tactile sensor, convert the external force data, and obtain a target joint torque; The target joint acceleration of the supporting robot is determined according to the target joint torque and the dynamic model, and the supporting robot is compliantly controlled based on the target joint acceleration.
9. The control method according to claim 8, It is characterized in that The kinematic modeling of the supporting robot to obtain the dynamic model of the supporting robot includes: Determining a mass coefficient according to the mass of the supporting robot, the joint positions of the supporting robot, and the joint speeds of the supporting robot; Determining a friction coefficient according to the friction force when the supporting robot moves and the joint position of the supporting robot; Determining a gravity coefficient according to the gravity acceleration of the supporting robot and the joint position of the supporting robot; A first product between the mass coefficient and the joint acceleration of the supporting robot is determined, and kinematic modeling of the supporting robot is performed according to the first product, the friction coefficient and the sum of the gravity coefficient to obtain a dynamic model of the supporting robot.
10. The control method according to claim 8, It is characterized in that The tactile sensor is provided with a plurality of tactile units, and the external force data is converted to obtain a target joint torque, including: Determine a third transformation matrix for converting the coordinate system of the assisting robot to the coordinate system of the joints of the assisting robot, and determine a fourth transformation matrix for converting the coordinate system of the joints of the assisting robot to the coordinate system of the tactile unit; Determine, according to the third transformation matrix and the fourth transformation matrix, a conversion function between a coordinate system where each of the tactile units is located and a coordinate system where the assisting robot is located, determine the differential of the conversion function with respect to a joint position of the assisting robot, and obtain a Jacobian matrix corresponding to each of the tactile units; A second product between the external force data collected by each of the tactile units and the corresponding Jacobian matrix is determined, and a target joint torque is obtained based on a sum of a plurality of the second products.
11. The control method according to claim 10, It is characterized in that The plurality of tactile units are cylindrically distributed on the robot arm, and a first distance between any two adjacent tactile units is equal. The fourth transformation matrix for converting the coordinate system where the joints of the assisting robot are located to the coordinate system where the tactile units are located includes: Based on the first distance corresponding to the tactile unit, a fourth transformation matrix for converting a coordinate system where the joints of the assisting robot are located to a coordinate system where the tactile unit is located is determined.
12. The control method according to claim 1, It is characterized in that The control method further comprises: In the process of the assisting robot following the target object, when an obstacle is identified within a preset angle range, detecting a target size of the obstacle; When the target size is outside a preset size range, the assisting robot is controlled to avoid the obstacle.
13. The control method according to claim 12, It is characterized in that The controlling the assisting robot to avoid the obstacle comprises: Detecting a second distance between the assisting robot and the obstacle, and when the second distance is less than or equal to a preset distance threshold, constructing a virtual space where the assisting robot is located; Determining the obstacle position of the obstacle and the set end position of the assisting robot in the virtual space; Determine a first force vector of the obstacle according to the obstacle position and the obstacle mass of the obstacle, and determine a second force vector of the end point position according to the end point position and the preset mass of the end point position; The first force vector and the second force vector are synthesized to obtain a target force vector, and the assisting robot is controlled to avoid the obstacle according to the target force vector.
14. The control method according to claim 1, It is characterized in that The tactile sensor is provided with a plurality of tactile units, and when an external force is applied to the tactile sensor, the assisting robot is compliantly controlled, including: When an external force is applied to the tactile sensor, obtaining current external force data of the tactile sensor and determining the activation number of the tactile unit; When the external force data indicates that the external force applied to the tactile sensor is greater than or equal to an external force threshold, and the activation quantity is greater than or equal to a preset quantity threshold, the assisting robot is compliantly controlled.
15. A supporting robot, It is characterized in that The assisting robot is provided with a control module, a mechanical arm and a target sensor for object perception, and the mechanical arm is provided with a tactile sensor: The control module is used for controlling the assisting robot to move toward the target object in response to recognizing that the target object has an intention to assist, controlling the mechanical arm to move to a target joint position when the assisting robot moves to a target moving position, and performing compliant control on the assisting robot when an external force is applied to the tactile sensor during or after the mechanical arm moves to the target joint position; The target moving position and the target joint position both follow the relative position change between the target object and the target sensor.
16. The assisting robot according to claim 15, Features: The supporting robot includes a main trunk, and the mechanical arm includes a first sub-arm and a second sub-arm. One end of the first sub-arm is movably connected to the main trunk, and the other end of the first sub-arm is movably connected to the second sub-arm. The tactile sensors are both provided on the first sub-arm and the second sub-arm.
17. The assisting robot according to claim 16, Features: The two ends of the first sub-arm are respectively connected to a first joint component and a second joint component, the other end of the first joint component is connected to a third joint component, and the third joint component is connected to the main trunk; One end of the second sub-arm is connected to the second joint component, the other end of the second sub-arm is connected to the fourth joint component, the fourth joint component is connected to the fifth joint component, the fifth joint component is connected to the sixth joint component, and the sixth joint component is connected to the end effector.
18. An electronic device comprising a memory and a processor, wherein the memory stores a computer program. It is characterized in that When the processor executes the computer program, the control method according to any one of claims 1 to 14 is implemented.
19. A computer-readable storage medium storing a computer program. It is characterized in that When the computer program is executed by a processor, the control method according to any one of claims 1 to 14 is implemented.
20. A computer program product comprising a computer program, It is characterized in that When the computer program is executed by a processor, the control method according to any one of claims 1 to 14 is implemented.
Citation Information
Patent Citations
Multifunctional intelligent rehabilitation robot for assisting stand and walk
CN101803988A
Indoor motion supporting robot system for old people and application method thereof
CN107877493A
Armrest type intelligent fall protection walking rehabilitation robot
CN109157381A
Self-adaptive control method applied to rehabilitation walking aid robot
CN112999026A
Robot arm operating device and robot system
CN114833795A
Cited By
Anti-falling method and device and robot
CN119871434A
Intelligent accompanying robot and nursing cooperation system and method based on AI and Internet of Things
CN121733597A
Intelligent accompanying robot based on ai and internet of things and nursing cooperation system and method
CN121733597B
Method for controlling robot, robot, and storage medium
CN122210667A
Control method for support robot, support robot, electronic device, computer storage medium, and computer program product
WO2025161865A1
