Control Method of Support Robot, Support Robot and Electronic Device

Abstract

An embodiment of the present application discloses a control method for a helping robot, a helping robot, and an electronic device. By responding to the recognition of the intention of a target object to need help, the helping robot is controlled to move towards the target object. When the helping robot moves to the target moving position, the robotic arm is controlled to move to the target joint position, where both the target moving position and the target joint position change following the relative position between the target object and the target sensor. Thus, the state of the target object can be sensed, and the helping task can be automatically executed according to the state of the target object. On this basis, during the process of the robotic arm moving to the target joint position or after the robotic arm moves to the target joint position, when an external force is applied to the tactile sensor, compliance control is performed on the robotic arm, enabling the robotic arm to conform to the posture of the target object, thereby improving the comfort of helping and achieving diverse interactions with the target object, with a relatively high degree of intelligence.

Description

Technical Field

[0001] This application relates to the technical field of robots, and particularly to a control method for a helping robot, a helping robot, and an electronic device. Background Art

[0002] In daily life, in order to enhance the balance ability, provide support, and improve stability when standing or walking, related auxiliary activity devices are widely used in various scenarios such as daily rehabilitation. Currently, the commonly used auxiliary activity devices are generally crutches or other simple walking aids, and the intelligence level of such auxiliary activity devices needs to be improved. Summary of the Invention

[0003] The following is an overview of the subject matter described in detail in this application. This overview is not intended to limit the scope of protection of the claims.

[0004] Embodiments of this application provide a control method for a helping robot, a helping robot, and an electronic device, which can automatically execute the helping task and have a relatively high intelligence level.

[0005] On the one hand, an embodiment of this application provides a control method for a helping robot. The helping robot is provided with a robotic arm and a target sensor for object perception, and a tactile sensor is arranged on the robotic arm. The control method includes:

[0006] In response to recognizing that the target object has the intention of needing help, controlling the helping robot to move towards the target object. When the helping robot moves to the target moving position, controlling the robotic arm to move to the target joint position, where both the target moving position and the target joint position change following the relative position between the target object and the target sensor;

[0007] During the process of the robotic arm moving to the target joint position or after the robotic arm moves to the target joint position, when an external force is applied to the tactile sensor, performing compliant control on the helping robot.

[0008] On the other hand, an embodiment of this application provides a helping robot. The helping robot is provided with a control module, a robotic arm, and a target sensor for object perception, and a tactile sensor is arranged on the robotic arm:

[0009] The control module is configured to control the assisting robot to move towards the target object in response to recognizing that the target object has an intention of needing assistance. When the assisting robot moves to the target moving position, it controls the robotic arm to move to the target joint position, and during the process of the robotic arm moving to the target joint position or after the robotic arm moves to the target joint position, when an external force is applied to the tactile sensor, perform compliant control on the assisting robot;

[0010] Wherein, both the target moving position and the target joint position change following the relative position between the target object and the target sensor.

[0011] Further, the control module is further configured to:

[0012] Obtain the current target pose data of the pose sensor, transform the target pose data based on a preset first transformation matrix to obtain the target moving position;

[0013] Control the assisting robot to move towards the target object based on the target moving position.

[0014] Further, the control module is further configured to:

[0015] Obtain the first pose data and the first covariance matrix predicted for the pose sensor at the previous moment, predict the current second pose data based on the first pose data, and predict the current second covariance matrix based on the first covariance matrix;

[0016] Obtain the measured pose data currently collected by the pose sensor, and determine the target gain based on the measured pose data and the second covariance matrix;

[0017] Correct the second pose data based on the target gain to obtain the current target pose data of the pose sensor.

[0018] Further, the control module is further configured to:

[0019] Obtain the current image data of the vision sensor, and determine the key point positions of multiple key points of the target object in the image data;

[0020] Convert the coordinate system where the key point positions are located to the coordinate system where the assisting robot is located, and based on the multiple transformed key point positions, determine the central point position of the multiple key points;

[0021] Transform the central point position based on a preset second transformation matrix to obtain the target joint position, and control the robotic arm to move based on the target joint position.

[0022] Furthermore, the control module is also used for:

[0023] Calibrating the parameters of the visual sensor to obtain the internal parameter matrix of the visual sensor;

[0024] Determining the installation position of the visual sensor in the assisting robot, and determining the external parameter matrix of the visual sensor according to the installation position;

[0025] Converting the coordinate system where the key point positions are located to the coordinate system where the target moving position is located based on the internal parameter matrix and the external parameter matrix.

[0026] Furthermore, the control module is also used for:

[0027] Performing pose recognition on the target object according to each of the key points, where the result of the pose recognition includes the intention that the target object needs assistance or the intention that the target object does not need assistance.

[0028] Furthermore, the control module is also used for:

[0029] Performing kinematic modeling on the assisting robot to obtain the dynamic model of the assisting robot, where the dynamic model is used to indicate the relationship between the joint torque of the assisting robot and the joint acceleration of the assisting robot;

[0030] Obtaining the current external force data of the tactile sensor, and converting the external force data to obtain the target joint torque;

[0031] Determining the target joint acceleration of the assisting robot according to the target joint torque and the dynamic model, and performing compliant control on the assisting robot based on the target joint acceleration.

[0032] Furthermore, the control module is also used for:

[0033] Determining the mass coefficient according to the mass of the assisting robot, the joint positions of the assisting robot, and the joint velocities of the assisting robot;

[0034] Determining the friction coefficient according to the friction force when the assisting robot moves and the joint positions of the assisting robot;

[0035] Determining the gravity coefficient according to the gravitational acceleration of the assisting robot and the joint positions of the assisting robot;

[0036] Determine a first product between the quality coefficient and the joint acceleration of the assisting robot, and perform kinematic modeling on the assisting robot according to the sum of the first product, the friction coefficient, and the gravity coefficient to obtain the dynamic model of the assisting robot.

[0037] Further, the control module is further configured to:

[0038] Determine a third transformation matrix for converting the coordinate system where the assisting robot is located to the coordinate system where the joints of the assisting robot are located, and determine a fourth transformation matrix for converting the coordinate system where the joints of the assisting robot are located to the coordinate system where the tactile unit is located;

[0039] According to the third transformation matrix and the fourth transformation matrix, determine a conversion function between the coordinate systems where each tactile unit is located and the coordinate system where the assisting robot is located, and determine the differential of the conversion function with respect to the joint positions of the assisting robot to obtain the Jacobian matrix corresponding to each tactile unit;

[0040] Determine a second product between the external force data collected by each tactile unit and the corresponding Jacobian matrix, and obtain the target joint torque based on the sum of multiple second products.

[0041] Further, multiple tactile units are cylindrically distributed on the robotic arm, and a first distance between any two adjacent tactile units is equal. The control module is further configured to:

[0042] Based on the first distance corresponding to the tactile unit, determine a fourth transformation matrix for converting the coordinate system where the joints of the assisting robot are located to the coordinate system where the tactile unit is located.

[0043] Further, the control module is further configured to:

[0044] During the process of the assisting robot following the target object, when an obstacle is recognized within a preset angular range, detect the target size of the obstacle;

[0045] When the target size is outside a preset size range, control the assisting robot to avoid the obstacle.

[0046] Further, the control module is further configured to:

[0047] Detect a second distance between the assisting robot and the obstacle, and when the second distance is less than or equal to a preset distance threshold, construct a virtual space where the assisting robot is located;

[0048] Determine the obstacle position of the obstacle and the set end position of the assisting robot in the virtual space;

[0049] Determine a first force vector of the obstacle according to the obstacle position and the obstacle mass of the obstacle, and determine a second force vector of the end position according to the end position and the preset mass of the end position;

[0050] Combine the first force vector and the second force vector to obtain a target force vector, and control the assisting robot to avoid the obstacle according to the target force vector.

[0051] Further, the tactile sensor is provided with a plurality of tactile units, and the control module is further configured to:

[0052] When an external force is applied to the tactile sensor, obtain the current external force data of the tactile sensor and determine the activation number of the tactile units;

[0053] When the external force data indicates that the external force received by the tactile sensor is greater than or equal to an external force threshold, and the activation number is greater than or equal to a preset number threshold, perform compliant control on the assisting robot.

[0054] On the other hand, an embodiment of the present application further provides an electronic device, including a memory and a processor. The memory stores a computer program, and when the processor executes the computer program, the control method of the above-mentioned assisting robot is implemented.

[0055] On the other hand, an embodiment of the present application further provides a computer-readable storage medium. The storage medium stores a computer program, and when the computer program is executed by a processor, the control method of the above-mentioned assisting robot is implemented.

[0056] On the other hand, an embodiment of the present application further provides a computer program product. The computer program product includes a computer program, and the computer program is stored in a computer-readable storage medium. The processor of the computer device reads the computer program from the computer-readable storage medium, and the processor executes the computer program, so that the computer device executes the control method of the above-mentioned assisting robot.

[0057] The embodiments of the present application at least include the following beneficial effects: By responding to the recognition of the intention of the target object to need assistance, controlling the assistance robot to move towards the target object, and when the assistance robot moves to the target moving position, controlling the robotic arm to move to the target joint position, where both the target moving position and the target joint position change following the relative position between the target object and the target sensor, so as to be able to sense the state of the target object and automatically perform the assistance task according to the state of the target object. On this basis, during the process of the robotic arm moving to the target joint position or after the robotic arm moves to the target joint position, when an external force is applied to the tactile sensor, compliant control is performed on the robotic arm, so that the robotic arm can conform to the posture of the target object, thereby improving the comfort of assistance. It can be seen that the control method provided by the present application can interact with the target object in a diversified manner and has a relatively high degree of intelligence.

[0058] Other features and advantages of the present application will be described in the following specification, and part of them will become obvious from the specification or be understood by implementing the present application. BRIEF DESCRIPTION OF THE DRAWINGS

[0059] The drawings are used to provide a further understanding of the technical solutions of the present application, and constitute a part of the specification. Together with the embodiments of the present application, they are used to explain the technical solutions of the present application and do not constitute a limitation to the technical solutions of the present application.

[0060] Figure 1 It is a schematic diagram of an optional implementation environment provided by the embodiments of the present application;

[0061] Figure 2 It is a flowchart of a control method for an assistance robot provided by the embodiments of the present application;

[0062] Figure 3 It is a schematic diagram of assisting a target object provided by the embodiments of the present application;

[0063] Figure 4 It is a schematic diagram of recognizing the intention of the target object to need assistance provided by the embodiments of the present application;

[0064] Figure 5 It is a schematic diagram of recognizing the intention of the target object to need assistance provided by the embodiments of the present application;

[0065] Figure 6 It is a schematic diagram of recognizing the intention of the target object to need assistance provided by the embodiments of the present application;

[0066] Figure 7 It is a schematic diagram of recognizing the intention of the target object to need assistance provided by the embodiments of the present application;

[0067] Figure 8Schematic diagram of the position transformation relationship provided by the embodiment of the present application;

[0068] Figure 9 Schematic diagram of the process for processing the pose data measured by the pose sensor provided by the embodiment of the present application;

[0069] Figure 10 Schematic diagram of the position of the center point provided by the embodiment of the present application;

[0070] Figure 11 Schematic diagram of converting the coordinate system where the joints of the assisting robot provided by the embodiment of the present application are located to the coordinate system where the tactile unit is located;

[0071] Figure 12 Schematic diagram of the process of compliant control provided by the embodiment of the present application;

[0072] Figure 13 Schematic diagram of obstacle avoidance of the assisting robot provided by the embodiment of the present application;

[0073] Figure 14 Schematic diagram of an optional overall process of the control method provided by the embodiment of the present application;

[0074] Figure 15 Schematic diagram of an optional overall process of the control method provided by the embodiment of the present application;

[0075] Figure 16 Schematic diagram of an optional overall process of the control method provided by the embodiment of the present application;

[0076] Figure 17 Schematic diagram of an optional structure of the assisting robot provided by the embodiment of the present application;

[0077] Figure 18 Schematic diagram of an optional structure of the assisting robot provided by the embodiment of the present application;

[0078] Figure 19 Schematic diagram of the structure of the assisting robot from another perspective provided by the embodiment of the present application;

[0079] Figure 20 Schematic diagram of the internal structure of the control cabinet of the assisting robot provided by the embodiment of the present application;

[0080] Figure 21 Schematic diagram of the structural connection of the assisting robot provided by the embodiment of the present application. Detailed implementation manners

[0081] In order to make the objectives, technical solutions and advantages of the present application more clear and understandable, the present application will be further described in detail below in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present application and are not used to limit the present application.

[0082] It should be noted that in each specific embodiment of the present application, when it comes to relevant processing that needs to be carried out based on data related to the characteristics of the target object, such as attribute information of the target object or a set of attribute information, the permission or consent of the target object will be obtained first. Moreover, the collection, use, and processing of these data will comply with relevant laws, regulations, and standards. Among them, the target object can be a user. In addition, when the embodiments of the present application need to obtain the attribute information of the target object, the separate permission or separate consent of the target object will be obtained through methods such as pop-up windows or jumping to a confirmation page. After clearly obtaining the separate permission or separate consent of the target object, the necessary data related to the target object for the normal operation of the embodiments of the present application will be obtained.

[0083] In the embodiments of the present application, the term "module" or "unit" refers to a computer program with a predetermined function or a part of a computer program, which works together with other related parts to achieve a predetermined goal, and can be fully or partially implemented by using software, hardware (such as processing circuits or memories), or a combination thereof. Similarly, one processor (or multiple processors or memories) can be used to implement one or more modules or units. In addition, each module or unit can be a part of the overall module or unit that includes the function of the module or unit.

[0084] To facilitate the understanding of the technical solutions provided by the embodiments of the present application, some key terms used in the embodiments of the present application are explained here first:

[0085] Computer Vision Technology (CV), computer vision is a science that studies how to enable machines to "see". Further, it refers to using cameras and computers to replace the human eye to perform machine vision such as identifying and measuring targets, and further performing graphic processing to make the computer process into an image that is more suitable for human eye observation or transmission to an instrument for detection. As a scientific discipline, computer vision studies related theories and technologies, and attempts to establish an artificial intelligence system that can obtain information from images or multi-dimensional data. Computer vision technology usually includes technologies such as image processing, image recognition, image semantic understanding, image retrieval, OCR, video processing, video semantic understanding, video content / behavior recognition, three-dimensional object reconstruction, 3D technology, virtual reality, augmented reality, simultaneous localization and mapping, and also includes common biometric recognition technologies such as face recognition and fingerprint recognition.

[0086] Artificial Intelligence (AI) is a theory, method, technology, and application system that uses digital computers or machines controlled by digital computers to simulate, extend, and expand human intelligence, perceive the environment, acquire knowledge, and use knowledge to obtain the best results. In other words, artificial intelligence is a comprehensive technology in computer science that attempts to understand the essence of intelligence and produce a new intelligent machine that can respond in a way similar to human intelligence. Artificial intelligence also studies the design principles and implementation methods of various intelligent machines, enabling the machines to have the functions of perception, reasoning, and decision-making. Artificial intelligence technology is an interdisciplinary subject with a wide range of fields, including both hardware-level and software-level technologies. The basic technologies of artificial intelligence generally include sensors, dedicated artificial intelligence chips, cloud computing, distributed storage, big data processing technology, pre-trained model technology, operation / interaction systems, mechatronics, etc. Among them, the pre-trained model, also known as the large model or the foundation model, can be widely applied to downstream tasks in various directions of artificial intelligence after fine-tuning. The software technologies of artificial intelligence mainly include several major directions such as computer vision technology, speech processing technology, natural language processing technology, and machine learning / deep learning.

[0087] Pose is used to describe the position and orientation of an object (such as coordinates) in a specified coordinate system. For example, pose is commonly used to describe the position and orientation of a robot in a space coordinate system. Among them, position refers to the positioning of a rigid body in space, and the position of a rigid body can be represented by a 3×1 matrix, that is, the position of the center of the rigid body coordinate system in the base coordinate system. And orientation refers to the orientation of a rigid body in space, and the orientation of a rigid body can be represented by a 3×3 matrix, that is, the orientation of the rigid body coordinate system in the base coordinate system.

[0088] In daily life, in order to enhance the balance ability, provide support, and improve stability when standing or walking, related assisted activity devices are widely used in various scenarios such as daily rehabilitation. Currently, the commonly used assisted activity devices are generally crutches or other simple walking aids, and the intelligence level of such assisted activity devices needs to be improved.

[0089] To solve the above problems, the embodiments of the present application provide a control method for a helping robot, a helping robot, and an electronic device, which can automatically execute the helping task and have a relatively high intelligence level.

[0090] Refer to Figure 1 , Figure 1 FIG.

[0091] Exemplarily, if the control module identifies that the target object has an intention to require assistance based on the obtained sensor data, the control module may control the assistance robot to move towards the target object based on the obtained sensor data. When the assistance robot moves to the target movement position, it may control the robotic arm to move to the target joint position. Among them, both the target movement position and the target joint position change following the relative position between the target object and the target sensor, so as to be able to sense the state of the target object and automatically perform the assistance task according to the state of the target object. On this basis, during the process of the robotic arm moving to the target joint position or after the robotic arm moves to the target joint position, when the control module senses that an external force is applied to the tactile sensor, it may obtain the interaction force data detected by the tactile sensor and perform compliant control on the assistance robot based on this interaction force data, so that the assistance robot can conform to the posture of the target object, thereby improving the comfort of assistance. It can be seen that the control method provided in this application can interact with the target object in a diversified manner and has a relatively high degree of intelligence.

[0092] In addition, the control module may communicate with the server, and the server updates the algorithm package corresponding to the control method to the control module in real time.

[0093] Figure 2 FIG. 8 is a flowchart of a control method for an assistance robot provided by an embodiment of the present application. The control method for the assistance robot may be executed by the control module of the assistance robot, or may also be executed in cooperation by the control module and the server of the assistance robot. In the embodiment of the present application, taking the control method for the assistance robot being executed by the control module of the assistance robot as an example for description, the control method for the assistance robot includes but is not limited to the following steps 201 to step 202.

[0094] Step 201, in response to identifying that the target object has an intention to require assistance, control the assistance robot to move towards the target object. When the assistance robot moves to the target movement position, control the robotic arm to move to the target joint position.

[0095] In a possible implementation, the target object may refer to a user object that can be perceived by the assisting robot and has the intention of needing assistance. The assisting intention may refer to the purpose for which the target object expects to receive the assistance of the assisting robot. For example, an object within the computer vision range of the assisting robot, or an object associated and bound to the terminal through biometric information. The assisting robot can sense and identify the target object through the installed target sensor and evaluate whether the perceived target object needs assistance. For the assisting robot, the manifestations of the assisting intention of the target user can be various. For example, the assisting intention can exist in ways such as voice language, body posture, physical interaction, signal instructions, etc. The specific way can be based on the perception method provided by the target sensor of the assisting robot.

[0096] In a possible implementation, the target moving position may refer to the location where the assisting robot needs to reach to complete the task of assisting the target object, such as the position close to the target object with the intention of needing assistance, or the target moving position may refer to the position of the assisting robot relative to the target object; and the target joint position may refer to the specific posture or joint angle that the robotic arm of the assisting robot needs to reach to complete the task of assisting the target object, or the target joint position may refer to the position of the robotic arm relative to the target object. Among them, both the target moving position and the target joint position change with the relative position change between the target sensor and the target object. Equivalently, during the process of the assisting robot moving towards the target object, the state of the target object can be sensed in real time through the target sensor, the relative position between the target sensor and the target object can be corrected, so as to adjust the target moving position and the target joint position in real time, and further be able to automatically adjust the assisting task to better fit the current state of the target object and improve the comfort of the assistance.

[0097] In a possible implementation, during the process of the assisting robot executing the assisting task of the target object, due to the real-time change of the state of the target object, the target moving position and the target joint position can also be adjusted correspondingly according to different intentions of the target object needing assistance. For example, the assisting postures (i.e., the target moving position and the target joint position) required by the assisting robot for two different assisting needs of assisting the target object to walk and assisting the target object to stand are different. By automatically sensing the state of the target object and adjusting the target moving position and the target joint position of the assisting robot in real time, different interaction methods, moving characteristics, and assisting methods can be provided to interact with the target object in a diversified manner and improve the degree of intelligence.

[0098] In a possible implementation, referring to Figure 3 , Figure 3Schematic diagram of assisting a target object provided by an embodiment of the present application. When a assisting robot needs to assist a target object to walk, the target moving position of the assisting robot can be located at the side rear of the target object. During the assisting process, the state of the target object is sensed in real time, and the target moving position is updated synchronously with the movement of the target object, so that the assisting robot keeps moving towards the target object, maintaining the relative position stability between the target sensor and the target object, and achieving the effect of the assisting robot following the target object for assistance. In addition, the target joint position can be adjusted in real time so that the assisting point of the robotic arm is maintained at the waist, hip and wrist of the target object, imitating the human assisting posture to guide the walking direction of the target object and maintain the walking rhythm, thereby improving the comfort of assistance.

[0099] In a possible implementation manner, when the assisting robot needs to assist the target object to stand up, the target moving position of the assisting robot can be located in front of the target object, and the target moving position is fixed relative to the target object to help the target object maintain balance, that is, the assisting robot moves towards the target object and can remain stable after moving to the target moving position. During the assisting process, the state of the target object is sensed in real time, and the target joint position is adjusted so that the assisting point of the robotic arm is maintained at the upper body of the target object such as the waist and arm to support the weight of the target object, enabling the target object to change from a sitting position or a squatting position to a standing position.

[0100] In a possible implementation manner, the intention of the target object to require assistance can be determined by sensing the state of the target object, that is, the relative position between the target sensor and the target object. The relative position between the target object and the target sensor can include the relative positions of the center of gravity position, joint position, torso position, etc. of the target object and the target sensor respectively. For example, if the center of gravity position of the target object is relatively low relative to the target sensor and there is no large movement in consecutive frames, it can be considered that the target object is trying to stand up, and thus it can be determined that the target object has the intention to require assistance to stand up; if the center of gravity position of the target object changes continuously relative to the target sensor in consecutive frames and the leg joint position has a coherent displacement in a certain direction relative to the target sensor, it can be considered that the object is walking, and thus it can be determined that the target object has the intention to require assistance to walk.

[0101] Step 202: When an external force is applied to the tactile sensor during the process of the robotic arm moving to the target joint position or after the robotic arm moves to the target joint position, perform compliant control on the assisting robot.

[0102] In a possible implementation, during the process of the robotic arm moving to the target joint position or after the robotic arm moves to the target joint position, when it is sensed that an external force is applied to the tactile sensor, it can be considered that the target object is in contact with the assisting robot. Therefore, through compliant control of the assisting robot, the compliant control can enable the assisting robot to make appropriate responses according to the motion posture of the target object. For example, it can control the assisting robot to maintain a fixed output force without being affected by external pressure, or it can control the assisting robot to change its position and speed correspondingly according to the preset impedance characteristics with the action of the external force to maintain gentle interaction with the target object, or to generate feedback on touch or contact while meeting specific postures. Specifically, the compliant control of the assisting robot can be to adjust the parameters of the whole body joints (including the upper and lower limbs) of the assisting robot, such as by adjusting the target joint position, target moving position, motion speed and acceleration of each joint, etc., to reduce the phenomenon of impact or excessive pressure on the target object during the assisting process and improve the comfort of the assistance.

[0103] In a possible implementation, during the assisting process, compliant control can be performed on the robotic arm of the assisting robot. Specifically, parameters such as the torque, position, speed, and acceleration of each joint of the robotic arm can be adjusted to change the motion trajectory and position of the robotic arm of the assisting robot. For example, in the case where it is sensed that an external force is applied to the tactile sensor of the robotic arm, the torque output of the corresponding joint can be adjusted in real time according to the position and direction of the applied external force to maintain the posture of the target object or interact gently with the target object, and at the same time, the dynamic impedance parameters of the corresponding joint can be increased to make the motion of the robotic arm smoother and slower and conform to the posture of the target object. In addition to performing compliant control on the robotic arm, compliant control of the whole body of the assisting robot can also be carried out by adjusting the whole body joints of the assisting robot to conform to the posture of the target object, so that the assisting robot can better imitate the limb movement characteristics of humans and improve the comfort during the assisting process. For example, when the assisting robot is a wheeled robot with a robotic arm, the degrees of freedom of the omnidirectional wheels (such as the forward distance, backward distance, and rotation angle of the omnidirectional wheels) and the moving acceleration of the omnidirectional wheels can be adjusted in the case where it is sensed that an external force is applied to the tactile sensor of the robotic arm to cooperate with the compliant motion of the robotic arm; when the assisting robot is a legged robot with a robotic arm, in the case where it is sensed that an external force is applied to the tactile sensor of the robotic arm, the height between the waist and the feet can be changed by adjusting the joint controller between the two legs and the waist to cooperate with the compliant motion of the robotic arm.

[0104] In a possible implementation, there are various types of target sensors, specifically including visual sensors and pose sensors. Among them, the pose sensor can detect the spatial position and orientation of the target object, and can also detect the spatial position and orientation of the assisting robot. When the pose sensor detects a change in the position of the assisting robot relative to the target object, it can adjust the target moving position of the assisting robot in real time to achieve or maintain the desired relative pose. Therefore, the target moving position can follow the change in the relative position between the target object and the pose sensor. The visual sensor can identify the target object and its pose. When the relative pose, i.e., the relative position, between the target object and the visual sensor changes, the action position of the robotic arm of the assisting robot can be adjusted in real time to respond to the pose change of the target object. Therefore, the target joint position follows the change in the relative position between the target object and the visual sensor.

[0105] In a possible implementation, at least one visual sensor and one pose sensor are provided. By integrating the data of multiple sensors, the measurement errors of individual sensors can be compensated for each other, improving the accuracy and reliability of object perception. For example, the pose sensor can include an inertial measurement unit, a lidar, an ultrasonic sensor, etc. Thus, the relative position between the target object and the pose sensor can be comprehensively determined based on the relative positions between each pose sensor and the target object respectively. Specifically, by collecting the sensor data of each pose sensor at the same moment, noise filtering is performed on all the sensor data, and then all the sensor data is fused and analyzed through a pre-trained neural network model to determine the relative position between the target object and the pose sensor. Correspondingly, the visual sensor can include a depth camera, a stereo camera, an infrared camera, etc. By integrating multiple visual sensors, the three-dimensional pose of the target object can be determined more accurately, and then the relative position between the visual sensor and the target object can be determined.

[0106] In a possible implementation, the assisting robot may be provided with a vision sensor, such as a camera. The terminal can sense and identify people within the computer vision range through the vision sensor set on the assisting robot, and identify the posture of the target object perceived. The vision sensor can capture image data of the current environment where the assisting robot is located in real time, and analyze the image data to determine whether there is a main body of the target object in the image data. Specifically, the human body recognition algorithm, such as the Mediapipe recognition framework, can be used to recognize the human body in the image data to determine the positions of the human body bone points, so that the key points of the target object can be detected in the image data. After determining the key points of the target object, the posture of the target object can be recognized to determine the human body posture of the target object, and then it can be judged whether the target object has the intention of needing assistance to perform the assistance task. Among them, human body posture recognition algorithms with different response rates and complexities can be selected based on the sampling frequency of the vision sensor. For example, if the video stream frame rate of the vision sensor is 30 frames per second, a model with medium model complexity can be selected to recognize the human body posture, so that the frequency of recognizing the human body posture at each video node is 28Hz, realizing real-time human body recognition of the current image data of the vision sensor. When it is detected that the target object does not have the intention of needing assistance, the assisting robot can continue to perform the current assistance task, or continue to recognize the posture of the target object to judge whether there is the intention of needing assistance.

[0107] Referring to Figure 4 , Figure 4 FIG. is a schematic diagram for identifying that the target object has the intention of needing assistance provided by an embodiment of the present application. The terminal may be deployed with a pre-trained pose estimation network model. After obtaining the image data captured by the vision sensor, the image data can be subjected to feature extraction to obtain first image data, and then the first image data is input into the pose estimation network model to obtain human key points. Then, according to the human key points, the current pose of the target object can be determined, and then the current pose of the target object is matched with the to-be-assisted pose with the intention of needing assistance. If the current pose matches the to-be-assisted pose, the terminal can consider that the target object has the intention of needing assistance. As Figure 4 shown, the to-be-assisted pose may be body postures such as the user making a semi-squat, squat-sit, walking or a specific gesture. It should be noted that if there are multiple user objects in the image data, the postures of each user object can be recognized, and the user object whose current pose matches the to-be-assisted pose is determined as the target object.

[0108] In addition, historical image data captured by a visual sensor at the previous moment can be obtained, feature extraction is performed on the historical image data at the previous moment to obtain second image data, then, the first image data and the second image data are stitched together to obtain third image data. Next, the third image data is input into a pose estimation network model to obtain target key points, and then the estimated pose of the target object is determined based on the target key points. By combining the action changes in consecutive frames, a more accurate object pose can be obtained. If the estimated pose matches the to-be-assisted object with an intention of needing assistance, the terminal can consider that the target object has an intention of needing assistance.

[0109] In a possible implementation manner, the assisting robot can also be provided with an acoustic sensor, such as a microphone. The terminal can sense the voice information emitted by the target object through the acoustic sensor provided by the assisting robot, and perform speech recognition on the voice information to determine whether the voice information has an intention of requesting assistance. For example, referring to Figure 5 , Figure 5 is a schematic diagram for identifying that the target object has an intention of needing assistance provided by an embodiment of the present application. When the user says sentences with a semantic meaning of needing assistance, such as "Please help me walk to the room" or "I need assistance", it can be considered that the target object has an intention of needing assistance, so as to control the assisting robot to perform an assisting task.

[0110] In a possible implementation manner, referring to Figure 6 , Figure 6 is a schematic diagram for identifying that the target object has an intention of needing assistance provided by an embodiment of the present application. The terminal can sense the interaction force applied from the outside through the tactile sensor provided on the robotic arm of the assisting robot. As Figure 6 shown, if the terminal senses that an external force is applied to the tactile sensor during the process of not controlling the assisting robot to perform an assisting task, at this time, the object with the smallest relative distance from the target sensor can be used as the target object, and it is considered that the target object has an intention of needing assistance. Or, the terminal can call a visual sensor (if the assisting robot is also provided with a visual sensor such as a camera) to sense and identify the poses of each object within the computer vision range, and use the object that meets the to-be-assisted pose as the target object, and consider that the target object has an intention of needing assistance.

[0111] In a possible implementation manner, referring to Figure 7 , Figure 7This is a schematic diagram for identifying the intention of a target object to require assistance provided by an embodiment of the present application. A help button can be set on the assistance robot, or a remote controller that can be detachably arranged with the assistance robot. When the terminal can sense the signal generated when the help button or the remote controller is triggered, it can be considered that the target object triggering the help button or the remote controller has the intention of requiring assistance. Alternatively, the terminal can call a vision sensor (if the assistance robot is also provided with a vision sensor such as a camera) to sense and identify the postures of each object within the computer vision range, and regard the object that meets the posture to be assisted as the target object, and consider that the target object has the intention of requiring assistance.

[0112] In a possible implementation manner, sensor data related to an object is captured by a target sensor. For example, the image data captured by a vision sensor and the pose data detected by a pose sensor are used for feature extraction and splicing to obtain multi-source sensor data. The multi-source sensor data is imported into a pre-trained deep learning model such as a convolutional neural network model, or a real-time object detection algorithm such as the YOLO (You Only Look Once) object detection algorithm, the SSD (Single Shot MultiBox Detector) object detection algorithm, etc., so as to quickly detect the human object in the sensor data (such as a frame image) and estimate their positions. Then, a pose estimation model such as the OpenPose human pose estimation algorithm model or the AlphaPose pose estimation algorithm based on human key point detection can be used to analyze the postures of the human objects in the sensor data, analyze the behavior patterns and action intentions of the human objects, and determine whether there is an intention of requiring assistance. When it is recognized that the target object has the intention of requiring assistance, an optimal path for the assistance robot to reach the target moving position can be calculated through a dynamic path planning algorithm, and the assistance robot can be controlled to move towards the target object to be assisted. Among them, in addition to being able to sense human objects, the target sensor can also sense non-human objects, that is, obstacle objects. For example, lidar, vision sensors, etc. sense the surrounding environment of the assistance robot, so that during the movement, the target sensor can also be used to sense the obstacles on the moving path to achieve the obstacle avoidance function. At the same time, during the movement, the target sensor is used to sense the target object in real time, correct the relative distance between the target object and the target sensor, and adjust the target moving position of the assisted target object. After the assistance robot reaches the target moving position, the manipulator can be controlled to move to the human assistance point, that is, the target joint position, and at the same time, the triggering situation of the tactile sensor on the manipulator and the relative distance between the target sensor and the target object are sensed to judge the state of the target object, and the position and force of the manipulator are adjusted in real time to adapt to the actions and balance states of the target object.

[0113] Specifically, the assisting robot and its control method provided by the embodiments of the present application can be applied to various human-machine interaction scenarios such as medical rehabilitation scenarios, daily care scenarios, public place assistance scenarios, rescue and disaster relief scenarios, etc., with a high degree of intelligence. For example, in a hospital or a rehabilitation center, the assisting robot can be controlled to help patients or the elderly perform pose change actions such as walking, standing, and sitting. Since it can sense the state of the target object and adjust the assisting plan in real time according to the state and needs of the target object, it can improve the comfort of assistance while assisting the object in rehabilitation and reduce the workload of physical therapists. Another example is to control the assisting robot to detect the activity state of the elderly in real time, help the elderly perform various exercises (such as going up and down stairs, standing, walking) indoors or outdoors, and provide assistance in case of emergencies (such as helping up when falling). Another example is to help passengers or customers with mobility difficulties reach their destinations in public places such as stations or shopping malls. Specifically, in the station scenario, it can assist passengers with mobility difficulties to move from the waiting area to the side of the car door and help them board the bus door pedal area, or help the passengers take a seat inside the car. In the shopping mall scenario, it can assist customers to board the steps of the escalator.

[0114] In a possible implementation manner, during the process of controlling the assisting robot to move towards the target object, the current target pose data of the pose sensor can be obtained, and the target pose data is transformed based on a preset first transformation matrix to obtain the target moving position; then, the assisting robot is controlled to move towards the target object based on the target moving position.

[0115] In a possible implementation manner, the pose sensor can detect the human pose data of the target object, that is, the target pose data can be the human pose data measured by the pose sensor for the target object at the current moment, or can be calculated based on the measured human pose data. The preset first transformation matrix can refer to a pose transformation matrix for converting from the human space coordinates to the space coordinates of the assisting robot, which is used to represent the pose transformation relationship between the target moving position of the assisting robot and the position where the target object is located, so as to obtain the expected position of the assisting robot relative to the target object, that is, the target moving position, through the conversion of the space coordinates where the target object is located, and then control the assisting robot to move to the target moving position near the target object to perform the assisting task for the target object.

[0116] In one possible implementation, multiple first candidate transformation matrices may be pre-set to accommodate different postures of the target object or different assistance requirements. For example, different transformation matrices may be selected based on the target object's motion pattern and posture to determine the corresponding target movement position of the assistance robot, thereby ensuring the safety and comfort of the assistance. After determining the target movement position, in the process of controlling the assistance robot to move toward the target object based on the target movement position, the position and posture data of the target object may be detected in real time, and the target movement position of the assistance robot may be updated in real time based on the first transformation matrix. Alternatively, the position and posture data of the target object may be detected in real time, the motion pattern and posture of the target object may be determined, and the first transformation matrix may be updated in real time to adjust the target movement position of the assistance robot.

[0117] Reference Figure 8 , Figure 8 The position transformation relationship diagram provided in the embodiment of the present application. The target position data of the target object at the current moment can be detected by the position sensor, wherein the state data of the target object, i.e., the position and direction, can be extracted from the target position data, such as the position coordinates (x h ,y h , 1), and the angle θ of the target object relative to the supporting robot h Assuming that the target object's assistance requirement is to assist walking, a first transformation matrix corresponding to the assistance walking can be determined from a plurality of first candidate transformation matrices, thereby based on the preset first transformation matrix [0.92, 0, 1] T The target moving position of the supporting robot is obtained by transforming the position of the target object. Therefore, the supporting robot can be controlled to move from the original position to the target object and to the target moving position, thereby performing the task of supporting the target object. Specifically, the target moving position (x base ,y base , the calculation process of 1) is shown in the following formula:

[0118]

[0119] Therefore, it can be seen that the target movement position of the supporting robot is related to the position of the target object relative to the supporting robot. As the supporting robot continues to move, the target pose data detected by the pose sensor continuously changes, and thus the determined target movement position also changes accordingly. In other words, the target movement position of the supporting robot changes with the relative position between the target object and the pose sensor. It should be noted that the target movement position can be determined based on the supporting robot's own spatial coordinate system. Therefore, although the target movement position is constantly changing, the target movement position at each moment can be relatively fixed relative to the world coordinate system or the coordinate system of the target object.

[0120] In a possible implementation, the target pose data may be obtained by synthesizing the pose data measured by the pose sensor at multiple moments. Specifically, the first pose data and the first covariance matrix predicted for the pose sensor at the previous moment may be obtained, the second pose data at the current moment may be predicted based on the first pose data, and the second covariance matrix at the current moment may be predicted based on the first covariance matrix; then the measured pose data currently collected by the pose sensor is obtained, and the target gain is determined based on the measured pose data and the second covariance matrix; next, the second pose data is corrected based on the target gain to obtain the current target pose data of the pose sensor.

[0121] In a possible implementation, due to the measurement error of the pose sensor, it is easy to confuse the target object (such as a person) with a non-target object (such as an obstacle), resulting in the inability to perform the helping task. Therefore, it is necessary to correct the human pose data directly measured by the pose sensor to obtain highly accurate target pose data. Refer to Figure 9 , Figure 9 FIG. is a schematic flowchart of data processing for the pose data measured by the pose sensor provided by the embodiment of the present application. As Figure 9 shown, for the correction process of the pose data, first, the pose data at the previous moment is used for prediction, including state prediction estimation of the pose data, and then prediction estimation of the covariance matrix corresponding to the predicted state quantity is performed. Then, the predicted state value is corrected using the measurement value at the current moment, including calculating the target gain, and then the predicted state value is corrected based on the target gain. Finally, the covariance matrix estimated by the previous prediction is updated. The human pose data may include the angles and directions of human joints, the three-dimensional coordinate positions of body key points, etc. Specifically, the human pose data detected by the pose sensor may be understood as the motion state of the target object, including the position and speed of the target object, that is The first pose data is the pose data predicted by predicting the pose data measured by the pose sensor at the previous moment. Since the motion state of the target object is not affected by external control inputs, based on the state equation of the correction system, the external control vector and the external control input matrix can be ignored. Thus, the second pose data predicted at the current moment can be obtained by multiplying the first pose data predicted at the previous moment by the state transition matrix, that is, the state of the target object at the current moment is predicted. The state transition matrix is used to describe the relationship of the correction system changing from the previous moment to the current moment. Specifically, the state transition matrix is:

[0122]

[0123] Wherein, ΔT represents the time interval for correcting the system, which represents the time difference between the previous moment and the current moment, and is used to correct the change of the state variable caused by the time interval.

[0124] According to the state equation of the corrected system after ignoring the influence of the external control input, the calculation formula for the second pose data can be obtained, which can be specifically shown as follows:

[0125]

[0126] Wherein, represents the predicted second pose data at the current moment, x k-1 represents the predicted first pose data at the previous moment, ω k represents the process noise for which the corrected system satisfies the Gaussian distribution.

[0127] In a possible implementation, the first covariance matrix refers to the matrix that measures the uncertainty of the predicted first pose data when predicting the pose sensor at the previous moment, and is used to represent the uncertainty of the state estimation at the previous moment in the corrected system. Among them, the first covariance matrix is an expression of the error distribution of the first pose data. The diagonal elements in the first covariance matrix represent the variances of the respective state variables of the first pose data (i.e., the elements of the first pose data), and the non-diagonal elements represent the covariances between different state variables, which are used to reflect the correlation between different state variables. Based on the corrected system, the second covariance matrix corresponding to the second pose data at the current moment can be obtained through the product operation of the state transition matrix and the first covariance matrix. Specifically, the calculation formula for the second covariance matrix can be shown as follows:

[0128]

[0129] Wherein, represents the second covariance matrix corresponding to the second pose data P k-1 represents the first covariance matrix corresponding to the first pose data x k-1 Q represents the noise covariance matrix corresponding to the process noise ω k of the corrected system.

[0130] In a possible implementation, the target gain is used to optimize the prediction accuracy by combining the predicted state (i.e., the second pose data) obtained from the pose sensor prediction with the observed quantity (i.e., the measured pose data) measured by the pose sensor, so as to balance between the predicted pose of the second pose and the measured pose data, correct the predicted data of the second pose, and obtain the target pose data. Specifically, the feedback gain matrix is calculated through the second covariance matrix and the observation matrix preset in the correction system, and then the target gain is calculated through the measured pose data and the feedback gain matrix. The calculation formula of the feedback gain matrix can be shown as follows:

[0131]

[0132] Where K k represents the feedback gain matrix, H represents the observation matrix, and R represents the observed quantity noise matrix. Specifically, the observation matrix can be expressed as:

[0133]

[0134] After obtaining the feedback gain matrix, the target gain can be obtained by balancing the measured pose data and the second pose data through the feedback gain matrix. The specific calculation formula can be shown as follows

[0135]

[0136] Where B represents the target gain, z k represents the measured pose data; it should be noted that the measured pose data can be obtained by low-pass correction of the initial pose data directly measured by the pose sensor. Specifically, the correction calculation formula of the measured pose data can be shown as follows:

[0137]

[0138] Where α represents the correction coefficient, and the correction coefficient represents the low-pass correction intensity, represents the second initial pose data directly measured by the pose sensor at the current moment, and z k-1 represents the first initial pose data directly measured by the pose sensor at the previous moment and is the historical measured pose data obtained by low-pass correction.

[0139] Next, the second pose data is corrected using the target gain, and the current target pose data of the pose sensor can be obtained. The specific correction formula for the target pose data x k can be shown as follows:

[0140]

[0141] After correcting the second pose data of the predicted state using the target gain, the second covariance matrix corresponding to the second pose data can be updated. The specific matrix update formula can be shown as follows:

[0142]

[0143] where P k represents the updated second covariance matrix. By correcting the feedback gain matrix K k of the system and the observation matrix H, the predicted second covariance matrix is updated, so that the updated second covariance matrix P k can be used to predict the target pose data at the next moment.

[0144] In a possible implementation, the relationship between the predicted state and the observed quantity in the correction system can be represented by an observation equation. Specifically, the observation equation can be shown as follows:

[0145] z k = Hx k + v k (9)

[0146] where v k represents the observation noise that satisfies the Gaussian distribution in the correction system.

[0147] In a possible implementation, during the process of controlling the manipulator to move to the target joint position, the current image data of the vision sensor can be obtained first to determine the key point positions of the target object in the image data; then, the coordinate system where the key point positions are located is converted to the coordinate system where the assisting robot is located, and based on the multiple converted key point positions, the central point position of the multiple key points is determined; the central point position is transformed based on a preset second transformation matrix to obtain the target joint position, and the manipulator is controlled to move based on the target joint position.

[0148] In a possible implementation, there can be multiple key points of the target object. Specifically, the key points of the target object can include significant feature points on the human body, such as joint points like eyes, ears, shoulders, waist, elbow joints, wrists, hip joints, knee joints, and ankles. When performing key point detection on the image data, the key points can be determined from multiple joint points based on the main part of the target object displayed in the image data. For example, if the main part of the target object displayed in the image data is the upper limb part of the target object, joint points such as the shoulder and waist can be selected as key points; if the main part of the target object displayed in the image data is the lower limb part of the target object, joint points such as the hip joint and knee joint can be selected as key points. Alternatively, when performing key point detection on the image data, the key points can be determined from multiple joint points based on the assistance requirement of the target object. For example, if the assistance requirement of the target object is to assist in standing up, joint points such as the shoulder, elbow joint, and wrist can be selected as key points; if the assistance requirement of the target object is to assist in walking, joint points such as the waist, hip joint, and elbow joint can be selected as key points.

[0149] In a possible implementation, the key points of the target object can be fixed joint points on the human body. When multiple fixed joint points of the target object cannot be detected from the image data, after re - determining the target movement based on the relative position between the pose sensor and the target object, after re - controlling the assistance robot to reach the new target movement position, the image data of the visual sensor is re - acquired for analysis. For example, the key points of the target object can be four key points on the human body, namely the left shoulder, right shoulder, left hip, and right hip. When the above four key points of the target object cannot be completely detected in the image data captured by the visual sensor, it can be considered that the assistance robot cannot accurately perceive the target object at present, and it is easy to have a perception error. At the same time, it can also be considered that the relative position between the assistance robot and the target object, such as the distance, is too close or too far, making it difficult to ensure the safe execution of the assistance task. Therefore, the target movement position can be re - determined based on the relative position between the pose sensor and the target object, and the assistance robot is controlled to move to the updated target movement position, and then the current image data is captured by the visual sensor until the left shoulder, right shoulder, left hip, and right hip four key points of the target object can be completely detected in the new image data.

[0150] In a possible implementation, the key point positions may refer to the coordinate points of each key point in the image data in the pixel coordinate system. Here, the pixel coordinate system may use the upper left corner of the image as the origin of the pixel coordinate system, with the horizontal rightward direction as the positive direction of the horizontal axis and the vertical downward direction as the positive direction of the vertical axis. After determining multiple key points through image recognition of the image data, the key point positions of each key point in the image data can be determined through image registration, that is, the two-dimensional coordinate points in the pixel coordinate system. Then, the coordinate system where the key point positions are located, namely the pixel coordinate system, is converted to the coordinate system where the assisting robot is located. At this time, the converted key point positions are represented as three-dimensional coordinate points of each key point in the coordinate system where the assisting robot is located. Then, the arithmetic mean of the multiple converted key point positions is calculated to obtain the central point position of the multiple key points, that is, the coordinates of the center point of the multiple key points. Then, the central point position is transformed through a preset second transformation matrix to obtain the target joint position. Here, the second transformation matrix may refer to the transformation matrix for the central point position to change to the position where the robotic arm needs to assist, that is, the central point position is used as a reference point for the expected assisting position of the assisting robot to determine the target joint position of the robotic arm.

[0151] In a possible implementation, referring to Figure 10 , Figure 10 is a schematic diagram of the central point position provided by an embodiment of the present application. Image data of the environment where the current assisting robot is located is captured through a vision sensor. As Figure 10 shown, due to the limitation of the shooting angle of the vision sensor, if the image data captured by the vision sensor is required to capture the main body of the target object, the relative distance between the vision sensor and the target object needs to be greater than the minimum shooting distance of the vision sensor. For example, if the vision sensor is a depth camera and the minimum shooting distance of this depth camera is 50 cm, then when the relative distance between the depth camera and the target object reaches 50 cm, the main body of the target object is shown in the image data captured by the vision sensor. Otherwise, the relative position between the assisting robot and the target object needs to be adjusted to re-capture the image data. If the main body of the target object is shown in the image data, then the key points of the target object can be detected for this image data. Through the left shoulder of the human body (such as Figure 10 the marked point A shown in Figure 10 ), the right shoulder (such as Figure 10 the marked point B shown in Figure 10 ), the left hip (such as Figure 10 the marked point C shown inFigure 10 For the target object shown in [figure], the center point is located at the waist of the target object. Thus, the waist of the target object can be used as a reference point for the desired support position of the manipulator of the support robot, that is, the reference point for the target joint position. By transforming the center point position through the second transformation matrix, the support positions on both sides of the waist can be obtained as the target joint positions (such as the marked point F shown in [figure]). Figure 10 as shown in [figure].

[0152] In a possible implementation, parameter calibration of the vision sensor can obtain the internal parameter matrix of the vision sensor. Then, determine the installation position of the vision sensor in the support robot, and determine the external parameter matrix of the vision sensor according to the installation position. Then, based on the internal parameter matrix and the external parameter matrix, transform the coordinate system where the key point positions are located to the coordinate system where the target movement position is located.

[0153] In a possible implementation, the internal parameter matrix of the vision sensor describes the internal optical characteristics of the vision sensor, including the focal length and the imaging center (optical center). The focal length of the vision sensor can be represented by the focal length of the x-axis and the focal length of the y-axis in pixels respectively, and the optical center can be represented by the projection position on the image plane. For example, after parameter calibration of the vision sensor, the internal parameter matrix of the vision sensor can be obtained, and the internal parameter matrix can be expressed as:

[0154]

[0155] where, f x represents the focal length of the x-axis, f y represents the focal length of the y-axis, and (c x , c y ) represents the position of the optical center.

[0156] In a possible implementation, the external parameter matrix of the vision sensor defines the spatial relationship between the coordinate system of the vision sensor and the coordinate system where the support robot (base) is located. The external parameter matrix includes a rotation matrix and a translation matrix. Among them, the rotation matrix is usually a 3X3 matrix, representing the rotation of the vision sensor on the coordinate system of the support robot (base), and the translation matrix is usually a 3X1 vector, used to represent the position of the vision sensor relative to the support robot (base). For example, according to the installation position of the vision sensor in the support robot, the external parameter matrix can be obtained, and it can be specifically expressed as:

[0157]

[0158] where, R represents the rotation matrix and T represents the translation matrix.

[0159] In a possible implementation, to convert the coordinate system where the key point positions are located to the coordinate system where the target movement position is located, it is possible to first convert from the coordinate system where the key point positions are located, i.e., the pixel coordinate system, to the coordinate system where the vision sensor is located, and then convert from the coordinate system where the vision sensor is located to the coordinate system where the assisting robot (base) is located. The specific calculation process can be shown as follows:

[0160]

[0161] where (u, v) represents the pixel coordinate system, i.e., the coordinate system where the key point positions are located; (X c , Y c , Z c ) represents the coordinate system where the vision sensor is located, and the origin of the coordinate system where the vision sensor is located is the optical center of the vision sensor. The Z-axis is parallel to the optical axis of the vision sensor, and Z c is the shooting direction of the vision sensor; (X B , Y B , Z B ) represents the coordinate system where the assisting robot (base) is located.

[0162] In a possible implementation, kinematic modeling of the assisting robot based on the joint variables of the assisting robot (such as displacement, velocity, acceleration, position) can obtain the dynamic model of the assisting robot, which is used to describe the relationship between the joint torques of the assisting robot and the joint accelerations of the assisting robot. Then, through the tactile sensors set on the robotic arm of the assisting robot, external force data is obtained, and the external force information at the force application point is converted to the coordinate system of the corresponding joints of the assisting robot through a transformation matrix, obtaining the torques required for the joints of the assisting robot to adapt to the external force, i.e., the target joint torques, so that the assisting robot can generate sufficient force to resist the external force while ensuring the compliance of the action and the stability of the assistance. Then, based on the obtained target joint torques and the established dynamic model, the target joint accelerations of the assisting robot are determined, and then the assisting robot is compliant controlled based on the target joint accelerations.

[0163] In a possible implementation, the mass coefficient can be determined first according to the mass of the assisting robot, the joint positions of the assisting robot, and the joint velocities of the assisting robot; then the friction coefficient can be determined according to the friction force when the assisting robot moves and the joint positions of the assisting robot; then, the gravity coefficient can be determined according to the gravitational acceleration of the assisting robot and the joint positions of the assisting robot; the first product between the mass coefficient and the joint accelerations of the assisting robot is determined, and kinematic modeling of the assisting robot is performed based on the sum of the first product, the friction coefficient, and the gravity coefficient, obtaining the dynamic model of the assisting robot.

[0164] In a possible implementation, the mass coefficient can represent the inertial properties of each joint of the assisting robot. The mass coefficient can be understood as the mass matrix in the robot dynamics model, which is used to describe the mass distribution of each joint of the assisting robot. Specifically, the mass coefficient of each joint in the assisting robot can be determined by the mass, joint position, and joint velocity of the corresponding joint of the assisting robot. Among them, the joint position of the assisting robot can be determined by the position encoder of each joint, and the mass coefficient can change with the change of the joint position. The friction coefficient can be understood as the friction matrix in the dynamics model, which represents the influence of the friction force on each joint when the assisting robot moves. The friction force directions and magnitudes corresponding to the joints at different joint positions are different. Therefore, the friction coefficients corresponding to the joints at different joint positions are different. The gravity coefficient can be understood as the gravity matrix in the dynamics model, which is used to represent the influence of gravity on each joint of the assisting robot. The gravity directions received by the joints at different joint positions are different, affecting the attitude transformation of each joint. The first product is obtained by multiplying the mass coefficient by the joint acceleration of the corresponding joint, and can represent the inertial force caused by the joint acceleration, that is, the inertial force that each joint needs to resist when moving, ensuring timely response when subjected to external forces. By performing kinematic modeling on the assisting robot through the sum of the first product, the friction coefficient, and the gravity coefficient, the specific formula of the obtained dynamics model can be shown as follows:

[0165]

[0166] Among them, M(q) represents the mass coefficient of the joint at joint position q, represents the first product, represents the joint velocity of the joint at the joint position, represents the friction coefficient of the joint at the joint position, represents the joint velocity of the joint at the joint position, g(q) represents the gravity coefficient of the joint at joint position q, and τ represents the joint torque of the joint at joint position q.

[0167] Next, based on the dynamics model, the target joint acceleration generated under the action of external forces can be deduced. The specific calculation formula of the target joint acceleration can be shown as follows:

[0168]

[0169] After obtaining the target joint acceleration, an adaptive control strategy can be generated through a proportional-integral-derivative controller, that is, a PID controller, to adjust the torque output of the corresponding joint to respond to external forces, so as to control the movement of the assisting robot based on the target joint acceleration.

[0170] In a possible implementation, the tactile sensor includes a plurality of tactile units. Therefore, a third transformation matrix for converting the coordinate system where the assisting robot is located to the coordinate system where the joints of the assisting robot are located can be determined first, and a fourth transformation matrix for converting the coordinate system where the joints of the assisting robot are located to the coordinate system where the tactile units are located can be determined. Then, according to the third transformation matrix and the fourth transformation matrix, a conversion function between the coordinate systems where each tactile unit is located and the coordinate system where the assisting robot is located can be determined, and the differential of the conversion function with respect to the joint positions of the assisting robot can be determined to obtain the Jacobian matrix corresponding to each tactile unit. Then, a second product between the external force data collected by each tactile unit and the corresponding Jacobian matrix is determined, and the target joint torque is obtained based on the sum of the plurality of second products.

[0171] In a possible implementation, assume that the assisting robot needs to perform an assisting task. At this time, the task space of the assisting task has a non-linear relationship with the joint space, that is, the coordinate points of the target joint positions required to perform the assisting task are located in the coordinate system where the assisting robot is located, while the coordinate points of the external force data received during the performance of the assisting task are located in the coordinate system where the tactile units are located. It is impossible to directly use the external force data to adjust the target joint positions to achieve compliant control. Therefore, a conversion function determined by the third transformation matrix and the fourth transformation matrix is required. The conversion function can describe how to convert the coordinate systems where each tactile unit is located to the coordinate system where the assisting robot is located. Then, by performing a differential operation on the joint positions of the assisting robot using the conversion function, the two different coordinate systems can be associated, and the Jacobian matrix of the corresponding joints can be obtained. The Jacobian matrix of the corresponding joints can be expressed as follows:

[0172]

[0173] where f represents the conversion function, J(q) represents the Jacobian matrix of the joint at the joint position q, and the Jacobian matrix of the corresponding joint can be obtained by taking the partial derivative of the joint position q with respect to the conversion function f.

[0174] Taking the differential of the conversion function with respect to the joint positions of the assisting robot as the Jacobian matrix of each tactile unit to represent the influence of the magnitude and direction of the external force received by each tactile unit on different joint positions. Specifically, assume that the direction of the axis of each joint i is represented by the unit vector z r i and the position of the origin of each joint coordinate system can be represented by the vector p i . Then, the Jacobian matrix J i (q) of joint i can be expressed as:

[0175]

[0176] where (x b, y b ) represents the coordinate system where the assisting robot is located, q k represents the k-th joint, where 1 < k ≤ n, and n represents the total number of all joints of the assisting robot, that is, the sum of the number of joints of the robotic arm and the number of joints of the base.

[0177] Since each joint of the assisting robot rotates around the z-axis, therefore, the pose transformation matrix T of each joint i i can be:

[0178]

[0179] Next, substituting the pose transformation matrices of the joints corresponding to each tactile unit into the Jacobian matrix of the corresponding joint, the Jacobian matrix corresponding to each tactile unit can be obtained. Then, multiplying the external force data collected by each tactile unit by the corresponding Jacobian matrix to obtain the second product. Using the Jacobian matrices of each tactile unit to convert the external force data received by the corresponding tactile unit to the overall joint, that is, the second product can represent the force transmitted from each tactile unit to the overall joint. Furthermore, by calculating the sum of each second product, it can represent the force exerted by the external world (i.e., the target object) during the interaction with the robotic arm during the assisting process, thereby mapping to obtain the target joint torque. Among them, the calculation formula of the target joint torque τ can be shown as the following formula:

[0180]

[0181] Among them, represents the Jacobian matrix of the tactile unit j of the robotic arm joint i, represents the external force data of the tactile unit j of the robotic arm joint i. Specifically, since the external force f z is applied in the z-axis direction, therefore the external force data can be represented in the following form:

[0182]

[0183] In a possible implementation, the joints of the assisting robot include multiple sub-arms of the robotic arm and joint components, and also include the base joint of the assisting robot. Therefore, compliant control can be achieved for the joint torques and joint accelerations of each sub-arm of the robotic arm and the joint components respectively. When the assisting robot is a wheeled robot, that is, the base joint of the assisting robot may include omnidirectional wheels mounted on the base. Among them, the joint torque of the omnidirectional wheel can be represented by the rotation angle of the omnidirectional wheel, and the joint acceleration of the omnidirectional wheel can be represented by the angular velocity, angular acceleration of the omnidirectional wheel, the torque of the motor driving the omnidirectional wheel to rotate, etc. Thus, compliant control can be achieved by adjusting the rotation direction, rotation speed, rotation acceleration, etc. of the omnidirectional wheel to conform to the movement trend of the assisted object. When the assisting robot is a legged robot, the base joint of the assisting robot may include an ankle joint, a knee joint, a hip joint, etc. Similarly, compliant control can be achieved by separately controlling the joint torques and joint accelerations of the ankle joint, the knee joint, and the hip joint.

[0184] In a possible implementation, to determine the fourth transformation matrix for converting the coordinate system where the joint of the assisting robot is located to the coordinate system where the tactile unit is located, it is necessary to perform kinematic modeling for each tactile unit of the tactile sensor with respect to its position on the robotic arm, and establish the relationship between the tactile unit and the kinematic chain on the robotic arm. Among them, each joint of the robotic arm can be approximately regarded as a cylinder. Therefore, it can be considered that multiple tactile units of the tactile sensor are cylindrically distributed on the surface of the robotic arm, and the first distance between any two adjacent tactile units is equal. Among them, the first distance is expressed as the distance between two adjacent tactile units in the direction of the length of the robotic arm link. At the same time, the attitude of the coordinate system where the tactile unit is located with respect to the coordinate system where the assisting robot is located is the same as the attitude of the coordinate system where the joint corresponding to the tactile unit is located with respect to the coordinate system where the assisting robot is located. Thus, based on the first distance corresponding to the tactile unit, the distance between each tactile unit and the starting end of the corresponding joint can be determined, and further, the fourth transformation matrix for converting the coordinate system where the joint of the assisting robot is located to the coordinate system where the tactile unit is located can be determined.

[0185] Refer to Figure 11 , Figure 11 is a schematic diagram for converting the coordinate system where the joint of the assisting robot provided in the embodiment of the present application is located to the coordinate system where the tactile unit is located. Assume the first distance is 0.015 meters, as Figure 11As shown in the figure, the robotic arm can be regarded as a cylinder. Since the first distance between any two adjacent tactile units is equal, equivalently, multiple tactile units distributed cylindrically on the surface of the robotic arm can be divided along the height of the cylinder to obtain multiple rings. The distance between two adjacent rings is fixed at 0.015 meters, and the tactile units on the same ring can be modeled as the same unit. Therefore, based on the first distance of the tactile units and the position of the tactile units (the rings they are on) on the robotic arm (cylinder), the conversion relationship between the tactile units and the joints they are on can be obtained, and then the fourth transformation matrix can be obtained. The fourth transformation matrix can be specifically expressed as:

[0186]

[0187] As Figure 11 shown, t i can be understood as the origin of the coordinate system of joint i where the tactile unit is located, and can be expressed as the origin of the coordinate system of tactile unit j in joint i. (0.015×j) can represent the distance between the origin of the coordinate system of tactile unit j and the origin of the coordinate system of joint i where it is located. can be expressed as the fourth transformation matrix for converting the coordinate system where the joint of the assisting robot is located to the coordinate system where the tactile unit is located.

[0188] In a possible implementation, referring to Figure 12 , Figure 12 is the schematic flowchart of the compliance control provided by the embodiment of the present application. The realization of full-body compliance control based on the tactile sensor can be achieved through the following steps. First, kinematic modeling needs to be performed on each tactile unit of the tactile sensor for its position on the robotic arm. Then, dynamic modeling of the assisting robot is carried out to construct a dynamic model. Then, based on the modeling of the tactile units and the dynamic model of the assisting robot, the Jacobian matrix can be calculated for each tactile unit, so that the external force data received by the tactile sensor can be mapped to the force conditions of each joint of the robotic arm and the whole body of the assisting robot through the Jacobian matrix conversion, and then the joint accelerations corresponding to the joints of the whole body of the assisting robot can be obtained for compliance control.

[0189] In a possible implementation, based on the perceived need for assistance of the target object, the assistance robot can be controlled to follow the target object to perform the assistance task. For example, when the intention of the target object to walk with assistance is perceived, the assistance robot can be controlled to update the target movement position in real time based on the relative distance between the pose sensor and the target object, and control the assistance robot to move towards the target object based on the updated target movement position, so as to realize the assistance robot following the target object to perform the assistance task. During the process of the assistance robot following the target object, obstacles can be detected through the target sensor. When an obstacle is recognized within a preset angle range, the data obtained by the target sensor can be used for feature extraction to determine the contour of the obstacle and then calculate the target size of the obstacle. Specifically, the environmental data of the surrounding environment can be obtained by using a pose sensor (such as a lidar), the environmental data can be segmented into point clouds, key features such as obstacles and the ground can be recognized, and then the segmented point cloud data can be clustered to form "clusters" of obstacles, so that the size of the obstacles can be further analyzed; or a depth image of the current environment can be obtained through a vision sensor (such as a depth camera), the distance data of the obstacle can be determined by calculating the depth of field of the depth image, and then the distance data can be recognized and the size of the obstacle can be estimated through a pre-trained neural network model. Among them, more accurate environmental data can be obtained by combining the data of multiple sensors. For example, the sensor data obtained by both the pose sensor and the vision sensor can be imported into a corrector or a particle corrector for multi-source data fusion analysis to determine the size of the obstacle. Then, the target size of the obstacle is compared with the preset size range. If the target size of the obstacle exceeds the preset size range, it indicates that the obstacle may interfere with the movement path of the assistance robot and affect the safety of the assistance. Therefore, it is necessary to control the assistance robot to avoid the obstacle.

[0190] In a possible implementation, the second distance between the assistance robot and the obstacle can be detected through the target sensors such as the pose sensor and / or the vision sensor set on the assistance robot. When the second distance is less than or equal to the preset distance threshold, it can be considered that the obstacle may affect the movement path of the assistance robot and the target object. Therefore, the assistance robot can be controlled to avoid the obstacle, and at the same time, the manipulator of the assistance robot can be controlled by compliant control to provide information feedback (such as applying external force, making sound prompts, etc.) to the target object to guide the target object to avoid the obstacle.

[0191] In a possible implementation, referring to Figure 13 , Figure 13Schematic diagram of obstacle avoidance for the assisting robot provided by the embodiment of this application. When it is considered that there are obstacles affecting the movement path of the assisting robot, a virtual space can be created, and the sensor data sensed by the target sensor of the assisting robot can be mapped into the virtual space, so that the obstacle position of the obstacle and the end position of the movement path of the assisting robot can be represented in the virtual space. Among them, both the obstacle and the key position can be modeled as spheres. Based on the obstacle position and the obstacle mass corresponding to the obstacle, the first force vector of the obstacle, that is, the repulsive force of the obstacle, can be determined. The first force vector is used to guide the assisting robot to move in a direction away from the obstacle position. Then, based on the end position and the corresponding preset mass, the second force vector of the end position is determined. The second force vector is used to guide the assisting robot to move towards the end position. Among them, the first force vector decreases as the second distance increases, and the second force vector decreases as the distance between the current position of the assisting robot and the end position increases. Then, the first force vector and the second force vector are synthesized to obtain the target force vector. Specifically, the two vectors can be added or weighted added to obtain the target force vector. According to the target force vector, the speed and direction of the assisting robot can be controlled, so that the assisting robot can avoid obstacles and move towards the set end position at the same time.

[0192] It should be noted that when the second distance between multiple obstacles and the assisting robot is less than or equal to the preset distance threshold, multiple obstacles and their corresponding obstacle positions can be determined in the virtual space, and the first force vector corresponding to each obstacle can be obtained respectively. Then, all the first force vectors and the second force vector can be synthesized to obtain the target force vector.

[0193] In a possible implementation manner, during the process of assisting the target object, the target sensor can also be used to detect the changes in the environment around the assisting robot in real time to respond to emergencies such as environmental changes and ensure the safety of users. For example, when the assisting robot assists the target object to cross the zebra crossing, the assisting robot can sense and identify dynamic obstacles such as vehicles and pedestrians and static obstacles such as road shoulders and flower beds in the surrounding environment, and guide the target object to adjust the walking speed, pause walking or change the walking path to avoid potential collisions.

[0194] In a possible implementation, the tactile sensor includes a plurality of tactile units, which can be evenly distributed on the surface of the robotic arm. The tactile sensor can collect external force data in real time. Based on the detection principles of different tactile sensors, it can be determined whether the tactile sensor is triggered through resistance change, capacitance change, piezoelectric effect, etc. When an external force is applied to the tactile sensor, the electrical signals of the tactile units will change, and these signals can be converted into recognizable external force data. At the same time, it can be analyzed which tactile units are activated, so as to determine the activation quantity of the tactile units. Specifically, the sensitivity or activation threshold can be set to determine whether a tactile unit is in an activated state. When the applied external force exceeds the sensitivity or activation threshold of the tactile unit, it can be considered that the tactile unit is in an activated state. Among them, when an external force is applied to the tactile sensor, the external force data can be continuously obtained through the tactile sensor. When the detected external force data indicates that the currently received external force is greater than or equal to the external force threshold and the activation quantity is greater than the preset quantity threshold, it can be considered that the current target object is performing a large-area force interaction behavior with the robotic arm. Therefore, it can be considered that the target object is currently being supported by the support robot, or it can be considered that the target object has a sudden situation such as unstable center of gravity. Therefore, the support robot can be compliant-controlled to adjust the target joint position and target joint acceleration of the robotic arm to conform to the posture of the target object.

[0195] In a possible implementation, when the external force data indicates that the external force received by the tactile sensor is greater than or equal to the external force threshold and the activation quantity is less than the preset quantity threshold, it can be considered that the target object is interacting with the robotic arm, but at this time the stability of the support is low. If the current posture of the support robot is changed, it is easy for the target object to lose support balance. Therefore, the support robot can be controlled to move to the target movement position or the robotic arm can be controlled to move to the target joint position according to the pre-set control strategy, without performing compliant control on the support robot.

[0196] In a possible implementation, when the external force data indicates that the change value of the external force received by the tactile sensor is greater than or equal to the first external force change threshold within the first preset time period and less than the second external force change threshold within the second preset time period after the first preset time period, it can be considered that it is determined that the target object has performed a force interaction behavior with the robotic arm, and the current support posture of the support robot and the supported posture of the target object are stable, and the support robot can be compliant-controlled.

[0197] The control method of the support robot provided by the embodiments of the present application will be described in detail below.

[0198] Refer to Figure 14 , Figure 14An optional overall flowchart of the control method provided by the embodiments of this application. Among them, the control method can be executed by a terminal, and the control method includes but is not limited to the following steps 1401 to 1410:

[0199] Step 1401: In response to recognizing that the target object has an intention to need assistance, obtain the first pose data and the first covariance matrix predicted for the pose sensor at the previous moment.

[0200] Step 1402: Predict the current second pose data based on the first pose data, and predict the current second covariance matrix based on the first covariance matrix.

[0201] Step 1403: Obtain the measured pose data currently collected by the pose sensor, and determine the target gain based on the measured pose data and the second covariance matrix.

[0202] Step 1404: Correct the second pose data based on the target gain to obtain the current target pose data of the pose sensor.

[0203] Step 1405: Transform the target pose data based on a preset first transformation matrix to obtain the target movement position.

[0204] Step 1406: Control the assisting robot to move towards the target object based on the target movement position.

[0205] Step 1407: When the assisting robot moves to the target movement position, obtain the current image data of the vision sensor, and determine the key point positions of multiple key points of the target object in the image data.

[0206] Step 1408: Convert the coordinate system where the key point positions are located to the coordinate system where the assisting robot is located, and determine the center point position of multiple key points based on multiple transformed key point positions.

[0207] Step 1409: Transform the center point position based on a preset second transformation matrix to obtain the target joint position, and control the movement of the robotic arm based on the target joint position.

[0208] In this step, both the target movement position and the target joint position change following the relative position between the target object and the target sensor;

[0209] Step 1410: During the process of the robotic arm moving to the target joint position or after the robotic arm moves to the target joint position, when an external force is applied to the tactile sensor, perform compliant control on the assisting robot.

[0210] Refer to Figure 15 , Figure 15It is an optional overall flowchart of the control method provided by the embodiments of the present application. Among them, the control method can be executed by a terminal, and the control method includes but is not limited to the following steps 1501 to 1512:

[0211] Step 1501: In response to recognizing that the target object has the intention of needing assistance, control the assisting robot to move towards the target object.

[0212] Step 1502: When the assisting robot moves to the target moving position, control the robotic arm to move to the target joint position.

[0213] In this step, both the target moving position and the target joint position change following the relative position between the target object and the target sensor.

[0214] Step 1503: During the process of the robotic arm moving to the target joint position or after the robotic arm moves to the target joint position, when an external force is applied to the tactile sensor, determine the mass coefficient according to the mass of the assisting robot, the joint position of the assisting robot, and the joint speed of the assisting robot.

[0215] Step 1504: Determine the friction coefficient according to the friction force when the assisting robot moves and the joint position of the assisting robot.

[0216] Step 1505: Determine the gravity coefficient according to the gravitational acceleration of the assisting robot and the joint position of the assisting robot.

[0217] Step 1506: Determine the first product between the mass coefficient and the joint acceleration of the assisting robot, and perform kinematic modeling on the assisting robot according to the sum of the first product, the friction coefficient, and the gravity coefficient to obtain the dynamic model of the assisting robot.

[0218] In this step, the dynamic model is used to indicate the relationship between the joint torque of the assisting robot and the joint acceleration of the assisting robot.

[0219] Step 1507: Obtain the current external force data of the tactile sensor.

[0220] Step 1508: Determine the third transformation matrix for converting the coordinate system where the assisting robot is located to the coordinate system where the joints of the assisting robot are located.

[0221] Step 1509: Based on the first distance corresponding to the tactile unit, determine the fourth transformation matrix for converting the coordinate system where the joints of the assisting robot are located to the coordinate system where the tactile unit is located.

[0222] In this step, multiple tactile units are distributed cylindrically on the robotic arm, and the first distance between any two adjacent tactile units is equal.

[0223] Step 1510: Determine the conversion function between the coordinate system where each tactile unit is located and the coordinate system where the assisting robot is located according to the third transformation matrix and the fourth transformation matrix, and determine the differential of the conversion function with respect to the joint positions of the assisting robot to obtain the Jacobian matrix corresponding to each tactile unit.

[0224] Step 1511: Determine the second product between the external force data collected by each tactile unit and the corresponding Jacobian matrix, and obtain the target joint torque based on the sum of multiple second products.

[0225] Step 1512: Determine the target joint acceleration of the assisting robot according to the target joint torque and the dynamic model, and perform compliant control on the assisting robot based on the target joint acceleration.

[0226] Refer to Figure 16 , Figure 16 which is an optional overall flowchart of the control method provided by the embodiment of the present application. Among them, the control method can be executed by a terminal, and the control method includes but is not limited to the following steps 1601 to step 1609:

[0227] Step 1601: In response to recognizing that the target object has an intention to be assisted, control the assisting robot to move towards the target object.

[0228] Step 1602: During the process of the assisting robot following the target object, when an obstacle is recognized within a preset angle range, detect the target size of the obstacle.

[0229] Step 1603: When the target size is outside the preset size range, detect the second distance between the assisting robot and the obstacle.

[0230] Step 1604: When the second distance is less than or equal to the preset distance threshold, construct the virtual space where the assisting robot is located.

[0231] Step 1605: Determine the obstacle position of the obstacle and the set end position of the assisting robot in the virtual space.

[0232] Step 1606: Determine the first force vector of the obstacle according to the obstacle position and the obstacle mass of the obstacle, and determine the second force vector of the end position according to the end position and the preset mass of the end position.

[0233] Step 1607: Synthesize the first force vector and the second force vector to obtain the target force vector, and control the assisting robot to avoid the obstacle according to the target force vector.

[0234] Step 1608: When the assisting robot moves to the target moving position, control the manipulator to move to the target joint position.

[0235] In this step, both the target moving position and the target joint position change following the relative position between the target object and the target sensor.

[0236] Step 1609: During the process of the robotic arm moving to the target joint position or after the robotic arm moves to the target joint position, when an external force is applied to the tactile sensor, perform compliant control on the assisting robot.

[0237] It can be understood that although the steps in each of the above flowcharts are shown in sequence according to the arrows, these steps are not necessarily executed in the order indicated by the arrows. Unless there is a clear description in this embodiment, the execution of these steps has no strict order limit, and these steps can be executed in other orders. Moreover, at least a part of the steps in the above flowcharts may include multiple steps or multiple stages. These steps or stages are not necessarily executed at the same time, but can be executed at different times. The execution order of these steps or stages is not necessarily sequential, but can be executed alternately or in turn with at least a part of other steps or steps or stages in other steps.

[0238] Refer to Figure 17 , Figure 17 FIG. is an optional structural schematic diagram of the assisting robot provided by an embodiment of the present application. The assisting robot 1700 includes:

[0239] A robotic arm 1701, provided with a tactile sensor;

[0240] A target sensor 1702, used for object perception;

[0241] A control module 1703, configured to control the assisting robot 1700 to move towards the target object in response to identifying that the target object has an intention to be assisted. When the assisting robot 1700 moves to the target moving position, control the robotic arm 1701 to move to the target joint position, and during the process of the robotic arm 1701 moving to the target joint position or after the robotic arm 1701 moves to the target joint position, when an external force is applied to the tactile sensor, perform compliant control on the assisting robot 1700; wherein, both the target moving position and the target joint position change following the relative position between the target object and the target sensor 1702.

[0242] In a possible implementation manner, the control module 1703 is further configured to:

[0243] Obtain the current target pose data of the pose sensor, perform transformation on the target pose data based on a preset first transformation matrix to obtain the target moving position;

[0244] Based on the target moving position, control the assisting robot 1700 to move towards the target object.

[0245] In a possible implementation, the control module 1703 is further configured to:

[0246] Obtain the first pose data and the first covariance matrix predicted for the pose sensor at the previous moment, predict the current second pose data based on the first pose data, and predict the current second covariance matrix based on the first covariance matrix;

[0247] Obtain the measured pose data currently collected by the pose sensor, and determine the target gain based on the measured pose data and the second covariance matrix;

[0248] Correct the second pose data based on the target gain to obtain the current target pose data of the pose sensor.

[0249] In a possible implementation, the control module 1703 is further configured to:

[0250] Obtain the current image data of the vision sensor, and determine the key point positions of multiple key points of the target object in the image data;

[0251] Convert the coordinate system where the key point positions are located to the coordinate system where the assisting robot 1700 is located, and based on the multiple converted key point positions, determine the center point position of the multiple key points;

[0252] Transform the center point position based on a preset second transformation matrix to obtain the target joint position, and control the manipulator 1701 to act based on the target joint position.

[0253] In a possible implementation, the control module 1703 is further configured to:

[0254] Calibrate the parameters of the vision sensor to obtain the internal parameter matrix of the vision sensor;

[0255] Determine the installation position of the vision sensor in the assisting robot 1700, and determine the external parameter matrix of the vision sensor according to the installation position;

[0256] Based on the internal parameter matrix and the external parameter matrix, convert the coordinate system where the key point positions are located to the coordinate system where the target moving position is located.

[0257] In a possible implementation, the control module 1703 is further configured to:

[0258] Perform pose recognition on the target object according to each key point, where the result of the pose recognition includes the intention that the target object needs assistance or the intention that the target object does not need assistance.

[0259] In a possible implementation, the control module 1703 is further configured to:

[0260] Perform kinematic modeling on the assisting robot 1700 to obtain the dynamic model of the assisting robot 1700, where the dynamic model is used to indicate the relationship between the joint torques of the assisting robot 1700 and the joint accelerations of the assisting robot 1700;

[0261] Obtain the current external force data of the tactile sensor, and perform conversion on the external force data to obtain the target joint torque;

[0262] Determine the target joint acceleration of the assisting robot 1700 according to the target joint torque and the dynamic model, and perform compliant control on the assisting robot 1700 based on the target joint acceleration.

[0263] In a possible implementation, the control module 1703 is further configured to:

[0264] Determine the mass coefficient according to the mass of the assisting robot 1700, the joint positions of the assisting robot 1700, and the joint velocities of the assisting robot 1700;

[0265] Determine the friction coefficient according to the friction force when the assisting robot 1700 moves and the joint positions of the assisting robot 1700;

[0266] Determine the gravity coefficient according to the gravitational acceleration of the assisting robot 1700 and the joint positions of the assisting robot 1700;

[0267] Determine the first product between the mass coefficient and the joint acceleration of the assisting robot 1700, and perform kinematic modeling on the assisting robot 1700 according to the sum of the first product, the friction coefficient, and the gravity coefficient to obtain the dynamic model of the assisting robot 1700.

[0268] In a possible implementation, the control module 1703 is further configured to:

[0269] Determine the third transformation matrix for converting the coordinate system where the assisting robot 1700 is located to the coordinate system where the joints of the assisting robot 1700 are located, and determine the fourth transformation matrix for converting the coordinate system where the joints of the assisting robot 1700 are located to the coordinate system where the tactile unit is located;

[0270] According to the third transformation matrix and the fourth transformation matrix, determine the transformation function between the coordinate systems where each tactile unit is located and the coordinate system where the assisting robot 1700 is located, and determine the differential of the transformation function with respect to the joint positions of the assisting robot 1700 to obtain the Jacobian matrix corresponding to each tactile unit;

[0271] Determine the second product between the external force data collected by each tactile unit and the corresponding Jacobian matrix, and obtain the target joint torque based on the sum of multiple second products.

[0272] In a possible implementation, multiple tactile units are cylindrically distributed on the robotic arm 1701, and the first distance between any two adjacent tactile units is equal. The control module 1703 is further configured to:

[0273] Based on the first distance corresponding to the tactile unit, determine a fourth transformation matrix for converting the coordinate system where the joint of the assisting robot 1700 is located to the coordinate system where the tactile unit is located.

[0274] In a possible implementation, the control module 1703 is further configured to:

[0275] During the process of the assisting robot 1700 following the target object, when an obstacle is recognized within a preset angular range, detect the target size of the obstacle;

[0276] When the target size is outside the preset size range, control the assisting robot 1700 to avoid the obstacle.

[0277] In a possible implementation, the control module 1703 is further configured to:

[0278] Detect the second distance between the assisting robot 1700 and the obstacle. When the second distance is less than or equal to a preset distance threshold, construct a virtual space where the assisting robot 1700 is located;

[0279] Determine the obstacle position of the obstacle and the set end position of the assisting robot 1700 in the virtual space;

[0280] Determine the first force vector of the obstacle according to the obstacle position and the obstacle mass of the obstacle, and determine the second force vector of the end position according to the end position and the preset mass of the end position;

[0281] Synthesize the first force vector and the second force vector to obtain a target force vector, and control the assisting robot 1700 to avoid the obstacle according to the target force vector.

[0282] In a possible implementation, the tactile sensor is provided with multiple tactile units, and the control module 1703 is further configured to:

[0283] When an external force is applied to the tactile sensor, obtain the current external force data of the tactile sensor and determine the activation number of the tactile unit;

[0284] When the external force data indicates that the external force received by the tactile sensor is greater than or equal to the external force threshold, and the activation number is greater than or equal to the preset number threshold, perform compliant control on the assisting robot 1700.

[0285] In a possible implementation, with reference to Figure 18 , Figure 18 is an alternative structural schematic diagram of the assisting robot 1700 provided by the embodiments of the present application. As Figure 18 shown, the assisting robot 1700 includes a main torso 1801. The robotic arm 1701 includes a first sub-arm 1802 and a second sub-arm 1803. One end of the first sub-arm 1802 is movably connected to the main torso 1801, and the other end of the first sub-arm 1802 is movably connected to the second sub-arm 1803. Tactile sensors are provided on both the first sub-arm 1802 and the second sub-arm 1803. Equivalently, the first sub-arm 1802 can move relative to the main torso 1801, and the second sub-arm 1803 can move relative to the first sub-arm 1802. The first sub-arm 1802 can serve as the main part of the robotic arm 1701 and can bear greater force relative to the second sub-arm 1803, facilitating assisting and supporting the user; while the second sub-arm 1803 acts as an extension of the first sub-arm 1802 and can perform more complex and delicate actions relative to the first sub-arm 1802, such as fine-tuning the assisting force or direction. In addition, the link length of the second sub-arm 1803 can be less than that of the first sub-arm 1802, thereby reducing the motion inertia of the second sub-arm 1803 to achieve more accurate motion control. Therefore, through the multi-stage movable connection of the first sub-arm 1802 and the second sub-arm 1803, a wider range of motion of the robotic arm 1701 can be provided, better imitating the functions of the human limbs, assisting the target object to move, and providing assisting support and protection. It should be noted that the tactile sensors can be respectively arranged on the first sub-arm 1802 and the second sub-arm 1803, so as to obtain more comprehensive external force data, and further more accurately identify the posture of the target object to perform compliant control on the assisting robot 1700.

[0286] In a possible implementation, a first joint component 1804 and a second joint component 1805 are respectively connected to both ends of the first sub-arm 1802. The first joint component 1804 is connected to a third joint component 1806 through a first connecting shaft, and the third joint component 1806 is connected to the main torso 1801 through a second connecting shaft; one end of the second sub-arm 1803 is connected to the second joint component 1805 through a third connecting shaft, the other end of the second sub-arm 1803 is connected to a fourth joint component 1807 through a fourth connecting shaft, the fourth joint component 1807 is connected to a fifth joint component 1808 through a fifth connecting shaft, the fifth joint component 1808 is connected to a sixth joint component 1809 through a sixth connecting shaft, and the sixth joint component 1809 is connected to an end effector 1810. As Figure 18As shown, the robotic arm 1701 of the assisting robot 1700 may include six joint components to achieve six degrees of freedom. Among them, the third joint component 1806 can rotate relative to the main torso 1801 around the second connecting axis, and the first joint component 1804 can rotate relative to the third joint component 1806 around the first connecting axis, so that the first sub-arm 1802 and the second joint component 1805 can move synchronously relative to the third joint component 1806. The second sub-arm 1803 can rotate relative to the second joint component 1805 around the third connecting axis, the fourth joint component 1807 can rotate relative to the second sub-arm 1803 around the fourth connecting axis, the fifth joint component 1808 can rotate relative to the fourth joint component 1807 around the fifth connecting axis, and the sixth joint component 1809 can rotate relative to the fifth joint component 1808 around the sixth connecting axis.

[0287] In a possible implementation, the sixth joint component 1809 is connected to an end effector 1810, and the end effector 1810 is detachably mounted on the sixth joint component 1809. The end effector 1810 can be a robotic gripper, which can facilitate helping the user transfer items, such as transferring heavy objects, backpacks, crutches, etc.; the end effector 1810 can also be a safety restraint device, such as a seat belt, which can improve the stability and safety during the movement of assisting the target object; the end effector 1810 can also be a support handle or handrail, which is convenient for the target object to grip to provide stable support for the target object; the end effector 1810 can also be a task controller for setting the assisting task, and the task controller can be communicatively connected to the control module 1703. The control module 1703 can control the assisting robot 1700 based on the assisting task triggered and set by the task controller. For example, the original assisting task is to assist the target object to walk. When reaching the position designated by the target object (such as next to a seat), the task controller of the end effector 1810 can be triggered to reset the assisting task to assist the target object to sit down, so that the control module 1703 can change the assisting task and control the assisting robot 1700 to assist the target object to sit on the seat.

[0288] In addition, the assisting robot 1700 can be provided with multiple robotic arms 1701, such as Figure 18 As shown, the assisting robot 1700 can be provided with two robotic arms 1701. And in order to reduce the collision between multiple robotic arms 1701 and achieve adjustments in different directions, the robotic arms 1701 can be respectively installed on both sides of the main torso 1801.

[0289] Such as Figure 18As shown, the assisting robot 1700 further includes a vision sensor 1811, a pose sensor, and a base 1812 for movement. The vision sensor 1811 can be installed on the front side of the main torso 1801, and the vision sensor 1811 can be located between the two robotic arms 1701, so as to obtain a wider field of view angle to capture image data for identifying human postures. Among them, the pose sensor can be installed on the front side of the base 1812 to sense the current environment situation of the assisting robot 1700, such as the position of the target object and the position of obstacles. When there are multiple pose sensors, they can be respectively arranged on the peripheral side walls of the base 1812 to increase the sensing range and obtain more accurate environmental data.

[0290] As Figure 19 shown, Figure 19 is a schematic structural diagram of another perspective of the assisting robot 1700 provided by the embodiment of the present application. In the base 1812, a pose sensor, a control cabinet 1901, an antenna 1902, a base display screen 1905 for interactively displaying the state of the base 1812, and a control display screen 1906 for interactively displaying the state of the robotic arm 1701 can be provided. Among them, both the base display screen 1905 and the control display screen 1906 are installed on the outer side of the control cabinet 1901, which is convenient for interacting with the user. The pose sensor can include a 3D lidar 1903 for detecting obstacles and a 2D lidar 1904 for detecting the position of the target object and the position of obstacles. The 2D lidar 1904 can be installed on the side wall of the base 1812, the 3D lidar 1903 can be installed above the control cabinet 1901, and the antenna 1902 can be installed on one side of the base 1812 close to the control cabinet 1901. As Figure 19 shown, the base 1812 can be a Mecanum mobile platform, and the joints of the assisting robot 1700 in the lower limb part can be an omnidirectional wheel 1813 and a movable connection mechanism of the base 1812. Among them, the main torso 1801 can be located in front of the control cabinet 1901. Due to the weight influence of the robotic arm 1701 and the main torso 1801, the center of gravity of the assisting robot 1700 moves forward. Therefore, the control cabinet 1901 can be placed behind the main torso 1801 to add weights to the base 1812 to balance the center of gravity position of the assisting robot 1700 and improve the movement stability of the assisting robot 1700.

[0291] As Figure 20 shown, Figure 20Schematic diagram of the internal structure of the control cabinet 1901 of the assisting robot 1700 provided by the embodiment of the present application. The assisting robot 1700 further includes a switch 2001 respectively connected to the antenna 1902 and the control module 1703, a storage battery 2002 for power supply, and an inverter 2003 for converting the voltage of the storage battery 2002 for power supply. The control module 1703, the switch 2001, the storage battery 2002, and the inverter 2003 are installed in the control cabinet 1901. The control module 1703 can perform data interaction with the server through the switch 2001 and the antenna 1902. For example, the sensor data can be uploaded to the server, or the updated algorithm program can be downloaded from the server.

[0292] As Figure 21 shown, Figure 21 Schematic diagram of the structural connection of the assisting robot 1700 provided by the embodiment of the present application. The storage battery 2002 is connected to the inverter 2003, so as to convert the 48V voltage of the storage battery 2002 into 221V AC voltage to supply power to the electrical load of the assisting robot 1700. The control module 1703 can be respectively communicatively connected to the tactile sensor, the pose sensor (including the 3D lidar 1903 and the 2D lidar 1904), the vision sensor 1811, the robotic arm control cabinet 2101, the base display screen 1905, and the control display screen 1906. Thus, the control module 1703 can obtain the data of each sensor and the interaction data of each display screen. The control module 1703 can generate a control instruction based on the obtained data and send it to the robotic arm control cabinet 2101, and then control the robotic arm 1701 and the base 1812 through the robotic arm control cabinet 2101. Specifically, the control module 1703 can establish connections with each component by using different data transmission methods. For example, the control module 1703 can transmit data with the robotic arm 1701 and the base 1812 through the TCP / IP protocol, can also transmit data with the tactile sensor and the vision sensor 1811 through the USB3.0 transmission protocol, and can also transmit data with the pose sensor, the base display screen 1905, and the pose display screen through the HDMI transmission protocol.

[0293] The control module 1703 can be an industrial control computer, which is used to execute the control methods of the foregoing embodiments. By responding to the recognition that the target object has the intention of needing assistance, it controls the assistance robot to move towards the target object. When the assistance robot moves to the target moving position, it controls the robotic arm to move to the target joint position. Among them, both the target moving position and the target joint position change following the relative position between the target object and the target sensor, so as to be able to sense the state of the target object and automatically execute the assistance task according to the state of the target object. On this basis, during the process of the robotic arm moving to the target joint position or after the robotic arm moves to the target joint position, when an external force is applied to the tactile sensor, compliant control is performed on the robotic arm, so that the robotic arm can conform to the posture of the target object, thereby improving the comfort of assistance. It can be seen that the control method provided by this application can interact with the target object in a diversified manner and has a relatively high degree of intelligence.

[0294] An embodiment of this application also provides an electronic device, including a memory and a processor. The memory stores a computer program, and when the processor executes the computer program, it implements the control methods of the foregoing embodiments.

[0295] An embodiment of this application also provides a computer-readable storage medium, which is used to store a computer program, and the computer program is used to execute the control methods of the foregoing embodiments.

[0296] An embodiment of this application also provides a computer program product, which includes a computer program stored in a computer-readable storage medium. The processor of the computer device reads the computer program from the computer-readable storage medium, and the processor executes the computer program, so that the computer device executes and implements the above control method.

[0297] The terms "first", "second", "third", "fourth", etc. (if any) in the specification of this application and the above drawings are used to distinguish similar objects and do not necessarily need to describe a specific order or sequence. It should be understood that such data can be interchanged under appropriate circumstances to describe the embodiments of this application, for example, it can be implemented in an order other than those illustrated or described here. In addition, the terms "including" and "having" and any variations thereof are intended to cover non-exclusive inclusion. For example, a process, method, system, product or device that includes a series of steps or units does not necessarily have to be limited to those steps or units clearly listed, but may include other steps or units not clearly listed or inherent to these processes, methods, products or devices.

[0298] It should be understood that in this application, "at least one (item)" means one or more, and "a plurality" means two or more. "And / or" is used to describe the association relationship of associated objects, indicating that there can be three relationships. For example, "A and / or B" can mean: only A exists, only B exists, and both A and B exist at the same time. Among them, A and B can be singular or plural. The character " / " generally indicates that the associated objects before and after are in an "or" relationship. "At least one (one) of the following" or similar expressions refer to any combination of these items, including any combination of single item (one) or plural items (ones). For example, at least one (one) of a, b, or c can mean: a, b, c, "a and b", "a and c", "b and c", or "a and b and c", where a, b, and c can be single or multiple.

[0299] It should be understood that in the description of the embodiments of this application, the meaning of "a plurality (or multiple items)" is more than two. Understandings such as greater than, less than, exceeding, etc. do not include the present number, and understandings such as above, below, within, etc. include the present number.

[0300] In several embodiments provided in this application, it should be understood that the disclosed systems, devices, and methods can be implemented in other ways. For example, the device embodiments described above are only illustrative. For example, the division of units is only a logical function division. In actual implementation, there can be other division methods. For example, multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the displayed or discussed coupling or direct coupling or communication connection to each other can be through some interfaces, and the indirect coupling or communication connection of devices or units can be in electrical, mechanical, or other forms.

[0301] The units described as separate components may or may not be physically separated, and the components displayed as units may or may not be physical units, that is, they can be located in one place, or distributed to multiple network units. Some or all of the units can be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0302] In addition, the functional units in each embodiment of this application can be integrated into one processing unit, or each unit can exist physically alone, or two or more units can be integrated into one unit. The above-mentioned integrated units can be implemented in the form of hardware or in the form of software functional units.

[0303] If an integrated unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, or all or part of this technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to enable a computer device (which can be a personal computer, a server, or a network device, etc.) to execute all or part of the steps of the methods of various embodiments of this application. The aforementioned storage medium includes: various media that can store program codes, such as USB flash drives, mobile hard disks, read-only memories (ROM for short), random access memories (RAM for short), magnetic disks, or optical discs.

[0304] It should also be understood that the various embodiments provided in the embodiments of this application can be combined arbitrarily to achieve different technical effects.

[0305] The above is a specific description of the preferred embodiments of this application, but this application is not limited to the above embodiments. Those skilled in the art can also make various equivalent deformations or substitutions without violating the spirit of this application. These equivalent deformations or substitutions are all included within the scope defined by the claims of this application.

Claims

1. A control method for a helping robot, characterized in that, The assisting robot is provided with a robotic arm and a target sensor for object perception. A tactile sensor is provided on the robotic arm. The tactile sensor is provided with a plurality of tactile units. The control method includes: In response to recognizing that the target object has an intention of needing assistance, controlling the assisting robot to move towards the target object. When the assisting robot moves to the target moving position, controlling the robotic arm to move to the target joint position, where both the target moving position and the target joint position change following the relative position between the target object and the target sensor; During the process of the robotic arm moving to the target joint position or after the robotic arm moves to the target joint position, when an external force is applied to the tactile sensor, performing kinematic modeling on the assisting robot to obtain the dynamic model of the assisting robot, acquiring the current external force data of the tactile sensor, determining a third transformation matrix for converting the coordinate system where the assisting robot is located to the coordinate system where the joints of the assisting robot are located, and determining a fourth transformation matrix for converting the coordinate system where the joints of the assisting robot are located to the coordinate system where the tactile unit is located; According to the third transformation matrix and the fourth transformation matrix, determining the transformation function between the coordinate system where each tactile unit is located and the coordinate system where the assisting robot is located, determining the partial derivative of the transformation function with respect to the joint position of the assisting robot to obtain the Jacobian matrix corresponding to each tactile unit; determining the second product between the external force data collected by each tactile unit and the corresponding Jacobian matrix, obtaining the target joint torque based on the sum of multiple second products, determining the target joint acceleration of the assisting robot according to the target joint torque and the dynamic model of the assisting robot, and performing compliant control on the assisting robot based on the target joint acceleration, where the dynamic model is used to indicate the relationship between the joint torque of the assisting robot and the joint acceleration of the assisting robot.

2. The control method according to claim 1, characterized in that The types of the target sensor include a vision sensor and a pose sensor. The target moving position changes following the relative position between the target object and the pose sensor, and the target joint position changes following the relative position between the target object and the vision sensor.

3. The control method according to claim 2, characterized in that, The controlling the assisting robot to move towards the target object includes: Acquiring the current target pose data of the pose sensor, performing transformation on the target pose data based on a preset first transformation matrix to obtain the target moving position; Controlling the assisting robot to move towards the target object based on the target moving position.

4. The control method according to claim 3, wherein The acquiring the current target pose data of the pose sensor includes: Acquiring the first pose data and the first covariance matrix predicted for the pose sensor at the previous moment, predicting the current second pose data based on the first pose data, and predicting the current second covariance matrix based on the first covariance matrix; Obtain the measured pose data currently collected by the pose sensor, and determine the target gain based on the measured pose data and the second covariance matrix; Correct the second pose data based on the target gain to obtain the current target pose data of the pose sensor.

5. The control method according to claim 2, characterized in that The controlling the manipulator to move to the target joint position includes: Obtain the current image data of the vision sensor, and determine the key point positions of multiple key points of the target object in the image data; Convert the coordinate system where the key point positions are located to the coordinate system where the assisting robot is located, and based on the multiple converted key point positions, determine the central point position of the multiple key points; Transform the central point position based on a preset second transformation matrix to obtain the target joint position, and control the manipulator to move based on the target joint position.

6. The control method according to claim 5, wherein The converting the coordinate system where the key point positions are located to the coordinate system where the target moving position is located includes: Calibrate the parameters of the vision sensor to obtain the internal parameter matrix of the vision sensor; Determine the installation position of the vision sensor in the assisting robot, and determine the external parameter matrix of the vision sensor according to the installation position; Convert the coordinate system where the key point positions are located to the coordinate system where the target moving position is located based on the internal parameter matrix and the external parameter matrix.

7. The control method according to claim 5, wherein Before the controlling the assisting robot to move towards the target object in response to recognizing that the target object has an intention to be assisted, the control method further includes: Perform pose recognition on the target object according to each of the key points, where the result of the pose recognition includes that the target object has an intention to be assisted or the target object does not have an intention to be assisted.

8. The control method according to claim 1, wherein The performing kinematic modeling on the assisting robot to obtain the dynamic model of the assisting robot includes: Determine the mass coefficient according to the mass of the assisting robot, the joint positions of the assisting robot, and the joint velocities of the assisting robot; Determine the friction coefficient according to the friction force when the assisting robot moves and the joint positions of the assisting robot; Determine the gravity coefficient according to the gravitational acceleration of the assisting robot and the joint positions of the assisting robot; Determine the first product between the mass coefficient and the joint acceleration of the assisting robot, and perform kinematic modeling on the assisting robot according to the sum of the first product, the friction coefficient, and the gravity coefficient to obtain the dynamic model of the assisting robot.

9. The control method according to claim 1, wherein Multiple of the tactile units are distributed cylindrically on the manipulator, and the first distance between any two adjacent tactile units is equal. The determining the fourth transformation matrix for converting the coordinate system where the joint of the assisting robot is located to the coordinate system where the tactile unit is located includes: Based on the first distance corresponding to the tactile unit, determine the fourth transformation matrix for converting the coordinate system where the joint of the assisting robot is located to the coordinate system where the tactile unit is located.

10. The control method according to claim 1, characterized in that, The control method further includes: During the process of the assisting robot following the target object, when an obstacle is recognized within a preset angular range, detect the target size of the obstacle; When the target size is outside the preset size range, control the assisting robot to avoid the obstacle.

11. The control method according to claim 10, characterized in that, The controlling the assisting robot to avoid the obstacle includes: Detect the second distance between the assisting robot and the obstacle. When the second distance is less than or equal to a preset distance threshold, construct the virtual space where the assisting robot is located; Determine the obstacle position of the obstacle and the set end position of the assisting robot in the virtual space; Determine the first force vector of the obstacle according to the obstacle position and the obstacle mass of the obstacle, and determine the second force vector of the end position according to the end position and the preset mass of the end position; Synthesize the first force vector and the second force vector to obtain a target force vector, and control the assisting robot to avoid the obstacle according to the target force vector.

12. The control method according to claim 1, characterized in that, The kinematic modeling of the assisting robot when an external force is applied to the tactile sensor includes: When an external force is applied to the tactile sensor, obtain the current external force data of the tactile sensor and determine the activation number of the tactile unit; When the external force data indicates that the external force received by the tactile sensor is greater than or equal to an external force threshold and the activation number is greater than or equal to a preset number threshold, perform kinematic modeling on the assisting robot.

13. A helping robot, characterized in that, The assisting robot is provided with a control module, a robotic arm, and a target sensor for object perception. The robotic arm is provided with a tactile sensor, and the tactile sensor is provided with a plurality of tactile units: The control module is configured to control the assisting robot to move towards the target object in response to recognizing that the target object has an intention of needing assistance. When the assisting robot moves to the target moving position, it controls the robotic arm to move to the target joint position, and during the process of the robotic arm moving to the target joint position or after the robotic arm moves to the target joint position, when an external force is applied to the tactile sensor, kinematic modeling is performed on the assisting robot to obtain the dynamic model of the assisting robot, the current external force data of the tactile sensor is acquired, a third transformation matrix for converting the coordinate system where the assisting robot is located to the coordinate system where the joints of the assisting robot are located is determined, and a fourth transformation matrix for converting the coordinate system where the joints of the assisting robot are located to the coordinate system where the tactile unit is located is determined; according to the third transformation matrix and the fourth transformation matrix, a conversion function between the coordinate systems where each tactile unit is located and the coordinate system where the assisting robot is located is determined, the partial derivative of the conversion function with respect to the joint position of the assisting robot is determined to obtain the Jacobian matrix corresponding to each tactile unit; the second product between the external force data collected by each tactile unit and the corresponding Jacobian matrix is determined, the target joint torque is obtained based on the sum of multiple second products, the target joint acceleration of the assisting robot is determined according to the target joint torque and the dynamic model of the assisting robot, and compliant control is performed on the assisting robot based on the target joint acceleration; Wherein, the dynamic model is used to indicate the relationship between the joint torque of the assisting robot and the joint acceleration of the assisting robot, and both the target moving position and the target joint position change following the relative position between the target object and the target sensor.

14. The assisting robot according to claim 13, wherein: The assisting robot includes a main torso, the robotic arm includes a first sub-arm and a second sub-arm, one end of the first sub-arm is movably connected to the main torso, the other end of the first sub-arm is movably connected to the second sub-arm, and the tactile sensors are provided on both the first sub-arm and the second sub-arm.

15. The assisting robot according to claim 14, wherein: Both ends of the first sub-arm are respectively connected with a first joint component and a second joint component, the other end of the first joint component is connected with a third joint component, and the third joint component is connected to the main torso; One end of the second sub-arm is connected to the second joint component, the other end of the second sub-arm is connected with a fourth joint component, the fourth joint component is connected with a fifth joint component, the fifth joint component is connected with a sixth joint component, and the sixth joint component is connected with an end effector.

16. An electronic device, comprising a memory and a processor, the memory storing a computer program, characterized in that, When the processor executes the computer program, it implements the control method according to any one of claims 1 to 12.

17. A computer-readable storage medium storing a computer program, characterized in that, When the computer program is executed by the processor, it implements the control method according to any one of claims 1 to 12.

18. A computer program product, comprising a computer program, characterized in that, When the computer program is executed by a processor, it implements the control method described in any one of claims 1 to 12.

Images