Positioning method for robotic system, and robotic system

Abstract

The present application provides a positioning method for a robotic system, and a robotic system. The method comprises: acquiring a first spatial relationship between a calibration reference object and a first robot by means of a first detection system disposed on the first robot, and acquiring a second spatial relationship between the calibration reference object and a second robot by means of a second detection system disposed on the second robot; and then determining the spatial relationship between the first robot and the second robot on the basis of the first spatial relationship and the second spatial relationship, such that the relative positions of a plurality of split robots can be calibrated.

Description

Positioning method of robot system and robot system

[0001] Cross-reference to Related Applications

[0002] This application claims priority to Chinese Patent Application No. 202410877637.2, filed on July 01, 2024, and to Chinese Patent Application No. 202411866191.X, filed on December 16, 2024, the entire contents of which are incorporated herein by reference. TECHNICAL FIELD

[0003] The present application relates to the field of robot system, and in particular, to a positioning method of robot system and robot system. BACKGROUND

[0004] Surgical robots are increasingly used in hospitals. At the same time, in order to reduce the invasion of the operating space, and to facilitate flexible preoperative positioning, split-type surgical robot systems have gradually emerged. In the surgical application process of the split-type surgical robot, the mutual positions of different robots need to be calibrated to realize the cooperative operation and avoidance between multiple robots. SUMMARY

[0005] Therefore, the present application provides a positioning method of robot system and robot system.

[0006] Specifically, the present application is implemented by the following technical solutions:

[0007] According to a first aspect of an embodiment of the present application, a positioning method of a robot system is provided, the robot system comprising a first robot and a second robot, the first robot being provided with a first detection system, the second robot being provided with a second detection system, the method comprising:

[0008] acquiring, by the first detection system, a first spatial relationship between a calibration reference and the first robot, and acquiring, by the second detection system, a second spatial relationship between the calibration reference and the second robot, wherein the calibration reference is within the detection range of the first detection system and the second detection system;

[0009] determining a spatial relationship between the first robot and the second robot based on the first spatial relationship and the second spatial relationship.

[0010] According to a second aspect of an embodiment of the present application, a positioning method of a robot system is also provided, the robot system comprising a first robot and a second robot, the first robot and the second robot being respectively provided with a first positioning module, the positioning method comprising:

[0011] obtaining a first relative pose between the first positioning module on the first robot and the second positioning module when the second positioning module is in a first position, the second positioning module comprising an inertial sensor;

[0012] obtaining a second relative pose between the first positioning module on the second robot and the second positioning module when the second positioning module is in a second position;

[0013] determining a third relative pose of the second positioning module between the first position and the second position using the inertial sensor;

[0014] determining a relative pose between the first robot and the second robot based on the first relative pose, the second relative pose and the third relative pose.

[0015] According to a third aspect of embodiments of the present application, a positioning method of a robot system is also provided, the robot system comprising a first robot and a second robot, a first positioning module being arranged on the first robot, the second robot being detachably connected with a second positioning module, the second positioning module comprising an inertial sensor, the method comprising:

[0016] obtaining a first relative pose between the first positioning module on the first robot and the second positioning module when the second positioning module is in a first position;

[0017] determining a fourth relative pose of the second positioning module between the first position and a third position using the inertial sensor, the third position being a position where the second positioning module is detachably connected on the second robot;

[0018] determining a relative pose between the first robot and the second robot based on the first relative pose and the fourth relative pose.

[0019] According to a fourth aspect of embodiments of the present application, a robot system is provided, comprising a first robot, a second robot and a control device. The control device comprises a processor and a memory, the memory being configured to store processor-executable instructions, the processor being configured to implement the positioning method of any one of the first aspect, the second aspect or the third aspect.

[0020] The details of one or more embodiments of the application are set forth in the accompanying drawings and the description below. Other features, objects, and advantages of the application will be apparent from the description, drawings, and claims. BRIEF DESCRIPTION OF DRAWINGS

[0021] FIG. 1a is a schematic diagram of a robot system according to an example embodiment of the present application;

[0022] FIG. 1b is a schematic diagram of another robotic system according to an example embodiment of the present application;

[0023] FIG. 2 is a schematic diagram of a robotic system according to an example embodiment of the present application;

[0024] FIG. 3 is a schematic diagram of the structure of a robotic system according to an example embodiment of the present application;

[0025] FIG. 4 is a schematic diagram of a manner of disposing a calibration reference according to an example embodiment of the present application;

[0026] FIG. 5 is a schematic diagram of another manner of disposing a calibration reference according to an example embodiment of the present application;

[0027] FIG. 6 is a schematic diagram of the structure of a calibration reference according to an example embodiment of the present application;

[0028] FIG. 7 is a schematic diagram of the structure of another calibration reference according to an example embodiment of the present application;

[0029] FIG. 8 is a schematic diagram of the structure of another calibration reference according to an example embodiment of the present application;

[0030] FIG. 9 is a flowchart of a positioning method of a robotic system according to an example embodiment of the present application;

[0031] FIG. 10 is a schematic diagram of yet another robotic system according to an example embodiment of the present application;

[0032] FIG. 11 is a schematic diagram of the structure of yet another robotic system according to an example embodiment of the present application;

[0033] FIG. 12 is a schematic diagram of the structure of still another robotic system according to an example embodiment of the present application;

[0034] FIG. 13 is a schematic diagram of the structure of still another robotic system according to an example embodiment of the present application;

[0035] FIG. 14 is a schematic diagram of the structure of still another robotic system according to an example embodiment of the present application;

[0036] FIG. 15 is a flowchart of a positioning method of another robotic system according to an example embodiment of the present application;

[0037] FIG. 16 is a flowchart of a positioning method of yet another robotic system according to an example embodiment of the present application;

[0038] FIG. 17 is a schematic diagram of the structure of an electronic device according to an example embodiment of the present application. DETAILED DESCRIPTION

[0039] The scheme in the embodiments of the present application will be described in detail below with reference to the drawings in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, rather than all the embodiments. Based on the embodiments in the present application, all other embodiments obtained by a person of ordinary skill in the art without creative labor fall within the scope of protection of the present application.

[0040] It should be noted that all directional indications, such as upper, lower, left, right, front, back, etc., in the embodiments of the present application are only used to explain the relative positional relationship, movement condition, etc. between components in a certain posture (as shown in the drawings), and if the certain posture changes, the directional indications will also change accordingly.

[0041] It should also be noted that when an element is referred to as being "fixed to" or "disposed on" another element, it can be directly on the other element or can have a middle element present at the same time. When an element is referred to as being "connected" to another element, it can be directly connected to the other element or can have a middle element present at the same time.

[0042] In addition, the description involving "first", "second", etc. in the present application is only for the purpose of description, and cannot be understood as indicating or implying the relative importance of the indicated technical features or implicitly indicating the number of the indicated technical features. Therefore, the features defined as "first", "second" can explicitly or implicitly include at least one of the features. In addition, the technical solutions of each embodiment can be combined with each other, but it must be based on the fact that a person of ordinary skill in the art can realize it, and when the combination of technical solutions appears to be contradictory or unachievable, it should be considered that the combination of technical solutions does not exist, and is also not within the scope of protection claimed by the present application.

[0043] The robot in the present application can be a surgical robot. The application of surgical robots in hospitals is becoming more and more common. At the same time, in order to reduce the occupation of the operating space, and to make it more convenient for positioning before surgery, split robot systems have gradually emerged. The split robot is composed of several independent robots (also known as mechanical arm systems). Each robot has an independent base. Different robots are placed at different positions beside the operating bed according to the needs of the surgery. The split surgical robot can be deployed more flexibly in the operating room, and is more convenient for clinical application.

[0044] Please refer to FIG. 1a, FIG. 1b and FIG. 2-FIG. 8. The first embodiment of the present application provides a robot system 10.

[0045] FIG. 1a shows a schematic diagram of the robot system 10. As shown in FIG. 1a, the robot system 10 includes a console 101, a first robot 102, a second robot 103 and a control device 104.

[0046] The control console 101 comprises a display unit for displaying the environment of the surgical instrument, an operation control mechanism and an armrest. The display unit has an observation window (also referred to as a stereoscopic display) for the doctor to observe. The operation control mechanism is used to control the surgical instrument. The armrest is used to support the doctor's arm. In addition, the control console 101 comprises other control switches which are convenient for the hands or feet to touch or press, and are used to perform various function operations to complete the human-computer interaction.

[0047] The first robot 102 comprises a single or multiple first mechanical arms 1021. The first mechanical arm 1021 comprises a plurality of jointed arms. The adjacent two jointed arms are relatively movable in a specific degree of freedom, so that the end of the first mechanical arm 1021 can reach a multi-degree of freedom (for example, 7 degrees of freedom, and different degrees of freedom can be achieved according to different instruments) movement. The end of the first mechanical arm 1021 is provided with an instrument driver, a surgical instrument or an image acquisition device. The image acquisition device can be an endoscope (for example, a three-dimensional endoscope), which is detachably installed on the instrument driver.

[0048] The second robot 103 comprises a single or multiple second mechanical arms 1031. The structure of the second mechanical arm 1031 and the instrument installed at the end thereof can be the same as or different from the first mechanical arm 1021. The second robot 103 can be a mechanical arm system used to complete an auxiliary manipulation function, and can be used to cooperate with the first robot 102 to perform a surgical operation or an auxiliary operation.

[0049] The control device 104 comprises at least one memory and at least one processor, and is in communication connection with the control console 101, the first robot 102 and the second robot 103, so as to realize the control among the control console 101, the first robot 102 and the second robot 103. The control device 104 can execute a plurality of programmed instructions (for example, a computer readable medium storing instructions) by using the processor, so as to realize some or all of the methods described in the embodiments of the present application. In the physical aspect, the control device 104 can also be integrated in the control console 101, the first robot 102 or the second robot 103, or can be a separate computer device. The embodiments of the present application do not make specific limitation on the structure of the control device 104.

[0050] In addition, as shown in FIG. 1b, the robotic system 10 can further include an image system 105, which can include a display screen and can display images captured by an endoscope. The endoscope can capture images (e.g., a surgical scene) and send the captured images to the image system 105. The endoscope can perform image processing (e.g., decoding) on the captured images by its own processor or by the processor of the image system 105. The processed images can also be sent to other image processing devices for further image processing (e.g., noise reduction, contrast enhancement, clarity improvement, etc.) or to the console 101. It should be noted that in the case where the robotic system 10 includes the image system 105, the control device 104 can also implement control of the image system 105, which can be an image system on the control device 104 or an image system on another device.

[0051] In order to realize the cooperative operation and avoidance of multiple robots in the surgical application process of the split robot system, the mutual positions between the robots need to be calibrated. Based on this, the embodiment of the present application provides a positioning method of a robot system, which obtains a first spatial relationship between a calibration reference and a first robot through a first detection system arranged on the first robot, and obtains a second spatial relationship between the calibration reference and a second robot through a second detection system arranged on the second robot, and then determines the spatial relationship between the first robot and the second robot according to the first spatial relationship and the second spatial relationship, so as to realize the mutual position calibration between multiple split robots.

[0052] In some embodiments, referring to FIGS. 1a and 2, the robotic system 10 includes a first robot 102 and a second robot 103, the first robot 102 is provided with a first detection system 106a, and the second robot 103 is provided with a second detection system 106b.

[0053] The first detection system 106a can detect the calibration reference 107 within the detection range A of the first detection system 106a, and obtain the first spatial relationship between the calibration reference 107 and the first robot 102; the second detection system 106b can detect the calibration reference 107 within the detection range B of the second detection system 106b, and obtain the second spatial relationship between the calibration reference 107 and the second robot 103, and then the spatial relationship between the first robot 102 and the second robot 103 can be determined according to the first spatial relationship and the second spatial relationship.

[0054] The calibration reference object 107 can be set at a position independent of the first robot 102 and the second robot 103, for example, a user can hold the calibration reference object 107 and make the calibration reference object 107 simultaneously located in the detection range of the first detection system 106a and the second detection system 106b.

[0055] It should be noted that the first detection system 106a and the second detection system 106b can be optical positioning systems, and the calibration reference object 107 can be a reflective ball jig. Specifically, in order to enable the optical positioning system to detect the relative relationship between the three-dimensional coordinate system of the reflective ball jig and the three-dimensional coordinate system of the optical positioning system, the reflective ball jig is embedded with at least three reflective balls. The optical positioning system sends corresponding light sources to the reflective ball jig, and calculates the spatial relationship between the reflective ball jig and the optical positioning system according to the light sources reflected by the reflective balls. In addition, the first detection system 106a and the second detection system 106b can also be binocular cameras, and the calibration reference object 107 can be a component provided with a structured image. The binocular camera is composed of two cameras, and two images of the same scene in the detection range of the binocular camera are captured. Each image corresponds to a camera view angle. The three-dimensional position of the structured part in space is determined by measuring the pixel difference of the structured part in the two images, and the spatial relationship between the calibration reference object 107 and the robot is determined. The structured image can be a two-dimensional code image or other images with regular (such as rectangular, circular, etc.) patterns.

[0056] It should be noted that the first robot 102 and the second robot 103 can be single-arm robots or multi-arm robots. For example, as shown in FIG. 2, the first robot 102 is a multi-arm robot having multiple mechanical arms, and the second robot 103 is a single-arm robot having one mechanical arm. For another example, as shown in FIG. 4, the first robot 102 and the second robot 103 are single-arm robots, each having only one mechanical arm. According to actual surgical requirements, the first robot 102 and the second robot 103 can be robots with different numbers of mechanical arms, and the embodiments of the present application do not limit the number of robots in the robot system.

[0057] It should be noted that the number of robots in the robot system described in the above embodiments is not limited to two, and the number of robots in the robot system is not specifically limited in the present application. Each robot is provided with a corresponding detection system, and each detection system detects the calibration reference object in its own detection range to obtain the spatial relationship between the calibration reference object and the robot. In the present application, the first robot refers to any robot in the robot system that has the same function as the first robot, and the second robot refers to any robot in the robot system that has the same function as the second robot.

[0058] In some embodiments, the first detection system 106a can be arranged in a region of the first robot 102 that is not covered by the sterile drape, and / or the second detection system 106b can be arranged in a region of the second robot 103 that is not covered by the sterile drape.

[0059] It should be noted that the sterile drape is a protective cover, and in the clinical application of the robot system 10, the sterile drape needs to be used to cover the relevant region of the robot system 10 to prevent harmful substances such as bacteria and viruses from entering and contaminating the robot system 10.

[0060] In the embodiments of the present application, either one or both of the first detection system 106a and the second detection system 106b are arranged in a region of the robot that is not covered by the sterile drape, so that the detection system can accurately obtain the spatial relationship between the calibration reference 107 and the robot without being affected by the sterile drape.

[0061] In some embodiments, the region not covered by the sterile drape can be the base of the robot, or the column of the robot, so that the first detection system 106a and the second detection system 106b can be arranged on the base or the column according to the needs, and the two detection systems can be arranged on the base or the column, or both of them can be arranged on the base or the column (for reference to FIG. 4).

[0062] It should be noted that in the clinical application of the robot system 10, a sterile cover needs to be used to cover the relevant areas in the first robot 102 and the second robot 103. As shown in FIG. 3, the first robot 102 includes a first base 1023, a first column 1022 arranged on the first base 1023, and a plurality of first mechanical arms 1021 arranged on the first column 1022, and the second robot 103 includes a second base 1033, a second column 1032 arranged on the second base 1033, and a plurality of second mechanical arms 1031 arranged on the second column 1032. For the first robot 102, a sterile cover is used to cover the first mechanical arm 1021, or a sterile cover is used to cover the first mechanical arm 1021 and the first column 1022, and correspondingly, the area in the first robot 102 that is not covered by the sterile cover includes the first base 1023 and / or the first column 1022, and therefore, the first detection system 106a can be arranged on the first base 1023 or the first column 1022 of the first robot 102. For the second robot 103, a sterile cover is used to cover the second mechanical arm 1031, or a sterile cover is used to cover the second mechanical arm 1031 and the second column 1032, and correspondingly, the area in the second robot 103 that is not covered by the sterile cover includes the second base 1033 and / or the second column 1032, and therefore, the second detection system 106b can be arranged on the second base 1033 and the second column 1032 of the second robot 103.

[0063] In some embodiments, the first detection system 106a can be connected with the first robot 102 through a first connecting structure, and / or the second detection system 106b can be connected with the second robot 103 through a second connecting structure. The first connecting structure can be fixedly installed or detachably installed on the first robot 102, and the first connecting structure is a fixed structure or a structure including at least one joint and movable relative to the first robot 102. According to the fixed structure of the first connecting structure or the motion parameters thereof, in combination with the mounting position between the first connecting structure and the first robot 102, the spatial relationship between the first detection system 106a and the first robot 102 (such as the base of the first robot 102) can be obtained. The second connecting structure can be the same as the first connecting structure, or according to the fixed structure of the second connecting structure or the motion parameters thereof, in combination with the mounting position between the second connecting structure and the second robot 103, the spatial relationship between the second detection system 106b and the second robot 103 (such as the base of the second robot 103) can be obtained.

[0064] In some embodiments, the calibration reference 107 can also be arranged on the first robot 102 and / or the second robot 103.

[0065] It can be understood that the calibration reference 107 can be arranged on the first robot 102 or the second robot 103, and when there are multiple calibration references 107, the calibration reference 107 can be arranged on the first robot 102 and the second robot 103 at the same time. As shown in FIG. 4, the first detection system 106a is arranged on the first robot 102 and has a corresponding detection range A, and the second detection system 106b is arranged on the second robot 103 and has a corresponding detection range B. If the calibration reference 107 is arranged on the second robot 103, and when the second robot 103 is in the detection range A of the first detection system 106a, the spatial relationship between the calibration reference 107 arranged on the second robot 103 and the first robot 102 can be obtained through the first detection system 106a, and since the calibration reference 107 is arranged on the second robot 103, the spatial relationship can be regarded as the spatial relationship between the second robot 103 and the first robot 102.

[0066] It can also be understood that if the calibration reference 107 is arranged on the first robot 102, and when the first robot 102 is in the detection range B of the second detection system 106b, the spatial relationship between the calibration reference 107 arranged on the first robot 102 and the second robot 103 can be obtained through the second detection system 106b, and since the calibration reference 107 is arranged on the first robot 102, the spatial relationship can be regarded as the spatial relationship between the first robot 102 and the second robot 103.

[0067] In the embodiments of the present application, the first robot 102 and / or the second robot 103 have the calibration reference 107, when the first robot 102 is in the detection range of the second detection system 106b, the spatial relationship between the calibration reference 107 arranged on the first robot 102 and the second robot 103 can be directly obtained through the second detection system 106b, and when the second robot 103 is in the detection range of the first detection system 106a, the spatial relationship between the calibration reference 107 arranged on the second robot 103 and the first robot 102 can be directly obtained through the first detection system 106a, which facilitates the position calibration of the first robot 102 and the second robot 103.

[0068] In some embodiments, the robot system 10 further comprises a calibration component 108 arranged independently of the first robot 102 and the second robot 103, and the calibration component 108 is provided with the calibration reference 107.

[0069] In the embodiments of the present application, as shown in FIG. 5, in the clinical application of the robot system 10, the calibration reference 107 can be arranged on the calibration component 108 and placed within the detection range of the first detection system 106a and the second detection system 106b, without the need for manual holding of the calibration reference 107, so as to obtain the spatial relationship between the calibration reference 107 and the robot through the detection system.

[0070] In some embodiments, the calibration component 108 comprises a third detection system 106c, and the first robot 102 and the second robot 103 are within the detection range of the third detection system 106c.

[0071] For example, as shown in FIG. 5, the third detection system 106c is arranged on the third column 1081 of the calibration component 108, and the first robot 102 and the second robot 103 are within the detection range C of the third detection system 106c.

[0072] It can be understood that, in the clinical application of the robot system 10, there can be a case as shown in FIG. 5, i.e., due to the movement of the second robot 103, the calibration reference component does not appear within the detection range B of the second detection system 106b, or the second detection system 106b is blocked by other equipment or medical staff and fails to detect the calibration reference component, at this time, the spatial relationship between the second robot 103 and the third detection system 106c can be obtained through the third detection system 106c arranged on the calibration component 108; for another example, due to the movement of the first robot 102, the calibration reference component does not appear within the detection range A of the first detection system 106a, or the first detection system 106a is blocked by other equipment or medical staff and fails to detect the calibration reference component, at this time, the spatial relationship between the first robot 102 and the third detection system 106c can be obtained through the third detection system 106c arranged on the calibration component 108.

[0073] In some embodiments, the calibration component 108 has a base that is movable relative to the first robot 102 and the second robot 103.

[0074] For example, as shown in FIG. 5, the calibration component 108 comprises a third base 1082 and a third column 1081, and the calibration reference 107 is arranged on the third column 1081, wherein the third base 1082 is movable relative to the first robot 102 and the second robot 103, so as to facilitate movement of the calibration reference 107 within the detection range of the first detection system 106a and the second detection system 106b.

[0075] In one embodiment, as shown in FIG. 6, the calibration reference 107 only includes one calibration reference component, the first detection system 106a is arranged on the first robot 102 and has a corresponding detection range A, the second detection system 106b is arranged on the second robot 103 and has a corresponding detection range B, wherein the detection range A and the detection range B have at least a partial overlapping area, and the calibration reference 107 is placed in the overlapping area, and for example, only the calibration reference component in the calibration reference 107 is in the overlapping area.

[0076] In another embodiment, as shown in FIG. 7, the calibration reference 107 includes a first calibration reference component 1071, a second calibration reference component 1072 and a connecting component 1073, the first calibration reference component 1071 and the second calibration reference component 1072 are connected through the connecting component 1073, the first calibration reference component 1071 is in the detection range of the first detection system 106a, and the second calibration reference component 1072 is in the detection range of the second detection system 106b.

[0077] It can be understood that in the clinical application of the robot system 10, the distance between the first robot 102 and the second robot 103 can be far, so that the detection range A of the first detection system 106a and the detection range B of the second detection system 106b do not have an overlapping area in the limited operating room space, in which case, as shown in FIG. 7, the first calibration reference component 1071 and the second calibration reference component 1072 connected through the connecting component 1073 can be in the detection range A of the first detection system 106a and the detection range B of the second detection system 106b respectively, so that the first calibration reference component 1071 can be detected by the first detection system 106a and the second calibration reference component 1072 can be detected by the second detection system 106b.

[0078] In another embodiment, the connecting component 1073 has a telescopic mechanism, so that the first calibration reference component 1071 can be in the detection range of the first detection system 106a and the second calibration reference component 1072 can be in the detection range of the second detection system 106b.

[0079] In the embodiment of the present application, the connecting component 1073 has a telescopic mechanism, and during the positioning of the robot system 10, the telescopic mechanism can be adjusted so that the first calibration reference component 1071 is in the detection range of the first detection system 106a and the second calibration reference component 1072 is in the detection range of the second detection system 106b, so as to adapt to various placement positions of the split robot in clinical applications, and improve the versatility of the positioning method.

[0080] In another specific embodiment, as shown in FIG. 8, the connecting assembly 1073 is a bendable telescopic rod, wherein a first rod part of the telescopic rod is provided with the first calibration reference assembly 1071 and has a variable length L1, a second rod part of the telescopic rod is provided with the second calibration reference assembly 1072 and has a variable length L2, and there is a bending angle θ between the first rod part and the second rod part. In the clinical application of the robot system 10, there can be a case that the first robot 102 and the second robot 103 are placed far apart and not in the same plane, in which case, the lengths and the bending angle of the telescopic rod can be adjusted so that the first calibration reference assembly 1071 and the second calibration reference assembly 1072 are respectively within the detection range A of the first detection system 106a and the detection range B of the second detection system 106b, so that the position of the first calibration reference assembly 1071 can be detected by the first detection system 106a and the position of the second calibration reference assembly 1072 can be detected by the second detection system 106b.

[0081] Based on the robot system described in the above embodiments, in combination with FIGS. 2-8, referring to FIG. 9, an embodiment of the present application provides a positioning method of a robot system, which comprises steps S910 and S920.

[0082] In step S910, the first spatial relationship between the calibration reference and the first robot is acquired by the first detection system, and the second spatial relationship between the calibration reference and the second robot is acquired by the second detection system.

[0083] In step S920, the spatial relationship between the first robot and the second robot is determined based on the first spatial relationship and the second spatial relationship.

[0084] As shown in FIG. 2, the calibration reference 107 is within the detection range of the first detection system 106a and the second detection system 106b, the first detection system 106a is used to detect the first spatial relationship between the calibration reference 107 and the first detection system 106a in three-dimensional space, and the second detection system 106b is used to detect the second spatial relationship between the calibration reference 107 and the second detection system 106b, wherein the first spatial relationship refers to the relative relationship of the three-dimensional coordinate system of the calibration reference 107 itself in the three-dimensional coordinate system of the first detection system 106a itself, and the second spatial relationship refers to the relative relationship of the three-dimensional coordinate system of the calibration reference 107 itself in the three-dimensional coordinate system of the second detection system 106b itself.

[0085] It can be understood that, since the first detection system 106a is arranged on the first robot 102, the coordinate system of the first detection system 106a itself can be regarded as the coordinate system of the first robot 102, and then the first spatial relationship can also be regarded as the spatial relationship between the calibration reference 107 and the first robot 102; similarly, since the second detection system 106b is arranged on the second robot 103, the coordinate system of the second detection system 106b itself can be regarded as the coordinate system of the second robot 103, and then the second spatial relationship can also be regarded as the spatial relationship between the calibration reference 107 and the second robot 103, and then the spatial relationship between the first robot 102 and the second robot 103 is determined through geometric operation according to the first spatial relationship and the second spatial relationship by taking the calibration reference 107 as a medium.

[0086] In some embodiments, the spatial relationship includes a position relationship and / or an orientation relationship.

[0087] It can be understood that the spatial relationship can include a position relationship or an orientation relationship, or both a position relationship and an orientation relationship, wherein the position relationship refers to the relative position relationship between the calibration reference 107 and the robot in a three-dimensional space, and the orientation relationship refers to the orientation relationship between the calibration reference 107 and the robot. Specifically, for the first spatial relationship, the position relationship refers to the relative position relationship between the calibration reference 107 and the first robot 102 in a three-dimensional space, and the orientation relationship refers to the orientation between the calibration reference 107 and the first robot 102; for the second spatial relationship, the position relationship refers to the relative position relationship between the calibration reference 107 and the second robot 103 in a three-dimensional space, and the orientation relationship refers to the orientation relationship between the calibration reference 107 and the second robot 103.

[0088] The embodiments of the present application obtain the spatial relationship between the calibration reference and the first robot 102 and the second robot 103 by using the first detection system 106a and the second detection system 106b, and then obtain the position relationship and / or the orientation relationship between the first robot 102 and the second robot 103, which can provide the position relationship and / or the orientation relationship between multiple robots for the cooperative operation and avoidance control of the split robot.

[0089] Exemplarily, when the spatial relationship comprises both the positional relationship and the orientation relationship between the calibration reference object 107 and the robot, the spatial relationship can also be referred to as a relative pose, i.e., the relative pose of the coordinate system of the calibration reference object 107 in the coordinate system of the robot. By acquiring the first relative pose of the coordinate system of the calibration reference object 107 in the coordinate system of the first robot 102 through the first detection system 106a and acquiring the second relative pose of the coordinate system of the calibration reference object 107 in the coordinate system of the second robot 103 through the second detection system 106b, the relative pose between the first robot 102 and the second robot 103 is determined based on the first relative pose and the second relative pose, which can simultaneously calibrate the position and the orientation between the first robot 102 and the second robot 103 and simultaneously provide detailed calibration information between multiple robots for cooperative operation and avoidance control of the split robot.

[0090] In a specific embodiment, as shown in FIG. 3, the first detection system 106a is arranged on the first stand column 1022 of the first robot 102, and the second detection system 106b is arranged on the second stand column 1032 of the second robot 103, or the first detection system 106a is arranged on the first base 1023 of the first robot 102, and the second detection system 106b is arranged on the second base 1033 of the second robot 103.

[0091] When the first detection system 106a is arranged on the first base 1023 or the first stand column 1022 of the first robot 102 and the second detection system 106b is arranged on the second base 1033 or the second stand column 1032 of the second robot 103, in the positioning method of the robot system provided in the embodiments of the present application, the first spatial relationship acquired by the first detection system 106a is the spatial relationship between the calibration reference object 107 and the base of the first robot 102, and the second spatial relationship acquired by the second detection system 106b is the spatial relationship between the calibration reference object 107 and the base of the second robot 103, so that the spatial relationship between the bases of the first robot 102 and the second robot 103 can be determined based on the first spatial relationship and the second spatial relationship, the mutual positions of the bases of multiple robots can be calibrated, and the avoidance control between multiple robots of the robot system 10 in clinical application is facilitated.

[0092] In a specific embodiment, as shown in FIG. 4, the second robot 103 is provided with a calibration reference 107, when the second robot 103 is within the detection range of the first detection system 106a, the positioning method provided by the embodiment of the application can directly obtain the spatial relationship between the calibration reference 107 provided on the second robot 103 and the first robot 102 through the first detection system 106a, and take the spatial relationship between the calibration reference 107 and the first robot 102 as the spatial relationship between the second robot 103 and the first robot 102; or, the first robot 102 is provided with a calibration reference 107, when the first robot 102 is within the detection range of the second detection system 106b, the positioning method provided by the embodiment of the application can directly obtain the spatial relationship between the calibration reference 107 provided on the first robot 102 and the second robot 103 through the second detection system 106b, and take the spatial relationship between the calibration reference 107 and the second robot 103 as the spatial relationship between the first robot 102 and the second robot 103.

[0093] In a specific embodiment, as shown in FIG. 5, the calibration component 108 provided independently of the first robot 102 and the second robot 103 is provided with a third detection system 106c, and the first robot 102 and the second robot 103 are within the detection range of the third detection system 106c, the positioning method provided by the embodiment of the application further includes the following steps:

[0094] obtaining a third spatial relationship between the first robot and the third detection system, and / or a fourth spatial relationship between the second robot and the third detection system through the third detection system;

[0095] determining the spatial relationship between the first robot and the second robot based on the third spatial relationship and the second spatial relationship, or the fourth spatial relationship and the first spatial relationship.

[0096] As shown in FIG. 5, the third detection system 106c is provided on the third column 1081 of the calibration component 108, and the first robot 102 and the second robot 103 are within the detection range C of the third detection system 106c, the third spatial relationship between the first robot 102 and the third detection system 106c, and / or the fourth spatial relationship between the second robot 103 and the third detection system 106c is obtained through the third detection system 106c, and then the spatial relationship between the first robot 102 and the second robot 103 is determined based on the third spatial relationship and the second spatial relationship, or the fourth spatial relationship and the first spatial relationship.

[0097] It should be further noted that the calibration reference 107 can be arranged on the first robot 102, and the spatial relationship between the calibration reference 107 and the third detection system 106c can be acquired by the third detection system 106c, and since the calibration reference 107 is arranged on the first robot 102, the spatial relationship can be regarded as the third spatial relationship between the first robot 102 and the third detection system 106c; similarly, the calibration reference 107 can be arranged on the second robot 103, and the spatial relationship between the calibration reference 107 and the third detection system 106c can be acquired by the third detection system 106c, and since the calibration reference 107 is arranged on the second robot 103, the spatial relationship can be regarded as the third spatial relationship between the second robot 103 and the third detection system 106c.

[0098] In the embodiment of the present application, the third detection system 106c is arranged on the calibration component 108, and in the case that the first detection system 106a or the second detection system 106b fails to detect the calibration reference 107, the third spatial relationship between the first robot 102 and the third detection system 106c or the fourth spatial relationship between the second robot 103 and the third detection system 106c can be acquired by the third detection system 106c, and based on the third spatial relationship and the second spatial relationship or the fourth spatial relationship and the first spatial relationship, the spatial relationship between the first robot 102 and the second robot 103 is determined by taking the calibration reference 107 as a medium, so that the positioning method can continuously calibrate the positions of the split robots in the clinical application of the robot system 10, and the reliability and versatility of the positioning method are improved.

[0099] In a specific embodiment, as shown in FIG. 6, the calibration reference 107 is arranged independently of the first robot 102 and the second robot 103, and the calibration reference 107 is located in the overlapping area between the detection range A of the first detection system 106a and the detection range B of the second detection system 106b, and the relative pose of the coordinate system {3} of the calibration reference 107 in the coordinate system {1} of the first detection system 106a is acquired by the first detection system 106a and the relative pose of the coordinate system {3} of the calibration reference 107 in the coordinate system {2} of the second detection system 106b is acquired by the first detection system 106a Since the poses of the detection systems and the robots are fixed and known, the coordinate systems {1} and {2} can be regarded as the coordinate system of the first robot 102 and the coordinate system of the second robot 103, and then the relative pose of the first robot 102 and the second robot 103 is determined according to the relative poses and ​

[0100] In a specific embodiment, as shown in FIG. 7, the calibration reference 107 includes a first calibration reference component 1071, a second calibration reference component 1072, and a connecting component 1073, the first calibration reference component 1071 and the second calibration reference component 1072 are connected through the connecting component 1073, the first calibration reference component 1071 is within the detection range of the first detection system 106a, and the second calibration reference component 1072 is within the detection range of the second detection system 106b. Therefore, in the case that the first calibration reference component 1071 is within the detection range of the first detection system 106a, the first spatial relationship between the first calibration reference component 1071 and the first robot 102 can be obtained through the first detection system 106a, and correspondingly, in the case that the second calibration reference component 1072 is within the detection range of the second detection system 106b, the second spatial relationship between the second calibration reference component 1072 and the second robot 103 can be obtained through the second detection system 106b.

[0101] Correspondingly, in the positioning method of the robot system provided in the embodiments of the present application, the step of determining the spatial relationship between the first robot and the second robot based on the first spatial relationship and the second spatial relationship includes the following sub-steps:

[0102] determining the spatial relationship between the first calibration reference component and the second calibration reference component;

[0103] determining the spatial relationship between the first robot and the second robot according to the first spatial relationship, the second spatial relationship, and the spatial relationship between the first calibration reference component and the second calibration reference component.

[0104] It can be understood that when there are two calibration reference components, the spatial relationship between the first robot 102 and the second robot 103 needs to be determined by first determining the spatial relationship between the first calibration reference 107 and the second calibration reference 107, and then determining the spatial relationship between the first robot 102 and the second robot 103 through geometric operation according to the three spatial relationships between the first calibration reference component 1071 and the first robot 102, the second calibration reference component 1072 and the second robot 103, and the first calibration reference 107 and the second calibration reference 107.

[0105] In a specific embodiment, the spatial relationship is the relative pose. As shown in FIG. 7, the first calibration reference component 1071 is within the detection range of the first detection system 106a, and the second calibration reference component 1072 is within the detection range of the second detection system 106b. The relative pose of the coordinate system {4} of the first calibration reference 107 in the coordinate system {1} of the first detection system 106a is obtained through the first detection system 106a and the relative pose of the coordinate system {3} of the second calibration reference 107 in the coordinate system {2} of the second detection system 106b is acquired by the second detection system 106b the relative pose of the coordinate system {4} of the first calibration reference assembly 1071 in the coordinate system {3} of the second calibration reference assembly 1072 is determined and the relative pose of the coordinate system {4} of the first calibration reference assembly 1071 in the coordinate system {3} of the second calibration reference assembly 1072 is determined and the relative pose of the first robot 102 and the second robot 103 is determined

[0106] In some embodiments, the step of determining the spatial relationship between the first calibration reference assembly and the second calibration reference assembly comprises the following sub-steps:

[0107] In the case that the calibration reference is in different poses, the shape parameters of the connecting assembly are determined according to the spatial relationship between the first calibration reference assembly and the first robot and the spatial relationship between the second calibration reference assembly and the second robot;

[0108] the spatial relationship between the first calibration reference assembly and the second calibration reference assembly is determined according to the shape parameters of the connecting assembly.

[0109] It should be understood that since the first calibration reference assembly 1071 and the second calibration reference assembly 1072 are connected by the connecting assembly 1073, in order to determine the spatial relationship between the first calibration reference assembly 1071 and the second calibration reference assembly 1072, the shape parameters of the connecting assembly 1073 can be determined first, and then the spatial relationship between the first calibration reference assembly 1071 and the second calibration reference assembly 1072 is determined according to the shape parameters of the connecting assembly 1073.

[0110] Specifically, in the case that the calibration reference object 107 is in the first pose, the spatial relationship between the first calibration reference assembly 1071 and the first robot 102 is acquired by the first detection system 106a, and the spatial relationship between the second calibration reference assembly 1072 and the second robot 103 is acquired by the second detection system 106b, then the pose of the calibration reference object 107 is adjusted so that the calibration reference object 107 is in a second pose different from the first pose, in the case that the calibration reference object 107 is in the second pose, the spatial relationship between the first calibration reference assembly 1071 and the first robot 102 is acquired by the first detection system 106a, and the spatial relationship between the second calibration reference assembly 1072 and the second robot 103 is acquired by the second detection system 106b, finally, the shape parameters of the connecting assembly 1073 are determined by using the spatial relationship between the first calibration reference assembly 1071 and the first robot 102 and the spatial relationship between the second calibration reference assembly 1072 and the second robot 103 in different poses of the calibration reference object 107, and the spatial relationship between the first calibration reference assembly 1071 and the second calibration reference assembly 1072 is determined according to the shape parameters of the connecting assembly 1073.

[0111] In a specific embodiment, the spatial relationship is a relative pose, in the case that the calibration reference object 107 is in the first pose, the relative pose of the coordinate system {4} of the first calibration reference object 107 in the coordinate system {1} of the first detection system 106a is acquired by the first detection system 106a and the relative pose of the coordinate system {3} of the second calibration reference object 107 in the coordinate system {2} of the second detection system 106b is acquired by the second detection system 106b In the case that the calibration reference object 107 is in the second pose, the relative pose of the coordinate system {4} of the first calibration reference object 107 in the coordinate system {1} of the first detection system 106a is acquired by the first detection system 106a and the relative pose of the coordinate system {3} of the second calibration reference object 107 in the coordinate system {2} of the second detection system 106b is acquired by the second detection system 106b

[0112] Further, the relative pose of the coordinate system {4} of the first calibration reference assembly 1071 in the coordinate system {3} of the second calibration reference assembly 1072 is defined wherein the relative pose includes the shape parameters of the connecting assembly 1073, in the case that the calibration reference object 107 is in the first pose and the second pose, the relative pose between the first robot 102 and the second robot 103 and remains unchanged, so it has equation (1):

[0113] The relative pose of the first calibration reference component 1071 in the coordinate system {3} of the second calibration reference component 1072 can be obtained according to equation (1). Thus, the shape parameter of the connecting component 1073 is determined.

[0114] In some embodiments, the calibration reference 107 further comprises a sensor for measuring the position information between the first calibration reference component 1071 and the second calibration reference component 1072, and the step of determining the spatial relationship between the first calibration reference component 1071 and the second calibration reference component 1072 comprises the following sub-steps:

[0115] The position information between the first calibration reference component and the second calibration reference component measured by the sensor is used to determine the shape parameter of the connecting component;

[0116] The spatial relationship between the first calibration reference component and the second calibration reference component is determined according to the shape parameter of the connecting component.

[0117] In the embodiments of the present application, the calibration reference 107 is provided with a sensor for measuring the position information between the first calibration reference component 1071 and the second calibration reference component 1072, as shown in FIG. 7, the sensor can be arranged on the connecting component 1073 of the calibration component 108, and then the position information between the first calibration reference component 1071 and the second calibration reference component 1072 measured by the sensor is used to determine the shape parameter of the connecting component 1073, and the spatial relationship between the first calibration reference component 1071 and the second calibration reference component 1072 is determined according to the shape parameter of the connecting component 1073.

[0118] In some embodiments, the connecting component 1073 is a telescopic rod, and the shape parameter of the connecting component 1073 comprises the length of the telescopic rod.

[0119] In the embodiments of the present application, as shown in FIG. 7, the connecting component 1073 is a telescopic rod with a variable length L, and accordingly, the shape parameter of the connecting component 1073 comprises the length L of the telescopic rod. For example, the relative pose of the coordinate system {4} of the first calibration reference component 1071 in the coordinate system {3} of the second calibration reference component 1072 can be defined based on the length of the telescopic rod as follows: As follows:

[0120] For the equation (1) provided in the above embodiments, the relative pose of the first calibration reference component 1071 in the coordinate system {3} of the second calibration reference component 1072 can be represented as follows: And Can be represented as follows:

[0121] The relative pose of the first calibration reference component 1071 in the coordinate system {3} of the second calibration reference component 1072 can be represented as follows: And Can be represented as follows:

[0122] According to equation (1) and the matrix representation of the relative pose described above, the following equation exists:

[0123] a 14 +L·a 11 +a 11 ·b 14 +a 12 ·b 24 +a 13 ·b 34 =c 14 +L·c 11 +c 11 ·d 14 +c 12 ·d 24 +c 13 ·d 34 (3)

[0124] in, express The element in the first row and fourth column, express The element in the first row and fourth column can be used to solve the shape parameters of the connecting component 1073, i.e. the length L of the telescopic rod, according to equation (3).

[0125] In some embodiments, the telescopic rod is a bendable telescopic rod, and the shape parameters of the connecting assembly 1073 also include the bending angle of the telescopic rod.

[0126] For example, as shown in Figure 8, the telescopic rod is a bendable telescopic rod. A first calibration reference component 1071 with a variable length L1 is provided on the first part of the telescopic rod, and a second calibration reference component 1072 with a variable length L2 is provided on the second part of the telescopic rod. A bending angle θ exists between the first and second parts. In the clinical application of the robot system 10, there may be situations where the first robot 102 and the second robot 103 are placed far apart and not on the same plane. In this case, the length and bending angle of the telescopic rod can be adjusted so that the first calibration reference component 1071 and the second calibration reference component 1072 can appear within the detection range A of the first detection system 106a and the detection range B of the second detection system 106b, respectively. This allows the first detection system 106a to detect the position of the first calibration reference component 1071, and the second detection system 106b to detect the position of the second calibration reference component 1072.

[0127] For example, the relative pose of the coordinate system {4} of the first calibration reference assembly 1071 in the coordinate system {3} of the second calibration reference assembly 1072 is defined based on the length L1 of the first rod part, the length L2 of the second rod part, and the bending angle θ of the telescopic rod As follows:

[0128] According to equation (1) and the relative pose and The matrix representation of the relative pose, the following equation exists:

[0129] wherein, is the element in the first three rows and three columns of is the element in the first three rows and three columns of Since and only relate to the bending angle θ of the telescopic rod, the bending angle θ can be calculated according to equation (4), and the length L1 of the first rod part and the length L2 of the second rod part can be calculated according to equation (5).

[0130] Please refer to FIGS. 10-14, the second embodiment of the present application provides a robot system 20.

[0131] Referring to FIG. 10, the robot system 20 includes a control console 201, a first robot 202, a second robot 203, and a control device 204. The control console 201 can include a display device for displaying a surgical instrument environment, and a processor, wherein the display device is provided with an observation window (also referred to as a stereoscopic display) for a doctor to observe, and the processor is used for information processing and program running; in addition, the control console 201 can also include part or all of an operation control mechanism, an armrest, and a control switch (not shown in the figure), wherein the action of the operation control mechanism corresponds to the action of the surgical tool, and by controlling the operation control mechanism, the pose of the surgical tool can be adjusted; the armrest is used to place the doctor's arm; the control switch can be a switch convenient for hands or feet to touch or press, which is used for various function operations to complete human-computer interaction.

[0132] ​The first robot 202 comprises at least one first mechanical arm 2021, which comprises a plurality of jointed arms, and adjacent two jointed arms are relatively movable with specific degrees of freedom, so that the end of the first mechanical arm 2021 can reach a multi-degree-of-freedom movement (for example, 7 degrees of freedom, which can be different according to different surgical tools). The end of the first mechanical arm 2021 is provided with a manipulator assembly for driving a surgical tool and the surgical tool (for example, a surgical instrument or an image acquisition device). Taking the image acquisition device as an example, the image acquisition device can be an endoscope (for example, a three-dimensional endoscope) and is detachably mounted on the manipulator assembly. The image (for example, a surgical picture) acquired by the image acquisition device can be displayed on the display device of the control console 201. It can be understood that the image acquisition device of the present disclosure can also be other types, for example, a laparoscope.

[0133] The second robot 203 also comprises at least one second mechanical arm 2031, which comprises a plurality of jointed arms, and adjacent two jointed arms are relatively movable with specific degrees of freedom, so that the end of the second mechanical arm 2031 can reach a multi-degree-of-freedom movement. The end of the second mechanical arm 2031 is provided with a manipulator assembly for driving a surgical tool and the surgical tool.

[0134] It can be understood that the first robot 202 and the second robot 203 are distributed on different bases. The two can operate cooperatively and cooperate with each other to perform surgical operations on the patient on the operating table 7.

[0135] Optionally, the above-mentioned robot system 20 can further comprise an image processing system 206, which comprises a display device (for example, a display screen), which can display the image acquired by the image acquisition device. The image acquisition device can acquire an image and send the acquired image to the image processing system 206, so that the display device contained in the image processing system 206 displays the acquired image. The image acquisition device can send the acquired image directly or through the control console 201 to the image processing system 206. The image acquisition device can process (for example, decode) the acquired image through its own processor or through the processor of the image processing system 206. The processed image can also be sent to other image processing devices for further image processing (for example, denoising, enhancing contrast, improving clarity, etc.).

[0136] In the related art, multiple robots included in a robot system can perform collaborative surgery, and in order to reduce the occupation of the surgery space and to facilitate flexible pre-surgery positioning, the arrangement of the multiple robots can be distributed on different bases. In order to realize the collaborative operation of the multiple robots and mutual avoidance during operation, the relative poses between the multiple robots need to be obtained for positioning. However, there are many obstacles (such as sterile covers, operating beds, medical staff, other instruments, etc.) between the multiple robots, which interfere with the positioning effect between the multiple robots. In the following, a specific scenario is described. In this specific scenario, the relative pose relationship between the multiple robots can be determined by directly recognizing the pose identification module between the multiple robots using machine vision. Taking two robots as an example, in the related art, the visual recognition module is fixed on one robot, and the pose identification module is fixed on the mechanical arm of the other robot. The one robot recognizes the pose identification module on the mechanical arm of the other robot through the visual recognition module, thereby determining the relative pose relationship between the two robots.

[0137] It should be noted that the above introduction of one scenario for positioning between multiple robots is only exemplary, and in actual application, other scenarios for positioning between multiple robots can exist, which are not limited.

[0138] In the above specific scenario, the mechanical arm of the robot is often covered by a sterile cover. The pose identification module is arranged on the mechanical arm, which is blocked by the sterile cover between the visual recognition module. The visual recognition module itself is also fixed in position, and its flexibility is poor. In addition to the blocking of the sterile cover, the visual recognition module and the pose identification module are also easily blocked by other obstacles, and the positioning effect is poor.

[0139] Based on this, referring to FIG. 11, a first robot 202 and a second robot 203 of an embodiment of the present application are respectively provided with first positioning modules. A control device 204 is configured to obtain a first relative pose between the first positioning module on the first robot 202 and the second positioning module when the second positioning module is in a first position, the second positioning module including an inertial sensor; obtain a second relative pose between the first positioning module on the second robot 203 and the second positioning module when the second positioning module is in a second position; determine a third relative pose of the second positioning module between the first position and the second position by using the inertial sensor of the second positioning module; and determine the relative pose between the first robot 202 and the second robot 203 based on the first relative pose, the second relative pose, and the third relative pose.

[0140] The technical scheme provided by the embodiments of the present application is that a first positioning module is arranged on the first robot 202 and a second positioning module is arranged on the second robot 203. The second positioning module can move between a first position close to the first positioning module on the first robot 202 and a second position close to the first positioning module on the second robot 203 (for example, moving to the first position and then moving to the second position), so as to collect a first relative pose between the first positioning module on the first robot at the first position and collect a second relative pose between the first positioning module on the second robot at the second position. The third relative pose of the second positioning module between the first position and the second position is determined by using the self-provided inertial sensor. Finally, the relative pose between the first robot and the second robot is calculated based on the first relative pose, the second relative pose and the third relative pose. The second positioning module can be flexibly moved to a position not blocked by other obstacles, so as to collect the pose information of the corresponding first positioning module at different positions, thereby achieving better positioning effect.

[0141] As an example, the first positioning module comprises a positioning mark, and the second positioning module further comprises a visual positioning module capable of collecting images.

[0142] The positioning mark and the visual positioning module can also be distributed in the opposite way. As another example, the first positioning module can comprise a visual positioning module capable of collecting images, and the second positioning module can further comprise a positioning mark.

[0143] As an example, when the second positioning module is in the first position, a first image containing the positioning mark can be acquired by the visual positioning module, and the first relative pose can be determined based on the first image; when the second positioning module is in the second position, a second image containing the positioning mark can be acquired by the visual positioning module, and the second relative pose can be determined based on the second image. The first image is related to the first robot, and specifically includes: if the first positioning module includes the positioning mark and the second positioning module includes the visual positioning module that can acquire images, the first image is an image containing the positioning mark included in the first positioning module and arranged on the first robot, which is acquired by the visual positioning module included in the second positioning module; if the first positioning module includes the visual positioning module that can acquire images and the second positioning module includes the positioning mark, the first image is an image containing the positioning mark included in the second positioning module, which is acquired by the visual positioning module included in the first positioning module and arranged on the first robot. The second image is related to the second robot, and specifically includes: if the first positioning module includes the positioning mark and the second positioning module includes the visual positioning module that can acquire images, the second image is an image containing the positioning mark included in the first positioning module and arranged on the second robot, which is acquired by the visual positioning module included in the second positioning module; if the first positioning module includes the visual positioning module that can acquire images and the second positioning module includes the positioning mark, the second image is an image containing the positioning mark included in the second positioning module, which is acquired by the visual positioning module included in the first positioning module and arranged on the second robot.

[0144] As an example, during movement of the second positioning module from the first initial position to the first position, it is continuously detected whether the positioning mark is included in the visual field range of the visual positioning module, and if so, the visual positioning module is triggered to acquire the first image; during movement of the second positioning module from the second initial position to the second position, it is continuously detected whether the positioning mark is included in the visual field range of the visual positioning module, and if so, the visual positioning module is triggered to acquire the second image.

[0145] As an example, when a first trigger instruction input by a user is detected, the visual positioning module is triggered to acquire the first image; when a second trigger instruction input by a user is detected, the visual positioning module is triggered to acquire the second image.

[0146] The embodiment in which the first positioning module includes the positioning mark and the second positioning module further includes the visual positioning module that can acquire images will be further described below in combination with FIG. 12.

[0147] Referring to FIG. 12, as an example, the first positioning module 208 on the first robot 202 can include a positioning mark 2082, and the first positioning module 208 on the second robot 203 can include a positioning mark 2084. When the second positioning module 205 is in the first position, the positioning mark 2082 can be within the sensing range of the second positioning module 205; and when the second positioning module 205 is in the second position, the positioning mark 2084 can be within the sensing range of the second positioning module 205.

[0148] The control device 204 includes at least one memory and at least one processor, and is in communication connection with the console 201, the first robot 202, and the second robot 203, and can realize control among the console 201, the first robot 202, and the second robot 203. The control device 204 can execute a plurality of programmed instructions (for example, a computer readable medium storing instructions) through the processor to realize some or all of the methods described in the embodiments of the present application. Physically, the control device 204 can also be integrated in the console 201, the first robot 202, the second robot 203, or the second positioning module, or can be a separate computer device, and the embodiments of the present application do not make specific limitations on the structure of the control device 204.

[0149] The control device 204 can obtain the first relative pose between the first positioning module on the first robot 202 and the second positioning module when the second positioning module is in the first position, and the second relative pose between the first positioning module on the second robot 203 and the second positioning module when the second positioning module is in the second position, in various ways. Referring to FIG. 12, as an example, the second positioning module 205 can also include a visual positioning module 2051 capable of collecting images, and the sensing range of the second positioning module 205 is determined based on the field of view of the visual positioning module 2051. When the second positioning module 205 is in the first position, a first image containing the positioning mark 2082 can be collected by the visual positioning module 2051 of the second positioning module 205, and the first relative pose between the first positioning module on the first robot 102 and the second positioning module can be determined based on the first image; when the second positioning module 205 is in the second position, a second image containing the positioning mark 2084 can be collected by the visual positioning module 2051 of the second positioning module, and the second relative pose between the first positioning module on the second robot 203 and the second positioning module can be determined based on the second image.

[0150] The above-mentioned visual positioning module 2051 can have various specific implementations. As an example, the visual positioning module 2051 can be a monocular visual positioning camera, or a multi-view visual positioning camera, or an optical tracker, etc.

[0151] It should be noted that the above description of the manner of obtaining the first relative pose and the second relative pose is only exemplary, and in actual application, other manners of obtaining the first relative pose and the second relative pose are not excluded. As another example, the second positioning module can further include a laser radar, and the sensing range of the second positioning module can be determined based on the measurement range of the laser radar. When the second positioning module is at the first position, the first relative pose between the first positioning module on the first robot 202 and the second positioning module can be measured by using the laser radar of the second positioning module. When the second positioning module is at the second position, the second relative pose between the first positioning module on the second robot 203 and the second positioning module can be measured by using the laser radar of the second positioning module. Therefore, the manner of obtaining the first relative pose and the second relative pose is not limited in particular.

[0152] When the second positioning module 205 includes the visual positioning module 2051, the visual positioning module 2051 can be used to collect the first image containing the positioning mark 2082 and the second image containing the positioning mark 2084 in various manners. As an example, during the movement of the second positioning module from the first initial position to the first position, it is continuously detected whether the visual positioning module of the second positioning module includes the positioning mark 2082 in the field of view. If the visual positioning module includes the positioning mark 2082 in the field of view, the visual positioning module is triggered to collect the first image containing the positioning mark 2082. During the movement of the second positioning module 205 from the second initial position to the second position, it is continuously detected whether the visual positioning module of the second positioning module includes the positioning mark 2084 in the field of view. If the visual positioning module includes the positioning mark 2084 in the field of view, the visual positioning module is triggered to collect the second image containing the positioning mark 2082. The visual positioning module can be automatically triggered to collect the image when it is detected that the visual positioning module 2051 of the second positioning module 205 includes the positioning mark 2082 or the positioning mark 2084 in the field of view. The efficiency of image collection is relatively high.

[0153] As an example, the first initial position can be any position in the surgical space, and the second initial position can be the first position or any position other than the first position, which is not limited in particular.

[0154] As another example, during the movement of the second positioning module 205 from the first initial position to the first position, if it is detected that the field of view of the visual positioning module 2051 of the second positioning module 205 includes the positioning mark 2082, and the visual positioning module collects the first image containing the positioning mark 2082, at this time the second positioning module 205 is at the first position, the inertial sensor of the second positioning module 205 can be triggered to start recording the inertial sensor data at the same time; during the movement of the second positioning module 205 from the second initial position to the second position, if it is detected that the field of view of the visual positioning module of the second positioning module 205 includes the positioning mark 2084, and the visual positioning module collects the second image containing the positioning mark 2084, at this time the second positioning module 205 is at the second position, the inertial sensor 2053 of the second positioning module 205 can be triggered to stop recording the inertial sensor data at the same time; finally, based on the inertial sensor data recorded by the inertial sensor 2053 during the period from the start of recording to the stop of recording, the third relative pose of the second positioning module 205 between the first position and the second position is determined. The inertial sensor data includes one or more of linear acceleration, angular velocity of the gyroscope, magnetic field and temperature. The visual positioning module 2051 of the second positioning module 205 can trigger the inertial sensor 2053 to start recording or stop recording the inertial sensor data at the same time of collecting the image containing the mark, associate the image collection with the start or stop of recording the inertial sensor data by the inertial sensor 2053, and improve the positioning efficiency.

[0155] As an example, another way of collecting the first image containing the positioning mark 2082 and the second image containing the positioning mark 2084 by the visual positioning module of the second positioning module 205 can include: when the first trigger instruction input by the user is detected, the visual positioning module 2051 of the second positioning module 205 can be triggered to collect the first image containing the positioning mark 2082; when the second trigger instruction input by the user is detected, the visual positioning module 2051 of the second positioning module 205 can be triggered to collect the second image containing the positioning mark 2084. When the visual positioning module 2051 of the second positioning module 205 is triggered to collect the image containing the positioning mark, the trigger instruction input by the user is manually triggered, and the collected image can better meet the needs of the user and be more accurate.

[0156] The first trigger instruction or the second trigger instruction input by the user can be generated in various ways. As an example, the first trigger instruction or the second trigger instruction can be a trigger instruction generated after the user presses a button. The button can be a button on the second positioning module or a button on another device, such as a button on the control device 204. As another example, the first trigger instruction or the second trigger instruction can be a trigger instruction generated after the user inputs a voice instruction. The voice instruction can be received and recognized by the second positioning module or by another device, such as the control device 204. Therefore, the specific generation method of the first trigger instruction or the second trigger instruction is not limited.

[0157] As another example, when the first trigger instruction is received and the visual positioning module 2051 collects the first image containing the positioning mark 2082, the second positioning module 205 is at the first position, and the inertial sensor 2053 of the second positioning module 205 can also be triggered to start recording inertial sensor data at the same time. When the second trigger instruction is received and the visual positioning module 2051 collects the second image containing the positioning mark 2084, the second positioning module 205 is at the second position, and the inertial sensor 2053 of the second positioning module 205 can also be triggered to stop recording inertial sensor data at the same time. Finally, based on the inertial sensor data recorded by the inertial sensor 2053 from the start of recording to the stop of recording, the third relative pose of the second positioning module between the first position and the second position is determined. The inertial sensor data includes one or more of linear acceleration, angular velocity of the gyroscope, magnetic field, and temperature.

[0158] After determining the third relative pose of the second positioning module between the first position and the second position, the relative pose between the first robot 202 and the second robot 203 can be determined based on the first relative pose, the second relative pose, and the third relative pose. As an example, the coordinates of the first robot 202 can be denoted as coordinate {1}, the coordinates of the second robot 203 can be denoted as coordinate {2}, the coordinates of the second positioning module at the first position can be denoted as coordinate {3}, and the coordinates of the second positioning module at the second position can be denoted as coordinate {4}. Let T represent the relative pose determined between two objects, then the relative pose between the first robot 202 and the second robot 203 can be represented by the following formula:

[0159] wherein, T is the relative pose between the first robot 202 and the second robot 203, T is the relative pose of the second positioning module between the first position and the second position (i.e., the third relative pose), a relative pose between the positioning mark 2082 and the second positioning module when the second positioning module is in the first position (i.e., the first relative pose described above), a relative pose between the positioning mark 2084 and the second positioning module when the second positioning module is in the second position (i.e., the second relative pose described above).

[0160] The positioning mark 2082 and the positioning mark 2084 described above can include various types. As an example, the positioning mark 2082 can include a mark of a type that can be an active light-emitting element, a passive reflective mark, a passive color structure, a 3D or 2D structure of a known shape, a fiducial, a two-dimensional code, a data matrix code, a checkerboard, or any combination thereof, and the like, wherein the active light-emitting element can include a color or infrared (IR) emitter, such as an LED, and the like. The passive reflective mark can be, for example, a reflective ball, a reflective sticker, and the like. The positioning mark 2084 can also include a mark of a type that can be an active light-emitting element, a passive reflective mark, a passive color structure, a 3D or 2D structure of a known shape, a fiducial, a two-dimensional code, a data matrix code, a checkerboard, or any combination thereof, and the like, wherein the active light-emitting element can include a color or infrared (IR) emitter, such as an LED, and the like. The passive reflective mark can be, for example, a reflective ball, a reflective sticker, and the like. Of course, the implementation of the present application is not limited thereto.

[0161] The following further describes an embodiment in which the first positioning module includes a visual positioning module that can capture images, and the second positioning module further includes a positioning mark, in conjunction with FIG. 13:

[0162] In view of the above-described case in which the second positioning module simultaneously includes an inertial sensor and a visual positioning module, the volume and weight of the second positioning module can be relatively large, and whether the user holds the second positioning module for movement or a mechanical arm drives the second positioning module for movement, the relatively large volume and weight will cause a relatively large burden on the object carrying the second positioning module.

[0163] Based on this, referring to FIG. 13, the first positioning module 208 on the first robot 202 in another embodiment of the application can include a visual positioning module 2083, and the first positioning module 208 on the second robot 203 can include a visual positioning module 2085; the second positioning module 205 can also include a positioning marker 2052 (at this time, the second positioning module 205 includes an inertial sensor 2053 and the positioning marker 2052, but does not include a visual positioning module 2051), and the control device 204 is configured to acquire a first relative pose between the visual positioning module 2083 and the second positioning module 205 when the second positioning module 205 is at a first position; the second positioning module is in the first position and is within the visual field range of the visual positioning module 2083; acquire a second relative pose between the visual positioning module 2085 and the second positioning module when the second positioning module 205 is at a second position; the second positioning module is in the second position and is within the visual field range of the visual positioning module 2085; determine a third relative pose of the second positioning module between the first position and the second position by using the inertial sensor 2053 of the second positioning module; and determine the relative pose between the first robot 202 and the second robot 203 based on the first relative pose, the second relative pose, and the third relative pose.

[0164] Compared with the positioning mode in which the positioning markers 2082 and 2084 are respectively arranged at the first robot 202 and the second robot 203 and the second positioning module 205 moves between the first position close to the positioning marker 2082 and the second position close to the positioning marker 2084 in the embodiment shown in FIG. 12, the positioning mode of the technical solution provided in this embodiment is that the vision positioning module 2083 and the vision positioning module 2085 are respectively arranged at the first robot 202 and the second robot 203 to replace the positioning markers 2082 and 2084, and the vision positioning module 2051 is replaced by the positioning marker 2052 on the second positioning module 205, and the second positioning module 205 moves between the first position and the second position. The second positioning module 205 only needs to include the inertial sensor 2053 and the positioning marker 2052, and does not need to include the vision positioning module or other types of positioning modules, and has a smaller volume and weight, thereby greatly reducing the burden on the object carrying the second positioning module 205 when moving. The vision positioning module or other types of positioning modules can be arranged on the first robot 202 and the second robot 203 respectively, so that the control device 204 collects the first relative pose between the second positioning module and the vision positioning module 2083 at the first position, and collects the second relative pose between the second positioning module and the vision positioning module 2085 at the second position, and determines the third relative pose of the second positioning module between the first position and the second position by using the inertial sensor of the second positioning module. Finally, the relative pose between the first robot and the second robot is calculated based on the first relative pose, the second relative pose, and the third relative pose. The second positioning module can also be flexibly moved to a position not blocked by other obstacles to facilitate the collection of the pose information of the corresponding positioning device at different positions, thereby achieving better positioning effect.

[0165] As an example, when the second positioning module 205 is at the first position, the first image containing the positioning marker 2052 can be collected by using the vision positioning module 2083, and the first relative pose between the vision positioning module 2083 and the second positioning module 205 is determined based on the first image. When the second positioning module is at the second position, the second image containing the positioning marker 2052 can be collected by using the vision positioning module 2085, and the second relative pose between the second positioning module 205 and the vision positioning module 2085 is determined based on the second image.

[0166] As an example, during the movement of the second positioning module 205 from the first initial position to the first position, it can be continuously detected whether the visual positioning module 2083 has a field of view range including the positioning mark 2052, and if so, the visual positioning module 2083 is triggered to collect the first image containing the positioning mark 2052; during the movement of the second positioning module 205 from the second initial position to the second position, it can be continuously detected whether the visual positioning module 2085 has a field of view range including the positioning mark 2052, and if so, the visual positioning module 2085 is triggered to collect the second image containing the positioning mark 2052.

[0167] As an example, when a first trigger instruction input by a user is detected, the visual positioning module 2083 is triggered to collect the first image containing the positioning mark 2052; when a second trigger instruction input by a user is detected, the visual positioning module 2085 is triggered to collect the second image containing the positioning mark 2052.

[0168] Considering that the visual positioning module 2083, the visual positioning module 2085, and the inertial sensor 2053 are not on the same device at this time (the visual positioning module 2083 and the visual positioning module 2085 are respectively located on the first robot 202 and the second robot 203, and the inertial sensor 2053 is on the second positioning module 205), but synchronization between the image collection of the visual positioning module and the start and stop of the recording of the inertial sensor data by the inertial sensor still needs to be achieved.

[0169] To solve this problem, as an example, at this time, the second positioning module 205 can further include a communication unit that can communicate with the first positioning module 208 in addition to the inertial sensor and the positioning mark; the communication unit can be used to transmit a synchronization positioning signal between the visual positioning module and the inertial sensor, and the synchronization positioning signal is used to synchronize the image collection of the visual positioning module and the start and stop of the recording of the inertial sensor data by the inertial sensor. For example, the communication unit can transmit a synchronization positioning signal between the visual positioning module 2083 and the inertial sensor 2053 of the second positioning module 205 to synchronize the first image collection of the visual positioning module 2083 and the start of the recording of the inertial sensor data by the inertial sensor 2053; the communication unit can also transmit a synchronization positioning signal between the visual positioning module 2085 and the inertial sensor 2053 of the second positioning module 205 to synchronize the second image collection of the visual positioning module 2085 and the stop of the recording of the inertial sensor data by the inertial sensor 2053.

[0170] As an example, the positioning mark 2052 on the second positioning module 205 can be a type of mark such as an active light-emitting element, a passive reflective mark, a passive color structure, a 3D or 2D structure of a known shape, a fiducial, a two-dimensional code, a data matrix code, a checkerboard, or any combination thereof, wherein the active light-emitting element can include a color or infrared (IR) emitter such as an LED or the like. The passive reflective mark can be, for example, a reflective ball, a reflective sticker, or the like.

[0171] The inertial sensor of the second positioning module described in any of the above embodiments can have various specific implementations. As an example, the inertial sensor of the second positioning module can include one or more of an acceleration sensor, a gyroscope angular velocity sensor, a magnetic field sensor, a temperature sensor, or a signal processor. As another example, the inertial sensor can further include a communication module that can be used to realize communication of the inertial sensor with the control device 204 or other objects.

[0172] Due to the fact that the mechanical arm of the first robot 202 or the second robot 203 is often covered by a sterile cover, to address this issue, the first positioning module described in any of the above embodiments can be respectively arranged on the base of the first robot 202 and the second robot 203. As an example, referring to FIG. 12, the positioning mark 2082 of the first positioning module 208 on the first robot 202 can be arranged on the base 2023 of the first robot 202, and the positioning mark 2084 of the first positioning module 208 on the second robot 203 can be arranged on the base 2033 of the second robot 203. Arranging the positioning mark on the base of the robot can avoid the shielding of the sterile cover, and the position of the base itself can also accurately reflect the position of the robot on the base. As another example, referring to FIG. 13, the visual positioning module 2083 can be arranged on the base of the first robot 202, and the visual positioning module 2085 can be arranged on the base of the second robot 203. As another example, the base 2023 can include a chassis and a lifting column, and the positioning mark 2082 (or the visual positioning module 2083) can be arranged on the chassis or the lifting column, which is not limited; the base 2033 can also include a chassis and a lifting column, and the positioning mark 2084 (or the visual positioning module 2085) can be arranged on the chassis or the lifting column, which is not limited.

[0173] The second positioning module described in any of the above embodiments can be moved in the surgical space (e.g., moved to the first position or the second position, or moved between the first position and the second position, or moved to any position outside the first position and the second position) in various ways. As an example, the second positioning module can be manually controlled by a user to move to the first position or the second position, for example, the user directly holds the second positioning module and moves the second positioning module by moving himself / herself. As another example, the robotic system can further include a third robot, and the second positioning module can be arranged on a mechanical arm of the third robot and can be controlled by the mechanical arm to move to the first position or the second position, for example, the mechanical arm can hold the second positioning module, which can be moved by displacement of the mechanical arm or by movement of the third robot itself. It should be noted that the above description of the way in which the second positioning module is moved in the surgical space is only exemplary, and in actual applications, other moving ways can exist and are not limited in particular.

[0174] It can be understood that the above robotic system can include only two robots, i.e., the first robot 202 and the second robot 203, and can also include three or more robots. When the robotic system includes three or more robots, the above-described way of determining the relative pose between the first robot and the second robot can be applied to the process of determining the relative pose between any two robots.

[0175] The first robot 202 or the second robot 203 described in any of the above embodiments can belong to various types. As an example, the first robot 202 or the second robot 203 can be a single-hole surgical robot. As another example, the first robot 202 or the second robot 203 can also be a multi-hole surgical robot. Therefore, the type of the first robot and the second robot is not limited in particular.

[0176] In another specific embodiment, as shown in FIG. 14, the second positioning module 205 includes a first calibration reference assembly 2055, a second calibration reference assembly 2056, and a connecting assembly 2057, and the first calibration reference assembly 2055 and the second calibration reference assembly 2056 are connected through the connecting assembly 2057. When the second positioning module 205 is in the first position, the first calibration reference assembly 2055 is in the detection range of the visual positioning module 2083 of the first robot 202; when the second positioning module 205 is in the second position, the second calibration reference assembly 2056 is in the detection range of the visual positioning module 2085 of the second robot 203.

[0177] Corresponding to the above embodiment of the robot system, the embodiment of the present application also provides a positioning method of a robot system, the robot system comprising a first robot and a second robot, and a first positioning module and a second positioning module are respectively arranged on the first robot and the second robot; referring to FIG. 15, the positioning method comprises steps S1510-S1540.

[0178] In step S1510, a first relative pose between the first positioning module on the first robot and the second positioning module when the second positioning module is in a first position is acquired.

[0179] The second positioning module comprises an inertial sensor.

[0180] In step S1520, a second relative pose between the first positioning module on the second robot and the second positioning module when the second positioning module is in a second position is acquired.

[0181] In step S1530, a third relative pose of the second positioning module between the first position and the second position is determined by using the inertial sensor.

[0182] In step S1540, a relative pose between the first robot and the second robot is determined based on the first relative pose, the second relative pose and the third relative pose.

[0183] As an example, the first positioning module comprises a positioning mark, and the second positioning module further comprises a visual positioning module capable of acquiring an image.

[0184] As an example, the acquiring of the first relative pose between the first positioning module on the first robot and the second positioning module when the second positioning module is in the first position comprises: acquiring a first image of the positioning mark contained in the first positioning module on the first robot by using the visual positioning module when the second positioning module is in the first position, and determining the first relative pose based on the first image; and the acquiring of the second relative pose between the first positioning module on the second robot and the second positioning module when the second positioning module is in the second position comprises: acquiring a second image of the positioning mark contained in the first positioning module on the second robot by using the visual positioning module when the second positioning module is in the second position, and determining the second relative pose based on the second image.

[0185] As an example, the acquiring, by the visual localization module, the first image of the positioning mark included in the first localization module on the first robot comprises: continuously detecting whether the positioning mark included in the first localization module on the first robot is included in the field of view of the visual localization module during movement of the second localization module from a first initial position to the first position, and if so, triggering the visual localization module to acquire the first image; and the acquiring, by the visual localization module, the second image of the positioning mark included in the first localization module on the second robot comprises: continuously detecting whether the positioning mark included in the first localization module on the second robot is included in the field of view of the visual localization module during movement of the second localization module from a second initial position to the second position, and if so, triggering the visual localization module to acquire the second image.

[0186] As an example, the acquiring, by the visual localization module, the first image of the positioning mark included in the first localization module on the first robot comprises: detecting a first trigger instruction input by a user, and triggering the visual localization module to acquire the first image; and the acquiring, by the visual localization module, the second image of the positioning mark included in the first localization module on the second robot comprises: detecting a second trigger instruction input by a user, and triggering the visual localization module to acquire the second image.

[0187] As an example, the determining, by the inertial sensor, the third relative pose of the second localization module between the first position and the second position comprises: detecting that the visual localization module is triggered to acquire the first image, and triggering the inertial sensor to start recording inertial sensor data; detecting that the visual localization module is triggered to acquire the second image, and triggering the inertial sensor to stop recording inertial sensor data; and determining the third relative pose of the second localization module between the first position and the second position based on the inertial sensor data recorded by the inertial sensor during the period from starting recording to stopping recording. The inertial sensor data comprises one or more of linear acceleration, angular velocity of a gyroscope, magnetic field, and temperature.

[0188] As an example, the positioning mark comprises a type of mark that can be an active light-emitting element, a passive reflective mark, a passive color structure, a 3D or 2D structure of a known shape, a fiducial, a two-dimensional code, a data matrix code, a checkerboard, or any combination thereof, etc., wherein the active light-emitting element can comprise a color or infrared (IR) emitter, such as an LED, etc. The passive reflective mark is, for example, a reflective ball, a reflective sticker, etc.

[0189] As an example, the second positioning module can be manually controlled by a user to move to the first position or the second position, or the robot system further comprises a third robot, the second positioning module is arranged on a mechanical arm of the third robot and can be controlled by the mechanical arm to move to the first position or the second position.

[0190] As an example, the first positioning module is respectively arranged on a base of the first robot and the second robot.

[0191] As an example, the first positioning module comprises a visual positioning module capable of capturing images, and the second positioning module further comprises a positioning mark.

[0192] As an example, the second positioning module further comprises a communication unit, and the determining the third relative pose of the second positioning module between the first position and the second position by using the inertial sensor comprises: triggering the inertial sensor to start recording inertial sensor data based on a first indication, the first indication being that the communication unit detects that the visual positioning module contained in the first positioning module on the first robot is triggered to capture images; triggering the inertial sensor to stop recording inertial sensor data based on a second indication, the second indication being that the communication unit detects that the visual positioning module contained in the first positioning module on the second robot is triggered to capture images; and determining the third relative pose of the second positioning module between the first position and the second position based on the inertial sensor data recorded in the period from starting recording to stopping recording.

[0193] In a specific embodiment, as shown in FIG. 14, the second positioning module 205 comprises a first calibration reference assembly 2055, a second calibration reference assembly 2056, and a connecting assembly 2057, the first calibration reference assembly 2055 and the second calibration reference assembly 2056 are connected through the connecting assembly 2057; when the second positioning module 205 is in the first position, the first calibration reference assembly 2055 is in the detection range of the visual positioning module 2083 of the first robot 202; when the second positioning module 205 is in the second position, the second calibration reference assembly 2056 is in the detection range of the visual positioning module 2085 of the second robot 203.

[0194] Therefore, when the first calibration reference component 2055 is in the detection range of the visual positioning module 2083 of the first robot 202, and the second positioning module 205 is in the first position, the relative pose between the first calibration reference component 2055 and the first robot 202, i.e., the first relative pose, can be obtained by the visual positioning module 2083 of the first robot 202. Correspondingly, when the second calibration reference component 2056 is in the detection range of the visual positioning module 2085 of the second robot 203, and the second positioning module 205 is in the second position, the relative pose between the second calibration reference component 2056 and the second robot 203, i.e., the second relative pose, can be obtained by the visual positioning module 2085 of the second robot 203. Further, the third relative pose of the second positioning module 205 between the first position and the second position can also be determined by using the inertial sensor.

[0195] In the positioning method of the robot system provided in the embodiments of the present application, the step of determining the third relative pose of the second positioning module 205 between the first position and the second position by using the inertial sensor includes the following sub-steps:

[0196] determining a first sub-relative pose of the first calibration reference component between the first position and the second position; or

[0197] determining a second sub-relative pose of the second calibration reference component between the first position and the second position.

[0198] It can be understood that, referring to FIG. 14, the second positioning module 205 includes the first calibration reference component 2055, the second calibration reference component 2056, and a connecting component 2057. When determining the third relative pose of the second positioning module 205 between the first position and the second position by using the inertial sensor, it can be determined that the first sub-relative pose of the first calibration reference component 2055 between the first position and the second position is determined, or the second sub-relative pose of the second calibration reference component 2056 between the first position and the second position is determined.

[0199] In some embodiments, the second positioning module 205 further includes a measurement sensor, which is used to measure the position information between the first calibration reference component 2055 and the second calibration reference component 2056. The step of determining the third relative pose of the second positioning module 205 between the first position and the second position by using the inertial sensor includes the following sub-steps:

[0200] determining the shape parameter of the connecting component by using the position information between the first calibration reference component and the second calibration reference component measured by the measurement sensor; and

[0201] The sixth relative pose of the first calibration reference component and the second calibration reference component when the second positioning module is in the first position is determined according to the shape parameter of the connecting component, and the third relative pose is obtained according to the sixth relative pose and the second sub relative pose; or

[0202] The seventh relative pose of the first calibration reference component and the second calibration reference component when the second positioning module is in the second position is determined according to the shape parameter of the connecting component, and the third relative pose is obtained according to the first sub relative pose and the seventh relative pose.

[0203] In the embodiment of the present application, referring to FIG. 14, the second positioning module 205 is provided with a measuring sensor for measuring the position information between the first calibration reference component 2055 and the second calibration reference component 2056. As shown in FIG. 14, the sensor can be arranged on the connecting component 2057 of the second positioning module 205, and then the shape parameter of the connecting component 2057 is determined by using the position information between the first calibration reference component 2055 and the second calibration reference component 2056 measured by the sensor.

[0204] After obtaining the shape parameter of the connecting component 2057, the position information of the first calibration reference component 2055 when the second positioning module 205 is in the first position can also be determined according to the first relative pose, and then the position information of the second calibration reference component 2056 when the second positioning module 205 is in the first position is determined according to the shape parameter and the position information of the first calibration reference component 2055 when the second positioning module 205 is in the first position, so as to determine the sixth relative pose of the first calibration reference component 2055 and the second calibration reference component 2056 when the second positioning module 205 is in the first position. After obtaining the sixth relative pose, the third relative pose can be obtained according to the sixth relative pose and the second sub relative pose.

[0205] Alternatively, after obtaining the shape parameter of the connecting component 2057, the position information of the second calibration reference component 2056 when the second positioning module 205 is in the second position can also be determined according to the second relative pose, and then the position information of the first calibration reference component 2055 when the second positioning module 205 is in the second position is determined according to the shape parameter and the position information of the second calibration reference component 2056 when the second positioning module 205 is in the second position, so as to determine the seventh relative pose of the first calibration reference component 2055 and the second calibration reference component 2056 when the second positioning module 205 is in the second position. After obtaining the seventh relative pose, the third relative pose can be obtained according to the seventh relative pose and the first sub relative pose.

[0206] Please refer to FIG. 10-FIG. 14, the application further provides a robot system 20, the robot system 20 includes a first robot 202 and a second robot 203, the first robot 202 is provided with a first positioning module 208, and the second robot 203 is detachably connected with a second positioning module 205, and the second positioning module 205 includes an inertial sensor 2053.

[0207] In the embodiment of the application, the first positioning module can be arranged only on the first robot 202, and the first positioning module is not arranged on the second robot 203, and the second robot 203 is detachably connected with the second positioning module; at this time, the control device 204 is used to acquire a first relative pose between the first positioning module on the first robot 202 and the second positioning module when the second positioning module is in a first position, and the second positioning module includes an inertial sensor; a fourth relative pose of the second positioning module between the first position and a third position is determined by using the inertial sensor of the second positioning module, wherein the third position is a position where the second positioning module is detachably connected on the second robot 203; and a relative pose between the first robot 202 and the second robot 203 is determined based on the first relative pose and the fourth relative pose.

[0208] The third position is located on a base of the second robot 203 or a position other than the base on the second robot 203.

[0209] When the second positioning module is detachably connected at a position other than the base of the second robot 203, a fifth relative pose between the second positioning module at the third position and the second robot 203 can be determined based on a kinematic relationship of a mechanical mechanism of the second robot 203.

[0210] At this time, the determination of the relative pose between the first robot 202 and the second robot 203 based on the first relative pose and the fourth relative pose can include:

[0211] The determination of the relative pose between the first robot 202 and the second robot 203 based on the first relative pose, the fourth relative pose and the fifth relative pose.

[0212] The technical scheme provided by the embodiment can detachably connect the movable second positioning module and the second robot 203. By mounting the movable second positioning module on the second robot 203, the inertial sensor of the second positioning module starts to record inertial sensor data. When the movable second positioning module is detached from the second robot 203, the second positioning module can be moved to a first position, so that the first relative pose between the first positioning module on the first robot 202 and the second positioning module when the second positioning module is in the first position is obtained. At this time, the inertial sensor of the second positioning module stops recording the inertial sensor data. Then, the fourth relative pose between the position of the movable second positioning module when the second positioning module is mounted on the second robot 203 and the first position is determined based on the inertial sensor data recorded by the inertial sensor of the second positioning module during the period from the start of recording to the stop of recording.

[0213] In the embodiment, as an example, the first positioning module can include a positioning mark, and the second positioning module can further include a visual positioning module capable of capturing images. As another example, the first positioning module includes a visual positioning module capable of capturing images, and the second positioning module further includes a positioning mark.

[0214] At this time, the manner in which the control device 204 obtains the first relative pose between the first positioning module on the first robot 202 and the second positioning module when the second positioning module is in the first position in the embodiment can refer to the manner of obtaining the first relative pose described in any of the above embodiments, and details are not described herein again.

[0215] The embodiment of the application further provides a positioning method of a robot system, the robot system including a first robot and a second robot, a first positioning module being arranged on the first robot, and the second robot being detachably connected with a second positioning module, the second positioning module including an inertial sensor. Referring to FIG. 16, the positioning method includes steps S1610-S1630.

[0216] In step S1610, a first relative pose between a first positioning module on the first robot and the second positioning module when the second positioning module is in a first position is obtained.

[0217] In step S1620, a fourth relative pose of the second positioning module between the first position and a third position is determined by using the inertial sensor.

[0218] The third position is a position at which the second positioning module is detachably connected to the second robot.

[0219] In step S1630, a relative pose between the first robot and the second robot is determined based on the first relative pose and the fourth relative pose.

[0220] As an example, the second positioning module is detachably connected to the base of the second robot, i.e., the third position is located on the base of the second robot.

[0221] As an example, the second positioning module is detachably connected to a position other than the base of the second robot, i.e., the third position is located on a position other than the base of the second robot. At this time, the positioning method further comprises:

[0222] determining a fifth relative pose between the second positioning module and the second robot at the second position based on the kinematic relationship of the mechanical mechanism of the second robot;

[0223] The determination of the relative pose between the first robot and the second robot based on the first relative pose and the fourth relative pose comprises:

[0224] The determination of the relative pose between the first robot and the second robot based on the first relative pose, the fourth relative pose and the fifth relative pose.

[0225] As an example, the first positioning module comprises a positioning mark, and the second positioning module further comprises a visual positioning module capable of acquiring images.

[0226] As an example, the first positioning module comprises a visual positioning module capable of acquiring images, and the second positioning module further comprises a positioning mark.

[0227] The present application also provides a computer readable storage medium having computer instructions stored thereon, wherein the computer instructions are executed by a processor to implement the positioning method of any one of the above embodiments. For example, the computer readable storage medium can be a ROM, a random access memory (RAM), a CD-ROM, a magnetic tape, a floppy disk, an optical data storage device, etc.

[0228] The present application also provides an electronic device, as shown in FIG. 17, which comprises a processor 1710 and a memory 1720. The memory 1720 is used to store processor-executable instructions, and the processor 1710 is configured to implement the positioning method of any one of the above embodiments.

[0229] The above description is only some or preferred embodiments of the present application, and neither the text nor the drawings can limit the scope of protection of the present application. Any equivalent structural transformation based on the content of the specification and drawings, or direct / indirect application in other related technical fields is included in the scope of protection of the present application.

Claims

A positioning method of a robot system, the robot system comprising a first robot and a second robot, the first robot being provided with a first detection system, the second robot being provided with a second detection system, the positioning method comprising: acquiring, by the first detection system, a first spatial relationship between a calibration reference and the first robot, and acquiring, by the second detection system, a second spatial relationship between the calibration reference and the second robot, wherein the calibration reference is within a detection range of the first detection system and the second detection system; determining a spatial relationship between the first robot and the second robot based on the first spatial relationship and the second spatial relationship. The positioning method according to claim 1, wherein, The calibration reference is provided on the first robot and / or the second robot. The positioning method according to claim 1, wherein, The robot system further comprises a calibration component provided independently of the first robot and the second robot, the calibration component being provided with the calibration reference. The positioning method according to claim 3, wherein, The calibration component has a base which is movable relative to the first robot and the second robot. The positioning method according to claim 3 or 4, wherein, The calibration component comprises a third detection system, the first robot and the second robot being within a detection range of the third detection system, the positioning method further comprising: acquiring, by the third detection system, a third spatial relationship between the first robot and the third detection system, and / or a fourth spatial relationship between the second robot and the third detection system; determining the spatial relationship between the first robot and the second robot based on the third spatial relationship and the second spatial relationship, or the fourth spatial relationship and the first spatial relationship. The positioning method according to claim 1, wherein, The calibration reference comprises a first calibration reference assembly, a second calibration reference assembly, and a connecting assembly, the first calibration reference assembly and the second calibration reference assembly being connected by the connecting assembly, the first calibration reference assembly being within a detection range of the first detection system, and the second calibration reference assembly being within a detection range of the second detection system. The positioning method according to claim 6, wherein, The first spatial relationship is a spatial relationship between the first calibration reference assembly and the first robot, and the second spatial relationship is a spatial relationship between the second calibration reference assembly and the second robot. The determination of the spatial relationship between the first robot and the second robot based on the first spatial relationship and the second spatial relationship comprises: determining a spatial relationship between the first calibration reference assembly and the second calibration reference assembly; determining the spatial relationship between the first robot and the second robot according to the first spatial relationship, the second spatial relationship, and the spatial relationship between the first calibration reference assembly and the second calibration reference assembly. The positioning method according to claim 7, wherein, The determination of the spatial relationship between the first calibration reference assembly and the second calibration reference assembly comprises: determining a shape parameter of the connecting assembly according to the spatial relationship between the first calibration reference assembly and the first robot, and the spatial relationship between the second calibration reference assembly and the second robot, when the calibration reference is in different postures. The shape parameter of the connecting assembly is determined according to the position information between the first calibration reference assembly and the second calibration reference assembly. The positioning method according to claim 7, wherein, The calibration reference further comprises a sensor for measuring the position information between the first calibration reference assembly and the second calibration reference assembly, and the determination of the spatial relationship between the first calibration reference assembly and the second calibration reference assembly comprises: The shape parameter of the connecting assembly is determined according to the position information between the first calibration reference assembly and the second calibration reference assembly measured by the sensor. The shape parameter of the connecting assembly is determined according to the position information between the first calibration reference assembly and the second calibration reference assembly. The positioning method according to any one of claims 6-9, wherein, The connecting assembly has a telescopic mechanism to enable the first calibration reference assembly to be within the detection range of the first detection system and the second calibration reference assembly to be within the detection range of the second detection system. The positioning method according to any one of claims 6-9, wherein, The connecting assembly is a telescopic rod, and the shape parameter of the connecting assembly comprises the length of the telescopic rod. The positioning method according to claim 11, wherein, The telescopic rod is a foldable telescopic rod, and the shape parameter of the connecting assembly further comprises the folding angle of the telescopic rod. The positioning method according to claim 1, wherein, The first detection system and the second detection system are optical positioning systems, and the calibration reference is a reflective ball jig; or, The first detection system and the second detection system are binocular cameras, and the calibration reference is a component provided with a structured image. A positioning method of a robot system, the robot system comprising a first robot and a second robot, a first positioning module being arranged on the first robot and a second positioning module being arranged on the second robot, the positioning method comprising: obtaining a first relative pose between the first positioning module on the first robot and the second positioning module when the second positioning module is at a first position, the second positioning module comprising an inertial sensor; obtaining a second relative pose between the first positioning module on the second robot and the second positioning module when the second positioning module is at a second position; determining a third relative pose of the second positioning module between the first position and the second position by using the inertial sensor; determining a relative pose between the first robot and the second robot based on the first relative pose, the second relative pose and the third relative pose. The positioning method according to claim 14, wherein, The first positioning module comprises a positioning mark, and the second positioning module further comprises a visual positioning module capable of capturing images; or, The first positioning module comprises a visual positioning module capable of capturing images, and the second positioning module further comprises a positioning mark. The positioning method according to claim 15, wherein The first relative pose between the first positioning module on the first robot and the second positioning module when the second positioning module is at a first position comprises: when the second positioning module is at the first position, a first image containing the positioning mark is obtained by the visual positioning module, and the first relative pose is determined based on the first image; The acquiring the second relative pose between the first positioning module on the second robot and the second positioning module when the second positioning module is at the second position comprises: acquiring a second image containing the positioning mark by the visual positioning module when the second positioning module is at the second position, and determining the second relative pose based on the second image. According to the positioning method in claim 16, wherein The acquiring the first image containing the positioning mark by the visual positioning module when the second positioning module is at the first position comprises: continuously detecting whether the positioning mark is included in the visual field range of the visual positioning module during the movement of the second positioning module from the first initial position to the first position, and triggering the visual positioning module to collect the first image if the positioning mark is included in the visual field range of the visual positioning module; The acquiring the second image containing the positioning mark by the visual positioning module when the second positioning module is at the second position comprises: continuously detecting whether the positioning mark is included in the visual field range of the visual positioning module during the movement of the second positioning module from the second initial position to the second position, and triggering the visual positioning module to collect the second image if the positioning mark is included in the visual field range of the visual positioning module. According to the positioning method in claim 16, wherein The acquiring the first image containing the positioning mark by the visual positioning module when the second positioning module is at the first position comprises: triggering the visual positioning module to collect the first image when a first trigger instruction input by a user is detected; The acquiring the second image containing the positioning mark by the visual positioning module when the second positioning module is at the second position comprises: triggering the visual positioning module to collect the second image when a second trigger instruction input by a user is detected. The positioning method according to claim 17 or 18, wherein, The determining the third relative pose of the second positioning module between the first position and the second position by the inertial sensor comprises: triggering the inertial sensor to start recording inertial sensor data when it is detected that the visual positioning module is triggered to collect the first image; triggering the inertial sensor to stop recording inertial sensor data when it is detected that the visual positioning module is triggered to collect the second image; determining the third relative pose of the second positioning module between the first position and the second position based on the inertial sensor data recorded by the inertial sensor during the period from the start of recording to the stop of recording. The positioning method according to claim 15, wherein, The second positioning module further comprises a communication unit capable of communicating with the first positioning module, and the positioning method further comprises: transmitting a synchronization positioning signal between the visual positioning module and the inertial sensor by the communication unit, the synchronization positioning signal being used to synchronize the image collection by the visual positioning module with the start or stop of recording inertial sensor data by the inertial sensor. The positioning method according to any one of claims 14-20, wherein, The second positioning module can be manually controlled by a user to move to the first position or the second position, or The robot system further comprises a third robot, and the second positioning module is arranged on a mechanical arm of the third robot and is capable of being controlled by the mechanical arm to move to the first position or the second position. The positioning method according to any one of claims 14-20, wherein, The first positioning module is arranged on a base of the first robot and the second robot respectively. The positioning method according to claim 14, wherein, The first positioning module is a visual positioning module capable of collecting images. The second positioning module comprises a first calibration reference component, a second calibration reference component and a connecting component, and the first calibration reference component and the second calibration reference component are connected through the connecting component. When the second positioning module is in the first position, the first calibration reference component is in the detection range of the visual positioning module of the first robot. When the second positioning module is in the second position, the second calibration reference component is in the detection range of the visual positioning module of the second robot. The positioning method according to claim 23, wherein, The first relative pose is the relative pose between the first calibration reference component and the first robot when the second positioning module is in the first position, the second relative pose is the relative pose between the second calibration reference component and the second robot when the second positioning module is in the second position, and the third relative pose is the relative pose between the first calibration reference component when the second positioning module is in the first position and the second calibration reference component when the second positioning module is in the second position. The determination of the third relative pose of the second positioning module between the first position and the second position by using the inertial sensor comprises: determination of a first sub-relative pose of the first calibration reference component between the first position and the second position; or determination of a second sub-relative pose of the second calibration reference component between the first position and the second position. The calibration reference further comprises a measurement sensor for measuring position information between the first calibration reference component and the second calibration reference component, and the determination of the third relative pose of the second positioning module between the first position and the second position by using the inertial sensor comprises: The positioning method according to claim 24, wherein, determination of a shape parameter of the connecting component by using the position information between the first calibration reference component and the second calibration reference component measured by the measurement sensor; determination of a sixth relative pose of the first calibration reference component and the second calibration reference component when the second positioning module is in the first position according to the shape parameter of the connecting component, and obtaining the third relative pose according to the sixth relative pose and the second sub-relative pose; or determination of a seventh relative pose of the first calibration reference component and the second calibration reference component when the second positioning module is in the second position according to the shape parameter of the connecting component, and obtaining the third relative pose according to the first sub-relative pose and the seventh relative pose. ​ A positioning method of a robot system, the robot system comprising a first robot and a second robot, a first positioning module being arranged on the first robot, the second robot being detachably connected with a second positioning module, the second positioning module comprising an inertial sensor, the positioning method comprising: acquiring a first relative pose between the first positioning module on the first robot and the second positioning module when the second positioning module is at a first position; determining a fourth relative pose of the second positioning module between the first position and a third position by using the inertial sensor, the third position being a position where the second positioning module is detachably connected on the second robot; determining a relative pose between the first robot and the second robot based on the first relative pose and the fourth relative pose. The positioning method according to claim 26, wherein, The third position is on a base of the second robot. The positioning method according to claim 26, wherein, The third position is a position other than the base on the second robot, and the positioning method further comprises: determining a fifth relative pose between the second positioning module and the second robot at the second position based on a kinematic relationship of a mechanical mechanism of the second robot; The determining of the relative pose between the first robot and the second robot based on the first relative pose and the fourth relative pose comprises: determining the relative pose between the first robot and the second robot based on the first relative pose, the fourth relative pose and the fifth relative pose. The positioning method according to claim 26, wherein, The first positioning module comprises a positioning mark, and the second positioning module further comprises a visual positioning module capable of acquiring images, or The first positioning module comprises a visual positioning module capable of acquiring images, and the second positioning module further comprises a positioning mark. A robot system comprising: a first robot; a second robot; and a control device, the control device comprising a processor and a memory, the memory being configured to store processor-executable instructions, the processor being configured to implement the positioning method of any one of claims 1 to 29. ​

Images