Method and apparatus for robotic system management

Abstract

Embodiments of the present disclosure provide methods, apparatuses, systems, and computer readable media for determining a position of a tracking device including a vision sensor and an inertial sensor. In the method, a first position of the tracking device relative to a reference marker is acquired based on images of the reference marker collected by the vision sensor. After movement of the tracking device, a second position of the tracking device relative to the first position is acquired by the inertial sensor. A position of the tracking device relative to the reference marker is determined based on the first position and the second position. Through these embodiments, position determination can be achieved in an accurate and efficient manner by a single tracking device.

Description

Technical Field

[0001] The exemplary embodiments of this disclosure generally relate to robot system management, and more specifically to methods, apparatus, systems, and computer-readable media for determining position via tracking devices in a robot system. Background Technology

[0002] With the development of computers and automatic control, robotic systems have been widely used in manufacturing to handle various types of work objects. For example, tools can be equipped at the end effector arms of a robotic system to handle work objects through operations such as cutting and grasping. Typically, a robotic system can have multiple robotic arms, each of which can rotate via corresponding joints at its end effector. The robotic system can be programmed to drive the tool along a path used to handle the work object based on demonstration programming (PBD) technology. However, PBD technology involves complex procedures and has shortcomings in position determination. Therefore, it is desirable to propose a more efficient solution for position determination, thereby enabling the management of the robotic system based on the determined position in an effective manner. Summary of the Invention

[0003] The exemplary embodiments disclosed herein provide solutions for robot system management.

[0004] In a first aspect, exemplary embodiments of this disclosure provide a method for determining the position of a tracking device, wherein the tracking device includes a visual sensor and an inertial sensor. The method includes: acquiring a first position of the tracking device relative to a reference marker based on an image of the reference marker collected by the visual sensor; acquiring a second position of the tracking device relative to the first position by the inertial sensor after movement of the tracking device; and determining the position of the tracking device relative to the reference marker based on the first and second positions. Through these embodiments, a tracking device including both a visual sensor and an inertial sensor is provided for position determination. In this respect, the tracking device can determine its position autonomously without the need for external sensors (such as signal sources that emit signals such as laser signals, magnetic signals, Wi-Fi signals, etc.). Furthermore, errors caused by obstacles blocking the signal source or interference with the signal source can be eliminated. Therefore, position determination can be achieved in a more convenient and efficient manner.

[0005] In some embodiments, obtaining the first position includes: obtaining the actual size of the reference marker; obtaining the virtual size of the reference marker in the image; and determining the first position based on the actual size, the virtual size, and the focus parameters of the vision sensor. Since the actual size and focus parameters of the reference marker are known in advance, and the virtual size can be easily measured by pixels included in the image, the first position can be determined in a simple and efficient manner.

[0006] In some embodiments, the method further includes: collecting an environmental image of the operating environment of the tracking device by a vision sensor after movement; and correcting a second position based on the environmental image and a previous environmental image collected by the vision sensor at a previous time point prior to the time point used to collect the environmental image. Through these embodiments, images collected during movement of the tracking device can further correct for potential errors in the inertial sensor. Therefore, the accuracy of position determination can be further improved.

[0007] In some embodiments, correcting the second position includes: identifying a pair of feature points from an environmental image and a previous environmental image, respectively, the pair of feature points corresponding to environmental features in the work environment, based on image analysis; determining an offset between the positions of the pair of feature points; and correcting the second position based on the offset. Through these embodiments, natural elements in the work environment, such as points and lines, can be used as environmental features to correct errors caused by inertial sensors. Since the environment includes a wealth of natural elements, the accuracy of position determination can be further improved effectively.

[0008] In some embodiments, a tracking device is attached to one of a tool deployed in a robotic system and a work object to be processed by the robotic system, and a reference marker is attached to the other of the tool and the work object. These embodiments allow for the accurate determination of the relative position between the tool and the work object to be processed by the tool.

[0009] In some embodiments, the method further includes determining a set of positions of a tracking device during the movement of the tool along a trajectory for handling a work object. Through these embodiments, the tracking device can be used to acquire a path for handling the work object, and thus an engineer can guide the tracking device along the trajectory. Compared to the conventional method of generating a path using a tracking tool and a signal source, the tracking device itself can efficiently and accurately record the tool path. Furthermore, the tool path can be converted into a robot path for driving the robotic system to handle the work object.

[0010] In some embodiments, the tracking device includes a telephone device, the visual sensor includes a camera equipped in the telephone device, and the inertial sensor includes an inertial measurement unit (IMU) equipped in the telephone device. Here, the telephone device can be reused as a tracking device, and therefore there is no need to design a dedicated tracking device. Through these embodiments, ordinary telephone devices such as smartphones can be used as tracking devices. Specifically, the camera device in the smartphone can function as a visual sensor for collecting images, and the IMU in the smartphone can function as an inertial sensor for determining the position of the smartphone.

[0011] In some embodiments, the method is implemented by an application installed in a telephone device, and the method further includes adjusting the processing parameters of the robot system via user input to the application. Through these embodiments, no changes are required to the hardware of the telephone device; only an application needs to be installed on the telephone device, which can then perform the aforementioned steps for location determination. Furthermore, engineers controlling the robot system can utilize the application to adjust the robot system's parameters. Compared to the conventional method of controlling a robot system by writing and running software code, this application provides a convenient way to manage the robot system.

[0012] In some embodiments, the reference marker includes a Quick Response (QR) code. Through these embodiments, QR codes can be used as reference markers. Compared to conventional measuring tools with large size and high precision, QR codes are easy to generate with smaller size and higher precision. Therefore, QR codes can facilitate location determination in a simple and efficient manner.

[0013] In a second aspect, exemplary embodiments of this disclosure provide a method for managing a robotic system via a tracking device, wherein the tracking device includes a vision sensor and an inertial sensor. The method includes: acquiring a first position of the tracking device relative to a tool marker based on an image of a tool marker collected by the vision sensor, the tool marker being attached to a tool deployed in the robotic system; acquiring a second position of the tracking device relative to the first position by the inertial sensor after movement of the tracking device, the tracking device being coupled to the tool during the movement; and determining a tool calibration position based on the first and second positions, the tool calibration position representing the position of the tracking device relative to the tool marker for calibrating the tool. Compared to conventional methods that calibrate tools by placing them in multiple poses within the robotic system during a complex calibration process, the proposed solution only requires attaching a tool marker to the tool and collecting measurements by the tracking device. The tool calibration position can then be automatically determined based on measurements from the vision sensor and the inertial sensor. Therefore, the workload for calibrating the tool can be significantly reduced.

[0014] In some embodiments, determining the tool calibration position includes: acquiring the position of a tool marker relative to the tool base of the robot system; and determining the tool calibration position based on the position of the tool marker and a first position and a second position. Through these embodiments, the position of the tool marker can be easily and accurately measured, and therefore the tool calibration position can be accurately determined.

[0015] In some embodiments, the method further includes: obtaining an object calibration position based on a robot marker and a work object marker, wherein the robot marker is attached to a tool, the work object marker is attached to a work object to be processed by the robot system, and the object calibration position represents the position of the work object marker relative to the frame base of the robot system for calibrating the work object. Compared to conventional methods that calibrate the work object by placing the robot system's arm in multiple poses around the work object using a calibration tool, the proposed solution only requires attaching the robot marker to the tool and the work object marker to the work object. The object calibration position can then be automatically determined. Therefore, the complexity level for calibrating the work object can be significantly reduced.

[0016] In some embodiments, obtaining the object calibration position includes: obtaining a third position of the tracking device relative to the robot marker based on an image of the robot marker collected by a vision sensor; obtaining a fourth position of the tracking device relative to the work object marker based on an image of the work object marker collected by the vision sensor after the tracking device has been moved to collect the image of the work object marker; obtaining a fifth position of the tracking device relative to the third position by an inertial sensor during the movement; and determining the object calibration position based on the position of the robot marker and the third, fourth, and fifth positions. Through these embodiments, the vision sensor can collect images of both the robot marker and the work object marker. Furthermore, the object calibration position can be automatically determined based on the images and measurements from the inertial sensor.

[0017] In some embodiments, the method further includes: acquiring multiple positions of a tracking device relative to a work object marker, the tracking device being coupled to a tool, and the multiple positions being determined during movement of the tool along a trajectory for handling the work object by a robotic system; and generating a tool path based on the multiple positions of the tracking device. Through these embodiments, the tracking device can be attached to a tool for handling the work object, and thus an engineer can guide the tool along a trajectory. Compared to the conventional method of generating a tool path from a tracking tool and a signal source, the tracking device itself can record the tool path in a more efficient and accurate manner. Furthermore, the tool path can be converted into a robotic path for driving the robotic system to handle the work object.

[0018] In some embodiments, acquiring multiple positions includes: acquiring a sixth position of the tracking device relative to the work object mark based on an image of the work object mark collected by a vision sensor; acquiring a seventh position of the tracking device relative to the sixth position by an inertial sensor during tool movement; and determining the position of the tracking device among multiple positions relative to the work object mark based on the sixth and seventh positions. Through these embodiments, a tracking device including both a vision sensor and an inertial sensor is provided for determining each position in a path. In this respect, the tracking device can determine its position autonomously without requiring an external signal source. Furthermore, errors in position determination caused by obstructions or interference with the signal source can be eliminated. Therefore, position determination can be achieved in a more convenient and efficient manner.

[0019] In some embodiments, the method further includes generating a robot path for handling a work object based on the tool calibration position, the object calibration position, and the tool path. Through these embodiments, the obtained tool path can be automatically converted into a robot path for controlling the movement of tools in the robotic system.

[0020] In some embodiments, the method further includes controlling the robot system to handle a work object based on a robot path. Through these embodiments, the robot system can drive the tool to precisely follow a trajectory, and thus the work object can be accurately processed into a desired shape.

[0021] In some embodiments, any of the tool markings, work object markings, and robot markings includes a Quick Response (QR) code. Through these embodiments, a QR code can be used as any of these markings. Compared to conventional calibration tools with large size and high precision, QR codes are easy to generate with smaller size and higher precision. Therefore, QR codes can facilitate calibration in a simple and efficient manner.

[0022] In a third aspect, an exemplary embodiment of this disclosure provides an apparatus for determining the position of a tracking device, the tracking device including a vision sensor and an inertial sensor. The apparatus includes: a first acquisition unit for acquiring a first position of the tracking device relative to a reference marker based on an image of the reference marker collected by the vision sensor; a second acquisition unit for acquiring a second position of the tracking device relative to the first position by the inertial sensor after movement of the tracking device; and a determination unit for determining the position of the tracking device relative to the reference marker based on the first position and the second position.

[0023] In some embodiments, the first acquisition unit includes: a real size acquisition unit for acquiring the real size of the reference mark; a virtual size acquisition unit for acquiring the virtual size of the reference mark in the image; and a position determination unit for determining a first position based on the real size, the virtual size, and the focusing parameters of the visual sensor.

[0024] In some embodiments, the apparatus further includes: a collection unit for collecting an environmental image of the working environment of the tracking device by a vision sensor after movement; and a correction unit for correcting a second position based on the environmental image and a previous environmental image, the previous environmental image being collected by the vision sensor at a previous time point prior to the time point used to collect the environmental image.

[0025] In some embodiments, the correction unit includes: an identification unit for identifying a pair of feature points from an environmental image and a previous environmental image based on image analysis, the pair of feature points corresponding to environmental features in the working environment; an offset determination unit for determining the offset between the positions of the pair of feature points; and a position correction unit for correcting a second position based on the offset.

[0026] In some embodiments, a tracking device is attached to one of a tool deployed in a robotic system and a work object to be handled by the robotic system, and a reference marker is attached to the other of the tool and the work object.

[0027] In some embodiments, the apparatus further includes a path determination unit for determining a set of positions of the tracking device during the movement of the tool along a trajectory for processing a work object.

[0028] In some embodiments, the tracking device includes a telephone device, the visual sensor includes a camera equipped in the telephone device, and the inertial sensor includes an inertial measurement unit equipped in the telephone device.

[0029] In some embodiments, the device is implemented by an application installed in a telephone device, and the device further includes an adjustment unit for adjusting the processing parameters of the robot system via user input to the application.

[0030] In some embodiments, the reference marker includes a Quick Response (QR) code.

[0031] In a fourth aspect, exemplary embodiments of the present disclosure provide an apparatus for managing a robot system according to embodiments of the present disclosure, wherein the tracking device includes a vision sensor and an inertial sensor. The apparatus includes: a first acquisition unit for acquiring a first position of the tracking device relative to a tool mark based on an image of a tool mark collected by the vision sensor, the tool mark being attached to a tool deployed in the robot system; a second acquisition unit for acquiring a second position of the tracking device relative to the first position by the inertial sensor after movement of the tracking device, the tracking device being coupled to the tool during the movement; and a determination unit for determining a tool calibration position based on the first and second positions, the tool calibration position representing the position of the tracking device relative to the tool mark for calibrating the tool.

[0032] In some embodiments, the determining unit includes: a tool position acquisition unit for acquiring the position of the tool mark relative to the tool base of the robot system; and a tool calibration determining unit for determining the tool calibration position based on the position of the tool mark and a first position and a second position.

[0033] In some embodiments, the apparatus further includes: an object acquisition unit for acquiring an object calibration position based on a robot marker and a work object marker, wherein the robot marker is attached to a tool, the work object marker is attached to a work object to be processed by the robot system, and the object calibration position represents the position of the work object marker relative to the frame base of the robot system for calibrating the work object.

[0034] In some embodiments, the object acquisition unit includes: a third acquisition unit for acquiring a third position of the tracking device relative to the robot marker based on an image of the robot marker collected by a vision sensor; a fourth acquisition unit for acquiring a fourth position of the tracking device relative to the work object marker based on an image of the work object marker collected by a vision sensor after the tracking device has been moved to collect the image of the work object marker; a fifth acquisition unit for acquiring a fifth position of the tracking device relative to the third position by an inertial sensor during the movement; and an object calibration determination unit for determining an object calibration position based on the position of the robot marker and the third, fourth, and fifth positions.

[0035] In some embodiments, the apparatus further includes: a position acquisition unit for acquiring a plurality of positions of the tracking device relative to a work object marker, the tracking device being coupled to the tool, and the plurality of positions being determined during movement of the tool along a trajectory for processing the work object by a robot system; and a generation unit for generating a tool path based on the plurality of positions of the tracking device.

[0036] In some embodiments, the position acquisition unit includes: a sixth acquisition unit for acquiring a sixth position of the tracking device relative to the work object mark based on an image of the work object mark collected by a vision sensor; a seventh acquisition unit for acquiring a seventh position of the tracking device relative to the sixth position by an inertial sensor during tool movement; and a path position determination unit for determining the position of the tracking device among a plurality of positions relative to the work object mark based on the sixth position and the seventh position.

[0037] In some embodiments, the apparatus further includes a generation unit for generating a robot path for processing a work object based on the tool calibration position, the object calibration position, and the tool path.

[0038] In some embodiments, the device further includes a control unit for controlling the robot system to process work objects based on robot paths.

[0039] In some embodiments, any one of the tool tag, work object tag, and robot tag includes a Quick Response (QR) code.

[0040] In a fifth aspect, exemplary embodiments of this disclosure provide a system for determining the location of a tracking device. The system includes a computer processor coupled to a computer-readable storage unit, the storage unit including instructions that, when executed by the computer processor, implement a method for determining the location of the tracking device.

[0041] In a sixth aspect, exemplary embodiments of the present disclosure provide a computer-readable medium having instructions stored thereon that, when executed on at least one processor, cause at least one processor to perform a method for determining the location of a tracking device.

[0042] In a seventh aspect, an exemplary embodiment of this disclosure provides a system for managing a robot system. The system includes a computer processor coupled to a computer-readable storage unit, the storage unit including instructions that, when executed by the computer processor, implement a method for managing the robot system.

[0043] In an eighth aspect, exemplary embodiments of the present disclosure provide a computer-readable medium having instructions stored thereon that, when executed on at least one processor, cause at least one processor to perform a method for managing a robotic system. Attached Figure Description

[0044] Figure 1A A schematic diagram is shown for determining the position along the trajectory used by a robotic system to process a work object;

[0045] Figure 1BA schematic diagram is shown for determining the position along the trajectory used by a robotic system to process a work object;

[0046] Figure 2 A schematic diagram of a process for determining the location of a tracking device according to an embodiment of the present disclosure is shown;

[0047] Figure 3 A schematic flowchart of a method for determining the location of a tracking device according to an embodiment of the present disclosure is shown;

[0048] Figure 4 A schematic diagram of a process for determining the location of a tracking device in an initial stage, according to an embodiment of the present disclosure, is shown.

[0049] Figure 5 A schematic diagram is shown illustrating a process for correcting a position acquired by an inertial sensor according to an embodiment of the present disclosure;

[0050] Figure 6 A schematic diagram is shown illustrating a process for determining the relative position between a tool and a work object according to an embodiment of the present disclosure;

[0051] Figure 7 A schematic diagram is shown illustrating a process for determining a tool path for processing a work object according to an embodiment of the present disclosure;

[0052] Figure 8 A schematic diagram is shown illustrating a process for controlling a robot system to handle work objects via a smartphone, according to an embodiment of the present disclosure.

[0053] Figure 9 A schematic flowchart of a method for managing a robot system via a tracking device according to an embodiment of the present disclosure is shown;

[0054] Figure 10 A schematic diagram illustrating a process for calibrating tools deployed in a robotic system according to an embodiment of the present disclosure is shown.

[0055] Figure 11 A schematic diagram is shown illustrating a process for calibrating a work object to be processed by a robotic system according to an embodiment of the present disclosure;

[0056] Figure 12 A schematic diagram of a process for obtaining a robot path for controlling a robot system, according to an embodiment of the present disclosure, is shown.

[0057] Figure 13A A schematic diagram of an apparatus for determining the location of a tracking device according to an embodiment of the present disclosure is shown;

[0058] Figure 13BA schematic diagram of an apparatus for managing a robot system according to an embodiment of the present disclosure is shown;

[0059] Figure 14 A schematic diagram of a system for implementing a method according to an embodiment of the present disclosure is shown.

[0060] Throughout the accompanying drawings, the same or similar reference numerals are used to indicate the same or similar elements. Detailed Implementation

[0061] The principles of this disclosure will now be described with reference to several exemplary embodiments illustrated in the accompanying drawings. Although exemplary embodiments of this disclosure are shown in the drawings, it should be understood that these embodiments are described merely to enable those skilled in the art to better understand and implement this disclosure, and not to limit the scope of this disclosure in any way.

[0062] In the robotics industry, robot programming refers to generating robot paths that guide a tool deployed at the end effector of a robotic system to handle a work object. PBD (Programming Path Generation) technology is one common solution for generating robot paths. PBD technology includes two modes: pass-through mode and non-touch mode. (See reference...) Figure 1A To describe the pass-through mode. In Figure 1A In this process, tool 140 is attached to the end effector arm of robot system 130 to handle workpiece 110. Tool 140, along with the robot arm, is directly moved by the engineer to guide it along a desired trajectory (including, for example, P1, P2, P3, P4, etc.). At this point, the arm's pose is recorded, and a robot path can then be generated to repeat the recorded pose. However, normal operation of robot system 130 should be interrupted by the above steps, and the engineer must enter the robot's work area to move the arm. For industrial robot systems, this straight-through mode reduces productivity and also presents serious safety issues.

[0063] In non-touch mode, the engineer holds a special tracking tool, rather than the robotic arm, without needing to enter the robot's work area. Typically, the engineer moves the tracking tool along a desired trajectory, and sensors around the tool can be used to track and record its posture. (Reference) Figure 1B To describe the non-touch mode. Figure 1BIn this process, the engineer can hold a tracking tool 122 and move along a trajectory used to handle the work object 110. One or more sensors 120 (such as laser sensors, magnetic sensors, Wi-Fi sensors, vision sensors, etc.) can be pre-deployed, and the sensors 120 should be carefully calibrated before the tracking tool 122 reaches positions P1, P2, P3, P4, etc. However, the sensors will not function when the tracking tool 122 is far away or when obstacles obstruct communication between the sensors 120 and the tracking tool 122. Furthermore, if a magnetic sensor is used for tracking, the operating range of the magnetic sensor is limited and it may be affected by ferromagnetic materials.

[0064] To at least partially address the aforementioned and other potential problems, embodiments of this disclosure provide a novel method for location determination. Typically, a handheld tracking device comprising both visual and inertial sensors is provided for tracking. Since all sensors are embedded within the tracking device, no auxiliary equipment is required around it. Therefore, the tracking device can operate over a large area. Furthermore, the tracking device will not be affected by ferromagnetic interference and / or obstacles in the working environment.

[0065] For the purpose of providing a brief description of this disclosure, reference will be made to... Figure 2 . Figure 2 A schematic diagram of a process 200 for determining the location of a tracking device according to an embodiment of the present disclosure is shown. Figure 2 In this system, the tracking device 210 includes a vision sensor 212 and an inertial sensor 214. Additionally, a reference marker 220 is provided to facilitate position determination. Here, the reference marker 220 may define coordinates used for position determination; that is, the determined position is relative to the reference marker 220. For example, if it is desired to determine the position of the tracking device 210 relative to a work object 110 to be processed by the robot system 130, the reference marker 220 may be attached to the work object 110.

[0066] An engineer can hold the tracking device 210 at point 232 and scan the reference marker 220. At this point, a first position 230 of the tracking device 210 relative to the reference marker 220 can be determined based on an image of the reference marker 220 collected by the vision sensor 212. Furthermore, the tracking device 210 can move to point 242, and the inertial sensor 214 can measure a second position 240 of the tracking device 210 relative to the first position 230. Additionally, a position 250 of the tracking device 210 relative to the reference marker 220 can be determined. Through these embodiments, the tracking device 210 can determine its position autonomously without the need for external sensors (such as signal sources that emit signals such as laser signals, magnetic signals, Wi-Fi signals, etc.). Furthermore, errors caused by obstacles between conventional tracking tools and sensors, or interference with signal sources, can be eliminated.

[0067] In the following text, reference will be made to Figure 3 To learn more details about the location determination. Figure 3 A schematic flowchart of a method for determining the position of a tracking device according to an embodiment of the present disclosure is shown. At block 310, a first position 230 of the tracking device 210 relative to the reference marker 220 is obtained based on an image of the reference marker 220 collected by the vision sensor 212. Here, the reference marker 220 is used to define reference coordinates for position determination. The reference marker 220 may be attached to the work object 110 when determining a robot path for handling the work object 110.

[0068] In some embodiments, the reference marker 220 can be attached to any location on the workpiece 110, for example, it can be attached to a corner that will not affect the processing. In some embodiments, the origin of the reference coordinates can be defined at a corner of the reference marker 220, at the center of the reference marker 220, or at another location associated with the reference marker 220. In some embodiments, the axis orientation of the reference coordinates relative to the reference marker 220 can be predetermined. For example, the x-axis and y-axis can be defined along the edge of the reference marker 220, and a right-handed coordinate system can be used.

[0069] In some embodiments, reference marker 220 may include a Quick Response (QR) code. Through these embodiments, a QR code can be used as reference marker 220. Compared to conventional measuring tools with large size and high precision, QR codes are easy to generate with small size and high precision. Therefore, QR codes can facilitate position determination in a simple and efficient manner. Furthermore, the offset between reference marker 220 and reference coordinates can be encoded in the QR code. Once the vision sensor 212 scans the QR code, the offset can be read, and then the reference coordinates can be generated in a simple and efficient manner.

[0070] The following text will refer to Figure 4 Learn more about determining the first position 230. Figure 4 A schematic diagram of a process 400 for determining the position of a tracking device in an initial stage according to an embodiment of the present disclosure is shown. First, the actual size 410 of a reference marker 220 can be obtained. Here, the actual size 410 can be measured by a measuring device with higher precision; for example, the actual size 410 can be expressed in units such as millimeters or micrometers. A vision sensor 212 can collect an image 420 of the reference marker 220 while scanning the reference marker 220. A virtual size 422 of the reference marker 220 in the image 420 can be determined. For example, the outline of the reference marker 220 can be identified from the image 420, and the width and length of the pixels within the outline can be taken into account as the virtual size 422. Furthermore, multiple focus parameters 430 of the vision sensor 212 can be obtained, and therefore a first position 440 can be determined based on the geometric relationship between the actual size 410, the virtual size 422, and the focus parameters 430 of the vision sensor 212. Since the actual size of the reference marker and the focus parameters are known in advance, and the virtual size can be easily measured by pixels included in the image, the first position can be determined in a simple and efficient manner.

[0071] At box 320, after the tracking device 210 moves, the inertial sensor 214 acquires a second position 240 of the tracking device 210 relative to the first position 230. Here, the inertial sensor 214 can measure the specific force, angular rate, and orientation of the tracking device 210 using a combination of accelerometers, gyroscopes, and magnetometers. Inertial navigation technology has been developed for many years, and therefore the second position 240 can be accurately determined. Furthermore, images collected by the visual sensor 212 during movement can be used to correct for potential errors caused by the inertial sensor 214.

[0072] In some embodiments, the visual sensor 212 can collect images of the operating environment of the tracking device 210 during movement. Here, the image collection rate can be much lower than that of the inertial sensor 214. If the visual sensor 212 is not obstructed by obstacles, the collected images can be used to correct a second position 240. Specifically, the second position 2400 can be based on an environmental image and a previous environmental image collected by the visual sensor 212 at a previous time point prior to the time point used to collect the environmental image. Through these embodiments, images collected by the tracking device 210 during movement can further correct for potential errors in the inertial sensor 214. Therefore, the accuracy of position determination can be further improved.

[0073] For more details, please refer to [link / reference]. Figure 5 . Figure 5A schematic diagram of a process 500 for correcting a position acquired by an inertial sensor according to an embodiment of the present disclosure is shown. In the correction step, the image is not required to include reference marker 220. Rather, the image may or may not include reference marker 220, as long as the image includes the same environmental features in the working environment. Figure 5 An environmental image 510 collected when the tracking device 210 is at point 242 is shown, as well as a previous environmental image 520 collected before the tracking device 210 reaches point 242. Both images 510 and 520 include the same environmental features (e.g., corners, lines, or other shapes of objects in the work environment). Based on image analysis, a pair of feature points 512 and 522 can be identified from the environmental image 510 and the previous environmental image 520, respectively, where the pair of feature points corresponds to the same environmental features in the work environment.

[0074] Furthermore, the offset between the positions of the pair of feature points 512 and 522 can be determined, and the second position 240 can then be corrected based on this offset. If the offset indicates that the tracking device 210 has moved 20 mm, while the second position 240 measured by the inertial sensor 214 is only 10 mm, the second position can be increased. For example, the second position 240 can be set based on 15 mm, which is the average of 20 mm and 10 mm. In another example, if the distance determined based on the offset is significantly lower than the distance determined by the inertial sensor 214, the second position 240 can be decreased. Through these embodiments, natural factors in the environment, such as points, lines, etc., can be used as environmental features to correct for errors caused by the inertial sensor 214. Since the environment includes a wealth of natural factors, the accuracy of position determination can be further improved in a simple and effective manner.

[0075] Refer again Figure 3 At box 330, the position 250 of the tracking device 210 relative to the reference mark 220 is determined based on the first position 230 and the second position 240. Here, the position 250 can be determined based on the sum of the first position 230 and the second position 240, so the position 250 of the tracking device 210 relative to the reference mark 220 can be accurately obtained by the tracking device 210 itself. Through these embodiments, a tracking device including both a visual sensor and an inertial sensor is provided for position determination. In this respect, the tracking device can determine its position independently without the need for external sensors (such as signal sources for emitting signals such as laser signals, magnetic signals, Wi-Fi signals, etc.). Furthermore, errors in position determination caused by obstacles between the tracking device and the sensor or interference with the signal source can be eliminated.

[0076] The preceding paragraphs described the steps for determining the position 250 of the tracking device 210 relative to the reference marker 220. Furthermore, the tracking device 210 can be used to measure the relative position between the tool 140 and the work object 110. Specifically, the tracking device 210 can be attached to one of the tool 140 deployed in the robot system 130 and the work object 110 to be processed by the robot system 130. Simultaneously, the reference marker 220 can be attached to the other of the tool 140 and the work object 110. Through these embodiments, the relative position between the tool 140 and the work object 110 can be determined efficiently.

[0077] The following text will refer to Figure 6 To learn more about the relative position obtained based on the aforementioned tracking device 210. Figure 6 A schematic diagram of a process 600 for determining the relative position between a tool and a work object according to an embodiment of the present disclosure is shown. Figure 6 In this process, reference mark 220 can be attached to work object 110, and tracking device 210 can be attached to tool 140. Therefore, tracking device 210 can accurately determine the relative position between tracking device 210 and reference mark 220. Although Figure 6 Only the case where the tracking device 210 is attached to the tool 140 and the reference mark 220 is attached to the work object 110 is shown, but the tracking device 210 can be attached to the work object 110 and the reference mark 220 can be attached to the tool 140. Relative positions can be determined in a similar manner, and details are omitted below.

[0078] Furthermore, the engineer can move the tool 140 along the trajectory used to process the work object 110, and thus the tracking device 210 can determine a set of positions of the tracking device 210 relative to the reference 220 during the movement. Figure 7 A schematic diagram of a process 700 for determining a tool path for processing a work object, according to an embodiment of the present disclosure, is shown. Figure 7 In this context, trajectory 710 represents the trajectory used to process workpiece 110. For example, trajectory 710 can be represented by a curve on workpiece 110 used to process workpiece 110 into a desired shape. At this point, the engineer can hold tool 140 and guide the tip of tool 140 to follow trajectory 710, and therefore, tracking device 210 can output tool path 720 based on the above method.

[0079] Through these embodiments, the tracking device can be attached to a tool used to handle a work object, allowing engineers to guide the tool along the handling path. Compared to the conventional method of generating tool paths based on pass-through and non-touch modes, the tracking device itself can accurately record the tool path.

[0080] To date, various types of smartphones are equipped with high-precision cameras and IMUs. Therefore, the tracking device 210 can be implemented using the phone device itself, the visual sensor 212 can be implemented using the camera integrated into the phone device, and the inertial sensor 214 can be implemented using the IMU configured within the phone device. Here, there is no need to design a dedicated tracking device. Through these embodiments, ordinary phone devices, such as smartphones, can be used as tracking devices. Specifically, the camera device in the smartphone can function as a visual sensor for collecting images, and the IMU in the smartphone can function as an inertial sensor for determining the smartphone's position. Therefore, the cost of the tracking device can be significantly reduced.

[0081] In some embodiments, an application may be developed and installed in a telephone device to implement method 300. Figure 8 A schematic diagram of a process 800 for handling work objects via a smartphone-controlled robot system, according to an embodiment of the present disclosure, is shown. Figure 8 In this embodiment, application 812 is installed in smartphone 810 to implement the aforementioned method 300. An engineer can attach reference mark 220 to work object 110 and attach smartphone 810 to tool 140 to generate tool path 720. Furthermore, application 812 can convert tool path 720 into robot path 820 for controlling robot system 130. Specifically, robot path 820 can be input into robot controller 830 to drive tool 140 to process work object 110 into a desired shape.

[0082] In some embodiments, application 812 can display robot path 820 to an engineer for further control. For example, robot path 820 may be shown in Table 1 below.

[0083] Table 1 Robot Path

[0084] 1 2 ……

[0085] In Table 1, the "XYZ" column represents the (x, y, z) position at a specific time point, the "Angle" column represents the orientation angle (angle 1, angle 2, angle 3) at a specific time point, and the "Parameter" column represents the control parameters of the robot system 130. For example, "Parameter 1" could represent the rotational speed of the tool 140, "Parameter 2" could represent whether the tool 140 contacts the work object 110, and so on. Engineers can read the robot path 820 from the smartphone 810 and adjust the robot system's processing parameters via user input to the application 812. For example, engineers can run the robot path 820 in a simulation program and check whether the posture of each arm in the robot system can be optimized. Then, engineers can optimize the parameters via the application 812 and input the optimized robot path into the robot controller 830.

[0086] These embodiments eliminate the need for any hardware modifications to the telephone equipment; instead, an application is installed on the equipment, which then performs the aforementioned steps for location determination. Furthermore, engineers controlling the robot system can utilize this application to adjust its parameters. Compared to the conventional method of controlling the robot system 130 by writing and running software code, this application provides a convenient way to manage the robot system. Although... Figure 8 The application 812 outputs robot path 820, but alternatively and / or additionally, tool path can be input into robot controller 830 and robot controller 830 can generate robot path 820.

[0087] In the initial phase of starting the robot system 130, the positions of the tool 140 and the workpiece 110 should be calibrated. Various calibration solutions have been proposed; for example, tool calibration involves aligning the tool to a user-defined position relative to the tool base of the robot system (also referred to as "tool 0," and which is a known parameter of the robot system). In conventional tool calibration, the tool 140 is moved to at least four different positions to collect values ​​for tool calibration. However, achieving high accuracy in tool calibration is difficult if the tool does not have a pointed structure. Even with a pointed structure, engineers require extensive experience and skill to place the tool in the desired position.

[0088] To at least partially address the aforementioned and other potential problems, a novel method for managing robotic systems is provided. Specifically, the tool can be calibrated using a tracking device 210 according to an embodiment of this disclosure. Further details will be referenced. Figure 9 The figure illustrates a schematic diagram of a process 900 for calibrating tools deployed in a robotic system according to an embodiment of the present disclosure. Figure 9In this configuration, tool marker 910 can be attached to tool 140 to indicate the coordinates for tool calibration. First, the engineer can hold the tracking device 210 at a specific point (e.g., ...). Figure 9 As shown in point 1), the tool mark 910 is scanned. At this point, the first position of the tracking device relative to the tool mark 910 can be obtained. Furthermore, the tracking device 210 can be positioned at another point (e.g., ...). Figure 9 The tool is attached to point 2 (as shown in the figure), and then the tool calibration position can be determined based on measurements from the tracking device 210.

[0089] Figure 10 A flowchart of a method 1000 for managing a robot system by a tracking device according to an embodiment of the present disclosure is shown. At block 1010, a first position of the tracking device 210 relative to the tool mark 910 is obtained based on an image of the tool mark 910 collected by a vision sensor 212. Here, the tool mark 910 is attached to a tool 140 deployed in the robot system 130. Detailed steps for obtaining the first position of the tracking device 210 are described below with reference to [reference needed]. Figure 4 The described steps are similar. Specifically, the actual size and virtual size of the tool mark 910 can be obtained, and therefore, the first position of the tracking device 210 can be determined based on the actual size, the virtual size, and the focusing parameters of the vision sensor 212 in the tracking device 210. Here, the first position of the tracking device 210 relative to the tool mark 910 can be determined by... instruct.

[0090] At box 1020, after the tracking device 210 moves, the inertial sensor 214 acquires a second position of the tracking device relative to the first position. Here, the tracking device is coupled to the tool during movement. In other words, the tracking device 210... Figure 9 The device moves from point 1 to point 2, and the second position can be directly measured by the inertial sensor 214. Here, the second position of the tracking device 210 relative to the first position can be determined by... instruct.

[0091] At box 1030, a tool calibration position is determined based on a first position and a second position. The tool calibration position represents the position of the tracking device 210 relative to the tool mark 910 for calibrating the tool 140. In some embodiments, the position of the tool mark 910 relative to the tool base of the robot system 130 can be obtained for tool calibration. In the robot system 130, the relationship between the tool mark 910 and the tool base (determined by…) is… The indication can be predefined by the physical characteristics of the tool mark 910, and therefore The tool calibration has known values. Furthermore, the tool calibration position can be based on the position of the tool markings, as well as a first position and a second position. In some embodiments, the following Formula 1 can be used to determine the tool calibration position.

[0092]

[0093] in Indicates the tool calibration position for calibrating the tool 910 deployed at the end effector of the robot system 130. Indicates the relationship between tool mark 910 and tool base. Indicates the first position of the tracking device 210 relative to the tool mark 910, and The tracking device 210 is positioned at a second position relative to the first position. In Formula 1 above, These are known parameters in robot system 130. Measured by vision sensor 212, Measured by inertial sensor 214, and therefore, the tool calibration position. It can be determined effectively and accurately.

[0094] Compared to conventional methods that calibrate tool 140 by placing it in multiple orientations during a complex calibration process, the proposed solution only requires attaching tool marker 910 to tool 140 and collecting measurements from tracking device 210. The tool calibration position can then be automatically determined based on measurements from vision sensor 212 and inertial sensor 214. Therefore, the workload for calibrating tool 140 can be significantly reduced.

[0095] In some embodiments, the work object 110 can be calibrated by attaching a robot marker to the tool 140 and a work object marker to the work object 110 to be processed by the robot system 130. The tracking device 210 can scan the tool marker and the work object marker separately to obtain an object calibration position for work object calibration based on measurements taken by the tracking device 210. Here, the object calibration position represents the position of the work object marker relative to the frame base of the robot system (also referred to as "base 0," and it is a known parameter of the robot system), and therefore can be used to calibrate the work object 110. Compared to the conventional method of calibrating the work object by placing the robot system's arm in multiple poses around the work object using a calibration tool, the proposed solution only requires attaching a robot marker to the tool and a work object marker to the work object. The object calibration position can then be automatically determined. Therefore, the complexity level for calibrating the work object can be significantly reduced.

[0096] The following text will refer to Figure 11To learn more about the calibration of work objects. Figure 11 A schematic diagram of a process 1100 for calibrating a work object to be processed by a robotic system, according to an embodiment of the present disclosure, is shown. Figure 11 In this process, the tracking device 210 can be placed at point 3 to scan the robot marker 1110 on the tool 140. At this point, a third position of the tracking device 210 relative to the robot marker 1110 can be determined based on the image of the robot marker 1110 collected by the vision sensor 212. Detailed steps for obtaining the third position of the tracking device 210 are described below with reference. Figure 4 The described steps are similar. Specifically, the actual and virtual dimensions of the robot marker 1110 can be obtained, and therefore, the third position of the tracking device 210 can be determined based on the actual size, the virtual size, and the focusing parameters of the vision sensor 212. Here, the third position of the tracking device 210 relative to the robot marker 1110 can be determined by... instruct.

[0097] Next, the engineer can move the tracking device 210 to... Figure 11 Point 4 is used to scan the work object mark 1120 on the work object 110. At this point, the fourth position of the tracking device 210 relative to the work object mark 1120 can be determined based on the image of the work object mark 1120 collected by the vision sensor 212 after the movement of the tracking device. Here, the tracking device is moved to point 4 to collect the image of the work object mark 1120. Specifically, the real size and virtual size of the work object mark 1120 can be acquired, and therefore, the fourth position of the tracking device 210 can be determined based on the real size, the virtual size, and the focusing parameters of the vision sensor 212. Here, the fourth position of the tracking device 210 relative to the work object mark 1120 can be determined by... instruct.

[0098] During the movement of tracking device 210 from point 3 to point 4, a fifth position of tracking device 210 relative to the third position can be determined by inertial sensor 214 after the movement of tracking device 210. Here, the fifth position can be directly measured by inertial sensor 214. The object calibration position can then be based on the position of robot marker 1110 and the third, fourth, and fifth positions. In some embodiments, the following formula 2 can be used to determine the object calibration position.

[0099]

[0100] in Indicates the object calibration position for calibrating the work object 110 to be processed by the robot system 130. Indicates the position of robot marker 1110, which can be read from robot system 130. The tracking device 210 is positioned relative to the robot marker 1110 at a third location. The tracking device 210 is positioned at a fourth location relative to the work object marker 1120, and The tracking device 210 is positioned at a fifth position relative to the third position. In Formula 2 above, It can be read directly from robot system 130, and at the same time... and Measurements can be taken by the vision sensor 212 and the inertial sensor 214. Therefore, the object calibration position can be determined efficiently and accurately. Through these embodiments, the tracking device 210 can collect measurements for object calibration, and thus, the object calibration position can be automatically determined based on the collected measurements.

[0101] It should be understood that tool calibration and workpiece calibration can be performed independently. Although the above paragraphs describe tool calibration prior to workpiece calibration, tool calibration can be performed after workpiece calibration or at the same time as workpiece calibration.

[0102] In some embodiments, any of the tool mark 910, work object mark 1120, and robot mark 1110 includes a QR code. Through these embodiments, a QR code can be used as any of these marks. Compared to conventional calibration tools with large size and high precision, QR codes are easy to generate with smaller size and higher precision. Therefore, QR codes can facilitate calibration in a simple and efficient manner.

[0103] In some embodiments, a tool path for processing work object 110 can be obtained. Here, the tool path can be obtained in a conventional manner (e.g., using...). Figure 1B(The tracking tool 122 and sensor 120 are shown). Alternatively and / or additionally, the tool path can be acquired by the tracking device 210. Specifically, when the tracking device 210 can be attached to the tool 140 to determine the tool path relative to the work object mark 1120 attached to the work object 110, the engineer can hold the tool 140 with the tracking device 210 attached thereto and move the tool 140 along a trajectory for the robot system 130 to process the work object 110. At this point, multiple positions of the tracking device 210 relative to the work object mark 1120 can be acquired. The tool path 720 can then be generated based on the multiple positions of the tracking device 210. Through these embodiments, the tracking device can be attached to the tool for processing the work object, and therefore, the engineer only needs to guide the tool along the trajectory and generate the tool path. Compared to the conventional method of generating the tool path by a pair of tracking tools and a signal source, the tracking device itself can accurately record the tool path.

[0104] In some embodiments, the detailed steps for determining a position among multiple positions are similar to those implemented by method 300. Initially, tracking device 210 may scan work object marker 1120 to obtain a sixth position of tracking device 210 relative to work object marker 1110. At this point, it can be based on... Figure 4 The process illustrated involves determining a sixth position from an image of the workpiece marker 1120. Furthermore, during the movement of the tool 140, the inertial sensor 214 can directly measure a seventh position of the tracking device 210 relative to the sixth position. Each of a plurality of positions of the tracking device can then be determined based on the sixth and seventh positions. Through these embodiments, a tracking device incorporating both visual and inertial sensors is provided for position determination. In this respect, the tracking device can determine its position autonomously without requiring an external signal source. Furthermore, errors in position determination caused by obstructions or interference with the signal source can be eliminated. Therefore, position determination can be performed in a more convenient and efficient manner.

[0105] It should be understood that the robot path can control the movement of the end effector arm, and the tool 140 deployed at the end effector arm can move together with the end effector arm. Therefore, the tool calibration position can be taken into account when generating the robot path. Furthermore, since the object calibration position represents the relationship between the work object marker 1120 and the frame base of the robot system 130, the object calibration position is also taken into account when generating the robot path. Therefore, a robot path for handling work objects can be generated based on the tool calibration position, the object calibration position, and the tool path.

[0106] Figure 12A schematic diagram of a process 1200 for obtaining a robot path for controlling a robot system, according to an embodiment of the present disclosure, is shown. Figure 12 As shown, the tool path 720 can be converted into the robot path 1210 based on the tool calibration position and the object calibration position. In particular, the following formula 3 can be used.

[0107]

[0108] in Instructions for the robot path used to control the robot system 130. Indicates the object calibration position for calibrating the work object 110 to be processed by the robot system 130. Indicates the tool calibration position for calibrating the tool 910 deployed at the end effector of the robotic system 130, and The tool path 720 is determined by the tracking device 210. Since all the parameters on the right side of Equation 3 are known, the tool path 720 can be automatically converted into the robot path 1210 to control the movement of the tool 140 in the robot system 130.

[0109] In some embodiments, the method further includes controlling the robot system 130 to process the work object 110 based on the robot path 1210. At this point, the tool 140 at the end effector of the robot system 130 can process the work object 110 into a desired shape. Through these embodiments, the robot system 130 can drive the tool 130 to precisely follow a trajectory, allowing the work object to be handled accurately as needed. Furthermore, embodiments of the invention can reduce the cost of deploying the robot system and the technical requirements for engineers. By using the tracking device 210, engineers can attach markers to the tool 140 and the work object 110, and collect measurements via the tracking device 210. Then, both the tool 140 and the work object 110 can be automatically calibrated based on the collected measurements before the tool 140 begins processing the work object 110.

[0110] The preceding paragraphs have described the detailed steps of methods 300 and 1000. In some embodiments of this disclosure, methods 300 and 1000 can be implemented by corresponding devices. Figure 13A A schematic diagram of a location determination apparatus 1300A for determining the location of a tracking device according to an embodiment of the present disclosure is shown. Figure 13AAs shown, device 1300A is used to determine the position of a tracking device, which includes a vision sensor and an inertial sensor. Device 1300A includes: a first acquisition unit 1310A for acquiring a first position of the tracking device relative to a reference marker based on an image of the reference marker collected by the vision sensor; a second acquisition unit 1320A for acquiring a second position of the tracking device relative to the first position by the inertial sensor after the tracking device has moved; and a determination unit 1330A for determining the position of the tracking device relative to the reference marker based on the first position and the second position.

[0111] In some embodiments, the first acquisition unit 1310A includes: a real size acquisition unit for acquiring the real size of the reference mark; a virtual size acquisition unit for acquiring the virtual size of the reference mark in the image; and a position determination unit for determining a first position based on the real size, the virtual size, and the focusing parameters of the visual sensor.

[0112] In some embodiments, the device 1300A further includes: a collection unit for collecting an environmental image of the working environment of the tracking device by a vision sensor after movement; and a correction unit for correcting a second position based on the environmental image and a previous environmental image collected by the vision sensor at a previous time point prior to the time point used to collect the environmental image.

[0113] In some embodiments, the correction unit includes: an identification unit for identifying a pair of feature points from an environmental image and a previous environmental image based on image analysis, the pair of feature points corresponding to environmental features in the working environment; an offset determination unit for determining the offset between the positions of the pair of feature points; and a position correction unit for correcting a second position based on the offset.

[0114] In some embodiments, a tracking device is attached to one of a tool deployed in a robotic system and a work object to be handled by the robotic system, and a reference marker is attached to the other of the tool and the work object.

[0115] In some embodiments, the apparatus 1300A further includes a path determination unit for determining a set of positions of the tracking device during the movement of the tool along a trajectory for processing a work object.

[0116] In some embodiments, the tracking device includes a telephone device, the visual sensor includes a camera equipped in the telephone device, and the inertial sensor includes an inertial measurement unit equipped in the telephone device.

[0117] In some embodiments, the device 1300A is implemented by an application installed in a telephone device, and the device further includes an adjustment unit for adjusting the processing parameters of the robot system via user input to the application.

[0118] In some embodiments, the reference marker includes a Quick Response (QR) code.

[0119] Figure 13B A schematic diagram of a robot management device 1300B for managing a robot system according to an embodiment of the present disclosure is shown. The tracking device includes a vision sensor and an inertial sensor. The device 1300B includes: a first acquisition unit 1310B for acquiring a first position of the tracking device relative to a tool mark based on an image of a tool mark collected by the vision sensor, the tool mark being attached to a tool deployed in the robot system; a second acquisition unit 1320B for acquiring a second position of the tracking device relative to the first position by the inertial sensor after movement of the tracking device, the tracking device being coupled to the tool during movement; and a determination unit 1330B for determining a tool calibration position based on the first and second positions, the tool calibration position representing the position of the tracking device relative to the tool mark for calibrating the tool.

[0120] In some embodiments, the determining unit 1330B includes: a tool position acquisition unit for acquiring the position of the tool mark relative to the tool base of the robot system; and a tool calibration determining unit for determining the tool calibration position based on the position of the tool mark and a first position and a second position.

[0121] In some embodiments, the apparatus 1300B further includes an object acquisition unit for acquiring an object calibration position based on a robot marker and a work object marker, wherein the robot marker is attached to a tool, the work object marker is attached to a work object to be processed by the robot system, and the object calibration position represents the position of the work object marker relative to the frame base of the robot system for calibrating the work object.

[0122] In some embodiments, the object acquisition unit includes: a third acquisition unit for acquiring a third position of the tracking device relative to the robot marker based on an image of the robot marker collected by a vision sensor; a fourth acquisition unit for acquiring a fourth position of the tracking device relative to the work object marker based on an image of the work object marker collected by a vision sensor after the tracking device has been moved to collect the image of the work object marker; a fifth acquisition unit for acquiring a fifth position of the tracking device relative to the third position by an inertial sensor during the movement; and an object calibration determination unit for determining an object calibration position based on the position of the robot marker and the third, fourth, and fifth positions.

[0123] In some embodiments, the apparatus 1300B further includes: a position acquisition unit for acquiring a plurality of positions of the tracking device relative to a work object marker, the tracking device being coupled to the tool, and the plurality of positions being determined during movement of the tool along a trajectory for processing the work object by a robot system; and a generation unit for generating a tool path based on the plurality of positions of the tracking device.

[0124] In some embodiments, the position acquisition unit includes: a sixth acquisition unit for acquiring a sixth position of the tracking device relative to the work object mark based on an image of the work object mark collected by a vision sensor; a seventh acquisition unit for acquiring a seventh position of the tracking device relative to the sixth position by an inertial sensor during tool movement; and a path position determination unit for determining the position of the tracking device among a plurality of positions relative to the work object mark based on the sixth position and the seventh position.

[0125] In some embodiments, the apparatus 1300B further includes a generation unit for generating a robot path for processing a work object based on the tool calibration position, the object calibration position, and the tool path.

[0126] In some embodiments, the apparatus 1300B further includes a control unit for controlling the robot system to process work objects based on robot paths.

[0127] In some embodiments, any of the tool tag, work object tag, and robot tag includes a Quick Response (QR) code.

[0128] In some embodiments of this disclosure, a system for implementing the methods 300 and 1000 described above is provided. Figure 14 A schematic diagram of a system 1400 for implementing a method according to an embodiment of the present disclosure is shown. The system 1400 includes a computer processor 1410 coupled to a computer-readable storage unit 1420, the storage unit 1420 including instructions 1422 that, when executed by the computer processor 1410, implement method 300 or 1000.

[0129] In some embodiments of this disclosure, a computer-readable medium for managing a camera system is provided. The computer-readable medium stores instructions thereon, and these instructions, when executed on at least one processor, can cause at least one memory to perform the methods for managing the camera system as described in the preceding paragraphs, details of which will be omitted below.

[0130] Generally, the various embodiments of this disclosure can be implemented in hardware or dedicated circuitry, software, logic, or any combination thereof. Some aspects can be implemented in hardware, while others can be implemented in firmware or software, which can be executed by a controller, microprocessor, or other computing device. Although various aspects of the embodiments of this disclosure are shown and described as block diagrams, flowcharts, or using some other illustrations, it should be understood that, as non-limiting examples, the blocks, apparatuses, systems, techniques, or methods described herein can be implemented in hardware, software, firmware, dedicated circuitry or logic, general-purpose hardware or controllers or other computing devices, or some combination thereof.

[0131] This disclosure also provides at least one computer program product tangibly stored on a non-transitory computer-readable storage medium. The computer program product includes computer-executable instructions, such as instructions included in a program module, which execute in a device on a target real or virtual processor to perform the functions described above. Figure 3 and Figure 10 The process or method described herein. Typically, a program module includes routines, programs, libraries, objects, classes, components, data structures, etc., that perform specific tasks or implement specific abstract data types. In various embodiments, the functionality of a program module can ideally be combined or split among program modules. The machine-executable instructions for a program module can be executed locally or in a distributed device. In a distributed device, the program module can reside in both local and remote storage media.

[0132] Program code used to perform the methods of this disclosure may be written in any combination of one or more programming languages. This program code may be provided to a processor or controller of a general-purpose computer, special-purpose computer, or other programmable data processing apparatus, such that when executed by the processor or controller, it causes the functions / operations specified in the flowcharts and / or block diagrams to be implemented. The program code may be executed entirely on a machine, partially on a machine, as a stand-alone software package, partially on a machine and partially on a remote machine, or entirely on a remote computer or server.

[0133] The above-described program code can be implemented on a machine-readable medium, which can be any tangible medium that contains or stores a program for use by or in connection with an instruction execution system, apparatus, or device. The machine-readable medium can be a machine-readable signal medium or a machine-readable storage medium. Machine-readable media can include, but is not limited to, electronic, magnetic, optical, electromagnetic, infrared, or semiconductor systems, apparatus, or devices, or any suitable combination thereof. More specific examples of machine-readable storage media include electrical connections having one or more wires, portable computer floppy disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fiber, portable optical disc read-only memory (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination thereof.

[0134] Furthermore, although the operations are described in a specific order, this should not be construed as requiring such operations to be performed in the specific order shown or sequentially, or requiring all shown operations to be performed to obtain the desired result. In some cases, multitasking and parallel processing can be advantageous. Similarly, although several specific implementation details are included in the foregoing discussion, these details should not be construed as limiting the scope of this disclosure, but rather as descriptions of features that may be specific to particular embodiments. Certain features described in the context of individual embodiments may also be implemented in combination in a single embodiment. On the other hand, various features described in the context of a single embodiment may also be implemented individually or in any suitable sub-combination in multiple embodiments.

[0135] Although the subject matter has been described in language specific to structural features and / or methodological behavior, it should be understood that the subject matter defined in the appended claims is not necessarily limited to the specific features or behaviors described above. Rather, the specific features and actions described above are disclosed as exemplary forms for implementing the claims.

Claims

1. A method for determining the position of a tracking device, the tracking device comprising a visual sensor and an inertial sensor, and the method comprising: The first position of the tracking device relative to the reference marker is obtained based on the image of the reference marker collected by the vision sensor; After the tracking device moves, the inertial sensor acquires a second position of the tracking device relative to the first position; as well as The position of the tracking device relative to the reference marker is determined based on the first position and the second position; The object calibration position is obtained based on robot tags and work object tags, wherein the robot tags are attached to tools deployed in the robot system, the work object tags are attached to work objects to be processed by the robot system, and the object calibration position represents the position of the work object tag relative to the frame base of the robot system for calibrating the work object; The tracking device is coupled to the tool, and multiple positions are acquired relative to the work object marker, and the multiple positions are determined during the movement of the tool along a trajectory for the robot system to process the work object; as well as A tool path is generated based on the multiple locations of the tracking device.

2. The method according to claim 1, wherein obtaining the first position comprises: Obtain the actual size of the reference mark; Obtain the virtual size of the reference marker in the image; as well as The first position is determined based on the actual size, the virtual size, and the focusing parameters of the vision sensor.

3. The method according to claim 1, further comprising: After the movement, the visual sensor collects an environmental image of the operating environment of the tracking device; as well as The second position is corrected based on the environmental image and a previous environmental image, which was collected by the visual sensor at a previous time point prior to the time point at which the environmental image was collected.

4. The method of claim 3, wherein correcting the second position comprises: Based on image analysis, a pair of feature points are identified from the environmental image and the previous environmental image, respectively, and the pair of feature points corresponds to environmental features in the working environment; as well as Determine the offset between the positions of the pair of feature points; as well as The second position is corrected based on the offset.

5. The method of claim 1, wherein the tracking device is attached to one of the tool deployed in the robot system and the work object to be processed by the robot system, and the reference mark is attached to the other of the tool and the work object.

6. The method according to claim 5, further comprising: During the movement of the tool along the trajectory used to process the work object, a set of positions of the tracking device are determined.

7. The method of claim 6, wherein the tracking device comprises a telephone device, the visual sensor comprises a camera disposed in the telephone device, and the inertial sensor comprises an inertial measurement unit disposed in the telephone device.

8. The method of claim 7, wherein the method is implemented by an application installed in the telephone device, and the method further comprises: The processing parameters of the robot system are adjusted via user input to the application.

9. The method of claim 1, wherein the reference marker comprises a Quick Response (QR) code.

10. An apparatus for determining the position of a tracking device, the tracking device including a visual sensor and an inertial sensor, and the apparatus comprising: The first acquisition unit is used to acquire a first position of the tracking device relative to the reference marker based on an image of the reference marker collected by the vision sensor; The second acquisition unit is used to acquire a second position of the tracking device relative to the first position by the inertial sensor after the tracking device has moved; A determining unit is configured to determine the position of the tracking device relative to the reference marker based on the first position and the second position; as well as An object acquisition unit is used to acquire an object calibration position based on a robot marker and a work object marker, wherein the robot marker is attached to a tool deployed in a robot system, the work object marker is attached to a work object to be processed by the robot system, and the object calibration position represents the position of the work object marker relative to the frame base of the robot system for calibrating the work object; The object acquisition unit includes: The third acquisition unit is used to acquire a third position of the tracking device relative to the robot mark based on an image of the robot mark collected by the vision sensor; The fourth acquisition unit is configured to acquire a fourth position of the tracking device relative to the work object mark based on an image of the work object mark collected by the vision sensor after the tracking device has been moved, the tracking device being moved to collect the image of the work object mark; The fifth acquisition unit is configured to acquire, during the movement, a fifth position of the tracking device relative to the third position using the inertial sensor; and An object calibration determination unit is used to determine the object calibration position based on the position of the robot marker and the third, fourth, and fifth positions.

11. The apparatus of claim 10, wherein the first acquiring unit comprises: A true size acquisition unit is used to acquire the true size of the reference mark; A virtual size acquisition unit is used to acquire the virtual size of the reference marker in the image; as well as A position determination unit is used to determine the first position based on the actual size, the virtual size, and the focusing parameters of the visual sensor.

12. The apparatus of claim 10, further comprising: A collection unit is used to collect environmental images of the operating environment of the tracking device by the visual sensor after the movement; as well as A correction unit is used to correct the second position based on the environmental image and a previous environmental image, the previous environmental image being collected by the visual sensor at a previous time point prior to the time point at which the environmental image was collected.

13. The apparatus of claim 12, wherein the calibration unit comprises: An identification unit is used to identify a pair of feature points from the environmental image and the previous environmental image respectively based on image analysis, the pair of feature points corresponding to environmental features in the working environment; as well as An offset determination unit is used to determine the offset between the positions of the pair of feature points; as well as A position correction unit is used to correct the second position based on the offset.

14. The apparatus of claim 10, wherein the tracking device is attached to one of the tool deployed in the robotic system and the work object to be processed by the robotic system, and the reference mark is attached to the other of the tool and the work object.

15. The apparatus of claim 14, further comprising: A path determination unit is used to determine a set of positions of the tracking device during the movement of the tool along a trajectory for processing the work object.

16. The apparatus of claim 15, wherein the tracking device comprises a telephone device, the visual sensor comprises a camera disposed in the telephone device, and the inertial sensor comprises an inertial measurement unit disposed in the telephone device.

17. The apparatus of claim 16, wherein the apparatus is implemented by an application installed in the telephone device, and the apparatus further comprises: An adjustment unit is used to adjust the processing parameters of the robot system via user input to the application.

18. The apparatus of claim 10, wherein the reference mark comprises a Quick Response (QR) code.

19. A system for determining the location of a tracking device, comprising: A computer processor coupled to a computer-readable storage unit, the storage unit including instructions that, when executed by the computer processor, implement the method according to any one of claims 1 to 9.

20. A computer-readable medium having instructions stored thereon, the instructions causing the at least one processor, when executed, to perform the method according to any one of claims 1 to 9.

Images