Adaptive mobile operating device and method

Abstract

An adaptive mobile operation device and method are provided. The method includes providing a mobile operation device including an operation apparatus, a sensor, and a processor to perform an operation on an object disposed on a carrier having a plurality of markers separated from each other; providing a base motion plan including a plurality of first pose-aware actions; detecting the plurality of markers by the sensor to obtain running marker information; generating, by the processor, a running motion plan based on the base motion plan, wherein the running motion plan includes a plurality of second pose-aware actions, and the plurality of second pose-aware actions are modified from the plurality of first pose-aware actions according to the running marker information; and further performing, by the processor, the running motion plan to control the operation apparatus to operate the object.

Description

Technical Field

[0001] This case relates to an adaptive mobile operation device and method, and more particularly to an adaptive mobile operation device and method utilizing multiple markers. Background Art

[0002] Material handling and logistics are two critical tasks in warehouses and factories. These tasks are typically performed manually, which can lead to safety risks and increase operating costs. While the gradual development of mobile operating devices has made it possible to automate these tasks, there are still challenges that need to be addressed.

[0003] The first challenge is how to navigate mobile operating devices. Using laser rangefinders or lidar coupled with advanced control algorithms, automated guided vehicles (AGVs) can now autonomously navigate to their target locations. However, their positional accuracy is limited to approximately 10 centimeters and their orientation accuracy to approximately 10 degrees.

[0004] The second challenge is how to locate the target object or operating area. This involves estimating the target object's pose, including its position and orientation. Typically, technologies such as computer vision and machine learning can achieve this under limited conditions (such as good lighting). However, due to the placement of the camera on the mobile device and the varying lighting conditions in warehouses or factories, these techniques may not provide stable estimates. Furthermore, these technologies are computationally intensive, making them unsuitable for mobile devices with limited battery power and computing power. Furthermore, if the operating scenario changes, the mathematical model must be rebuilt, which is a complex process. Technologies such as ArUco and ALVAR, which use square planar fiducial markers, are commonly used to detect the pose of markers placed on objects. After fixing the marker's size, the marker's position and orientation, or pose, are determined by its size and shape in the camera image. While this method provides accurate position estimates (typically within one to two millimeters), its orientation estimates are highly dependent on short-term environmental conditions, such as lighting and fluctuations.

[0005] The third challenge is motion planning. This planning involves moving a mobile manipulator to a specific location and using the manipulator to perform an operation. Traditionally, "teaching" is a technique used on production lines to enable fixed manipulators to perform repetitive tasks such as picking and placing and tightening. Engineers guide and program the manipulator through a series of movements that represent the task. However, because the movement of the mobile platform (AGV) generates position and orientation errors, there is also a positional and orientation offset between the manipulator and the target object. Therefore, traditional "teaching" techniques are not suitable for mobile manipulators.

[0006] In addition to the above challenges, artificial intelligence and machine learning are also common technologies used in academic research to solve the above problems. However, considering the financial costs, it is actually difficult for small companies to have a dedicated research team in this area. Therefore, it is more appropriate to solve these problems by providing a low-cost architecture.

[0007] Therefore, there is a need to provide an adaptive mobile operation device and method to address the deficiencies in the prior art. Summary of the Invention

[0008] The purpose of this application is to provide an adaptive mobile operation device and method to overcome at least one of the above-mentioned deficiencies.

[0009] Another object of this case is to provide an adaptive mobile operating device and method, which classifies the actions of an operating object into gesture-sensing actions and non-gesture-sensing actions, and further associates the gesture-sensing actions with the position and orientation information obtained by detecting markers, thereby achieving high-precision gesture-sensing actions while adopting an adaptive mobile operating device with a low-cost architecture.

[0010] To achieve the above-mentioned objectives, the present invention provides an adaptive mobile operation method, comprising the following steps: providing a mobile operation device, which includes an operating device, a sensor, and a processor to perform operations on an object placed on a carrier, and the carrier is provided with multiple markers separated from each other; providing a basic motion plan, which includes multiple first posture sensing actions; the sensor detects the multiple markers to obtain operation marker information; the processor generates an operation motion plan based on the basic motion plan, wherein the operation motion plan includes multiple second posture sensing actions, and the multiple second posture sensing actions are modified from the multiple first posture sensing actions according to the operation marker information; and the processor further executes the operation motion plan to control the operating device to operate the object.

[0011] In one embodiment, each of the first posture perception actions of the basic motion plan includes variables corresponding to a plurality of tags and basic tag information.

[0012] In one embodiment, the method further includes the following steps: the processor calculates a difference between the basic tag information and the running tag information; and the processor generates a plurality of second gesture sensing actions according to the plurality of first gesture sensing actions and the difference.

[0013] In one embodiment, the running mark information and the basic mark information respectively include positions and orientations between a plurality of marks and the sensor.

[0014] In one embodiment, the operating device further includes an end effector and a joint.

[0015] In one embodiment, the first and second gesture sensing actions respectively include moving the end effector relative to the position and orientation of the object.

[0016] In one embodiment, the first and second gesture sensing actions each include at least one of the following actions: moving the end effector to a target gesture; the end effector traversing a trajectory; and the end effector moving in association with the operation marker information.

[0017] In one embodiment, the object is placed at a fixed position on the carrier.

[0018] In one embodiment, the markings include visual markings or fiducial markings.

[0019] In one embodiment, the sensor comprises a camera.

[0020] To achieve the above-mentioned objectives, the present invention provides an adaptive mobile operating device, including an operating device, a sensor, and a processor, wherein the processor is coupled to the operating device and the sensor, and is configured to perform the following steps: capturing a basic motion plan, which includes multiple first posture sensing actions; driving the sensor to detect multiple markers set on a carrier to obtain operation marker information; generating an operation motion plan based on the basic motion plan, wherein the operation motion plan includes multiple second posture sensing actions, and the multiple second posture sensing actions are modified from the multiple first posture sensing actions according to the operation marker information; and executing the operation motion plan to control the operating device to operate an object set on the carrier.

[0021] In one embodiment, the sensor comprises a camera.

[0022] In one embodiment, the markings include visual markings or fiducial markings. BRIEF DESCRIPTION OF THE DRAWINGS

[0023] Figure 1 A schematic diagram showing an adaptive mobile operating system according to an embodiment of the present invention.

[0024] Figure 2 A schematic diagram showing the basic structure of the adaptive mobile operating device according to an embodiment of the present invention.

[0025] Figure 3 A flow chart showing the exercise plan of an embodiment of the present invention.

[0026] Figure 4 A schematic diagram showing the configuration of the operating area of ​​an embodiment of this case.

[0027] Figure 5 A schematic diagram showing the arrangement of cameras and markers in an embodiment of this case is shown.

[0028] Figure 6 A flowchart for generating a basic exercise plan is shown in an embodiment of the present invention.

[0029] Figure 7 A flowchart showing the execution of a motion plan at runtime in an embodiment of the present invention is shown.

[0030] Figure 8 A flow chart showing the operation process when operating a target object in an embodiment of this case.

[0031] Figure 9 The process of obtaining marking information by the camera sensor in this embodiment is shown.

[0032] Figure 10 The positional relationship of the markers in the basic motion plan and the running motion plan of the embodiment of the present invention is shown.

[0033] Figure 11 A schematic diagram showing the calculation of position and orientation offsets in an embodiment of the present invention.

[0034] The description of the accompanying drawings is as follows:

[0035] 101: Flooring

[0036] 111, 201, 413: AGV components

[0037] 112, 205, 411: Operating device

[0038] 113: Operation Tools

[0039] 114, 204, 521: Sensors

[0040] 115, 522: Effective field of view

[0041] 121, 401, 501: carrier

[0042] 122, 404: Target object

[0043] 123: Collider

[0044] 124, 125, 402, 403, 511, 512: Mark

[0045] 202: Processor

[0046] 203: Distance measuring object

[0047] 207: Joints

[0048] 206, 412: End effector

[0049] 301-304: Steps

[0050] 601-610: Steps

[0051] 701-702, 711-716, 721: steps

[0052] 801-805: steps

[0053] 901-906: steps

[0054] 1001: base motion plan

[0055] 1002: run motion plan

[0056] 1101, 1102, 1103: inter-marker positional relationship DETAILED DESCRIPTION

[0057] Some exemplary embodiments embodying the features and advantages of the present application are described in detail in the following description. It should be understood, however, that the application can be practiced in a variety of alternate configurations, and that the explanation and drawings are not intended to limit the application in any way.

[0058] The present application provides a system architecture for manipulating an object (e.g., picking, placing, or modifying the object) in a warehouse or factory production line, so that engineers or operators can easily and simply design a motion plan at an affordable economic cost.

[0059] The present application includes four parts, including (1) system design, (2) adaptive mobile manipulator construction, (3) teaching-based adaptive mobile manipulation design, and (4) algorithms for obtaining positioning information from multiple markers.

[0060] (1) System Design

[0061] The system of the present application includes an environment entity setup, which contains an adaptive mobile manipulator, a carrier for setting an object, and markers set on the carrier and separated from each other. Generally, the carrier is a rigid shelf in a warehouse or factory, and different shelves are distinguished by their identification codes, i.e., shelf IDs. Please refer to Figure 1Fig. 1 is a schematic diagram showing an adaptive mobile operating system according to an embodiment of the present application. The adaptive mobile operating system comprises three main parts: (a) a floor 101 of a warehouse or factory, (b) an adaptive mobile operating device, and (c) a carrier 121, a target object 122, a collision object 123, and markers 124, 125. The adaptive mobile operating device comprises an AGV component 111, an operating device 112, and a sensor 114. The operating device 112 has an operating tool 113, and the sensor 114 has an effective field of view 115. Depending on different environmental settings and actual requirements, the sensor 114 can be a camera sensor, such as a 2D / RGB camera, and the markers 124, 125 can be, for example, visual or fiducial markers, but are not limited thereto.

[0062] Since the carrier 121 is specially designed to accommodate the target object 122, under the assumption that the relative positions among the carrier 121, the target object 122, the collision object 123, and the markers 124, 125 are fixed, as long as the pose of the carrier 121 is known, the poses of the other objects can be obtained by calculation based on the pose of the carrier 121. In addition, the two markers 124, 125 are horizontally arranged on the carrier 121 and have substantially the same height from the floor 101 and substantially face the same direction. In an embodiment, to obtain the best results, the size of the markers is at least 35 mm, with a white frame of at least 3 mm, and the distance between the center points of the two markers is at least 100 mm apart from each other.

[0063] Although Figure 1 Although only one adaptive mobile operating device and one carrier are shown in Fig. 1, this is only by way of example, and different numbers and types of adaptive mobile operating devices and carriers can be arranged in the operating area at the same time, without being limited thereto. In addition, multiple pairs of markers can also be arranged on a single carrier, such as those facing different directions or being different types, to correspond to different mobile operating devices.

[0064] The purpose of the "operation task" in the present application is to move the mobile operating device to a position close to the carrier 121 and perform an operation on the target object 122. Therefore, before performing the operation on the target object 122, the adaptive mobile operating device is moved and approaches the carrier 121 by the AGV component 111 to achieve an operable pose of the target object 122, and to make the corresponding pair of specific markers fall within the effective field of view 115, and also to make the target object 122 fall within the reach of the operating device 112. This part can be achieved using conventional navigation technology.

[0065] Under this premise, the following assumptions can be made. First, the operation task is divided into two parts, navigation (by AGV component 111) and operation (by operation device 112). Second, AGV component 111 can navigate adaptive mobile operation device to a target position and orientation with sufficient accuracy so that target object 122 falls within the reach of operation device 112, but some degree of error (e.g., position and orientation offset) is allowed. Finally, since target object 122 is disposed on designed carrier 121, the pose of target object 122 relative to carrier 121 is fixed, that is, the pose of target object 122 can be obtained by calculation after the pose information of carrier 121 is known.

[0066] (2) Basic architecture of adaptive mobile operation device

[0067] Please refer to Figure 2 , which shows the basic structure of the adaptive mobile operation device of the embodiment. This mobile operation device is similar to those commonly used in industry and includes the following elements electrically coupled to each other: AGV component 201, processor 202, ranging object 203, sensor 204, and operation device 205, wherein operation device 205 has end effector (EFF) 206 and at least one joint 207, and processor 202 is configured to perform calculations and communications related to operating target objects. Since the present application focuses on the execution of operation tasks, only components related to operation tasks, such as processor 202, sensor 204, operation device 205, and EFF 206, are described, without limitation.

[0068] (3) Teaching-based adaptive operation

[0069] (3.1) Motion planning

[0070] On the basis of the foregoing, the operation task of the present application is defined as the procedure of operating a target object using an operation device without direct physical contact with the target object after the adaptive mobile operation device has reached the pose of the target object that can be operated. The operation task includes a series of operation actions, which are defined as a "motion plan" in the present application. Please refer to Figure 3, which shows a flow chart of the motion plan of an embodiment of the present invention. The motion plan includes n actions, namely action 1 (step 301), action 2 (step 302), action 3 (step 303), and up to action n (step 304), wherein possible actions include, but are not limited to, (a) moving to a target joint state, (b) moving the EFF to a target posture, (c) the EFF traversing a trajectory, (d) the position offset of the EFF movement relative to the coordinates of the operating device, (e) the position offset of the EFF movement relative to the coordinates of the target object, and (f) performing an action on the target object (e.g., opening / closing the gripper) and / or performing other related actions (e.g., turning on / off the light, starting / stopping the conveyor belt, etc.). It should be noted that the trajectory of (c) is formed by a series of (b). In addition, the motion plan may also include a collision body 123 to prevent the operating device from colliding with it, so it is not limited to this.

[0071] For simplicity, this application will only describe and discuss the actions listed in Table 1 below, but this is not intended to be limiting. Actions in the motion plan are categorized as gesture-aware and non-gesture-aware. Gesture-aware actions are those directly related to manipulating a target object after the operating device and EFF have reached the reachable target object. Note that actions adjusted online using a wrist-mounted camera or other sensors can also be incorporated into this framework.

[0072] Table 1 Actions included in the exercise plan

[0073]

[0074] (3.2) Using “teaching” to generate basic movement plans and their execution

[0075] See also Figure 4 , which shows a schematic diagram of the operating area configuration of an embodiment of the present invention. As shown in the figure, a carrier, such as a rigid shelf, is labeled 401, two square reference marks are labeled 402 and 403, a target object is labeled 404, an operating device is labeled 411, an EFF is labeled 412, and an AGV component is labeled 413. As can be observed from the figure, the relative posture between the target object 404 and the rigid shelf 401 is fixed, whereas the relative posture between the operating device 411 and the EFF 412 and the rigid shelf 401 depends on the posture (i.e., position and orientation) of the mobile operating device.

[0076] Therefore, if a motion plan can be provided for the operating device 411 to construct a specific operating posture for performing such an operating task, this motion plan can be used as a basis for modification to construct other different operating postures. Therefore, this motion plan is defined as a "basic motion plan," and the purpose of "teaching" in this case is to generate a basic motion plan. This basic motion plan can be generated manually, by calculation, or through teaching. This section describes how to use teaching to generate a basic motion plan and how to execute it.

[0077] (3.2.1) Environment Settings

[0078] In order to adjust the actions in the basic motion plan using position and orientation corrections, additional information is required. In this case, since each posture-sensing action is associated with a pair of fiducial markers, please refer to Figure 5 , which shows a schematic diagram of the camera and marker configuration in this embodiment. When the adaptive mobile operating device stops moving, a pair of reference markers 511 and 512 on carrier 501 must fall within effective field of view 522 of camera sensor 521. In one embodiment, a single pair of markers is used to associate with the entire basic motion plan, but this is not limiting. Different pairs of markers can also be used for different movements, which is also feasible.

[0079] (3.2.2) Basic movement plan and its digital representation

[0080] Given an arbitrary manipulation device pose, the motion plan for executing the manipulation task can be modified into a base motion plan by adding marker information to each pose-aware action. Generating the base motion plan requires an additional step of detecting marker pairs to obtain the base marker information associated with each pose-aware action. This can be achieved using a square fiducial marker technique. This technique uses image frames from an RGB camera to provide a stream of estimated poses (positions and orientations) of the markers. However, because these values ​​may vary, the following sections describe how to apply filters to the resulting pose stream and improve detection. Table 2 below shows the data structures used for each action and how various variables of different types are represented in a computer system.

[0081] Table 2 Actions and their digital representations

[0082]

[0083] Based on the previous paragraph, the previously defined motion plan for basic motion planning can be expanded into an ordered list of actions, where the gesture-sensing actions are stored along with the positions of a pair of fiducial markers. The details of the data structure for basic motion planning in the computer system are as follows:

[0084] Basic exercise plan = [a]

[0085] a=Pose EFF |Trajectory|Offset target |JS|Offset manipulator |Action other

[0086] Pose EFF =((p,q),(p L , p R ))

[0087] Trajectory=([(p,q)],(p L , p R ))

[0088] Offset target =((x, y, z), (p L , p R ))

[0089] JS=[j]

[0090] Offset manipulator =(x, y, z)

[0091] p, p L , p R =(x, y, z)

[0092] q=(rx,ry,rz,rw)

[0093] x, y, z, rx, ry, rz, rw, j are real numbers.

[0094] The meaning of each mark is as follows:

[0095] [u]: A sorted list of "u"s

[0096] a: an action

[0097] |: or

[0098] Pose EFF :EFF stance

[0099] Trajectory: EFF trajectory

[0100] Offset target : EFF's relative movement offset to the target object's coordinates

[0101] JS: Joint status of the operating device

[0102] Offsetmanipulator : EFF relative to the movement of the operating device coordinates

[0103] Action other : Other actions that do not affect the status of the operating device

[0104] p, p L , p R : Position, position of left mark, position of right mark

[0105] q: orientation (Euler angle or quaternion)

[0106] It should be noted that, unless otherwise mentioned, it is assumed that the origin is (0,0,0) and the world coordinate system is aligned with the operating device.

[0107] (3.2.3) Using teaching to generate basic movement plans

[0108] When programming a manipulator, teaching is the process of specifying the state of the manipulator (mostly joint states) by moving the manipulator to the desired configuration, rather than providing numerical values ​​for the joint states. In this case, this concept is further extended to the entire motion plan, and the user guides the manipulator through a series of movements in this program.

[0109] See also Figure 6 , which shows a flowchart for generating a basic exercise plan according to an embodiment of the present invention. At the beginning of the tutorial (step 601), a blank sorted list "actions" is provided for storing the exercise plan. Next, in step 602, the user specifies the next action in the exercise plan, or the user completes the exercise plan generation. At the same time, a blank dictionary data structure ({}) "curr_action" is provided. In decision step 603, the user's input in step 602 is examined.

[0110] If the action is a posture-sensing action, the corresponding variables in Table 2 are collected in step 604 and stored in "curr_action" together with the action type. In this step, the variables can be directly collected from the operating device after the user has operated it to reach the desired posture; then, step 605 uses the known technology as described above to collect the basic marking information of the left and right marks, and the basic marking information is stored in "curr_action" together with the marking ID. Here, the marking ID is provided by the user. The content of the algorithm collecting a series of samples and using a filter to filter out extreme values ​​to obtain a better value will be presented in Section 4.1. Afterwards, in step 606, "curr_action" is appended to the end of "actions". Based on this, a posture-sensing action associated with the basic marking information in the basic motion plan can be generated, which is defined as the first posture-sensing action.

[0111] Similarly, if the action is a non-gesture-aware action, the variables corresponding to Table 2 are collected in step 607 and stored in "curr_action" along with the action type. In this step, the variables can be collected (1) directly from the operating device that has been manipulated by the user to reach the desired joint state, and (2) through user keyboard input (e.g., EFF position offset, open / close grip, or other options). The system performs the corresponding action based on the received user input, and then "curr_action" is appended to the end of "actions" in step 608. Based on this, the non-gesture-aware action in the basic motion plan can be generated, which is defined as the first non-gesture-aware action.

[0112] If the user is indicated as having completed the process of generating a basic exercise plan, "actions" is then flattened into a string data structure and stored in step 609 with a unique name specified by the user for later use, and the process ends (step 610).

[0113] (3.3) Adjust the basic exercise plan to a running exercise plan

[0114] See also Figure 7, which shows a flow chart for executing a motion plan at runtime. When an operational scenario is generated, the system retrieves the existing basic motion plan and modifies it into an operational motion plan that matches the operational scenario. In step 701, the basic motion plan, including the first gesture-aware action and the first non-gesture-aware action, as described in Section 3.2, is input. This basic motion plan can be retrieved from computer storage data by name using a table lookup. Next, each element (i.e., each action) in the motion plan is processed. First, in step 702, a check is performed to see if the number of actions is zero. If so, the process ends and moves to the end state (step 721). Otherwise, the first element (action) is retrieved in step 702 and treated as curr_action. Next, in step 712, a determination is made as to whether the action is a gesture-aware action. If it is not, the process proceeds to step 715. Otherwise, in step 713, the processor drives the sensor to detect the flag associated with the action and obtains run flag information through the filter and algorithm described in Section 4.1. The run flag information is then used in step 714 to calculate the position and orientation offset using the algorithm described in Section 4.3 and to modify the action. The modified action is then executed in step 715, and the first action is removed in step 716. Each action in the basic motion plan is processed using this process.

[0115] In other words, the running motion plan is derived from a modified base motion plan. First, based on the running tag information obtained by the sensor, the first gesture-sensing action in the base motion plan is modified into a different gesture-sensing action, which is defined as the second gesture-sensing action in the running motion plan. Furthermore, the first non-gesture-sensing action in the base motion plan remains unchanged and is executed directly during the running phase. This is defined as the second non-gesture-sensing action in the running motion plan.

[0116] Accordingly, please refer to Figure 8 , which shows a flow chart of the operation process for manipulating a target object in an embodiment of the present invention. The process begins at step 801. Then, at step 802, the processor retrieves the generated basic motion plan. Then, at step 803, the processor drives the sensor to detect the marker to obtain operation marker information. Then, at step 804, the processor modifies the basic motion plan, which includes a first gesture-sensing action and a first non-gesture-sensing action, into an operation motion plan, which includes a second gesture-sensing action and a second non-gesture-sensing action, based on the operation marker information. Finally, as shown in step 805, the operation motion plan is executed to control the operating device to manipulate the target object.

[0117] (4) Algorithms for modifying basic motion plans

[0118] (4.1) Filtering algorithm to obtain stable marker positions

[0119] See also Figure 9 , which shows the process of acquiring marker information by the camera sensor in the embodiment of the present case. In step 901, an image stream is acquired using an RGB camera. Then, as shown in step 902, the image in the acquired image stream is processed in step 903 using existing square fiducial marker positioning technology (such as ArUco or AR Tracker Alvar), and a data stream with the marker ID and the position and orientation of the marker is output. Then, in step 904, the orientation of each consecutive k data points of the marker is removed and regenerated. Here, during the teaching period, k is set to 300, and during the running period, k is set to 30. Then, the data from step 904 is processed by the algorithm in step 905 to filter out extreme values ​​and output in step 906.

[0120] The filtering algorithm in step 905 is as follows.

[0121] enter:

[0122]

[0123] Algorithm:

[0124] 1. Calculation

[0125]

[0126] 2. According to P m and The distance descending order P m

[0127] 3. According to the P sorted in the previous step m , so that P′ m Be the first

[0128] 4. Calculation

[0129] Output:

[0130]

[0131] The meaning of each mark is as follows:

[0132] M: marker group to be located

[0133] m: mark m

[0134] P m : k samples with a specific label m

[0135] pmi : the i-th sample in P with position (x mi , y mi , z mi ) m

[0136] C m : final position of marker m

[0137] Other notations for temporary variables are self-explanatory.

[0138] Then, the output of the filtering algorithm is used to modify the base motion plan.

[0139] It is noted that by setting three markers in L-shape (or more markers), the three-dimensional position offset and pitch, yaw and roll can be determined, and the above information can be used to grasp the situation that the height of the shelf is changed, and the modification of the motion plan is similar to the foregoing, and thus is not described again.

[0140] (4.2) Algorithm for obtaining position and orientation offset between base motion plan and running motion plan

[0141] In the base motion plan, the base marker information is associated with each action, which is used together with the running marker information detected during running to calculate the position and orientation offset, and then modify the base motion plan, i.e., the first pose perception action is modified to the second pose perception action. Please refer to Figure 10 , which shows the difference in the positions of the markers in the base motion plan and the running motion plan in the embodiment of the present case. As shown in the figure, a and b are the left and right marker positions in the base motion plan 1001, and c and d are the left and right marker positions in the running motion plan 1002.

[0142] Please refer to Figure 11 , which shows a schematic diagram for calculating the position and orientation offset in the embodiment of the present case. It is noted that the position is based on the coordinates of the mobile operation device. The reference numeral 1101 indicates the relationship between the marker positions, wherein the position offset is from c to a, and the orientation offset is theta z ​Here, since the floor in a factory or warehouse environment is assumed to be flat, the height information (Z) is removed, and the relationship indicated by label 1102 is obtained, where a, b, c, and d correspond to a', b', c', and d' respectively, and a', b', c', and d' only contain two-dimensional information (X and Y). It should be noted that this is a projection onto the XY plane. Next, a' and c' are translated as O(0,0) and applied to b' (a' to 0) and d' (c' to 0) at the same time, finally generating a", b", c", and d" in relationship 1103. The following equations can then be used to calculate the position and orientation offsets.

[0143] enter:

[0144] a=(x a ,y a , z a ), b=(x b ,y b , z b ), c=(x c ,y c , z c ), d=(x d ,y d , z d )

[0145] Position offset:

[0146] (Δ x , Δ y ,0)=(x c -x a ,y c -y a ,0)

[0147] Azimuth offset:

[0148]

[0149] in:

[0150] r=(x r ,y r )=(x b -x a ,y b -y a )

[0151] s=(x s ,y s )=(x d -x c ,y d -y c )

[0152] r·s=x r xs +y r y s

[0153]

[0154] (4.3) Algorithms for modifying the basic motion plan

[0155] At this point, the basic motion plan can be obtained by position offset (Δ x , Δ y , 0) and azimuth offset theta z The motion plan is adjusted to execute the desired action. Within the motion plan, only gesture-sensing actions require modification, including, but not limited to, "moving the EFF to a certain gesture," "traversing a trajectory," and "offsetting the EFF's position relative to the target object's coordinates" (see Table 1). The calculations for these adjustments are described in Sections 4.3.1 and 4.3.2.

[0156] (4.3.1) EFF posture and trajectory

[0157] For the "Move EFF to a pose" action, a single EFF pose needs to be modified. On the other hand, the "EFF traverses a track" action involves a series of EFF poses, each of which must be recalculated. Both can use the following algorithm to calculate the new target EFF pose.

[0158] enter:

[0159] EFF posture in basic exercise plan

[0160] pose(p, q), p=(x, y, z), q=(q x ,q y ,q z ,q w )

[0161] Marking information in the basic exercise plan

[0162] l=(x l ,y l , z l ), r=(x r ,y r , z r )

[0163] According to the position offset (Δ x , Δ y ,0)

[0164] According to Section 4.2, the azimuth offset θ = theta z

[0165] Quaternion rotation equivalent to θ q r

[0166] Algorithm:

[0167] 1. Translate l, p as the origin of the XY plane: l' = (0, 0, z l )

[0168] p′=(x p′ ,y p′ , z p )=(xx a ,yy a , z)

[0169] 2. Rotate p′ by θ on the Z axis:

[0170] p"=(x p′ cosθ-y p′ sinθ,x p′ sinθ+y p′ cosθ,z p )=(x p″ ,y p″ , z p )

[0171] 3. Translate back to p″ and add the offset to get the new target pose:

[0172] p n =(x p″ +x a +Δ x ,y p″ +y a +Δ y , z p )

[0173] 4. Use q r Applied to q, where x represents quaternion multiplication

[0174] q n =q r ×q

[0175] Output:

[0176] Final EFF stance (p n ,q n )

[0177] (4.3.2) Mobile Position Offset

[0178] The action type “position offset of EFF movement relative to the coordinates of the target object” can be calculated using the following equation to calculate the new EFF movement offset.

[0179] enter:

[0180] EFF Movement in Basic Movement Planning

[0181] Δ=(Δ x , Δ y , Δ z )

[0182] Rotation (relative to the Z axis) θ = theta z

[0183] Algorithm:

[0184] New EFF shift Δ′ = (Δ′ x , Δ′ y , Δ′ z )

[0185] Where: Δ′ x =Δ x cosθ-Δ y sinθ

[0186] Δ′ y =Δ x sinθ+Δ y cosθ

[0187] Δ′ z =Δ z

[0188] In summary, this architecture provides a procedure for generating a basic motion plan based on basic marker information, wherein the basic motion plan can be adjusted to an operating motion plan using the method provided for compensating for position and orientation offsets by using the operating marker information obtained by the method provided in this case using two square reference markers.

[0189] In short, this case has the following advantages:

[0190] 1. Low cost: The cost of setting up the system is low, including the cost of the RGB camera and printed labels.

[0191] 2. Easy to set up: The marker can be simply set within the camera's field of view without the need for special measurement and alignment.

[0192] 3. Accuracy: The multi-marker system in this case can provide good accuracy in finding position offsets and orientation offsets relative to the basic motion plan.

[0193] 4. The ("teaching") method used to generate basic movement plans can be actually applied in industry without the support of a research team.

[0194] 5. Only local operating information is required: In this case, only local operating information needs to be used and stored. Compared with building an accurate global 3D environment map of the entire field, the cost is much lower and it also makes it easier to reconfigure the environment.

[0195] It should be noted that the above description is merely a preferred embodiment for the purpose of illustrating the present invention. The present invention is not limited to the described embodiment. The scope of the present invention is determined by the appended claims. Furthermore, the present invention is subject to various modifications by those skilled in the art, but all modifications are within the scope of the appended claims.

Claims

1. An adaptive mobile operation method, comprising the following steps: A mobile operating device is provided, comprising an operating apparatus, a sensor, and a processor for performing an operation on an object placed on a carrier, wherein a plurality of marks separated from each other are provided on the carrier; Providing a basic movement plan, including a plurality of first posture sensing actions, wherein each of the plurality of first posture sensing actions includes a basic marking information corresponding to the plurality of markings disposed on the carrier; The sensor detects the multiple marks set on the carrier to obtain operation mark information; The processor generates an operation motion plan based on the basic motion plan, wherein the operation motion plan includes a plurality of second gesture sensing actions, and the plurality of second gesture sensing actions are modified from the plurality of first gesture sensing actions according to the operation tag information; and The processor further executes the operation motion plan to control the operating device to operate the object.

2. The method according to claim 1, further comprising the steps of: The processor calculates a difference between the base tag information and the running tag information; and The processor generates the plurality of second gesture sensing actions according to the plurality of first gesture sensing actions and the difference. 3 . The method of claim 1 , wherein the running mark information and the basic mark information respectively comprise positions and orientations between the plurality of marks and the sensor. The method of claim 1 , wherein the operating device further comprises an end effector and a joint.

5. The method of claim 4, wherein the first gesture-sensing action and the second gesture-sensing action each comprise moving the end effector relative to the position and orientation of the object.

6. The method of claim 5, wherein the first gesture sensing action and the second gesture sensing action each comprise at least one of the following actions: Moving the end effector to a target posture; The end effector traverses a trajectory; and The end effector moves in association with the operation mark information. The method of claim 1 , wherein the object is placed at a fixed position on the carrier.

8. The method of claim 1, wherein the marking comprises a visual marking or a fiducial marking.

9. The method of claim 1, wherein the sensor comprises a camera.

10. An adaptive mobile operating device comprising: an operating device; a sensor; as well as A processor is coupled to the operating device and the sensor and is configured to perform the following steps: Retrieving a basic motion plan, including a plurality of first posture sensing actions, wherein each of the plurality of first posture sensing actions includes basic marking information corresponding to a plurality of marks disposed on a carrier; driving the sensor to detect the multiple marks disposed on the carrier to obtain operation mark information; generating an operation motion plan based on the basic motion plan, wherein the operation motion plan includes a plurality of second posture sensing actions, and the plurality of second posture sensing actions are modified from the plurality of first posture sensing actions according to the operation tag information; as well as The operation motion plan is executed to control the operating device to operate an object disposed on the carrier. The adaptive mobile operating device as claimed in claim 10 , wherein the sensor comprises a camera.

12. The adaptive mobile operating device of claim 10, wherein the marking comprises a visual marking or a fiducial marking.

Images