Fixing tool, position detection method, and robot system

Abstract

To provide a new technique for easily recognizing the position and size of an object.SOLUTION: A fixing tool fixes an object that becomes an object of recognition of a position, and comprises a hand 14 fitted to the object, and a marker 16 integrated with the band. In the marker 16, a pattern 17 that can be detected as an image by a camera is formed. The pattern 17 is configured so as to be capable of performing calculation a position of the marker on the basis of the image. The band 14 has an identification region 14a that can be detected as an image by a camera and surrounds one portion of the object for a semicircle or more. The identification region 14a is configured so as to be capable of calculating the size of one portion of the object to which the band is fitted on the basis of the image.SELECTED DRAWING: Figure 1

Description

[Technical Field]

[0001] The present invention relates to a technology for easily recognizing the position and size of an object. [Background technology]

[0002] A robot system has been devised that determines the position and orientation of an object based on an image of the object captured by a camera and grasps the object with a hand at the tip of a robot arm (see Patent Document 1). This robot system uses a trained machine learning model to estimate multiple feature points of the object contained in the input image and determines the position and orientation of the object from the multiple feature points. This makes it possible to perform tasks such as grasping a cup by its handle and placing it on a table. [Prior art documents] [Patent documents]

[0003] [Patent Document 1] Japanese Patent Publication No. 2024-32055 Summary of the Invention [Problem to be solved by the invention]

[0004] However, the above-mentioned robot system has an information processing device that generates multiple training object models using basic shape data of the object, generates multiple simulation images for multiple scenes in which the training object models are respectively arranged, and generates a ground truth feature map for each simulation image. In other words, a ground truth feature map must be generated for each object, which requires an advanced information processing device. Therefore, there is room for further improvement in terms of versatility and simplification of the information processing device.

[0005] The present invention has been made in view of the above circumstances, and one of its exemplary purposes is to provide a new technique for easily recognizing the position and size of an object. [Means for solving the problem]

[0006] In order to solve the above problems, a fixing device according to one embodiment of the present invention is a fixing device that is fixed to an object whose position is to be recognized, and comprises a band that is attached to the object, and a marker that is integrated with the band, the marker having a pattern that can be detected by a camera as an image, and the pattern is configured so that the position of the marker can be calculated based on the image, and the band has an identification region that can be detected by a camera as an image and surrounds a portion of the object for more than halfway around, and the identification region is configured so that the size of the portion of the object to which the band is attached can be calculated based on the image.

[0007] According to this aspect, the object to which the fixing device is fixed is placed at any location, and the position of the object and the size of a part of the object can be calculated by capturing an image of the marker pattern or identification area with a camera.

[0008] The band may be attached to a transparent device having a cylindrical portion as the target. The identification area may be provided so that the diameter of the cylindrical portion can be calculated. In the case of a transparent device, it may be difficult to recognize the shape and position with a camera as is. Therefore, by providing an identification area on the band that can be detected by a camera, it becomes easier to calculate the size of a portion of the target to which the band is attached.

[0009] The band may be an elastic or flexible member that conforms to the outer surface of the object, allowing one band to be used for a variety of objects within a certain range of sizes and shapes, rather than requiring a variety of standardized bands for each object.

[0010] The identification area may be a strip-shaped area that can be detected by a camera, which makes it easy to form the identification area.

[0011] The pattern may be configured so that the orientation of the marker can be calculated based on the image, thereby allowing the orientation of the marker to be calculated from the image detected by the camera, even if the object is not always placed in a constant orientation.

[0012] Another aspect of the present invention is a position detection method, which includes placing an object to which the above-described fixture is attached, capturing an image of the placed object with a camera, calculating the position of the marker from a pattern included in the captured image, and calculating the diameter of a portion of the object from an identification region included in the captured image.

[0013] According to this aspect, the object to which the fixture is fixed is placed at any location, and the position of the object and the diameter of a part of the object can be calculated by capturing an image of the marker pattern or identification area with a camera.

[0014] The band may be attached a predetermined distance below the opening of the cylindrical part of the object, so that the position of the opening can be estimated by calculating the position of the marker.

[0015] Yet another aspect of the present invention is a robot system including a camera that captures an image of an object to which the above-described fixture is attached, a movement mechanism that moves an arm to an arbitrary position relative to the object, a calculation unit that calculates the position of a marker from a pattern included in the captured image and calculates the diameter of a portion of the object from an identification region included in the captured image, and a control unit that controls the movement mechanism to move the arm to a predetermined position based on the calculated information on the position and diameter of the object.

[0016] According to this aspect, the object to which the fixture is fixed is placed at a desired location, and the pattern of the marker or the identification area is captured by a camera, thereby allowing the position of the object and the diameter of a portion of the object to be calculated. Furthermore, based on the calculated position and diameter of the object, the arm can be moved to a predetermined position above the object, for example.

[0017] The target object may be a cylindrical laboratory instrument with an open top. The laboratory instrument may be, for example, a transparent container such as a test tube, a flask, or a beaker. The arm may have a mounting portion to which a pipette for supplying a predetermined amount of liquid to the laboratory instrument can be attached. This allows the predetermined amount of liquid to be automatically supplied to the laboratory instrument simply by placing the laboratory instrument with the fixture attached in front of the camera of the robot system. Also, even if there are multiple laboratory instruments with fixtures attached and each is placed in an arbitrary location, the predetermined amount of liquid can be automatically supplied to each of them.

[0018] Any combination of the above components and conversion of the present invention into a method, device, system, etc. are also valid aspects of the present invention. [Effects of the Invention]

[0019] According to the present invention, the position and size of an object can be easily recognized. [Brief explanation of the drawings]

[0020] [Figure 1] 1 is a schematic diagram of a test container to which an image marker band according to the present embodiment is attached. [Figure 2] 2(a) to 2(c) are schematic diagrams for explaining image processing according to this embodiment. [Figure 3] FIG. 10 is a diagram for explaining the positional relationship between a camera and a container. [Figure 4] FIG. 2 is a block diagram illustrating the software architecture of an image processing system including a calculation unit according to the present embodiment. [Figure 5] 5(a) and 5(b) are schematic diagrams for explaining the definition of the position parameters and the orientation parameters of the container relative to the camera. [Figure 6]FIG. 6(a) is a graph plotting the absolute error e when the distance xc in the planar direction is changed, FIG. 6(b) is a graph plotting the absolute error e when the distance zc in the depth direction is changed, FIG. 6(c) is a graph plotting the absolute error e when the roll angle θc is changed, and FIG. 6(d) is a graph plotting the absolute error e when the pitch angle φc is changed. [Figure 7] 1 is a schematic diagram showing a general configuration of a robot system according to an embodiment of the present invention; DETAILED DESCRIPTION OF THE INVENTION

[0021] The present invention will be described below based on preferred embodiments with reference to the drawings. The same or equivalent components, parts, and processes shown in each drawing will be given the same reference numerals, and redundant explanations will be omitted where appropriate. Furthermore, the embodiments are illustrative and do not limit the invention, and all features and combinations thereof described in the embodiments are not necessarily essential to the invention.

[0022] The fixing device according to this embodiment can be used for various purposes that can be realized by recognizing the position and orientation of the object to be fixed. As an example of such a purpose, the case where the fixing device is attached to a laboratory instrument and the position and orientation of the laboratory instrument are recognized will be described below.

[0023] In recent years, there has been a growing need for laboratory automation in chemical and biological experiments to reduce the workload of researchers and improve experimental efficiency. In automating experiments, it is important to accurately grasp the position and shape of instruments in the laboratory for safety and speeding up the process. Conventional automation systems use dedicated vessels and fix their positions to efficiently perform specific experiments.

[0024] However, this method requires the robot to be taught the location of each container, which increases the system's implementation costs and limits its flexibility and versatility for experimental processes.In addition, because laboratories often handle transparent, cylindrical containers (e.g., reagent bottles and test tubes), image recognition of these containers also poses a challenge.

[0025] Therefore, the present inventors have devised a fixture (hereinafter referred to as an "image marker band") that can be attached to a transparent reagent container to recognize the diameter and center position of the container, and a position detection method using the fixture. Furthermore, the image marker band according to this embodiment uses a stretchable material for the band portion, making it easy to attach to various containers already present in the laboratory. Furthermore, the position detection method using the image marker band does not require prior registration of container dimensions, thereby reducing the costs of system implementation and changes to experimental procedures. Furthermore, the present inventors have devised a robot system that autonomously pipettes reagent containers to which image marker bands are attached.

[0026] (Image marker band) FIG. 1 is a schematic diagram of a test container to which an image marker band according to this embodiment is attached. The image marker band 10 shown in FIG. 1 is a fixture that is fixed to an object (container 12) whose position is to be recognized. The image marker band 10 includes a band 14 that is attached to the container 12 and a marker 16 that is integrated with the band 14. The marker 16 has a pattern 17 that can be detected by a camera as an image. The pattern 17 is configured so that the position and orientation of the marker 16 can be calculated based on the image. This allows the orientation of the marker to be calculated from the image detected by the camera, even if the object is not always placed in a constant orientation. The band 14 has an identification region 14a that can be detected by a camera as an image and surrounds a portion of the container 12 over more than half the circumference. The identification region 14a is configured so that the size of the portion of the container 12 to which the band 14 is attached can be calculated based on the image.

[0027] The marker 16 according to this embodiment is a small, high-resolution marker. It is preferable that the marker 16 be capable of measuring three-dimensional position and orientation. Specifically, it is a 10 mm × 10 mm DX Marker (registered trademark). This marker has a pattern 17 printed at high resolution on a 0.7 mm thick glass substrate using photolithography. Specifically, the pattern 17 includes a two-dimensional barcode 17a capable of registering (identifying) multiple types of IDs and black dots 17b formed at each of the four corners for calculating the position and orientation. Furthermore, since the main portion of this marker is made of a glass substrate, it has excellent chemical resistance and weather resistance. The band 14 is a rubber band with a width of 10.5 mm and a thickness of 0.4 mm. The identification region 14a of the band 14 is a strip-shaped portion colored red extending from the center in the circumferential direction. This allows the identification region 14a to be easily formed and easily detected by a camera. The shape of the identification area 14a is not limited to a strip shape, and may be any shape that can be identified by a camera.

[0028] The image marker band 10 is preferably one that can be attached to a transparent container 12 having a cylindrical portion as the target object, but it may also be one that can be attached to an opaque container that does not necessarily have a cylindrical portion. For example, the outer periphery of the portion where the image marker band 10 is attached may be elliptical or polygonal. The identification area 14a is provided so that the diameter of the cylindrical portion can be calculated. In the case of a transparent container 12, it may be difficult to recognize the shape and position with a camera as is. Therefore, by providing the identification area 14a in the band 14 that can be detected by a camera, it becomes easier to calculate the size (diameter) of the portion of the container to which the band 14 is attached.

[0029] Band 14 is preferably an elastic or flexible material that deforms along the outer surface of the object, but any material that can be fixed to container 12 will do. For example, it may be something like a sticker. Band 14 does not necessarily have to be long enough to encircle the entire circumference of the container, and may be a C-shaped piece of plastic or rubber. This allows one band to be used for a variety of objects within a certain range of sizes and shapes, without having to prepare various standardized bands for each object.

[0030] (Image Processing) Next, processing of the image captured from the image marker band 10 will be described. Figures 2(a) to 2(c) are schematic diagrams for explaining image processing according to this embodiment. Figure 3 is a diagram for explaining the positional relationship between the camera and the container.

[0031] First, the container 12 shown in FIG. 1 is imaged by the camera 18, and the marker 16 of the image marker band 10 is detected from the RGB image (image size 1080p) acquired from the camera. To achieve rapid processing, a software ROI (Region of Interest) is applied centered on the detected marker 16. The range of the software ROI is determined by the distance x in the planar direction (arrow x direction) of the marker shown in FIG. m and the distance z in the depth direction (arrow z direction) m This allows the entire image marker band 10 to fit within the range (see FIG. 2(a)).

[0032] Next, a color similar to the linear identification area 14a drawn on the band 14 is overlaid on the marker 16 in the image so that the marker 16 is completely hidden (see FIG. 2(b)). The image is then binarized by thresholding the RGB values, and contour detection is performed. If multiple contours are detected, the contour where the center of the marker 16 and the coordinates in the image overlap is determined as the contour of the image marker band 10 (see FIG. 2(c)). The number of horizontal pixels occupied by the entire image marker band 10 is defined as S c , the number of horizontal pixels occupied by the marker 16 is S m It is defined as:

[0033] Then, the diameter d of the container 12 shown in FIG. 3 is calculated by the following formula (1): c Calculate. dc=(S c / S m )×k(z m )-2α Equation (1) Here, k is a coefficient for correcting the difference in dpi due to the depth direction in the image, and Z mIn addition, α is a constant representing the thickness of the band 14, and by subtracting this, the diameter d of the container 12 itself can be calculated. c In the image marker band 10 according to this embodiment, the thickness α of the band 14 is 0.4 mm.

[0034] Next, the coordinates (x c ,z c ) is calculated. x c =x m -(d c / 2)×sinφ c Formula (2) z c =z m +(d c / 2)×cosφ c Formula (3) where φ c is the rotation angle of the container 12 and the marker 16.

[0035] (Image Processing System) 4 is a block diagram illustrating the software architecture of an image processing system including a calculation unit according to this embodiment. The image processing system 20 includes a camera 18 and a personal computer 22. The LEAG library 24 manufactured by League Solutions Inc. is used to recognize the image marker band 10, and the OpenCV library 26 is used to detect the contour of the image marker band 10 and to recognize the center position of the container 12. Each library is executed on an OS 28. Furthermore, an application 30 performs each calculation for the image processing described above.

[0036] (Experimental Method) Using the above-mentioned method, the recognition performance of the center position of the container 12 to which the image marker band 10 is attached was investigated by an experiment. c Three different types of reagent containers (d c(=18mm, 35mm, 50mm) were used. Figures 5(a) and 5(b) are schematic diagrams for explaining the definition of the position parameters and orientation parameters of the container relative to the camera. Each parameter is the distance x in the planar direction. c , distance z in the depth direction c , roll angle θ c , pitch angle φ c In the experiment, the coordinates of the center position of the reagent container (x c ,z c ), roll angle θ c , pitch angle φ c was changed and the absolute error e between the estimated value by the above method and the true value was measured.

[0037] The absolute error e is defined by the following equation (4). e=(x e 2 +z e 2 ) 1 / 2 Formula (4) where x e is the distance x in the planar direction c Recognition error with z e is the depth distance z c The measurement was performed 10 times for each parameter.

[0038] (Experimental results) Each figure in Figure 6 shows the diameter d c Three different types of reagent containers are placed at a distance x c , distance z in the depth direction c , roll angle θ c , pitch angle φ c The absolute error between the estimated value and the true value is plotted when the distance x in the planar direction is changed. c Fig. 6(b) shows a graph plotting the absolute error e when the depth distance z c Fig. 6(c) shows a graph plotting the absolute error e when the roll angle θ c 6(d) shows a graph plotting the absolute error e when the pitch angle φ cThis is a graph plotting the absolute error e when σ is changed. In the figure, σ is the standard deviation (1σ) of the absolute error e.

[0039] (Distance in the plane x c (Recognition performance for changes in Distance x in the plane direction c Only the depth distance z c = 250 mm, roll angle θ c =0°, pitch angle φ c When position recognition is performed under fixed conditions, the recognition error (absolute error e) is the distance x in the planar direction. c The accuracy was ±3.3 mm or less within the range of ±200 mm, with a standard deviation of 0.818 mm (see Figure 6(a)). c As the error increases, the recognition error (absolute error e) tends to increase. This is thought to be due to the influence of camera lens distortion in the edge areas of the image.

[0040] (Distance z c (Recognition performance for changes in Depth distance z c Only the distance x in the planar direction is changed. c =0mm, roll angle θ c =0°, pitch angle φ c When position recognition is performed under fixed conditions, the recognition error (absolute error e) is the distance z in the depth direction. c The accuracy was within ±2.2 mm in the range of 50 to 360 mm, with a standard deviation of 0.760 mm (see Figure 6(b)). c If the distance exceeds 360 mm, it becomes difficult to detect the marker itself, making it difficult to recognize its position accurately.

[0041] (Roll angle θ c (Recognition performance for changes in Roll angle θ c Only the distance x in the planar direction is changed. c = 0 mm, depth distance z c =250mm, pitch angle φ cWhen position recognition is performed under fixed conditions, the recognition error (absolute error e) is c The accuracy was within ±3.0 mm in the range of ±31°, with a standard deviation of 0.723 mm (see Figure 6(c)).

[0042] (pitch angle φ c (Recognition performance for changes in Pitch angle φ c Only the distance x in the planar direction is changed. c = 0 mm, depth distance z c = 250 mm, roll angle θ c When position recognition is performed under fixed conditions, the recognition error (absolute error e) is c The pitch angle φ was within ±2.5 mm in the range of ±52°, with a standard deviation of 0.766 mm (see Figure 6(d)). c When the angle exceeded ±52°, the image marker band could no longer fit within the camera's field of view, making position recognition impossible.

[0043] From the above results, it can be seen that the position detection method using image marker bands according to this embodiment can accurately recognize the position and orientation of test containers within the range in which test containers are placed during normal experimental work.

[0044] (Robot System) Next, an experiment was conducted to transfer liquid between containers based on the information on the center positions of the containers detected by the above-mentioned method. Fig. 7 is a schematic diagram showing the general configuration of the robot system according to this embodiment.

[0045] The robot system 100 according to this embodiment includes a camera 18 that captures images of the test containers 12a, 12b to which the image marker bands 10a, 10b are attached, a movement mechanism 34 that moves the arm 32 to any position relative to the test containers 12a, 12b, a calculation unit that calculates the positions of the markers 16a, 16b from patterns included in the captured image and calculates the diameter of a portion of the test containers 12a, 12b from the identification regions 14a, 14b included in the captured image, and a control unit that controls the movement mechanism 34 so that the arm 32 moves to a predetermined position based on the calculated position and diameter information of the test containers 12a, 12b. The calculation unit and control unit according to this embodiment are provided in a personal computer 22.

[0046] The test vessels 12a and 12b are cylindrical vessels with open tops, such as transparent vessels such as test tubes, flasks, beakers, etc. The arm 32 has a mounting portion 38 to which an electric pipette 36 is attached, which supplies a predetermined amount of liquid to the laboratory vessel.

[0047] The robot system 100 can recognize the center position of each test container 12a, 12b and transfer liquid between the containers using an electric pipette 36 attached to a mounting portion 38 of the arm 32. For example, the robot system 100 can aspirate liquid from the test container 12a and then dispense the aspirated liquid into another test container 12b. The test containers 12a, 12b are positioned so that they do not overlap when viewed from the camera 18 and within a range that allows the camera 18 to recognize the entire image marker bands 10a, 10b.

[0048] By attaching image marker bands to the test containers, the robot system 100 can accurately aspirate liquid from and discharge liquid into the test containers even if the position or orientation of the test containers is changed. Also, by assigning the ID of the test container to the marker on the image marker band, operations such as aspirating liquid from one test container and discharging liquid into another test container can be performed accurately and smoothly even without information such as the diameter or position of the test containers.

[0049] In this way, the robot system 100 according to this embodiment can calculate the position of the test container and the size (diameter) of a part of the test container (the part where the image marker band is attached) by placing the test container with the image marker band fixed at any location and capturing an image of the marker pattern or identification area with a camera. Furthermore, based on the calculated position and diameter of the test container, the arm can be moved to a predetermined position above the test container.

[0050] Furthermore, the robot system 100 according to this embodiment can automatically supply a predetermined amount of liquid to a test container by simply placing the test container with an image marker band attached in front of the camera of the robot system. Even if there are multiple test containers with image marker bands attached and each is placed in a desired location, a predetermined amount of liquid can be automatically supplied to each of them. Note that the object of transfer and supply is not limited to liquid, and powder or solid materials can also be used.

[0051] 7, the image marker band 10a is preferably attached a predetermined distance L below the opening 13 of the cylindrical part of the test container 12a. This allows the position of the opening 13 to be estimated by calculating the position of the marker 16a. This allows the tip of the electric pipette 36 to be moved to an accurate position relative to the liquid surface, ensuring reliable aspirating of the liquid.

[0052] Furthermore, by managing the position where the image marker band is attached to the container (position from the opening), it is possible to always keep the height at which reagents etc. are dripped / injected uniformly. It is also possible to calculate the liquid level from the estimated diameter of the container and drip from a constant height above the liquid level. As a result, it is possible to achieve uniformity in the reagent synthesis process, which is difficult to manage manually.

[0053] (Position detection method) The following position detection method can be realized using the image marker band and robot system described above. In this method, a container 12 with the image marker band 10 attached is placed in a predetermined area that can be imaged by a camera 18, an image of the placed container 12 is taken by the camera 18, the position and orientation of the marker 16 are calculated from the pattern 17 included in the image, and the diameter of a portion of the container 12 is calculated from the identification area 14a included in the image.

[0054] In this embodiment, an example has been described in which an image marker band is attached to a general-purpose container currently used in many laboratories. In this case, the image marker band is made of a flexible and stretchable material, so it can be adapted to containers of various sizes. Furthermore, because the size and height of the container can be recognized, for example, after a sample is placed in the robot, it can autonomously perform operations such as grasping the container and placing it in a mixer or analyzer without having to input information about the shape and position of the container in advance.

[0055] Furthermore, by providing an ID function to the central pattern 17a of the image marker band, containers containing samples such as reagents can be easily registered and managed, reducing mistakes of people mixing up reagents during experiments. Furthermore, when programming experimental procedures, a system can be constructed in which containers with image marker bands attached can be "pre-registered" by showing them to the camera. Since not only the ID but also size information can be read instantly at this time, even if multiple image marker bands appear to overlap after the container is placed on the table, the pre-registered information can be used to deal with the situation without any problems.

[0056] Furthermore, the object to which the image marker band is attached may be not only a small object such as a laboratory vessel, but also a large object such as a drum, a utility pole, or a tree. For example, by applying the robot system according to this embodiment, a robot can transport and stack a large number of drums placed at a port to a predetermined location. Alternatively, a robot can string electrical wires or install electrical components on utility poles in a city. Alternatively, a robot can cut down and transport a large number of trees standing in a forest.

[0057] Although the present invention has been described above with reference to the above-mentioned embodiments, the present invention is not limited to the above-mentioned embodiments, and suitable combinations and substitutions of the configurations of the embodiments are also included in the present invention. Furthermore, it is possible to suitably rearrange the combinations and processing orders in the embodiments based on the knowledge of those skilled in the art, and to make modifications to the embodiments such as various design changes, and such modified embodiments are also included in the scope of the present invention. [Explanation of symbols]

[0058] 10 image marker band, 12 container, 14 band, 14a identification area, 16 marker, 17 pattern, 18 camera, 20 image processing system, 32 arm, 34 movement mechanism, 36 electric pipette, 38 attachment part, 100 robot system.

Claims

1. A fixture that is fixed to an object whose position is to be recognized, a band attached to the object; a marker integrated with the band; The marker has a pattern formed thereon that can be detected by a camera as an image, the pattern is configured so that the position of the marker can be calculated based on the image; The band has an identification area that can be detected by a camera as an image and surrounds a part of the object over at least half a circumference, The fixing device is characterized in that the identification area is configured so that the size of the part of the object to which the band is attached can be calculated based on the image.

2. The band is attached to a transparent device having a cylindrical portion as the object, The fixture according to claim 1 , wherein the identification area is provided so that the diameter of the cylindrical portion can be calculated.

3. 3. The fixing device according to claim 1, wherein the band is an elastic or flexible member that deforms along the outer surface of the object.

4. 3. The fixture according to claim 1, wherein the identification area is a strip-shaped area that can be detected by a camera.

5. 3. The fixture according to claim 1, wherein the pattern is configured so that the orientation of the marker can be calculated based on the image.

6. An object to which the fixing device according to claim 1 or 2 is attached is placed, Taking an image of the placed object with a camera; calculating the position of the marker from the pattern included in the captured image; calculating a diameter of a part of the object from the identification region included in the captured image; A position detection method comprising:

7. 7. The position detection method according to claim 6, wherein the band is attached a predetermined distance below the opening of the cylindrical portion of the object.

8. a camera that captures an image of an object to which the fixing device according to claim 1 or 2 is attached; a movement mechanism that moves the arm to an arbitrary position relative to the object; a calculation unit that calculates the position of the marker from the pattern included in the captured image and calculates the diameter of a portion of the object from the identification region included in the captured image; a control unit that controls the movement mechanism so that the arm moves to a predetermined position based on the calculated information on the position and diameter of the object; A robot system comprising:

9. The object is a cylindrical laboratory instrument with an open top, 9. The robot system according to claim 8, wherein the arm has a mounting portion for mounting a pipette for supplying a predetermined amount of liquid to the laboratory equipment.

Images