Rotation state derivation device and robot system

Abstract

To realize a rotating state derivation device capable of suitably derive the rotating state of a globe.SOLUTION: A rotating state derivation device (4) according to one embodiment of the present disclosure is a device for deriving a rotating state of a globe on the basis of image data of the globe detected by a sensor (3). When a distance between the sensor (3) fixed to a mobile body and the globe is greater than or equal to a preset first threshold. the rotating state derivation device extracts a feature part of the globe derives the rotating state of the globe on the basis of time-series displacement of the extracted feature part. Meanwhile, when the distance between the sensor (3) and the globe is less than the first threshold, the rotating state derivation device estimates an attitude of the glove at a current time point and derives the rotating state of the globe on the basis of data indicating the rotating state of the globe having been derived in the past.SELECTED DRAWING: Figure 3

Description

Technical Field

[0006] ,

[0001] The present disclosure relates to a rotation state derivation device and a robot system.

Background Art

[0002] Generally, when deriving the rotation state of a sphere, as disclosed in Patent Document 1, in each of a plurality of image data obtained by detecting the sphere with a sensor, a mark provided on the sphere is extracted as a feature part, and the rotation state of the sphere is derived based on the displacement of the feature part over time.

Prior Art Documents

Patent Documents

[0003]

Patent Document 1

Summary of the Invention

Problems to be Solved by the Invention

[0004] The applicant of the present application has found the following problems. When the sphere approaches the sensor, the area of the sphere that can be detected by the sensor becomes narrow. For example, the feature part of the sphere may fall outside the detection range of the sensor, and the rotation state of the sphere may not be derived well.

[0005] The present disclosure has been made in view of such problems, and realizes a rotation state derivation device and a robot system capable of deriving the rotation state of a sphere well.

Means for Solving the Problems

[0006] A rotation state derivation device according to an aspect of the present disclosure is a rotation state derivation device that derives the rotation state of a sphere based on image data obtained by detecting the sphere with a sensor, and when the distance between the sensor fixed to the moving body and the sphere is equal to or greater than a preset first threshold value, extracts a feature part of the sphere, and derives the rotation state of the sphere based on the displacement of the extracted feature part over time. If the distance between the sensor and the sphere is less than the first threshold, the orientation of the sphere at the current time is estimated based on data indicating the rotation state of the sphere derived in the past, and the rotation state of the sphere is derived. [Effects of the Invention]

[0007] According to this disclosure, a rotation state derivation device and a robot system capable of accurately deriving the rotation state of a sphere can be realized. [Brief explanation of the drawing]

[0008] [Figure 1] This is a block diagram showing the configuration of the robot system according to Embodiment 1. [Figure 2] This is a diagram illustrating the detection range of the sensor in the robot system of Embodiment 1. [Figure 3] This is a flowchart showing the processing flow of the rotation state derivation device in the robot system of Embodiment 1. [Figure 4] (a) is a diagram illustrating the detection range of the sensor when the fingertips of the robot's hand are touching the basketball, (b) is a diagram showing image data detected by the sensor when the fingertips of the robot's hand are touching the basketball, (c) is a diagram illustrating the detection range of the sensor when the robot's hand is grasping the basketball, and (d) is a diagram showing image data detected by the sensor when the robot's hand is grasping the basketball. [Figure 5] This diagram illustrates the normal process for deriving the rotation state. [Figure 6] This is a flowchart illustrating the control flow when a robot dribbles a basketball. [Modes for carrying out the invention]

[0009] The following describes specific embodiments applying this disclosure with reference to the drawings. However, this disclosure is not limited to the following embodiments. Also, the following description and drawings have been simplified as appropriate.

[0010] <Embodiment 1> First, the configuration of the robot system of this embodiment will be described. Figure 1 is a block diagram showing the configuration of the robot system of this embodiment. As shown in Figure 1, the robot system 1 of this embodiment includes a robot 2, a sensor 3, a rotation state derivation device 4, and a database (DB) 5. These robot 2, sensor 3, rotation state derivation device 4, and DB 5 are connected via a network 6. Here, the network 6 is a wired or wireless communication line, for example, the internet.

[0011] Robot 2 can be composed of, for example, a general humanoid robot. In other words, although detailed illustrations are omitted, robot 2 comprises a head, torso, arms, hands, and legs. Here, the arms and hands of robot 2 constitute a robotic arm, and the hands of robot 2 constitute the hand portion of the robotic arm.

[0012] Robot 2 is configured to perform predetermined actions by controlling actuators 2a, which are provided at the joints of the head, torso, arms, hands, and legs, with a control unit 2b. For example, Robot 2 may be configured to perform actions that mimic those of a basketball player (such as grasping a basketball, dribbling, and free throws).

[0013] Sensor 3 can be configured, for example, as a three-dimensional distance measuring sensor. Figure 2 is a diagram illustrating the detection range (i.e., field of view) of the sensor in the robot system of this embodiment. In Figure 2, the detection range R of sensor 3 is shown by a dashed line.

[0014] The sensor 3 is fixed, for example, as shown in FIG. 2, to the base portion of the hand portion 2d of the robot arm 2c, that is, to the wrist portion of the hand portion in the robot 2, and has a detection range R on the palm side.

[0015] The rotation state derivation device 4, although details will be described later, for example, as shown in FIG. 2, derives the rotation state (for example, the amount of rotation and the direction of rotation of the basketball 10, etc.) of the basketball 10 based on the image data detected by the sensor 3 for the basketball 10. The DB5 stores the image data and the data indicating the rotation state of the basketball 10 derived in the past.

[0016] Next, the processing flow of the rotation state derivation device 4 in the robot system 1 of the present embodiment will be described. FIG. 3 is a flowchart showing the processing flow of the rotation state derivation device in the robot system of the present embodiment. Here, in the present embodiment, it is assumed that the rotation state of the basketball 10 is derived.

[0017] First, the rotation state derivation device 4 acquires the image data detected by the sensor 3 for the basketball 10 (S1). Then, the rotation state derivation device 4 calculates the distance to the basketball 10 (that is, the distance between the sensor 3 and the basketball 10) based on the image data, and acquires the amount of rotation of the basketball 10 included in the data indicating the rotation state of the basketball 10 derived last time from the DB5.

[0018] Next, the rotation state derivation device 4 determines whether the distance to the basketball 10 is greater than or equal to the first threshold value and the rotation amount of the basketball 10 is less than the second threshold value (S2). Here, FIG. 4(a) is a diagram for explaining the detection range of the sensor and the like in a state where the fingertip of the hand portion of the robot touches the basketball, FIG. 4(b) is a diagram showing the image data detected by the sensor in a state where the fingertip of the hand portion of the robot touches the basketball, FIG. 4(c) is a diagram for explaining the detection range of the sensor and the like in a state where the robot's hand portion holds the basketball, and FIG. 4(d) is a diagram showing the image data detected by the sensor in a state where the robot's hand portion holds the basketball.

[0019] As shown in FIG. 2, when the distance to the basketball 10 is greater than or equal to the first threshold value and substantially the entire area of the basketball 10 exists within the detection range R of the sensor 3, the sensor 3 can detect substantially the entire area of the basketball 10. Therefore, the displacement of the pattern (i.e., the feature portion) of the basketball 10 can be detected well.

[0020] On the other hand, as shown in FIGS. 4(a) and 4(c), when the distance to the basketball 10 is less than the first threshold value and substantially the entire area of the basketball 10 does not exist within the detection range R of the sensor 3, as shown in FIGS. 4(b) and 4(d), the sensor 3 cannot detect substantially the entire area of the basketball 10. Therefore, there may be a case where the displacement of the pattern of the basketball 10 cannot be detected well.

[0021] Further, for example, when the pattern such as the basketball 10 has regularity such as symmetry, when the rotation amount of the basketball 10 is greater than or equal to the second threshold value and the displacement between the pattern of the basketball 10 shown in the image data detected by the sensor 3 last time and the pattern of the basketball 10 shown in the image data detected by the sensor 3 this time cannot be clearly recognized, there may be a case where the displacement of the mutual patterns cannot be detected well.

[0022] Therefore, if the distance to basketball 10 is greater than or equal to the first threshold, and the amount of rotation of basketball 10 is less than the second threshold (YES in S2), the rotation state derivation device 4 performs the normal rotation state derivation process (S3). Figure 5 is a diagram illustrating the flow of the normal rotation state derivation process.

[0023] In detail, first, the rotation state derivation device 4 acquires three-dimensional point cloud data based on image data as shown in Figure 5(a) detected by the sensor 3. Then, the rotation state derivation device 4 uses a method such as the RANSAC (Random Sample Consensus) algorithm to acquire point cloud data from the three-dimensional point cloud data that shows the spherical shape of the basketball 10 and the position (i.e., coordinates) of the basketball 10 in the environment, as shown in Figure 5(b).

[0024] Next, the rotation state derivation device 4 acquires a two-dimensional image as shown in Figure 5(c) based on the image data detected by the sensor 3. Then, the rotation state derivation device 4 performs feature extraction from the two-dimensional image using methods such as SIFT (Scale-Invariant Feature Transform) or FAST (Features from Accelerated Segment Test) to acquire the pattern of the basketball 10 as shown in Figure 5(d).

[0025] Next, the rotation state derivation device 4 extracts point cloud data corresponding to the pattern of the basketball 10, and derives the rotation state of the basketball 10 based on the displacement of the point cloud data corresponding to the pattern of the basketball 10 acquired this time, as shown in Figure 5(f), relative to the point cloud data corresponding to the pattern of the basketball 10 acquired last time, as shown in Figure 5(e).

[0026] On the other hand, if the distance to the basketball 10 is greater than or equal to the first threshold and the amount of rotation of the basketball 10 is not less than the second threshold (NO in S2), that is, if the distance to the basketball 10 is less than the first threshold or the amount of rotation of the basketball 10 is greater than or equal to the second threshold, the rotation state derivation device 4 does not perform the normal rotation state derivation process, but estimates the posture of the basketball 10 at the current time based on data indicating the rotation state of the basketball 10 that has been derived in the past, and derives the rotation state of the basketball 10 (S4).

[0027] In detail, for example, the rotation state derivation device 4 acquires data from DB5 indicating the rotation state of the basketball 10 that was derived in the previous and the time before that, and based on this data, estimates the orientation of the basketball 10 at the current time and generates point cloud data for that orientation.

[0028] The rotation state derivation device 4 then extracts point cloud data (second point cloud data) from the point cloud data of the basketball 10's posture at the estimated current time, corresponding to the point cloud data (first point cloud data) based on image data as shown in Figure 4(b) or Figure 4(d) in which the basketball 10 was detected by the sensor 3, and derives the rotation state of the basketball 10 by matching the second point cloud data with the first point cloud data.

[0029] In this manner, if the distance to basketball 10 is less than the first threshold, or if the amount of rotation of basketball 10 is greater than or equal to the second threshold, the current orientation of basketball 10 is estimated based on previously derived data indicating the rotation state of basketball 10, and the rotation state of basketball 10 is derived.

[0030] In other words, this method derives the rotational state of basketball 10 at the current time, without relying solely on image data detected by sensor 3. Therefore, it is possible to accurately derive the rotational state of basketball 10. Here, matching can be performed using methods such as ICP (Iterative Closest Point).

[0031] Next, based on the data showing the rotation state of the basketball 10 derived as described above, we will explain the control flow when the robot 2 dribbles the basketball 10. Figure 6 is a flowchart showing the control flow when the robot dribbles the basketball.

[0032] First, the control unit 2b of robot 2 acquires data from DB5 indicating the current rotation state of basketball 10 (S11). Then, the control unit 2b of robot 2 determines whether the amount of topspin of basketball 10 included in the data indicating the current rotation state of basketball 10 is within a preset rotation range (S12).

[0033] If the amount of topspin of the basketball 10 is within the specified rotation range (YES in S12), the control unit 2b of the robot 2 controls each actuator 2a of the robot 2 to perform a normal dribble without adding topspin or backspin to the rotation of the basketball 10 (S13). In other words, the control unit 2b of the robot 2 basically controls each actuator 2a.

[0034] On the other hand, if the amount of topspin of the basketball 10 is not within the rotation range, that is, if the amount of topspin of the basketball 10 is outside the rotation range (NO in S12), the control unit 2b of the robot 2 determines whether the amount of topspin of the basketball 10 is greater than the rotation range (S14).

[0035] If the amount of topspin of the basketball 10 is greater than the specified rotation range (YES in S14), the control unit 2b of the robot 2 controls each actuator 2a of the robot 2 so that backspin is added to the rotation of the basketball 10 (S15).

[0036] On the other hand, if the amount of topspin of the basketball 10 is not greater than the rotation range (NO in S14), that is, if the amount of topspin of the basketball 10 is less than the rotation range, the control unit 2b of the robot 2 controls each actuator 2a of the robot 2 so that topspin is added to the rotation of the basketball 10 (S16).

[0037] In the robot 2 of this embodiment, when dribbling the basketball 10, the basketball 10 approaches the sensor 3 and the distance to the basketball 10 falls below the first threshold. However, based on the data indicating the rotation state of the basketball 10 at the current time, which has been successfully derived as described above, the robot 2 can be made to dribble the basketball 10. Therefore, the robot 2 can dribble the basketball 10 with high accuracy.

[0038] In this embodiment, the rotation state derivation device 4 estimates the current orientation of the basketball 10 based on previously derived data indicating the rotation state of the basketball 10, and derives the rotation state of the basketball 10, when the distance to the basketball 10 is less than the first threshold, or the amount of rotation of the basketball 10 is greater than or equal to the second threshold.

[0039] In other words, this time, the rotation state of the basketball 10 at the current time is derived without relying solely on the image data detected by sensor 3. Therefore, the rotation state of the basketball 10 can be derived accurately.

[0040] Furthermore, the robot system 1 of this embodiment causes the robot 2 to dribble the basketball 10 based on the data indicating the rotation state of the basketball 10 at the current time, which was derived as described above. Therefore, the robot 2 can dribble the basketball 10 with high precision.

[0041] Here, it is advisable to supervise learning the operation of each actuator 2a of the robot 2 when it makes a free throw of the basketball 10, in relation to the rotation state of the basketball 10 when the robot 2 places the basketball 10 on the hand part 2d to make a free throw (i.e., the arrangement of the pattern when the basketball 10 is stationary).

[0042] This allows the robot 2 to operate each actuator 2a according to the rotation state of the basketball 10 when it is placed on the hand unit 2d, enabling the robot 2 to throw free throws with high precision toward the hoop using the basketball 10.

[0043] <Embodiment 2> In Embodiment 1, when the amount of rotation of the basketball 10 is greater than or equal to a second threshold, the current posture of the basketball 10 is estimated based on data indicating the rotation state of the basketball 10 derived in the past, and the rotation state of the basketball 10 is derived. However, for example, the rotation state of the basketball 10 may be derived as follows.

[0044] For example, similar to the normal rotation state derivation process, the rotation state derivation device 4 matches the point cloud data corresponding to the pattern of the basketball 10 acquired in the previous session with the point cloud data corresponding to the pattern of the basketball 10 acquired in the current session. Then, by removing the point clouds that successfully matched, it eliminates the point cloud data corresponding to the pattern that has regularity.

[0045] In other words, the rotation state derivation device 4 extracts point cloud data corresponding to unique patterns such as letters, which have a lower matching success rate than the point cloud data corresponding to patterns with regularity, from the point cloud data of the basketball 10 corresponding to patterns acquired in the previous and current sessions.

[0046] The rotation state derivation device 4 then derives the rotation state of the basketball 10 based on the displacement of the point cloud data corresponding to the unique patterns such as letters on the basketball 10, which was acquired in the previous session, and the point cloud data corresponding to the unique patterns such as letters on the basketball 10, which was acquired in the current session.

[0047] In this way, point cloud data corresponding to the regular pattern of the basketball 10 is excluded, and the rotation state of the basketball 10 is derived using point cloud data corresponding to unique patterns such as letters. Therefore, even if a regular pattern like the basketball 10 exists, the rotation state of the basketball 10 can be derived accurately.

[0048] Here, methods such as ICP are used for matching point cloud data, but when the number of data points is large, the computational load on the rotation state derivation device 4 becomes high, so it is advisable to downsample during matching. In this case, it is advisable to change the amount of downsampling depending on the matching.

[0049] For example, in order to extract point cloud data of a basketball 10 based on image data detected by sensor 3, when matching point cloud data based on image data previously detected by sensor 3 with point cloud data based on image data currently detected by sensor 3, it is preferable to set the downsampling amount as the first reduction amount.

[0050] Furthermore, when matching point cloud data of parts corresponding to patterns with regularity, the downsampling amount should be set to a second reduction amount which is smaller than the first reduction amount, and when matching point cloud data of parts corresponding to unique patterns, the downsampling amount should be set to a third reduction amount which is smaller than the second reduction amount (for example, no downsampling). This reduces the computational burden on the rotation state derivation device 4.

[0051] <Other Embodiments> In the embodiments described above, the disclosure was explained as a hardware configuration, but the disclosure is not limited thereto. The disclosure can also be implemented by having a CPU (Central Processing Unit) execute a program for each process.

[0052] Here, a program includes a set of instructions (or software code) that, when loaded into a computer, cause the computer to perform one or more functions. A program may be stored on a non-temporary computer-readable medium or a physical storage medium. Examples, but not limited to, include random-access memory (RAM), read-only memory (ROM), flash memory, solid-state drive (SSD) or other memory technologies, CD-ROM, digital versatile disc (DVD), Blu-ray® disc or other optical disc storage, magnetic cassette, magnetic tape, magnetic disk storage or other magnetic storage devices. A program may be transmitted over a temporary computer-readable medium or a communication medium. Examples, but not limited to, include temporary computer-readable medium or a communication medium that includes electrically, optically, acoustically or otherwise propagating signals.

[0053] This disclosure is not limited to the embodiments described above, and may be modified as appropriate without departing from the spirit of the invention. For example, in the above embodiment, since the sensor 3 is fixed to the hand portion 2d of the robot 2, the position and orientation of the sensor 3 can be derived based on the rotation angles of each joint of the robot 2. However, if the position and orientation of the sensor 3 cannot be derived, the position and orientation of the sensor 3 can be derived based on the point cloud data of the environmental portion other than the basketball 10 shown in the image data detected by the sensor 3. For example, in the above embodiment, the rotational state of the basketball 10 was derived, but any sphere would suffice. Also, in the above embodiment, the sensor 3 is fixed to the robot 2, but the sensor 3 can be fixed to any moving object. For example, in the above embodiment, if the amount of rotation of the basketball 10 is greater than or equal to the second threshold, the rotation state derivation device 4 estimates the current orientation of the basketball 10 based on data indicating the rotation state of the basketball 10 derived in the past and derives the rotation state of the basketball 10, or derives the rotation state of the basketball 10 based on the displacement of point cloud data of parts corresponding to unique patterns such as letters on the basketball 10. However, at least if the distance to the basketball 10 is greater than or equal to the first threshold, it is sufficient to estimate the current orientation of the basketball 10 based on data indicating the rotation state of the basketball 10 derived in the past and derive the rotation state of the basketball 10. [Explanation of Symbols]

[0054] 1. Robot System 2 robot, 2a actuator, 2b control unit, 2c robot arm, 2d hand unit 3 sensors 4. Rotational State Derivation Device 5 Databases 6 Network 10 Basketball R sensor detection range

Claims

1. A rotation state derivation device that derives the rotation state of a sphere based on image data detected by a sensor on the sphere, If the distance between the sensor fixed to the moving body and the sphere is greater than or equal to a preset first threshold, the characteristic part of the sphere is extracted, and the rotational state of the sphere is derived based on the time-series displacement of the extracted characteristic part. A rotation state derivation device that, when the distance between the sensor and the sphere is less than the first threshold, estimates the orientation of the sphere at the current time based on data indicating the rotation state of the sphere derived in the past, and derives the rotation state of the sphere.

2. The rotation state derivation device according to claim 1, wherein if the amount of rotation of the sphere included in the previously derived data indicating the rotation state of the sphere is greater than or equal to a preset second threshold, the device estimates the orientation of the sphere at the current time based on the previously derived data indicating the rotation state of the sphere and derives the rotation state of the sphere.

3. The rotation state derivation device according to claim 1, wherein if the amount of rotation of the sphere included in the previously derived data indicating the rotation state of the sphere is greater than or equal to a preset second threshold, a feature portion of the sphere is extracted, a feature portion having regularity is excluded from the extracted feature portion, and the rotation state of the sphere is derived based on the time-series displacement of the remaining feature portion.

4. The aforementioned sensor is a three-dimensional distance measuring sensor. A rotation state derivation device according to any one of claims 1 to 3, wherein, when the distance between the sensor and the sphere is less than the first threshold, a second point cloud data of the region corresponding to the first point cloud data from the estimated point cloud data of the sphere's orientation at the current time is matched with a first point cloud data based on image data of the sphere detected by the sensor, and the rotation state of the sphere is derived.

5. A rotation state derivation device according to claim 1 or 2, A robot with a robotic arm, A sensor fixed to the base of the hand portion of the robot arm, A control unit that controls the robot arm based on the rotation state of the sphere derived from image data of the sphere detected by the sensor, Equipped with, If the amount of top spin of the sphere included in the data indicating the rotation state of the sphere is greater than a preset rotation range, the control unit controls the robot arm so that back spin is added to the sphere. A robot system in which, if the amount of top spin of the sphere included in the data indicating the rotation state of the sphere is less than the rotation range, the control unit controls the robot arm so that top spin is added to the sphere.

Images