Holding member and information processing system

Abstract

A system is provided, including at least one information processing device, which individually or cooperatively implements: a first image acquisition function of obtaining a first image of an object; an angle information acquisition function of obtaining angle information indicating an angle of the object in the first image; a dictionary data generation function of generating dictionary data based on the first image and the angle information; a second image acquisition function of obtaining a second image of an object different from the object of the first image; and an angle estimation function of estimating the angle of the object in the second image based on the second image and the dictionary data.

Description

[0001] This divisional application is filed on November 28, 2017, with application number 201780074636.2, and entitled "System, Fixture, Information Processing Apparatus, Information Processing Method and Program". Technical Field

[0002] This invention relates to systems, jigs, information processing devices, information processing methods, and programs. Background Technology

[0003] Image-based object recognition has become a common technique in recent years. In image-based object recognition, for example, features are extracted from an image captured by an imaging device, and these features are matched with features pre-registered as dictionary data to identify objects in the image. Here, changes in the angle of an object in the image also change the feature values; therefore, dictionary data needs to be prepared for each angle of the object to increase the usability of object recognition.

[0004] PTL 1 and PTL 2 are examples of techniques for preparing dictionary data for each angle of an object in image-based object recognition. PTL 1 describes a technique for recognizing an object based on eight images obtained by rotating the object at 45-degree intervals. PTL 2 describes a technique for learning an object model by identifying common parts from a large number of images obtained by rotating the object in horizontal and zenith directions in increments of five degrees.

[0005] [List of Citations]

[0006] [Patent Literature]

[0007] [PTL 1]

[0008] Japanese Patent Publication No. 2003-346152

[0009] [PTL 2]

[0010] U.S. Patent Application Publication No. 2013 / 0202212 Summary of the Invention

[0011] [Technical Issues]

[0012] These techniques are used to identify objects in images; that is, to estimate what the objects in an image are, without intending to extract additional information from the image. However, given the diverse fields where object recognition techniques have been used in recent years, providing additional information about objects based on images may be advantageous.

[0013] Therefore, the object of the present invention is to provide novel and improved systems, fixtures, information processing devices, information processing methods and programs that can estimate the angle of an object based on an image.

[0014] [Solution to the problem]

[0015] One aspect of the present invention provides a system comprising one or more information processing devices, which individually or collaboratively perform the following functions: a first image acquisition function for acquiring a first image of an object; an angle information acquisition function for acquiring angle information indicating the angle of the object in the first image; a dictionary data generation function for generating dictionary data based on the first image and the angle information; a second image acquisition function for acquiring a second image of an object different from the first image; and an angle estimation function for estimating the angle of the object in the second image based on the second image and the dictionary data.

[0016] In another aspect, the present invention provides a clamp capable of being attached to a platform device, the platform device including a base portion rotatable about a first axis, a pair of supports fixed at positions symmetrical to the first axis relative to the base portion, a pair of arms respectively connected to the pair of supports for pivoting about a second axis perpendicular to the first axis on opposite sides of the base portion, a retainer fixed between the edges of the pair of arms on opposite sides of the pair of supports, and a control unit providing the angle of rotation of the base portion about the first axis and the angle of pivoting of the pair of arms about the second axis. The clamp includes: an attachment member, a retainer capable of being attached to the platform device, an object retainer for attaching an object, and a connecting member connecting the attachment member and the object retainer and defining the positional relationship between the attachment member and the object retainer such that when the attachment member is attached to the retainer of the platform device, the object attached to the object retainer is located near the intersection of the first axis and the second axis.

[0017] In addition, another aspect of the present invention provides an information processing apparatus, comprising: a processor that performs the following functions: an image acquisition function that acquires multiple images of an object that are different from each other; an angle information acquisition function that acquires angle information common to the multiple images; an angle information indicating the angle of the object; and a dictionary data generation function that generates dictionary data based on the multiple images and the angle information common to the multiple images.

[0018] In addition, another aspect of the present invention provides an information processing apparatus, comprising: a processor that performs the following functions: a dictionary data acquisition function that acquires a first image of an object and dictionary data generated based on angle information indicating the angle of the object in the first image; an image acquisition function that acquires a second image of an object different from the first image; and an angle estimation function that estimates the angle of the object in the second image based on the second image and the dictionary data.

[0019] In another aspect, the present invention provides an information processing method, including the steps of acquiring multiple images of an object that are different from each other, the steps of acquiring angle information common to the multiple images, the angle information indicating the angle of the object, and the steps of generating dictionary data using a processor based on the multiple images and the angle information common to the multiple images.

[0020] In addition, another aspect of the present invention provides an information processing method, including the steps of acquiring dictionary data generated based on a first image of an object and angle information indicating the angle of the object in the first image, the steps of acquiring a second image of an object different from the first image, and the steps of using a processor to estimate the angle of the object in the second image based on the second image and the dictionary data.

[0021] In addition, another aspect of the present invention provides a program for enabling a processor to perform functions, including: an image acquisition function for acquiring multiple images of objects that are different from each other, an angle information acquisition function for acquiring angle information common to the multiple images, an angle information indicating the angle of the object, and a dictionary data generation function for generating dictionary data based on the multiple images and the angle information common to the multiple images.

[0022] In addition, another aspect of the present invention provides a program for enabling a processor to perform functions, including: a dictionary data acquisition function for acquiring dictionary data generated based on a first image of an object and angle information indicating the angle of the object in the first image; a second image acquisition function for acquiring a second image of an object different from the first image; and an angle estimation function for estimating the angle of the object in the second image based on the second image and the dictionary data. Attached Figure Description

[0023] Figure 1 This is a schematic diagram illustrating a system according to a first embodiment of the present invention.

[0024] Figure 2 It is shown Figure 1 The diagram shows the functional configuration of the terminal in the system.

[0025] Figure 3 It is shown Figure 1 A schematic perspective view of the configuration of the platform equipment and fixtures used in the system shown.

[0026] Figure 4 It is along Figure 3 The sectional view taken from line II.

[0027] Figure 5 This is a conceptual diagram used to describe the dictionary data generated in the first embodiment of the present invention.

[0028] Figure 6 It is used to describe Figure 1A diagram illustrating a schematic configuration of the robot in the system shown.

[0029] Figure 7 It is shown Figure 1 The diagram shows the functional configuration of the robot in the system.

[0030] Figure 8 This is a flowchart illustrating an example of dictionary data generation processing in the first embodiment of the present invention.

[0031] Figure 9 This is a flowchart illustrating an example of the identification process in the first embodiment of the present invention.

[0032] Figure 10 This is a flowchart illustrating an example of the trimming process in the first embodiment of the present invention.

[0033] Figure 11 It is used to describe Figure 10 The diagram shows a concept of the pruning process.

[0034] Figure 12 This is a flowchart illustrating an example of dictionary data update processing in the first embodiment of the present invention.

[0035] Figure 13 This is a block diagram illustrating the functional configuration of a robot according to a second embodiment of the present invention.

[0036] Figure 14 This is a schematic diagram used to describe the third embodiment of the present invention.

[0037] Figure 15 This is a block diagram illustrating the functional configuration of a robot according to a third embodiment of the present invention.

[0038] Figure 16 This is a block diagram illustrating an example of the hardware configuration of an information processing apparatus in an embodiment of the present invention. Detailed Implementation

[0039] In the following description, some embodiments of the invention will be described in detail with reference to the accompanying drawings. Note that in this specification and the drawings, repeated descriptions will be omitted by providing the same reference numerals to component elements having substantially the same functional configuration.

[0040] (First Embodiment)

[0041] Figure 1 This is a schematic diagram illustrating system 10 according to a first embodiment of the present invention. (See reference) Figure 1 System 10 includes a terminal 100, a database 200, and a robot 300 interconnected via a network NW. A camera 150 and a platform device 160 are connected to the terminal 100. The robot 300 includes a camera 310 and a robotic arm 320.

[0042] In system 10, camera 150 captures an image of an object obj attached to platform device 160 via clamp 170, described later. Terminal 100 acquires the image from camera 150 and angle information indicating the angle of object obj from platform device 160. Note that in the following description, unless otherwise specified, the angle of object obj is an angle in three-dimensional space, such as an angle indicated by the amount of rotation along three axes in an orthogonal coordinate system. Terminal 100 generates dictionary data based on the acquired image and angle information (and the identification information of object obj). The generated dictionary data is stored in database 200.

[0043] On the other hand, robot 300 uses camera 310 to capture an image of object obj, while robotic arm 320 holds object obj. Robot 300 identifies object obj in the image based on dictionary data obtained from database 200 and the captured image, and further estimates the angle of object obj in the image.

[0044] In this way, robot 300 can identify object obj and further estimate the angle of object obj held by robotic arm 320. This angle indicates, for example, the degree of rotation of object obj relative to a reference pose. Robot 300 can, for example, control robotic arm 320 to rotate object obj based on the estimated angle, thereby setting object obj to a desired pose.

[0045] System 10 can be used, for example, to automate the arrangement or organization of goods using robot 300. System 10 can also be used, for example, to specify how to rotate object obj in order to read information (such as printed codes and radio frequency identification (RFID)) arranged at predetermined portions of object obj. Note that the use of system 10 is not limited to these examples and can have a variety of other uses.

[0046] (Configuration for generating dictionary data)

[0047] Figure 2 It is shown Figure 1 A block diagram illustrating the functional configuration of terminal 100 in the system shown. (Reference) Figure 2Terminal 100 includes an image acquisition unit 110, an angle information acquisition unit 120, and a dictionary data generation unit 130. Terminal 100 may be, for example, a personal computer, tablet computer, smartphone, etc., and the functions of components implemented by the hardware configuration of an information processing device described later. Specifically, for example, the functions of the image acquisition unit 110, the angle information acquisition unit 120, and the dictionary data generation unit 130 are implemented by a processor included in the information processing device. The dictionary data 210 generated by the dictionary data generation unit 130 is stored in a database 200 connected to terminal 100 via a network. The functions of database 200 are implemented by storing one or more information processing devices connected to the network. Note that in the case where terminal 100 includes multiple processors, the multiple processors can cooperate to implement the functions of the components. Additionally, as described later, some or all of the functions implemented by the processor of terminal 100 may also be implemented by a server. The functions of the components will be described below.

[0048] Image acquisition unit 110 acquires an image of object obj captured by camera 150. Here, camera 150 is an example of an imaging device that captures images of objects. Specifically, camera 150 is, for example, a digital camera including an image sensor, and image acquisition unit 110 receives image data generated by camera 150. Although camera 150 is connected to terminal 100 via a wired communication interface such as Universal Serial Bus (USB) (in the illustrated example), in another example, camera 150 can be connected to terminal 100 via a wireless communication interface such as Bluetooth (registered trademark). Alternatively, camera 150 can be built into terminal 100 and can transmit image data to image acquisition unit 110 via a bus.

[0049] The angle information acquisition unit 120 acquires angle information indicating the angle of the object obj from the platform device 160. Here, in this embodiment, the angle information acquired by the angle information acquisition unit 120 of the terminal 100 indicates the angle of the object obj based on the coordinate system of the platform device 160. Note that the case where the angle information acquisition unit 120 generates the angle information of the object obj, sends the angle information to the platform device 160, and provides the angle information to the dictionary data generation unit 130 is also included in the case of "angle information acquisition unit 120 acquiring angle information". In this case, the platform device 160 sets the angle of the object obj based on the angle information received from the angle information acquisition unit 120. In this embodiment, the platform device 160 is an example of a holding component for holding the object obj. Similar to the camera 150, the platform device 160 can also be connected to the terminal 100 via a wired communication interface or via a wireless communication interface.

[0050] As described above, the angle of object obj is an angle in three-dimensional space, such as an angle indicated by the rotation amount of the three axes in an orthogonal coordinate system. Therefore, the angle information acquisition unit 120 represents the angle information by, for example, a rotation amount equal to the difference between the current pose and the reference pose of object obj. Here, the reference pose is, for example, the pose of object obj when platform device 160 is reset. Alternatively, the reference pose may be the pose of object obj when image acquisition unit 110 first acquires an image of object obj to generate dictionary data 210.

[0051] The dictionary data generation unit 130 generates dictionary data 210 based on the image acquired by the image acquisition unit 110, the identification information of the object obj, and the angle information acquired by the angle information acquisition unit 120. Here, the identification information of the object obj can be specified in any way. For example, the identification information of the object obj can be specified based on information input by the user to the terminal 100. Alternatively, the identification information of the object obj can be specified by matching the image acquired by the image acquisition unit 110 with dictionary data separately provided for image-based object recognition. Or, the dictionary data generation unit 130 can assign the identification information to the object obj, which is typically included in multiple images acquired by the image acquisition unit 110.

[0052] Note that known techniques in image-based object recognition can be appropriately used in this embodiment to combine the image and identification information of the object obj in the information used to generate dictionary data 210. For example, the dictionary data generation unit 130 can use appropriate methods used in image-based object recognition to extract feature quantities from the image, and can associate the extracted feature quantities with the identification information and angle information of the object obj. Alternatively, for example, the dictionary data generation unit 130 can use the identification information of the object obj, which has been classified and labeled by appropriate methods used in image-based object recognition.

[0053] Furthermore, although dictionary data 210 is generated based on the identification information of object obj in the description of this embodiment, it is possible to generate dictionary data 210 without using the identification information of object obj. For example, if system 10 is provided for a single type of object obj, dictionary data 210 may not include the identification information of object obj. On the other hand, if dictionary data 210 includes the identification information of object obj as in this embodiment, multiple types of object obj can be identified, and then the angle of object obj can be further estimated.

[0054] (Configuration of platform equipment and fixtures)

[0055] In the following description, the configuration of the platform device 160 used with the terminal 100 in the system 10 according to this embodiment and the configuration of the fixture 170 for attaching the object obj to the platform device 160 will be further described.

[0056] Figure 3 It is shown Figure 1 A schematic perspective view of the configuration of the platform device 160 and fixture 170 used in the system shown. Figure 4 It is along Figure 3 A sectional view taken from line II. (Reference) Figure 3 and Figure 4 Platform device 160 includes a base 161, a pair of supports 162, a pair of arms 163, a pair of pins 164, a holder 165, a beam 166, and a control unit 167. Fixture 170 includes an attachment member 171, a connecting member 172, an object holder 173, and a background plate 174. Note that... Figure 3 Background panel 174 is not shown. Each component will be described below.

[0057] In platform device 160, base 161 is, for example, a rotary table. Base 161 is driven by a motor (not shown) controlled by control unit 167 and rotates about axis A1. Here, axis A1 is parallel to the optical axis of camera 150 (e.g., ...). Figure 4 A pair of support members 162 are fixed to the base 161 at positions symmetrical with respect to axis A1. Therefore, the midpoints of the pair of support members 162 substantially coincide with axis A1. Pins 164 are located on opposite sides of the base 161 for connecting a pair of arms 163 to the pair of support members 162, respectively. Pins 164 are located on axis A2, which is orthogonal to axis A1. Each of the pair of arms 163 is pivotable about axis A2. Specifically, the pair of support members 162 and pins 164 are connected by gears, or pins 164 and the pair of arms 163 are connected by gears. A motor (not shown), controlled by control unit 167, is connected to the gears, and the pair of arms 163 pivot about axis A2.

[0058] The retainer 165 is secured to the edges of a pair of arms 163 by beams 166 on opposite sides of a pair of supports 162. Although the retainer 165 is a component with a camera mounted on it, in the case where the platform device 160 is used as an automated platform for a camera, in this embodiment, as described later, the attachment member 171 of the clamp 170 is attached to the retainer 165. As described above, the retainer 165 pivots about axis A2 when the pair of arms 163 revolve. Here, while the retainer 165 rotates about axis A2 according to the configuration of the pair of arms 163, the attachment surfaces 165s of the retainer 165 remain facing axis A2.

[0059] Control unit 167 is, for example, a microcontroller integrated into platform device 160, and control unit 167 controls the motor as described above to control the rotation of base 161 and the pivoting of a pair of arms 163. The control unit controls motor 167, for example, according to a predetermined program or instructions from terminal 100. Thus, control unit 167 sets the rotation angle of base 161 about axis A1 and the pivoting angle of the pair of arms 163 about axis A2. Angle information acquisition unit 120 of terminal 100 acquires, for example, angle information indicating the value of the angle set by control unit 167.

[0060] The aforementioned platform device 160 is primarily a means of automating the translation (rotation about axis A1) and tilt (switching about axis A2) of the camera attached to the retainer 165. This embodiment aims to use the platform device 160 to automatically set the angle of the object obj, thereby effectively generating dictionary data 210 covering various angles. However, when the object obj is directly attached to the retainer 165 of the platform device 160, when the pair of arms 163 pivot, the retainer 165 swings about axis A2, resulting in the position of the object obj becoming significantly away from the optical axis of the camera 150 (e.g., ...). Figure 4 (As shown by axis A3 in the diagram). Therefore, as described below in this embodiment, object obj is attached to platform device 160 via clamp 170.

[0061] In the clamp 170, the attachment member 171 is a member that can be attached to the retainer 165 of the platform device 160. For example, the attachment member 171 is provided with an attachment structure corresponding to the structure for fixing a camera mounted on the retainer 165. Specifically, if the retainer 165 is provided with screws for fixing the camera, the attachment member 171 is provided with screw holes. Alternatively, the attachment member 171 may be provided with an attachment structure that can be used regardless of the structure of the retainer 165. Specifically, the attachment member 171 may be provided with clips for clamping the retainer 165, straps wrapped around the retainer 165, etc.

[0062] The object holder 173 is a component for attaching an object obj. For example, the object holder 173 is provided with an attachment structure that can fix the object obj while minimizing the contact area with it. This is because the contact area between the attachment structure and the object obj can become an occlusion area in the image of the object obj captured by the camera 150. Specifically, the object holder 173 may be provided with clips for holding the object obj, hooks for gripping the object obj, adhesive surfaces for attaching the object obj, etc. Additionally, the object holder 173 may be provided with a magnet for attaching the object obj as a magnet.

[0063] Connecting member 172 connects attachment member 171 and object holder 173. Furthermore, connecting member 172 defines the positional relationship between attachment member 171 and object holder 173 such that when attachment member 171 is attached to holder 165 of platform device 160, object obj attached to object holder 173 is positioned near the intersection of axes A1 and A2. For example, connecting member 172 is connected to attachment member 171 such that when connecting member 171 is attached to holder 165, it extends along a pair of arms 163. In this case, the length of connecting member 172 in the direction along the pair of arms 163 is substantially equal to the distance obtained by subtracting half the thickness of attachment member 171 and object holder 173 plus the thickness of the object from the distance between holder 165 and axis A2. Connecting member 172 may have a structure that allows its length to be adjusted along the direction of arms 163. This allows the length of connecting member 172 to be adjusted according to the size of object obj so that the center of object obj is close to the intersection of axes A1 and A2.

[0064] As described above, the object obj, attached to the platform device 160 by the clamp 170, is located near the intersection of axes A1 and A2. Therefore, even when the base 161 of the platform device 160 rotates about axis A1 or when the pair of arms 163 pivot about axis A2, the position of the object obj remains substantially unchanged and does not significantly deviate from the optical axis of the camera 150 (e.g., ...). Figure 4 (As shown in axis A3). Therefore, when the control unit 167 of the platform device 160 sets the rotation angle of the base 161 about axis A1 and the angle of the pivot of the pair of arms 163 about axis A2, in this embodiment, the angle can be considered as the amount of rotation of the object obj about axis A1 and axis A2.

[0065] Note that when using platform device 160 and fixture 170, although object obj does not rotate about axis A3 (i.e., the optical axis of camera 150) which is perpendicular to axes A1 and A2, rotation about axis A3 can be precisely supplemented by planar rotation of the image captured by camera 150. Furthermore, although the description above simplifies by stating that object obj is on the optical axis of camera 150, object obj may not actually be on the optical axis of camera 150.

[0066] Background plate 174 is attached to connecting member 172 or object holder 173 and provides a background for object obj. For example, background plate 174 may be provided with an attachment structure for selectively attaching screens. Screens may include multiple screens formed of, for example, different materials. Examples of materials include paper, cloth, and film. Screens may also include multiple screens of different colors or different reflective properties. Screens can be interchanged to provide multiple interchangeable backgrounds for object obj with different materials, colors, reflective properties, etc. Additionally, background plate 174 may be detachably attached to connecting member 172 or object holder 173, for example. In this case, multiple background plates 174 can be selectively attached to provide multiple interchangeable backgrounds for object obj with different materials, colors, reflective properties, etc. Specifically, background plate 174 may include, for example, multiple background plates 174, wherein the surface facing object obj is formed of different materials. Examples of materials include paper, cloth, and film. Background plate 174 may also include multiple background plates 174, wherein the surface facing object obj has different colors or different reflective properties.

[0067] (Conceptual description of dictionary data)

[0068] Figure 5 This is a conceptual diagram used to describe the dictionary data generated in the first embodiment of the present invention. Figure 5 Dictionary data 210 is shown associated with an object obj (the connector in the illustrated example) specified by specific identification information. In the illustrated example, the angle of object obj is a vector indicated by the amount of rotation about three axes (X-axis, Y-axis, and Z-axis) of an orthogonal coordinate system in three-dimensional space. Regarding the angle of object obj, dictionary data 210 includes at least N... X ×N Y ×N Z An element, which is formed by dividing the entire circumference into N elements with respect to a rotation (rot_X) about the X-axis. X An element that divides the entire circumference into N elements with respect to a rotation (rot_Y) about the Y-axis. Y The elements, and N that divides the entire circumference into rotations about the Z-axis (rot_Z). Z Defined by an element. Each element is associated with information corresponding to at least one image of the object obj. Here, the information corresponding to the image of the object obj can be, for example, feature quantities extracted from the image captured by the camera 150 when the object obj is at an angle indicated by the rotation amount (rot_X, rot_Y, rot_Z).

[0069] Note that in this example, the width of the division regarding the rotation of the axes (rot_X, rot_Y, rot_Z) can be different (i.e., N). X N Yand N Z At least one of them can be different from the others. Additionally, the rotation amounts may not be uniformly divided. For example, in the angle estimation of object obj described later, where there is an angle that cannot be estimated with high reliability, the division width of the rotation amounts near that angle can be set smaller than the division width of other parts.

[0070] For example, if the camera 310 of robot 300 captures an image of object obj at an unknown angle, the feature quantities extracted from the captured image and the feature quantities associated with the elements of dictionary data 210 can be matched to estimate the angle of object obj.

[0071] Here, dictionary data 210 may include multiple elements generated based on the angle information of the same object obj and multiple different images. In this case, the number of elements in dictionary data 210 is greater than N. X ×N Y ×N Z For example, the environmental conditions for capturing multiple images can vary among multiple images associated with the same angular information. Environmental conditions can be, for example, the arrangement of background or light. Generating dictionary data 210 under multiple different environmental conditions can provide dictionary data 210 that can estimate the angle of object obj under various environmental conditions.

[0072] In the above scenario, the image acquisition unit 110 of terminal 100 acquires multiple different images of object obj. For example, when the control unit 167 of platform device 160 sets the same angle before and after exchanging the background of object obj using the background plate 174 of fixture 170, the image acquisition unit 110 can acquire an image of object obj. In this case, the dictionary data generation unit 130 generates multiple elements of dictionary data 210 based on multiple images with different backgrounds, identification information of object obj common to the multiple images, and angle information indicating the angle of object obj common to the multiple images.

[0073] (Robot configuration)

[0074] Figure 6 It is used to describe Figure 1 A schematic diagram of the configuration of robot 300 in the system shown. (Reference) Figure 6The robot 300 includes a camera 310, a robotic arm 320, a control unit 330, a sensor 340, and a motor 350. The robot 300 can, for example, use the robotic arm 320 to hold an object obj, and, under the control of the control unit 330, use the camera 310 to capture images of the object obj. In this embodiment, the robotic arm 320 is also an example of a holding component for holding the object obj, similar to the platform device 160. The control unit 330 is implemented via a hardware configuration of, for example, an information processing device described later.

[0075] Sensor 340 includes sensors for acquiring various measurements used by robot 300 or transmitted from robot 300 to another device. Specifically, sensor 340 may include an accelerometer, an angular velocity sensor, a geomagnetic sensor, and / or a Global Navigation Satellite System (GNSS) receiver. Sensor 340 may also include laser range scanners, such as depth sensors and laser imaging detection and ranging (LIDAR).

[0076] Motor 350 activates each component of robot 300 under the control of control unit 330. Motor 350 may include, for example, motors (actuators) for activating joint structures (not shown) to change the robot's posture or move robot 300. Motor 350 may also include motors for rotating wheels to move robot 300. Note that the configuration of each component of robot 300, including motor 350, can be a suitable configuration based on known methods of robot design. Here, robot 300 may not change posture or may not move. Similarly, robot 300 may not include joint structures (except for robotic arm 320) or may not include wheels.

[0077] (Configuration of estimated object angle)

[0078] Figure 7 It is shown Figure 1 A block diagram illustrating the functional configuration of robot 300 in the system shown. (Reference) Figure 7In addition to camera 310 and robotic arm 320, robot 300 also includes image acquisition unit 331, dictionary data acquisition unit 332, object recognition / angle estimation unit 333, result output unit 334, dictionary data update unit 335, robotic arm control unit 336, and angle information acquisition / angle estimation unit 337. Components other than camera 310 and robotic arm 320 are implemented by a processor of an information processing device, such as the control unit 330 that implements robot 300. Note that if control unit 330 includes multiple processors, the multiple processors can cooperate to implement the functions of the components. Additionally, as described later, some or all of the functions implemented by the processor of control unit 330 can also be implemented by a server. The functions of the components will be described below. Note that the functions related to updating dictionary data will be described in detail later with reference to flowcharts, and these functions will be described simply here.

[0079] Image acquisition unit 331 acquires an image of the object obj captured by camera 310. Here, in this embodiment, camera 310 is also an example of an imaging device for capturing images of objects, similar to camera 150. Although the images captured by camera 150 and the images captured by camera 310 include the same type of object obj, the images are different from each other. Specifically, camera 310 is, for example, a digital camera including an image sensor, and image acquisition unit 331 receives image data generated by camera 310. For example, robot 300 uses robotic arm 320 to hold object obj. In this case, the image acquired by image acquisition unit 331 includes the object obj held by robotic arm 320. Alternatively, image acquisition unit 331 may include object obj that is not held by robotic arm 320 but is placed on a table, floor, etc. Although in the illustrated example camera 310 is built into robot 300 and camera 310 sends image data to image acquisition unit 331 via a bus, camera 310 can be externally connected to robot 300 via a wired communication interface or a wireless communication interface.

[0080] The dictionary data acquisition unit 332 acquires dictionary data 210 from a database 200 connected to the robot 300 via a network. As described above, the dictionary data 210 is generated based on the image and angle information (and the identification information of the object obj) of the object obj. The robot 300 uses the dictionary data 210 to estimate the angle of the object obj held by the robotic arm 320. Note that the dictionary data acquisition unit 332 may not acquire the entire dictionary data 210. For example, when the dictionary data 210 is generated for multiple types of objects and the object obj included in the image acquired by the image acquisition unit 331 has already been identified, the dictionary data acquisition unit 332 selectively acquires elements associated with the identification information of the object obj in the dictionary data 210.

[0081] The object recognition / angle estimation unit 333 estimates the angle of the object obj in the image based on the image of the object obj acquired by the image acquisition unit 331 and the dictionary data 210 acquired by the dictionary data acquisition unit 332. When the dictionary data 210 is generated for multiple types of objects and the object obj included in the image acquired by the image acquisition unit 331 is not identified, the object recognition / angle estimation unit 333 uses image-based object recognition to specify the identification information of the object obj. Known techniques can be applied to image-based object recognition, and details will not be described. For example, when the dictionary data 210 is generated for multiple types of objects, or when the object obj included in the image acquired by the image acquisition unit 331 has already been identified, the object recognition / angle estimation unit 333 does not perform object recognition.

[0082] On the other hand, the object recognition / angle estimation unit 333 estimates the angle of object obj by, for example, matching the image acquired by the image acquisition unit 331 with elements of the dictionary data 210. In this case, the angle associated with the element of the dictionary data 210 that has the highest score in the match is estimated as the angle of object obj in the image. As will be described later, the dictionary data 210 used to estimate the angle of object obj may include a large number of elements. Therefore, the object recognition / angle estimation unit 333 can trim the dictionary data 210 based on the image acquired by the image acquisition unit 331, and can match the trimmed dictionary data 210 with the image. Here, in this embodiment, trimming is a process of determining which dictionary data 210 to exclude from the match is a process with a lower processing load than matching used to estimate the angle of object obj.

[0083] The result output unit 334 outputs the result recognized by the object recognition / angle estimation unit 333. As described above, although the robot 300 can use the estimated angle of the object obj for its operation, such as for controlling the robotic arm 320, the robot 300 can also output the estimation result in some form if needed. More specifically, for example, the estimation result can be displayed as an image on the robot 300's display, or it can be output as sound from a speaker. Additionally, the estimation result can be transmitted via a network from a communication device included in the robot 300 to another device. The result output unit 334 controls the output of the estimation result. Note that the result output unit 334 is not provided when it is not necessary to output the estimation result.

[0084] The dictionary data update unit 335 updates the dictionary data 210 based on the estimation result of the angle of the object obj estimated by the object recognition / angle estimation unit 333 and the re-estimation result re-estimated by the angle information acquisition / angle estimation unit 337, which will be described later. More specifically, if the reliability of the angle estimated by the object recognition / angle estimation unit 333 does not exceed a threshold, the dictionary data update unit 335 updates the dictionary data 210 based on the re-estimation result of the angle re-estimated by the angle information acquisition / angle estimation unit 337. Note that in the following description, the estimation function of the angle estimated by the object recognition / angle estimation unit 333 will also be referred to as the "first angle estimation function," and the re-estimation function of the angle re-estimated by the angle information acquisition / angle estimation unit 337 will also be referred to as the "second angle estimation function." The angle estimation functions may not be performed independently of each other. For example, as described later, the angle information acquisition / angle estimation unit 337 re-estimates the angle by using the estimation result of the angle estimated by the object recognition / angle estimation unit 333. That is, there exists a case where the "first angle estimation function" is run alone, and there also exists a case where the "second angle estimation function" is called the "first angle estimation function".

[0085] The robotic arm control unit 336 controls the robotic arm 320 of the robot 300 that holds the object obj. When the dictionary data update unit 335 updates the dictionary data 210, the robotic arm control unit 336 controls the robotic arm 320 to rotate the object obj. Note that the rotation here refers to changing the angle of the object obj. The rotation of the object obj is an example of the physical operation of the object obj that is being performed with respect to the re-estimation of the angle of the object obj.

[0086] The angle information acquisition / angle estimation unit 337 acquires angle information indicating the angle of the object obj from the robot arm control unit 336. Here, in this embodiment, the angle information 300 acquired by the robot's angle information acquisition / angle estimation unit 337 indicates the angle of the object obj based on the coordinate system of the robot 300 or the robot arm 320. Therefore, in this embodiment, the angle information acquired from the robot arm control unit 336 may not be directly associated with the angle information in the dictionary data 210. Therefore, in this embodiment, the angle information acquisition / angle estimation unit 337 calculates the rotation amount Δθ of the object obj based on the angle information before and after the robot arm control unit 336 controls the robot arm 320 to rotate the object obj, and the rotation amount Δθ is used for the re-estimation of the angle described later.

[0087] The angle information acquisition / angle estimation unit 337 re-estimates the angle θ1 of the object obj in the image (first image) before the rotation of the object obj, based on the angle θ2 of the object obj after the rotation (estimated by the object recognition / angle estimation unit 333 based on the image (second image) and dictionary data 210) and based on the rotation amount Δθ. Simply put, θ1 = θ2 - Δθ. Here, the rotation amount Δθ is an example of a physical operation quantity with respect to the object obj. Note that each of the angles θ1, θ2, and Δθ can be, for example, a rotation element including each axis of the coordinate system. Figure 5 The vectors of rot_X, rot_Y, and rot_Z in the example.

[0088] If the reliability of the angle θ2 of object obj estimated by object recognition / angle estimation unit 333 based on the image after rotation of object obj (second image) and dictionary data 210 exceeds a threshold, dictionary data update unit 335 updates dictionary data 210 based on angle information indicating that angle θ1 is re-estimated by angle information acquisition / angle estimation unit 337 (based on angle θ2) and based on the image before rotation of object obj (first image).

[0089] On the other hand, if the reliability of the angle θ2 of object obj estimated by object recognition / angle estimation unit 333 based on the image (second image) after rotation of object obj and dictionary data 210 does not exceed a threshold, the robotic arm control unit 336 controls the robotic arm 320 to further rotate object obj by a rotation amount Δθ', and object recognition / angle estimation unit 333 estimates angle θ3 of object obj based on the image (third image) after rotation of object obj and dictionary data 210. If the reliability of angle θ3 exceeds the threshold, angle information acquisition / angle estimation unit 337 re-estimates angle θ1 based on angle θ3 and total rotation amount (Δθ+Δθ'), and dictionary data update unit 335 updates dictionary data 210 based on the re-estimation result.

[0090] In this way, once the angle θ1 is re-estimated with sufficient reliability, the dictionary data update unit 335 updates the dictionary data 210 based on the image (first image) before the rotation of the object obj, using angle θ1. Specifically, the dictionary data update unit 335 adds or replaces elements of the dictionary data 210. As a result, when the camera 310 later captures an image of the object obj at angle θ1 under similar environmental conditions, angle θ1 may be estimated with high reliability without re-estimation.

[0091] (Processing flow example)

[0092] In the following text, reference will be made to Figure 8 and Figure 12 An example describing the processing flow in system 10 according to this embodiment.

[0093] Figure 8 This is a flowchart illustrating an example of a dictionary data generation process according to a first embodiment of the present invention. (Reference) Figure 8 In the registration process, the image acquisition unit 110 of terminal 100 first acquires an image (S101), and the angle information acquisition unit 120 acquires angle information (step S103). Either step S101 or S103 can be run first, or they can be run in parallel. For example, once the image acquisition unit 110 acquires the image captured by camera 150 in real time, the angle information acquisition unit 120 can acquire angle information from platform device 160. Furthermore, once the angle information acquisition unit 120 sends the angle information to platform device 160, the image acquisition unit 110 can acquire the image captured by camera 150 in real time. Alternatively, the image acquisition unit 110 can acquire the images captured by camera 150 sequentially over time, and the angle information acquisition unit 120 can acquire the angle information set in platform device 160 sequentially over time.

[0094] Next, the dictionary data generation unit 130 of terminal 100 associates the image acquired in step S101 with the angle information acquired in step S103. For example, if both the image and angle information are acquired in real time, the dictionary data generation unit 130 associates the image with the angle information acquired substantially simultaneously. On the other hand, if the image and angle information are acquired with a time lag or later, the dictionary data generation unit 130 associates the image with the angle information including a public key. In this case, the key may be, for example, a timestamp, or a sequence number separately assigned from the timestamp.

[0095] Next, the dictionary data generation unit 130 generates dictionary data 210 based on the image and angle information associated with each other in step S105 (step S107). Here, as already described, the dictionary data generation unit 130 can apply known image-based object recognition techniques to generate the dictionary data 210. Furthermore, if, for example, the dictionary data generation unit 130 continuously acquires a set of substantially the same image and angle information, the dictionary data generation unit 130 can determine that the information is redundant and skip the generation of dictionary data 210.

[0096] Figure 9 This is a flowchart illustrating an example of the identification process in the first embodiment of the present invention. (See reference...) Figure 9The image acquisition unit 331 of the robot 300 first acquires an image in the recognition process (step S301). As described above, the image acquisition unit 331 acquires an image captured by the camera 310, and the image includes, for example, an object obj held by the robotic arm 320. Next, the dictionary data acquisition unit 332 acquires dictionary data 210 from the database 200 (step S303).

[0097] Next, the object recognition / angle estimation unit 333 identifies the object obj based on the image acquired in step S301 and the dictionary data 210 acquired in step S303 (step S305). Note that known techniques can be applied to image-based object recognition, and details will not be described. Furthermore, as described above, in cases where dictionary data 210 is generated for, for example, a single type of object, or where the object obj included in the image has already been identified, the object recognition in step S305 can be skipped.

[0098] Next, the object recognition / angle estimation unit 333 trims the dictionary data 210 (step S307). For example, in... Figure 5 The diagram shows 52 elements (i.e., N) by dividing the entire circumference into rotations (rot_X, rot_Y, rot_Z) about each axis. X =N Y =N Z When dictionary data 210 is generated using ), the generated dictionary data 210 includes at least 52 3 = 140,608 elements. When multiple different images are associated with the same angle, the number of elements becomes even larger to generate dictionary data 210 as described above. The processing load for matching all elements of dictionary data 210 is enormous, and the benefits of pruning dictionary data 210 are significant.

[0099] Figure 10 This is a flowchart illustrating an example of the trimming process in the first embodiment of the present invention. Figure 11 It is used to describe Figure 10 The diagram shows a conceptual illustration of the pruning process. (Reference) Figure 10 The object recognition / angle estimation unit 333 first determines the trimming process corresponding to the object obj (step S331). For example, the trimming process corresponding to the object obj is pre-set and stored in the database 200 along with the dictionary data 210. During operation... Figure 9 In the case of step S305 shown, the object recognition / angle estimation unit 333 determines the trimming process based on the object recognition result in step S305.

[0100] The following steps S333 and S335 are based on Figure 11The example shown illustrates a processing procedure performed on the object obj corresponding to a trimming process. The processing performed here may vary depending on the object type. In this example, the object recognition / angle estimation unit 333 masks the image (step S333) and further reduces the image's color (step S335). Next, the object recognition / angle estimation unit 333 performs trimming (step S337). In the example shown, for example, multiple feature portions are extracted from the masked image with reduced color, and elements in dictionary data 210 that have positional relationships between multiple similar extracted feature portions that differ from the image are removed from the matching target.

[0101] exist Figure 11 In the example shown, object obj is a connector. The trimming process set up in the example focuses on the color of the cables (cable 1 to cable 3). Figure 10 In step S333 shown, the portion of the image other than the cables is masked (the mask is in...). Figure 11 (Indicated as MSK). This eliminates the shadow effect of the terminal cover present in the shielded portion. Furthermore, although not in Figure 1 The text indicates that in step S335, the colors of the image are reduced to represent the difference in cable colors at the two edges (cable 1 and cable 3). This allows for easy extraction of the cables at the two edges (cable 1 and cable 3) as each element of the image and two feature portions in the dictionary data 210.

[0102] In addition, Figure 10 In step S337, the dictionary data 210 is trimmed based on the masked image with reduced colors. Specifically, for example, when viewed from the image, cable 1 is located in the upper right when viewed from the image of cable 3. On the other hand, when viewed from element group 210b (connector rotated about the viewing axis) of the dictionary data 210, cable 1 is located in the upper left. Additionally, when viewed from element group 210c (connector reversed), cable 1 is located in the lower left. Therefore, element groups 210b and 210c are removed from the matching target in step S337. As a result, matching is performed only on element group 210a (e.g., in the image, cable 1 is located in the upper right when viewed from the image of cable 1).

[0103] Back Figure 9 After trimming the dictionary data 210 in step S307, the object recognition / angle estimation unit 333 matches the image with the dictionary data 210 (step S309). The matching can be, for example, template matching. Note that known techniques can be applied to image matching, and details will not be described. Note that although the object-based score is calculated based on known matching results in image-based object recognition, the object-based angle score is calculated in step S307.

[0104] Next, the object recognition / angle estimation unit 333 estimates the angle of object obj based on the matching result in step S309 (S311). The estimation result in step S311 may be, for example, an angle indicated by angle information associated with an element of dictionary data 210 that has the highest score calculated in the matching in step S309.

[0105] Next, the object recognition / angle estimation unit 333 determines whether the score calculated in the matching in step S309 exceeds a threshold (step S313). Here, the score compared with the threshold is, for example, the highest matching score. Alternatively, it can be determined whether a certain highest percentage of the matching score (e.g., 10%) exceeds the threshold. If the matching score does not exceed the threshold in the determination in step S313 (No), the dictionary data update unit 335 updates the dictionary data 210 (step S315). On the other hand, if the matching score exceeds the threshold in the determination in step S313 (Yes), the process of updating the dictionary data 210 may not be run. If necessary, the result output unit 334 outputs the estimation result from step S311.

[0106] Figure 12 This is a flowchart illustrating an example of dictionary data update processing in a first embodiment of the present invention. (Reference) Figure 12 The robot 300's angle information acquisition / angle estimation unit 337 first stores the angle information of the object obj provided by the robotic arm control unit 336 in the update process (step S351). Here, the angle stored in step S351 is, for example, based on the angle of the object obj in the coordinate system indicated by the robotic arm 320. Next, the robotic arm control unit 336 controls the robotic arm 320 to rotate the object obj (step S353).

[0107] After rotating object obj, estimate the angle of object obj (step S355). The processing in step S355 corresponds to, for example... Figure 9 The processing steps S301 to S311 are shown. Specifically, the image acquisition unit 331 acquires a rotated image (second image) of the object obj, and the object recognition / angle estimation unit 333 estimates the angle of the object obj in the rotated image (second image). Note that the dictionary data 210 acquired in the previously executed step S303 can be used, and it can be assumed that the object obj has been identified in the previously executed step S305.

[0108] Next, the dictionary data update unit 335 determines whether the estimated matching score in step S355 exceeds a threshold (step S357). This determination can be made, for example... Figure 9The process proceeds as shown in step S313. If the matching score does not exceed the threshold in the determination of step S357 (No), the processing of steps S353 and S355 is re-run. That is, the robotic arm control unit 336 controls the robotic arm 320 to further rotate the object obj (step S353), and the object recognition / angle estimation unit 333 estimates the angle of the object obj in the rotated image (third image) (step S355).

[0109] On the other hand, if the matching score exceeds the threshold in step S357 (yes), the angle information acquisition / angle estimation unit 337 re-estimates the initial angle θ1 from the angle θ2 estimated in step S355 and the rotation amount Δθ of the object obj (step S359). Here, the initial angle θ1 is the angle of the object obj before rotation, and it is the angle that the object recognition / angle estimation unit 333 could not estimate with sufficient reliability. On the other hand, the angle θ2 is the angle of the object obj estimated by the object recognition / angle estimation unit 333 based on the image of the object obj after rotation (second image) and dictionary data 210, and it is proven in step S357 that the angle was estimated with sufficient reliability. In addition, the rotation amount Δθ is calculated based on the angle information of the object obj stored in step S353 and the angle information of the object obj provided by the robotic arm control unit 336 in step S353.

[0110] Note that, as a result of step S357, if the processing of steps S353 and S355 is repeated N times, the angle information acquisition / angle estimation unit 337 estimates the angle θ from the last run of step S355. N+1 The total rotation Δθ of object obj in step S353, which is run N times. TTL The initial angle θ1 is re-estimated. The total rotation Δθ is calculated based on the angle information of object obj stored in step S353 and the angle information of object obj provided by the robot arm control unit 336 in step S353. TTL .

[0111] Next, the dictionary data update unit 335 will update the angle information corresponding to the initial angle θ1 re-estimated in step S359 with the data in... Figure 9 The image of object obj before rotation (first image) obtained in step S301 is associated with it (step S361). Furthermore, in step S361, dictionary data update unit 335 updates dictionary data 210 based on the associated image and angle information (step S363). Here, updating dictionary data 210 includes adding elements to dictionary data 210 and / or replacing elements of dictionary data 210.

[0112] In step S363 above, the dictionary data update unit 335 adds elements of the dictionary data 210 based on image and angle information. As a result, when the camera 310 of the robot 300 later captures an image of the object obj at angle θ1 under similar environmental conditions, the angle θ1 can be estimated with high reliability. Note that, for example, if the dictionary data 210 is specific to the robot 300 and it is expected that the environmental conditions when the camera 310 captures an image of the object obj will not change, the dictionary data update unit 335 can replace elements of the dictionary data 210 based on image and angle information.

[0113] The dictionary data 210 can be updated as described above to accumulate additional dictionary data 210 regarding the angle or environmental conditions of the object obj, because it is difficult to make a highly reliable estimate for the object obj using the initially generated dictionary data 210. In this way, the robot 300, which uses the dictionary data 210 to estimate the angle of the object obj, can autonomously enhance the dictionary data 210 to improve the robustness of the estimate.

[0114] (Example of validation processing before update)

[0115] Here, for reference Figure 12 The described dictionary data update process may include additional processing, which is a validation process performed before updating dictionary data 210. For the first example, validation can be run to check if... Figure 12 The dictionary data update process (shown as step S371 "Verification Process 1") runs prior to step S351. In the verification process according to the first example, the image acquisition unit 331 re-acquires an image of the object obj in step S353 before the object obj rotates. The object recognition / angle estimation unit 333 estimates the angle of the object obj in the re-acquired image. If the estimated matching score exceeds a threshold (with... Figure 9 (If the estimation in step S311 is different), cancel the dictionary data update process, and at least do not run the dictionary data update in step S363.

[0116] For example, in Figure 9 In step S301, the image acquired by the image acquisition unit 331 may exhibit unexpected variations due to chance factors (such as a delay in the focus of the camera 310 or a momentary change in lighting conditions (e.g., caused by lightning or a flash)), which may reduce the reliability of the estimation. The verification process, as shown in the first example, is effective in preventing the updating of dictionary data 210 based on information with low reproducibility due to chance factors.

[0117] Additionally, for the second example, it is possible to... Figure 12Following step S361, a process is run to verify whether the dictionary data is updated based on the prepared angle information and image (shown as step S373, "Verification Process 2"). In the verification process according to the second example, the dictionary data update unit 335 generates temporary dictionary data based on the angle information and image associated in step S361. Next, the robotic arm control unit 336 controls the robotic arm 320 to rotate the object obj, which is the opposite of step S353. This causes the object obj to return to the original angle θ1. Furthermore, the image acquisition unit 331 acquires a new image of the object obj that has returned to the original angle θ1, and the object recognition / angle estimation unit 333 estimates the angle of the object obj in the new image acquired by the image acquisition unit 331 based on the temporary dictionary data generated by the dictionary data update unit 335. Here, if the original angle θ1 can be estimated and the matching score exceeds a threshold, the dictionary data update unit 335 updates the dictionary data 210 of step S363. Otherwise, the dictionary data 210 of step S363 is not updated.

[0118] The second example is effective for preventing updates to dictionary data 210 that do not contribute to improving the reliability of angle estimation. Based on the environmental conditions when camera 310 captures an image of object obj, even if dictionary data 210 is updated based on an image acquired by image acquisition unit 331, the reliability of angle estimation in similar images acquired later may not be improved. The verification process in the second example is effective for preventing an increase in the size of dictionary data 210 due to unnecessary elements that may not contribute to improving the reliability of angle estimation.

[0119] (Other modifications)

[0120] Note that although the angle information acquisition / angle estimation unit 337 re-estimates the angle of object obj after rotation in this example, the angle information acquisition / angle estimation unit 337 can also re-estimate the angle in another example after robot 300 is moved along with object obj by motor 350. The environmental conditions when camera 310 captures images can be changed by the movement of robot 300, and the angle can be estimated with high reliability without rotating object obj. Note that the configuration for moving robot 300 is described in more detail in the third embodiment described later.

[0121] Furthermore, the movement of robot 300 can be combined with the rotation of object obj. For example, if sufficient reliability is still not obtained in the re-estimation of the angle after the rotation of object obj, the angle information acquisition / angle estimation unit 337 can re-estimate the angle after robot 300 moves together with object obj. For example, the re-estimation process may be effective if the environmental conditions when camera 310 captures an image of object obj are significantly different from the environmental conditions of camera 150 when generating dictionary data 210.

[0122] By distributing functions to Figure 1 , 2 and Figure 7 The terminal 100, database 200, and robot 300 in the example shown implement the functions of system 10 according to this embodiment. In another example, most of the functions of system 10 can be implemented by a server. That is, the functions implemented by the processors of the terminal 100 and robot 300 described in the example can also be implemented by the processor of a server including database 200. In this case, terminal 100 sends an image of object obj captured by camera 150 and angle information of object obj obtained from platform device 160 to the server, and the server associates them to generate dictionary data 210. On the other hand, robot 300 sends an image of object obj captured by camera 310 to the server, and the server estimates the angle of object obj based on the image. Robot 300 receives the angle estimation result from the server. If the reliability of the estimated angle does not exceed a threshold, the server can request robot 300 to rotate object obj to re-estimate the angle, and acquire an image of object obj after rotation. Note that the number of servers implementing the functions may not be one, and multiple servers distributed across the network can implement the functions. In addition, the server implementing the functions may be a device separate from the memory including database 200.

[0123] (Second Embodiment)

[0124] Next, a second embodiment of the present invention will be described. Note that repeated descriptions can be omitted by providing common reference numerals to components having a similar configuration to that of the first embodiment.

[0125] Figure 13 This is a block diagram illustrating the functional configuration of a robot 300a according to a second embodiment of the present invention. (See reference) Figure 13In this embodiment, robot 300a implements all the functions of generating dictionary data 210 and estimating the angle of object obj using dictionary data 210. Specifically, the processing of control unit 330 of robot 300a implements image acquisition units 110 and 331, angle information acquisition / angle estimation units 120 and 337, dictionary data generation / update units 130 and 335, dictionary data acquisition unit 332, object recognition / angle estimation unit 333, result output unit 334, and robotic arm control unit 336. Note that when control unit 330 includes multiple processors, the multiple processors can cooperate to realize the functions of the components. In addition, as described later, some or all of the functions implemented by the processors of control unit 330 can also be implemented by a server. In addition, database 200 is stored in the memory of control unit 330 of robot 300a. These components will be further described below.

[0126] Image acquisition units 110 and 331 have reference Figure 2 The image acquisition unit 110 and reference described Figure 7 The functions of both image acquisition units 331 are described below. Specifically, image acquisition units 110 and 331 provide the image of object obj captured by camera 310 to dictionary data generation / update units 130 and 335 to generate dictionary data 210, and provide the image to object recognition / angle estimation unit 333 to estimate the angle of object obj using dictionary data 210.

[0127] Angle information acquisition / angle estimation units 120 and 337 have reference Figure 2 The described angle information acquisition unit 120 and reference Figure 7 The functions of the angle information acquisition / angle estimation unit 337 are described below. Specifically, angle information acquisition / angle estimation units 120 and 337 provide angle information acquired from the robotic arm control unit 336 to dictionary data generation / update units 130 and 335 to generate dictionary data 210. Furthermore, angle information acquisition / angle estimation units 120 and 337 calculate the rotation amount Δθ of object obj based on the angle information acquired from the robotic arm control unit 336, and further estimate the initial angle θ1 based on the rotation amount Δθ and the angle θ2 estimated by the object recognition / angle estimation unit 333, in order to update the dictionary data 210.

[0128] Note that in this embodiment, the angle information acquired by the angle information acquisition / angle estimation unit 337 of the robot 300 can indicate the angle of the object obj based on the coordinate system of the robotic arm 320. In this case, the angle of the object obj indicated by the angle information acquired by the angle information acquisition / angle estimation unit 337 can be changed not only by the amount of rotation of the robotic arm 320 set by the robotic arm control unit 336, but also by the amount of operation of other components of the robot 300 (such as the arm) connected to the robotic arm 320. In addition, which surface of the object obj is held by the robotic arm 320 can also change at any time. Therefore, even when the robotic arm 320 holds the object obj in the same way as when generating dictionary data 210, it is beneficial to use dictionary data 210 to estimate the angle of the object obj in the image.

[0129] Dictionary data generation / update units 130 and 335 have reference Figure 2 The dictionary data generation unit 130 and reference described Figure 7 The functions of both dictionary data update units 335 are described. Specifically, dictionary data generation / update units 130 and 335 generate dictionary data 210 based on images acquired by image acquisition units 110 and 331 and angle information acquired by angle information acquisition / angle estimation units 120 and 337. Furthermore, dictionary data generation / update units 130 and 335 update dictionary data 210 based on the estimated angle of object obj estimated by object recognition / angle estimation unit and the re-estimated angle by angle information acquisition / angle estimation units 120 and 337, so that dictionary data 210 can be used to estimate the angle of object obj.

[0130] As shown in the second embodiment, the functionality of system 10 according to the first embodiment can be implemented by a single device such as robot 300a. In this case, it can also be said that system 10 is implemented by a single device. Similarly, the configuration of system 10 can be implemented by various device configurations. For example, system 10 may include multiple robots 300, and each robot 300 may generate dictionary data 210 and estimate the angle of an object by using the dictionary data 210. In this case, the dictionary data 210 stored in database 200 is shared by multiple robots 300.

[0131] Additionally, for example, in the second embodiment, the server including database 200 can implement the functions implemented by the control unit 330 of robot 300a. In this case, robot 300a, which generates dictionary data, sends an image of object obj captured by camera 310 and angle information of object obj obtained from robotic arm control unit 336 to the server, and the server concatenates them to generate dictionary data 210. On the other hand, robot 300a, which estimates angles, sends an image of object obj captured by camera 310 to the server, and the server estimates the angle of object obj based on the image. Robot 300a receives the angle estimation result from the server. The server can also request robot 300a to rotate object obj to re-estimate the angle, and, provided that the reliability of the estimated angle does not exceed a threshold, acquire an image of object obj after rotation.

[0132] (Third Embodiment)

[0133] Next, a third embodiment of the invention will be described. Note that repeated descriptions will be omitted by providing common reference numerals for components having a configuration similar to that of the second embodiment.

[0134] Figure 14 This is a schematic diagram used to illustrate a third embodiment of the present invention. (Reference) Figure 14 In this embodiment, robot 300b moves relative to object obj instead of using a robotic arm to hold the object. In the example shown, the movement of robot 300b includes rotation (REV) around the object. In this case, object obj rotates about axis A1 in the image captured by camera 310. The movement of robot 300b also includes tilting (TLT) relative to object obj with respect to camera 310. In this case, object obj rotates about axis A2 in the image captured by camera 310.

[0135] Figure 15 This is a block diagram illustrating the functional configuration of a robot 300b according to a third embodiment of the present invention. The robot 300b according to this embodiment and... Figure 13 Unlike robot 300a, robot 300b includes a motor control unit 339, which controls motor 350 in place of robotic arm control unit 336 controlling robotic arm 320.

[0136] Motor control unit 339 controls motor 350 of robot 300. (See reference...) Figure 6 The motor 350 includes a motor for activating the joint structure of the robot 300 or rotating the wheels of the robot 300b to move the robot 300b or change the posture of the robot 300b. (See reference...) Figure 14The motor control unit 339 controls the motor 350 to rotate the robot 300b around the object obj and / or tilt the robot 300b relative to the object obj of the camera 310.

[0137] Angle information acquisition / angle estimation units 120 and 337b acquire angle information indicating the angle of the object obj. Here, angle information is acquired, for example, using multiple time-series images acquired by image acquisition unit 331 during the movement of robot 300 and camera 310, to run image-based simultaneous localization and mapping (SLAM). Note that SLAM can also be run using measurements from other sensors 340 included in robot 300a, such as depth sensors and laser range scanners. In this case, angle information acquisition / angle estimation units 120 and 337b use SLAM to specify the amount of movement of camera 310, and then acquire the angle information of object obj based on a separately specified positional relationship between camera 310 and object obj. Alternatively, angle information acquisition / angle estimation units 120 and 337b can specify the amount of movement of camera 310 based on the value of motor 350 controlled by motor control unit 339.

[0138] This embodiment allows the use of the angle information obtained as described above to generate dictionary data 210. Furthermore, if the object recognition / angle estimation unit 333 cannot estimate the angle with sufficient reliability based on the dictionary data 210, the motor control unit 339 can control the motor 350 to rotate the object obj in the image, thereby re-estimating the angle and updating the dictionary data 210. In this embodiment, the relative movement of the camera 310 relative to the object obj is an example of the physical operation of the object obj performed when re-estimating its angle.

[0139] According to the configuration of the third embodiment of the present invention described above, even when the object obj is large or when the object obj is small but cannot move, dictionary data 210 for estimating the angle of the object obj can be generated. Here, the robot 300b may also include, as referenced... Figure 7 The robotic arm 320 and robotic arm control unit 336 are described above, and the robotic arm 320 can be used to rotate the object obj, as in the first and second embodiments, while the object obj can be held.

[0140] Note that in the example of the third embodiment described above, although the robot 300b performs all the functions of generating dictionary data 210 and estimating the angle of object obj using dictionary data 210, as in the second embodiment, other examples are possible. For example, the robot 300 in the system 10 according to the first embodiment may include a motor control unit 339 in place of the robotic arm control unit 336, or may include a motor control unit 339 in addition to the robotic arm control unit 336.

[0141] For example, if the size of the platform device 160 (or robot 300) used to generate dictionary data 210 is different from the size of the robot 300 used to estimate the angle of object obj using dictionary data 210, it is possible that although the platform device 160 (or robotic arm 320) can be used to rotate object obj to generate dictionary data 210, it is difficult to rotate object obj when updating dictionary data 210. The opposite situation may also exist.

[0142] Furthermore, for example, as described above, if the robot 300 includes a motor control unit 339 in addition to the robotic arm control unit 336, the motor control unit 339 can control the motor 350 to move the camera 310 together with the object obj. In this case, the robotic arm control unit 336 controls the robotic arm 320 to prevent changes in the angle of the object obj in the image. Specifically, the robotic arm control unit 336 maintains the positional relationship between the robotic arm 320 and the camera 310, as well as the angle of the object obj held by the robotic arm 320, while the motor control unit 339 controls the motor 350 to move the robot 300.

[0143] In this way, camera 310 can move along with object obj to change, for example, the environmental conditions when camera 310 captures an image, without changing the angle of object obj in the image. As a result, for example, in situations where it is difficult to reliably estimate the angle of object obj based on dictionary data 210 under certain environmental conditions, a highly reliable estimation can be achieved by changing the environmental conditions. Furthermore, when generating dictionary data 210, multiple elements that associate common angle information with multiple images acquired under different environmental conditions can be included in the dictionary data 210, thereby improving the robustness of angle estimation.

[0144] In this example, the motor control unit 339 first controls the motor 350 to move the camera 310 and the object obj together during the dictionary data 210 update process. After moving the camera 310 and the object obj, the image acquisition unit 331 acquires an image (second image) after the movement of the object obj, and the object recognition / angle estimation unit 333 re-estimates the angle of the object obj in the image after the movement (second image). If the matching score in the estimation exceeds a threshold, the dictionary data update unit 335 updates the dictionary data based on the angle information corresponding to the re-estimated angle of the object obj and the image acquired by the image acquisition unit 331 before the movement of the object obj (first image). In this example, the movement of the camera 310 and the object obj is a physical operation of the object obj running in the re-estimation of the object obj's angle. Furthermore, in this example, the object recognition / angle estimation unit 333 performs both the "first angle estimation function" and the "second angle estimation function".

[0145] (Example of hardware configuration for an information processing device)

[0146] Next, we will refer to Figure 16 An example describing the hardware configuration of an information processing apparatus according to an embodiment of the present invention. Figure 16 This is a block diagram illustrating an example of the hardware configuration of an information processing apparatus according to an embodiment of the present invention.

[0147] The information processing device 900 includes a processor 901, a memory 903, an input device 905, an output device 907, and a bus 909. The information processing device 900 may also include a storage device 911, a driver 913, a connection port 915, and a communication device 917.

[0148] Processor 901 includes, for example, processing circuitry such as a central processing unit (CPU), digital signal processor (DSP), application-specific integrated circuit (ASIC), and / or field-programmable gate array (FPGA). Processor 901 functions as an operation processing device and control device, and controls the operation of information processing device 900 according to a program recorded in memory 903, storage device 911, or removable recording medium 919.

[0149] The memory 903 includes, for example, read-only memory (ROM) and random access memory (RAM). ROM stores, for example, programs and operating parameters for the processor 901. RAM mainly stores, for example, programs deployed when the processor 901 is running and parameters for running the programs.

[0150] Input device 905 is, for example, a user-operated device such as a mouse, keyboard, touchpad, button, and various switches. Input device 905 may not be integrated with information processing device 900 and may be, for example, a remote control that transmits control signals wirelessly. Input device 905 includes input control circuitry that generates input signals based on user input information and outputs the input signals to processor 901.

[0151] Output device 907 includes means for outputting information to a user using senses such as sight, hearing, and touch. Output device 907 may include, for example, display devices (such as liquid crystal displays (LCDs) and organic electroluminescent (EL) displays), voice output devices (such as speakers and headphones), and vibrators. Output device 907 outputs the results obtained in the processing of information processing device 900 in the form of images (such as text and pictures), sounds (such as speech and audio), or vibrations.

[0152] Storage device 911 includes, for example, magnetic storage devices such as hard disk drives (HDDs), semiconductor storage devices, optical storage devices, magneto-optical storage devices, etc. Storage device 911 stores, for example, programs for processor 901, various types of data read during program execution or generated by program execution, and various types of data acquired from external sources.

[0153] Drive 913 is a reader / writer for removable recording media 919, such as magnetic disks, optical disks, magneto-optical disks, and semiconductor memories. Drive 913 reads information recorded on the installed removable recording media 919 and outputs that information to memory 903. Drive 913 also writes various types of data to the installed removable recording media 919.

[0154] Connection port 915 is a port used to connect external connection device 921 to information processing device 900. Connection port 915 may include, for example, a USB port, an IEEE 1394 port, a Small Computer System Interface (SCSI) port, etc. Connection port 915 may also include an RS-232C port, an optical audio port, a High Definition Multimedia Interface (HDMI) (registered trademark) port, etc. External connection device 921 can be connected to connection port 915 to exchange various types of data between information processing device 900 and external connection device 921.

[0155] Communication device 917 is connected to network 923. Note that network 923 can be, for example, an open communication network (such as the Internet) in which a large number of unspecified devices are connected, or it can be, for example, a closed communication network, such as Bluetooth (registered trademark) in which a limited number of devices are connected, such as two devices. Communication device 917 may include, for example, a local area network (LAN), Bluetooth (registered trademark), Wi-Fi, or a communication card for wireless USB (WUSB). Communication device 917 uses a predetermined protocol according to network 923 to send signals, data, etc., to other information processing devices, and to receive signals, data, etc., from other information processing devices.

[0156] An example of the hardware configuration of the information processing apparatus 900 has been described above. Each constituent element can be provided using general-purpose components, or it can be provided using hardware specific to the function of each constituent element. Furthermore, those skilled in the art can appropriately modify the configuration of the information processing apparatus 900 according to the level of technology available at the time of implementation.

[0157] Embodiments of the present invention may include, for example, the system, fixture and information processing apparatus described above, the information processing method operated by the information processing apparatus, the program for operating the information processing apparatus, and the non-transitory tangible medium for recording the program.

[0158] Although some embodiments of the invention have been described in detail above with reference to the accompanying drawings, the invention is not limited to these examples. It will be apparent to those skilled in the art that various changes or modifications can be made within the technical concept described in the claims, and it should be understood that such changes and modifications obviously fall within the technical scope of the invention.

[0159] [List of Reference Symbols]

[0160] 10...System, 100...Terminal, 110...Image Acquisition Unit, 120...Angle Information Acquisition Unit, 130...Dictionary Data Generation Unit, 150...Camera, 160...Platform Equipment, 161...Base, 162...Support Member, 163...Arm, 164...Pin, 165...Retainer, 167...Control Unit, 170...Clamp, 171...Attachment Member, 172...Connecting Member, 173...Object Retainer, 174...Background Plate, 200...Database, 2 10...Dictionary data, 300, 300a, 300b...Robot, 310...Camera, 320...Robotic arm, 330...Control unit, 331...Image acquisition unit, 332...Dictionary data acquisition unit, 333...Object recognition / angle estimation unit, 334...Result output unit, 335...Dictionary data update unit, 336...Robotic arm control unit, 337...Angle information acquisition / angle estimation unit, 339...Motor control unit, 340...Sensor, 350...Motor.

Claims

1. A retaining member for holding an object, comprising: Platform equipment; as well as A clamp capable of being attached to the platform device. The platform equipment includes: The base portion rotates about the first axis. A pair of support members, fixed at positions symmetrical to the first axis relative to the base portion. A pair of arms, each connected to one of the pair of supports, are configured to pivot about a second axis perpendicular to the first axis on opposite sides of the base portion. A retainer, located on opposite sides of the pair of supports, fixed between the edges of the pair of arms, and The control unit sets the angle of rotation of the base portion about the first axis and the angle of pivoting of the pair of arms about the second axis. The clamp includes: An attachment member capable of being attached to the retainer of the platform device. An object retainer for attaching the object, and A connecting member connects the attachment member and the object holder, and defines the positional relationship between the attachment member and the object holder such that when the attachment member is attached to the holder of the platform device, the object attached to the object holder is located at the intersection of the first axis and the second axis.

2. The retaining component according to claim 1, further comprising: A pair of pins, located on the second shaft, are used to connect the pair of arms to the pair of supports, respectively.

3. The holding member according to claim 2, wherein The pair of supports and the pair of pins are connected by gears, or the pair of pins and the pair of arms are connected by gears.

4. The retaining member according to claim 1, further comprising: A beam is used to secure the retainer between the edges of the pair of arms.

5. The retaining member according to claim 1, wherein, The connecting member has a structure that extends along the pair of arms of the platform device when the attachment member is attached to the retainer of the platform device, and the structure is adjustable in length along the direction of the pair of arms.

6. The retaining member according to claim 1, further comprising: Background panel, which provides an interchangeable background for the object attached to the object holder.

7. The retaining member of claim 1, wherein, When the object is held by the holding member, the holding member is also configured to indicate the angle information of the object.

8. The holding member according to claim 1, wherein The retaining component can be communicatively connected to the terminal device wirelessly or via a wired connection.

9. An information processing system, comprising: Holding member for holding an object according to any one of claims 1-8; as well as One or more information processing devices, individually or collaboratively performing the following functions, including: The first image acquisition function retrieves the first image of the object. An angle information acquisition function that obtains angle information indicating the angle of the object in the first image from the holding component. Dictionary data generation function that generates dictionary data based on the first image and the angle information. A second image acquisition function that acquires a second image of the object that is different from the first image, and An angle estimation function that estimates the angle of an object in the second image based on the second image and the dictionary data.

Images