Holding device

Abstract

Provided is a holding device capable of stably holding an object by determining the orientation of the object while holding the object. The holding device comprises: a housing including an opening; a contact unit covering the opening of the housing, formed of an elastic body having light transmittance, and configured to come into contact with an object to be held and maintain the contact; a photographing unit installed within the housing and configured to acquire images of the surface of the contact unit and the object to be held; and a recognition device configured to recognize the state of contact between the contact unit and the object to be held, on the basis of the images of the surface of the contact unit and the object to be held, the images being acquired by the photographing unit.

Description

Maintenance device

[0001] The present invention relates to a maintenance device.

[0002] In robotic arms and the like, a holding device is used that detects and holds a holding object using a tactile sensor. Patent Document 1 discloses a tactile sensor including a transmitting portion having a first surface capable of contacting a holding object and a second surface which is the back surface of the first surface, an imaging portion capable of capturing an image of an object existing on the first surface side of the transmitting portion from the second surface side, and a reflecting portion arranged on the second surface side of the transmitting portion to reflect light from at least a portion of the transmitting portion and guide it into the capturing field of view of the imaging portion.

[0003] The above information may be provided as background information to aid in understanding the present disclosure. No claim or determination is made as to whether any of the above is applicable as prior art related to the present disclosure.

[0004] [Patent Document]

[0005] International Publication No. 2021 / 001992

[0006] In maintaining objects using robotic arms, it is required to recognize various objects, determine their posture, and maintain them stably.

[0007] One embodiment of the present disclosure can address the problems and / or disadvantages mentioned above and provide at least the advantages described below. Accordingly, one embodiment of the present disclosure can provide a holding device capable of stably fixing an object by determining its position. One embodiment of the present disclosure will be described below, some of which may be clearly understood through the description of the present disclosure.

[0008] The purpose of the present invention is to provide a holding device that can stably maintain an object by identifying the position of the object while holding the object.

[0009] According to one embodiment of the present disclosure, a maintenance device is provided.

[0010] According to one embodiment of the present disclosure, a holding device may include a housing including an opening, a contact portion that covers the opening of the housing, is formed of an elastic body having light-transmitting properties, and is configured to contact a holding object and maintain contact with the holding object, a photographing portion that is installed within the housing and is configured to acquire an image of a surface of the contact portion and the holding object, and a recognition device that is configured to recognize a state of contact between the contact portion and the holding object based on an image of the surface of the contact portion and the holding object acquired by the photographing portion.

[0011] The above contact portion includes a suction hole for sucking air, and the holding object can be sucked by sucking air from the suction hole by aligning the position of the suction hole with the position of the holding object.

[0012] In addition, a marker having a predetermined geometric pattern is arranged on the surface of the contact portion that comes into contact with the object to be maintained, and the recognition device can recognize the contact state of the object to be maintained on the contact portion based on the deformation of the marker in the image captured by the photographing unit.

[0013] Additionally, the surface of the contact portion that comes into contact with the maintenance target may include a film formed of a resin having light-transmitting properties.

[0014] In addition, a marker having a predetermined pattern is attached to the film formed on the contact portion, and the recognition device can recognize the contact state of the object to be maintained on the contact portion based on the deformation amount of the marker in the image captured by the photographing unit.

[0015] In addition, the above-described contact portion may further include a lighting unit that irradiates light from inside the housing for photographing by the photographing unit.

[0016] The above-mentioned photographing unit may include a group of light-receiving elements configured to output an electric signal according to light reception, and a plurality of optical systems that form a plurality of images on the group of light-receiving elements.

[0017] The plurality of optical systems of the above-described photographing unit may include a lens and an aperture plate having a plurality of apertures for allowing a plurality of incident lights to enter the lens so that the light passing through the lens forms a plurality of images on the light receiving element group.

[0018] The plurality of optical systems of the above photographing unit may include a first optical system that sets a focal length to a surface of the contact unit that contacts the object to be maintained and forms an image on the light-receiving element group, and a second optical system that sets a focal length to a position further than the surface of the contact unit that contacts the object to be maintained and forms an image on the light-receiving element group.

[0019] The above lens may include one lens including a first portion having a long-distance focal distance set and a second portion having a short-distance focal distance set.

[0020] The above light-receiving element group includes an image sensor, and an image can be formed from the first part of the lens to the first part of the image sensor, and an image can be formed from the second part of the lens to the second part of the image sensor.

[0021] The above lens may include a metalens.

[0022] According to the present invention, a holding device capable of stably maintaining an object by identifying the position of the object while holding the object can be realized.

[0023] The effects that can be obtained from the present disclosure are not limited to the effects mentioned above, and other effects that are not mentioned can be clearly understood by a person having ordinary skill in the art to which the present disclosure belongs from the description below.

[0024] The above-described aspects or other aspects, configurations and / or advantages of various embodiments of the present disclosure may be further clarified by the following detailed description taken in conjunction with the accompanying drawings.

[0025] FIG. 1 is a drawing showing the appearance of a maintenance device having the function of an optical tactile sensor (OTS) according to one embodiment of the present disclosure.

[0026] FIG. 2 is a cross-sectional view showing the internal structure of the holding device illustrated in FIG. 1 according to one embodiment of the present disclosure.

[0027] FIG. 3 is a drawing showing a suction structure of a maintenance device according to one embodiment of the present disclosure.

[0028] FIG. 4 is a drawing showing the structure of a floor plate according to one embodiment of the present disclosure.

[0029] FIG. 5 is a drawing showing the configuration of an optical system of a maintenance device photographing unit according to one embodiment of the present disclosure.

[0030] FIG. 6 is a drawing showing an example of a configuration of a photographing unit according to one embodiment of the present disclosure.

[0031] FIG. 7 is a diagram schematically showing a configuration for obtaining two images, one at a close range and one at a long range, from a photographing unit according to one embodiment of the present disclosure.

[0032] FIG. 8 is a diagram schematically showing another configuration for obtaining two images, one at a close range and one at a long range, from a photographing unit according to one embodiment of the present disclosure.

[0033] FIG. 9 is a drawing showing a state in which a maintenance device according to one embodiment of the present disclosure detects a connector as a target object and a socket as a mounting location of the connector.

[0034] FIG. 10 is a drawing showing a state in which a holding device according to one embodiment of the present disclosure holds a connector.

[0035] FIG. 11 is a drawing showing a state in which a connector maintained by a maintenance device according to one embodiment of the present disclosure is mounted on a socket.

[0036] FIG. 12 is a drawing showing a state in which a holding device according to one embodiment of the present disclosure releases the holding of a connector.

[0037] FIG. 13 is a diagram showing an example of a hardware configuration of a processing device according to one embodiment of the present disclosure.

[0038] FIG. 14 is a diagram showing the functional configuration of a processing device according to one embodiment of the present disclosure.

[0039] FIG. 15 is a drawing showing an example of displacement of a marker according to one embodiment of the present disclosure, wherein FIG. 15(A) is a drawing showing a marker in an initial state, FIG. 15(B) is a drawing showing a marker state at time t, and FIG. 15(C) is a drawing showing a state in which the marker positions of FIGS. 15(A) and (B) are superimposed.

[0040] FIG. 16 is a drawing showing an example of dot overlapping when one dot is displaced according to one embodiment of the present disclosure.

[0041] FIG. 17 is a drawing showing the correspondence of a plurality of dots according to one embodiment of the present disclosure.

[0042] FIG. 18 is a drawing specifying dots excluded from the tracking target in the example of FIG. 15(C) according to one embodiment of the present disclosure.

[0043] FIG. 19 is a drawing showing an example of a first displacement according to one embodiment of the present disclosure.

[0044] FIG. 20 is a drawing showing an example of a second displacement according to one embodiment of the present disclosure.

[0045] FIG. 21 is a graph showing the relationship between the force acting on the contact portion of the holding device and the amount of deformation of the contact portion according to one embodiment of the present disclosure.

[0046] FIG. 22a, FIG. 22b, and FIG. 22c are drawings showing an example of operation of a holding device according to one embodiment of the present disclosure, wherein FIG. 22a is a drawing showing a state in which the holding device holds an object, FIG. 22b is a drawing showing a state in which the holding device is about to install an object at an installation location, and FIG. 22c is a drawing showing a state in which the object is correctly installed at the installation location.

[0047] Throughout the attached drawings, similar reference numbers may be assigned to similar parts, components and / or structures.

[0048] Hereinafter, embodiments of the present invention will be described in detail with reference to the attached drawings. This disclosure is intended to assist in comprehensively understanding the various embodiments of the present invention as defined by the claims and their equivalents. While it includes various specific details for this purpose, these are merely illustrative. Accordingly, those skilled in the art will recognize that various changes and modifications can be made to the various embodiments described in this disclosure without departing from the scope and spirit of the present invention. Furthermore, for clarity and conciseness, descriptions of well-known functions and configurations may be omitted.

[0049] The terms and words used in this disclosure and claims are not limited to their dictionary meanings, but rather have been used by the inventors to clearly and consistently describe the present invention. Therefore, the description of the various embodiments described in this disclosure is merely illustrative, and those skilled in the art should note that it is not intended to limit the scope of the present invention, which is defined by the appended claims and their equivalents.

[0050] Additionally, the singular forms "a," "an," and "the" can include plural meanings unless the context clearly dictates otherwise. For example, the expression "a component surface" can include one or more of those surfaces.

[0051] Each block of a flowchart and combination of flowcharts may be executed by one or more computer programs, and the computer programs may include instructions. The one or more computer programs may be stored in a single memory device or may be distributed and stored across multiple different memory devices.

[0052] The functions or operations described in the present disclosure may be processed by a single processor or a plurality of processors. The single processor or the plurality of processors are circuits that perform operations and may include circuits such as an application processor (AP, e.g., a central processing unit (CPU)), a communication processor (CP, e.g., a modem), a graphics processing unit (GPU), a neural network processing unit (NPU, e.g., an artificial intelligence (AI) chip), a Wi-Fi chip, a Bluetooth® chip, a global positioning system (GPS) chip, a near field communication (NFC) chip, a connectivity chip, a sensor controller, a touch controller, a fingerprint sensor controller, a display driver integrated circuit (IC), an audio codec chip, a universal serial bus (USB) controller, a camera controller, an image processing IC, a microprocessor unit (MPU), a system on a chip (SoC), an IC, and the like.

[0053] <System Configuration>

[0054] FIG. 1 is a drawing showing the appearance of a retaining device having the function of an optical tactile sensor (OTS) according to one embodiment of the present disclosure. FIG. 2 is a cross-sectional view showing the internal structure of the retaining device illustrated in FIG. 1 according to one embodiment of the present disclosure. The retaining device (1) of the present embodiment includes a housing (10), a contact portion (20), and a bottom plate (30). The retaining device (1) maintains contact with a retaining object (e.g., a connector (101a) of FIGS. 9 and 10) (hereinafter referred to as “object”) on the bottom plate (30) side. A suction tube (40), a photographing portion (50), and a lighting portion (60) are accommodated inside the housing (10).

[0055] In the present disclosure, a holding device may refer to a device configured to move an external object by absorbing or gripping it. However, it should be noted that the present disclosure is not limited thereto.

[0056] Although not shown, the holding device (1) is installed at the tip of the arm (robot arm) and is movable. In addition, a suction device (not shown) that sucks air on the side of the surface of the holding device (1) that comes into contact with the object is connected to the holding device (1). The holding device (1) sucks and holds the object by sucking air with the suction device. A detailed description of a method for holding the object by the holding device (1) will be described later. The holding device (1), the arm (robot arm), and the suction device are controlled by a control device (not shown). The control device is realized by a computer, for example. The holding device (1) moves by the operation of the arm based on the control of the control device, holds the object, and installs it at an installation position.

[0057] In addition, the holding device (1) is installed outside the housing (10) and has a processing device (not shown). According to one embodiment, the holding device (1) may constitute a holding system including a processing device (e.g., a processing device (200) of FIG. 13). The processing device (e.g., a processing device (200) of FIG. 13) may detect an object or recognize a state of an object held by the contact unit (20) based on image data captured (or acquired) by the photographing unit (50). Here, an example of recognizing the state of an object may include recognizing a contact state between the contact unit (20) and the holding object (e.g., a connector (101a) of FIGS. 9 and 10). The processing device is implemented by, for example, a computer. In one embodiment, the processing device and the control device may be implemented by the same computer.

[0058] That is, in the present disclosure, the holding device (1) may be configured to recognize the contact state of the contact portion (20) and the holding object (e.g., the connector (101a) of FIGS. 9 and 10). In the present disclosure, the processing device may be referred to as a recognition device. Alternatively, the recognition device of the present disclosure may include at least one means for recognizing the contact state of the holding object and the contact portion (20). For example, the means for recognizing the contact state of the holding object and the contact portion (20) may include attaching a marker to the contact portion (20) and recognizing the amount of deformation of the marker when the contact portion (20) and the holding object (e.g., the connector (101a) of FIGS. 9 and 10) come into contact. This will be described in detail below.

[0059] Hereinafter, recognition in the present disclosure may include estimation. Hereinafter, the maintenance object may refer to an external object or the connector (101a) of FIGS. 9 and 10. However, this is merely an example, and the maintenance object may refer to an object that comes into contact with the contact portion of the maintenance device.

[0060] The housing (10) illustrated in Fig. 1 is rectangular and has one open surface (the lower surface in the drawing). The open surface of the housing (10) is the surface that comes into contact with the object. The housing (10) is made of, for example, resin or metal. A suction hole (11) is formed in a portion of the housing (the upper surface in the drawing). A suction device (not illustrated) is connected to the suction hole (11).

[0061] The bottom plate (30) is a plate-shaped member covering the opening surface of the housing (10). The bottom plate (30) has a three-layer structure consisting of a first layer (31), a second layer (32), and a third layer (33). The first layer (31) to the third layer (33) have a certain degree of hardness and are formed of a member (e.g., a resin such as acrylic) that has transparency to visible light, etc. A through hole (35) is formed in the first layer (31) of the bottom plate (30). A through hole (34) is formed in the third layer (33). A space (air passage) connecting the through hole (35) and the through hole (34) is formed in the second layer (32). A detailed description of the configuration of the bottom plate (30) will be described later.

[0062] The contact portion (20) is composed of a transparent elastic body (21) and a film (22) covering the surface of the transparent elastic body (21). The transparent elastic body (21) is formed by a material having transparency to visible light or the like and elasticity. The transparent elastic body (21) is formed using, for example, a gel such as an elastomer. Examples of the elastomer include optically transparent elastomers such as silicone rubber, polyurethane, plastisol, natural rubber, polyisoprene, and polyvinyl chloride, and other thermoplastic elastomers.

[0063] The film (22) has transparency to visible light, etc., and is formed of a deformable resin, etc. The film (22) is formed by attaching a film to the surface of a transparent elastic body (21) or applying a film-forming material to the surface of the transparent elastic body (21) after attachment. By forming the film (22), the surface of the transparent elastic body (21) can be protected.

[0064] A marker with a geometric pattern is attached to the transparent elastic body (21) or film (22) to visually recognize the deformation caused by the contact portion (20) coming into contact with the target object. The details of the marker will be described later. In addition, a suction hole (23), which is a through hole, is formed in the transparent elastic body (21) and film (22) at a position corresponding to the through hole (35) of the first layer (31) of the bottom plate (30).

[0065] The suction tube (40) is a tubular member whose one end is connected to the suction hole of the housing and whose other end is connected to the through hole of the third layer of the bottom plate. The suction tube (40) is made of, for example, resin or metal. It may be the same member as the housing (10). By the suction tube (40), a path is formed through which air passes from the through hole (34) of the third layer (33) of the bottom plate (30) to the suction hole (11) of the housing (10).

[0066] The photographing unit (50) is a camera. The photographing unit (50) photographs an object by passing through the base plate (30) and the contact portion (20). In addition, the photographing unit (50) photographs a marker attached to the transparent elastic body (21) or film (22) of the contact portion (20). The photographing unit (50) includes a light-receiving element that outputs an electric signal according to light reception, and a plurality of optical systems that form a plurality of images on the light-receiving element. The details of the photographing unit (50) will be described later.

[0067] The lighting unit (60) irradiates light for shooting by the photographing unit (50) from inside the housing (10) toward the contact unit (20). The light from the lighting unit (60) illuminates the bottom plate (30) and the contact unit (20), and at the same time, illuminates the target object by transmitting through the bottom plate (30) and the contact unit (20). The lighting unit (60) includes a light source (61) and a diffusion plate (62). The light source (61) is realized using, for example, an LED (Light Emitting Diode). The diffusion plate (62) is formed of, for example, a resin having light-transmitting properties, and diffuses the transmitted light. By interposing the diffusion plate (62), the light emitted from the light source (61) is diffused, thereby uniformly illuminating the target object and the marker attached to the contact unit (20).

[0068] <Suction structure of the maintenance device (1)>

[0069] FIG. 3 is a drawing showing a suction structure of a maintenance device (1) according to one embodiment of the present disclosure.

[0070] Referring to Fig. 3, a suction hole (23) is formed in the contact portion (20). A through hole (35) is formed in the first layer (31) of the bottom plate (30), a through hole (34) is formed in the third layer (33), and a passage connecting the through hole (35) of the first layer (31) and the through hole (34) of the third layer (33) is formed by the second layer (32). A suction tube (40) is a tube connecting the through hole (34) of the third layer (33) of the bottom plate (30) and the suction hole (11) of the housing (10). The suction hole (11) of the housing (10) is connected to a suction device (not shown). When the suction device is operated, air around the suction hole (23) of the contact portion (20) is sucked from the suction hole (23), and is sucked out through the passage inside the bottom plate (30), through the suction tube (40), and into the suction hole (11) (see arrow in Fig. 3). By the suction structure configured as described above, an object located at the position of the suction hole (23) of the contact portion (20) is sucked into the contact portion (20) and held in the holding device (1).

[0071] <Composition of the floor plate (30)>

[0072] FIG. 4 is a drawing showing the structure of a floor plate (30) according to one embodiment of the present disclosure. The floor plate (30) is configured by overlapping a first layer (31), a second layer (32), and a third layer (33). The first layer (31) is a plate-shaped member and has a through hole (35) formed therein. The second layer (32) is a frame member with a largely open central portion. The third layer (33) is a plate-shaped member and has a through hole (34) formed therein. The second layer (32) is formed to rotate around the outer side of a position corresponding to the through hole (35) of the first layer (31) and the through hole (34) of the third layer (33). Accordingly, by overlapping the first layer (31), the second layer (32), and the third layer (33), a space is formed between the first layer (31) and the third layer (33) from the through hole (35) to the through hole (34). This space becomes a passage through which air is sucked when the holding device (1) holds the object. In addition, although the second layer (32) is a frame member here, the shape of the second layer (32) may be a shape that forms a passage through which air flows from the through hole (35) of the first layer (31) to the through hole (34) of the third layer (33) when overlapping the first layer (31), the second layer (32), and the third layer (33), and the specific shape is not particularly limited. For example, the second layer (32) may be a plate-shaped member having an elongated hole extending from the through hole (35) of the first layer (31) to the through hole (34) of the third layer (33).

[0073] <Optical system of the maintenance device (1)>

[0074] FIG. 5 is a diagram illustrating the configuration of an optical system of a photographing unit (50) of a holding device (1) according to one embodiment of the present disclosure. The optical system of the holding device (1) is realized in the photographing unit (50). The photographing unit (50) has two focal lengths, one for a long distance and one for a short distance, and can simultaneously photograph a subject at a long distance and a subject at a short distance. The photographing unit (50) can be configured to acquire images of the surface of the contact unit (20) and the object to be held.

[0075] Referring to Fig. 5, an example of a long-distance shooting range is indicated by a dashed line, and an example of a short-distance shooting range is indicated by a broken line. The long-distance subject is a subject located at a location that has passed through the base plate (30) and the contact portion (20), and specifically, is an object by the holding device (1). The short-distance subject is the outer surface (the side in contact with the object) of the contact portion (20). The configuration that realizes a short-distance focal length in the shooting portion (50) is an example of the first optical system, and the configuration that realizes a long-distance focal length is an example of the second optical system.

[0076] FIG. 6 is a diagram showing an example of a configuration of a photographing unit (50) according to one embodiment of the present disclosure. The photographing unit (50) includes an image sensor (51), a lens (52), and an aperture plate (53). The image sensor (51) converts light from a subject into an electrical signal. The image sensor (51) is a group of light-receiving elements, and for example, a CIS (Contact Image Sensor) is used. The lens (52) gathers light from a subject onto the image sensor (51) and forms an image. For example, a metal lens is used as the lens (52). The photographing unit (50) includes one lens (52), and in this one lens (52), a long-distance and a close-distance focus are set. The aperture plate (53) has two openings (53a, 53b) and adjusts light incident on the lens (52).

[0077] The lens (52) has a long-distance focus and a close-distance focus set at different positions. For example, in FIG. 6, the long-distance focus is set on the left side of the lens (52), and the close-distance focus is set on the right side. The opening (53a) of the diaphragm plate (53) corresponds to the position where the long-distance focus of the lens (52) is set. The opening (53b) of the diaphragm plate (53) corresponds to the position where the close-distance focus of the lens (52) is set. Therefore, in the example shown in FIG. 6, light from a long-distance subject is incident on the right side of the lens (52) (see the arrow of the dashed line in the drawing), and light from a close-distance subject is incident on the left side of the lens (52) (see the arrow of the dotted line in the drawing). In addition, the image of the long-distance subject and the image of the close-distance subject can be imaged on different areas on the image sensor (51), respectively. The image sensor (51) transmits an electric signal based on the formed image to a processing device (not shown). The processing device generates image data of a distant subject and image data of a nearby subject based on the electric signal received from the image sensor (51). As a result, these objects are detected.

[0078] In the embodiment shown in Fig. 6, the distance from the image sensor (51) to the aperture plate (53) (the subject-side surface of the lens (52)) is 1.5 mm. In addition, the distance from the aperture plate (53) to the surface of the contact portion (20) that contacts the object (the lower surface in the drawing) is 4.5 mm. In addition, the distance from the image sensor (51) to the surface of the contact portion (20) that contacts the object is 6 mm. In addition, the field of view on the surface of the contact portion (20) that contacts the object is 25 mm (12.5 mm × 2). The focal length of the lens (52) is set to, for example, 4.5 mm for a near distance, which is the distance to the film (22) of the contact portion (20) (the surface that contacts the object), and 10 to 20 mm for a far distance.

[0079] FIG. 7 is a diagram schematically showing a configuration for obtaining two images, one at a close range and one at a long range, from a photographing unit (50) according to one embodiment of the present disclosure.

[0080] Referring to Fig. 7, among the configurations of the photographing unit (50), the lens (52) and the aperture plate (53) are illustrated, and the image sensor (51) is omitted. Here, it is explained that the image is formed on the side opposite to the subject of the lens (the lower side in the drawing) of the lens (52). Hereinafter, the side of the lens (52) where the image is formed on the side opposite to the subject is called the 'imaging surface'. In addition, the photographing unit (50) simultaneously photographs both a subject at a close distance and a subject at a long distance, but in Fig. 7, only one subject is described for simplicity.

[0081] Referring to Fig. 7, the light emitted from a subject at a close range reaches the imaging plane of a lens (52) as indicated by a dashed arrow. The light emitted from the subject at a close range passes through one opening (53b) of the diaphragm plate (53) and is focused on a portion of the imaging plane of the lens (52) (the right area in the drawing). Hereinafter, the image of the subject at a close range that is focused on the imaging plane of the lens (52) is referred to as a "close-range subject image."

[0082] In addition, the light emitted from a subject at a distant location reaching the imaging plane of the lens (52) is indicated by an arrow of a dashed line. The light emitted from the subject at a distant location is focused on a part of the imaging plane of the lens (52) (the left area in the drawing) through the other opening (53a) of the diaphragm plate (53). Hereinafter, the image of the subject at a distant location focused on the imaging plane of the lens (52) is referred to as a “distant subject image.” The near-distance subject image and the far-distance subject image are each formed in a circular or oval shape on the imaging plane. In addition, although the near-distance subject image and the far-distance subject image are actually different in shape and size, since the difference is slight, they are described herein as having the same shape and size.

[0083] Here, it is desirable that the near-distance subject image and the far-distance subject image do not overlap on the focusing surface of the lens (52). In order to focus the near-distance subject image and the far-distance subject image without overlapping, it is necessary to set the distance (X) between the aperture (53a) and the aperture (53b) in the diaphragm plate (53) as follows.

[0084] First, if the incident angle of light on the lens (52) is θ0, the refraction angle is θ1, and the refractive index of the lens (52) is N,

[0085] Sinθ0 = N x Sinθ1

[0086] am.

[0087] Next, if the maximum value of Sinθ0 is MaxSinθ0=1, the maximum value of Sinθ1, MaxSinθ1, is

[0088] MaxSinθ1=1 / N

[0089] am.

[0090] Next, assume a point A at a position corresponding to the opening (53a) of the aperture plate (53) on the image plane of the lens (52), and a point B at a position corresponding to the opening (53b). In addition, on the straight line connecting points A and B, the edge position on the side closer to point B in the image of a close-up subject is set as point C. Point C is the point closest to point B in the image of a close-up subject. Then, if the distance between points A and C is Isdmax, and the thickness of the lens (52) (i.e., the distance between the subject-side surface of the lens (52) and the image plane) is d, then

[0091] Isdmax=d / N

[0092] am.

[0093] Therefore, when the distance (X) between the aperture (53a) and the aperture (53b) is greater than the value below, the near-distance subject image and the far-distance subject image do not interfere.

[0094] X=2×Isdmax=2d / N

[0095] FIG. 8 is a schematic diagram illustrating another configuration for obtaining two images of a near distance and a far distance in a photographing unit (50) according to one embodiment of the present disclosure. In the configuration illustrated in FIG. 7, the aperture plate (53) is in contact with the subject-side surface of the lens (52), but in the configuration illustrated in FIG. 8, there is a space between the aperture plate (53) and the subject-side surface of the lens (52). Between the aperture plate (53) and the lens (52), a partition plate (54) is arranged between the aperture (53a) and the aperture (53b) of the aperture plate (53), which divides the space including the aperture (53a) and the space including the aperture (53b).

[0096] Even in the configuration illustrated in Fig. 8, among the configurations of the photographing unit (50), the lens (52) and the aperture plate (53) are illustrated, and the image sensor (51) is omitted. In addition, it is assumed that the image is formed on the imaging surface, which is the opposite side of the lens (52) from the subject (the lower side in the drawing). In addition, the photographing unit (50) simultaneously photographs both a subject at a close distance and a subject at a long distance, but in Fig. 8, only one subject is described for the sake of simplicity.

[0097] Referring to Fig. 8, the light emitted from a subject at a close distance reaching the imaging plane of a lens (52) is indicated by a dashed arrow. The light emitted from the subject at a close distance enters the lens (52) through one opening (53b) of the diaphragm plate (53) and is imaged on a part of the imaging plane (the right area in the drawing). In addition, the light emitted from a subject at a far distance reaching the imaging plane of the lens (52) is indicated by a dashed arrow. The light emitted from a subject at a far distance is imaged on a part of the imaging plane of the lens (52) through the other opening (53a) of the diaphragm plate (53). The near-distance subject image and the far-distance subject image are each formed in a circular or elliptical shape on the imaging plane. Also, although the close-up subject image and the far-distance subject image actually have different shapes and sizes, the difference is small, so here they are described as having the same shape and size.

[0098] In the configuration shown in Fig. 8, in order to focus the image of a close-range subject and the image of a far-range subject without overlapping, it is necessary to set the distance (X) between the aperture (53a) and the aperture (53b) in the aperture plate (53) as follows.

[0099] First, the distance between the aperture plate (53) and the subject is d0, the distance between the aperture plate (53) and the lens (52) is d1, and the thickness of the lens (52) (i.e., the distance between the subject-side surface of the lens (52) and the imaging surface) is d2.

[0100] And, as in the case of the configuration shown in Fig. 7, if the incident angle of light to the lens (52) is θ0, the refraction angle is θ1, and the refractive index of the lens (52) is N,

[0101] Sinθ0=N×Sinθ1

[0102] am.

[0103] And, if the maximum value of Sinθ0 is MaxSinθ0=1, the maximum value of Sinθ1, MaxSinθ1, is

[0104] MaxSinθ1=1 / N

[0105] am.

[0106] Also, as in the case of the configuration illustrated in Fig. 7, point A at a position corresponding to the opening (53a) of the aperture plate (53) on the image plane of the lens (52) and point B at a position corresponding to the opening (53b) are assumed. Then, on the straight line connecting points A and B, the edge position on the side closer to point B in the image of the near-distance subject is set to point C. If the distance between points A and C is Isdmax,

[0107] Isdmax=d2 / N

[0108] am.

[0109] In addition, if the size of the direction along the straight line connecting points A and B of the subject is Y, and the distance (X) between the aperture (53a) and the aperture (53b) is greater than the value below, the near-distance subject image and the far-distance subject image do not interfere.

[0110] X=(Y / 2)×d1 / d0+2×Isdmax

[0111] =(Y / 2)×d1 / d0+2×d2 / N

[0112] <Operation of the maintenance device (1)>

[0113] Next, the operation of the holding device (1) for holding the object will be described. Here, the operation of the holding device (1) for holding the connector portion of the FFC, which is the object, will be described, using as an example the process of holding a connector on which an FFC (flexible flat cable) is installed and mounting it in a socket installed on a substrate.

[0114] FIG. 9 is a drawing showing a state in which a retaining device (1) according to one embodiment of the present disclosure detects a connector as a target and a socket as a mounting location of the connector. FIG. 10 is a drawing showing a state in which a retaining device (1) according to one embodiment of the present disclosure holds a connector. FIG. 11 is a drawing showing a state in which a connector held by a retaining device (1) according to one embodiment of the present disclosure is mounted on a socket. FIG. 12 is a drawing showing a state in which a retaining device (1) according to one embodiment of the present disclosure releases holding of a connector.

[0115] In the initial state, the FFC (101) on which the connector (101a) is installed and the board (102) on which the socket (102a) is installed are arranged at a predetermined initial position. In this state, the holding device (1) first moves the photographing unit (50) to an initial position where the connector (101a) and the socket (102a) can be photographed, according to the arrangement of the FFC (flexible flat cable) (101) and the board (102). In particular, although not illustrated, the holding device (1) is mounted on an arm (robot arm) and is movable according to the operation of the arm. As illustrated in Fig. 9, in the initial position, the holding device (1) photographs the connector (101a) and the socket (102a), which are distant subjects, by the photographing unit (50) (see the photographing range indicated by the dashed-dotted line). Therefore, the control device (not illustrated) detects the connector (101a) and the socket (102a) at a distant location.

[0116] Next, the holding device (1) moves by the movement of the arm portion, and the contact portion (20) of the holding device (1) contacts the connector (101a). At this time, as illustrated in Fig. 10, the holding device (1) photographs the connector (101a) and the socket (102a), which are close-range subjects, by the photographing portion (50) (see the photographing range indicated by the broken line). Then, a control device (not illustrated) controls the movement of the arm portion, and moves the holding device (1) so that the position of the suction hole (23) of the contact portion (20) overlaps with the position of the connector (101a). In this state, when the suction device (not illustrated) is operated by the control of the control device, air is sucked from the suction hole (23) (see the arrow in Fig. 10), and the connector (101a) is held by being sucked by the contact portion (20).

[0117] Next, the holding device (1) moves by the movement of the arm portion, and the connector (101a) held by the holding device (1) is mounted on the socket (102a). At this time, as illustrated in Fig. 11, the holding device (1) photographs the connector (101a) and the socket (102a), which are close-range subjects, by the photographing unit (50) (see the photographing range indicated by the broken line). Then, a control device (not illustrated) controls the movement of the arm portion, and moves the holding device (1) so as to push the connector (101a) into the socket (102a) in a state where the position of the connector (101a) sucked by the suction hole (23) of the contact portion (20) overlaps with the position of the socket (102a).

[0118] Next, the suction device stops operating, releasing the suction of the connector (101a) by the retaining device (1). Then, the retaining device (1) is operated by the control of the arm portion by the control device and is separated from the connector (101a). At this time, as illustrated in Fig. 12, the retaining device (1) photographs the connector (101a) and the socket (102a), which are distant objects, by the photographing unit (50) (see the photographing range indicated by the dashed-dotted line). Then, the control device (not illustrated) confirms that the connector (101a) is mounted on the socket (102a) based on the acquired image.

[0119] In the present disclosure, the captured image may mean an image of a holding object (e.g., a connector (101a) of FIG. 10) held in a contact portion (20) or a surface of the contact portion (20) or a holding object obtained (or captured) by a photographing portion (50).

[0120] <Composition of the processing device>

[0121] In the present disclosure, the processing device may be referred to as a recognition device.

[0122] FIG. 13 is a diagram showing an example of a hardware configuration of a processing device according to one embodiment of the present disclosure. The processing device (200) includes one or more processors (201) as a calculation means, a main memory (202) as a memory means, and an auxiliary memory (203). The processor (201) executes various processes by reading and executing a program stored in the auxiliary memory (203) into the main memory (202). For example, a CPU (Central Processing Unit), an MPU (Micro Processing Unit), a GPU (Graphics Processing Unit), a DSP (Digital Signal Processor), etc. are used as the processor (201). For example, a RAM (Random Access Memory) is used as the main memory (202). For example, a magnetic disk device, an SSD (Solid State Drive), etc. are used as the auxiliary memory (203). In addition, the processing device (200) has an interface (204) for exchanging data with external devices such as the photographing unit (50), arm part (robot arm), suction device, and display device of the maintenance device (1). The configuration illustrated in Fig. 13 is merely an example, and the hardware configuration of the processing device (200) is not limited to the configuration example of Fig. 13.

[0123] FIG. 14 is a diagram showing the functional configuration of a processing device (200) according to one embodiment of the present disclosure. The processing device (200) includes a marker detection unit (210), a marker information storage unit (220), a displacement calculation unit (230), a conversion processing unit (240), a drawing unit (250), and an estimation processing unit (260). The processing device (200) can recognize the state of an object held in a holding device (1), in other words, the contact state between a contact unit (20) and the holding object (e.g., a connector (101a) of FIG. 10), based on an image captured (or acquired) by a photographing unit (50).

[0124] In the present disclosure, the estimation processing unit (260) may be named identically to the recognition processing unit.

[0125] The marker detection unit (210) interprets the image captured (or acquired) by the photographing unit (50) and detects a marker based on a geometric pattern attached to the contact unit (20). The function of the marker detection unit (210) is realized, for example, by the processor (201) illustrated in FIG. 13 executing a program.

[0126] The marker information storage unit (220) stores positional information of a marker in an image of the marker captured (or acquired) by the photographing unit (50). The marker information storage unit (220) stores information on the initial position of the geometric pattern constituting the marker and information on the position at which the marker was last detected from the image. The marker information storage unit (220) is realized by, for example, the main memory device (202) and the auxiliary memory device (203) illustrated in FIG. 13.

[0127] The displacement calculation unit (230) calculates the displacement of the position of the geometric pattern constituting the marker. The displacement calculation unit (230) is realized, for example, by the processor (201) illustrated in FIG. 13 executing a program.

[0128] The displacement calculation unit (230) first calculates the displacement relative to the position of the pattern acquired immediately before, or in other words, the last detected position stored in the marker information storage unit (220). This displacement is hereinafter referred to as the first displacement. The first displacement indicates how the marker moved from the previous state. The first displacement indicates a change in the force applied to the contact portion (20) by an object coming into contact with the contact portion (20).

[0129] Second, the displacement calculation unit (230) calculates the displacement relative to the initial position of the pattern constituting the marker. This displacement is hereinafter referred to as the second displacement. The second displacement indicates how the marker has moved from the initial position. The second displacement indicates how much force the contact portion (20) receives from the object coming into contact with the contact portion (20).

[0130] The conversion processing unit (240) calculates the force acting on the contact portion (20) based on the first displacement and the second displacement calculated by the displacement calculation unit (230). Here, “based on the first displacement and the second displacement” may be the same as based on the deformation. In other words, the conversion processing unit (240) converts the displacement of the marker calculated by the displacement calculation unit (230) into a force acting on the contact portion (20). The force acting on the contact portion (20) may include a pressure acting in a direction perpendicular to the surface of the contact portion (20), a shear force acting in a direction along the surface of the contact portion (20), a torque acting to rotate on the surface of the contact portion (20), etc. The conversion processing unit (240) is realized, for example, by the processor (201) illustrated in FIG. 13 executing a program. When the holding device (1) holds an object and performs an operation, various forces may be applied to the contact portion (20) depending on the posture of the object or the operation performed on the object. By obtaining the force applied to the contact portion (20) by the conversion processing unit (240), the state (position, posture, etc.) of the object held by the holding device (1) or the contact state of the object in contact with the contact portion (20) can be recognized.

[0131] The drawing unit (250) draws the second displacement calculated by the displacement calculation unit (230) onto the image of the marker, thereby generating an image representing the displacement. The generated image is displayed, for example, on a display device connected to the processing device (200), and is used for a user to visually confirm the state of the object being held by the holding device (1). The drawing unit (250) is realized, for example, by the processor (201) illustrated in FIG. 13 executing a program.

[0132] The estimation processing unit (260) recognizes the state of the object held by the holding device (1) based on the force acting on the contact portion (20) calculated by the conversion processing unit (240). The recognized state of the object (e.g., contact state) may include the posture of the object, the amount of deformation by which the object pushes the surface of the contact portion (20), etc. The estimation processing unit (260) is realized by, for example, the processor (201) illustrated in FIG. 13 executing a program.

[0133] <Marker displacement detection>

[0134] Next, a method for detecting displacement of a marker attached to a contact portion (20) will be described. Detection of marker displacement includes, for example, 1. marker detection processing, 2. marker correspondence processing between the initial image and the current image, and 3. displacement amount calculation processing. Existing methods can be used as specific methods for these processes. Specifically, for example, the FingerVision method described in the following document can be used.

[0135] ·project / FingerVision / Software - Akihiko's Tech Note (akihikoy.net)·http: / akihikoy.net / notes / ?project%2FFingerVision%2FSoftware%2FMarkerTrack·GitHub - akihikoy / fingervision: Data processing programs for the vision-based tactile sensor FingerVision

[0136] In this embodiment, since black circles (dots) were used as geometric markers, the black circles were detected during the marker detection process. For specific detection processing, for example, cv::SimpleBlobDetector from OpenCV (an open source image processing library) can be used. Through this detection, the center image coordinates of each circle (dot) are obtained.

[0137] Fig. 15 is a diagram showing an example of marker displacement according to one embodiment of the present disclosure. Fig. 15(A) is a diagram showing a marker in an initial state, Fig. 15(B) is a diagram showing a marker state at time t, and Fig. 15(C) is a diagram showing a state in which the marker positions of Fig. 15(A) and Fig. 15(B) are superimposed.

[0138] Referring to Fig. 15(A), an example in which a pattern in which dots are arranged in a grid shape is used as a marker is shown. The grid lines indicated by dotted lines in Fig. 15(A) indicate the tracking range of each dot. Referring to Fig. 15(B), at time t, the holding device (1) holds the object, and the surface of the contact portion (20) is deformed, so that the dot pattern attached to the surface of the contact portion (20) is distorted. For example, by comparing the positions of each dot shown in Fig. 15(A) with the positions of each dot shown in Fig. 15(B), and obtaining the difference in positions for each corresponding dot as shown in Fig. 15(C), it is possible to recognize how the surface of the contact portion (20) is deformed with respect to the initial state at time t.

[0139] FIG. 16 is a diagram illustrating an example of dot correspondence when a single dot is displaced according to one embodiment of the present disclosure. Here, an example of determining a correspondence between dots using a first displacement is illustrated. As a method for determining the correspondence between individual dots in two images, such as those in FIG. 15(A) and FIG. 15(B), a method that takes into account the overlapping of dots in the two images can be considered, for example.

[0140] Referring to Fig. 16, among the image frames captured (or acquired) by the shooting unit (50), the coordinates of the i-th dot in the j-th image frame are p i,j Similarly, the coordinates of the i-th dot in the previous j-1th image frame are p i,j-1 The coordinates of the dot p are represented as a vector, but here, no special notation is used to indicate that it is a vector, and it is simply p i,j , p i,j-1 It is written as follows. In addition, the area of ​​the region where the ith dot in the jth image frame and the j-1th image frame overlaps is a i,jThe displacement calculation unit (230) of the processing device (200) compares the j-th image frame and the j-1-th image frame, and determines that dots having overlapping areas in the two image frames are the same dots in these two image frames.

[0141] FIG. 17 is a drawing showing the correspondence of a plurality of dots according to one embodiment of the present disclosure.

[0142] Referring to Figure 17, the coordinates p of the i-1th dot i-1 , the coordinates of the i-th dot p i , coordinates p of the i+1th dot i+1 For , the detection result of the j-1th image frame and the detection result of the jth image frame are overlapped and shown. Here, the i+1th dot in the jth image frame is noted. Then, this dot has an overlapping area with two areas of the i-th dot and the i+1th dot in the j-1th image frame. In this case, for example, the dot with a larger overlapping area can be determined to be the same dot. In the example of Fig. 17, the i-1th dot in the j-1th image frame and the i+1th dot in the jth image frame are determined to be the same dot.

[0143] In addition, when dots arranged in a grid shape are significantly displaced due to deformation of the contact portion (20), there are cases where it is not possible to determine identical dots based solely on the size of the overlapping area using the above-described method. To anticipate such a situation, a tracking range is set for each dot. The broken line frame illustrated in Fig. 15 (A) represents the tracking range of each dot. Dots that reach outside the tracking range due to displacement are excluded from the tracking target (target of displacement detection).

[0144] FIG. 18 is a drawing specifying dots excluded from the tracking target in the example of FIG. 15 (C) according to one embodiment of the present disclosure.

[0145] Referring to Fig. 18, dots excluded from the tracking target are indicated by diagonal marks. As a result of each dot being displaced from the initial state shown in Fig. 15 (A) as shown in Fig. 15 (B), among the 4x4 dot columns, the 3rd dot from the left in the 1st upper row, the 2nd dot from the left in the 1st lower row, and the 1st dot to the right in the 1st lower row have reached outside the tracking range. Therefore, as shown in Fig. 15 (C) and Fig. 18, these 3 dots are excluded from the tracking target.

[0146] The displacement calculation unit (230) of the processing device (200) has been described to calculate a first displacement and a second displacement for each dot. The first displacement is the difference between the position of each dot in the previous image frame and the position of each dot in the current image frame. The second displacement is the difference between the position of each dot in the initial state image frame and the position of each dot in the current image frame.

[0147] FIG. 19 is a drawing showing an example of a first displacement according to one embodiment of the present disclosure.

[0148] Referring to Figure 19, the coordinates p of the i-1th dot i-1 , the coordinates of the i-th dot p i , coordinates p of the i+1th dot i+1 , coordinates p of the k-1th dot k-1 , kth dot p k , k+1th dot p k+1, the initial position, the detection result of the j-1th image frame, and the detection result of the jth image frame are superimposed and shown. In Fig. 19, the initial position of each dot is indicated by a dashed line, the detection result of the j-1th image frame is indicated by a thick line, and the detection result of the jth image frame is indicated by a thin line. In Fig. 19, an arrow is shown for each dot connecting the center of the dot which is the detection result of the j-1th image frame and the center of the dot which is the detection result of the j-1th image frame. This arrow is an arrow indicating the correspondence relationship of each dot obtained by the first displacement of each dot.

[0149] FIG. 20 is a drawing showing an example of a second displacement according to one embodiment of the present disclosure.

[0150] Referring to Figure 20, the coordinates p of the i-1th dot i-1 , the coordinates of the i-th dot p i , coordinates p of the i+1th dot i+1 , coordinates p of the k-1th dot k-1 , the coordinates of the kth dot p k , coordinates p of the k+1th dot k+1 For , the initial position, the detection result of the j-1th image frame, and the detection result of the jth image frame are superimposed. The initial position is the position in the 0th image frame, with coordinates p i,0 dot of coordinate p k,0 It is described as a dot of. In Fig. 20, the initial position of each dot is indicated by a thick line, the detection result of the j-1th image frame is indicated by a dashed line, and the detection result of the jth image frame is indicated by a thin line. In addition, in Fig. 20, an arrow is shown connecting the center of the dot which is the detection result of the jth image frame and the center of the dot at the initial position in the display of each dot. This arrow is an arrow indicating the correspondence relationship of each dot obtained by the second displacement of each dot.

[0151] The drawing unit (250) of the processing device (200) can generate an image representing the second displacement, as shown in Fig. 20, based on the second displacement calculated by the displacement calculation unit (230), and display the image on the display device. In this way, the user can visually recognize the second displacement of each dot while viewing the image displayed on the display device.

[0152] The conversion processing unit (240) of the processing device (200) has been described to calculate the force acting on the contact portion (20) based on the second displacement of the dots constituting the marker. The displacement amount of the dots corresponds to the deformation amount of the contact portion (20).

[0153] FIG. 21 is a predicted diagram assuming a relationship between the force acting on the contact portion (20) of the holding device (1) and the amount of deformation of the contact portion (20) when it is assumed that the transparent elastic body used in the contact portion (20) according to one embodiment of the present disclosure is deformed with very ideal hysteresis.

[0154] Referring to Fig. 21, the vertical axis represents the force applied to the contact portion (20). The horizontal axis represents the displacement of the transparent elastic body. In addition, in the same drawing, the curve indicated by the arrow pointing to the upper right represents the trend when a load is applied to the contact portion (20), and the curve indicated by the arrow pointing to the lower left represents the trend when the load applied to the contact portion (20) is limited. The straight line drawn with a dotted line in the drawing is a straight line connecting the lowest point and the highest point of the curves within the area surrounded by the two curves. If the slope of this straight line is k, this straight line is thought to be the trend of a spring having a diagonal spring constant k.

[0155] It is assumed that the transparent elastic material used in the contact portion (20) is deformed with a very ideal hysteresis as shown in Fig. 21. At this time, the contact portion (20) can be displaced in the x, y, and z-axis directions, respectively. Fig. 21 shows an example of deformation in the x-axis direction. At this time, the diagonal spring constant is kx If we do, k x is obtained by the following equation (Equation 1).

[0156]

[0157] These spring constants exist for each of the x, y, and z-axis directions. Therefore, if all diagonal spring constants are organized and called the spring constant matrix K, K can be expressed by the following equation (Equation 2).

[0158]

[0159] This spring constant K represents the ratio of the displacement and load of the transparent elastic body used in the contact portion (20), and is therefore also captured as the resolution of the optical tactile sensor in the holding device (1).

[0160] Any marker (m) of the contact part (20) i ) the force (f) generated i ) is a marker (m i ) and the three-dimensional displacement amount is dx i , dy i , dz i As a result, the spring constant K mentioned above is expressed by the following equation (Mathematical Equation 3). In addition, the force (f i ) is represented as a vector, but except in mathematical expressions, no special notation is used to indicate that it is a vector, and it is simply f i is written as follows. The force f below avg The same applies to .

[0161]

[0162] And, the average force (f) acting on the entire contact area (20) avg ) is expressed by the following equation (Mathematical Equation 4), where the number of dots of the marker attached to the contact portion (20) is N.

[0163]

[0164] In addition, any marker (m) located around the center of the image captured (or acquired) by the camera unit (50) i ) that generates the torque (τ) i ) is the marker (m) from the center position of the image i ) is the distance to r i If , it is expressed by the following equation (mathematical equation 5). In addition, the torque τ i and distance r i is represented as a vector, but except in mathematical expressions, no special notation is used to indicate that it is a vector, and it is simply written as τ i , r i is written as . The following force τ avg The same applies to .

[0165]

[0166] (step, (assumed)

[0167] At this time, the average torque (τ) acting on the contact part (20) avg ) has the number of dots as N, and the torque τ i It is expressed by the following equation (Equation 6) using .

[0168]

[0169] <Object status detection>

[0170] When the holding device (1) holds an object and performs a task, the contact portion (20) is deformed according to the force applied to the contact portion (20). The deformation of the contact portion (20) is calculated based on an image of a marker of the contact portion (20) photographed (or acquired) by the photographing unit (50). And, from the calculated deformation of the contact portion (20), the contact state of the object held by the holding device is recognized.

[0171] FIG. 22a, FIG. 22b, and FIG. 22c are drawings showing an example of operation of a holding device (1) according to one embodiment of the present disclosure. FIG. 22a is a drawing showing a state in which the holding device (1) holds an object, FIG. 22b is a drawing showing a state in which the holding device (1) is about to install the object at an installation location, and FIG. 22c is a drawing showing a state in which the object is correctly installed at the installation location.

[0172] In Fig. 22a, the holding device (1) holds an object. The object is a connector (101a) attached to an FFC (101). Although not shown, the holding device (1) is connected to an arm portion (robot arm) and moves toward the installation position of the connector (101a) by the operation of the arm portion. The installation position of the connector (101a) is a socket (102a) installed on a substrate (102).

[0173] Here, it is possible to consider a case where the connector (101a) is not mounted on the socket (102a) because the position of the connector (101a) is incorrect. In the example shown in Fig. 22b, the connector (101a) is in contact with the edge of the socket (102a) and is not mounted correctly. In this state, when an operation is performed to push the connector (101a) into the socket (102a), the connector (101a) receives a reaction force from the edge of the socket (102a), and a torque is applied to the contact portion (20) that is adsorbing the connector (101a). The contact portion (20) is deformed by the torque received from the connector (101a). The deformation of the contact portion (20) is detected as a displacement of a dot of a marker attached to the contact portion (20) in an image of the contact portion (20) captured (or acquired) by the photographing unit (50).

[0174] The processing device (200) calculates the force acting on the contact portion (20) based on the displacement of the dot of the detected marker, and recognizes that the connector (101a) is not properly mounted on the socket (102a) and is receiving a force that causes a part of the connector (101a) to return. The processing device (200) stops the operation of the arm portion in the control device. Then, the processing device (200) specifies the position of the socket (102a) based on the image captured (or acquired) by the photographing unit (50), and instructs the control device to operate the arm portion to mount the connector (101a) at the newly specified position of the socket (102a). In this way, the position of the connector (101a) is adjusted, and the connector (101a) is properly mounted on the socket (102a), as illustrated in FIG. 22C.

[0175] <Variations>

[0176] While the embodiments of the present invention have been described above, the technical scope of the present invention is not limited to the above embodiments. For example, in the above embodiment, the marker attached to the contact portion (20) is a pattern of dots arranged in a grid shape. In contrast, various geometric patterns suitable for detecting deformation occurring in the contact portion (20) based on an image captured by the imaging unit (50) can be used as markers.

[0177] In addition, in the above embodiment, the floor plate (30) has a three-layer structure, and the frame-shaped second layer (32) is sandwiched between the plate-shaped first layer (31) and the third layer (33), thereby forming an air flow path in the suction mechanism. In this regard, the floor plate (30) is not limited to the three-layer structure as long as it supports the contact portion (20) by an elastic body and realizes an air flow path in the suction mechanism. For example, the two-layer structure may be formed by two plate-shaped members, and a groove or a concave portion may be formed in one or both members to realize an air flow path. In this case, a through hole corresponding to the through hole (35) of the first layer (31) and the through hole (34) of the third layer (33) shown in Fig. 4 is formed in the plate-shaped member of the two layers, and a groove or a concave portion is formed so as to connect these two through holes when the plate-shaped members of the two layers are overlapped.

[0178] In addition, the bottom plate (30) and the contact portion (20) can be configured to be detachable and interchangeable with respect to the holding device (1). In this way, a plurality of bottom plates (30) and contact portions (20) having different positions of the suction holes (23) are prepared in advance, and by exchanging the bottom plates (30) and contact portions (20) according to the type of work, the position at which the object is held in the contact portion (20) can be changed.

[0179] In addition, in the above embodiment, the photographing unit (50) is configured to include a plurality of optical systems having a near-distance focal length and a far-distance focal length set. In contrast, the photographing unit (50) may be configured to include a plurality of optical systems having the same focal length set. With such a configuration, it is possible to obtain a plurality of images with parallax for a single photographing target (e.g., a holding target), and based on the difference in the way the photographing target appears due to parallax, the shape of the photographing target can be three-dimensionally grasped. In addition, various modifications or alternative configurations that do not depart from the scope of the technical idea of ​​the present invention are included in the present invention.

[0180] It will be appreciated that the various embodiments of the present disclosure may be implemented in the form of hardware, software, or a combination of hardware and software.

[0181] Software according to various embodiments of the present disclosure may be stored in a non-transitory computer-readable storage medium. The non-transitory computer-readable storage medium stores one or more computer programs (software modules), and the one or more computer programs may include computer-executable instructions that, when executed singly or collectively by one or more processors of an electronic device, cause the electronic device to perform the methods of the present invention.

[0182] The software according to various embodiments of the present disclosure may be stored in volatile or non-volatile storage forms. For example, it may be stored in the form of a storage device, such as read-only memory (ROM), whether erasable or rewritable, or in the form of random access memory (RAM), memory chips, devices, or integrated circuits. Furthermore, the software may be stored in the form of optically or magnetically readable media, such as compact discs (CDs), digital versatile discs (DVDs), magnetic disks, or magnetic tapes. The storage devices and storage media will be understood as examples of non-transitory machine-readable storage media suitable for storing computer programs containing instructions for implementing various embodiments of the present disclosure. Accordingly, various embodiments may provide a program including code for implementing a device or method as recited in any of the claims of the present disclosure, and a non-transitory machine-readable storage medium storing such a program.

[0183] Above, the detailed description of this document has described specific embodiments, but it will be obvious to those skilled in the art that various modifications are possible within the scope of this document.

Claims

1. In the maintenance device (1), a housing (10) including an opening; A contact portion (20) that covers the opening of the housing, is formed by an elastic body having light transmission properties, and is configured to contact a maintenance object (101a) and maintain contact with the maintenance object; A photographing unit (50) installed in the housing and configured to acquire images of the surface of the contact portion and the object to be maintained, and A holding device including a recognition device configured to recognize a contact state between the contact portion and the holding object based on an image of the surface of the contact portion and the holding object obtained by the photographing unit.

2. In the first paragraph, the contact part includes a suction hole for sucking air, and a holding device that sucks the holding object by sucking air from the suction hole by aligning the position of the suction hole with the position of the holding object.

3. In the first or second paragraph, a marker having a predetermined geometric pattern is arranged on the surface of the contact portion that comes into contact with the object to be maintained. A holding device configured to recognize a contact state of a holding object held in the contact portion based on the deformation of the marker in an image captured by the photographing portion.

4. A holding device comprising a film formed by a light-transmitting resin on the surface of the contact portion that comes into contact with the object to be held in accordance with claim 1 or 2.

5. In the fourth paragraph, a marker according to a predetermined pattern is arranged on the film formed on the contact portion, A holding device configured to recognize a contact state of the holding object held on the contact portion based on the deformation amount of the marker in the image captured by the photographing portion.

6. A maintenance device according to claim 1, further comprising a lighting unit that irradiates light for photographing by the photographing unit from inside the housing to the contact unit.

7. In paragraph 1, the photographing unit A group of light-receiving elements that output electrical signals according to light reception, A holding device comprising a plurality of optical systems for forming a plurality of images on the above light-receiving element group.

8. In paragraph 7, the plurality of optical systems of the photographing unit one lens, and A holding device including an aperture plate having a plurality of apertures for allowing a plurality of incident lights to enter the lens so that the light passing through the lens forms a plurality of images on the light receiving element group.

9. In paragraph 8, the plurality of optical systems of the photographing unit A first optical system that focuses on the light-receiving element group by adjusting the focal length to the surface that contacts the object to be maintained at the contact portion, and A holding device including a second optical system that focuses an image on the light-receiving element group at a focal length further than the surface of the contact part that contacts the object to be held.

10. In paragraph 8, A holding device comprising a lens, wherein the lens comprises a first part having a long-distance focal length set and a second part having a short-distance focal length set.

11. In paragraph 10, The above photodetector group includes an image sensor, A holding device in which an image is formed from the first part of the lens to the first part of the image sensor, and an image is formed from the second part of the lens to the second part of the image sensor.

12. In paragraph 8, The above lens is a maintenance device including a metalens.

Images