Fixed condition determination system, glove, information processing device, fixed condition determination method, and fixed condition determination program

Abstract

A fixing state determination system (500) is provided with a glove (200) that is worn by a worker (10) and holds an object (300). An imaging device (201) is disposed at a position corresponding to a position between a pair of fingers holding an object (300) in a glove (200), and captures an object image (25). A position and orientation estimation unit (110) estimates the position and orientation of an object (300). A range calculation unit (120) calculates a fingertip movement range, which is the movement range of a fingertip of a worker (10). The contact position calculation unit (130) calculates the contact position of the fingertip on the object (300) on the basis of the position and orientation of the object (300) and the fingertip movement range. The determination unit (140) determines the fixing state of the object (300) on the basis of excitation force information (26) of the excitation force for the object (300) at the contact position of the fingertip and the displacement of the object (300).

Description

Technical Field

[0001] The present disclosure relates to a fixing condition determination system, a glove, an information processing device, a fixing condition determination method, and a fixing condition determination program. Background Art

[0002] In order to remotely check the fixing status of the equipment, there are gloves for on-site workers to wear.

[0003] In conventional gloves, the displacement of the force applied to the contact surface and the vibration of the fingertips can be measured. However, it is not possible to obtain information such as the position and posture of the object relative to the glove or the position of the fingertips in the object. Therefore, it is not possible to obtain information such as the direction of the force applied to the object and the accurate displacement of the object.

[0004] Patent Document 1 discloses a robot finger that has a CCD camera disposed inside a fingertip portion covered with an elastic body to detect contact with an object and measures deformation of the elastic body.

[0005] Prior art literature

[0006] Patent Literature

[0007] Patent Document 1: Japanese Patent Application Publication No. 2010-221359 Summary of the invention

[0008] Problem that the invention aims to solve

[0009] In order to inspect the fixing condition of equipment such as looseness of equipment, workers on site wear gloves and perform work such as holding an object with the gloves and vibrating it.

[0010] In this case, by accurately grasping the position of the fingertip in the object, the fixing status can be determined with higher accuracy.

[0011] Even if the technology of Patent Document 1 is applied to the glove, it is impossible to accurately obtain the contact position of the fingertip in the object, and it is impossible to grasp information such as the direction of the force applied to the object and the accurate displacement of the object. Therefore, there is a problem that the fixing state of the object cannot be determined with high accuracy.

[0012] The present disclosure aims to accurately obtain the contact position of a fingertip in an object, thereby enabling a more accurate determination of the fixing state of the object.

[0013] Means used to solve problems

[0014] The fixing condition determination system disclosed in the present invention comprises a glove worn by an operator to hold an object, and the fixing condition determination system determines the fixing condition of the object, wherein the fixing condition determination system comprises: a photographing device, which is arranged at a position in the glove corresponding to between a pair of fingers holding the object, and photographs the object held by the pair of fingers as an object image; a position and posture estimation unit, which uses the object image to estimate the position and posture of the object; a range calculation unit, which uses the object image to analyze the movement of the operator's fingertips and calculates the movement range of the position of the fingertips of each finger of the operator, namely, the fingertip movement range; a contact position calculation unit, which calculates the contact position of the fingertip in the object based on the position and posture of the object and the fingertip movement range; and a determination unit, which determines the fixing condition of the object based on information on the excitation force on the object at the contact position of the fingertip, namely, the excitation force information and the displacement of the object.

[0015] Effects of the Invention

[0016] The fixing state determination system of the present disclosure aims to determine the fixing state of an object with higher accuracy by accurately acquiring the contact position of a fingertip in the object. BRIEF DESCRIPTION OF THE DRAWINGS

[0017] Figure 1 This is a diagram showing the overall configuration of the fixing state determination system according to the first embodiment.

[0018] Figure 2 This is a diagram showing a configuration example of an information processing device according to Embodiment 1.

[0019] Figure 3 This is a diagram showing a configuration example of a glove according to the first embodiment.

[0020] Figure 4 This is a diagram showing a state where the glove according to the first embodiment is gripping an object.

[0021] Figure 5 This is a flowchart showing an operation example of the fixed state determination system according to the first embodiment.

[0022] Figure 6 This is a diagram showing an example of a target image according to the first embodiment.

[0023] Figure 7 1 is a diagram showing an image of the angle of the proximal phalanx of each finger in the range calculation process according to the first embodiment.

[0024] Figure 8 This is a diagram showing an image of a fingertip movement range in the range calculation process according to the first embodiment.

[0025] Fig. 9 1 is a diagram showing an image of a contact position in the contact position calculation process according to the first embodiment.

[0026] Fig.10 This is a diagram showing a configuration example of a glove according to Modification 1 of Embodiment 1.

[0027] Fig.11 This is a diagram showing a configuration example of a glove according to a second modification of the first embodiment.

[0028] Fig.12 This is a diagram showing a configuration example of a glove according to a third modification of the first embodiment.

[0029] Fig.13 This is a diagram showing a configuration example of a glove according to a fourth modification of the first embodiment.

[0030] Fig.14 This is a diagram showing a configuration example of an information processing device according to a fifth modification of the first embodiment. DETAILED DESCRIPTION

[0031] Hereinafter, the present embodiment is described using figures. In each figure, the same reference numerals are used for the same or corresponding parts. In the description of the embodiment, the description is appropriately omitted or simplified for the same or corresponding parts. The arrows in the figure mainly show the flow of data or the process. In addition, in the following figures, the size relationship of each component is sometimes different from the actual situation. In addition, in the description of the embodiment, directions or positions such as top, bottom, left, right, front, back, front, and back are sometimes shown. These expressions are for the convenience of description and do not limit the configuration, direction, and orientation of the device, appliance, or component.

[0032] Implementation method 1.

[0033] ***Structure description***

[0034] Figure 1 1 is a diagram showing an overall configuration example of a fixing state determination system 500 according to the present embodiment.

[0035] The fixing condition determination system 500 is a system for determining the fixing condition of the object 300 by causing the worker 10 to wear gloves and grasp the object 300 to vibrate the object 300 .

[0036] The fixing state determination system 500 includes a glove 200 worn by the worker 10 to grasp the object 300, and an information processing device 100 that communicates with the glove 200. The communication between the glove 200 and the information processing device 100 may be either priority or wireless.

[0037] The detailed structures of the information processing device 100 and the glove 200 are described below.

[0038] Figure 2 1 is a diagram showing a configuration example of the information processing device 100 according to the present embodiment.

[0039] The information processing device 100 is a computer. The information processing device 100 includes a processor 910 and other hardware such as a memory 921, an auxiliary storage device 922, an input interface 930, an output interface 940, and a communication device 950. The processor 910 is connected to other hardware through wired communication or wireless communication to control the other hardware.

[0040] The information processing device 100 includes a position and posture estimation unit 110, a range calculation unit 120, a contact position calculation unit 130, a determination unit 140, and a storage unit 160 as functional elements. The storage unit 160 stores an object database 161.

[0041] The functions of the position and posture estimation unit 110, the range calculation unit 120, the contact position calculation unit 130, and the determination unit 140 are implemented by software. The storage unit 160 is provided in the memory 921. In addition, the storage unit 160 may be provided in the auxiliary storage device 922, or may be provided in the memory 921 and the auxiliary storage device 922 in a distributed manner.

[0042] Processor 910 is a device that executes a fixing condition determination program. The fixing condition determination program is a program that realizes the functions of position and posture estimation unit 110, range calculation unit 120, contact position calculation unit 130, and determination unit 140. In addition, the fixing condition determination program also includes a program that realizes the function of the gloves described later.

[0043] The processor 910 is an IC that performs calculation processing. Specific examples of the processor 910 are CPU, DSP, and GPU. IC is the abbreviation of Integrated Circuit. CPU is the abbreviation of Central Processing Unit. DSP is the abbreviation of Digital Signal Processor. GPU is the abbreviation of Graphics Processing Unit.

[0044] The memory 921 is a storage device for temporarily storing data. A specific example of the memory 921 is SRAM or DRAM. SRAM is the abbreviation of Static Random Access Memory. DRAM is the abbreviation of Dynamic Random Access Memory.

[0045] The auxiliary storage device 922 is a storage device for storing data. A specific example of the auxiliary storage device 922 is an HDD. In addition, the auxiliary storage device 922 may also be a removable storage medium such as an SD (registered trademark) memory card, CF, NAND flash memory, a floppy disk, an optical disk, a high-density disk, a Blu-ray (registered trademark) optical disk, or a DVD. In addition, HDD is the abbreviation of Hard Disk Drive. SD (registered trademark) is the abbreviation of Secure Digital. CF is the abbreviation of CompactFlash (registered trademark; compact flash memory). DVD is the abbreviation of Digital Versatile Disk.

[0046] The input interface 930 is a port connected to an input device such as a mouse, a keyboard, or a touch panel. Specifically, the input interface 930 is a USB terminal. In addition, the input interface 930 may also be a port connected to a LAN. USB is the abbreviation of Universal Serial Bus. LAN is the abbreviation of Local Area Network.

[0047] The output interface 940 is a port for connecting a cable to an output device such as a display. Specifically, the output interface 940 is a USB terminal or an HDMI (registered trademark) terminal. Specifically, the display is an LCD. The output interface 940 is also called a display interface. HDMI (registered trademark) is an abbreviation for High Definition Multimedia Interface. LCD is an abbreviation for Liquid Crystal Display.

[0048] The communication device 950 has a receiver and a transmitter. The communication device 950 is connected to a communication network such as a LAN, the Internet, WiFi (registered trademark), or a telephone line. Specifically, the communication device 950 is a communication chip or NIC. NIC is an abbreviation of Network Interface Card.

[0049] The fixed status determination program is executed in the information processing device 100. The fixed status determination program is read into the processor 910 and executed by the processor 910. In the memory 921, not only the fixed status determination program but also the OS is stored. OS is the abbreviation of Operating System. The processor 910 executes the fixed status determination program while executing the OS. The fixed status determination program and the OS may also be stored in the auxiliary storage device 922. The fixed status determination program and the OS stored in the auxiliary storage device 922 are loaded into the memory 921 and executed by the processor 910. In addition, part or all of the fixed status determination program may also be incorporated into the OS.

[0050] The information processing device 100 may include a plurality of processors instead of the processor 910. The plurality of processors share the execution of the fixed state determination program. Each processor is a device that executes the fixed state determination program in the same manner as the processor 910.

[0051] Data, information, signal values, and variable values ​​used, processed, or output by the fixed status determination program are stored in the memory 921 , the auxiliary storage device 922 , or a register or cache memory in the processor 910 .

[0052] The "unit" of each unit of the position and posture estimation unit 110, the range calculation unit 120, the contact position calculation unit 130, and the determination unit 140 may be rewritten as "circuit", "process", "procedure", "processing", or "line". The fixed condition determination program enables the computer to perform position and posture estimation processing, range calculation processing, contact position calculation processing, determination processing, and image setting processing. The "processing" of the position and posture estimation processing, range calculation processing, contact position calculation processing, and determination processing may be rewritten as "program", "program product", "computer-readable storage medium storing the program", or "computer-readable recording medium recording the program". In addition, the fixed condition determination method is a method performed by the information processing device 100 executing the fixed condition determination program.

[0053] The fixed state determination program may be stored in a computer-readable recording medium and provided. In addition, the fixed state determination program may be provided as a program product.

[0054] Figure 3 2 is a diagram showing a configuration example of a glove 200 according to the present embodiment.

[0055] Figure 4 This is a diagram showing a state where the glove 200 according to the present embodiment is gripping an object 300 .

[0056] When the worker 10 grasps the object 300 and vibrates it, the glove 200 is worn on the hand of the worker 10 grasping the object 300 .

[0057] exist Figure 3 , a glove 200 is shown worn on the right hand of the worker 10. Figure 3 The left figure is a figure of the glove 200 viewed from the palm side. Figure 3 The right figure of FIG. 1 is a figure which looks at the glove 200 from the back of the hand.

[0058] In addition, Figure 4 , a situation in which the worker 10 wears the glove 200 and grasps the object 300 is shown.

[0059] The glove 200 includes an imaging device 201 , a force sensor 202 , a sensor data collection module 203 , and a marker 204 .

[0060] The imaging device 201 is disposed in the glove 200 at a position corresponding to between a pair of fingers gripping the object 300 , and captures an image of the object 300 gripped by the pair of fingers as the object image 25 .

[0061] The imaging device 201 is, for example, a device in which a module having an optical imaging element is arranged on a substrate. The imaging device 201 is, for example, a CCD camera.

[0062] Usually, people use the thumb and fingers to hold objects. Figure 3 In the embodiment, the operator 10 holds the object with fingers including the thumb and the index finger. Therefore, the photographing device 201 is arranged at a position between the thumb and the index finger on the palm side of the glove 200. In addition, any other position may be used as long as it is a position for photographing the held object 300. It may be arranged at the base of any finger. Alternatively, it may be arranged near the center of the palm.

[0063] The mark 204 is arranged at a position corresponding to the inner side of the proximal phalanx of each finger of the worker 10 in the glove 200. Specifically, the mark 204 is printed at a position corresponding to the inner side of the proximal phalanx of each finger of the worker 10 in the glove 200.

[0064] The imaging device 201 captures an object image 25 including the position of the mark when the glove 200 grasps the object. That is, the object image 25 also includes the mark 204 printed on each finger of the pair of fingers.

[0065] The force sensor 202 is disposed in the glove 200 at a position corresponding to the inner side of the fingertip of the worker 10. The force sensor 202 acquires information on the excitation force applied to the object 300 as the excitation force information 26. The excitation force information 26 is, for example, the magnitude of the excitation force.

[0066] The sensor data collection module 203 collects various sensor data and sends them to the information processing device 100. Specifically, the sensor data collection module 203 collects the object image 25 from the imaging device 201 and collects the excitation force information 26 from the touch force sensor 202. Then, the sensor data collection module 203 sends the object image 25 and the excitation force information 26 to the information processing device 100 through wired communication or wireless communication.

[0067] The sensor data collection module 203 is a computer that collects sensor data such as the object image 25 and the excitation force information 26 and transmits the data to the information processing device 100 .

[0068] For example, the information processing device 100 includes hardware such as an input interface for inputting sensor data, a processor, and a communication device for communicating with the information processing device 100. Each hardware is the same as described in the information processing device 100.

[0069] ***Action description***

[0070] Next, the operation of the fixed state determination system 500 of this embodiment will be described. The operation process of the fixed state determination system 500 corresponds to the fixed state determination method. In addition, the program that realizes the operation of the fixed state determination system 500 corresponds to the fixed state determination program.

[0071] Figure 5 1 is a flowchart showing an operation example of the fixed state determination system 500 according to the present embodiment.

[0072] <Photography Process>

[0073] In step S101 , the imaging device 201 captures an image of the object 300 held by a pair of fingers as the object image 25 .

[0074] Specifically, the imaging device 201 captures the object 300 grasped by a pair of fingers and the marks 204 printed on the respective fingers of the pair of fingers.

[0075] The sensor data collection module 203 sends the object image 25 to the information processing device 100 .

[0076] Figure 6 1 is a diagram showing an example of a target image 25 captured by the imaging device 201 according to the present embodiment.

[0077] like Figure 6 As shown, depending on the shape of the object 300, the contact portion between the object 300 and the fingertip may not be captured.

[0078] Although not shown in the figure, the target image 25 also includes a mark 204 printed on each of the pair of fingers.

[0079] <Position and attitude estimation processing>

[0080] In step S102 , the position and posture estimation unit 110 estimates the position and posture of the object 300 using the object image 25 . The position and posture estimation unit 110 estimates the position and posture of the object 300 by comparing the object image 25 with the image candidates 61 stored in the object database 161 .

[0081] Specifically, it is as follows.

[0082] In the object database 161 , images of a plurality of combinations of positions and postures of the object 300 are stored as image candidates 61 .

[0083] The image candidates 61 are a plurality of three-dimensional stereo data obtained by capturing the object 300 at various angles and positions.

[0084] The position and posture estimation unit 110 compares the target image 25 with the image candidates 61 and selects the image candidate 61 having the highest similarity, thereby estimating the position and posture of the object 300 in the target image 25 .

[0085] <Range calculation processing>

[0086] In step S103 , the range calculation unit 120 analyzes the movement of the fingertips of the operator 10 using the object image 25 , and calculates the fingertip movement range R, which is the movement range of the positions of the fingertips of the operator's fingers. Step S103 includes steps S31 and S32 .

[0087] Specifically, it is as follows.

[0088] Figure 7 1 is a diagram showing an image of the angle of the proximal phalanx of each finger in the range calculation process according to the present embodiment.

[0089] In step S31, the range calculation unit 120 calculates the angle of the proximal phalanx of each finger of the operator 10 using the position of the marker 204 on the proximal phalanx in the object image 25. Figure 7 In the example of FIG. 1 , a case where an angle θ1 of the proximal phalanx of the thumb and an angle θ2 of the proximal phalanx of the index finger are calculated is shown.

[0090] Figure 8 1 is a diagram showing an image of a fingertip movement range R in the range calculation process according to the present embodiment.

[0091] In step S32, the range calculation unit 120 calculates the fingertip movement range R based on the angle of the proximal phalanx of each finger of the operator 10. The range calculation unit 120 calculates the fingertip movement range R of each finger based on the angle of the proximal phalanx of each finger using information on the general finger joint structure.

[0092] <Contact position calculation processing>

[0093] In step S104 , the contact position calculation unit 130 calculates the contact position P of the fingertip on the object 300 based on the position and posture of the object 300 and the fingertip movement range R.

[0094] Fig. 9 1 is a diagram showing an image of a contact position P in the contact position calculation process according to the present embodiment.

[0095] The contact position calculation unit 130 calculates the surface position of the object 300 based on the position and posture of the object 300. Then, the contact position calculation unit 130 calculates the contact position P between the object 300 and the fingertip based on the intersection of the surface position of the object 300 and the fingertip movement range R.

[0096] <Judgment Process>

[0097] In step S105 , the determination unit 140 determines the fixing condition of the object 300 based on the excitation force information 26 , which is information of the excitation force on the object 300 at the contact position P of the fingertip, and the displacement of the object 300 . Step S105 includes steps S51 and S52 .

[0098] Specifically, it is as follows.

[0099] In step S51, the operator 10 wears the glove 200, grasps the object 300, and excites the object 300. The sensor data collection module 203 sends the magnitude of the excitation force obtained by the touch force sensor to the information processing device 100 as the excitation force information 26. The excitation force information 26 is obtained by the touch force sensor 202 and is sent from the glove 200 to the information processing device 100 via the sensor data collection module 203.

[0100] In step S51 , the determination unit 140 determines the fixing status of the object 300 based on the contact position P, the excitation force information 26 , and the displacement of the object 300 .

[0101] The displacement of the object 300 is acquired from the object image 25 by the determination unit 140 .

[0102] The determination unit 140 calculates the magnitude of the excitation force and the direction of the excitation force at the contact position P, obtains the displacement as the response of the object 300 , and integrates these calculation results to thereby determine the fixing condition or state of the object 300 .

[0103] As described above, the fixing condition can be evaluated as an index based on the relationship between the displacement of the object 300 and the exciting force.

[0104] **Other structures**

[0105] <Variation 1>

[0106] Fig.10 This is a diagram showing a configuration example of a glove 200 according to a first modification of the present embodiment.

[0107] The glove 200 may include a motion sensor for detecting the motion of the fingertips on the outer sides of the fingertips of the worker 10 .

[0108] The range calculation unit 120 calculates the fingertip movement range R based on the angle of the proximal phalanx of each finger of the worker 10 and the movement of the fingertip detected by the motion sensor 205 .

[0109] The motion sensor 205 is, for example, an acceleration sensor capable of measuring acceleration. The range calculation unit 120 estimates the fingertip position based on the displacement from the initial position. The number of motion sensors used may be changed according to changes in the assumed gripping method.

[0110] According to the glove 200 of the first modification, the fingertip position estimation accuracy is improved, and the excitation position can be estimated accurately and stably.

[0111] <Variation 2>

[0112] Fig.11 This is a diagram showing a configuration example of a glove 200 according to a second modification of the present embodiment.

[0113] The glove 200 may also include a fixing device 206, which is provided on each of a pair of fingers in the glove 200 to fix the imaging device 201. The fixing device 206 includes, for example, a ring 261 provided on each of a pair of fingers in the glove 200, and a rigid body 262 connecting the ring 261 and the imaging device 201.

[0114] According to the glove 200 of the second modification, the ring 261 fixed to the finger is connected to the imaging device 201 via the rigid body 262. Therefore, the estimation accuracy of the position of the finger and the imaging device 201 is improved, and the estimation accuracy of the fingertip position is also improved.

[0115] <Variation 3>

[0116] Fig.12 This is a diagram showing a configuration example of a glove 200 according to a third modification of the present embodiment.

[0117] The mark 204 may also have a protruding shape.

[0118] As a mark corresponding to the proximal phalanx, a three-dimensional object with different shapes on each finger may be formed or fixed on the surface of the glove. By using a protrusion with a characteristic shape as a mark for position detection set on the palm side of the proximal phalanx, the angle range of the proximal phalanx that can be detected by the camera is widened, and the fingertip position estimation can be performed more stably.

[0119] <Variation 4>

[0120] Fig.13 This is a diagram showing a configuration example of a glove 200 according to a fourth modification of the present embodiment.

[0121] The photographing device 201 may also be provided with multiple devices. That is, multiple cameras on the gloves may also be provided. By configuring two or more camera modules as optical imaging elements on the gloves, the angle range of the markers that can be detected by the camera can be widened, and the fingertip position estimation can be performed more stably.

[0122] <Variant 5>

[0123] In this embodiment, the functions of the position and posture estimation unit 110, the range calculation unit 120, the contact position calculation unit 130, and the determination unit 140 are implemented by software. As a modified example, the functions of the position and posture estimation unit 110, the range calculation unit 120, the contact position calculation unit 130, and the determination unit 140 may also be implemented by hardware.

[0124] Specifically, the fixed state determination system 500 includes an electronic circuit 909 instead of the processor 910 .

[0125] Fig.14 This is a diagram showing a configuration example of an information processing device 100 according to a fifth modification of the present embodiment.

[0126] The electronic circuit 909 is a dedicated electronic circuit that realizes the functions of the position and posture estimation unit 110, the range calculation unit 120, the contact position calculation unit 130, and the determination unit 140. Specifically, the electronic circuit 909 is a single circuit, a composite circuit, a programmed processor, a parallel programmed processor, a logic IC, a GA, an ASIC, or an FPGA. GA is the abbreviation of Gate Array. ASIC is the abbreviation of Application Specific Integrated Circuit. FPGA is the abbreviation of Field-Programmable Gate Array.

[0127] The functions of the position and posture estimation unit 110 , the range calculation unit 120 , the contact position calculation unit 130 , and the determination unit 140 may be implemented by one electronic circuit or may be implemented by distributing them to a plurality of electronic circuits.

[0128] As another variation, some of the functions of the position and posture estimation unit 110, the range calculation unit 120, the contact position calculation unit 130, and the determination unit 140 may be implemented by electronic circuits, and the remaining functions may be implemented by software. In addition, some or all of the functions of the position and posture estimation unit 110, the range calculation unit 120, the contact position calculation unit 130, and the determination unit 140 may also be implemented by firmware.

[0129] The processor and the electronic circuit are also referred to as processing circuits, respectively. That is, the functions of the position and posture estimation unit 110, the range calculation unit 120, the contact position calculation unit 130, and the determination unit 140 are implemented by the processing circuits.

[0130] In addition, the computer mounted on the glove 200 is also the same as that of the fifth modification.

[0131] **Description of the Effects of the Embodiment**

[0132] In the fixing state determination system of the present embodiment, a worker on site wears a glove having a tactile force sensor, grasps an object and vibrates it, and based on the sensor information at this time, tactile information is presented to an inspector at a remote location.

[0133] For example, in a hand-eye system that detects the position of an object by using a camera installed in the robot when the robot grasps the object, the fingertips in the grasping process may enter the blind spot of the object, so the position of the fingertips cannot be grasped. Therefore, the position / direction of the excitation force applied from the finger to the object cannot be grasped, and the judgment accuracy may be reduced.

[0134] On the other hand, according to the fixing condition determination system of this embodiment, the contact position of the fingertip in the object can be accurately obtained. Therefore, information such as the direction of the force applied to the object and the accurate displacement of the object can be accurately grasped. Therefore, the fixing condition of the object can be determined with high accuracy.

[0135] In the fixing state determination system of this embodiment, the following gloves are used: different marks are printed on the palm side corresponding to the bones (proximal phalanx) at the base of the fingers for each finger, and a small camera is built into the space between the thumb and the index finger. Then, the following processing is performed.

[0136] (1) Using the shape data of the inspection object, the camera calculates the position of the object.

[0137] (2) Calculate the angle of the proximal phalanx of each finger based on the markers captured by the camera.

[0138] (3) Based on the angle information of the proximal phalanx, a range of candidates for the fingertip position is calculated.

[0139] (4) Calculate the contact position based on the relationship between the position / shape information of the object and the candidate fingertip position.

[0140] (4) Based on the changes in the contact position and contact force information of the fingertip during the excitation operation, the change in three-dimensional excitation force is calculated and integrated with the information on the displacement of the object to calculate the fixing status of the object.

[0141] As described above, according to the fixing state determination system of the present embodiment, the fingertip position estimation accuracy is improved, and the excitation position can be estimated accurately and stably.

[0142] In the above embodiment 1, each part of the fixed state determination system is described as an independent functional module. However, the structure of the fixed state determination system may not be the structure of the above embodiment. The functional module of the fixed state determination system may be any structure as long as it can realize the functions described in the above embodiment. In addition, the fixed state determination system may be a system composed of multiple devices instead of one device.

[0143] Furthermore, a plurality of parts in Embodiment 1 may be implemented in combination. Alternatively, one part of this embodiment may be implemented. In addition, this embodiment may be implemented in combination in whole or in part.

[0144] That is, in Embodiment 1, it is possible to freely combine the various embodiments or to modify any constituent element of the various embodiments, or to omit any constituent element in the various embodiments.

[0145] In addition, the above embodiments are essentially preferred illustrations and are not intended to limit the scope of the present disclosure, the scope of the application objects of the present disclosure, and the scope of the use of the present disclosure. The above embodiments can be modified in various ways as needed. For example, the process described using a flowchart or a timing diagram can also be appropriately modified.

[0146] Description of Reference Numerals

[0147] 10 operator, 25 object image, 26 excitation force information, 61 image candidate, 100 information processing device, 110 position posture estimation unit, 120 range calculation unit, 130 contact position calculation unit, 140 determination unit, 160 storage unit, 161 object database, 200 glove, 201 shooting device, 202 touch force sensor, 203 sensor data collection module, 204 marker, 205 motion sensor, 206 fixing device, 261 ring, 262 rigid body, 300 object, 500 fixing status determination system, 909 electronic circuit, 910 processor, 921 memory, 922 auxiliary storage device, 930 input interface, 940 output interface, 950 communication device.

Claims

1. A fixing condition determination system, comprising gloves worn by a worker to hold an object, the fixing condition determination system determining a fixing condition of the object, wherein: The fixing condition determination system comprises: a photographing device disposed in the glove at a position corresponding to between a pair of fingers holding the object, and photographing the object held by the pair of fingers as an object image; a position and posture estimating unit that estimates the position and posture of the object using the object image; a range calculation unit that analyzes the movement of the operator's fingertips using the object image and calculates a fingertip movement range that is a movement range of the positions of the fingertips of the operator's fingers; a contact position calculation unit that calculates a contact position of the fingertip in the object based on the position and posture of the object and the movement range of the fingertip; as well as The determination unit determines the fixing state of the object based on the excitation force information which is information of the excitation force on the object at the contact position of the fingertip and the displacement of the object.

2. The fixing condition determination system according to claim 1, wherein: The glove has a mark at a position corresponding to the inner side of the proximal phalanx of each finger of the operator. The imaging device captures an image of the object including the position of the mark when the glove grips the object. The range calculation unit calculates the angle of the proximal phalanx of each finger of the operator using the position of the marker in the target image, and calculates the fingertip movement range based on the angle of the proximal phalanx of each finger of the operator.

3. The fixing condition determination system according to claim 2, wherein: The glove includes a motion sensor on the outer side of the fingertips of each finger of the operator to detect the motion of the fingertips. The range calculation unit calculates the fingertip movement range based on an angle of a proximal phalanx of each finger of the operator and a movement of the fingertip detected by the motion sensor.

4. The fixing condition determination system according to any one of claims 1 to 3, wherein: The fixed state determination system includes an object database storing images of a plurality of combinations of positions and postures of the object as image candidates. The position and posture estimation unit estimates the position and posture of the object by comparing the object image with the image candidates accumulated in the object database.

5. The fixing condition determination system according to any one of claims 1 to 4, wherein: The glove includes a force sensor at a position corresponding to the inner side of the operator's fingertips. The determination unit acquires the excitation force on the object at the contact position of the fingertip from the force sensor, and acquires the displacement of the object from the object image.

6. A glove provided in a fixing condition determination system, the fixing condition determination system comprising a glove worn by an operator to hold an object and an information processing device communicating with the glove, the fixing condition determination system determining the fixing condition of the object, wherein: The gloves have: an imaging device disposed at a position corresponding to between a pair of fingers holding the object, and capturing the object held by the pair of fingers as an object image; and A communication device that sends the object image to the information processing device, the information processing device uses the object image to estimate the position and posture of the object, uses the object image to calculate the movement range of the position of the fingertips of each finger of the operator, that is, the fingertip movement range, and calculates the contact position of the fingertip in the object based on the position and posture of the object and the fingertip movement range.

7. The glove according to claim 6, wherein: The glove has a mark at a position corresponding to the inner side of the proximal phalanx of each finger of the operator. The imaging device captures an image of the object including the position of the mark when the glove grips the object. The information processing device calculates the angle of the proximal phalanx of each finger of the operator using the position of the marker in the target image, and calculates the fingertip movement range based on the angle of the proximal phalanx of each finger of the operator.

8. The glove according to claim 7, wherein: The glove includes a motion sensor on the outer side of the fingertips of each finger of the operator to detect the motion of the fingertips. The information processing device calculates the fingertip movement range based on the angle of the proximal phalanx of each finger of the operator and the movement of the fingertip detected by the motion sensor.

9. The glove according to claim 7 or 8, wherein: The glove includes a fixing tool provided on each of the pair of fingers in the glove to fix the imaging device.

10. The glove according to claim 9, wherein: The fixing device has a shape including a ring provided on each of the pair of fingers in the glove and a rigid body connecting the ring and the imaging device.

11. The glove according to any one of claims 7 to 10, wherein: The mark has a protrusion shape.

12. The glove according to any one of claims 6 to 11, wherein: A plurality of the above-mentioned imaging devices are provided.

13. The glove according to any one of claims 6 to 12, wherein: The glove includes a force sensor at a position corresponding to the inner side of the operator's fingertips. The information processing device acquires information of an excitation force on the object at a contact position of the fingertip from the force sensor as excitation force information, and acquires a displacement of the object from the object image.

14. An information processing device provided in a fixing condition determination system, the fixing condition determination system comprising a glove worn by an operator to hold an object and an information processing device communicating with the glove, the fixing condition determination system determining the fixing condition of the object, wherein: The information processing device comprises: a position and posture estimation unit that estimates the position and posture of the object using an object image captured by a camera, the camera being disposed in the glove at a position corresponding to between a pair of fingers holding the object, and capturing the object held by the pair of fingers as the object image; a range calculation unit that analyzes the movement of the operator's fingertips using the object image and calculates a fingertip movement range that is a movement range of the positions of the fingertips of the operator's fingers; a contact position calculation unit that calculates a contact position of the fingertip in the object based on the position and posture of the object and the movement range of the fingertip; as well as A determination unit determines a fixing condition of the object based on an excitation force of the excitation applied to the object at a contact position of the fingertip and a displacement of the object.

15. A method for determining a fixing condition, used in a fixing condition determination system, the fixing condition determination system comprising gloves worn by an operator to hold an object, the fixing condition determination system determining a fixing condition of the object, wherein: The imaging device disposed in the glove at a position corresponding to between a pair of fingers holding the object captures the object held by the pair of fingers as an object image, The computer estimates the position and posture of the object using the object image, The computer analyzes the movement of the operator's fingertips using the object image, calculates the movement range of the positions of the operator's fingertips, that is, the fingertip movement range, The computer calculates the contact position of the fingertip in the object based on the position and posture of the object and the movement range of the fingertip, The computer determines the fixing condition of the object based on the excitation force of the excitation on the object at the contact position of the fingertip and the displacement of the object.

16. A fixing condition determination program used in a fixing condition determination system, the fixing condition determination system comprising gloves worn by an operator to hold an object, the fixing condition determination system determining the fixing condition of the object, wherein: The fixing state determination program causes the computer to execute the following processing: Position and posture estimation processing, estimating the position and posture of the object using an object image captured by a camera, the camera being arranged in the glove at a position corresponding to between a pair of fingers holding the object, and capturing the object held by the pair of fingers as the object image; Range calculation processing, using the object image, analyzing the movement of the operator's fingertips, and calculating the movement range of the positions of the fingertips of the operator's fingers, that is, the fingertip movement range; A contact position calculation process of calculating a contact position of the fingertip in the object based on the position and posture of the object and the movement range of the fingertip; as well as The determination process determines the fixing condition of the object based on the excitation force of the excitation on the object at the contact position of the fingertip and the displacement of the object.

Images