Fixation state determination system, glove, information processing device, fixation state determination method and fixation state determination program

Abstract

Fixation state determination system (500) comprising a glove (200) worn by an operator (10) to grasp a target object (300) and determining a fixation state of the target object (300), wherein the fixation state determination system (500) comprises: a photographic device (201) which is placed at a position of the glove (200) corresponding to a section between a pair of fingers which grasp the target object (300) in order to photograph the target object (300) grasped by the pair of fingers as a target image (25); a position and orientation determination unit (110) to determine the position and orientation of the target object (300) using the target image (25); an area calculation unit (120) to analyze a movement of a fingertip of the operator (10) using the target image (25), and to calculate a fingertip movement area which is a movement area of ​​a position of a fingertip of each finger of the operator (10); a contact position calculation unit (130) to calculate a contact position of the fingertip on the target object (300) based on the position and orientation of the target object (300) and the fingertip movement range; and a determination unit (140) to determine the fixation state of the target object (300) on the basis of a displacement of the target object (300) and excitation force information (26), which is information about an excitation force at the contact position of the fingertip for the target object (300).

Description

Technical field

[0001] The present disclosure relates to a fixation state determination system, a glove, an information processing device, a fixation state determination method and a fixation state determination program. State of the art

[0002] There is a glove that an operator wears on site to remotely check the fixation status of a device or apparatus.

[0003] Using a glove, it's possible to measure changes in force applied to a contact surface and the vibration of a fingertip. However, information such as the position and orientation of a target object relative to the glove and the position of the fingertip on the target object cannot be obtained. Therefore, it's not possible to obtain information such as the direction or orientation of the force acting on the target object or its precise displacement.

[0004] Patent literature 1 discloses a robot hand finger in which a CCD camera is attached to the inside of a part of the fingertip covered with an elastic body in order to detect contact with a target object, thereby measuring the deformation of the elastic body.

[0005] Patent literature 2 discloses a production monitoring system. The production monitoring system comprises a production work area. Technicians equipped with gloves containing force sensors and / or markers for motion tracking can work within the production work area. Motion tracking cameras can surround the production work area. Motion data from the cameras or force data from the gloves can be sent to a remote workstation, where it is processed and analyzed. The workstation can generate quality assessments or training results based on the motion or force data. Reference list patent literature Patent Literature 1: JP 2010-221359 A Patent literature 2: US 2020 / 0 117 271 A1 Summary of the invention: Technical problem

[0006] To check a device's fixation state, such as rattling, an operator on site wears a glove and performs a procedure such as grasping a target object with the glove and applying vibrations to it.

[0007] By accurately recording the position of a fingertip on the target object, the fixation state at that time can be determined with greater precision.

[0008] Even when the technique described in patent literature 1 is applied to the glove, the contact position of a fingertip on the target object cannot be accurately determined, making it impossible to obtain information such as the direction of a force acting on the target object and its exact displacement. This raises the problem that the fixation state of the target object cannot be determined with high precision.

[0009] The purpose of the present disclosure is to determine a fixation state of a target object with higher precision by accurately preserving the contact position of a fingertip on the target object. Solution to the problem

[0010] The task is solved by a fixation state determination system comprising a glove worn by an operator to grasp a target object, and which determines a fixation state of the target object, wherein the fixation state determination system comprises: a photographic device placed at a position on the glove corresponding to a section between a pair of fingers grasping the target object, in order to photograph the target object grasped by the pair of fingers as a target image; a position and orientation determination unit to determine the position and orientation of the target object using the target image; an area calculation unit to analyze a movement of an operator's fingertip using the target image, and to calculate a fingertip movement area, which is a movement range of a position of a fingertip of each of the operator's fingers; a contact position calculation unit to calculate a contact position of the fingertip on the target object based on the position and orientation of the target object and the fingertip movement range; and a determination unit to determine the fixation state of the target object based on a displacement of the target object and excitation force information, which is information about an excitation force at the contact position of the fingertip for the target object. Advantageous effects of the invention

[0011] One purpose of the present disclosure is to determine a fixation state of a target object with higher precision by accurately preserving the contact position of a fingertip on the target object. Brief description of the drawings Fig. Figure 1 is a representation illustrating an overall configuration example for a fixation state determination system according to embodiment 1. Fig. Figure 2 is a representation illustrating a configuration example of an information processing device according to embodiment 1. Fig. Figure 3 is a representation illustrating a configuration example of a glove according to embodiment 1. Fig. Figure 4 is a representation to illustrate a situation in which the glove according to embodiment 1 grasps a target object. Fig. Figure 5 is a flowchart illustrating an operational example of the fixation state determination system according to embodiment 1. Fig. Figure 6 is a representation illustrating an example of a target image according to embodiment 1. Fig. Figure 7 is a representation illustrating the concept of an angle of a proximal phalanx of each finger in a range calculation process according to embodiment 1. Fig. Figure 8 is a representation to illustrate a concept of a fingertip movement range in the range calculation process according to embodiment 1. Fig. Figure 9 is a representation to illustrate a concept of a contact position in a contact position calculation process according to embodiment 1. Fig. Figure 10 is a representation illustrating a configuration example of a glove according to modification 1 of embodiment 1. Fig. Figure 11 is a representation illustrating a configuration example of a glove according to modification 2 of embodiment 1. Fig. Figure 12 is a representation illustrating a configuration example of a glove according to modification 3 of embodiment 1. Fig. Figure 13 is a representation illustrating a configuration example of a glove according to modification 4 of embodiment 1. Fig. Figure 14 is a representation illustrating an example of an information processing device according to modification 5 of embodiment 1. Description of embodiments

[0012] The present embodiment is described below with reference to the drawings. In the drawings, identical or equivalent parts are designated by the same reference numerals. In the description of the embodiment, an explanation of the identical or equivalent part may be omitted or simplified where appropriate. Arrows in the drawings primarily indicate data flows or process sequences. Furthermore, the dimensional relationships between components may differ from their actual dimensions in the drawings below. The description of the embodiment may specify a direction or orientation, or positions such as "top," "bottom," "left," "right," "front," and "back." These representations are for descriptive purposes only and are not intended to restrict the positioning, direction, or orientation of equipment, tools, components, etc. Design 1.*** Description of configurations ***

[0013] Fig. Figure 1 is a representation illustrating a configuration example for a fixation state determination system 500 according to the present embodiment.

[0014] The fixation state determination system 500 is a system in which an operator 10 wears a glove, grasps a target object 300 and exerts an excitation force on the target object 300, thereby determining a fixation state of the target object 300.

[0015] The fixation state determination system 500 is equipped with a glove 200, worn by the operator 10 to grasp the target object 300, and an information processing device 100 that communicates with the glove 200. Communication between the glove 200 and the information processing device 100 can be either wired or wireless.

[0016] The following describes detailed configurations of the information processing unit 100 and the glove 200.

[0017] Fig. Figure 2 is a representation showing a configuration example of the information processing device 100 according to the present embodiment.

[0018] The Information Processing Unit 100 is a computer. The Information Processing Unit 100 is equipped with a Processor 910 as well as other hardware components, such as a Main Memory 921, an Auxiliary Memory 922, an Input Interface 930, an Output Interface 940, and a Communication Unit 950. The Processor 910 is connected to the other hardware components via wired or wireless communication and controls them.

[0019] The information processing unit 100 is equipped with a position and location determination unit 110, an area calculation unit 120, a contact position calculation unit 130, a determination unit 140, and a storage unit 160 as functional elements. A target object database 161 is stored in the storage unit 160.

[0020] The functions of the position and orientation determination unit 110, the area calculation unit 120, the contact position calculation unit 130, and the determination unit 140 are implemented by software. The memory unit 160 is provided in the main memory 921. The memory unit 160 can be provided in the auxiliary memory unit 922, or it can be provided in both the main memory 921 and the auxiliary memory unit 922 by distribution.

[0021] The 910 processor is a device on which a fixation state determination program runs. This fixation state determination program implements the functions of the 110 position and orientation determination unit, the 120 area calculation unit, the 130 contact position calculation unit, and the 140 determination unit. It should be noted that the fixation state determination program includes a program that implements a glove function, which will be described later.

[0022] The 910 processor is an integrated circuit (IC) that performs computational processing. Specific examples of the 910 processor include a CPU, a DSP, and a GPU. It should be noted that IC stands for Integrated Circuit; DSP for Digital Signal Processor; and GPU for Graphics Processing Unit.

[0023] Main memory (RAM) is a storage device in which data is temporarily stored. A concrete example of main memory is SRAM or DRAM. It should be noted that SRAM stands for Static Random-Access Memory; and DRAM for Dynamic Random-Access Memory.

[0024] The Auxiliary Storage Device 922 is a storage device in which data is stored. A specific example of the Auxiliary Storage Device 922 is a hard disk drive (HDD). The Auxiliary Storage Device 922 can be a portable storage medium, such as an SD (registered trademark), a memory card, a CF (compact disk), NAND flash memory, a flexible disk, an optical disk, a compact disk, a Blu-ray disc (registered trademark), and a DVD. It should be noted that HDD stands for Hard Disk Drive, SD (registered trademark) for Secure Digital, CF for CompactFlash (registered trademark), and DVD for Digital Versatile Disc.

[0025] The 930 input interface is a port for connecting to an input device such as a mouse, keyboard, or touch panel. Specifically, the 930 input interface is a USB port. The 930 input interface can also be used to connect to a LAN. Note that USB stands for Universal Serial Bus, and LAN stands for Local Area Network.

[0026] The 940 output interface is a connector used to connect a cable to an output device, such as a display. Specifically, the 940 output interface is either a USB port or an HDMI port (registered trademark). The display in question is an LCD. The 940 output interface is also referred to as a display unit interface. Note that HDMI (registered trademark) stands for High Definition Multimedia Interface, and LCD stands for Liquid Crystal Display.

[0027] The Communication Device 950 comprises a receiver and a transmitter. The Communication Device 950 is connected to a communication network, such as a LAN, the Internet, Wi-Fi (registered trademark), or a telephone line. Specifically, the Communication Device 950 is a communication chip or NIC. Note that NIC stands for Network Interface Card.

[0028] The fixer state determination program runs in information processing unit 100. The fixer state determination program is read by processor 910 and executed by processor 910. Not only the fixer state determination program, but also an operating system (OS) is stored in main memory 921. Note that OS stands for operating system. Processor 910 executes the fixer state determination program while the OS is running. The fixer state determination program and the OS can be stored in auxiliary memory unit 922. The fixer state determination program and the OS stored in auxiliary memory unit 922 are loaded into main memory 921 and executed by processor 910. The fixer state determination program can be partially or completely integrated into the OS.

[0029] The Information Processing Unit 100 can be equipped with multiple processors that replace the Processor 910. These multiple processors jointly execute the fixing state determination program. Each processor is a separate unit that executes the fixing state determination program just like the Processor 910.

[0030] Data, information, signal values ​​and variable values ​​used, processed or output by the fixing state determination program are stored in the main memory 921, in the auxiliary memory device 922 or in a register or cache memory in the processor 910.

[0031] The term "unit" in each of the terms: Position-and-Position Determination Unit 110; Area Calculation Unit 120; Contact Position Calculation Unit 130; and Determination Unit 140 may be replaced by "circuit," "stage," "procedure," "process," or "circuit." The Fix-State Determination Program causes the computer to perform a Position-and-Position Determination Process, an Area Calculation Process, a Contact Position Calculation Process, a Determination Process, and an Image Setting Process. The term "process" in each of the terms: Position-and-Position Determination Process; Area Calculation Process; Contact Position Calculation Process; and Determination Process may be replaced by "program," "program product," "program-stored computer-readable media," or "program-recorded computer-readable media."The fixation state determination procedure is a procedure that is carried out by the information processing unit 100, which executes the fixation state determination program.

[0032] The fixer state determination program can be stored on a computer-readable data carrier. Alternatively, the fixer state determination program can be provided as a software product.

[0033] Fig. Figure 3 is a representation showing a configuration example of the glove 200 according to the present embodiment.

[0034] Fig. Figure 4 is a representation to illustrate a situation in which the glove 200, according to the present embodiment, grasps the target object 300.

[0035] If the operator 10 is to grasp the target object 300 and exert a stimulating force on it, the glove 200 is pulled over one hand of the operator 10, who is to grasp the target object 300.

[0036] Fig. Figure 3 shows glove 200, which is worn on the right hand of operator 10. The left part of Fig. Figure 3 shows glove 200 as seen from the palm. The right part of Fig. Figure 3 shows glove 200 as seen from the back of the hand.

[0037] Fig. Figure 4 shows how the operator 10 wears the glove 200 to grasp the target object 300.

[0038] The glove 200 is equipped with a photography device 201, contact force sensors 202, a sensor data acquisition module 203 and markers 204.

[0039] The photographic device 201 is placed at a position of the glove 200 corresponding to a section between a pair of fingers grasping the target object 300, and photographs the target object 300 grasped by a pair of fingers as a target image 25.

[0040] The photographic apparatus 201, for example, is a device formed by placing a module with an optical image sensor on a substrate. The photographic apparatus 201 is, for example, a CCD camera.

[0041] Normally, people grasp an object with their thumb and a finger other than their thumb. Fig. 3. The operator 10 grasps the object with their fingers, including the thumb and index finger. Thus, the photographic device 201 is positioned on the glove 200 corresponding to a palm-side section of the thumb and index finger. Any other position is sufficient, provided it is located where the grasped target object 300 is to be photographed. The photographic device 201 can be placed at the base of a finger. Alternatively, the photographic device 201 can also be placed near the center of the palm.

[0042] The marker 204 is placed at a position on the glove 200 that corresponds to the inside of a proximal finger joint of each finger of operator 10. Specifically, the marker 204 is printed at a position on the glove 200 that corresponds to the inside of a proximal finger joint of each finger of operator 10.

[0043] The photographic device 201 photographs the target image 25, including the position of the marker, when the glove 200 grasps the target object. That is, the target image 25 contains the marker 204, which is printed on each finger of the pair of fingers.

[0044] Each contact force sensor 202 is positioned on the glove 200 at a location corresponding to the inside of the fingertip of the operator 10. The contact force sensor 202 acquires information about an excitation force of the target object 300 as excitation force information 26. The excitation force information 26 expresses, for example, the magnitude of the excitation force.

[0045] The sensor data acquisition module 203 acquires various types of sensor data and transmits the data to the information processing unit 100. Specifically, the sensor data acquisition module 203 acquires the target image 25 from the photographing unit 201 and the excitation force information 26 from the contact force sensor 202. The sensor data acquisition module 203 then transmits the target image 25 and the excitation force information 26 to the information processing unit 100 via wired or wireless communication.

[0046] The sensor data acquisition module 203 is a computer that acquires sensor data such as the target image 25 and the excitation force information 26 and transmits the sensor data to the information processing unit 100.

[0047] The information processing unit 100, for example, is equipped with hardware components such as an input interface for sensor data, a processor, and a communication device for communication with the information processing unit 100. The explanation of the individual hardware components corresponds to the explanation of the information processing unit 100. *** Description of Functionality ***

[0048] The operating functions of the fixation state determination system 500 according to the present embodiment are described below. An operating sequence of the fixation state determination system 500 corresponds to the fixation state determination method. A program that implements operations of the fixation state determination system 500 corresponds to the fixation state determination program.

[0049] Fig. Figure 5 is a flowchart that illustrates an operational example of the fixation state determination system 500 according to the present embodiment. < Photography process >

[0050] In step S101, the photographic device 201 photographs the target object 300, which is being grasped by a pair of fingers, as the target image 25.

[0051] Specifically, the photographic apparatus 201 performs a photograph to contain the target object 300, which is grasped by the pair of fingers, as well as the markers 204, which are printed on the individual fingers of the pair of fingers.

[0052] The sensor data acquisition module 203 transmits the target image 25 to the information processing unit 100.

[0053] Fig. Figure 6 is a representation illustrating an example of the target image 25 using the photographic device 201 according to the present embodiment.

[0054] As in Fig. As shown in Figure 6, depending on the shape of the target object, 300 contact sections of the target object 300 cannot be photographed with the fingertips.

[0055] Although not shown, target image 25 contains the markers 204 printed on each finger of the pair of fingers. < Position and orientation determination process >

[0056] In step S102, the position and location determination unit 110 determines the position and location of the target object 300 using the target image 25. The position and location determination unit 110 compares the target image 25 with image candidates 61 that are accumulated in the target object database 161, and thus determines the position and location of the target object 300.

[0057] Specifically, this is as follows.

[0058] Individual images from a multitude of combinations of position and location of the target object 300 are accumulated as image candidates 61 in the target object database 161.

[0059] The image candidates 61 are a large number of three-dimensional data obtained by photographing the target object 300 in combinations from different angles and different positions.

[0060] The position and location determination unit 110 compares the target image 25 with image candidates 61 and selects an image candidate 61 that has the highest similarity, thereby determining the position and location of the target object 300 in the target image 25. < Area calculation process >

[0061] In step 103, the area calculation unit 120 analyzes a movement of an operator's fingertip 10 using the target image 25 and calculates a fingertip movement area R, which is the movement range of a position of the fingertip of each of the operator's fingers. Step S103 includes steps S31 and S32.

[0062] Specifically, this is as follows.

[0063] Fig. Figure 7 is a representation illustrating a concept for a proximal finger segment of each finger in the area calculation process according to the present embodiment.

[0064] In step S31, the area calculation unit 120 calculates the angle of the proximal finger joint of each finger of operator 10 using the position of marker 204 on the proximal finger joint in target image 25. The example in Fig. Figure 7 shows how to calculate an angle θ1 of the proximal phalanx of the thumb and an angle θ2 of the proximal phalanx of the index finger.

[0065] Fig. Figure 8 is a representation to illustrate a concept of a range of motion R in the range calculation process according to the present embodiment.

[0066] In step S32, the area calculation unit 120 calculates the fingertip movement range R based on the angle of the proximal phalanx of each finger of operator 10. The area calculation unit 120 calculates the fingertip movement range R of each finger based on the angle of the proximal phalanx of the finger using information about the joint structure of the finger in a normal state. < Contact position calculation process >

[0067] In step S104, the contact position calculation unit 130 calculates a contact position P of the fingertip on the target object 300 based on the position and orientation of the target object 300 and the fingertip movement range R.

[0068] Fig. Figure 9 is a representation to illustrate a concept of the contact position P in the contact position calculation process according to the present embodiment.

[0069] The contact position calculation unit 130 calculates a surface position of the target object 300 from the position and orientation of the target object 300. Then, the contact position calculation unit 130 calculates the contact position P between the target object 300 and the fingertip from an intersection between the surface position of the target object 300 and the fingertip movement range R. <Determination process>

[0070] In step S105, the determination unit 140 determines the fixation state of the target object 300 based on the displacement of the target object 300 and the excitation force information 26, which is information about the excitation force at the contact position P of the fingertip for the target object 300. Step S105 includes steps S51 and S52.

[0071] Specifically, this is as follows.

[0072] In step S51, the operator 10 puts on the glove 200, grasps the target object 300, and applies the excitation force to the target object 300. The sensor data acquisition module 203 then transmits a value of the excitation force, acquired by the contact force sensors, to the information processing device 100 as the excitation force information 26. The excitation force information 26 is acquired via the contact force sensors 202 and transmitted from the glove 200 to the information processing device 100 via the sensor data acquisition module 203.

[0073] In step S52, the determination unit 140 determines the fixation state of the target object 300 based on the contact position P, the excitation force information 26 and the displacement of the target object 300.

[0074] The displacement of the target object 300 is acquired by the determination unit 140 from the target image 25.

[0075] The determination unit 140 calculates the magnitude and direction of the excitation force at the contact position P, acquires the displacement, which is a response of the target object 300, and integrates these calculation results to determine the fixation state or fixation situation of the target object 300.

[0076] As described above, the fixation state can be evaluated as an indicator using a relationship between the displacement of the target object 300 and the excitation force. *** Other Configurations ***< Modification 1 >

[0077] Fig. Figure 10 is a representation that shows an example of a configuration of a glove 200 according to modification 1 of the present embodiment.

[0078] The glove 200 can be equipped with a motion sensor on the outside of a fingertip of each finger of an operator 10 to detect the movement of the fingertip.

[0079] A range calculation unit 120 calculates a fingertip movement range R based on an angle of the proximal finger joint of each finger of the operator 10 and the movement of the fingertip detected by a motion sensor 205.

[0080] The motion sensor 205, for example, is an accelerometer that can measure acceleration. The area calculation unit 120 determines a fingertip position based on the displacement from the starting position. The number of motion sensors used can be changed to account for variations in an assumed gripping method.

[0081] With the glove 200 according to modification 1, the position determination accuracy of the fingertip is improved, and an excitation position can be determined accurately and stably. < Modification 2 >

[0082] Fig. Figure 11 is a representation that shows an example of a configuration of a glove 200 according to modification 2 of the present embodiment.

[0083] The glove 200 can be provided with a fixing tool 206, which is attached to each finger of a pair of fingers of the glove 200, in order to fix a photographic device 201. The fixing tool 206 is provided, for example, with a ring 261, which is attached to each finger of the pair of fingers of the glove 200, and with a rigid body 262, which connects the ring 261 to the photographic device 201.

[0084] With the glove 200 according to modification 2, since the ring 261, which is fixed to the finger, and the photographing device 201 are connected to the rigid body 262, the position determination accuracy of the finger and the photographing device 201 is improved, and the determination accuracy of the fingertip position is also improved. < Modification 3 >

[0085] Fig. Figure 12 is a representation that shows an example of a configuration of a glove 200 according to modification 3 according to the present embodiment.

[0086] Each Marker 204 can have a protruding shape.

[0087] The markers corresponding to the proximal finger joints can be three-dimensional objects of varying shapes formed between the fingers or fixed to a glove surface. Using a protrusion with a characteristic projecting shape as a position-detection marker, placed on the palm side of the proximal finger joint, expands the angular range of the proximal finger joint for which a marker can be detected by a camera, thus enabling more stable determination of the fingertip position. < Modification 4 >

[0088] Fig. Figure 13 is a representation that shows an example of a configuration of a glove 200 according to modification 4 of the present embodiment.

[0089] The glove can be equipped with a variety of photographic devices. This means that multiple cameras can be attached to the glove. If two or more camera modules are mounted on the glove as optical image sensors, the angle range in which the cameras can detect the markers is increased, allowing for more stable determination of the fingertip's position. < Modification 5 >

[0090] In the present embodiment, the functions of the position and orientation determination unit 110, the area calculation unit 120, the contact position calculation unit 130, and the determination unit 140 are implemented by software. According to a modification, the functions of the position and orientation determination unit 110, the area calculation unit 120, the contact position calculation unit 130, and the determination unit 140 can be implemented by hardware.

[0091] Specifically, a fixing state determination system 500 is equipped with an electronic circuit 909 instead of a processor 910.

[0092] Fig. Figure 14 is a representation that shows an example of a configuration of an information processing device 100 according to modification 5 of the present embodiment.

[0093] The electronic circuit 909 is a dedicated electronic circuit that implements the functions of the position and orientation determination unit 110, the range calculation unit 120, the contact position calculation unit 130, and the determination unit 140. The electronic circuit 909 is, in particular, a single circuit, a composite circuit, a programmed processor, a parallel-programmed processor, a logic IC, a GA, an ASIC, or an FPGA. It should be noted that GA stands for Gate Array; ASIC for Application Specific Integrated Circuit; and FPGA for Field-Programmable Gate Array.

[0094] The functions of the position and orientation determination unit 110, the area calculation unit 120, the contact position calculation unit 130 and the determination unit 140 can be implemented by an electronic circuit, or can be implemented by a multitude of electronic circuits through distribution.

[0095] According to a further modification, some of the functions of the position and orientation determination unit 110, the area calculation unit 120, the contact position calculation unit 130, and the determination unit 140 can be implemented by an electronic circuit, and the remaining functions can be implemented by software. Furthermore, some or all of the functions of the position and orientation determination unit 110, the area calculation unit 120, the contact position calculation unit 130, and the determination unit 140 can be implemented by firmware.

[0096] The processor and the electronic circuit are also referred to as a processing circuit. This means that the functions of the position and orientation determination unit 110, the area calculation unit 120, the contact position calculation unit 130, and the determination unit 140 are implemented by a processing circuit.

[0097] The description for modification 5 also applies to the computer installed in glove 200. *** Description of the effect of the embodiment ***

[0098] In the fixation state determination system according to the present embodiment, the operator puts on a glove with a contact force sensor, grasps a target object, and applies an excitation force to the target object. The haptic information, based on the sensor data thus acquired, is presented to a person responsible for testing at a remote location.

[0099] For example, if a hand-eye system is used when a robot hand grasps an object, and the object's position is captured by a camera attached to the robot hand unit, the fingertips enter a blind spot formed by the target object, making it impossible to determine their positions. Therefore, the application position and direction of any excitation force exerted by the fingers on the target object cannot be obtained, which can consequently impair the accuracy of the grasp.

[0100] On the other hand, a fixation state determination system according to the present embodiment can accurately obtain the contact positions of the fingertips on the target object. This allows for the precise acquisition of information such as the direction of a force acting on the target object and its exact displacement. As a result, the fixation state of the target object can be determined with high accuracy.

[0101] The fixation state determination system according to the present embodiment uses a glove in which different markers are printed on sections of the palm side, corresponding to the bones at the base of the fingers (proximal phalanges). A compact camera is installed in the glove in a space between the thumb and index finger. The following processes are then carried out. (1) The position of the target object is calculated by the camera using the shape data of a test object. (2) Using the marker displayed on the camera, the angle of the proximal phalanx of each finger is calculated. (3) An area that is a candidate for a fingertip position is calculated on the basis of the angular information of the proximal phalanx. (4) A contact position is calculated from a relationship between position and shape information of the target object and the candidate fingertip position. (5) A change in a three-dimensional excitation force is calculated from a change in the fingertip contact position and a change in the contact force information obtained during an excitation process, and the calculated change is integrated with the information about the displacement of the target object to calculate the fixed state of the target object.

[0102] As described above, the fixation state determination system according to the present embodiment improves the fingertip position determination accuracy, so that the excitation position can be determined accurately and stably.

[0103] In the embodiment described above, the individual units of the fix state determination system are described as independent functional blocks. However, the configuration of the fix state determination system need not necessarily be limited to a configuration like that described above. The functional blocks of the fix state determination system can have any configuration, as long as they can implement the functions described in the embodiment above. The fix state determination system can be formed by a plurality of devices instead of just one.

[0104] Several parts of embodiment 1 can be implemented in practice in combination. Alternatively, only a part of the embodiment can be implemented in practice. The embodiment can be implemented in practice as a whole or in parts in any combination.

[0105] That is, in embodiment 1, the individual embodiments can be freely combined; any component of each embodiment can be modified; and any component of each embodiment can be omitted.

[0106] The embodiment described above is a substantially preferred example and is not intended to limit the scope of this disclosure, the scope of any applied product of this disclosure, or any application area of ​​this disclosure. Various modifications can be made to the embodiment described above as needed. The processes illustrated by means of flowcharts or sequence diagrams may be modified as necessary. Reference symbol list

[0107] 10: Operator; 25: Target image; 26: Excitation force information; 61: Image candidate; 100: Information processing unit; 110: Position and orientation determination unit; 120: Area calculation unit; 130: Contact position calculation unit; 140: Determination unit; 160: Storage unit; 161: Target object database; 200: Glove; 201: Photographing device; 202: Contact force sensor; 203: Sensor data acquisition module; 204: Marker; 205: Motion sensor; 206: Fixing tool; 261: Ring; 262: Rigid body; 300: Target object; 500: Fixation state determination system; 909: Electronic circuit; 910: Processor; 921: Working memory; 922: Auxiliary memory device; 930: Input interface; 940: Output interface; 950: Communication device.

Claims

[1] Fixation state determination system (500) comprising a glove (200) worn by an operator (10) to grasp a target object (300) and determining a fixation state of the target object (300), wherein the fixation state determination system (500) comprises: a photographic device (201) which is placed at a position of the glove (200) corresponding to a section between a pair of fingers which grasp the target object (300) in order to photograph the target object (300) grasped by the pair of fingers as a target image (25); a position and orientation determination unit (110) to determine the position and orientation of the target object (300) using the target image (25); an area calculation unit (120) to analyze a movement of a fingertip of the operator (10) using the target image (25), and to calculate a fingertip movement area which is a movement area of ​​a position of a fingertip of each finger of the operator (10); a contact position calculation unit (130) to calculate a contact position of the fingertip on the target object (300) based on the position and orientation of the target object (300) and the fingertip movement range; and a determination unit (140) to determine the fixation state of the target object (300) on the basis of a displacement of the target object (300) and excitation force information (26), which is information about an excitation force at the contact position of the fingertip for the target object (300). [2] Fixation state determination system (500) according to claim 1, wherein the glove (200) has a marker (204) at a position corresponding to the inside of a proximal finger joint of each finger of the operator (10), wherein the photographic device (201) photographs the target image (25) which contains the position of the marker (204) at the time when the glove (200) grasps the target object (300), and wherein the range calculation unit (120) calculates an angle of the proximal phalanx of each finger of the operator (10) using the position of the marker (204) in the target image (25), and calculates the fingertip movement range based on the angle of the proximal phalanx of each finger of the operator (10). [3] Fixation state determination system (500) according to claim 2, wherein the glove (200) includes a motion sensor (205) on an outside of the fingertip of each finger of the operator (10) to detect the movement of the fingertip, and wherein the area calculation unit (120) calculates the fingertip movement area based on the angle of the proximal finger joint of each finger of the operator (10) and the movement of the fingertip detected by the motion sensor (205). [4] Fixation state determination system (500) according to any one of claims 1 to 3, comprising: a target object database (161) in which individual images of a multitude of combinations of the position and location of the target object (300) are accumulated as image candidates (61), wherein the position and location determination unit (110) compares the target image (25) with the image candidates (61) accumulated in the target object database (161) and thereby determines the position and location of the target object (300). [5] Fixation state determination system (500) according to one of claims 1 to 4, wherein the glove (200) includes a contact force sensor (202) in a position corresponding to the inside of the fingertip of the operator (10), and wherein the determining unit (140) acquires the excitation force at the contact position of the fingertip for the target object (300) from the contact force sensor (202), and acquires a displacement of the target object (300) from the target image (25). [6] Glove (200) provided for a fixation state determination system (500) comprising the glove (200) worn by an operator (10) to grasp a target object (300) and an information processing device (100) for communication with the glove (200), wherein the fixation state determination system (500) determines a fixation state of the target object (300), where the glove comprises (200): a photographic device (201) positioned at a location corresponding to a section between a pair of fingers grasping the target object (300) in order to photograph the target object (300) grasped by the pair of fingers as a target image (25); and a communication device (950) to transmit the target image (25) to the information processing device (100), wherein the information processing device (100) determines a position and orientation of the target object (300) using the target image (25); calculates a fingertip movement range, which is a movement range of a position of a fingertip of each finger of the operator (10), using the target image (25); and A contact position of the fingertip on the target object (300) is calculated based on the position and orientation of the target object (300) and the fingertip movement range. [7] Glove (200) according to claim 6, comprising: a marker (204) at a position corresponding to the inside of a proximal finger joint of each finger of the operator (10), wherein the photographic device (201) photographs the target image (25) which contains the position of the marker (204) at the time when the glove (200) grasps the target object (300), and wherein the information processing device (100) calculates an angle of the proximal finger joint of each finger of the operator (10) using the position of the marker (204) in the target image (25), and calculates the fingertip movement range based on the angle of the proximal finger joint of each finger of the operator (10). [8] Glove (200) according to claim 7, comprising: a motion sensor (205) on the outside of the fingertip of each finger of the operator (10) to detect movement of the fingertip, wherein the information processing device (100) calculates the fingertip movement range on the basis of the angle of the proximal finger joint of each finger of the operator (10) and the movement of the fingertip detected by the motion sensor (205). [9] Glove(200) according to claim 7 or 8, comprising: a fixing tool (206) attached to each finger of the pair of fingers of the glove (200) to fix the photographic device (201). [10] Glove (200) according to claim 9, wherein the fixing tool (206) comprises a ring (261) attached to each finger of the pair of fingers of the glove (200) and a rigid body (262) connecting the ring (261) to the photographing device (201). [11] Glove (200) according to one of claims 7 to 10, wherein the marker (204) has a protruding shape. [12] Glove (200) according to any one of claims 6 to 11, wherein the photographing device (201) comprises a plurality of photographing devices. [13] Glove (200) according to any one of claims 6 to 12, comprising: a contact force sensor (202) at a position corresponding to the inside of the fingertip of the operator (10), wherein the information processing device (100) acquires information about an excitation force at a contact position of the fingertip for the target object (300) from the contact force sensor (202) as an excitation force information (26) and acquires a displacement of the target object (300) from the target image (25). [14] Information processing device (100) provided for a fixation state determination system (500) comprising a glove (200) worn by an operator (10) to grasp a target object (300) and an information processing device (100) for communication with the glove (200), wherein the fixation state determination system (500) determines a fixation state of the target object (300), and wherein the information processing device (100) comprises: a position and orientation determination unit (110) for determining a position and orientation of the target object (300) using a target image (25) photographed by a photographing device (201) placed at a position of the glove (200) corresponding to a section between a pair of fingers grasping the target object (300) in order to photograph the target object (300) grasped by the pair of fingers as the target image (25); an area calculation unit (120) to analyze a movement of a fingertip of the operator (10) using the target image (25), and to calculate a fingertip movement area which is a movement area of ​​a position of a fingertip of each finger of the operator (10); a contact position calculation unit (130) to calculate a contact position of the fingertip on the target object (300) based on the position and orientation of the target object (300) and the fingertip movement range; and a determination unit (140) to determine the fixation state of the target object (300) on the basis of a displacement of the target object (300) and an excitation force of excitation at the contact position of the fingertip for the target object (300). [15] Fixation state determination method applied by a fixation state determination system (500) comprising a glove (200) worn by an operator (10) to grasp a target object (300) and determining a fixation state of the target object (300), wherein the fixation state determination method comprises: a photographic device (201) which is placed at a position of the glove (200) corresponding to a section between a pair of fingers which grasp the target object (300), which photographs the target object (300) grasped by the pair of fingers as a target image (25); by a computer, determining the position and location of the target object (300) using the target image (25); by the computer, analyzing a movement of a fingertip of the operator (10) using the target image (25), and calculating a fingertip movement range which is a movement range of a position of a fingertip of each finger of the operator (10); by the computer, calculating a contact position of the fingertip on the target object (300) based on the position and orientation of the target object (300) and the fingertip movement range; and by the computer, determining the fixation state of the target object (300) on the basis of a displacement of the target object (300) and an excitation force of excitation at the contact position of the fingertip for the target object (300). [16] Fixation state determination program applied by a fixation state determination system (500) comprising a glove (200) worn by an operator (10) to grasp a target object (300) and determining a fixation state of the target object (300), wherein the fixation state determination program causes a computer to execute: a position and orientation determination process of determining a position and orientation of the target object (300) using a target image (25) photographed by a photographing device (201) placed at a position of the glove (200) corresponding to a section between a pair of fingers grasping the target object (300) in order to photograph the target object (300) being grasped by the pair of fingers as the target image (25); a range calculation process of analyzing a movement of a fingertip of the operator (10) using the target image (25), and calculating a fingertip movement range which is a movement range of a position of a fingertip of each finger of the operator (10); a contact position calculation process of calculating a contact position of the fingertip on the target object (300) based on the position and orientation of the target object (300) and the fingertip movement range; and a determination process of determining the fixation state of the target object (300) on the basis of a displacement of the target object (300) and an excitation force of excitation at the contact position of the fingertip for the target object (300).

Images