Information processor, information processing system, information processing method and program

Abstract

To provide an information processor, an information processing system, an information processing method, and a program which can reduce errors of work of a worker when the worker remotely operates a robot arm.SOLUTION: In an information processor, a storage part stores object data of one or a plurality of articles, and hardness data of the article corresponding to the object data. An object data reception part receives the object data of a gripping object article of a robot arm operated by a worker from an object measuring device. A hardness estimation part estimates the hardness data of the gripping object article from the object data of the gripping object article, on the basis of the information stored in the storage part. A force tactile data generating part generates force tactile data indicating what force tactile sense is presented by the worker when the worker grips the gripping object article, on the basis of the estimated hardness data of the gripping object. The force tactile data transmitting part transmits the generated force tactile data to a force tactile presenting device, and allows the worker to present the force tactile sense based on the force tactile data.SELECTED DRAWING: Figure 1

Description

[Technical Field]

[0001] The present disclosure relates to an information processing device, an information processing system, an information processing method, and a program. [Background technology]

[0002] Patent Document 1 discloses a technology for controlling remote operation of a robot, which provides feedback information such as vibrations measured by a sensor attached to the robot's gripping mechanism (i.e., the robot arm) to the robot operator, thereby improving the accuracy of the work. [Prior art documents] [Patent documents]

[0003] [Patent Document 1] Japanese Patent Publication No. 2022-155623 Summary of the Invention [Problem to be solved by the invention]

[0004] The technology disclosed in Patent Document 1 has a problem in that when a worker remotely controls a robot arm, the worker makes operational mistakes. For example, such operational mistakes are mistakes such as the remotely controlled robot arm crushing or denting the object to be grasped, and are particularly likely to occur when the object to be grasped is soft (i.e., fruit or vegetables).

[0005] In consideration of such issues, the present disclosure aims to provide an information processing device, an information processing system, an information processing method, and a program that can reduce operator errors when remotely operating a robot arm. [Means for solving the problem]

[0006] The information processing device of the present disclosure includes: a storage unit that stores object data of one or more articles and hardness data of articles corresponding to each of the object data of the one or more articles in association with each other; an object data receiving unit that receives object data of a target object to be grasped, which is an object to be grasped by a robot arm operated by a worker, from the object measuring device; a hardness estimation unit that estimates hardness data of the object to be grasped from object data of the object to be grasped based on the information stored in the storage unit; a force-tactile data generation unit that generates force-tactile data indicating what force-tactile sensations should be presented to the worker when the worker operates the robot arm to grasp the object to be grasped, based on the estimated hardness data of the object to be grasped; and The device is equipped with a force-tactile data transmission unit that transmits the generated force-tactile data to a force-tactile presentation device and causes the force-tactile presentation device to present a force-tactile sensation based on the generated force-tactile data to the worker.

[0007] The information processing system of the present disclosure includes: a storage unit that stores object data of one or more articles and hardness data of articles corresponding to each of the object data of the one or more articles in association with each other; an object measurement unit that measures object data of a target object to be grasped, which is an object to be grasped by a robot arm operated by a worker; a hardness estimation unit that estimates hardness data of the object to be grasped from object data of the object to be grasped based on the information stored in the storage unit; a force-tactile data generation unit that generates force-tactile data indicating what force-tactile sensations should be presented to the worker when the worker operates the robot arm to grasp the object to be grasped, based on the estimated hardness data of the object to be grasped; and and a haptic sensation presentation unit that presents a haptic sensation to the worker based on the generated haptic sensation data.

[0008] The information processing method of the present disclosure includes: The computer storing object data of one or more articles and hardness data of articles corresponding to each of the object data of the one or more articles in association with each other; receiving object data of a target object to be grasped, which is an object to be grasped by a robot arm operated by a worker, from the object measuring device; Estimating hardness data of the object to be grasped from object data of the object to be grasped based on the stored information; generating force-tactile data indicating what force-tactile sensations should be presented to the worker when the worker operates the robot arm to grasp the object to be grasped, based on the estimated hardness data of the object to be grasped; The generated haptic data is transmitted to a haptic presentation device, and a haptic sensation based on the generated haptic data is presented to the worker by the haptic presentation device.

[0009] The program of the present disclosure is storing object data of one or more articles and hardness data of articles corresponding to each of the object data of the one or more articles in association with each other; receiving object data of a target object to be grasped, which is an object to be grasped by a robot arm operated by a worker, from the object measuring device; Estimating hardness data of the object to be grasped from object data of the object to be grasped based on the stored information; generating force-tactile data indicating what force-tactile sensations should be presented to the worker when the worker operates the robot arm to grasp the object to be grasped, based on the estimated hardness data of the object to be grasped; The generated haptic data is transmitted to a haptic presentation device, and a computer is caused to execute a process of causing the haptic presentation device to present to the worker a haptic sensation based on the generated haptic data. [Effects of the Invention]

[0010] The present disclosure provides an information processing device, an information processing system, an information processing method, and a program that can reduce operator errors when remotely operating a robot arm. [Brief explanation of the drawings]

[0011] [Figure 1] 1 is a block diagram showing an example of the configuration of an information processing device 10 according to the present embodiment. [Figure 2] 1 is a schematic diagram showing an example of the configuration of an information processing system 2 according to the present embodiment. [Figure 3] 1 is a block diagram showing an example of the configuration of an object measuring device 100, a haptic data generating device 200, and a haptic presentation device 300 of an information processing system 2 according to the present embodiment. FIG. [Figure 4] FIG. 10 is a diagram showing an example of the types of one or more articles stored in the memory unit 203 of the information processing system 2 according to this embodiment, the wavelength band data of the one or more articles, and the hardness data of the articles corresponding to each of the wavelength band data of the one or more articles. [Figure 5] 10 is a flowchart showing an example of the operation of the object measuring device 100 of the information processing system 2 according to the present embodiment. [Figure 6] 10 is a flowchart showing an example of the operation of the haptic data generating device 200 of the information processing system 2 according to the present embodiment. [Figure 7] 10 is a flowchart showing an example of the operation of the haptic sense presentation device 300 of the information processing system 2 according to the present embodiment. [Figure 8] 1 is a schematic diagram showing an example of the configuration of an information processing system R2, which is a comparative example of the information processing system 2 according to the present embodiment. [Figure 9] FIG. 1 is a block diagram showing an example of the configuration of a computer 1000 according to the present embodiment. DETAILED DESCRIPTION OF THE INVENTION

[0012] Hereinafter, embodiments of the present disclosure will be described in detail with reference to the drawings. In each drawing, the same or corresponding elements are designated by the same reference numerals, and for clarity of explanation, duplicate explanations will be omitted as necessary.

[0013] (First embodiment) First, an information processing device 10 according to the first embodiment will be described with reference to FIG. 1 is a diagram showing an example of the configuration of an information processing device 10 according to this embodiment. As shown in Fig. 1, the information processing device 10 includes a storage unit 11, an object data receiving unit 12, a hardness estimating unit 13, a haptic data generating unit 14, and a haptic data transmitting unit 15.

[0014] The memory unit 11 stores object data for one or more objects and hardness data for the objects corresponding to each of the object data for the one or more objects in association with each other. The object data receiving unit 12 receives object data for a grasp target object, which is an object to be grasped by a robot arm operated by a worker, from the object measuring device. The hardness estimation unit 13 estimates the hardness data of the grasp target object from the object data of the grasp target object based on the information stored in the memory unit 12. The haptic data generating unit 14 generates haptic data indicating what kind of haptic sensations should be presented to the worker when the worker operates the robot arm to grasp the grasp target object. The haptic data transmitting unit 15 transmits the generated haptic data to a haptic presentation device, and causes the haptic presentation device to present a haptic sensation to the worker based on the generated haptic data.

[0015] The information processing device 10 according to the first embodiment considers the hardness data of an object to be grasped when the worker operates a robot arm to grasp the object to be grasped, and causes the force-tactile presentation device to present the force-tactile sensation to the worker. Therefore, when the worker remotely controls the robot arm, it is possible to reduce the occurrence of operator errors, such as the robot arm crushing or denting the object to be grasped. For example, when the object to be grasped is a fruit or vegetable with a low hardness, the information processing device 10 can reduce the occurrence of a situation in which the fruit or vegetable to be grasped is crushed during sorting of the fruit or vegetable from a remote location, soiling the conveyor, and losing its value.

[0016] (Second embodiment) Next, the configuration of an information processing system 2 according to the second embodiment will be described with reference to Fig. 2 to Fig. 4. Fig. 2 is a schematic diagram showing an example of the configuration of the information processing system 2 according to the embodiment. Fig. 3 is a block diagram showing an example of the configuration of the information processing system 2 according to the embodiment.

[0017] The information processing system 2 is a system made up of one or more devices, and is a system that embodies the configuration of the information processing device 10 according to the first embodiment. The information processing device 10 corresponds to a haptic data generating device 200, which will be described later.

[0018] As shown in Figures 2 and 3, the information processing system 2 includes an object measuring device 100, a force-tactile data generating device 200, a force-tactile presentation device 300, a worker monitor 400, a worker vision camera 500, a remote control actuator 600, and a robot arm 700.

[0019] In the information processing system 2, the robot arm 700 uses a gripping mechanism to grip an object (X in FIG. 2). Here, the object to be gripped by the robot arm 700 is referred to as a gripping target object. The object is, for example, a fruit or vegetable such as an apple. The remote control actuator 600 receives an operation from an operator (Y in FIG. 2) and remotely controls the robot arm 700. Thus, the operator can operate the robot arm 700 using the remote control actuator 600 to cause the robot arm 700 to grip the gripping target object. In addition, the operator monitor 400 displays an image captured by the operator vision camera 500. The operator vision camera 500 captures an image of the vicinity of the gripping target object of the robot arm 700. The operator can operate the robot arm 700 while viewing the image displayed on the operator monitor 400.

[0020] The object measuring device 100 is a hyperspectral camera and is installed so that it can measure an object to be grasped by the robot arm 700. The object measuring device 100 communicates with the force-tactile data generating device 200. The object measuring device 100 includes an object measuring unit 101 and an object data transmitting unit 102.

[0021] The object measurement unit 101 measures wavelength band data indicating the wavelength band of the surface of the object to be grasped by capturing an image using a hyperspectral camera. The wavelength band data of the surface of the object to be grasped is included in object data indicating the characteristics of the object to be grasped.

[0022] The object data transmission unit 102 transmits the wavelength band data of the object to be grasped measured by the object measurement unit 101 to the force-tactile data generation device 200.

[0023] The haptic data generating device 200 is, for example, a server. The haptic data generating device 200 communicates with the object measuring device 100, the haptic presentation device 300, the worker monitor 400, and the worker vision camera 500. The haptic data generating device 200 includes an object data receiving unit 201, an object data accumulation unit 202, a memory unit 203, a hardness estimation unit 204, a haptic data generating unit 205, a time adjustment unit 206, a haptic data transmitting unit 207, and a device operation acquisition unit 208.

[0024] The haptic data generating device 200 corresponds to the information processing device 10 according to the first embodiment. The object data receiving unit 201, memory unit 203, hardness estimation unit 204, haptic data generation unit 205, and haptic data transmission unit 207 of the haptic data generating device 200 correspond to the object data receiving unit 12, memory unit 11, hardness estimation unit 13, haptic data generation unit 14, and haptic data transmission unit 15 of the information processing device 10, respectively, in this order.

[0025] The object data receiving unit 201 receives wavelength band data of the object to be grasped from the object measuring device 100. The wavelength band data is a specific example of object data in the first embodiment. The object data storage unit 202 stores the wavelength band data of the object to be grasped received by the object data receiving unit 201.

[0026] The storage unit 203 is a database that stores wavelength band data of one or more articles and hardness data of articles corresponding to the wavelength band data of the one or more articles in association with each other. The wavelength band data of each article is data measured by a hyperspectral camera in a learning environment and stored. The hardness data of each article is data measured by a hardness meter in a learning environment and stored. The information processing system 2 may also include a learning device (not shown) for learning the information stored in the storage unit 203.

[0027] 4 is a diagram showing an example of one or more types of items (item types), wavelength band data for the one or more items, and item hardness data corresponding to each of the wavelength band data for the one or more items stored in the storage unit 203 of the information processing system 2. The item hardness data includes at least one of estimated moisture content (%) and softness (P). This diagram shows three types of wavelength band data for the item type of apple, and hardness data (estimated moisture content (%) and softness (P)) corresponding to each of the three types of wavelength band data. The higher the softness (P), the lower the hardness of the item, i.e., the softer it is determined to be. The lower the estimated moisture content (%), the harder the item is determined to be.

[0028] The hardness estimation unit 204 estimates hardness data of the object to be grasped from the wavelength band data of the object to be grasped stored by the object data storage unit 202 based on the data stored in the storage unit 203 .

[0029] The force-tactile data generation unit 205 generates force-tactile data indicating what kind of force-tactile sensations should be presented to the worker when the worker operates the robot arm 700 to grasp the object to be grasped, based on the hardness data of the object to be grasped estimated by the hardness estimation unit 204. For example, the force-tactile data generation unit 205 generates force-tactile data that presents the worker with a force-tactile sensation such that the lower the hardness of the object to be grasped, the smaller the repulsion when attempting to close the robot arm 700. On the other hand, the force-tactile data generation unit 205 generates force-tactile data that presents the worker with a force-tactile sensation such that the higher the hardness of the object to be grasped, the greater the repulsion when attempting to close the robot arm 700. Here, the force-tactile data generation unit 205 generates force-tactile data that gradually adjusts the repulsion when attempting to close the robot arm 700 depending on the hardness of the object to be grasped. Additionally, the haptic data generation unit 205 may generate haptic data based on the output range of the haptic presentation device 300, the worker's perception, etc., in order to present a haptic sensation that is easily recognized by the worker.

[0030] The time adjustment unit 206 adjusts the timing so that the force and tactile sensations are presented to the worker at an appropriate timing before the worker operates the robot arm 700 to grasp the object to be grasped. Specifically, the time adjustment unit 206 adjusts the timing of transmitting the force and tactile data to the force and tactile presentation device 300 based on data such as the image from the worker vision camera 500 that enables the worker to visually recognize the object to be grasped, the worker's working speed, and communication delays involved in transmitting the force and tactile data to the force and tactile presentation device 300.

[0031] The haptic data transmission unit 207 transmits the haptic data generated by the haptic data generation unit 205 and whose transmission timing has been adjusted by the time adjustment unit 206 to the haptic presentation device 300. In doing so, the haptic data transmission unit 207 causes the haptic presentation device 300 to present a haptic sensation to the worker based on the generated haptic data. In this case, the haptic data transmission unit 207 causes the haptic presentation device 300 to present a haptic sensation to the worker at an appropriate timing before the worker operates the robot arm 700 to grasp the object to be grasped. The equipment operation acquisition unit 208 acquires feedback information on the operation of the robot arm 700, such as the amount of force applied by the worker, from the remote control actuator 600. In this case, the force-tactile data generation unit 205 generates force-tactile data in consideration of the feedback information on the operation of the robot arm 700 acquired by the equipment operation acquisition unit 208.

[0032] The haptic feedback device 300 is a glove-type haptic device worn by a worker performing a gripping operation, and provides the worker with a sense of force (haptic feedback). The haptic feedback device 300 includes a haptic data receiving unit 301 and a haptic feedback unit 302.

[0033] The haptic data receiving unit 301 receives haptic data from the haptic data generating device 200 . The haptic sense presentation unit 302 presents a haptic sense to the worker based on the haptic data received by the haptic data receiving unit 301. For example, the haptic sense presentation unit 302 presents a haptic sense to the worker by vibration, pressure, or the like.

[0034] Next, the operation of the information processing system 2 according to the second embodiment will be described with reference to FIGS.

[0035] FIG. 5 is a flowchart showing an example of the operation of the object measuring device 100 of the information processing system 2 according to this embodiment. As shown in FIG. 5, first, in step S101, the object measuring device 100 is started up by the operator.

[0036] Next, in step S102, the object measuring unit 101 of the object measuring device 100 measures waveband data indicating the waveband of the surface of the object to be grasped by capturing an image with the hyperspectral camera. Next, in step S103, the object data transmitting unit 102 transmits the wavelength band data of the object to be grasped to the haptic data generating device 200.

[0037] FIG. 6 is a flowchart showing an example of the operation of the haptic data generating device 200 of the information processing system 2 according to this embodiment.

[0038] As shown in FIG. 6, in step S201, the object data receiving unit 201 of the force and tactile data generation device 200 receives the wavelength band data of the object to be grasped from the object measuring device 100. Next, in step S202, the object data storage unit 202 stores the received wavelength band data of the object to be grasped.

[0039] Next, in step S203, the hardness estimation unit 204 determines whether the wavelength band data of the object to be grasped matches the wavelength band data of one or more objects pre-stored in the storage unit 203. Here, the wavelength band data of one or more objects stored in the storage unit 203 is linked to the hardness data of the corresponding object. If it is determined that the wavelength band data of the object to be grasped does not match the wavelength band data of one or more objects pre-stored in the storage unit 203 (NO in step S203), the process proceeds to step S204. On the other hand, if it is determined that the wavelength band data of the object to be grasped matches the wavelength band data of one or more objects pre-stored in the storage unit 203 (YES in step S203), the process proceeds to step S205.

[0040] In step S204, the hardness estimation unit 204 estimates hardness data of the object to be grasped, associates the estimated hardness data with the wavelength band data of the object to be grasped, and stores the associated data in the storage unit 203. Specifically, the hardness estimation unit 204 acquires wavelength band data of multiple objects before and after the wavelength band data of the object to be grasped, and hardness data corresponding to each of the wavelength band data of the multiple objects before and after, from the storage unit 203. The hardness estimation unit 204 estimates the hardness data of the object to be grasped from the multiple wavelength band data before and after and the transition of the hardness data corresponding to them. The hardness estimation unit 204 associates the estimated hardness data of the object to be grasped with the wavelength band data of the object to be grasped, and stores the associated data in the storage unit 203. Then, the process proceeds to step S205.

[0041] In step S205, the hardness estimation unit 204 acquires from the memory unit 203 hardness data associated with wavelength band data that matches the wavelength band data of the object to be grasped, and estimates the acquired hardness data as the hardness data of the object to be grasped.

[0042] In step S206, the force-tactile data generation unit 205 generates force-tactile data indicating what kind of force-tactile sensations should be presented to the worker when the robot arm 700 grasps the object to be grasped by the worker's operation, based on the estimated hardness data of the object to be grasped. For example, the force-tactile data generation unit 205 generates force-tactile data that presents the worker with a force-tactile sensation such that the lower the hardness of the object to be grasped, the smaller the repulsion when attempting to close the robot arm 700. On the other hand, the force-tactile data generation unit 205 generates force-tactile data that presents the worker with a force-tactile sensation such that the higher the hardness of the object to be grasped, the greater the repulsion when attempting to close the robot arm 700. In addition, in step S206, the force-tactile data generation unit 205 may generate force-tactile data based on the output range of the force-tactile presentation device 300, the worker's perception, etc., in order to present a force-tactile sensation that is easy for the worker to recognize.

[0043] In step S207, the time adjustment unit 206 adjusts the timing of transmitting the haptic data to the haptic presentation device 300 based on the video data acquired by the worker vision camera 500. For example, the time adjustment unit 206 adjusts the timing so that the haptic data is transmitted to the haptic presentation device 300 at an appropriate timing before the worker operates the robot arm 700 to grasp the object to be grasped. Note that the time adjustment unit 206 may adjust the timing of transmitting the haptic data to the haptic presentation device 300 based on data such as the worker's working speed and communication delays involved in transmitting the haptic data to the haptic presentation device 300. Next, in step S208, the haptic data transmission unit 207 transmits to the haptic presentation device 300 the haptic data corresponding to the object to be grasped, the transmission timing of which has been adjusted.

[0044] FIG. 7 is a flowchart showing an example of the operation of the haptic sense presentation device 300 of the information processing system 2 according to this embodiment.

[0045] As shown in FIG. 7, first, in step S301, the haptic sense presentation device 300 is activated by the operator. Next, in step S302, the haptic data receiving unit 301 of the haptic presentation device 300 receives the haptic data corresponding to the object to be grasped from the haptic data generating device 200. Next, in step S303, the haptic sense presentation unit 302 presents a haptic sense to the worker based on the received haptic sense data corresponding to the object to be grasped. For example, the haptic sense presentation unit 302 presents a haptic sense to the worker by vibration, pressure, or the like.

[0046] Next, a comparison between the information processing system 2 and an information processing system R2, which is a comparative example of the information processing system 2 according to this embodiment, will be described with reference to FIGS.

[0047] 8 is a schematic diagram showing an example of the configuration of an information processing system R2, which is a comparative example of the information processing system 2 according to this embodiment. As shown in FIG. 8, the information processing system R2, which is the comparative example, includes an operator monitor 400, an operator vision camera 500, a remote control actuator 600, and a robot arm 700, similar to the information processing system 2. However, errors can occur when operators perform work remotely. For example, when sorting items to be grasped from a remote location (e.g., fruits and vegetables), a situation can occur in which the items to be grasped are crushed, soiling the conveyor and causing the value of the items to be grasped to be lost.

[0048] On the other hand, as shown in FIG. 2 , an information processing system 2 according to this embodiment further includes an object measuring device 100, a haptic data generating device 200, and a haptic presentation device 300, compared to the information processing system R2. When a worker operates a robot arm to grasp an object to be grasped, the information processing system 2 presents haptic sensations to the worker, taking into account the hardness data of the object to be grasped. For example, when the object to be grasped is soft, the information processing system 2 minimizes the load on the hand operating the remote-controlled actuator 600 for operating the robot arm 700. By doing so, the information processing system 2 can reduce the occurrence of worker errors, such as the robot arm 700 crushing or denting the object to be grasped. For example, when the object to be grasped is a fruit or vegetable with a low hardness, the information processing system 2 can reduce the occurrence of a situation in which the fruit or vegetable is crushed during sorting, soiling the conveyor, and losing its value.

[0049] <Hardware configuration> Each of the devices (e.g., object measuring device 100, force-tactile data generating device 200, and force-tactile presentation device 300) included in the information processing device 10 and information processing system 2 in the above-described embodiment is configured by hardware or software, or both, and may be configured by one piece of hardware or software, or by multiple pieces of hardware or software.

[0050] Specifically, each of the devices (e.g., the object measuring device 100, the haptic data generating device 200, and the haptic presentation device 300) included in the information processing device 10 and the information processing system 2 in the above-described embodiment is configured by a computer including a processor 1001 and a memory 1002. FIG. 9 is a block diagram showing an example of the configuration of the computer 1000 according to this embodiment. As shown in FIG. 9, the processor 1001 may be, for example, a microprocessor, an MPU (Micro Processing Unit), or a CPU (Central Processing Unit). The processor may include multiple processors. The memory 1002 is configured by a combination of volatile memory and non-volatile memory. The memory 1002 may include storage located away from the processor. In this case, the processor 1001 may access the memory via an I / O interface (not shown). The processor 1001 executes one or more programs including instructions for causing the computer to execute the algorithms described using the drawings.

[0051] The program includes instructions (or software code) that, when loaded into a computer, cause the computer to perform one or more functions described in the embodiments. The program may be stored in a non-transitory computer-readable medium or a tangible storage medium. By way of example and not limitation, computer-readable media or tangible storage media include random-access memory (RAM), read-only memory (ROM), flash memory, solid-state drive (SSD) or other memory technologies, CD-ROM, digital versatile disc (DVD), Blu-ray® disc or other optical disk storage, magnetic cassette, magnetic tape, magnetic disk storage or other magnetic storage devices. The program may also be transmitted on a transitory computer-readable medium or communication medium. By way of example and not limitation, transitory computer-readable media or communication media include electrical, optical, acoustic, or other forms of propagated signals.

[0052] Although the present disclosure has been described above with reference to the embodiments, the present disclosure is not limited to the above-described embodiments. Various modifications that can be understood by those skilled in the art can be made to the configuration and details of the present disclosure within the scope of the present disclosure. Furthermore, each embodiment can be combined with other embodiments as appropriate.

[0053] Each drawing is merely an example for describing one or more embodiments. Each drawing may relate not only to one particular embodiment, but also to one or more other embodiments. As will be understood by those skilled in the art, various features or steps described with reference to any one drawing can be combined with features or steps shown in one or more other drawings to create, for example, an embodiment not explicitly shown or described. Not all features or steps shown in any one drawing are necessary to describe an exemplary embodiment, and some features or steps may be omitted. The order of steps described in any drawing may be changed as appropriate.

[0054] The present invention is not limited to the above-described embodiment, and can be modified as appropriate within the scope of the invention.

[0055] In the information processing system 2, the "item" is not limited to fruit and vegetables such as apples, but may also be processed foods, wood whose hardness changes depending on the state of the item, or human skin, muscle, fat or organs. The information processing system 2 is not limited to being used for work by a worker in real space, but may also be used for work by a worker in virtual reality.

[0056] In information processing system 2, object measuring device 100 is not limited to a hyperspectral camera, but may be any of various sensors such as an RGB camera, a 3D sensor, a weighing scale, or a sound collector. The data measured by object measuring device 100 is not limited to wavelength band data of the object's surface, but may be object data that indicates the characteristics of the object. For example, the object data includes any of image analysis data analyzed from an image including the object captured by an RGB camera, unevenness data of the object's surface measured by a 3D sensor, weight data of the object measured by a weighing scale, and ultrasonic data of the object's surface measured by a sound collector. Here, in information processing system 2, when the type of object data measured by object measuring device 100 is changed, the type of object data stored in memory unit 203 must also be changed.

[0057] In the information processing system 2, there may be multiple object measuring devices 100 instead of one device, and the object to be grasped may be measured from multiple directions. The object measuring device 100 may not be a single device, but may be a combination of multiple types of object measuring devices 100.

[0058] In the information processing system 2, the haptic data generating unit 205 may excessively vary the strength of the haptic sensation in the generated haptic data in order to represent the difference in the estimated softness of the object to be grasped.

[0059] In the information processing system 2, the object data storage unit 202 of the haptic data generation device 200 may store time series data indicating the time series of when the robot arm 700 grasps the object to be grasped (e.g., the time series of when the wavelength band data of the object to be grasped is received) in consideration of the case where there are multiple objects to be grasped. In this case, subsequent processing by the hardness estimation unit 204 and the like may be performed in accordance with the time series data. For example, as shown in the explanation of the operation of the haptic data generation device 200 in FIG. 6, in step S202, the object data storage unit 202 may store the time series data in association with the wavelength band data of the object to be grasped. The haptic data generation device 200 performs the processing from step S203 onwards in accordance with the time series of the object to be grasped.

[0060] In the information processing system 2, the data used to generate the force-tactile data corresponding to the object to be grasped is not limited to the hardness data of the object to be grasped, but may also be the coefficient of friction of the surface of the object to be grasped (i.e., the slipperiness of the surface of the object to be grasped). Specifically, the storage unit 203 of the force-tactile data generating device 200 pre-stores object data of one or more objects associated with the coefficient of friction of the surface of the one or more objects. The hardness estimation unit 204 estimates the coefficient of friction of the surface of the object to be grasped from the object data of the object to be grasped based on the information pre-stored in the storage unit 203. The force-tactile data generating unit 205 generates force-tactile data indicating the type of force-tactile sensation that will be presented to the operator when the robot arm 700 grasps the object to be grasped by operating the operator, based on the estimated coefficient of friction of the surface of the object to be grasped.

[0061] The haptic sensation is, for example, a repulsion against the worker. The stronger the repulsion against the worker, the weaker the gripping force of the robot arm 700 operated by the worker. The degree of repulsion is adjusted so that the gripping force of the robot arm 700 falls within a predetermined range when the coefficient of friction of the surface of the object to be grasped is lower than a predetermined threshold (i.e., when the surface of the object to be grasped is slippery). The reason for setting this range is that when the surface of the object to be grasped is slippery, if the gripping force of the robot arm 700 is too weak, the robot arm 700 may not be able to grasp the object to be grasped. On the other hand, if the gripping force of the robot arm 700 is too strong, slippage between the robot arm 700 and the object to be grasped becomes more likely, and the robot arm 700 may not be able to grasp the object to be grasped.

[0062] However, simply weakening the repulsion against the worker may not be enough to control the gripping force of the robot arm 700 within a specified range. For example, the equipment operation acquisition unit 208 acquires feedback information about the operation of the robot arm 700, such as the amount of force applied by the worker, from the remote control actuator 600. The force and haptic data generation unit 205 may then control the degree of repulsion in consideration of the feedback information about the operation.

[0063] Alternatively, both the hardness data of the object to be grasped and the friction coefficient of the surface of the object to be grasped may be used. Specifically, the memory unit 203 of the haptic data generating device 200 pre-stores object data of one or more objects, hardness data of the one or more objects, and the friction coefficient of the surface of the one or more objects, in association with each other. The hardness estimation unit 204 estimates the hardness data of the object to be grasped and the friction coefficient of the surface of the object to be grasped from the object data of the object to be grasped, based on the information pre-stored in the memory unit 203. The haptic data generating unit 205 generates haptic data indicating what kind of haptic sensations will be presented to the worker when the robot arm 700 grasps the object to be grasped by the worker's operation, based on the estimated hardness data of the object to be grasped and the friction coefficient of the surface of the object to be grasped.

[0064] A part or all of the above-described embodiments can be described as, but not limited to, the following supplementary notes. (Appendix 1) a storage unit that stores object data of one or more articles and hardness data of articles corresponding to each of the object data of the one or more articles in association with each other; an object data receiving unit that receives object data of a target object to be grasped, which is an object to be grasped by a robot arm operated by a worker, from the object measuring device; a hardness estimation unit that estimates hardness data of the object to be grasped from object data of the object to be grasped based on the information stored in the storage unit; a force-tactile data generation unit that generates force-tactile data indicating what force-tactile sensations should be presented to the worker when the worker operates the robot arm to grasp the object to be grasped, based on the estimated hardness data of the object to be grasped; and a haptic data transmission unit that transmits the generated haptic data to a haptic presentation device and causes the haptic presentation device to present to the worker a haptic sensation based on the generated haptic data. Information processing device. (Appendix 2) The data includes any of wavelength band data of the surface of the item measured by a hyperspectral camera, image analysis data analyzed from an image including the item photographed by an RGB camera, unevenness data of the surface of the item measured by a 3D sensor, weight data of the item measured by a weighing scale, and ultrasonic data of the surface of the item measured by a sound collector. 10. The information processing device according to claim 1. (Appendix 3) The article comprises: It is either fruit and vegetables, processed foods, wood, skin, muscle, fat, or organs. 10. The information processing device according to claim 1. (Appendix 4) The haptic data generation unit Generate haptic data based on the estimated hardness data of the object to be grasped, as well as on at least one of the output range of the haptic presentation device and the worker's perception of the haptic sensation. 10. The information processing device according to claim 1. (Appendix 5) The storage unit storing the object data of the one or more articles and the friction coefficients of the surfaces of the articles corresponding to the object data of the one or more articles in association with each other; The hardness estimation unit estimating a coefficient of friction of the surface of the object to be grasped from object data of the object to be grasped based on the stored information; The haptic data generation unit Generate force-tactile data corresponding to the object to be grasped based on the estimated hardness data of the object to be grasped and the friction coefficient of the surface of the object to be grasped. 10. The information processing device according to claim 1. (Appendix 6) The present invention further includes a time adjustment unit that adjusts the timing of transmission of the generated haptic data to the haptic presentation device so that the haptic sensation can be presented to the worker at an appropriate timing before the worker grasps the object to be grasped. 10. The information processing device according to claim 1. (Appendix 7) a storage unit that stores object data of one or more articles and hardness data of articles corresponding to each of the object data of the one or more articles in association with each other; an object measurement unit that measures object data of a target object to be grasped, which is an object to be grasped by a robot arm operated by a worker; a hardness estimation unit that estimates hardness data of the object to be grasped from object data of the object to be grasped based on the information stored in the storage unit; a force-tactile data generation unit that generates force-tactile data indicating what force-tactile sensations should be presented to the worker when the worker operates the robot arm to grasp the object to be grasped, based on the estimated hardness data of the object to be grasped; and a haptic sensation presentation unit that presents a haptic sensation to the worker based on the generated haptic sensation data. Information processing system. (Appendix 8) The object data is The data includes any of wavelength band data of the surface of the item measured by a hyperspectral camera, image analysis data analyzed from an image including the item photographed by an RGB camera, unevenness data of the surface of the item measured by a 3D sensor, weight data of the item measured by a weighing scale, and ultrasonic data of the surface of the item measured by a sound collector. 8. The information processing system of claim 7. (Appendix 9) The article comprises: It is either fruit and vegetables, processed foods, wood, skin, muscle, fat, or organs. 8. The information processing system of claim 7. (Appendix 10) The haptic data generation unit Generate haptic data based on the estimated hardness data of the object to be grasped, as well as on at least one of the output range of the haptic presentation device and the worker's perception of the haptic sensation. 8. The information processing system of claim 7. (Appendix 11) The storage unit storing the object data of the one or more articles and the friction coefficients of the surfaces of the articles corresponding to the object data of the one or more articles in association with each other; The hardness estimation unit estimating a coefficient of friction of the surface of the object to be grasped from object data of the object to be grasped based on the stored information; The haptic data generation unit Generate force-tactile data corresponding to the object to be grasped based on the estimated hardness data of the object to be grasped and the friction coefficient of the surface of the object to be grasped. 8. The information processing system of claim 7. (Appendix 12) The present invention further includes a time adjustment unit that adjusts the timing of transmission of the generated haptic data to the haptic presentation device so that the haptic sensation can be presented to the worker at an appropriate timing before the worker grasps the object to be grasped. 8. The information processing system of claim 7. (Appendix 13) The computer storing object data of one or more articles and hardness data of articles corresponding to each of the object data of the one or more articles in association with each other; receiving object data of a target object to be grasped, which is an object to be grasped by a robot arm operated by a worker, from the object measuring device; Estimating hardness data of the object to be grasped from object data of the object to be grasped based on the stored information; generating force-tactile data indicating what force-tactile sensations should be presented to the worker when the worker operates the robot arm to grasp the object to be grasped, based on the estimated hardness data of the object to be grasped; The generated haptic data is transmitted to a haptic presentation device, and a haptic sensation based on the generated haptic data is presented to the worker by the haptic presentation device. Information processing methods. (Appendix 14) The object data is The data includes any of wavelength band data of the surface of the item measured by a hyperspectral camera, image analysis data analyzed from an image including the item photographed by an RGB camera, unevenness data of the surface of the item measured by a 3D sensor, weight data of the item measured by a weighing scale, and ultrasonic data of the surface of the item measured by a sound collector. 14. The information processing method according to claim 13. (Appendix 15) The article comprises: It is either fruit and vegetables, processed foods, wood, skin, muscle, fat, or organs. 14. The information processing method according to claim 13. (Appendix 16) Furthermore, in addition to the estimated hardness data of the object to be grasped, force-tactile data is generated based on at least one of the output range of the force-tactile presentation device and the worker's perception of the force sensation. 14. The information processing method according to claim 13. (Appendix 17) Furthermore, the object data of the one or more articles is associated with a friction coefficient of the surface of each of the articles corresponding to the object data of the one or more articles, and the associated data is stored; estimating a coefficient of friction of the surface of the object to be grasped from object data of the object to be grasped based on the stored information; Generate force-tactile data corresponding to the object to be grasped based on the estimated hardness data of the object to be grasped and the friction coefficient of the surface of the object to be grasped. 14. The information processing method according to claim 13. (Appendix 18) Furthermore, the timing of transmitting the generated haptic data to the haptic presentation device is adjusted so that the haptic sensation can be presented to the worker at an appropriate timing before the worker grasps the object to be grasped. 14. The information processing method according to claim 13. (Appendix 19) storing object data of one or more articles and hardness data of articles corresponding to each of the object data of the one or more articles in association with each other; receiving object data of a target object to be grasped, which is an object to be grasped by a robot arm operated by a worker, from the object measuring device; Estimating hardness data of the object to be grasped from object data of the object to be grasped based on the stored information; generating force-tactile data indicating what force-tactile sensations should be presented to the worker when the worker operates the robot arm to grasp the object to be grasped, based on the estimated hardness data of the object to be grasped; transmitting the generated haptic data to a haptic presentation device, and causing the computer to execute a process of presenting a haptic sensation based on the generated haptic data to the worker using the haptic presentation device; program. (Appendix 20) The object data is The data includes any of wavelength band data of the surface of the item measured by a hyperspectral camera, image analysis data analyzed from an image including the item photographed by an RGB camera, unevenness data of the surface of the item measured by a 3D sensor, weight data of the item measured by a weighing scale, and ultrasonic data of the surface of the item measured by a sound collector. 19. The program described in Appendix 19. (Appendix 21) The article comprises: It is either fruit and vegetables, processed foods, wood, skin, muscle, fat, or organs. 19. The program described in Appendix 19. (Appendix 22) and causing the computer to further execute a process of generating haptic data based on at least one of the output range of the haptic sense presentation device and the worker's perception of the haptic sensation in addition to the estimated hardness data of the object to be grasped. The program described in Appendix 19. (Appendix 23) storing the object data of the one or more articles and the friction coefficients of the surfaces of the articles corresponding to the object data of the one or more articles in association with each other; estimating a coefficient of friction of the surface of the object to be grasped from object data of the object to be grasped based on the stored information; and causing the computer to further execute a process of generating force-tactile data corresponding to the object to be grasped based on the estimated hardness data of the object to be grasped and the coefficient of friction of the surface of the object to be grasped. 19. The program described in Appendix 19. (Appendix 24) The computer is further caused to execute a process of adjusting a timing for transmitting the generated haptic data to the haptic presentation device so that the haptic sensation can be presented to the worker at an appropriate timing before the worker grasps the object to be grasped. 19. The program described in Appendix 19. [Explanation of symbols]

[0065] 2. Information Processing Systems 10. Information processing equipment 11 Object data acquisition unit 12 Hardness estimation section 13 Haptic data generation unit 14 Haptic data supply unit 100 Object measuring device 101 Object Measurement Department 102 Object data transmission unit 200 Force and tactile data generation device 201 Object data receiving unit 202 Object Data Storage Unit 203 Storage section 204 Hardness estimation section 205 Force and haptic data generation unit 206 Time Adjustment Unit 207 Force and haptic data transmission unit 208 Equipment operation acquisition section 300 Force tactile presentation device 301 Force and haptic data receiver 302 Force tactile presentation section 400 Operator monitor 500 Worker Vision Camera 600 Remote Control Actuator 700 Robot Arm 1000 computers 1001 processor 1002 memory

Claims

1. a storage unit that stores object data of one or more articles and hardness data of articles corresponding to each of the object data of the one or more articles in association with each other; an object data receiving unit that receives object data of a target object to be grasped, which is an object to be grasped by a robot arm operated by a worker, from the object measuring device; a hardness estimation unit that estimates hardness data of the object to be grasped from object data of the object to be grasped based on the information stored in the storage unit; a force-tactile data generation unit that generates force-tactile data indicating what force-tactile sensations should be presented to the worker when the worker operates the robot arm to grasp the object to be grasped, based on the estimated hardness data of the object to be grasped; and a haptic data transmission unit that transmits the generated haptic data to a haptic presentation device and causes the haptic presentation device to present to the worker a haptic sensation based on the generated haptic data. Information processing device.

2. The object data is The data includes any one of wavelength band data of the surface of the item measured by a hyperspectral camera, image analysis data analyzed from an image including the item photographed by an RGB camera, unevenness data of the surface of the item measured by a 3D sensor, weight data of the item measured by a weighing scale, and ultrasonic data of the surface of the item measured by a sound collector. The information processing device according to claim 1 .

3. The article comprises: It is either fruit and vegetables, processed foods, wood, skin, muscle, fat, or organs. The information processing device according to claim 1 .

4. The haptic data generation unit Generate haptic data based on the estimated hardness data of the object to be grasped, as well as on at least one of the output range of the haptic presentation device and the worker's perception of the haptic sensation. The information processing device according to claim 1 .

5. The storage unit storing object data of one or more articles and a friction coefficient of the surface of each of the articles corresponding to the object data of the one or more articles in association with each other; The hardness estimation unit estimating a coefficient of friction of the surface of the object to be grasped from object data of the object to be grasped based on the stored information; The haptic data generation unit Generate force-tactile data corresponding to the object to be grasped based on the estimated hardness data of the object to be grasped and the friction coefficient of the surface of the object to be grasped. The information processing device according to claim 1 .

6. The present invention further includes a time adjustment unit that adjusts the timing of transmission of the generated haptic data to the haptic presentation device so that the haptic sensation can be presented to the worker at an appropriate timing before the worker grasps the object to be grasped. The information processing device according to claim 1 .

7. a storage unit that stores object data of one or more articles and hardness data of articles corresponding to each of the object data of the one or more articles in association with each other; an object measurement unit that measures object data of a target object to be grasped, which is an object to be grasped by a robot arm operated by a worker; a hardness estimation unit that estimates hardness data of the object to be grasped from object data of the object to be grasped based on the information stored in the storage unit; a force-tactile data generation unit that generates force-tactile data indicating what force-tactile sensations should be presented to the worker when the worker operates the robot arm to grasp the object to be grasped, based on the estimated hardness data of the object to be grasped; and a haptic sensation presentation unit that presents a haptic sensation to the worker based on the generated haptic sensation data. Information processing system.

8. The object data is The data includes any one of wavelength band data of the surface of the item measured by a hyperspectral camera, image analysis data analyzed from an image including the item photographed by an RGB camera, unevenness data of the surface of the item measured by a 3D sensor, weight data of the item measured by a weighing scale, and ultrasonic data of the surface of the item measured by a sound collector. The information processing system according to claim 7 .

9. The computer storing object data of one or more articles and hardness data of articles corresponding to each of the object data of the one or more articles in association with each other; receiving object data of a target object to be grasped, which is an object to be grasped by a robot arm operated by a worker, from the object measuring device; Estimating hardness data of the object to be grasped from object data of the object to be grasped based on the stored information; generating force-tactile data indicating what force-tactile sensations should be presented to the worker when the worker operates the robot arm to grasp the object to be grasped, based on the estimated hardness data of the object to be grasped; The generated haptic data is transmitted to a haptic presentation device, and a haptic sensation based on the generated haptic data is presented to the worker by the haptic presentation device. Information processing methods.

10. storing object data of one or more articles and hardness data of articles corresponding to each of the object data of the one or more articles in association with each other; receiving object data of a target object to be grasped, which is an object to be grasped by a robot arm operated by a worker, from the object measuring device; Estimating hardness data of the object to be grasped from object data of the object to be grasped based on the stored information; generating force-tactile data indicating what force-tactile sensations should be presented to the worker when the worker operates the robot arm to grasp the object to be grasped, based on the estimated hardness data of the object to be grasped; transmitting the generated haptic data to a haptic presentation device, and causing the computer to execute a process of presenting a haptic sensation based on the generated haptic data to the worker using the haptic presentation device; program.

Images