Physical quantity change detection device, robot system, and physical quantity change detection method

Abstract

The present invention provides a physical quantity change detection device, a robot system, and a physical quantity change detection method that can reduce the processing load and increase the processing speed. [Solution] The physical quantity change detection device 1 comprises a structure 3 that outputs a speckle pattern of light L when light L is incident on it, and a structure 3 whose output speckle pattern changes when a physical quantity related to the structure 3 changes due to the external environment, and an optical neural network unit 21 that performs neural network calculations on the input light and outputs output light representing the calculation result, and a processing unit 4 that acquires a value corresponding to the change in the physical quantity by an acquisition process that includes inputting at least a part of the speckle pattern to the optical neural network unit 21.

Description

Technical Field

[0001] The present invention relates to a physical quantity change detection device, a robot system, and a physical quantity change detection method.

Background Art

[0002] Patent Document 1 describes a detection device including a flexible member having light transmissibility and flexibility, a laser light source that emits laser light to the flexible member, and an imaging device that images a speckle pattern of the laser light. In this device, when the flexible member is deformed, a change occurs in the speckle pattern, and the presence or absence of deformation of the flexible member is detected based on the speckle pattern imaged by the imaging device.

Prior Art Documents

Patent Documents

[0003]

Patent Document 1

Summary of the Invention

Problems to be Solved by the Invention

[0004] In the above-described device, only the presence or absence of deformation of the flexible member is detected based on the speckle pattern. However, for example, it is conceivable to extract more advanced feature amounts from the speckle pattern. However, since the speckle pattern is a high-density signal based on the interaction of light and is a signal reflecting minute changes in an object, the amount of information contained in the speckle pattern is extremely large. Therefore, a great deal of processing is required for feature amount extraction from the speckle pattern, and there is a risk that the processing load and the time required for processing will increase.

[0005] Therefore, an object of the present invention is to provide a physical quantity change detection device, a robot system, and a physical quantity change detection method capable of reducing the processing load and speeding up the processing.

Means for Solving the Problems

[0006] The physical quantity change detection device of the present invention is a physical quantity change detection device comprising: [1] "a structure that outputs a speckle pattern of light when light is incident on it, wherein the speckle pattern output from the structure changes when a physical quantity related to the structure changes due to the external environment; an optical neural network unit that performs neural network calculations on input light and outputs output light representing the calculation results, and a processing unit that acquires a value corresponding to the change in the physical quantity by an acquisition process that includes inputting at least a part of the speckle pattern to the optical neural network unit."

[0007] In this physical quantity change detection device, when physical quantities related to a structure (e.g., pressure, shape, temperature, etc.) change due to the external environment, the speckle pattern output from the structure changes. The processing unit then acquires a value corresponding to the change in physical quantity through an acquisition process that includes inputting at least a portion of the speckle pattern into the optical neural network unit. This makes it possible to detect changes in physical quantities related to the structure from this value. Although the amount of information contained in the speckle pattern is extremely large, by applying neural network calculations to the speckle pattern while it is still in its optical state, it is possible to reduce the processing load and increase the speed compared to, for example, converting the speckle pattern into an electrical signal and applying neural network calculations using an electrical neural network. Therefore, this physical quantity change detection device can reduce the processing load and increase the speed.

[0008] The physical quantity change detection device of the present invention may also be [2] "the physical quantity change detection device according to [1], wherein the structure includes a multimode waveguide capable of propagating multiple modes of light, and when a physical quantity relating to the multimode waveguide changes, the speckle pattern output from the multimode waveguide changes." In this case, a speckle pattern can be output using a multimode waveguide.

[0009] The physical quantity change detection device of the present invention may also be [3] "the physical quantity change detection device according to [2], wherein the multimode waveguide is composed of a multimode fiber." In this case, a speckle pattern can be output using the multimode fiber.

[0010] The physical quantity change detection device of the present invention may also be the "physical quantity change detection device according to [2] or [3], wherein the optical neural network section includes an optical integrated circuit in which an optical waveguide is formed on a substrate." In this case, the optical neural network section can be realized using an optical integrated circuit. Furthermore, by using a multimode waveguide, the output from the multimode waveguide can be suitably input to the optical input section of the optical integrated circuit.

[0011] The physical quantity change detection device of the present invention may also be [5] "the physical quantity change detection device according to [4], wherein the processing unit further comprises an electrical neural network unit arranged downstream of the optical neural network unit, and the acquisition process further includes processing the output from the optical neural network unit by the electrical neural network unit." In this case, a part of the acquisition process can be processed by the electrical neural network unit, which is relatively easier to implement than the optical neural network unit. Furthermore, processing that is suitable for processing by an optical neural network can be processed by the optical neural network unit, and processing that is suitable for processing by an electrical neural network can be processed by the electrical neural network unit.

[0012] The physical quantity change detection device of the present invention may also be [6] "the physical quantity change detection device according to any one of [2] to [5], wherein the structure further includes a multicore fiber disposed between the multimode waveguide and the optical neural network section." In this case, by using a multicore fiber, a speckle pattern can be suitably input to the optical neural network section.

[0013] The physical quantity change detection device of the present invention may also be "the physical quantity change detection device according to any one of [2] to [6], further comprising a single-mode fiber disposed in front of the multimode waveguide." In this case, the single-mode fiber can suitably guide light into the multimode waveguide.

[0014] The physical quantity change detection device of the present invention may also be the physical quantity change detection device according to any one of [2] to [7], further comprising a soft material member arranged around the multimode waveguide. In this case, the multimode waveguide can be protected. Furthermore, deformation of the multimode waveguide can be mitigated compared to when the soft material member is not arranged.

[0015] The physical quantity change detection device of the present invention may also be the "physical quantity change detection device according to [8]", wherein the multimode waveguide includes a plurality of waveguide portions, each capable of propagating light of a plurality of modes, and the soft material member is arranged around the plurality of waveguide portions. In this case, a speckle pattern can be suitably generated in the multimode waveguide.

[0016] The physical quantity change detection device of the present invention may also be

[10] "the physical quantity change detection device according to [1], wherein the structure includes a light-transmitting soft material member, and when a physical quantity relating to the soft material member changes, the speckle pattern output from the soft material member changes." In this case, the soft material member can be used to output the speckle pattern.

[0017] The physical quantity change detection device of the present invention may also be

[11] "the physical quantity change detection device according to

[10] , wherein the processing unit further comprises an electrical neural network unit arranged downstream of the optical neural network unit, and the acquisition process further includes processing the output from the optical neural network unit by the electrical neural network unit." In this case, a part of the acquisition process can be processed by the electrical neural network unit, which is relatively easier to implement than the optical neural network unit. Furthermore, processing that is suitable for processing by an optical neural network can be processed by the optical neural network unit, and processing that is suitable for processing by an electrical neural network can be processed by the electrical neural network unit.

[0018] The physical quantity change detection device of the present invention may also be the "physical quantity change detection device according to

[10] , wherein the acquisition process consists only of inputting at least a portion of the speckle pattern to the optical neural network unit." That is, the acquisition process does not have to include processing by the electrical neural network unit, and may consist only of processing by the optical neural network unit, for example. In this case, it is possible to effectively reduce the processing load and increase the speed.

[0019] The physical quantity change detection device of the present invention may also be

[13] "a physical quantity change detection device according to any one of

[10] to

[12] wherein an object that scatters or attenuates light is arranged inside the soft material member." In this case, a speckle pattern can be suitably generated in the soft material member.

[0020] The physical quantity change detection device of the present invention may also be

[14] "the physical quantity change detection device according to any one of

[10] to

[13] , wherein the soft material member is formed in the shape of a flat plate having a main surface and side surfaces, and the speckle pattern output from the main surface of the soft material member is input to the optical neural network unit." In this case, the device can be miniaturized.

[0021] The physical quantity change detection device of the present invention may be "

[15] the soft material member has an incident surface which is an end surface on one side in a predetermined direction and a reflection surface which is an end surface on the other side in the predetermined direction, the light enters the incident surface and travels inside the soft material member, is reflected by the reflection surface and returns inside the soft material member, and exits from the incident surface as the speckle pattern and is input to the optical neural network unit, the physical quantity change detection device according to any one of

[10] to

[13] ". In this case, the device can be miniaturized.

[0022] The physical quantity change detection device of the present invention may be "

[16] the optical neural network unit includes an optical diffraction type neural network unit that utilizes light diffraction, the physical quantity change detection device according to any one of [1] to

[15] ". In this case, the optical neural network unit can be realized by using an optical diffraction type neural network unit that utilizes light diffraction.

[0023] The physical quantity change detection device of the present invention may be "

[17] the optical diffraction type neural network unit includes a plurality of mask layers, and each of the plurality of mask layers generates a predetermined diffraction for the light when the light passes through or transmits through the mask layer, the physical quantity change detection device according to

[16] ". In this case, the optical diffraction type neural network unit can be realized by using a plurality of mask layers.

[0024] The physical quantity change detection device of the present invention may be "

[18] the optical diffraction type neural network unit includes a spatial light modulator, the physical quantity change detection device according to

[16] ". In this case, the optical diffraction type neural network unit can be realized by using a spatial light modulator.

[0025] The physical quantity change detection device of the present invention may be the one described in

[19] "the optical diffraction type neural network unit is configured using hologram technology, the physical quantity change detection device described in

[16] ". In this case, the optical diffraction type neural network unit can be realized using hologram technology.

[0026] The physical quantity change detection device of the present invention may be the one described in

[20] "the optical diffraction type neural network unit is configured including a metasurface, the physical quantity change detection device described in

[16] ". In this case, the optical diffraction type neural network unit can be realized using the metasurface.

[0027] The physical quantity change detection device of the present invention may be the one described in

[21] "the optical diffraction type neural network unit is configured including a dynamically controllable metasurface, the physical quantity change detection device described in

[16] ". In this case, the optical diffraction type neural network unit can be realized using the dynamically controllable metasurface.

[0028] The physical quantity change detection device of the present invention may be the one described in

[22] "the optical diffraction type neural network unit consists of a single mask layer, and when the light passes through or penetrates the mask layer, the mask layer causes predetermined diffraction of the light, the processing unit further has an electrical neural network unit arranged at the subsequent stage of the optical neural network unit, and the acquisition processing further includes processing the output from the optical neural network unit by the electrical neural network unit, the physical quantity change detection device described in

[16] ". In this case, the optical diffraction type neural network unit can be realized using a single mask layer. Also, a part of the acquisition processing can be processed by an electrical neural network unit that is relatively easy to realize compared to the optical neural network unit. Also, the processing suitable for the optical neural network can be processed by the optical neural network unit, and the processing suitable for the electrical neural network can be processed by the electrical neural network unit.

[0029] The physical quantity change detection device of the present invention may also be described as

[23] "the physical quantity change detection device according to any one of [1] to

[22] , wherein the processing unit further comprises an electrical neural network unit arranged downstream of the optical neural network unit, and the acquisition process further includes processing the output from the optical neural network unit by the electrical neural network unit." In this case, a part of the acquisition process can be processed by the electrical neural network unit, which is relatively easier to implement than the optical neural network unit. Furthermore, processing that is suitable for processing by an optical neural network can be processed by the optical neural network unit, and processing that is suitable for processing by an electrical neural network can be processed by the electrical neural network unit.

[0030] The physical quantity change detection device of the present invention may also be

[24] "the processing unit detects a change in one of the physical quantities by the acquisition process, as described in any one of [1] to

[23] ." In this case, a change in one physical quantity can be detected.

[0031] The physical quantity change detection device of the present invention may also be

[25] "the physical quantity change detection device according to any one of [1] to

[23] , wherein the processing unit detects changes in a plurality of physical quantities by the acquisition process." In this case, changes in a plurality of physical quantities can be detected.

[0032] The physical quantity change detection device of the present invention may also be

[26] "a physical quantity change detection device according to any one of [1] to

[25] , further comprising a light source that outputs the light." In this case, a light source can be incorporated into the physical quantity change detection device.

[0033] The robot system of the present invention is

[27] "a robot system comprising a physical quantity change detection device described in any one of [1] to

[26] and a robot hand capable of grasping an object, wherein the physical quantity change detection device comprises a plurality of the structures, the plurality of structures are arranged at a plurality of locations on the robot hand, and the grasping state of the object by the robot hand is identified based on the change in the physical quantity in the plurality of structures." With this robot system, for the reasons described above, the processing load can be reduced and the processing speed can be increased.

[0034] The robot system of the present invention may also be

[28] "the robot system according to

[27] , wherein the physical quantity change detection device comprises a plurality of processing units, and at least a portion of the speckle pattern output from the plurality of structures is input to the optical neural network unit of each of the plurality of processing units." In this case, the gripping state of the object can be identified by processing the speckle pattern output from the plurality of structures with the plurality of optical neural network units.

[0035] The robot system of the present invention may also be

[29] "the robot system according to

[27] in which at least a portion of the speckle patterns output from the plurality of structures is input to the optical neural network unit of the processing unit." In this case, the gripping state of the object can be identified by processing the speckle patterns output from the plurality of structures with a single optical neural network unit.

[0036] The robot system of the present invention may also be

[30] "the robot system according to any one of

[27] to

[29] wherein the processing unit is located within the robot hand." In this robot system, since the processing unit includes an optical neural network unit, the processing unit can be miniaturized and it becomes possible to place the processing unit within the robot hand.

[0037] The physical quantity change detection device of the present invention may also be

[31] "the physical quantity change detection device according to any one of [1] to

[26] , wherein the speckle pattern is converted into at least one optical signal including a plurality of information from among the intensity information, wavelength information, phase information, and polarization information included in the speckle pattern, and the at least one optical signal is input to the optical neural network unit." In this case, a value corresponding to the change in the physical quantity can be obtained by utilizing the plurality of information from among the intensity information, wavelength information, phase information, and polarization information included in the speckle pattern.

[0038] The present invention provides a method for detecting changes in physical quantities, comprising:

[32] "a structure that outputs a speckle pattern of light when light is incident on it, wherein the speckle pattern output from the structure changes when a physical quantity relating to the structure changes due to the external environment; and an acquisition process that includes inputting at least a portion of the speckle pattern to the optical neural network unit, thereby acquiring a value corresponding to the change in the physical quantity."

[0039] In this physical quantity change detection method, when physical quantities related to a structure (e.g., pressure, shape, temperature, etc.) change due to the external environment, the speckle pattern output from the structure changes. Then, through an acquisition process that includes inputting at least a portion of the speckle pattern into an optical neural network, a value corresponding to the change in physical quantity is obtained. This makes it possible to detect changes in physical quantities related to the structure from this value. Although the amount of information contained in the speckle pattern is extremely large, by applying neural network calculations to the speckle pattern while it is still in its optical state, it is possible to reduce the processing load and speed up the process compared to, for example, converting the speckle pattern into an electrical signal and applying neural network calculations using an electrical neural network. Therefore, this physical quantity change detection method can reduce the processing load and speed up the process. [Effects of the Invention]

[0040] According to the present invention, it is possible to provide a physical quantity change detection device, a robot system, and a physical quantity change detection method that can reduce the processing load and increase the processing speed. [Brief explanation of the drawing]

[0041] [Figure 1] This is a configuration diagram of a physical quantity change detection device according to an embodiment. [Figure 2] (a) is a diagram showing an example of an optical image at point A1 in Figure 1, and (b) is a diagram showing an example of an optical image at point A2 in Figure 1. [Figure 3] This is a diagram showing the configuration of a physical quantity change detection device attached to a robot hand. [Figure 4] This is a diagram illustrating the configuration of the robot system in the first example. [Figure 5] This is a flowchart illustrating the processing example in the first example. [Figure 6] This is a diagram illustrating the configuration of the robot system in the second example. [Figure 7] This is a flowchart illustrating the processing example in the second example. [Figure 8] This diagram illustrates the difference in deformation of multimode fibers with and without soft material components. [Figure 9] This is a diagram showing the configuration of a physical quantity change detection device relating to the first modified example. [Figure 10] This is a diagram showing the configuration of a physical quantity change detection device related to the second modified example. [Figure 11] This is a diagram showing the configuration of a physical quantity change detection device related to other variations. [Figure 12] This is a diagram showing the configuration of a physical quantity change detection device related to other variations. [Figure 13] This is a diagram showing the configuration of a physical quantity change detection device related to other variations. [Figure 14] (a), (b), and (c) are diagrams illustrating other variations. [Figure 15] (a) and (b) are diagrams illustrating other variations. [Figure 16] (a) and (b) are diagrams illustrating other variations. [Modes for carrying out the invention]

[0042] Embodiments of the present invention will be described in detail below with reference to the drawings. In the following description, the same or equivalent elements will be denoted by the same reference numerals, and redundant explanations will be omitted.

[0043] As shown in Figure 1, the physical quantity change detection device 1 comprises a light source 2, a structure 3, and a processing unit 4. The light source 2 outputs light L. The light source 2 is, for example, a small laser diode with a low threshold current, and outputs laser light as light L.

[0044] Light L output from light source 2 is incident on structure 3. Structure 3 is composed of a single-mode fiber 11, a multi-mode fiber (multi-mode waveguide) 12, a multi-core fiber 13, and a soft material member 14.

[0045] The single-mode fiber 11 is optically connected to the light source 2 so that light L from the light source 2 is input to the single-mode fiber 11. The single-mode fiber 11 is positioned between the light source 2 and the multimode fiber 12. The single-mode fiber 11 consists of a fiber with a narrow core diameter, and only light L of a single propagation mode propagates within the single-mode fiber 11.

[0046] The multimode fiber 12 is optically connected to the single-mode fiber 11 such that the light L output from the single-mode fiber 11 is input to the multimode fiber 12. The multimode fiber 12 consists of a fiber with a larger core diameter than the single-mode fiber 11, and light L in multiple propagation modes propagates within the multimode fiber 12.

[0047] The multi-core fiber 13 is optically connected to the multi-mode fiber 12 such that the light L output from the multi-mode fiber 12 is input to the multi-core fiber 13. That is, one end of the multi-mode fiber 12 is optically connected to the single-mode fiber 11, and the other end of the multi-mode fiber 12 is optically connected to the multi-core fiber 13. The multi-core fiber 13 is positioned between the multi-mode fiber 12 and the optical neural network section 21, which will be described later. The multi-core fiber 13 has multiple cores 13a (Figure 2(b)), and light L propagates within each core 13a.

[0048] The soft material member 14 is arranged around the multimode fiber 12. The soft material member 14 is made of a material that is softer than the multimode fiber 12, for example. Examples of materials for the soft material member 14 include resin materials and transparent urethane. In this example, the soft material member 14 is formed to have a first surface 14a that is curved in a substantially hemispherical shape on one side and a second surface 14b that is flat on the other side. The multimode fiber 12 is arranged so that a portion of it is embedded within the soft material member 14. This portion of the multimode fiber 12 extends within the soft material member 14 along the first surface 14a.

[0049] When light L is incident on structure 3, it outputs a speckle pattern S of light L (Figures 2(a) and 2(b)). The speckle pattern S is an irregular optical image (spot pattern) caused by the interaction of light (interference, scattering, etc.). The speckle pattern S is generated when coherent light, such as laser light, interferes when scattered on a rough surface. When physical quantities related to structure 3 change due to the external environment, the speckle pattern S output from structure 3 changes. In this example, when physical quantities related to the multimode fiber 12 change due to the external environment, the speckle pattern S output from the multimode fiber 12 changes. The external environment refers to the environment outside the multimode fiber 12 (structure 3), such as the pressure (force) applied to the multimode fiber 12 and the ambient temperature of the multimode fiber 12. The physical quantities related to the multimode fiber 12 refer to, for example, the pressure (force) applied to the multimode fiber 12, the shape of the multimode fiber 12, and the temperature of the multimode fiber 12.

[0050] When physical quantities related to the multimode fiber 12 (structure 3) change due to the external environment, the speckle pattern S output from the multimode fiber 12 changes. Therefore, the speckle pattern S contains a lot of information about the physical quantities in question. The speckle pattern S contains this information in the form of intensity information (luminance information), wavelength information, phase information, and polarization information. The speckle pattern S output from the multimode fiber 12 changes according to the shape of the multimode fiber 12. In the multimode fiber 12, changes in physical quantities related to the multimode fiber 12 (e.g., the shape and temperature of the multimode fiber 12, the pressure acting on the multimode fiber 12, etc.) appear as changes in the shape of the multimode fiber 12. Therefore, based on the speckle pattern S output from the multimode fiber 12, physical quantities related to the multimode fiber 12 (e.g., the shape and temperature of the multimode fiber 12, the pressure acting on the multimode fiber 12, etc.) can be detected.

[0051] Figure 2(a) shows an example of an optical image at point A1 in Figure 1, and Figure 2(b) shows an example of an optical image at point A2 in Figure 1. As shown in Figure 2(a), the generated speckle pattern S propagates in the multimode fiber 12. As shown in Figure 2(b), in the multiple cores 13a of the multicore fiber 13, a portion of the speckle pattern S output from the multimode fiber 12 (a portion of the speckle pattern S that is incident on the cores 13a) propagates.

[0052] The processing unit 4 includes an optical neural network unit 21 and a light detection unit 22. The optical neural network unit 21 performs neural network calculations on the input light and outputs output light representing the calculation results. The processing unit 4 obtains values ​​corresponding to changes in the physical quantities of the multimode fiber 12 (structure 3) through an acquisition process that includes inputting at least a portion of the speckle pattern S to the optical neural network unit 21. In this example, the speckle pattern S of the light L output from each core 13a of the multicore fiber 13 of structure 3 is input to the optical neural network unit 21 via a plurality of optical fibers 5. The plurality of optical fibers 5 are provided in the same number as the plurality of cores 13a of the multicore fiber 13, and transmit the speckle pattern S output from the cores 13a to the optical neural network unit 21.

[0053] The optical neural network unit 21 performs neural network calculations on the input light (optical signal) and outputs the calculation results as output light. In this example, the optical neural network unit 21 is composed of an optical integrated circuit 33 (optical neural network circuit) in which an optical waveguide 32 is formed on a substrate 31. For example, the substrate 31 is a silicon substrate, and the optical integrated circuit 33 is a silicon optical integrated circuit. The optical waveguide 32 is formed over the entire substrate 31, including the part that constitutes the optical integrated circuit 33. In the optical integrated circuit 33, a neural network calculation is performed on the input light L (speckle pattern S) as it passes through the optical waveguide 32.

[0054] A neural network is a mathematical model that mimics the neural network of the human brain. In this example, the optical integrated circuit 33 uses phase modulation to perform neural network calculations on light L passing through the optical waveguide 32. For example, the optical waveguide 32 has multiple input ports corresponding to multiple input layers of the neural network, and multiple output ports corresponding to multiple output layers of the neural network. Neural network calculations are performed on the light L passing through the optical waveguide 32 from the time it is input to the input port until it is output from the output port. Alternatively, instead of modulating the phase of the light L passing through the optical waveguide 32, the intensity of the light L passing through the optical waveguide 32 may be modulated magnetically or transmissively.

[0055] The optical neural network unit 21 is designed (trained) to output a predetermined output light when a predetermined speckle pattern S is input. Therefore, when the speckle pattern S from the multi-core fiber 13 is input to the optical neural network unit 21, the light output from the optical neural network unit 21 can be detected, and based on the detection result, changes in the physical quantities of the multimode fiber 12 (structure 3) can be detected (measured). Thus, in this example, the processing unit 4 inputs the speckle pattern S to the optical neural network unit 21 and obtains a value corresponding to the change in the physical quantities of the multimode fiber 12 (acquisition process). In this example, a portion of the speckle pattern S generated in structure 3 (multimode fiber 12) (a portion input to the core 13a of the multi-core fiber 13) is input to the optical neural network unit 21.

[0056] The light detection unit 22 detects the light L output from the optical neural network unit 21. The light detection unit 22 is, for example, a photodiode array and has multiple detection regions 22a. Each of the multiple detection regions 22a detects multiple output lights from the optical neural network unit 21 (lights output from the multiple output ports described above).

[0057] The processing unit 4 detects changes in physical quantities (e.g., changes in pressure) occurring in the multimode fiber 12 (structure 3) due to the external environment, based on the detection results of the photodetector 22. The processing unit 4 is configured to include, for example, an integrated circuit C which is an ASIC (Application Specific Integrated Circuit), and the integrated circuit C performs the detection process.

[0058] Figure 3 is a diagram showing the configuration of the physical quantity change detection device 1 attached to the robot hand 51. The physical quantity change detection device 1 and the robot hand 51 constitute a robot system 50. The robot hand 51 is provided, for example, at the end of a robot arm configured to be movable and / or operable. In this example, the robot hand 51 has drivable fingers 52, and the physical quantity change detection device 1 constitutes the tip of the fingers 52.

[0059] More specifically, the physical quantity change detection device 1 is integrated with a part 52a that constitutes the finger 52 on the second surface 14b of the soft material member 14. The finger 52 contacts an object, for example, on the first surface 14a of the soft material member 14. When the soft material member 14 is pressed by the object, the multimode fiber 12 is pressed through the soft material member 14. As a result, the pressure (force) acting on the multimode fiber 12 changes. When the pressure acting on the soft material member 14 changes and the pressure acting on the multimode fiber 12 changes, the physical quantity change detection device 1 detects the change in pressure acting on the multimode fiber 12 based on the change in the speckle pattern S. In the example of Figure 3, the processing unit 4 is located outside the finger 52, but the processing unit 4 may be located inside the finger 52.

[0060] Figure 4 is a diagram of the configuration of the robot system 50 according to the first example. In the first example, the robot hand 51 has multiple fingers 52 (three in this example) 52A, 52B, and 52C, and is capable of grasping the object B with fingers 52A, 52B, and 52C. The physical quantity change detection device 1 is equipped with the same number of structures 3 and processing units 4 as fingers 52A, 52B, and 52C (three in this example). The three structures 3 are attached to the tips of fingers 52A, 52B, and 52C, respectively. The three processing units 4 are connected to the three structures 3, and the speckle patterns S output from the three structures 3 are input to the optical neural network section 21 of the three processing units 4. Each processing unit 4 detects changes in physical quantities (changes in pressure in this example) occurring in the connected structure 3. That is, in this example, information from one structure 3 (finger 52) is processed (decoded) by one processing unit 4.

[0061] Figure 5 is a flowchart illustrating an example of processing in the first example. In this example, when the robot hand 51 performs a gripping operation on object B, the gripping state of object B is identified based on the detection result by the physical quantity change detection device 1. First, the robot hand 51 grips object B with its fingers 52A, 52B, and 52C (S1). In step S1, as shown in Figure 4, the fingers 52A, 52B, and 52C of the robot hand 51 come into contact with object B on the first surface 14a of the soft material member 14.

[0062] Next, the three processing units 4 detect changes in the pressure acting on fingers 52A, 52B, and 52C (S2A, S2B, S2C). Steps S2A, S2B, and S2C are processes performed in the processing units 4 connected to the structures 3 attached to fingers 52A, 52B, and 52C, respectively. Step S2A detects the pressure P(A) acting on finger 52A, step S2B detects the pressure P(B) acting on finger 52B, and step S2C detects the pressure P(C) acting on finger 52C.

[0063] Next, it is determined whether the gripping state of object B by the robot hand 51 is good or not (S3). The processing in step S3 is performed, for example, by the control unit of the robot having the robot hand 51. The pressures P(A), P(B), and P(C) detected in steps S2A, S2B, and S2C are input to the control unit. In step S3, the gripping state of object B is estimated based on the pressures P(A), P(B), and P(C), and it is determined whether the gripping state of object B is good or not based on the estimation result.

[0064] If the judgment determines that the gripping state of object B is good (YES in step S3), the system identifies that the gripping state is good and terminates the process. If the judgment determines that the gripping state of object B is not good (NO in step S3), the process proceeds to step S4. In step S4, the force applied to fingers 52A, 52B, and 52C is adjusted. After the completion of step S4, the process of step S1 is executed again. Through the above process, the object can be gripped well.

[0065] Figure 6 is a diagram of the configuration of the robot system 50 according to the second example. In the second example, the physical quantity change detection device 1 has only one processing unit 4, and the speckle patterns S output from multiple (three in this example) structures 3 are input to the optical neural network section 21 of the single processing unit 4. The processing unit 4 acquires values ​​corresponding to the changes in physical quantities (changes in pressure in this example) occurring in the three structures 3. In other words, in this example, information from multiple structures 3 (fingers 52) is processed (decoded) together by a single processing unit 4. By decoding the information from multiple fingers 52A, 52B, and 52C together, the gripping state of the object B and the interaction of fingers 52A, 52B, and 52C are identified. In other respects, the second example is configured the same as the first example.

[0066] Figure 7 is a flowchart illustrating a processing example in the second example. In this example, when the robot hand 51 performs a gripping operation on object B, the gripping state of object B is identified based on the detection result by the physical quantity change detection device 1. First, the robot hand 51 grips object B with its fingers 52A, 52B, and 52C (S11). In step S11, as shown in Figure 6, the fingers 52A, 52B, and 52C of the robot hand 51 come into contact with object B on the first surface 14a of the soft material member 14.

[0067] Next, in the processing unit 4, values ​​corresponding to the changes in pressure acting on fingers 52A, 52B, and 52C are acquired (S12). Subsequently, it is estimated whether the gripping state of object B by the robot hand 51 is good or not (S13). The processing in step S13 is performed, for example, by the control unit of a robot having the robot hand 51. The control unit is input with values ​​corresponding to the changes in pressure acting on fingers 52A, 52B, and 52C acquired in step S12. In step S13, the gripping state of object B is estimated based on these values, and it is determined whether the gripping state of object B is good or not based on the estimation result.

[0068] If the judgment determines that the gripping state of object B is good (YES in step S13), the system identifies that the gripping state is good and terminates the process. If the judgment determines that the gripping state of object B is not good (NO in step S13), the process proceeds to step S14. In step S14, the force applied to fingers 52A, 52B, and 52C is adjusted. After the completion of step S15, the process in step S11 is executed again. Through the above process, the object can be gripped well. [Mechanism of Action and Effects]

[0069] In the physical quantity change detection device 1, when physical quantities related to the structure 3 (e.g., pressure, shape, temperature, etc.) change due to the external environment, the speckle pattern S of the light L output from the structure 3 changes. The processing unit 4 then acquires a value corresponding to the change in physical quantities through an acquisition process that includes inputting at least a portion of the speckle pattern S into the optical neural network unit 21. This makes it possible to detect changes in physical quantities related to the structure 3 from this value. Although the amount of information contained in the speckle pattern S is extremely large, by applying neural network calculations to the speckle pattern S while it is still in the optical state, it is possible to reduce the processing load and increase the speed compared to, for example, converting the speckle pattern S into an electrical signal and applying neural network calculations using an electrical neural network. Therefore, the physical quantity change detection device 1 can reduce the processing load and increase the speed. In addition, by performing dimensionality compression of information by the optical neural network unit 21, it is possible to simplify the detector (light detection unit 22). For example, even if reproducing a speckle pattern S requires information with a resolution of 1000 x 1000 pixels, if the data processed by the optical neural network unit 21 has a resolution of 100 x 100 pixels, the detector only needs to have 100 x 100 pixels. Furthermore, it can detect changes in physical quantities with high precision (high spatial resolution), similar to the human nervous system. In addition, because it utilizes light, it can achieve low power consumption.

[0070] Structure 3 includes a multimode fiber 12, and when a physical quantity related to the multimode fiber 12 changes, the speckle pattern S output from the multimode fiber 12 changes. This makes it possible to output the speckle pattern S using the multimode fiber 12.

[0071] The optical neural network section 21 is configured to include an optical integrated circuit 33 in which an optical waveguide 32 is formed on a substrate 31. This allows the optical neural network section 21 to be realized using the optical integrated circuit 33. Furthermore, by using a multimode fiber 12, the output from the multimode fiber 12 can be suitably input to the optical input section (input port) of the optical integrated circuit 33. In other words, when using the optical integrated circuit 33, it is necessary to input the signal to the input port of the optical waveguide 32. Directly inputting high-resolution information into the optical waveguide 32 would require a large-scale optical integrated circuit 33, but realizing such an optical integrated circuit 33 is difficult. Therefore, when using the optical integrated circuit 33, input using optical fibers (multimode fiber 12, multicore fiber 13) that can input information after reducing the amount of information is suitable.

[0072] Structure 3 includes a multi-core fiber 13 positioned between the multimode fiber 12 and the optical neural network section 21. This allows the speckle pattern S to be suitably input to the optical neural network section 21 by using the multi-core fiber 13.

[0073] Structure 3 includes a single-mode fiber 11 positioned between the light source 2 and the multimode fiber 12. This allows the single-mode fiber 11 to suitably guide the light L from the light source 2 to the multimode fiber 12.

[0074] Structure 3 includes a soft material member 14 placed around the multimode fiber 12. This protects the multimode fiber 12. Furthermore, it can mitigate deformation of the multimode fiber 12 compared to when the soft material member 14 is not present.

[0075] This point will be explained with reference to Figure 8. As shown in the upper part of Figure 8, if the soft material member 14 is not provided, the multimode fiber 12 may deform relatively easily to the point of fracture. In contrast, as shown in the lower part of Figure 8, if the soft material member 14 is provided, the load is distributed by the soft material member 14, which can mitigate the deformation of the multimode fiber 12 (structure 3), and it is possible to suppress the deformation of the multimode fiber 12 to the point of fracture. In other words, if the soft material member 14 is provided, the amount of deformation of the multimode fiber 12 will be smaller even if pressed with the same force. This makes it possible to extend the upper limit of the pressing force that can be detected by the physical quantity change detection device 1. Thus, by providing the soft material member 14, the deformation of the multimode fiber 12 can be suppressed (the influence of the external environment on the multimode fiber 12 can be distributed).

[0076] In the first example described above, the physical quantity change detection device 1 is equipped with multiple processing units 4, and the speckle patterns S output from multiple structures 3 are input to the optical neural network units 21 of each of the multiple processing units 4. In this case, the gripping state of the object B can be identified by processing the speckle patterns S output from the multiple structures 3 with the multiple optical neural network units 21.

[0077] In the second example described above, the physical quantity change detection device 1 has only one processing unit 4, and the speckle patterns S output from multiple structures 3 are input to the optical neural network unit 21 of the single processing unit 4. In this case, by processing the speckle patterns S output from multiple structures 3 with the single optical neural network unit 21, the gripping state of the object B can be identified.

[0078] The robot system 50 comprises a physical quantity change detection device 1 and a robot hand 51 capable of grasping an object B. The physical quantity change detection device 1 comprises multiple structures 3, which are arranged at multiple locations on the robot hand 51. Based on the changes in physical quantities in the multiple structures 3, the grasping state of the object B by the robot hand 51 is identified. According to the robot system 50, for the reasons described above, the processing load can be reduced and the processing speed can be increased.

[0079] The processing performed by the physical quantity change detection device 1 described above can be considered a physical quantity change detection method. This physical quantity change detection method comprises the steps of: a structure 3 that outputs a speckle pattern S of light L when light L is incident on it, and the speckle pattern S output from the structure 3 changes when a physical quantity related to the structure 3 changes due to the external environment, light L output from a light source 2 is incident on the structure 3 and the structure 3 outputs a speckle pattern S; and an acquisition process that includes inputting at least a part of the speckle pattern S to an optical neural network unit 21 that performs neural network calculations on the input light and outputs output light representing the calculation result, thereby acquiring a value corresponding to the change in the physical quantity. According to this physical quantity change detection method, for the reasons described above, the processing load can be reduced and the processing speed can be increased.

[0080] The first modified physical quantity change detection device 1A shown in Figure 9 comprises a light source 2, a structure 3A, and a processing unit 4A. The structure 3A is composed of a multimode fiber 16 and a soft material member 17. The multimode fiber 16 is optically connected to the light source 2 so that light L from the light source 2 is input to the soft material member 17. Multiple propagation modes of light L can propagate within the multimode fiber 16.

[0081] The soft material member 17 is formed from, for example, a material that is light-transmitting and flexible. Examples of materials for the soft material member 17 include resin materials and transparent urethane. The soft material member 17 is formed in a flat plate shape having a light-entering surface 17a on one side and a light-exiting surface 17b on the other side.

[0082] In the above embodiment, the speckle pattern S is generated in the multimode fiber 12, but in the first modified example, the speckle pattern S is generated in the soft material member 17. In the first modified example, when a physical quantity related to the soft material member 17 changes due to the external environment, the speckle pattern S output from the soft material member 17 changes. The structure 3 outputs the speckle pattern S from the output surface 17b of the soft material member 17. For example, when the soft material member 17 is pressed by an object, the pressure (force) acting on the soft material member 17 changes. The physical quantity change detection device 1 detects the change in pressure acting on the soft material member 17 based on the change in the speckle pattern S when the pressure acting on the soft material member 17 from the object changes.

[0083] The processing unit 4A includes an optical neural network unit 21A, a photodetector unit 22A, and an electrical neural network unit 23. In the first modified example, the processing unit 4A acquires values ​​corresponding to changes in physical quantities related to the soft material member 17 (structure 3A) through an acquisition process in which the entire speckle pattern S is input to the optical neural network unit 21A. That is, in this example, the entire speckle pattern S output from the emission surface 17b of the soft material member 17 is input to the optical neural network unit 21A.

[0084] The optical neural network section 21A is composed of a diffractive neural network section 60 consisting of a single mask layer 61. The optical diffractive neural network section 60 uses the diffraction of light to perform neural network calculations on input light (optical signal) and outputs the result of the calculation as output light. The mask layer 61 is formed in a planar shape, and the entire speckle pattern S is input to the mask layer 61. The mask layer 61 has an internal structure for diffracting light L (speckle pattern S), and generates predetermined diffraction in the light L when it passes through or is transmitted through the mask layer 61. The mask layer 61 can be manufactured, for example, by a 3D printer.

[0085] The mask layer 61 is designed to output a predetermined output light when a predetermined speckle pattern S is input. Therefore, by detecting the light L output from the mask layer 61 when the speckle pattern S from the soft material member 17 is input to the mask layer 61, it is possible to detect (measure) changes in physical quantities related to the soft material member 17 (structure 3A) based on the detection result. Thus, in this first modified example, the processing unit 4A inputs the speckle pattern S to the optical neural network unit 21A to acquire values ​​corresponding to changes in physical quantities related to the soft material member 17 (acquisition process).

[0086] The light detection unit 22A detects the light L output from the mask layer 61. The light detection unit 22A is, for example, an image sensor and has multiple pixels (detection areas). The light detection unit 22A detects the light L output from the mask layer 61 at each pixel.

[0087] The electrical neural network unit 23 performs neural network calculations on the input electrical signal and outputs the calculation result as an electrical signal. The electrical neural network unit 23 has an input layer 23a, an intermediate layer 23b coupled to the input layer 23a, and an output layer 23c coupled to the intermediate layer 23b. The input layer 23a has multiple nodes 23a1, the intermediate layer 23b has multiple nodes 23b1, and the output layer 23c has multiple nodes 23c1.

[0088] In the electrical neural network section 23, each connection between nodes 23a1, 23b1, and 23c1 has a weight. When data is input to the input layer 23a, a sum-of-accumulate operation is performed at each node 23b1 and 23c1, and finally the values ​​of each node 21c1 in the output layer 21c are output as output data. In the electrical neural network section 23, the connection weights between the input layer 23a and the hidden layer 23b, and the connection weights between the hidden layer 23b and the output layer 23c are learned based on the target values ​​(training data).

[0089] Multiple nodes 23a1 of the input layer 23a of the electrical neural network unit 23 are input to the detection results (electrical signals) of multiple pixels of the photodetector unit 22. The electrical neural network unit 23 performs neural network calculations on the input data and outputs the calculation results from the output layer 23c. The output from the electrical neural network unit 23 becomes the output of the processing unit 4A. Thus, in this first modified example, the acquisition process for obtaining values ​​corresponding to changes in physical quantities related to the soft material member 17 includes not only inputting the speckle pattern S to the optical neural network unit 21A, but also processing the output from the optical neural network unit 21A (mask layer 61) by the electrical neural network unit 23.

[0090] The physical quantity change detection device 1A of the first modified example can also reduce the processing load and increase the speed, similar to the embodiment described above. In the first modified example, the structure 3A includes a light-transmitting soft material member 17, and when a physical quantity related to the soft material member 17 changes, the speckle pattern S output from the soft material member 17 changes. This makes it possible to output the speckle pattern S using the soft material member 17.

[0091] The processing unit 4A has an electrical neural network unit 23 located after the optical neural network unit 21A, and the acquisition process includes processing the output from the optical neural network unit 21A by the electrical neural network unit 23. This allows a portion of the acquisition process to be processed by the electrical neural network unit 23, which is relatively easier to implement compared to the optical neural network unit 21A. Furthermore, processing suitable for optical neural network processing can be handled by the optical neural network unit 21A, and processing suitable for electrical neural network processing can be handled by the electrical neural network unit 23.

[0092] The optical neural network section 21A is configured to include an optical diffraction type neural network section 60 that utilizes the diffraction of light. This allows the optical neural network section 21A to be realized using the optical diffraction type neural network section 60 that utilizes the diffraction of light. When using the optical diffraction type neural network section 60 as in the first modified example, losses can be suppressed by inputting light that propagates spatially rather than light that propagates through a waveguide into the optical diffraction type neural network section 60. This is because when light propagating through a waveguide is input into the optical diffraction type neural network section 60, losses may occur due to reflections at coupling points, etc. The optical diffraction type neural network section 60 may also include nonlinear elements (e.g., image intensifiers). In this case, the accuracy of feature extraction can be improved.

[0093] The optical diffraction neural network section 60 consists of a single mask layer 61, and the mask layer 61 generates predetermined diffraction in the light L when it passes through or is transmitted through the mask layer 61. This makes it possible to realize the optical diffraction neural network section 60 using a single mask layer 61.

[0094] The physical quantity change detection device 1B of the second modified example shown in Figure 10 comprises a light source 2, a structure 3A, and a processing unit 4B. The processing unit 4B of the second modified example does not have an electrical neural network unit 23, but has an optical neural network unit 21B and a photodetection unit 22A. In the second modified example, the acquisition process by the processing unit 4B consists only of inputting the speckle pattern S to the optical neural network unit 21B. That is, the acquisition process does not include processing by the electrical neural network unit, and is composed only of processing by the optical neural network unit 21A.

[0095] The optical neural network section 21B is composed of an optical diffraction type neural network section 60 consisting of a plurality of mask layers 62. Each of the plurality of mask layers 62 is formed in a planar shape and is arranged in a line along the direction of propagation of light L. The speckle pattern S is input to the mask layer 62 located at the forefront (on the soft material member 17 side) of the plurality of mask layers 62. Each mask layer 62 has an internal structure for diffracting light L (speckle pattern S), and generates predetermined diffraction in the light L when it passes through or is transmitted through the mask layer 62.

[0096] In this example, the optical diffraction neural network section 60 is configured as a D2NN (Diffractive Deep Neural Network) that applies deep learning-based neural network calculations to the input light. In this case, the optical diffraction neural network section 60 utilizes the diffraction and interference of light to perform neural network calculations on the input light. Diffraction of light is the phenomenon where light bends around an obstacle, and interference of light is the phenomenon where two or more waves reinforce or cancel each other out when they reach the same location. The mask layer 62 utilizes these principles, and the diffraction mask pattern is designed so that the diffraction of light behaves as a neural network calculation. More specifically, each point in each mask layer 62 functions as an artificial neuron in the neural network. Diffraction occurs when light passes through the mask layer 62. At the focal plane after diffraction, each neuron interacts with neurons in other mask layers 62. By setting the interactions of each neuron so that deep learning-based neural network calculations are performed, the diffraction mask pattern of multiple mask layers 62 is designed.

[0097] The optical diffraction neural network section 60 (multiple mask layers 62) is designed to output a predetermined output light when a predetermined speckle pattern S is input. Therefore, when the speckle pattern S from the soft material member 17 is input to the optical diffraction neural network section 60, the light L output from the optical diffraction neural network section 60 can be detected, and based on the detection result, changes in physical quantities related to the soft material member 17 (structure 3A) can be detected (measured). Thus, in this second modified example, the processing unit 4B inputs the speckle pattern S to the optical neural network section 21B to obtain a value corresponding to the change in physical quantities related to the soft material member 17.

[0098] The photodetector 22A detects light L output from the optical diffraction neural network section 60 (multiple mask layers 62). The photodetector 22A detects the light L output from the optical diffraction neural network section 60 at multiple pixels. Based on the detection results of the photodetector 22A, the processing unit 4B detects changes in physical quantities occurring in the soft material member 17 (structure 3A) due to the external environment. For example, a feature map, which is data that associates the detection results of the photodetector 22A with changes in physical quantities, is stored in advance, and by referring to this feature map, changes in physical quantities can be grasped from the detection results of the photodetector 22A. This processing may be performed, for example, by the integrated circuit C (Figure 1) described above.

[0099] The second modified physical quantity change detection device 1B also achieves the same reduction in processing load and speed as the above embodiment. Furthermore, the acquisition process consists only of inputting the speckle pattern S to the optical neural network unit 21B. This effectively reduces the processing load and speeds up the process. In addition, electrical processing can be reduced.

[0100] The optical diffraction neural network section 60 includes a plurality of mask layers 62, and each mask layer 62 generates a predetermined diffraction in the light L when it passes through or is transmitted through the mask layer 62. This makes it possible to realize the optical diffraction neural network section 60 using a plurality of mask layers 62.

[0101] The present invention is not limited to the above embodiments and modifications. For example, the materials and shapes of each component are not limited to those described above, but can be made from a variety of materials and shapes. In the above embodiment, the structure 3 outputs a speckle pattern S corresponding to changes in physical quantities related to the multimode fiber 12, and in the first and second modifications, it outputs a speckle pattern S corresponding to changes in physical quantities related to the soft material member 17. However, it may also output a speckle pattern S corresponding to changes in physical quantities of elements other than the multimode fiber 12 and the soft material member 17.

[0102] In the first and second modified examples, the optical diffraction neural network section 60 may include a spatial light modulator. A spatial light modulator is a device that electrically controls the spatial distribution of light, such as amplitude, phase, and polarization. This control is achieved, for example, using a liquid crystal. In this case, the optical diffraction neural network section 60 can be realized using a spatial light modulator. Figure 11 shows an example in the first modified example in which the optical diffraction neural network section 60 is configured to include a spatial light modulator 65.

[0103] In the first and second modified examples, the optical diffraction neural network section 60 may be constructed using hologram technology. Hologram technology is used to record and reproduce optical wavefronts. Phase information is recorded on the recording member on which the hologram is recorded, and the wavefront can be reproduced by irradiating the hologram with reference light. In this case, the optical diffraction neural network section 60 can be realized using hologram technology. The hologram can also function as a layer for performing neural network calculations, similar to the mask layers 61 and 62. Since phase information can be recorded on the hologram, the wavefront can be effectively controlled. Figure 11 shows an example in the first modified example in which the optical diffraction neural network section 60 is constructed using hologram technology (recording member 66 on which the hologram is recorded).

[0104] In the first and second modified examples, the optical diffraction neural network section 60 may be configured using a metasurface. A metasurface is an artificially formed fine periodic structure that allows control of the reflection phase in the electromagnetic field reflected off its surface. In this case, the optical diffraction neural network section 60 can be realized using a metasurface. The metasurface may be a static metasurface whose reflection characteristics do not change, or it may be a metasurface whose reflection characteristics can be dynamically controlled. Figure 11 shows an example in the first modified example in which the optical diffraction neural network section 60 is configured to include a metasurface 67.

[0105] In the examples shown in Figures 9 to 11, the speckle pattern S output from the emission surface 17b, which is the side surface of the soft material member 17, is input to the optical neural network units 21A and 21B. However, the speckle pattern S output from the main surface 17c of the soft material member 17 may also be input to the optical neural network units 21A and 21B. Figure 12 shows an example in the first modified example where the speckle pattern S output from the main surface 17c is input to the optical neural network unit 21A. In this case, the physical quantity change detection device 1 can be miniaturized. Furthermore, by adjusting the positional relationship between the processing unit 4 and the structure 3, interference between the processing unit 4 and the gripping unit (structure 3) can be avoided.

[0106] In the examples shown in Figures 9 to 11, the light L from the light source 2 is incident on the structure 3 on the opposite side from the processing units 4A and 4B, but the light L from the light source 2 may also be incident on the structure 3 on the same side as the processing units 4A and 4B. Figure 13 shows an example in the first modification in which the light L from the light source 2 is incident on the structure 3 on the same side as the processing unit 4A. In the example in Figure 13, the soft material member 17 has an incident surface 17d, which is one end face in a predetermined direction (left-right direction in the figure), and a reflective surface 17e, which is the other end face in the predetermined direction. The light L output from the light source 2 is incident on the incident surface 17d via the multimode fiber 16 and travels through the inside of the soft material member 17. The light L is then reflected by the reflective surface 17e and returns through the inside of the soft material member 17, and is emitted from the incident surface 17d as a speckle pattern S and input to the optical neural network unit 21A. In this case, the physical quantity change detection device 1 can be miniaturized. Furthermore, by adjusting the positional relationship between the processing unit 4 and the structure 3, interference between the processing unit 4 and the gripping unit (structure 3) can be avoided.

[0107] As shown in Figure 14, objects that scatter or attenuate light may be placed inside the soft material member 14. In the example in Figure 14(a), a plurality of granular scatterers 71 are arranged randomly inside the soft material member 14. The scatterers 71 are formed from, for example, air bubbles, resin beads, or silica, and scatter light. In the example in Figure 14(b), a plurality of granular scatterers 71 are arranged regularly inside the soft material member 14. In the example in Figure 14(a), the sizes of the plurality of scatterers 71 vary, while in the example in Figure 14(b), the sizes of the plurality of scatterers 71 are uniform. In the example in Figure 14(c), a plurality of linear markers 72 are arranged regularly inside the soft material member 14. The markers 72 are formed from, for example, ink or resin that does not transmit light, and attenuate light.

[0108] In these cases, a speckle pattern S can be suitably generated in the soft material member 14. More specifically, when the scatterers 71 or markers 72 are arranged randomly, the influence on the speckle pattern S can be greatly increased even when the displacement of the structure 3 is small. This can mitigate the concentration of changes in the speckle pattern S in a part of the image, and enable efficient learning from the features of the entire image. When the scatterers 71 or markers 72 are arranged regularly, the shape information of the structure 3 can be pre-assigned to the speckle pattern S. This enables information processing that adds shape information to the information of the speckle pattern S, which can be used for analysis of the direction of displacement of the structure 3, etc. These processes can be performed simultaneously with speckle pattern generation (signal generation). Therefore, unlike electrical calculations, no additional computational load is incurred by this process.

[0109] In the example in Figure 1, a soft material member 14 is arranged around a single multimode fiber 12. However, multiple multimode fibers 12 may be arranged as shown in Figure 15(a), or they may be branched into multiple parts as shown in Figure 15(b). In the example in Figure 15(a), multiple multimode fibers 12 are arranged to pass through the soft material member 14. In the example in Figure 15(b), the multimode fiber 12 is branched within the soft material member 14. In either case, the multimode fiber 12 includes multiple waveguide portions 12a, each capable of propagating multiple modes of light, and the soft material member 14 is arranged around the multiple waveguide portions 12a.

[0110] In these cases, a speckle pattern S can be suitably generated in the multimode fiber 12 (multimode waveguide). More specifically, by arranging multiple waveguide portions 12a inside the soft material member 14, the influence on the speckle pattern S can be increased even when the displacement of the structure 3 is small. Furthermore, it is possible to reduce the sensing area with low sensitivity and to perform information processing that takes into account the positional information of the multimode fiber 12 (for example, it is possible to recognize the direction in which a force is applied). In addition, it is possible to increase the number of input nodes to the optical neural network section 21 and add spatial positional information of the speckle generation section to each node. Note that the input nodes to the optical neural network section 21 may be consolidated into one. Even in that case, spatial positional information can be reliably added by arranging multiple waveguide portions 12a as described above.

[0111] In the examples of Figures 4 and 5, the robot hand 51 had multiple fingers 52A, 52B, and 52C, but the robot hand 51 may also be configured as a gripper type, as shown in Figure 16. In the examples of Figures 16(a) and 16(b), the robot hand 51 has a pair of flat gripper sections 53. The pair of gripper sections 53 are drivable to move closer to each other, and grip the object B by sandwiching it between them. The processing unit 4 is located inside the gripper section 53, and the structure 3 is attached to the gripper section 53. In the example of Figure 16(a), the structure 3 is attached to the gripper section 53 so that it is located inside the pair of gripper sections 53, and the structure 3 is attached to the gripper section 53 so that it protrudes towards the tip of the pair of gripper sections 53.

[0112] In the robot system 50, the processing unit 4 is configured to include the optical neural network unit 21, which allows the processing unit 4 to be miniaturized and to be placed inside the robot hand 51 as shown in the above structure. When the robot hand 51 needs to enter a narrow space, the gripping part needs to be thin. In the robot system 50, the speckle pattern S generated from the structure 3 is spatially parallel processed in optical form by the optical neural network unit 21, which makes it possible to configure the processing unit 4 to be small and simple, and to make the gripping part thinner.

[0113] In the physical quantity change detection device 1 of the above embodiment, the processing unit 4 may have an electrical neural network unit located downstream of the optical neural network unit 21 (optical integrated circuit 33). In this case, the acquisition process by the processing unit 4 may include processing the output from the optical neural network unit 21 by the electrical neural network unit. In this case, a part of the acquisition process can be processed by the electrical neural network unit, which is relatively easier to implement than the optical neural network unit 21. Furthermore, processing that is suitable for processing by an optical neural network can be processed by the optical neural network unit 21, and processing that is suitable for processing by an electrical neural network can be processed by the electrical neural network unit.

[0114] The processing unit 4 may detect a change in one physical quantity (e.g., pressure) through the acquisition process, or it may detect changes in multiple physical quantities (e.g., pressure and temperature) through the acquisition process. In the latter case, the physical quantity change detection device 1 functions as a multimodal sensor that detects multiple physical quantities.

[0115] In the physical quantity change detection device 1 of the above embodiment, the speckle pattern S output from the structure 3 was directly input to the optical neural network unit 21. However, after being output from the structure 3, the speckle pattern S may be converted into at least one optical signal containing multiple pieces of information from among the intensity information, wavelength information, phase information, and polarization information contained in the speckle pattern S, and this at least one optical signal may be input to the optical neural network unit 21. In this case, a value corresponding to the change in the physical quantity can be obtained by utilizing multiple pieces of information from among the intensity information, wavelength information, phase information, and polarization information contained in the speckle pattern S. For example, when using wavelength information, the signal light may be split by wavelength to simultaneously obtain information from multiple wavelength bands. For example, when using polarization information, a signal may be obtained simultaneously from light in multiple polarization states, similar to when using wavelength information. Wavelength information and polarization information may also be used in combination.

[0116] The single mask layer 61 of the first modification may be composed of multiple layer portions within a single layer. In this case, the multiple layer portions function similarly to the multiple mask layers 62 of the second modification, thereby enabling the mask layer 61 to function as a D2NN. Such a mask layer 61 can be manufactured, for example, by a silicon process. As an example, a structure utilizing the refractive index difference between silicon and silicon nitride (SiN) can be considered.

[0117] In the physical quantity change detection device 1 of the above embodiment, the single-mode fiber 11 may be omitted, and the light L from the light source 2 may be directly input to the multimode fiber 12. In the physical quantity change detection device 1, the multicore fiber 13 may be omitted, and the speckle pattern S from the multimode fiber 12 may be directly input to the optical neural network unit 21. In this case, the light (speckle pattern S) from the multimode fiber 12 may be beam-shaped or wavefront-controlled using mirrors, lenses, gratings, etc., and then input to the optical neural network unit 21 via free space. In the physical quantity change detection device 1, the soft material member 14 may be omitted.

[0118] In the first modified physical quantity change detection device 1A, the optical diffraction type neural network section 60 may be configured to include a plurality of mask layers 62, as in the second modified example. In other words, the processing unit 4A may have an optical neural network section 21A configured to include an optical diffraction type neural network section 60 consisting of a plurality of mask layers 62, and an electrical neural network section 23 arranged downstream of the optical neural network section 21A.

[0119] In the above embodiment, the multimode waveguide was constructed using a multimode fiber 12, but the multimode waveguide may be constructed using materials other than the multimode fiber 12. For example, the multimode waveguide may be a resin optical waveguide formed by a 3D printer, an optical waveguide formed on a silicon substrate, or an optical waveguide formed from a transparent polymer. In the first modified example, the soft material member 17 was formed in a flat plate shape, but the soft material member 17 may be formed in any shape. The physical quantity change detection device 1 does not necessarily have to include a light source 2. For example, light L output from a light source 2 located outside the physical quantity change detection device 1 may be incident on the structure 3A. [Explanation of symbols]

[0120] 1,1A,1B...Physical quantity change detection device, 2...Light source, 3,3A...Structure, 4,4A,4B...Processing unit, 11...Single-mode fiber, 12...Multimode fiber (multimode waveguide), 12a...Waveguide section, 13...Multicore fiber, 14...Soft material component, 16...Multimode fiber, 17...Soft material component, 17c...Main surface, 17d...Incident surface, 17e...Reflective surface, 21,21A,21B...Optical neural network section, 23...Electrical neural network section, 31...Substrate, 32...Optical waveguide, 33...Optical integrated circuit, 50...Robot system, 51...Robot hand, 60...Optical diffraction neural network section, 61,62...Mask layer, 71...Scatterer (object), 72...Marker (object), B...Target object, L...Light, S...Speckle pattern.

Claims

1. A structure that outputs a speckle pattern of light when light is incident on it, wherein the speckle pattern output from the structure changes when a physical quantity related to the structure changes due to the external environment, A physical quantity change detection device comprising: an optical neural network unit that performs neural network calculations on input light and outputs output light representing the calculation result; and a processing unit that acquires a value corresponding to the change in the physical quantity by an acquisition process that includes inputting at least a portion of the speckle pattern to the optical neural network unit.

2. The physical quantity change detection device according to claim 1, wherein the structure includes a multimode waveguide capable of propagating multiple modes of light, and when a physical quantity relating to the multimode waveguide changes, the speckle pattern output from the multimode waveguide changes.

3. The physical quantity change detection device according to claim 2, wherein the multimode waveguide is composed of multimode fibers.

4. The physical quantity change detection device according to claim 2 or 3, wherein the optical neural network section includes an optical integrated circuit having optical waveguides formed on a substrate.

5. The physical quantity change detection device according to claim 4, wherein the processing unit further comprises an electrical neural network unit disposed downstream of the optical neural network unit, and the acquisition process further includes processing the output from the optical neural network unit by the electrical neural network unit.

6. The physical quantity change detection device according to claim 2 or 3, wherein the structure further includes a multicore fiber disposed between the multimode waveguide and the optical neural network section.

7. The physical quantity change detection device according to claim 2 or 3, wherein the structure further includes a single-mode fiber disposed prior to the multimode waveguide.

8. The physical quantity change detection device according to claim 2 or 3, wherein the structure further includes a soft material member disposed around the multimode waveguide.

9. The multimode waveguide includes a plurality of waveguide portions, each capable of propagating multiple modes of light. The physical quantity change detection device according to claim 8, wherein the soft material member is arranged around the plurality of waveguide portions.

10. The physical quantity change detection device according to claim 1, wherein the structure includes a light-transmitting soft material member, and when a physical quantity relating to the soft material member changes, the speckle pattern output from the soft material member changes.

11. The physical quantity change detection device according to claim 10, wherein the processing unit further comprises an electrical neural network unit disposed downstream of the optical neural network unit, and the acquisition process further includes processing the output from the optical neural network unit by the electrical neural network unit.

12. The physical quantity change detection device according to claim 10, wherein the acquisition process consists only of inputting at least a portion of the speckle pattern to the optical neural network unit.

13. The physical quantity change detection device according to claim 10, wherein an object that scatters or attenuates light is arranged inside the soft material member.

14. The soft material member is formed in the shape of a flat plate having a main surface and side surfaces. The physical quantity change detection device according to claim 10, wherein the speckle pattern output from the main surface of the soft material member is input to the optical neural network unit.

15. The soft material member has an incident surface which is one end face in a predetermined direction, and a reflective surface which is the other end face in the predetermined direction. The physical quantity change detection device according to claim 10, wherein the light is incident on the incident surface, travels through the interior of the soft material member, is reflected by the reflective surface and returns through the interior of the soft material member, and is emitted from the incident surface as a speckle pattern and input to the optical neural network unit.

16. The physical quantity change detection device according to any one of claims 1 to 3, wherein the optical neural network section includes an optical diffraction type neural network section that utilizes the diffraction of light.

17. The optical diffraction neural network section includes a plurality of mask layers, The physical quantity change detection device according to claim 16, wherein each of the plurality of mask layers generates a predetermined diffraction in the light when the light passes through or is transmitted through the mask layer.

18. The physical quantity change detection device according to claim 16, wherein the optical diffraction type neural network section is configured to include a spatial light modulator.

19. The physical quantity change detection device according to claim 16, wherein the optical diffraction type neural network section is configured using hologram technology.

20. The physical quantity change detection device according to claim 16, wherein the optical diffraction type neural network section is configured to include a metasurface.

21. The physical quantity change detection device according to claim 16, wherein the optical diffraction type neural network section is configured to include a dynamically controllable metasurface.

22. The aforementioned optical diffraction neural network section consists of a single mask layer, The mask layer generates predetermined diffraction in the light when the light passes through or is transmitted through the mask layer. The physical quantity change detection device according to claim 16, wherein the processing unit further comprises an electrical neural network unit disposed downstream of the optical neural network unit, and the acquisition process further includes processing the output from the optical neural network unit by the electrical neural network unit.

23. The physical quantity change detection device according to any one of claims 1 to 3, wherein the processing unit further comprises an electrical neural network unit disposed downstream of the optical neural network unit, and the acquisition process further comprises processing the output from the optical neural network unit by the electrical neural network unit.

24. The physical quantity change detection device according to any one of claims 1 to 3, wherein the processing unit detects a change in one of the physical quantities by the acquisition process.

25. The physical quantity change detection device according to any one of claims 1 to 3, wherein the processing unit detects changes in a plurality of physical quantities by the acquisition process.

26. A physical quantity change detection device according to any one of claims 1 to 3, further comprising a light source that outputs the aforementioned light.

27. A physical quantity change detection device according to any one of claims 1 to 3, Equipped with a robotic hand capable of grasping an object, The physical quantity change detection device comprises a plurality of the above structures, The aforementioned multiple structures are arranged at multiple locations on the robot hand, A robotic system in which the gripping state of the object by the robot hand is identified based on the changes in the physical quantities in the plurality of structures.

28. The physical quantity change detection device comprises a plurality of processing units, The robot system according to claim 27, wherein at least a portion of the speckle pattern output from the plurality of structures is input to the optical neural network section of the plurality of processing units, respectively.

29. The robot system according to claim 27, wherein at least a portion of the speckle pattern output from the plurality of structures is input to the optical neural network section of the processing unit.

30. The robot system according to claim 27, wherein the processing unit is located within the robot hand.

31. A physical quantity change detection device according to any one of claims 1 to 3, wherein the speckle pattern is converted into at least one optical signal containing a plurality of information among intensity information, wavelength information, phase information, and polarization information contained in the speckle pattern, and the at least one optical signal is input to the optical neural network unit.

32. A structure that outputs a speckle pattern of light when light is incident on it, wherein the speckle pattern output from the structure changes when a physical quantity related to the structure changes due to the external environment, and light is incident on the structure and the structure outputs the speckle pattern, A method for detecting a change in a physical quantity, comprising: using an optical neural network unit that performs neural network calculations on input light and outputs output light representing the calculation results, and obtaining a value corresponding to the change in the physical quantity by an acquisition process that includes inputting at least a portion of the speckle pattern to the optical neural network unit.

Images