Neural network device, control method, and program

Abstract

To start an execution of a predetermined control with low power consumption.SOLUTION: A neural network device comprises a light receiving unit that receives light via an optical neural network and a control unit that executes a predetermined control triggered by detecting a predetermined object based on a signal corresponding to the light received by the light receiving unit.SELECTED DRAWING: Figure 1

Description

[Technical field]

[0001] The present technology relates to a neural network device, a control method, and a program, and in particular to a technology using an optical neural network. [Background technology]

[0002] Conventionally, a neural network device has been proposed in which a neural network including multiple intermediate layers is constructed between an input layer and an output layer, and parameters are learned using this neural network, thereby outputting a score for input information (see, for example, Patent Document 1). [Prior art documents] [Patent documents]

[0003] [Patent Document 1] Patent Publication No. 2021-162917 Summary of the Invention [Problem to be solved by the invention]

[0004] However, in the above-mentioned neural network device, the load of the processing by the neural network tends to increase, and the power consumption also increases accordingly. Therefore, when performing a predetermined control based on the result of the processing by the neural network, there is a risk of the power consumption increasing.

[0005] Therefore, an object of the present technology is to start executing a predetermined control with low power consumption. [Means for solving the problem]

[0006] The neural network device of the present technology includes a light receiving unit that receives light through an optical neural network, and a control unit that performs predetermined control in response to the detection of a predetermined object based on a signal corresponding to the light received by the light receiving unit. This makes it possible for the neural network device to operate the optical neural network, which operates in an energy-saving manner, and to stop the device to be controlled before carrying out a predetermined control. [Brief description of the drawings]

[0007] [Figure 1] FIG. 1 is a diagram illustrating a configuration of a neural network device according to a first embodiment. [Diagram 2] FIG. 1 is a diagram illustrating the optical circuit within an ODDNN. [Diagram 3] FIG. 2 is a diagram illustrating light emitted from an ODDNN. [Figure 4] 11 is a flowchart showing a procedure of an imaging process. [Diagram 5] FIG. 13 is a diagram illustrating the configuration of a neural network device according to a second embodiment. [Figure 6] FIG. 13 is a diagram illustrating the configuration of a neural network device according to a third embodiment. [Figure 7] 1 is an enlarged view of the spectroscopic section, ODDNN, and light receiving section. [Figure 8] FIG. 11 is a diagram illustrating a specific example of object detection. [Figure 9] FIG. 13 is a diagram illustrating the configuration of a neural network device according to a third embodiment. [Figure 10] FIG. 11 is a diagram illustrating a specific example of object detection. [Figure 11] 11 is a flowchart showing a procedure of an imaging process. [Figure 12] 13 is a diagram illustrating a spectroscopic unit according to a modified example. FIG. [Figure 13] FIG. 2 is a diagram illustrating a configuration of an imaging light receiving section. [Figure 14] FIG. 13 is a diagram illustrating the configuration of a modified neural network device. [Figure 15] FIG. 1 is a diagram illustrating the configuration of OCNN. [Figure 16] FIG. 1 is a diagram illustrating the configuration of OCNN. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0008] The embodiments will be described below in the following order. <1. First embodiment> <2. Second embodiment> <3. Third embodiment> <4. Fourth embodiment> <5. Example of neural network device configuration> <6. Summary of the embodiment> <7. This Technology>

[0009] <1. First embodiment> 1 is a diagram for explaining the configuration of a neural network device 1 according to a first embodiment. The neural network device 1 is a device that performs control using the detection of an object based on a signal received via an optical neural network (hereinafter, abbreviated as ONN).

[0010] Here, the ONN is a neural network formed by an optical circuit, and examples of the ONN include an optical diffractive deep neural network (hereinafter, referred to as ODDNN: Optical Diffractive Deep Neural Networks), an optical convolutional neural network (OCNN: Optical Convolutional Neural Networks), etc. In the first embodiment, a case where an ODDNN is applied as an ONN will be described.

[0011] As shown in FIG. 1, the neural network device 1 includes a half mirror 11, an ODDNN 12, a light receiving unit 13, an image sensor 14, and a control unit 15.

[0012] In the neural network device 1, external light is incident on the half mirror 11. The half mirror 11 transmits and reflects the incident light (here, light reflected by an object OB) at a certain ratio, and, for example, guides the transmitted light to the ODDNN 12 and guides the reflected light to the imaging element 14.

[0013] The ODDNN 12 is a thin, translucent, flat glass layer in which a large number of sub-wavelength-sized bubbles, graphene, and other heterogeneous phases are arranged. Although FIG. 1 illustrates a case in which the ODDNN 12 is composed of one flat glass layer, the ODDNN 12 may be composed of multiple flat glass layers spaced apart at a predetermined interval.

[0014] Fig. 2 is a diagram explaining the optical circuit in the ODDNN 12. Fig. 3 is a diagram explaining the light emitted from the ODDNN 12. Fig. 2 shows an example in which light is incident from the entrance surface on the left side of the figure and light is emitted from the exit surface on the right side of the figure.

[0015] As shown in Figure 2, when light is incident on the ODDNN12 from the entrance surface, the incident light is repeatedly reflected, diffracted, and absorbed by the heterogeneous phases present inside, and then emitted from the exit surface. At this time, in the ODDNN12, a region where a certain amount of light is concentrated is generated on the exit surface according to the incident light.

[0016] For example, as shown in FIG. 3, when light from the number "A" is incident as an object OB, on the exit surface of the ODDNN 12, the light is most concentrated in the area circled in the figure.

[0017] In this way, in the ODDNN 12, the area where the light is concentrated on the exit surface side differs for each light incident thereon, that is, for each light reflected by the object OB and incident thereon (also simply referred to as the light of the object OB). In other words, in the ODDNN 12, the intensity distribution of the light emitted from the exit surface differs for each light of the object OB incident thereon.

[0018] Therefore, in the ODDNN 12, light from one or more objects OB is made incident in advance, and the type of light intensity distribution at the exit surface is learned. This makes it possible for the neural network device 1 to detect the learned object OB as a target object based on the learning result and the intensity distribution of light emitted from the exit surface of the ODDNN 12 when light from an unknown object OB is incident thereon.

[0019] In this way, the ODDNN 12 functions as a neural network, but since it does not require a power source, electronic circuits, sensors, etc., it is possible to build an energy-saving, low-load neural network.

[0020] In addition, since the incident light is emitted after being repeatedly reflected, diffracted, and absorbed, the ODDNN 12 is capable of operating at the speed of light.

[0021] The light receiving unit 13 is disposed opposite to the emission surface of the ODDNN 12. The light receiving unit 13 is a two-dimensional array of multiple light receiving elements (e.g., diodes) so that the light emitted from the emission surface of the ODDNN 12 can be detected for each predetermined range. The light receiving unit 13 outputs to the control unit 15 a signal according to the intensity of the light received by each light receiving element.

[0022] However, the light receiving unit 13 only needs to have a resolution capable of detecting an object based on the light emitted from the emission surface of the ODDNN 12, and is composed of light receiving elements with a significantly smaller number of pixels than the image sensor 14. Therefore, the light receiving unit 13 can operate with less energy than the image sensor 14.

[0023] The imaging element 14 is, for example, a CCD (Charge Coupled Device) type or CMOS (Complementary Metal-Oxide-Semiconductor) type image sensor, and has a plurality of pixels having photoelectric conversion elements arranged two-dimensionally. The imaging element 14 captures an image of a predetermined imaging range through the half mirror 11 at regular intervals according to a frame rate to generate image data. The image captured by the imaging element 14 may be either a still image or a moving image, or may be interval imaging, etc.

[0024] The control unit 15 is configured with, for example, a microcomputer having a CPU (Central Processing Unit), a ROM (Read Only Memory), and a RAM (Random Access Memory), and performs overall control of the neural network device 1. The control unit 15 functions as a sensor control unit 21 and an imaging control unit 22 in the first embodiment.

[0025] The sensor control unit 21 detects a predetermined object using a learning result learned in advance based on the signal input from the light receiving unit 13, i.e., the light intensity distribution on the exit surface of the ODDNN 12. The imaging control unit 22 causes the imaging element 14 to start imaging using the detection of the predetermined object as a trigger.

[0026] The sensor control unit 21 also detects and sets a range in which an object is detected as a ROI (Region Of Interest) based on a signal input from the light receiving unit 13. Then, the imaging control unit 22 causes the imaging element 14 to perform an imaging operation targeting only the ROI.

[0027] Here, as described above, the ODDNN 12 can detect an object based on the light intensity distribution at the exit surface. The ODDNN 12 can also detect the approximate position of the object based on the light intensity distribution at the exit surface. Therefore, in the neural network device 1, light for objects at different positions is made incident on the ODDNN 12, and the type of light intensity distribution at the exit surface is learned in advance. The learning results are stored in the ROM or RAM of the control unit 15. The sensor control unit 21 then detects the ROI of the object based on the signal input from the light receiving unit 13 and the learning results.

[0028] Thereafter, when the imaging device 14 captures an image of an object, the imaging control unit 22 performs a predetermined image analysis on the object and calculates the ROI of the object. Then, for the next frame and thereafter, the imaging control unit 22 performs image analysis on the captured image of the ROI calculated in the previous frame to perform object recognition processing, and calculates the ROI for the recognized object. Note that the image analysis for setting the ROI can use a known analysis method, so a description thereof will be omitted.

[0029] By using this method, the detection of a specific object is used as a trigger to start imaging by the image sensor 14, so that until the specific object is detected, only the light receiving unit 13, which consumes less power than the image sensor 14, needs to be operated, thereby making it possible to save energy.

[0030] 4 is a flow chart showing the procedure of the imaging process. As shown in FIG. 4, in step S1, the imaging control unit 22 stops the imaging element 14, and the sensor control unit 21 starts the operation of the light receiving unit 13. In the following step S2, the sensor control unit 21 executes an object detection process to detect an object based on a signal input from the light receiving unit 13. Here, based on the signal input from the light receiving unit 13, i.e., the light intensity distribution on the exit surface of the ODDNN 12, it is determined whether an object has been detected using a learning result learned in advance.

[0031] Then, in step S3, the sensor control unit 21 determines whether or not a predetermined object has been detected in step S2. If a predetermined object has not been detected (No in step S3), the sensor control unit 21 repeats the processes of steps S2 and S3.

[0032] On the other hand, if a specified object is detected (Yes in step S3), in step S4, the imaging control unit 22 sets the range in which the specified object is detected as the ROI based on the signal input from the light receiving unit 13.

[0033] After that, in step S5, the imaging control unit 22 causes the imaging element 14 to start imaging only the ROI set in step S4.

[0034] <2. Second embodiment> 5 is a diagram for explaining the configuration of a neural network device 100 according to the second embodiment. Note that in the second embodiment, configurations different from those in the first embodiment will be explained in detail, and the same configurations will be denoted by the same reference numerals and explanations thereof will be omitted.

[0035] As shown in FIG. 5, the neural network device 100 includes a half mirror 101, an ODDNN 102, and a light receiving unit 103 in addition to the half mirror 11, the ODDNN 12, the light receiving unit 13, the image sensor 14, and the control unit 15.

[0036] In the neural network device 100 , a half mirror 101 and an ODDNN 102 are disposed between the ODDNN 12 and the light receiving section 13 .

[0037] The light emitted from the ODDNN 12 is incident on the half mirror 101. The half mirror 101 transmits and reflects the incident light at a constant ratio, and for example, guides the transmitted light to the ODDNN 102 and guides the reflected light to the light receiving unit 103.

[0038] The ODDNN 102, like the ODDNN 12, is a thin, translucent, flat glass layer in which a large number of sub-wavelength-sized bubbles, graphene, and other heterogeneous phases are arranged. One or more ODDNNs 102 are provided, and in the example of FIG. 5, four ODDNNs 102 are provided. Since light emitted from the output surface of ODDNN12 is incident on ODDNN102, when learning the light intensity distribution of the target object, the learning is performed in a state where light emitted from the output surface of ODDNN12 is incident on ODDNN102.

[0039] The ODDNN 12 and the ODDNN 102 are configured to detect different objects from each other. In addition, when multiple ODDNN 102 are provided, the multiple ODDNN 102 may be configured to detect different objects from each other, or multiple ODDNN 102 may be configured to detect one target portion.

[0040] For example, the ODDNN 12 is configured to be able to detect objects more roughly than the ODDNN 102. Specifically, the ODDNN 12 is configured to be able to detect vehicles including automobiles, trucks, motorcycles, and bicycles. On the other hand, the ODDNN 102 is configured to be able to detect detailed objects such as automobiles, trucks, motorcycles, and bicycles.

[0041] The light receiving unit 103 is a two-dimensional array of a plurality of light receiving elements (e.g., diodes) so that the light emitted from the emission surface of the ODDNN 12 can be detected in each predetermined range. The light receiving unit 103 outputs a signal to the control unit 15 according to the intensity of the light received by each light receiving element.

[0042] The control unit 15 is configured to include, for example, a microcomputer having a CPU, ROM, and RAM, and performs overall control of the neural network device 100. The control unit 15 functions as a sensor control unit 21 and an imaging control unit 22 in the second embodiment.

[0043] The sensor control unit 21 detects a predetermined object using a learning result learned in advance based on the signal input from the light receiving unit 13, i.e., the light intensity distribution on the exit surface of the ODDNN 102. Therefore, here, it is determined whether a detailed object (e.g., a car) has been detected. The imaging control unit 22 causes the imaging element 14 to start imaging using the detection of the predetermined object as a trigger.

[0044] Further, the sensor control unit 21 detects and sets, as the ROI, a range in which a rough target object (e.g., a vehicle) is detected based on a signal input from the light receiving unit 103. Then, the imaging control unit 22 causes the imaging element 14 to perform an imaging operation targeting only the ROI.

[0045] Thereafter, when an object is imaged by the image sensor 14, the imaging control unit 22 performs a predetermined image analysis on the object and calculates the ROI of the object. Then, even in the next frame and thereafter, the imaging control unit 22 performs image analysis on the captured image of the ROI calculated in the previous frame, performs object recognition processing, and calculates the ROI for the recognized object.

[0046] By using such a method, the neural network device 100 can differentiate between the object that is the target of the trigger and the object that is the target of the ROI, making it possible to perform imaging that focuses on an area that is not necessarily the object that is the target of the trigger. Furthermore, the neural network device 100 also makes it possible to capture an image focused on an object (other vehicle) other than the object (automobile) that is the subject of the trigger. Furthermore, in the neural network device 100, the detection of a specific object is used as a trigger to start imaging by the image sensor 14. Therefore, until the specific object is detected, it is necessary to operate only the light receiving units 13, 103, which consume less power than the image sensor 14, thereby enabling energy savings.

[0047] It should be noted that the imaging process in the second embodiment has the same flow as that in the first embodiment, and therefore the description thereof will be omitted.

[0048] <3. Third embodiment> 6 is a diagram for explaining the configuration of a neural network device 200 according to the third embodiment. Note that in the third embodiment, configurations different from those in the first embodiment will be explained in detail, and the same configurations will be denoted by the same reference numerals and explanations thereof will be omitted.

[0049] As shown in FIG. 6, the neural network device 200 includes a spectroscopic unit 201 in addition to the half mirror 11, the ODDNN 12, the light receiving unit 13, the image sensor 14, and the control unit 15.

[0050] In the neural network device 200, external light is incident on the half mirror 11. The half mirror 11 guides the transmitted light to the spectroscopic section 201, and guides the reflected light to the imaging element .

[0051] The spectroscopic unit 201 is a device that splits the incident light, and is a prism in the third embodiment. The spectroscopic unit 201 splits the incident light by utilizing the difference in refractive index for each wavelength. The split light is incident on the ODDNN 12.

[0052] 7 is an enlarged view of the spectroscopic unit 201, the ODDNN 12, and the light receiving unit 13. When light of different wavelength regions is incident on one ODDNN 12, crosstalk due to the light of different wavelength regions may occur within the ODDNN 12, resulting in a decrease in the detection accuracy of the target object.

[0053] Therefore, in the third embodiment, a plurality of ODDNNs 12 (six in FIG. 7) are provided. Specifically, light in different wavelength ranges (for example, light in first to sixth wavelength ranges) is incident on the plurality of ODDNNs 12a to 12f, respectively.

[0054] In this case, the ODDNNs 12a to 12f are selected to be optimized for the incident wavelength range light. Here, "optimization" refers to adjusting the arrangement of the different phases in the flat glass layers, the number of flat glass layers, and the intervals between the flat glass layers so that the detection accuracy of the target object based on the incident wavelength range light is higher than that based on the light of other wavelength ranges.

[0055] For example, the ODDNN 12a optimized for light in a first wavelength range is arranged so that light in the first wavelength range is incident thereon. Similarly, the ODDNNs 12b to 12f optimized for light in the second to sixth wavelength ranges are arranged so that light in the second to sixth wavelength ranges is incident thereon. This makes it possible for the ODDNNs 12a to 12f to reduce the influence of crosstalk caused by the incidence of light in different wavelength ranges, and to further improve the detection accuracy of the target object.

[0056] In the neural network device 200, light from an object that mainly reflects light in the corresponding wavelength range is made incident on each of the ODDNNs 12a to 12f, and the type of light intensity distribution region on the exit surface is learned in advance. The learning result is stored in the ROM or RAM of the control unit 15.

[0057] The light receiving units 13 are provided in the same number as the ODDNNs 12, and are arranged to face the emission surfaces of the ODDNNs 12. That is, the light receiving unit 13a is arranged to face the emission surface of the ODDNNs 12a. Similarly, the light receiving units 13b to 13f are arranged to face the emission surfaces of the ODDNNs 12b to 12f, respectively. The light emitted from the ODDNNs 12a to 12f is guided to the corresponding light receiving units 13a to 13f.

[0058] In the third embodiment, the control unit 15 functions as a sensor control unit 21 and an imaging control unit 22.

[0059] The sensor control unit 21 detects a predetermined object based on the signal input from the light receiving unit 13, that is, the light intensity distribution on the exit surface of the ODDNN 12, using a learning result previously learned.

[0060] FIG. 8 is a diagram for explaining a specific example of object detection. As shown in FIG. 8, assume that a person, a tree, a car, a house, a dog, and a bicycle are set as objects. In this case, for example, the person and the dog mainly reflect light in a first wavelength range. The tree mainly reflects light in a second wavelength range. The car and the bicycle mainly reflect light in a third wavelength range. The house mainly reflects light in a fourth wavelength range.

[0061] Therefore, for example, when the target object is an automobile, the sensor control unit 21 detects the automobile based on a signal input from the light receiving unit 13a corresponding to the ODDNN 12a optimized for light in the first wavelength range.

[0062] For example, if the target object is a bicycle with a person on it, the sensor control unit 21 detects the bicycle based on a signal input from the light receiving unit 13c corresponding to the ODDNN 12c optimized for light in the third wavelength range. The sensor control unit 21 detects the person based on a signal input from the light receiving unit 13a corresponding to the ODDNN 12a optimized for light in the first wavelength range. The sensor control unit 21 then determines that the bicycle is a bicycle with a person on it based on the positional relationship between the detected bicycle and the person.

[0063] In this way, the neural network device 200 can detect an object based on the intensity distribution of light emitted from the emission surfaces of the ODDNNs 12 optimized for different wavelength ranges. At this time, in the neural network device 200, light in an optimized wavelength range is made incident on the ODDNNs 12, so crosstalk with light in other wavelength ranges can be reduced and the accuracy of object detection can be improved.

[0064] The imaging control unit 22 causes the imaging element 14 to start imaging when a predetermined object (for example, an automobile) is detected as a trigger.

[0065] The sensor control unit 21 detects and sets, as an ROI, a range in which the object is detected based on a signal input from the light receiving unit 13 corresponding to the ODDNN 12 optimized for light mainly reflected by the object. The imaging control unit 22 then causes the imaging element 14 to perform an imaging operation targeting only the ROI.

[0066] Thereafter, when an object is imaged by the image sensor 14, the imaging control unit 22 performs a predetermined image analysis on the object and calculates the ROI of the object. Then, even in the next frame and thereafter, the imaging control unit 22 performs image analysis on the captured image of the ROI calculated in the previous frame, performs object recognition processing, and calculates the ROI for the recognized object.

[0067] The imaging process in the third embodiment is the same as that in the first embodiment, and therefore the description thereof will be omitted. In steps S2 and S4, the object or ROI is detected based on a signal input from the light receiving unit 13a corresponding to the ODDNN 12a optimized for light in a wavelength range that is mainly reflected by the object.

[0068] <4. Fourth embodiment> 9 is a diagram for explaining the configuration of a neural network device 300 according to the fourth embodiment. Note that in the fourth embodiment, configurations different from those in the first and third embodiments will be explained in detail, and the same configurations will be denoted by the same reference numerals and explanations thereof will be omitted.

[0069] 9, the neural network device 300 includes an irradiation unit 301 in addition to a half mirror 11, an ODDNN 12, a light receiving unit 13, an image sensor 14, and a control unit 15. In the fourth embodiment, the ODDNN 12 includes a plurality of ODDNNs 12a-12f optimized for light in the incident wavelength range, as in the third embodiment. The light receiving unit 13 includes a plurality of light receiving units 13a-13f that receive light emitted from the emission surfaces of the ODDNNs 12a-12f.

[0070] The irradiating unit 301 is driven under the control of the control unit 15 and can irradiate light in a predetermined wavelength range. The irradiating unit 301 can also irradiate light in different wavelength ranges (or wavelengths) by switching between them. The irradiating unit 301 can also simultaneously irradiate light in multiple wavelength ranges, such as visible light.

[0071] When the light emitted from the irradiation unit 301 is reflected by the object OB, the reflected light is incident on the half mirror 11. The half mirror 11 guides the transmitted light to the ODDNN 12, and guides the reflected light to the imaging element .

[0072] In the fourth embodiment, the control unit 15 functions as a sensor control unit 21, an imaging control unit 22, and an irradiation control unit 23.

[0073] The irradiation control unit 23 causes the irradiation unit 301 to irradiate light in a predetermined wavelength range. The sensor control unit 21 detects the object based on a signal input from the light receiving unit 13 corresponding to the ODDNN 12 optimized for the wavelength range of the irradiated light.

[0074] FIG. 10 is a diagram for explaining a specific example of object detection. In the example shown in FIG. 10, the same object as in FIG. 8 is set. As shown in FIG. 10, for example, when an automobile is set as the object, the irradiation control unit 23 causes the irradiation unit 301 to irradiate light in the third wavelength range, and the sensor control unit 21 detects the automobile based on a signal detected from the light receiving unit 13c corresponding to the ODDNN 12c optimized for light in the third wavelength range. At this time, the light receiving unit 13 corresponding to the ODDNN 12 optimized for light other than the third wavelength range may be stopped.

[0075] When the target object is a bicycle with a person on it, the irradiation control unit 23 causes the irradiation unit 301 to irradiate light in the third wavelength range, and the sensor control unit 21 detects the bicycle based on a signal input from the light receiving unit 13c corresponding to the ODDNN 12c optimized for the light in the third wavelength range. The irradiation control unit 23 also switches the irradiation unit 301 to irradiate light in the first wavelength range, and the sensor control unit 21 detects the person based on a signal input from the light receiving unit 13a corresponding to the ODDNN 12a optimized for the light in the first wavelength range. At this time, the sensor control unit 21 determines that the bicycle is ridden by a person based on the positional relationship between the detected bicycle and the person.

[0076] The imaging control unit 22 causes the imaging element 14 to start imaging when a predetermined object is detected as a trigger.

[0077] The sensor control unit 21 also detects and sets, as an ROI, a range in which a predetermined object is detected based on a signal input from the light receiving unit 13. Then, the imaging control unit 22 causes the imaging element 14 to perform an imaging operation targeting only the ROI.

[0078] Thereafter, when an object is imaged by the image sensor 14, the imaging control unit 22 performs a predetermined image analysis on the object and calculates the ROI of the object. Then, even in the next frame and thereafter, the imaging control unit 22 performs image analysis on the captured image of the ROI calculated in the previous frame, performs object recognition processing, and calculates the ROI for the recognized object.

[0079] 11 is a flowchart showing the procedure of the imaging process. As shown in FIG 11, in step S11, the imaging control unit 22 stops the imaging element 14, and the sensor control unit 21 starts the operation of the light receiving unit 13.

[0080] In the next step S12, the irradiation control unit 23 causes the irradiation unit 301 to irradiate light in a wavelength range that is mainly reflected by the object to be detected. Then, in step S13, the sensor control unit 21 executes an object detection process to detect the object based on a signal input from the light receiving unit 13 corresponding to the ODDNN 12 optimized for the light irradiated from the irradiation unit 301.

[0081] Then, in step S14, the sensor control unit 21 determines whether or not an object was detected in step S12. If a predetermined object is not detected (No in step S14), the sensor control unit 21 repeats the processes of steps S13 and S14.

[0082] On the other hand, if a specified object is detected (Yes in step S14), in step S15, the sensor control unit 21 sets the range in which the specified object is detected as the ROI based on the signal input from the light receiving unit 13.

[0083] After that, in step S16, the imaging control unit 22 causes the imaging element 14 to start imaging only of the ROI set in step S14.

[0084] <5. Other configuration examples of neural network device> It should be noted that the embodiment is not limited to the specific example described above, and various modified configurations may be adopted.

[0085] In the above embodiment, detection of an object based on a signal corresponding to light received by the light receiving unit 13 is used as a trigger to start imaging by the image sensor 14. However, the neural network device may be configured to perform predetermined control other than imaging using detection of an object based on a signal corresponding to light received by the light receiving unit 13 as a trigger.

[0086] In the above embodiment, the ROI is set based on a signal corresponding to the light received by the light receiving unit 13, but the image sensor 14 may be caused to capture an image without setting the ROI.

[0087] In the third embodiment described above, the spectroscopic unit 201 that spectroscopically separates light is provided as a light guiding unit that guides light in the optimized wavelength range to the ODDNN 12. In the fourth embodiment described above, the irradiation unit 301 that can irradiate light in a predetermined wavelength range is provided as a light guiding unit that guides light in the optimized wavelength range to the ODDNN 12. However, the light guiding unit may be something other than the spectroscopic unit 201 and the irradiation unit 301 as long as it can guide light in the optimized wavelength range to the ODDNN 12.

[0088] In the third embodiment described above, a prism is used as the spectroscopic unit 201. However, the spectroscopic unit 201 is not limited to a prism as long as it can split light. For example, as shown in FIG. 12, the spectroscopic unit 201 may be a diffraction grating that splits light.

[0089] In the third and fourth embodiments described above, a plurality of ODDNNs 12 optimized for different wavelength regions are provided. However, only one ODDNN 12 optimized for any one of the wavelength regions may be provided. In this case, only one light receiving unit 13 may be provided.

[0090] In the above-described embodiment, the light receiving unit 13 and the image sensor 14 are provided as separate bodies. However, the light receiving unit 13 and the image sensor 14 may be stacked. For example, as shown in Fig. 13, the image receiving unit 400 includes a dual bandpass filter 401 and an image sensor 402. The dual bandpass filter 401 is a filter that passes, for example, visible light and infrared light and cuts light in other wavelength ranges. Light that has passed through the dual bandpass filter 401 is incident on the image sensing element 402. The image sensing element 402 is packaged (integrated) by stacking an image sensing element section 410 and an ODDNN section 420. The imaging element section 410 includes, for example, color filters 411 arranged in a Bayer array and imaging elements (diodes) 412 corresponding to the color filters 411. This allows the imaging element section 410 to capture a full-color image based on the visible light out of the visible light and infrared light that have passed through the dual bandpass filter 401. Furthermore, the imaging element section 410 transmits the visible light out of the visible light and infrared light that have passed through the dual bandpass filter 401. The ODDNN unit 420 includes an ODDNN 421 and a light receiving unit 422. The ODDNN 421 is optimized for infrared light, and is configured to be able to detect an object based on the infrared light incident thereon after passing through the imaging element unit 410. The light receiving unit 422 is disposed opposite the emission surface of the ODDNN 421, and outputs a signal according to the intensity of the light emitted from the emission surface of the ODDNN 421 to the control unit 15. Such a configuration makes it possible to reduce the size of the neural network device.

[0091] Also, an ANN (Artificial Neural Network) may be executed using the light emitted from the emission surface of the ODDNN 12. The ANN may be a known DNN (Deep Neural Network), CNN (Convolutional Neural Network), RNN (Recurrent Neural Network), SNN (Spiking Neural Networks), or the like. For example, as shown in FIG. 14, a neural network device 500 includes a half mirror 11, an ODDNN 12, a light receiving unit 13, an image sensor 14, a control unit 15, and an irradiation unit 301, similar to the fourth embodiment.

[0092] The control unit 15 functions as a sensor control unit 21, an imaging control unit 22, an irradiation control unit 23, and an ANN unit 24. The ANN unit 24 executes an ANN based on a signal input from the light receiving unit 13, i.e., the intensity distribution of light emitted from the emission surface of the ODDNN 12.

[0093] In the neural network device 500 having such a configuration, first, as in the above-described embodiment, the object detection process using the ODDNN 12 is executed. For example, in the object detection process using the ODDNN 12, the object is roughly detected. Specifically, the sensor control unit 21 detects vehicles including passenger cars, trucks, motorcycles, and bicycles.

[0094] The ANN unit 24 performs detailed detection of the object by executing the ANN based on the intensity distribution of the light emitted from the emission surface of the ODDNN 12. The ANN unit 24 performs detection to distinguish between, for example, passenger cars, trucks, motorcycles, bicycles, etc. As a result, the neural network device 500 can reduce the processing load and power consumption caused by the ANN, and perform target object detection at high speed and with low power consumption.

[0095] In the above embodiment, the case where the ODDNN12 is used as the ONN has been described. However, the OCNN may be used as the ONN. When the OCNN is used, the composition (optical characteristics) of the OCNN may be configured to be changeable. 15, OCNN 600 has two optical neural network layers 601 and two LCD (Liquid Crystal Display) layers 602 arranged alternately. In OCNN 600, the control unit 15 controls the light transmittance of each pixel of LCD layer 602, thereby making it possible to change the composition. 16, the OCNN 700 has a plurality of metalens optical CNN neurons 701 arranged therein, and a MEMS (Micro Electro Mechanical Systems) actuator 702 is provided on the outer periphery of each metalens optical CNN neuron 701. In the OCNN 700, the composition of the metalens optical CNN neuron 701 can be changed by the control unit 15 controlling the MEMS actuator 702. In this way, by using the OCNNs 600 and 700 whose composition can be changed, it is possible to set optimal compositions for different wavelength ranges without providing multiple ODDNNs 12, and the size of the device can be reduced.

[0096] <6. Summary of the embodiment> As described above, the neural network device of the embodiment (1, 100, 200, 300) includes a light receiving unit 13 that receives light via an optical neural network (ODDNN 12), and a control unit 15 that performs predetermined control in response to detection of a predetermined object based on a signal corresponding to the light received by the light receiving unit 13. This makes it possible for the neural network device to operate the optical neural network, which operates in an energy-saving manner, and to stop the device to be controlled before carrying out a predetermined control. Therefore, the neural network device can start executing a predetermined control with low power consumption.

[0097] Furthermore, the control unit 15 starts imaging by the imaging element 14 when a target object is detected based on a signal corresponding to the light received by the light receiving unit 13 as a trigger. This allows the neural network device to reduce power consumption since it does not need to operate the image sensor 14 until a specific object is detected.

[0098] Furthermore, based on a signal corresponding to the light received by the light receiving unit 13, the control unit 15 sets the range in which a predetermined object is detected as a region of interest. This makes it possible for the neural network device to capture an image of the range in which the target object exists.

[0099] Furthermore, the control unit 15 causes the imaging element to capture an image of the region of interest as an imaging range. This makes it possible for the neural network device to capture an image of the range in which the target object exists.

[0100] In addition, multiple optical neural networks (ODDNN12, 102) are provided, and a specified object is detected based on a signal corresponding to light received by the light receiving unit 13 via one of the optical neural networks (ODDNN102), and a region of interest is detected based on a signal corresponding to light received by the light receiving unit via the other optical neural network (ODDNN12). This allows the neural network device to capture images of objects other than the object that serves as the trigger.

[0101] In addition, the optical neural network is an optical diffraction deep neural network (ODDNN12). As a result, in the neural network device, no power is consumed by the optical neural network, thereby enabling low power consumption.

[0102] In addition, the optical neural network is an optical convolutional neural network (OCNN 600, 700) whose optical properties can be changed. This makes it possible to set optimal compositions for different wavelength ranges in the neural network device without providing multiple ODDNNs 12, thereby enabling the device to be made more compact.

[0103] The neural network device also includes an irradiation unit 301 capable of irradiating light in a predetermined wavelength range. This enables the neural network device to improve the accuracy of the wavelength range of light guided to the ODDNN 12, thereby improving the accuracy of detecting an object.

[0104] Moreover, the irradiating unit 301 is capable of irradiating light in a plurality of wavelength ranges, and the optical neural network is optimized for each of the plurality of wavelength ranges that the irradiating unit 301 is capable of irradiating. As a result, even if there is an object that mainly reflects different wavelength ranges, the neural network device can accurately detect the object by using the ODDNN 12 corresponding to each wavelength range.

[0105] In addition, the optical neural network (ODDNN12) is optimized for light in the wavelength range that is primarily reflected by the object, and the irradiating unit 301 irradiates light in the wavelength range that is primarily reflected by the object to be detected, and the control unit 15 detects the object based on a signal input from the light receiving unit 13 that receives the light emitted from the optical neural network optimized for light in the wavelength range that is primarily reflected by the object.

[0106] The neural network device also includes a spectroscopic section 201 that disperses light and guides it to the optical neural network. This allows the neural network device to guide light in an optimized wavelength range to the ODDNN 12 with a simple configuration and without consuming power.

[0107] Also, an imaging element (imaging element section 410) and an optical neural network (ODDNN section 420) are stacked. This allows space saving and miniaturization to be achieved.

[0108] The imaging element (imaging element section 410) receives visible light, and the optical diffraction deep neural network (ODDNN section 420) receives infrared light that has passed through the imaging element. This makes it possible to capture images and detect objects in different wavelength ranges, thereby enabling space saving and miniaturization.

[0109] In the detection method according to the present technology described above, control is triggered by detection of a predetermined object based on a signal corresponding to light received by a light receiving unit via an optical neural network. In the program related to the present technology described above, the neural network device is caused to execute a control process triggered by the detection of a specified object based on a signal corresponding to light received by a light receiving unit via an optical neural network.

[0110] Such a program can be recorded in advance in a HDD as a recording medium built into a device such as a computer device, or in a ROM within a microcomputer having a CPU. Alternatively, the software may be temporarily or permanently stored (recorded) on a removable recording medium such as a flexible disk, a CD-ROM (Compact Disc Read Only Memory), an MO (Magneto Optical) disk, a DVD (Digital Versatile Disc), a Blu-ray Disc (registered trademark), a magnetic disk, a semiconductor memory, a memory card, etc. Such removable recording media may be provided as a so-called package software. Furthermore, such a program can be installed in a personal computer or the like from a removable recording medium, or can be downloaded from a download site via a network such as a LAN (Local Area Network) or the Internet.

[0111] In addition, such a program is suitable for providing the neural network device of the embodiment in a wide range of applications. For example, by downloading the program to a mobile terminal device such as a smartphone or tablet, a mobile phone, a personal computer, a game device, a video device, a PDA (Personal Digital Assistant), or the like, the neural network device of the present technology can be made to function.

[0112] It should be noted that the effects described in this specification are merely examples and are not limiting, and other effects may also be obtained.

[0113] <7. This Technology> The present technology can also be configured as follows. (1) a light receiving unit that receives light via an optical neural network; a control unit that performs a predetermined control in response to a detection of a predetermined object based on a signal corresponding to the light received by the light receiving unit; A neural network device comprising: (2) The control unit is The detection of an object based on a signal corresponding to the light received by the light receiving unit is used as a trigger to start imaging with the imaging element. A neural network device according to (1). (3) The control unit sets a range in which a predetermined object is detected as a region of interest based on a signal corresponding to the light received by the light receiving unit. A neural network device according to (2). (4) The control unit is The region of interest is imaged by the imaging element as an imaging range. A neural network device according to (3). (5) The optical neural network is provided in plurality, A predetermined object is detected based on a signal corresponding to the light received by the light receiving unit via any one of the optical neural networks, and the region of interest is detected based on a signal corresponding to the light received by the light receiving unit via the other optical neural network. A neural network device according to (3) or (4). (6) The optical neural network is an optical diffractive deep neural network. A neural network device according to any one of (1) to (5). (7) The optical neural network is an optical convolutional neural network whose optical properties can be changed. A neural network device according to any one of (1) to (5). (8) Equipped with an irradiation unit capable of irradiating light in a predetermined wavelength range A neural network device according to any one of (1) to (7). (9) The irradiation unit is capable of irradiating light in a plurality of wavelength ranges, The optical neural network is optimized for each of a plurality of wavelength ranges that the irradiation unit can irradiate. A neural network device according to (8). (10) The optical neural network is optimized for light in a wavelength range that is primarily reflected by the object, The irradiating unit irradiates light in a wavelength range that is mainly reflected by an object to be detected, The control unit detects the object based on a signal input from the light receiving unit that receives light emitted from the optical neural network optimized for light in a wavelength range that is mainly reflected by the object. A neural network device according to (9). (11) A spectroscopic unit that disperses light and guides it to the optical neural network. A neural network device according to any one of (1) to (7). (12) The imaging element and the optical neural network are stacked. A neural network device according to any one of (2) to (11). (13) The imaging element receives visible light, The optical neural network receives infrared light that has passed through the image sensor. A neural network device according to (12). (14) Control is triggered by the detection of a specific target based on a signal corresponding to the light received by the light receiving unit via an optical neural network. Control methods. (15) Control is triggered by the detection of a specific target based on a signal corresponding to the light received by the light receiving unit via an optical neural network. A program that causes a neural network device to execute processing. [Explanation of symbols]

[0114] 1, 100, 200, 300 Neural network device 12 ODDNN 13 Light receiving part 14 Image sensor 15 Control section 201 Spectroscopy section 301 Irradiation unit

Claims

1. a light receiving unit that receives light via an optical neural network; a control unit that performs a predetermined control in response to a detection of a predetermined object based on a signal corresponding to the light received by the light receiving unit; A neural network device comprising:

2. The control unit is The detection of the object based on a signal corresponding to the light received by the light receiving unit is used as a trigger to start imaging with the imaging element.

2. The neural network device according to claim 1.

3. The control unit sets a range in which a predetermined object is detected as a region of interest based on a signal corresponding to the light received by the light receiving unit.

3. The neural network device according to claim 2.

4. The control unit is The region of interest is imaged by the imaging element as an imaging range.

4. The neural network device according to claim 3.

5. The optical neural network is provided in plurality, A predetermined object is detected based on a signal corresponding to the light received by the light receiving unit via one of the optical neural networks, and the region of interest is detected based on a signal corresponding to the light received by the light receiving unit via the other optical neural network.

4. The neural network device according to claim 3.

6. The optical neural network is an optical diffractive deep neural network.

2. The neural network device according to claim 1.

7. The optical neural network is an optical convolutional neural network whose optical properties can be changed.

2. The neural network device according to claim 1.

8. Equipped with an irradiation unit capable of irradiating light in a predetermined wavelength range 2. The neural network device according to claim 1.

9. The irradiation unit is capable of irradiating light in a plurality of wavelength ranges, The optical neural network is optimized for each of a plurality of wavelength ranges that the irradiation unit can irradiate.

9. The neural network device according to claim 8.

10. The optical neural network is optimized for light in a wavelength range that is primarily reflected by the object, The irradiating unit irradiates light in a wavelength range that is mainly reflected by an object to be detected, The control unit detects the object based on a signal input from the light receiving unit that receives light emitted from the optical neural network optimized for light in a wavelength range that is mainly reflected by the object.

10. The neural network device of claim 9.

11. A spectroscopic unit that disperses light and guides it to the optical neural network.

2. The neural network device according to claim 1.

12. The imaging element and the optical neural network are stacked.

3. The neural network device according to claim 2.

13. The imaging element receives visible light, The optical neural network receives infrared light that has passed through the image sensor.

13. The neural network device of claim 12.

14. Control is triggered by the detection of a specific target based on a signal corresponding to the light received by the light receiving unit via an optical neural network. Control methods.

15. Control is triggered by the detection of a specific target based on a signal corresponding to the light received by the light receiving unit via an optical neural network. A program that causes a neural network device to execute processing.

Images