Optical convolution calculation device and operation method thereof

Abstract

Disclosed is an optical convolution calculation apparatus including: an image-based spatial optical modulator (SLM) configured to modulate multi-wavelength light based on spatial domain image data and output modulated light; an optical demultiplexer configured to output first to Nth sub-lights having first to Nth wavelengths, respectively, based on the modulated light; an optical convolution processor configured to receive the first to Nth sub-lights and output the first to Nth inverse-transformed lights; and an optical multiplexer configured to align paths of the first to the Nth inverse-transformed light.

Description

Technical Field

[0001] The embodiments of the present disclosure described herein relate to an optical convolution computing device, and more particularly, to an optical convolution computing device and a method of operating the optical convolution computing device. Background Art

[0002] Convolutional neural network (CNN) is an artificial neural network structure used for image classification and reasoning. With the development of wireless communication, convolutional neural network is widely used in fields that require image processing devices, such as autonomous driving and the Internet of Things. The calculations required for convolutional neural network are becoming more and more complex, and the amount of calculation is also increasing, but the energy efficiency of electronic computers is low and the calculation speed is low.

[0003] The optical convolution computing device can optically perform convolution computing of a convolutional neural network. A typical optical convolution computing device utilizes an optical 4f system that uses a single wavelength coherent light source. The optical convolution computing device uses two spatial optical modulators (SLMs), and each SLM can receive multiple spatial domain image data and one Fourier kernel data.

[0004] In addition, when an optical convolution computing device using a single-wavelength coherent light source performs calculations in parallel by applying multiple cores, the number of image sensors required to identify each calculation result of the parallel calculation increases, or the accuracy of the calculation results decreases due to interference when synthesizing the calculation results. Summary of the invention

[0005] Embodiments of the present disclosure relate to an optical convolution computing device and a method of operating the optical convolution computing device.

[0006] According to an embodiment of the present disclosure, an optical convolution computing device includes: an image-based spatial optical modulator (SLM), configured to modulate multi-wavelength light based on spatial domain image data and output modulated light; an optical demultiplexer, configured to output first to Nth sub-lights having first to Nth wavelengths, respectively, based on the modulated light; an optical convolution processor, configured to receive the first to Nth sub-lights and output first to Nth inverse-transformed lights; and an optical multiplexer, configured to align paths from the first to Nth inverse-transformed lights, and N is a natural number greater than or equal to 2.

[0007] According to an embodiment, the optical convolution computing device further includes: a multi-wavelength light generator configured to output multi-wavelength light having a first wavelength to an Nth wavelength.

[0008] According to an embodiment, the optical convolution computing device further includes: an image sensor configured to sense the synthesized light.

[0009] According to an embodiment, the optical convolution computing device also includes: a multi-wavelength light generator, configured to output multi-wavelength light having a first wavelength to an Nth wavelength; and an image sensor, configured to sense synthetic light, wherein the optical convolution processor includes: a first transformation device to an Nth transformation device, configured to perform Fourier transform on the first sub-light to the Nth sub-light, respectively, and output the first transformed light to the Nth transformed light, respectively; a first core-based SLM to an Nth core-based SLM, configured to receive the first core data to the Nth core data, respectively, and the first core data to the Nth core data respectively modulate the first transformed light to the Nth transformed light to respectively output the first core product light to the Nth core product light; a first inverse transformation device to an Nth inverse transformation device, configured to perform inverse Fourier transform on the first core product light to the Nth core product light, respectively, and output the first inverse transformed light to the Nth inverse transformed light, respectively.

[0010] According to an embodiment, each of the first to Nth transforming means includes at least one lens.

[0011] According to an embodiment, the image-based SLM is a transmissive SLM, multi-wavelength light is input to the image-based SLM in a first direction, and the image-based SLM outputs modulated light to an optical demultiplexer in the first direction, the optical demultiplexer includes first to N-th optical devices arranged in parallel in the first direction, and the first to N-th optical devices respectively output first to N-th sub-light in a second direction perpendicular to the first direction.

[0012] According to an embodiment, a first optical device among the first to Nth optical devices includes at least one dichroic mirror, and each of the remaining optical devices among the first to Nth optical devices includes at least one beam splitter or at least one polarization beam splitter.

[0013] According to an embodiment, the Nth optical device among the first to Nth optical devices receives modulated light, outputs light from the modulated light corresponding to the Nth wavelength among the first to Nth wavelengths as the Nth sub-light in the second direction, and outputs the remaining light from the modulated light excluding the light corresponding to the Nth wavelength to the (N-1)th optical device among the first to Nth optical devices in the first direction.

[0014] According to an embodiment, each of the first core-based SLM to the Nth core-based SLM is a transmissive SLM, and the first core-based SLM among the first core-based SLM to the Nth core-based SLM receives the first transformed light among the first transformed light to the Nth transformed light in a second direction, and modulates the first transformed light based on the first core data among the first core data to the Nth core data and outputs the first core product light to the first inverse transformation device among the first inverse transformation device to the Nth inverse transformation device in the second direction.

[0015] According to an embodiment, the optical multiplexer includes first to Nth optical devices configured to receive first to Nth inverse converted light in a second direction and align paths of the first to Nth inverse converted light to a third direction.

[0016] According to an embodiment, the third direction is the same as the first direction.

[0017] According to an embodiment, the image-based SLM is a reflective SLM, and the optical convolution computing device also includes: an incident device configured to change the optical path of the multi-wavelength light received from the multi-wavelength light generator in a first direction to a second direction perpendicular to the first direction, and the image-based SLM is configured to output the modulated light to the optical demultiplexer in a direction opposite to the second direction.

[0018] According to an embodiment, each of the first core-based SLM to the Nth core-based SLM is a reflective SLM, and the optical convolution computing device also includes: a first incident device to an Nth incident device respectively arranged parallel to the optical demultiplexer in a first direction, the first incident device to the Nth incident device being configured to respectively receive the first core integration light to the Nth core integration light from the first core-based SLM to the Nth core-based SLM in a direction opposite to the first direction, and respectively change the optical path of the first core integration light to the Nth core integration light to the opposite direction of the second direction, so as to respectively output the first core integration light to the Nth core integration light to the optical multiplexer.

[0019] According to an embodiment, a multi-wavelength light generator includes: a first diffraction grating configured to receive first to N-th input lights in different first to N-th directions, and align optical paths of the first to N-th input lights in an (N+1)-th direction to output multi-wavelength light.

[0020] According to an embodiment, the optical demultiplexer includes: a second diffraction grating configured to receive modulated light and output first to Nth sub-lights having first to Nth wavelengths respectively in different (N+2)th to (2N+1)th directions.

[0021] According to an embodiment, the first transformation device among the first transformation device to the Nth transformation device receives the first sub-light in the (N+2)th direction, and the optical convolution computing device also includes: a first prism to an (N-1)th prism, which are configured to respectively receive the second sub-light to the Nth sub-light, change the optical paths of the second sub-light to the Nth sub-light among the first sub-light to the Nth sub-light, and output the second sub-light to the Nth sub-light to the second inverse transformation device to the Nth inverse transformation device among the first inverse transformation device to the Nth inverse transformation device, respectively.

[0022] According to an embodiment, the optical convolution calculation also includes a third diffraction grating, and wherein the third diffraction grating is configured to respectively receive the first inverse transformed light to the Nth inverse transformed light, and align the paths of the first inverse transformed light to the Nth inverse transformed light in a single direction to output as a synthetic light.

[0023] According to an embodiment, a multi-wavelength light generator outputs multi-wavelength light by aligning optical paths of first to N-th input lights.

[0024] According to an embodiment of the present disclosure, an optical convolution computing device includes: a digital demultiplexer, configured to modulate multi-wavelength light based on spatial domain image data, and output a first sub-light to an Nth sub-light in a first direction to an Nth direction, respectively; a first transformation device to an Nth transformation device, configured to perform Fourier transform on the first sub-light to the Nth sub-light, respectively, and output a first transformed light to an Nth transformed light; a first digital device to an Nth digital device, configured to receive a first core data to an Nth core data and modulate the first transformed light to the Nth transformed light based on the first core data to the Nth core data to output a first core product light to an Nth core product light; a first inverse transformation device to an Nth inverse transformation device, configured to perform inverse Fourier transform on the first core product light to the Nth core product light, respectively, and output the first inverse transformed light to the Nth inverse transformed light; an optical multiplexer, configured to align the paths of the first inverse transformed light to the Nth inverse transformed light and output a synthesized light; and an image sensor, configured to sense the synthesized light, and N is a natural number greater than or equal to 2.

[0025] According to an embodiment, at least one of the digital demultiplexer or each of the first to Nth digital devices includes a digital micromirror device (DMD). BRIEF DESCRIPTION OF THE DRAWINGS

[0026] The above and other objects and features of the present disclosure will become apparent by describing in detail embodiments of the present disclosure with reference to the accompanying drawings.

[0027] Figure 1 is a block diagram showing a general optical convolution computing device.

[0028] Figure 2 is a block diagram showing a general optical convolution computing device that performs parallel computing.

[0029] Figure 3 It is shown by Figure 2 A diagram of sensed image data sensed by an image sensor.

[0030] Figure 4 is a block diagram showing an optical convolution computing device according to an embodiment of the present disclosure.

[0031] Figure 5An optical convolution computing device using a transmissive spatial optical modulator (SLM) according to an embodiment of the present disclosure is shown.

[0032] Figure 6 An optical convolution computing device using a reflective SLM according to an embodiment of the present disclosure is shown.

[0033] Figure 7 An optical convolution calculation device using a diffraction grating according to an embodiment of the present disclosure is shown.

[0034] Figure 8 An optical convolution computing device to which a digital device is applied according to an embodiment of the present disclosure is shown.

[0035] Fig. 9 It is shown by Figure 4 A diagram of sensed image data sensed by an image sensor.

[0036] Fig.10 is a flowchart illustrating the operation of an optical convolution computing device according to an embodiment of the present disclosure. DETAILED DESCRIPTION

[0037] Hereinafter, embodiments of the present disclosure will be described in detail and clearly to an extent that a person having ordinary skill in the art can easily realize the present disclosure.

[0038] The terms "unit", "module" and the like to be used below and the functional blocks shown in the accompanying drawings may be implemented in the form of software components, hardware components or a combination thereof. Below, in order to clearly describe the technical ideas of the inventive concept, descriptions associated with the same components will be omitted. As used herein, including in the claims, "or" used in a list of items ending with phrases such as "at least one of..." or "one or more of..." or "one or two of..." indicates an inclusive list, so that, for example, a list of at least one of A, B or C indicates A or B or C or AB or AC or BC or ABC (i.e., A and B and C).

[0039] Figure 1 is a block diagram showing a general optical convolution computing device 10. Figure 1 The optical convolution computing device 10 includes an image-based spatial optical modulator (SLM) 11, a transformation device 12, a kernel-based SLM 13, an inverse transformation device 14 and an image sensor 15.

[0040] The general optical convolution computing device 10 can perform convolution computing on spatial domain image data (SID) and kernel data using at least one optical device. The optical convolution computing device 10 can be included and used in a convolutional neural network (CNN). In the optical convolution computing device used in the convolutional neural network, the spatial domain image data (SID) can refer to image data that is the target of processing and analysis by the convolutional neural network. The kernel data can correspond to at least one of the kernels included in the convolutional neural network.

[0041] The convolution calculation follows the convolution theorem as shown in Equation 1.

[0042] [Equation 1]

[0043]

[0044] In this case, f(x, y) may refer to a spatial domain image function corresponding to the spatial domain image data, g(x, y) may refer to a spatial domain kernel function corresponding to the spatial domain kernel data, F(x′, y′) may refer to a Fourier domain image function, and G(x′, y′) may refer to a Fourier domain kernel function.

[0045] The Fourier domain image function is a function that applies the Fourier transform to the spatial domain image function, and the Fourier domain kernel function is a function that applies the Fourier transform to the spatial domain kernel function. The Fourier domain kernel function may correspond to kernel data used in a convolutional neural network.

[0046] Referring again to Equation 1, convolving the spatial domain image function with the spatial domain kernel function may be equivalent to performing element-wise multiplication of the Fourier domain image function and the Fourier domain kernel function, and then performing an inverse Fourier transform on the result of the element-wise multiplication.

[0047] In general, an SLM is a transformation device that modulates incident light in a spatial pattern corresponding to an electrical or optical input. The modulation can affect the phase, intensity, polarization, or direction of the incident light, and can be implemented using a variety of materials that exhibit various electro-optical or magneto-optical effects, or by materials that modulate light by a change in form.

[0048] The image-based SLM 11 may receive spatial domain image data f(x, y). In addition, single wavelength light having a first wavelength λ1 may be incident on the image-based SLM 11. The single wavelength light may be coherent light.

[0049] The image-based SLM 11 may modulate the single wavelength light based on the spatial domain image data f(x, y) and output the modulated light ML. For example, the image-based SLM 11 may modulate the optical amplitude or optical frequency of the single wavelength light to generate the modulated light ML having a waveform corresponding to the spatial domain image data f(x, y). The modulated light ML output from the image-based SLM 11 may refer to light transmitted through the image-based SLM 11 or light reflected from the image-based SLM 11.

[0050] The transforming device 12 may receive the modulated light ML. In detail, the modulated light ML may be incident on the transforming device 12. The transforming device 12 may include at least one optical device. For example, the transforming device 12 may include a single lens or a multi-lens system. The modulated light ML may pass through at least one optical device. The transforming device 12 may perform Fourier transform on the modulated light ML using at least one optical device and output the transformed light TL.

[0051] The core-based SLM 13 may receive Fourier domain core data G(x', y') and transformed light TL. In detail, the transformed light TL may be incident on the core-based SLM 13. The core-based SLM 13 may modulate the transformed light TL based on the Fourier domain core data G(x', y') and output core product light KML. The core product light KML may refer to light transmitted or reflected from the core-based SLM 13.

[0052] Specifically, the kernel-based SLM 13 can perform element-by-element multiplication on the transformed light TL and the Fourier domain kernel data G(x',y'). Since the transformed light TL is in the Fourier domain, the element-by-element multiplication of the kernel-based SLM 13 follows the same principle as the element-by-element multiplication between Fourier domain functions.

[0053] The inverse transformation device 14 may receive the kernel product light KML. In detail, the kernel product light KML may be incident on the inverse transformation device 14. The inverse transformation device 14 may include at least one optical device. For example, the inverse transformation device 14 may include a single lens or a multi-lens system. The inverse transformation device 14 may perform an inverse Fourier transform on the kernel product light KML using at least one optical device and output an inverse transformed light ITL. In this case, the inverse transformed light ITL may correspond to the result of the convolution between the spatial domain functions as described in Equation 1.

[0054] The image sensor 15 may sense the inverse converted light ITL and generate a digital signal DS in the form of image data. For example, the image sensor 15 may be a camera.

[0055] Since the Fourier domain kernel data G(x', y') input to the kernel-based SLM 13 corresponds to only one kernel, the time required for the optical convolution calculation device 10 to perform the convolution calculation may increase with the number of kernels.

[0056] Figure 2 is a block diagram showing a general optical convolution computing device 20 that performs parallel computing. Figure 2 The optical convolution computing device 20 includes an image-based SLM 21, a segmentation unit 22, a kernel-based SLM unit 23, an inverse transformation unit 24 and an image sensor unit 25. Figure 2 The image-based SLM 21 corresponds to Figure 1 Image-based SLM 11.

[0057] The general optical convolution computing device 20 can perform convolution computing on the first to N-th kernel data in parallel.

[0058] First, the image-based SLM 21 may modulate single wavelength light having a first wavelength λ1 based on the spatial domain image data f(x, y) to output modulated light ML.

[0059] The segmentation unit 22 may receive the modulated light ML. The segmentation unit 22 may perform Fourier transform on the modulated light ML and output first to N-th sub-lights SL1 to SLN. The first to N-th sub-lights SL1 to SLN may each contain the same information, such as the modulated light ML. The segmentation unit 22 may include a transforming device and a segmenting device. The transforming device may correspond to Figure 1 The transforming device 12 may perform a Fourier transform on the modulated light ML. The splitting device may split the modulated light ML or the modulated light after the Fourier transform into N light beams. For example, the splitting device may include a diffraction grating, a beam splitter, or a similar component. Here, "N" is a natural number of 2 or more. The operation of performing the Fourier transform on the modulated light ML and the operation of splitting the modulated light ML into N light beams may be performed simultaneously or sequentially.

[0060] The core-based SLM unit 23 can receive the first sub-light SL1 to the Nth sub-light SLN. The core-based SLM unit 23 can perform element-by-element multiplication based on the first core data to the Nth core data and the first sub-light SL1 to the Nth sub-light SLN. The element-by-element multiplication can be performed in parallel. For example, the core-based SLM unit 23 can perform element-by-element multiplication of the first core data and the first sub-light SL1 and output the first core product light KML1. The core-based SLM unit 23 can perform element-by-element multiplication of the second core data and the second sub-light SL2 and output the second core product light KML2. Repeat this process for the third core data to the Nth core data and the third sub-light to the Nth sub-light.

[0061] In an embodiment, the core-based SLM unit 23 may include first to N-th core-based SLMs 23-1 to 23-N. Each of the first to N-th core-based SLMs 23-1 to 23-N may correspond to Figure 1 The core-based SLM 13 of FIG. 1 may be a core-based SLM 13. The first core-based SLM 23-1 may receive the first sub-light SL1, perform element-by-element multiplication based on the first sub-light SL1 and the first core data, and output the first core product light KML1. The second core-based SLM 23-2 may receive the second sub-light SL2, perform element-by-element multiplication based on the second sub-light SL2 and the second core data, and output the second core product light KML2. The third core-based SLM 23-3 to the Nth core-based SLM 23-N may be similarly operated to output the third core product light KML3 to the Nth core product light KMLN, respectively.

[0062] In some embodiments, the core-based SLM unit may be composed of a single SLM. The SLM may simultaneously receive the first to N-th sub-lights SL1 to SLN and perform modulation operations on the first to N-th sub-lights SL1 to SLN in parallel. In this case, the single SLM may have a larger area and a higher resolution than each of the first to N-th core-based SLMs 23-1 to 23-N.

[0063] The inverse transform unit 24 may perform inverse Fourier transform on the first to Nth kernel product lights KML1 to KMLN, respectively, and output first to Nth inverse transformed lights ITL1 to ITLN.

[0064] Specifically, the inverse transform unit 24 may include a first inverse transform device 24-1 to an Nth inverse transform device 24-N. Each of the first inverse transform device 24-1 to the Nth inverse transform device 24-N may correspond to Figure 1 The first to N-th inverse transforming devices 24-1 to 24-N may respectively perform inverse Fourier transform on the first to N-th kernel product lights KML1 to KMLN, and respectively output first to N-th inverse transformed lights ITL1 to ITLN.

[0065] The image sensor unit 25 may sense the first to Nth inverse converted light ITL1 to ITLN to generate first to Nth digital signals DS1 to DSN in an image data format.

[0066] In some embodiments, the image sensor unit 25 may include first to N-th image sensors 25-1 to 25-N. Each of the first to N-th image sensors 25-1 to 25-N may correspond to Figure 1The image sensor 15 may include an image sensor 15. The first image sensor 25-1 may sense the first inverse transformed light ITL1 to generate a first digital signal DS1. The second image sensor 25-2 may sense the second inverse transformed light ITL2 to generate a second digital signal DS2. The third image sensor 25-3 to the N-th image sensor 25-N may sense the third inverse transformed light to the N-th inverse transformed light to generate a third digital signal DS3 to an N-th digital signal DSN, respectively. In this case, the output data of the convolutional neural network may be generated by an electronic device that synthesizes the first digital signal DS1 to the N-th digital signal DSN. However, the “N” image sensors 25-1 to 25-N may require additional space and energy.

[0067] In some embodiments, the image sensor unit 25 may be composed of a single image sensor. Such an image sensor may have a larger area and a higher resolution than each of the first to N-th image sensors 25-1 to 25-N.

[0068] Figure 3 It is shown by Figure 2 FIG. 1 is a diagram of first sensed image data sensed by an image sensor. Figure 2 In the case where the image sensor unit 25 is composed of a single image sensor, Figure 3 The illustrated first sensed image data is generated by a single image sensor.

[0069] The wavelengths of the first to Nth inverse transformed light ITL1 to ITLN may be the same. The image sensor unit 25 may include a sensing plane. The sensing plane may be substantially perpendicular to the incident direction of the first to Nth inverse transformed light ITL1 to ITLN, and may receive the first to Nth inverse transformed light ITL1 to ITLN. Even if the area of ​​the sensing plane of the image sensor unit 25 is large enough to accommodate the first to Nth inverse transformed light ITL1 to ITLN incident in parallel, the actual image sensor may not be able to capture the first to Nth inverse transformed light in a completely parallel manner. This is due to the spatial difference between the sensing plane and the optical device included in the inverse transform unit 24, and changes in the refractive index of the air and other factors.

[0070] Reference again Figure 3 , the first inverse-converted light and the second inverse-converted light may be incident on the image sensor unit 25 at a specific angle.

[0071] The amplitude of image data generated by the image sensor unit 25 based on the first inverse converted light and the second inverse converted light is shown in Equation 2.

[0072] [Equation 2]

[0073]

[0074] In this case, E(x, t) refers to the amplitude function of the first sensed image data, A0 refers to the light amplitude of the first inverse-converted light, B0 refers to the light amplitude of the second inverse-converted light, k0 refers to the spatial frequency corresponding to the first wavelength λ1, and r refers to the wavelength from the light source (e.g., Figure 2 ω0 refers to the angular frequency of the first inverse transformed light and the second inverse transformed light.

[0075] In addition, the image sensor unit 25 can measure the time from the first time t1 to the exposure time t s As a result, the image sensor unit 25 obtains the intensity of the incident light based on the first time t1 and the exposure time t s The amplitude of the first sensed image data generated by the light detected between φ and φ is expressed in Equation 3.

[0076] [Equation 3]

[0077]

[0078] Since Δr(x) is linear in x, an interference pattern may appear in the first sensing image data. Figure 3 As shown, the first sensed image data may show dark shadows that are vertically spaced apart from each other.

[0079] For ease of explanation, this specification focuses on sensing only the first inverse-converted light and the second inverse-converted light; however, the scope of the present disclosure is not limited thereto. As described above, the image sensor unit 25 may generate sensing image data from three or more lights having the same wavelength incident at different angles. In this case, the sensing image data may include more interference patterns.

[0080] Figure 4 is a block diagram showing an optical convolution computing device 100 according to an embodiment of the present disclosure. Figure 4 , the optical convolution computing device 100 includes a multi-wavelength light generator 110, an image-based SLM 120, an optical demultiplexer 130, a transform unit 140, a core-based SLM unit 150, an inverse transform unit 160, an optical multiplexer 170, and an image sensor 180. In an embodiment, the transform unit 140, the core-based SLM unit 150, and the inverse transform unit 160 may be integrated into an optical convolution processor.

[0081] The multi-wavelength light generator 110 may output multi-wavelength light having a first wavelength λ1 to an N-th wavelength λN. For example, the multi-wavelength light generator 110 may receive a first input light IL1 to an N-th input light ILN. The first input light IL1 to the N-th input light ILN may have a first wavelength λ1 to an N-th wavelength λN, respectively. The multi-wavelength light generator 110 may generate coherent multi-wavelength light by synthesizing the first input light IL1 to the N-th input light ILN. For example, the multi-wavelength light generator 110 may include at least one optical device, and the at least one optical device may output one multi-wavelength light by synthesizing the first input light IL1 to the N-th input light ILN.

[0082] The image-based SLM 120 may receive the spatial domain image data SID. The image-based SLM 120 may output modulated light ML by modulating the multi-wavelength light output from the multi-wavelength light generator 110 based on the spatial domain image data SID.

[0083] In some embodiments, the image-based SLM 120 may include one or both of a transmissive SLM and a reflective SLM.

[0084] In some embodiments, the image-based SLM 120 may be included in a digital micromirror device (DMD).

[0085] For example, the optical convolution computing device 100 may modulate the multi-wavelength light by transmitting or reflecting the multi-wavelength light through the image-based SLM 120 to output modulated light ML having a waveform of a function corresponding to the spatial domain image data SID.

[0086] The optical demultiplexer 130 may include at least one optical device. The optical demultiplexer 130 may output first to N-th sub-lights SL1 to SLN using at least one optical device based on the input modulated light ML. The first to N-th sub-lights SL1 to SLN may have first to N-th wavelengths λ1 to λN, respectively. In detail, the optical demultiplexer 130 may split the modulated light ML into the first to N-th sub-lights SL1 to SLN, each of which contains the same information but has a different wavelength.

[0087] In some embodiments, the optical demultiplexer 130 may include at least one of a beam splitter, a polarization beam splitter, or a dichroic mirror.

[0088] In some embodiments, the optical multiplexer 170 may include at least one of a diffraction grating or a DMD.

[0089] The transform unit 140 may correspond to Figure 2The transforming unit 140 may include first to N-th transforming devices 140-1 to 140-N. The first to N-th transforming devices 140-1 to 140-N may receive the first to N-th sub-lights SL1 to SLN, respectively. Each of the first to N-th transforming devices 140-1 to 140-N may perform Fourier transform on the first to N-th sub-lights SL1 to SLN, respectively, using at least one optical device. The first to N-th transforming devices 140-1 to 140-N may output the first to N-th transformed lights TL1 to TLN, respectively.

[0090] In some embodiments, each of the first to Nth transforming devices 140 - 1 to 140 -N may include at least one lens.

[0091] In some embodiments, the relationship between the spatial frequency of the spatial domain image data SID and the coordinates on the Fourier plane of the nth channel is shown in Equation 4.

[0092] [Equation 4]

[0093]

[0094] In this case, k x It can refer to the horizontal axis spatial frequency of the spatial domain image data SID, k y It can refer to the vertical axis spatial frequency of the spatial domain image data SID, x n' Can refer to the horizontal axis coordinate of the nth channel on the Fourier plane, y n' may refer to the vertical axis coordinate on the Fourier plane of the nth channel, λn may refer to the wavelength of light corresponding to the nth channel, and f 1,n It may refer to the focal length of an optical device (eg, a lens) included in the transformation device.

[0095] The core-based SLM unit 150 may include first to N-th core-based SLMs 150-1 to 150-N. The first to N-th core-based SLMs 150-1 to 150-N may receive first to N-th core data, respectively, and modulate first to N-th transformed light TL1 to TLN based on the first to N-th core data to output first to N-th core integrated light KML1 to KMLN. The first to N-th core-based SLMs 150-1 to 150-N may correspond to the first to N-th core data, respectively. Figure 2 The first core-based SLM 23-1 to the Nth core-based SLM 23-N.

[0096] The inverse transform unit 160 may include a first inverse transform device 160-1 to an Nth inverse transform device 160-N. The inverse transform unit 160 may correspond to Figure 2 The first to N-th inverse transforming devices 160-1 to 160-N may receive the first to N-th core integrated light KML1 to KMLN, respectively. The first to N-th inverse transforming devices 160-1 to 160-N may respectively perform inverse Fourier transform on the first to N-th core integrated light KML1 to KMLN using at least one optical device. The first to N-th inverse transforming devices 160-1 to 160-N may respectively output the first to N-th inverse transformed light ITL1 to ITLN.

[0097] In some embodiments, each of the first to Nth transform devices 140-1 to 140-N and the first to Nth inverse transform devices 160-1 to 160-N may include at least one single lens. When N is 3, the relationship between the focal lengths of the single lenses may be shown in Equation 5.

[0098] [Equation 5]

[0099]

[0100] In this case, f 1.1 It may refer to the focal length of the lens included in the first transforming device 140-1. 2.1 f may refer to the focal length of the lens included in the first inverse transform device 160-1. 1.2 It may refer to the focal length of the lens included in the second transforming device 140-2. 2.2 f may refer to the focal length of the lens included in the second inverse transform device 160-2. 1.3 It may refer to the focal length of the lens included in the third transforming device 140-3. 2.3 It may refer to the focal length of the lens included in the third inverse transform device 160 - 3 .

[0101] The optical multiplexer 170 may include at least one optical device. The optical multiplexer 170 may align optical paths of the first to Nth inverse converted lights ITL1 to ITLN using the at least one optical device and output the synthesized light RL.

[0102] In some embodiments, the optical multiplexer 170 may include at least one of a beam splitter, a polarization beam splitter, or a dichroic mirror.

[0103] In some embodiments, the optical multiplexer 170 may include at least one of a diffraction grating or a DMD.

[0104] The image sensor 180 may sense the synthetic light RL. The image sensor 180 may generate a digital signal DS in an image data format based on the light intensity of the synthetic light RL. The digital signal DS may correspond to image data processed by a convolutional neural network using the optical convolution computing device 100 according to the present disclosure.

[0105] Figure 5 An optical convolution computing device 200 to which a transmissive spatial optical modulator (SLM) is applied according to an embodiment of the present disclosure is shown. In the optical convolution computing device 200, at least one of an image-based SLM or a kernel-based SLM is a transmissive SLM. Figure 5 A plan view formed by a first direction D1 and a second direction D2 perpendicular to the first direction D1 is shown, and a third direction D3 may mean a direction perpendicular to both the first direction D1 and the second direction D2.

[0106] For ease of explanation, the case where "N" is 3 is shown, but the scope of the present disclosure is not limited thereto. Even when "N" is 2 or greater than 3, the optical convolution computing device 200 can operate similarly. In addition, Figure 5 The components shown in the figure may correspond to Figure 4 Those components having similar reference numerals in the drawings (for example, when the latter part of the reference numeral is the same).

[0107] The multi-wavelength light generator 210 may align optical paths of the first to third input lights IL1 to IL3. In detail, the multi-wavelength light generator 210 may synthesize the first to third input lights IL1 to IL3 to output multi-wavelength light.

[0108] The multi-wavelength light generator 210 may include an optical device 210 - 1 that aligns an optical path of the second input light IL2 with an optical path of the first input light IL1 .

[0109] For example, the optical device 210 - 1 may transmit the first input light IL1 as it is in the second direction D2 , which is a direction of an optical path of the first input light IL1 , and may output the second input light IL2 in the second direction D2 .

[0110] The multi-wavelength light generator 210 may further include an optical device 210 - 2 that aligns an optical path of the third input light IL3 with an optical path of the first input light IL1 .

[0111] For example, the optical device 210 - 2 may transmit the light output from the optical device 210 - 1 as it is in the second direction D2 , which is a direction of the optical path of the first input light IL1 , and may output the third input light IL3 in the second direction D2 .

[0112] Therefore, the multi-wavelength light generator 210 may generate the first to third input lights IL1 to IL3 having aligned optical paths (ie, synthesized) as multi-wavelength light.

[0113] In some embodiments, the first to third input lights IL1 to IL3 may be applied with first to third weight values, respectively. The first to third weight values ​​may be included in weight data of a convolutional neural network using a convolutional computing device according to the present disclosure.

[0114] For example, the multi-wavelength optical generator 210 may further include first to third weight-based SLMs SLM3-1 to SLM3-3. The first to third weight-based SLMs SLM3-1 to SLM3-3 may modulate the angular frequencies of the first to third input lights IL1 to IL3 based on first to third weight values, respectively.

[0115] However, the scope of the present disclosure is not limited thereto, and the first to third input lights IL1 to IL3 input to the optical convolution computing device 200 may be modulated based on the first to third weight values ​​at any stage before being synthesized in the optical multiplexer.

[0116] The image-based SLM1 220 may transmit the multi-wavelength light incident in the second direction D2. In detail, the image-based SLM1 220 may modulate the multi-wavelength light based on the spatial domain image data SID, and may transmit the modulated light in the second direction D2.

[0117] The optical demultiplexer may include a plurality of optical devices 230-1 to 230-3. The plurality of optical devices 230-1 to 230-3 may split modulated light incident in the second direction D2 into first to third sub-lights, each based on its wavelength, and may output the first to third sub-lights in the first direction D1.

[0118] Specifically, a plurality of optical devices 230-1 to 230-3 may be arranged parallel to each other in the second direction D2. The optical device 230-3 may receive the modulated light in the second direction D2, output a component (i.e., light) corresponding to the third wavelength λ3 of the modulated light in the first direction D1 as a third sub-light, and output the remaining components (e.g., components corresponding to the first wavelength λ1 and the second wavelength λ2 of the modulated light) in the second direction D2. The optical device 230-2 may output a component corresponding to the second wavelength λ2 of the light received from the optical device 230-3 in the first direction D1 as a second sub-light, and output the remaining components (e.g., components corresponding to the first wavelength λ1 of the modulated light) in the second direction D2. The optical device 230-1 may output the light received from the optical device 230-2 (e.g., a component corresponding to the first wavelength λ1 of the modulated light) in the first direction D1 as a first sub-light.

[0119] In some embodiments, each of the plurality of optical devices 230 - 1 to 230 - 3 may include at least one of a beam splitter, a polarization beam splitter, or a dichroic mirror.

[0120] exist Figure 5 In the illustrated embodiment, the optical device 230 - 1 includes a dichroic mirror, and each of the remaining optical devices 230 - 2 and 230 - 3 includes a beam splitter or a polarizing beam splitter.

[0121] The first sub-light output from the optical device 230 - 1 may pass through the first transforming device 240 - 1 , the first core-based SLM 250 - 1 , and the first inverse transforming device 260 - 1 , which are arranged parallel to each other in the first direction D1 .

[0122] For example, the first core-based SLM 250-1 may be a transmissive SLM. The first core-based SLM 250-1 may perform element-wise multiplication on the first transformed light output from the first transform device 240-1 based on the first core data and output the first core product light in the first direction D1.

[0123] The second sub-light output from the optical device 230 - 2 may pass through the second transforming device 240 - 2 , the second core-based SLM 250 - 2 , and the second inverse transforming device 260 - 2 , which are arranged parallel to each other in the first direction D1 .

[0124] For example, the second core-based SLM 250-2 may be a transmissive SLM. The second core-based SLM 250-2 may perform element-wise multiplication on the second transformed light output from the second transform device 240-2 based on the second core data and output second core product light in the first direction D1.

[0125] The third sub-light output from the optical device 230 - 3 may pass through the third transforming device 240 - 3 , the third core-based SLM 250 - 3 , and the third inverse transforming device 260 - 3 , which are arranged parallel to each other in the first direction D1 .

[0126] For example, the third core-based SLM 250-3 may be a transmissive SLM. The third core-based SLM 250-3 may perform element-wise multiplication on the third transformed light output from the third transform device 240-3 based on the third core data and output the third core product light in the first direction D1.

[0127] The optical multiplexer may include a plurality of optical devices 270 - 1 to 270 - 3 .

[0128] For example, a plurality of optical devices 270-1 to 270-3 may change the optical paths of the first inverse-converted light to the third inverse-converted light respectively incident in the first direction D1. In detail, the optical multiplexer may change the optical paths of the first inverse-converted light to the third inverse-converted light to align with a direction different from the first direction D1 (e.g., the second direction D2) so as to be synthesized. The optical multiplexer may output the synthesized light in a direction different from the first direction D1 (e.g., the second direction D2). In detail, the synthesized light may refer to light synthesized from the first inverse-converted light to the third inverse-converted light.

[0129] The optical device 270-3 may be arranged in parallel with the third inverse conversion device 260-3 in the first direction D1. The optical device 270-3 may receive the third inverse conversion light in the first direction D1 and output the third inverse conversion light in the second direction D2.

[0130] The optical device 270-2 may be arranged parallel to the second inverse conversion device 260-2 in the first direction D1 and parallel to the optical device 270-3 in the second direction D2. The optical device 270-2 may receive the second inverse conversion light in the first direction D1 and output the second inverse conversion light in the second direction D2. In addition, the optical device 270-2 may transmit the third inverse conversion light input in the same direction D2 along the second direction D2. In detail, the optical device 270-2 may align the optical paths of the second inverse conversion light and the third inverse conversion light.

[0131] The optical device 270-1 may be arranged parallel to the first inverse conversion device 260-1 in the first direction D1 and parallel to the optical device 270-2 in the second direction D2. The optical device 270-1 may receive the first inverse conversion light in the first direction D1 and output the first inverse conversion light in the second direction D2. In addition, the optical device 270-1 may transmit the second inverse conversion light and the third inverse conversion light input in the second direction D2 in the second direction D2. In detail, the optical device 270-1 may align the optical path of the first inverse conversion light with the optical paths of the second inverse conversion light and the third inverse conversion light.

[0132] In some embodiments, each of the plurality of optical devices 270 - 1 to 270 - 3 may include at least one of a beam splitter, a polarization beam splitter, or a dichroic mirror.

[0133] In detail, the optical multiplexer may output the first to third inverse-converted lights whose optical paths are aligned with each other as the synthesized light in the second direction D2.

[0134] The image sensor 280 may be arranged in parallel with the plurality of optical devices 270-1 to 270-3 in the second direction D2. The image sensor 280 may sense the synthesized light.

[0135] Figure 6 An optical convolution computing device 300 to which a reflective SLM is applied according to an embodiment of the present disclosure is shown. In the optical convolution computing device 300, at least one of an image-based SLM and a kernel-based SLM is a reflective SLM. Figure 6 The third direction D3 may refer to a direction perpendicular to both the first direction D1 and the second direction D2 , and is a plan view formed by the first direction D1 and the second direction D2 perpendicular to the first direction D1 .

[0136] For ease of explanation, the case where "N" is 3 is shown, but the scope of the present disclosure is not limited thereto. Even when "N" is 2 or greater than 3, the optical convolution computing device 300 can operate similarly. In addition, Figure 6 The components shown in the figure may correspond to Figure 4 Those components having similar reference numerals in the drawings (for example, when the latter part of the reference numeral is the same).

[0137] The multi-wavelength optical generator 310 may correspond to Figure 5 The multi-wavelength light generator 210 may receive the first input light IL1 in the first direction D1, and may receive the second input light IL2 and the third input light IL3 in the second direction D2. The multi-wavelength light generator 310 may change the optical paths of the second input light IL2 and the third input light IL3 to be aligned with the first direction D1.

[0138] The optical convolution computing device 300 may further include a first incident device. The first incident device may allow the multi-wavelength light output from the multi-wavelength light generator 310 to be incident on the image-based SLM 320, and may allow the light reflected from the image-based SLM 320 to be output to the optical demultiplexer without overlapping with the optical path of the multi-wavelength light.

[0139] For example, the first incident device may include at least one of a beam splitter, a polarization beam splitter, or a dichroic mirror. The first incident device may be arranged in parallel to the multi-wavelength light generator 310 in the first direction D1. The first incident device may receive the multi-wavelength light in the first direction D1 and output the multi-wavelength light to the image-based SLM 320 in the opposite direction of the second direction D2. In addition, the first incident device may transmit the modulated light reflected from the image-based SLM 320 in the second direction D2 and output the modulated light to the optical demultiplexer.

[0140] The optical demultiplexer may include a plurality of optical devices 330-1 to 330-3. The plurality of optical devices 330-1 to 330-3 may be arranged in parallel to the first incident device in the second direction D2.

[0141] The optical device 330-1 may receive the modulated light in the second direction D2. The optical device 330-1 may output a component corresponding to the first wavelength λ1 of the modulated light as a first sub-light in the first direction D1, and may output the remaining components in the second direction D2.

[0142] The optical device 330-2 may receive the light transmitted from the optical device 330-1 in the second direction D2. The optical device 330-2 may output a component corresponding to the second wavelength λ2 of the input light as a second sub-light in the first direction D1, and may output the remaining components in the second direction D2.

[0143] The optical device 330 - 3 may receive the light transmitted from the optical device 330 - 2 in the second direction D2 and output the received light as third sub-light in the first direction D1 .

[0144] The first sub-light may be incident on the first core-based SLM 350-1 through the first transformation device 340-1 and the second incident device. The second incident device is arranged in parallel to the optical device 330-1 in the first direction D1. The second incident device may output the first transformed light output from the first transformation device 340-1 to the first core-based SLM 350-1 as it is, and may output the first core product light reflected from the first core-based SLM 350-1 in a different direction (e.g., the opposite direction of the second direction D2) so as not to overlap with the optical path of the first transformed light. Subsequently, the first core product light may be incident on the optical multiplexer through the first inverse transformation device 360-1 arranged in parallel with the second incident device.

[0145] The second sub-light may be incident on the second core-based SLM 350-2 through the second transformation device 340-2 and the third incident device. The third incident device is arranged in parallel to the optical device 330-2 in the first direction D1. The third incident device may output the second transformed light output from the second transformation device 340-2 to the second core-based SLM 350-2 as it is, and may output the second core product light reflected from the second core-based SLM 350-2 in a different direction (e.g., the opposite direction of the second direction D2) so as not to overlap with the optical path of the second transformed light. Subsequently, the second core product light may be incident on the optical multiplexer through the second inverse transformation device 360-2 arranged in parallel with the third incident device.

[0146] The third sub-light may be incident on the third core-based SLM 350-3 through the third transforming device 340-3 and the fourth incident device. The fourth incident device is arranged in parallel to the optical device 330-3 in the first direction D1. The fourth incident device may output the third transformed light output from the third transforming device 340-3 to the third core-based SLM 350-3 as it is, and may output the third core product light reflected from the third core-based SLM 350-3 in a different direction (e.g., the opposite direction of the second direction D2) so as not to overlap with the optical path of the third transformed light. Subsequently, the third core product light may be incident on the optical multiplexer through the third inverse transforming device 360-3 arranged in parallel with the fourth incident device.

[0147] In some embodiments, each of the second to fourth incident means may include at least one of a beam splitter, a polarization beam splitter, or a dichroic mirror.

[0148] The optical multiplexer may include a plurality of optical devices 370 - 1 to 370 - 3 .

[0149] The optical device 370 - 3 may reflect the third inversely converted light incident in the opposite direction of the second direction D2 and output the third inversely converted light in the first direction D1 .

[0150] The optical device 370-2 may receive the second inversely transformed light incident in the opposite direction of the second direction D2 and output the second inversely transformed light in the first direction D1, and may receive the third inversely transformed light incident in the first direction D1 and output the third inversely transformed light as is in the first direction D1. In detail, the optical device 370-2 may align the optical path of the second inversely transformed light with the optical path of the third inversely transformed light.

[0151] The optical device 370-1 may receive the first inversely transformed light incident in the opposite direction of the second direction D2 and output the first inversely transformed light in the first direction D1, and may receive the second inversely transformed light and the third inversely transformed light incident in the first direction D1 and output the second inversely transformed light and the third inversely transformed light as is in the first direction D1. In detail, the optical device 370-1 may align the optical path of the first inversely transformed light with the optical paths of the second inversely transformed light and the third inversely transformed light.

[0152] The optical multiplexer may output the first to third inverse-converted lights whose optical paths are aligned as synthesized light in the first direction D1.

[0153] The image sensor 380 may be arranged in parallel with the optical multiplexer in the first direction D1. The image sensor 380 may sense the synthesized light incident in the first direction D1.

[0154] In some embodiments, a reflective SLM may also use a polarizer or a wave plate.

[0155] Figure 7 FIG. 4 shows an optical convolution computing device 400 using a diffraction grating according to an embodiment of the present disclosure. Figure 7 , the optical convolution computing device 400 uses at least one diffraction grating. Figure 7 A plan view formed by a first direction D1 and a second direction D2 perpendicular to the first direction D1 is shown, and a third direction D3 may mean a direction perpendicular to both the first direction D1 and the second direction D2.

[0156] For ease of explanation, the case where "N" is 3 is shown, but the scope of the present disclosure is not limited thereto. Even when "N" is 2 or greater than 3, the optical convolution computing device 400 can operate similarly. In addition, Figure 7 The components shown in the figure may correspond to Figure 4 Those components having similar reference numerals in the drawings (for example, when the latter part of the reference numeral is the same).

[0157] The multi-wavelength light generator 410 may include a first diffraction grating DG1. The first diffraction grating DG1 may change the optical paths of the first to third input lights IL1 to IL3 incident in different directions to one direction (e.g., the second direction D2). In detail, the first diffraction grating DG1 may synthesize (or combine) the first to third input lights IL1 to IL3 and output the synthesized light as multi-wavelength light in the second direction D2.

[0158] In some embodiments, the first to third input lights IL1 to IL3 may be lights to which first to third weight values ​​are applied, respectively. The first to third weight values ​​may be included in weight data of a convolutional neural network using a convolutional computing device according to the present disclosure.

[0159] For example, the multi-wavelength optical generator 410 may further include first to third weight-based SLMs SLM3-1 to SLM3-3. The first to third weight-based SLMs SLM3-1 to SLM3-3 may modulate the angular frequencies of the first to third input lights IL1 to IL3 based on first to third weight values, respectively.

[0160] However, the scope of the present disclosure is not limited thereto, and the first to third input lights IL1 to IL3 input to the optical convolution computing device 400 may be modulated based on the first to third weight values ​​at any stage before being synthesized in the optical multiplexer.

[0161] The image-based SLM 420 may correspond to Figure 4 The image-based SLM 120. The image-based SLM 420 may modulate the multi-wavelength light and output the modulated light in the second direction D2.

[0162] The optical demultiplexer 430 may include a second diffraction grating DG2. The second diffraction grating DG2 may output modulated light incident in the second direction D2 as first to third sub-lights in three different directions for each of the first to third wavelengths λ1 to λ3. For example, the second diffraction grating DG2 may output the first sub-light in the first direction D1. The second diffraction grating DG2 may output the second and third sub-lights in directions different from the first direction D1.

[0163] The optical convolution computing device 400 may further include a first prism WP1 and a second prism WP2. The first prism WP1 may change the optical path of the third sub-light into the first direction D1. The second prism WP2 may change the optical path of the second sub-light into the first direction D1.

[0164] The first sub-light may sequentially pass through the first transforming device 440-1, the first core-based SLM 450-1, and the first inverse transforming device 460-1 arranged in parallel in the first direction D1. The second sub-light passing through the second prism WP2 may sequentially pass through the second transforming device 440-2, the second core-based SLM 450-2, and the second inverse transforming device 460-2 arranged in parallel in the first direction D1. The third sub-light passing through the first prism WP1 may sequentially pass through the third transforming device 440-3, the third core-based SLM 450-3, and the third inverse transforming device 460-3 arranged in parallel in the first direction D1.

[0165] In some embodiments, the second diffraction grating DG2 may output the first sub-light in a fourth direction different from the first direction D1. In this case, the optical convolution computing device 400 may include first to third prisms (however, Figure 7 The first to third prisms may receive the first to third sub-lights in three different directions, respectively, and may change the optical paths of the first to third sub-lights into the first direction D1.

[0166] The optical multiplexer 470 may include a third diffraction grating DG3. The third diffraction grating DG3 may align optical paths of the first to third inverse-converted lights incident from different directions to a single direction to output as a synthesized light.

[0167] In detail, for example, the third diffraction grating DG3 may receive the third inverse-converted light in the first direction D1. The third diffraction grating DG3 may receive the first inverse-converted light and the second inverse-converted light in directions other than the first direction D1, respectively.

[0168] In this case, the optical convolution computing device 400 may further include a fourth prism WP4 and a fifth prism WP5. The fourth prism WP4 may change the optical path of the second inversely transformed light so as to be incident on the third diffraction grating DG3, so that the second inversely transformed light may be synthesized with the third inversely transformed light. The fifth prism WP5 may change the optical path of the first inversely transformed light so as to be incident on the third diffraction grating DG3, so that the first inversely transformed light may be synthesized with the third inversely transformed light.

[0169] The image sensor 480 may include a sensing plane. The sensing plane may be arranged perpendicular to the synthesized light. The image sensor 480 may sense the light intensity of the synthesized light through the sensing plane.

[0170] Figure 8 FIG. 5 shows an optical convolution computing device 500 to which a digital device is applied according to an embodiment of the present disclosure. Figure 8 , the optical convolution computing device 500 uses at least one digital device. Figure 8A plan view formed by a first direction D1 and a second direction D2 perpendicular to the first direction D1 is shown, and a third direction D3 may mean a direction perpendicular to both the first direction D1 and the second direction D2.

[0171] For ease of explanation, the case where "N" is 3 is shown, but the scope of the present disclosure is not limited thereto. Even when "N" is 2 or greater than 3, the optical convolution computing device 500 can operate similarly. In addition, Figure 8 The components shown in can correspond to Figure 4 Those components having similar reference numerals in the drawings (for example, when the latter part of the reference numeral is the same).

[0172] The multi-wavelength optical generator 510 may correspond to Figure 5 The multi-wavelength optical generator 210. The multi-wavelength optical generator 510 may output the multi-wavelength light to the digital demultiplexer by aligning the optical paths of the first to third input lights IL1 to IL3.

[0173] The digital demultiplexer can receive the multi-wavelength light in any direction. The digital demultiplexer can modulate the multi-wavelength light based on the spatial domain image data. In addition, the digital demultiplexer can output the modulated multi-wavelength light as the first sub-light to the third sub-light in three different directions.

[0174] In detail, for example, the digital demultiplexer may output the third sub-light in the first direction D1. The digital demultiplexer may output the first sub-light and the second sub-light in directions different from the first direction D1, respectively.

[0175] In some embodiments, the digital demultiplexer may include a DMD (digital micromirror device).

[0176] The optical convolution computing device 500 may further include a fifth prism WP5 and a sixth prism WP6. The fifth prism WP5 may change the optical path of the first sub-light into the first direction D1. The sixth prism WP6 may change the optical path of the second sub-light into the first direction D1.

[0177] The first sub-light may pass through the first transforming device 540-1 arranged in parallel to the fifth prism WP5 in the first direction D1, and may be incident on the first digital device DD1. The first digital device DD1 may receive the first transformed light output from the first transforming device 540-1 in the first direction D1. The first digital device DD1 may modulate the first transformed light to correspond to the element-by-element multiplication based on the first kernel data. The first digital device DD1 may output the first kernel product light in a direction different from the first direction D1 (e.g., a fourth direction).

[0178] The second sub-light may pass through the second transforming device 540-2 arranged in parallel to the sixth prism WP6 in the first direction D1, and may be incident on the second digital device DD2. The second digital device DD2 may receive the second transformed light output from the second transforming device 540-2 in the first direction D1. The second digital device DD2 may modulate the second transformed light to correspond to the element-by-element multiplication based on the second kernel data. The second digital device DD2 may output the second kernel product light in a direction different from the first direction D1 (e.g., a fourth direction).

[0179] The third sub-light may pass through the third transform device 540-3 arranged in parallel to the digital demultiplexer in the first direction D1, and may be incident on the third digital device DD3. The third digital device DD3 may receive the third transformed light output from the third transform device 540-3 in the first direction D1. The third digital device DD3 may modulate the third transformed light to correspond to the element-by-element multiplication based on the third kernel data. The third digital device DD3 may output the third kernel product light in a direction different from the first direction D1 (e.g., a fourth direction).

[0180] The optical convolution operation device 500 may further include seventh to ninth prisms WP7 to WP9. The seventh to ninth prisms WP7 to WP9 may align the optical paths of the first to third core integration lights to the same direction. The optical paths of the first to third core integration lights do not overlap with each other.

[0181] For example, the seventh prism WP7 can change the optical path of the first core light to the opposite direction of the first direction D1. The eighth prism WP8 can change the optical path of the second core light to the opposite direction of the first direction D1. The ninth prism WP9 can change the optical path of the third core light to the opposite direction of the first direction D1.

[0182] In addition, the first inverse transformation device 560-1 is arranged in parallel with the seventh prism WP7 in the first direction D1, and can integrate light by the first core incident in the opposite direction of the first direction D1 through the seventh prism WP7. The second inverse transformation device 560-2 is arranged in parallel with the eighth prism WP8 in the first direction D1, and can integrate light by the second core incident in the opposite direction of the first direction D1 through the eighth prism WP8. The third inverse transformation device 560-3 is arranged in parallel with the ninth prism WP9 in the first direction D1, and can integrate light by the third core incident in the opposite direction of the first direction D1 through the ninth prism WP9.

[0183] The optical multiplexer may include a plurality of optical devices 570-1 to 570-3. The optical multiplexer may output a synthesized light by aligning optical paths of the first inverse-converted light to the third inverse-converted light using the plurality of optical devices 570-1 to 570-3. The optical multiplexer may correspond to Figure 5 Optical multiplexer.

[0184] For example, the optical multiplexer may align optical paths of the first inverse-converted light to the third inverse-converted light in the second direction D2 and may output the synthesized light in the second direction D2.

[0185] The image sensor 580 may sense light intensity of the synthetic light incident in the second direction D2.

[0186] For ease of explanation, based on Figures 5 to 8 The embodiments shown in the embodiment are described, but the scope of the present disclosure is not limited thereto. As long as the optical paths do not overlap, the optical paths may be different from those described. In addition, some configurations for changing the optical paths may be added or omitted.

[0187] Fig. 9 Is used to describe Figure 4 FIG. 10 is a diagram of the sensed image data sensed by the image sensor 180. Fig. 9 , shows second sensed image data generated by the image sensor 180.

[0188] First, in Figure 4 , the wavelengths of the first inverse-converted light ITL1 to the Nth inverse-converted light ITLN may correspond to the first wavelength λ1 to the Nth wavelength λN, respectively. Figure 4 As shown, the image sensor 180 may include a sensing plane. Even if the sensing plane is large enough to accommodate the first to Nth inverse-converted lights in parallel, the actual image sensor 180 may not allow the first to Nth inverse-converted lights to be incident completely parallel to the sensing plane of the image sensor 180 due to factors such as spatial misalignment between the sensing plane and the optical device included in the inverse conversion unit 160, a change in the refractive index of air, and the like.

[0189] Reference again Fig. 9 , the first inverse-converted light and the second inverse-converted light may be incident on the image sensor 180 at a specific angle.

[0190] The amplitude of the second sensed image data generated by the image sensor 180 based on the first inverse converted light and the second inverse converted light is shown in Equation 6.

[0191] [Equation 6]

[0192]

[0193] In this case, E(x, t) refers to the amplitude function of the second sensed image data, A1 refers to the optical amplitude of the first inverse-converted light, A2 refers to the optical amplitude of the second inverse-converted light, k1 refers to the spatial frequency corresponding to the first wavelength λ1, k2 refers to the spatial frequency corresponding to the second wavelength λ2, and r refers to the wavelength from the light source (e.g., Figure 5 The optical multiplexer of the present invention is used to represent the distance from the first inverse transformed light to the origin (x=0) of the image sensor 180, Δr(x) refers to the distance difference between the first inverse transformed light and the second inverse transformed light, ω1 refers to the angular frequency of the first inverse transformed light, and ω2 refers to the angular frequency of the second inverse transformed light.

[0194] In addition, the image sensor 180 may measure the time from the first time t1 to the exposure time t s The intensity of the incident light. As a result, it is represented by sensing from the start time t0 to the exposure time t s A function of the amplitude of the second sensed image data generated by the light incident on the image sensor 180 is shown in Equation 7.

[0195] [Equation 7]

[0196]

[0197] In the case where the wavelengths of the first inverse-converted light and the second inverse-converted light are different, a beat phenomenon may occur when the first inverse-converted light and the second inverse-converted light overlap. In this case, it can be assumed that the beat period is longer than the exposure time t s Hundreds to billions of times shorter.

[0198] Therefore, due to Compare is much smaller, so Equation 7 can be approximated as Equation 8.

[0199] [Equation 8]

[0200]

[0201] In this case, an amplitude function of the second sensed image data sensed by the image sensor 180 is expressed in Equation 9.

[0202] [Equation 9]

[0203]

[0204] In this case, x" indicates a horizontal axis coordinate on a sensing plane of the image sensor, and y" indicates a vertical axis coordinate on the sensing plane of the image sensor.

[0205] Therefore, with Figure 3 Different from the first sensing image data, the second sensing image data may not have the interference pattern.

[0206] In detail, the optical convolution computing device according to the present disclosure can use an optical multiplexer to synthesize the operation results (inverse transformation light) while performing calculations on multiple kernel data in parallel. Therefore, the optical convolution computing device can reduce the area or energy consumption of the image sensor, or can reduce the amount of calculation of the electronic computer while performing multiple calculations in parallel.

[0207] For ease of explanation, the case where only the first inverse-converted light and the second inverse-converted light are sensed is described, but the scope of the present disclosure is not limited thereto. As described above, the image sensor may generate sensed image data from three or more lights having the same wavelength and incident at different angles. In this case, the sensed image data may include more interference patterns.

[0208] Fig.10 is a flowchart showing the operation of an optical convolution computing device according to an embodiment of the present disclosure. Fig.10 , the operations are described in the process Figure 4 The optical convolution computing device 100 may include a multi-wavelength light generator, an image-based SLM, an optical demultiplexer, a first transform device to an N-th transform device, a first core-based SLM to an N-th core-based SLM, a first inverse transform device to an N-th inverse transform device, an optical multiplexer, and an image sensor.

[0209] In S110, the optical convolution computing device may generate multi-wavelength light having a first wavelength to an Nth wavelength by a multi-wavelength light generator.

[0210] In S120, the optical convolution computing device may modulate the multi-wavelength light based on the spatial domain image data by the image-based SLM and output the modulated light.

[0211] In S130, the optical convolution computing device may output first to N-th sub-lights having first to N-th wavelengths, respectively, by using at least one optical device based on the modulated light by the optical demultiplexer.

[0212] In S140, the optical convolution calculation device may use at least one optical device to perform Fourier transform on the first to N-th sub-lights respectively through the first to N-th transform devices and output the first to N-th transformed lights respectively.

[0213] In S150, the optical convolution computing device may modulate the first to Nth transformed lights based on the first to Nth core data respectively by the first to Nth core-based SLMs and output the first to Nth core-product lights respectively.

[0214] In S160, the optical convolution calculation device can use at least one optical device to perform Fourier inverse transform on the first to Nth kernel product lights respectively by the first to Nth inverse transform devices, and output the first to Nth inverse transformed lights.

[0215] In S170, the optical convolution calculation device may align paths of the first inverse-converted light to the Nth inverse-converted light using at least one optical device through an optical multiplexer, and output a synthesized light.

[0216] In S180, the optical convolution computing device may sense the synthesized light by using an image sensor.

[0217] According to an embodiment of the present disclosure, an optical convolution computing device and a method of operating the optical convolution computing device are provided.

[0218] In addition, an optical convolution computing device and an operating method thereof are provided, which improves computing performance and increases energy efficiency by applying multiple kernels based on a multi-wavelength light source to perform calculations in parallel and optically synthesize the results of the convolution calculations.

[0219] The above description is a detailed embodiment for carrying out the present disclosure. Embodiments in which the design change is simple or easy to change may be included in the present disclosure as well as in the above-mentioned embodiments. In addition, the technology that is easy to change and implement by using the above-mentioned embodiments may be included in the present disclosure. Therefore, the scope of the present disclosure should not be limited to the above-mentioned embodiments, and should be limited not only by the described claims, but also by the claims equivalent to the claims of the present disclosure.

Claims

1. An optical convolution computing device, comprising: an image-based spatial optical modulator (SLM) configured to modulate the multi-wavelength light based on spatial domain image data and output the modulated light; an optical demultiplexer configured to output first to N-th sub-lights having first to N-th wavelengths, respectively, based on the modulated light; An optical convolution processor configured to receive the first to N-th sub-lights and output the first to N-th inverse-converted lights; as well as an optical multiplexer configured to align paths of the first inverse-converted light to the Nth inverse-converted light; Here, N is a natural number greater than or equal to 2.

2. The optical convolution computing device according to claim 1, further comprising: The multi-wavelength light generator is configured to output multi-wavelength light having a first wavelength to an Nth wavelength.

3. The optical convolution computing device according to claim 1, further comprising: An image sensor is configured to sense the synthesized light.

4. The optical convolution computing device according to claim 1, further comprising: a multi-wavelength light generator configured to output multi-wavelength light having a first wavelength to an Nth wavelength; as well as an image sensor configured to sense the resultant light, Wherein, the optical convolution processor comprises: The first to Nth transforming devices are configured to perform Fourier transform on the first to Nth sub-lights, respectively, and output first to Nth transformed lights, respectively; The first core-based SLM to the Nth core-based SLM are configured to receive the first core data to the Nth core data, respectively, and modulate the first converted light to the Nth converted light based on the first core data to the Nth core data, respectively, to output the first core integrated light to the Nth core integrated light, respectively; The first to Nth inverse transform devices are configured to respectively perform inverse Fourier transform on the first to Nth kernel product lights, and respectively output the first to Nth inverse transformed lights.

5. The optical convolution computing device according to claim 4, wherein: Each of the first to Nth transforming means includes at least one lens.

6. The optical convolution computing device according to claim 1, wherein: The image-based SLM is a transmissive SLM, wherein the multi-wavelength light is input to the image-based SLM in a first direction, and the image-based SLM outputs the modulated light to the optical demultiplexer in the first direction, The optical demultiplexer includes first to Nth optical devices arranged in parallel in a first direction, and The first optical device to the Nth optical device of the optical demultiplexer output the first sub-light to the Nth sub-light in a second direction perpendicular to the first direction respectively.

7. The optical convolution computing device according to claim 6, wherein: The first optical device among the first to Nth optical devices of the optical demultiplexer includes at least one dichroic mirror, and each of the remaining optical devices among the first to Nth optical devices of the optical demultiplexer includes at least one beam splitter or at least one polarization beam splitter.

8. The optical convolution computing device according to claim 6, wherein: The first optical device to the Nth optical device of the optical demultiplexer: receiving the modulated light; outputting light corresponding to an Nth wavelength from the first wavelength to an Nth wavelength from the modulated light in a second direction as an Nth sub-light; as well as The remaining light from the modulated light excluding the light corresponding to the Nth wavelength is output in a first direction to the (N-1)th optical device among the first to Nth optical devices of the optical demultiplexer.

9. The optical convolution computing device according to claim 4, wherein: Each of the first to Nth core-based SLMs is a transmissive SLM, and Among them, the first core-based SLM to the first core-based SLM in the Nth core-based SLM: receiving the first converted light from among the first converted light to the Nth converted light in the second direction; and The first converted light is modulated based on the first core data among the first to N-th core data and the first core product light is output in a second direction to the first inverse conversion device among the first to N-th inverse conversion devices.

10. The optical convolution computing device according to claim 4, wherein: The optical multiplexer comprises: The first to N-th optical devices are configured to receive the first to N-th inverse-converted lights in the second direction and align paths of the first to N-th inverse-converted lights to the third direction.

11. The optical convolution computing device according to claim 10, wherein: The third direction is the same as the first direction.

12. The optical convolution computing device according to claim 1, wherein: The image-based SLM is a reflective SLM, The optical convolution computing device further includes: an incident device configured to change the optical path of the multi-wavelength light received from the multi-wavelength light generator from the first direction to a second direction perpendicular to the first direction, and Wherein, the image-based SLM is configured to output the modulated light to the optical demultiplexer in a direction opposite to the second direction.

13. The optical convolution computing device according to claim 12, wherein: Each of the first to Nth core-based SLMs in the optical convolution processor is a reflective SLM, and The optical convolution computing device further includes a first incident device to an Nth incident device respectively arranged in parallel with the optical demultiplexer in a first direction, and Wherein, the first incident device to the Nth incident device are configured as follows: receiving first to N-th core-integrated lights from a first core-based SLM to an N-th core-based SLM respectively in a direction opposite to the first direction; and The optical paths of the first core integrated light to the Nth core integrated light are respectively changed to directions opposite to the second direction, so as to output the first core integrated light to the Nth core integrated light to the optical multiplexer respectively.

14. The optical convolution computing device according to claim 1, wherein: Also included is a multi-wavelength optical generator, wherein the multi-wavelength optical generator includes: The first diffraction grating is configured to receive the first to N-th input lights in different first to N-th directions and align optical paths of the first to N-th input lights in the (N+1)-th direction to output the multi-wavelength light.

15. The optical convolution computing device according to claim 14, wherein: The optical demultiplexer includes a second diffraction grating configured to receive the modulated light and output first to N-th sub-lights having first to N-th wavelengths in different (N+2)-th to (2N+1)-th directions, respectively.

16. The optical convolution computing device according to claim 15, wherein: The first transforming device in the optical convolution processor to the first transforming device in the Nth transforming device receives the first sub-light in the (N+2)th direction, and The optical convolution computing device also includes: The first prism to the (N-1)th prism are configured to respectively receive the second to Nth sub-lights of the first to Nth sub-lights, change the optical paths of the second to Nth sub-lights, and output the second to Nth sub-lights to the second to Nth inverse transformation devices of the first to Nth inverse transformation devices in the optical convolution processor, respectively.

17. The optical convolution computing device according to claim 14, further comprising: a third diffraction grating, and Wherein, the third diffraction grating is configured as: Receiving first to Nth inverse-converted lights respectively; and Paths of the first inverse-converted light to the Nth inverse-converted light are aligned in a single direction to be output as a composite light.

18. The optical convolution computing device according to claim 2, wherein: The multi-wavelength light generator outputs the multi-wavelength light by aligning optical paths of the first input light to the Nth input light.

19. An optical convolution computing device, comprising: a digital demultiplexer configured to modulate the multi-wavelength light based on the spatial domain image data, and output first to N-th sub-lights in a first direction to an N-th direction, respectively; The first to N-th transforming devices are configured to perform Fourier transform on the first to N-th sub-lights, respectively, and output first to N-th transformed lights; The first to N-th digital devices are configured to receive the first to N-th core data and modulate the first to N-th converted light based on the first to N-th core data to output the first to N-th core integrated light; The first to N-th inverse transform devices are configured to perform inverse Fourier transform on the first to N-th kernel product lights, respectively, and output the first to N-th inverse transformed lights; an optical multiplexer configured to align paths of the first inverse-converted light to the Nth inverse-converted light and output a synthesized light; as well as an image sensor configured to sense the synthesized light, Here, N is a natural number greater than or equal to 2.

20. The optical convolution computing device according to claim 19, wherein: Each of the first to N-th digital devices or at least one of the digital demultiplexers includes a digital micromirror device (DMD).

Images