Method and apparatus for performing optical convolution operation based on spatial domain

Abstract

The invention relates to a method and apparatus for performing spatial domain-based optical convolution operations. A method of performing a convolution operation according to an embodiment of the present disclosure may include: performing a first optical Fourier transform on a spatial domain image; performing a second optical Fourier transform on the spatial domain kernel; performing an element-by-element multiplication operation between a result of the first optical Fourier transform and a result of the second optical Fourier transform; calculating a convolution result by performing a third optical Fourier transform on a result of the element-by-element product operation; and obtaining data based on the convolution result.

Description

Technical Field

[0001] The present disclosure relates to optical systems for implementing convolutional optical artificial neural networks, and methods and devices for performing convolution operations of spatial domain kernels and spatial images. Background Art

[0002] As one of the feedforward artificial neural network structures, convolutional artificial neural networks show high accuracy in image classification and reasoning. Convolutional artificial neural networks are widely used in fields such as autonomous driving and the Internet of Things that require image information processing. With the development of wireless communication technology, its utilization is also increasing. Therefore, due to the increasing number of training images and more complex image information, the amount of information processing required is increasing exponentially. However, the image information processing speed of existing electronic computers is slow in terms of the development of energy efficiency and computing speed compared to the increase in the amount of information, and is approaching its limit. Therefore, the demand for computing systems with high processing speed and high energy efficiency is increasing.

[0003] A convolutional artificial neural network is composed of a combination of convolutional layers, pooling layers, fully connected layers, activation layers, etc. Since the convolutional layers that perform convolution operations among these layers account for a high proportion of the energy consumption of the artificial neural network, research is actively being conducted to improve energy efficiency and computing speed by replacing convolution calculations performed by electronic computers with optical convolution computing devices. Summary of the invention

[0004] The technical purpose of the present disclosure is to provide a method and device for performing optical convolution calculation based on the spatial domain.

[0005] The technical object of the present disclosure is to provide a method and apparatus for optically performing a convolution operation of a spatial domain image and a spatial domain kernel by optically Fourier transforming the spatial domain kernel and performing an element-wise product operation using an opto-optical modulator.

[0006] The technical objectives to be achieved by the present disclosure are not limited to the above-mentioned technical objectives, and other technical objectives not described herein will be clearly understood by those skilled in the relevant art from the following description.

[0007] A method for performing a convolution operation according to one aspect of the present disclosure may include: performing a first optical Fourier transform on a spatial domain image; performing a second optical Fourier transform on a spatial domain kernel; performing an element-by-element product operation between a result of the first optical Fourier transform and a result of the second optical Fourier transform; calculating a convolution result by performing a third optical Fourier transform on a result of the element-by-element product operation; and obtaining data based on the convolution result.

[0008] According to another aspect of the present disclosure, a device for performing a convolution operation may include a processor and a memory, and the processor may be configured to: perform a first optical Fourier transform on a spatial domain image; perform a second optical Fourier transform on a spatial domain kernel; perform an element-by-element product operation between a result of the first optical Fourier transform and a result of the second optical Fourier transform; calculate a convolution result by performing a third optical Fourier transform on a result of the element-by-element product operation; and obtain data based on the convolution result.

[0009] As one or more non-transitory computer-readable media storing one or more instructions according to another aspect of the present disclosure, the one or more instructions can be executed by one or more processors and control a device for performing a convolution operation to perform the following operations: perform a first optical Fourier transform on a spatial domain image; perform a second optical Fourier transform on a spatial domain kernel; perform an element-by-element product operation between a result of the first optical Fourier transform and a result of the second optical Fourier transform; calculate a convolution result by performing a third optical Fourier transform on a result of the element-by-element product operation; and obtain data based on the convolution result.

[0010] In various aspects of the present disclosure, an element-wise product operation may be performed by the optical-optical modulator.

[0011] Additionally, in various aspects of the present disclosure, the wavelength of the control beam associated with the spatial domain kernel and the wavelength of the signal beam associated with the spatial domain image may be selected based on the modulation characteristics of the optical-optical modulator.

[0012] In addition, in various aspects of the present disclosure, when the wavelength of the control beam and the wavelength of the signal beam are different, a first optical Fourier transform may be performed by the first lens, and a second optical Fourier transform may be performed by the second lens.

[0013] In addition, in various aspects of the present disclosure, the focal length of the first lens may be selected based on the wavelength of the signal beam and the dimension of the Fourier domain, and the focal length of the second lens may be selected based on the wavelength of the control beam and the dimension of the Fourier domain.

[0014] In addition, in various aspects of the present disclosure, in the case where the wavelength of the control light beam and the wavelength of the signal light beam are the same, the first optical Fourier transform and the second optical Fourier transform may be performed by the same lens.

[0015] Additionally, in various aspects of the present disclosure, the spatial domain image may correspond to a function generated by applying spatial light intensity modulation to the signal beam, and the spatial domain kernel may correspond to a function generated by applying spatial light intensity modulation to the control beam.

[0016] Additionally, in various aspects of the present disclosure, data may be obtained based on measurements taken by a camera that receives a convolution result as input.

[0017] Additionally, in various aspects of the present disclosure, an element capable of emitting light may be used to block a control light beam associated with the spatial domain kernel from being input to the camera.

[0018] According to the present disclosure, a method and apparatus for performing optical convolution calculation based on the spatial domain may be provided.

[0019] According to the present disclosure, a method and apparatus for optically performing a convolution operation of a spatial domain image and a spatial domain kernel by optically Fourier transforming the spatial domain kernel and performing an element-wise product operation using an optical-optical modulator may be provided.

[0020] According to the present disclosure, for an optical convolution operation based on the spatial domain, by optically processing the Fourier transform process that must be executed in an electronic computer, it has the advantages of improving energy efficiency and achieving fast calculation speed.

[0021] Effects achievable through the present disclosure are not limited to the above-described effects, and other effects not described herein may be clearly understood by those skilled in the relevant art from the following description. BRIEF DESCRIPTION OF THE DRAWINGS

[0022] Figure 1 The operation structure of the existing optical convolution operator based on the spatial domain kernel is shown.

[0023] Figure 2 The operation structure of the existing Fourier domain kernel-based optical convolution operator is shown.

[0024] Figure 3 The operation structure of the optical convolution operator based on the spatial domain kernel according to an embodiment of the present disclosure is shown.

[0025] Figure 4 The optical structure of the optical convolution operator based on the spatial domain according to an embodiment of the present disclosure is shown.

[0026] Figure 5 An example of the structure of an optical convolution operator based on a spatial domain considering the wavelengths of a signal beam and a control beam according to an embodiment of the present disclosure is shown.

[0027] Figure 6 Another example of the structure of the spatial domain-based optical convolution operator considering the wavelengths of the signal beam and the control beam according to an embodiment of the present disclosure is shown.

[0028] Figure 7 The structure of an optical convolution operator based on the spatial domain considering optical path correction according to an embodiment of the present disclosure is shown.

[0029] Figure 8 An operational flow chart of a method for performing an optical convolution operation according to an embodiment of the present disclosure is shown.

[0030] Fig. 9 is a block diagram illustrating an apparatus according to an embodiment of the present disclosure. DETAILED DESCRIPTION

[0031] Since the present disclosure can be variously changed and has multiple embodiments, specific embodiments are shown in the drawings and described in detail in the specific embodiments. However, the present disclosure is not limited to specific embodiments, and should be understood to include all changes, equivalents and substitutes included in the thought and technical scope of the present disclosure. Similar reference numerals in the drawings refer to the same or similar functions in multiple aspects. For a clearer description, the shapes and sizes of the elements in the drawings, etc. may be exaggerated. The detailed description of the exemplary embodiments described below refers to the drawings showing specific embodiments as examples. These embodiments are described in detail to enable technicians in the relevant fields to implement the embodiments. It should be understood that the various embodiments are different from each other, but they do not need to be mutually exclusive. For example, without departing from the scope and spirit of the present disclosure in conjunction with the embodiments, the specific shapes, structures and characteristics described herein can be implemented in other embodiments. In addition, it should be understood that the position or arrangement of each element in each disclosed embodiment can be changed without departing from the scope and spirit of the embodiments. Therefore, the specific embodiments described below are not regarded as restrictive meanings, and if properly described, the scope of the exemplary embodiments is limited only by the attached claims and any scope equivalent to the scope claimed by these claims.

[0032] In the present disclosure, terms such as first, second, etc. may be used to describe various elements, but elements should not be limited by the terms. These terms are only used to distinguish one element from another element. For example, without departing from the scope of the present disclosure, the first element may be referred to as the second element, and similarly, the second element may also be referred to as the first element. The term "and / or" includes a combination of multiple related description items or any item in multiple related description items.

[0033] When an element in the present disclosure is referred to as being "connected" or "linked" to another element, it should be understood that it can be directly connected or linked to the other element, but another element may exist between them. At the same time, when an element is referred to as being "directly connected" or "directly linked" to another element, it should be understood that another element does not exist between them.

[0034] Since the structural units shown in the embodiments of the present disclosure are shown independently to represent different characteristic functions, it does not mean that each structural unit is composed of a separate hardware structural unit or a software structural unit. In other words, since each structural unit is included by being listed as each structural unit for the convenience of description, at least two structural units of each structural unit can be combined to form a structural unit, or a structural unit can be divided into multiple structural units to perform functions, and the integrated embodiment and separate embodiment of each structural unit are also included in the scope of the rights of the present disclosure, unless they exceed the essence of the present disclosure.

[0035] The terms used in this disclosure are only used for the purpose of describing specific embodiments and are not intended to limit the present disclosure. Unless the context clearly indicates otherwise, singular expressions include plural expressions. In the present disclosure, it should be understood that terms such as "including", "having" and the like are only intended to specify the presence of features, numbers, steps, operations, elements, parts or combinations thereof described in this specification, and do not preclude the possibility of the presence or addition of one or more other features, numbers, steps, operations, elements, parts or combinations thereof. In other words, the description of "including" a specific configuration in the present disclosure does not exclude configurations other than the corresponding configuration, and it means that additional configurations may be included in the technical concept of the present disclosure or the scope of the embodiments of the present disclosure.

[0036] Some elements of the present disclosure are not necessary elements for performing necessary functions in the present disclosure, and may be optional elements used only to improve performance. In addition to elements used only for performance improvement, the present disclosure may be implemented by including only configuration units necessary for realizing the essence of the present disclosure, and in addition to optional elements used only for performance improvement, structures including only necessary elements are also included in the scope of the present disclosure.

[0037] Hereinafter, embodiments of the present disclosure will be described in detail with reference to the accompanying drawings. When describing the embodiments of the present specification, when it is determined that a detailed description of the configuration or function of the relevant disclosure may obscure the main purpose of the present specification, such detailed description will be omitted, and the same reference numerals are used for the same elements in the accompanying drawings, and repeated descriptions of the same elements will be omitted.

[0038] To meet the demand for computing systems with high processing speed and high energy efficiency, research on convolutional artificial neural networks based on optical convolution operations is ongoing.

[0039] In this regard, unlike most existing convolutional artificial neural networks that use spatial domain kernels, the optical convolution computing device is optically designed to perform calculations using Fourier domain kernels. In order to apply the spatial domain kernel to the computing device, it is necessary to input / insert the Fourier domain kernel obtained by Fourier transforming the spatial domain kernel using an electronic computer, etc. into the convolution computing device. This Fourier transform process has the disadvantage of providing additional work to the electronic computer, etc., thereby offsetting the advantages of the optical convolution computing device.

[0040] In addition, there are systems that use Fourier domain kernels directly for learning instead of spatial domain kernels, but the Fourier domain kernel is usually tens to hundreds of times larger than the spatial domain kernel, so it cannot be said to be a suitable computing structure to achieve high energy efficiency and fast computing speed.

[0041] Therefore, there is a need for an optical convolution computing device that can perform convolution computing by directly applying a spatial domain kernel to a spatial domain image.

[0042] In view of the above points, in the present disclosure, a method and apparatus are provided for optically performing Fourier transform on a spatial domain kernel and optically performing a convolution operation of a spatial domain image (i.e., an input image) and the spatial domain kernel by performing element-by-element product calculations using an optical-optical modulator.

[0043] The existing optical convolution calculation method using spatial domain images and Fourier domain kernels has disadvantages that offset the advantages of optical convolution calculation devices (such as high energy efficiency and fast calculation speed). In contrast, the optical convolution calculation method and device proposed in the present disclosure have the technical effect of minimizing unnecessary calculations and energy consumption that should be performed by electronic computers, etc. in existing optical convolution calculation devices.

[0044] Typically, the optical convolution operator is applied to two functions (e.g., f(x, y), g (x, y)) is Fourier transformed in order to compute the convolution of the two functions, and after performing the element-by-element product of the two Fourier transformed functions (e.g., F(x′, y′), G(x′, y′)), the convolution theorem can be used, which states that the result is the same as the inverse Fourier transform.

[0045] Its relational expression can be expressed as the following Equation 1.

[0046] [Equation 1]

[0047]

[0048] The general operation structure of an optical convolution calculator using Equation 1 can be expressed as Figure 1 shown.

[0049] Figure 1 The operation structure of the existing optical convolution operator based on the spatial domain kernel is shown.

[0050] Reference Figure 1 , the input image, that is, the spatial domain image (f(x, y)) may be transmitted in the form of amplitude of light through a first spatial light modulator (SLM), and it may be optically Fourier transformed through a lens or the like (S110).

[0051] In addition, the spatial domain kernel (g(x, y)) may be Fourier transformed using an electronic computer or the like, and may be input / inserted into the second SLM for element-wise product as a function of amplitude (S120).

[0052] Then, through the second SLM, an element-by-element product between the Fourier transformed image (F(x′, y′)) (i.e., the result in S110) and the Fourier domain kernel (G(x′, y′)) can be performed, and the result can be optically Fourier transformed and output as a convolution result (S130).

[0053] Finally, a camera or the like may be used to acquire data corresponding to the convolution result.

[0054] For this method, since operations such as electronic computers are required to perform Fourier transforms on the spatial domain kernel (g(x, y)), the advantages of the optical convolution calculator, which aims to achieve high energy efficiency and fast computational speed by minimizing the role of electronic devices, may be offset.

[0055] Figure 2 The operation structure of the existing Fourier domain kernel-based optical convolution operator is shown.

[0056] Reference Figure 2 , through the first SLM, the input image, that is, the spatial domain image (f(x, y)) can be transmitted in the form of amplitude of light, and it can be optically Fourier transformed through a lens, etc. (S210).

[0057] In addition, the Fourier domain kernel (G(x′, y′)) obtained by direct learning in the Fourier domain may be input / inserted into the second SLM for element-wise product ( S220 ).

[0058] Then, through the second SLM, an element-by-element product between the Fourier transformed image (F(x′, y′)) (i.e., the result in S210) and the Fourier domain kernel (G(x′, y′)) can be performed, and the result can be optically Fourier transformed and output as a convolution result (S230).

[0059] Finally, a camera or the like may be used to acquire data corresponding to the convolution result.

[0060] In the case of this method, although it has advantages in that Fourier transform using an electronic computer is not performed, the data size of the Fourier domain kernel is tens to hundreds of times larger than that of the spatial domain kernel, which is disadvantageous in constructing an efficient artificial neural network.

[0061] Therefore, there is a need to develop an optical computing system that directly uses the spatial domain kernel but does not require additional calculation / processing by an electronic computer to perform Fourier transform.

[0062] Figure 3 The operation structure of the optical convolution operator based on the spatial domain kernel according to an embodiment of the present disclosure is shown.

[0063] Reference Figure 3 , which is different from the existing convolution operator calculation methods (e.g. Figure 1 and Figure 2 ), the spatial domain kernel can be optically Fourier transformed and can be applied to convolution.

[0064] Through the first SLM, the input image, that is, the spatial domain image (f(x, y)) can be transmitted in the form of amplitude of light, and it can be optically Fourier transformed through a lens, etc. In addition, through the second SLM, the spatial domain kernel can be transmitted in the form of amplitude of light, and it can be optically Fourier transformed through a lens, etc. (S310).

[0065] An optical-optical modulator may be used to perform an element-by-element product in the Fourier domain on an optical Fourier transformed image (F(x′, y′)) and an optical Fourier transformed spatial domain kernel (G(x′, y′)), and the result may be optically Fourier transformed and output as a convolution result (S320).

[0066] Finally, a camera or the like may be used to acquire data corresponding to the convolution result.

[0067] In the case of this method, it is based on a spatial domain kernel with a small data size and optically processes the Fourier transform, so it has the advantage of not requiring the Fourier transform of the kernel to be calculated / processed by an electronic computer. Therefore, the advantages of an optical convolution computing device with high energy efficiency and high parallelism can be maximized.

[0068] Figure 4 The optical structure of the optical convolution operator based on the spatial domain according to an embodiment of the present disclosure is shown.

[0069] Figure 4 The optical structure shown corresponds to the optical structure of a spatial domain optical convolution operator, which can directly convolve the spatial domain kernel and the input image (ie, the spatial domain image).

[0070] Two parallel light beams, referred to as a signal beam and a control beam, may each be incident on a spatial light intensity modulator (SLM).

[0071] For example, the signal beam is incident on the first SLM, and the first SLM can transmit data with an optical amplitude having a function of the spatial domain image. In addition, the control beam is incident on the second SLM, and the second SLM can transmit data with an optical amplitude having a function of the spatial domain kernel.

[0072] As described above, the two parallel lights whose light intensities have been modulated may each be optically Fourier transformed through a lens.

[0073] The optical-optical modulator may be located in a plane where the optical Fourier transform light is generated, and an element-wise product operation between two optical Fourier transform functions may be performed by the optical-optical modulator.

[0074] In this regard, since the control beam intensity is sufficiently lower than the saturation power, the signal beam transmittance (T(λ)) of the optical-optical modulator is λ / (λ) ... s )) can be expressed as the following Equation 2.

[0075] [Equation 2]

[0076] T(λ s )=A·P(λ c )+T0

[0077] In Equation 2, λ c and λ s denote the wavelengths of the control beam and the signal beam, respectively, P(λ c ) represents the light intensity of the control beam, A represents T(λ s ) According to P(λ c ), T0 represents the transmittance of the signal beam when the control beam is not inserted.

[0078] In Equation 2, it is assumed that the light intensity of the signal light beam does not affect the transmittance of the optical-optical modulator.

[0079] For example, when an optical-optical modulator (such as a smart pixel) is used, the intensity of the signal rarely affects the transmittance, but when a saturable absorber is used as the optical-optical modulator, the intensity of the signal beam must be low enough not to exhibit saturable absorption characteristics that depend on the intensity of the signal beam. In other words, the transmittance (T(λ)) of the optical-optical modulator s )) must be an independent function of the signal beam intensity. A saturable absorber as one of the optical-optical modulators has a positive value A, and an optical limiter has a negative value A. In a smart pixel, A can be positive or negative depending on the design of the device.

[0080] The wavelengths of the signal beam and the control beam should be selected based on / suitable for the modulation characteristics of the optical-optical modulator, and the two wavelengths may be the same or different depending on the type of the optical-optical modulator. For example, for a two-dimensional saturable absorber (such as graphene or smart pixels), the transmittance of a signal beam with a different wavelength may be modulated according to the output of the adjusted light, but if the absorption wavelength band of the saturable absorber is narrow, the wavelengths of the control beam and the output light may need to be designed to be the same.

[0081] When the light intensity of the signal beam incident on the optical modulator is called P in (λ s ), the light intensity (P out (λ s )) can be expressed as the following Equation 3.

[0082] [Equation 3]

[0083] P out (λ s )=A·P(λ c )·P in (λ s )+T0·P in (λ s )

[0084] The signal beam transmitted or reflected from the optical-optical modulator may pass through a lens again, undergo an optical Fourier transform, and be measured using a camera.

[0085] The light intensity measured by the camera based on the above equation 3 The function of can be expressed as the following equation 4.

[0086] [Equation 4]

[0087]

[0088] The final measured light intensity function is shown in Equation 4. represents the term used to obtain the result of the convolution of two functions. In addition, The term is an unnecessary calculation result, and the corresponding value can be removed by correction to use only the convolution calculation result. Here A and T0 are constants that can be measured in advance during the optical system calibration stage, and f(x, y) is a spatial domain image function, so it can be easily corrected.

[0089] After correcting Equation 4, the convolution result value can be derived by converting to an optical amplitude function, and this can be expressed as the following Equation 5.

[0090] [Equation 5]

[0091]

[0092] In the following, specific examples for implementing the spatial domain kernel and the optical convolution operator on a spatial domain image will be described.

[0093] Figure 5 An example of the structure of an optical convolution operator based on a spatial domain considering the wavelengths of a signal beam and a control beam according to an embodiment of the present disclosure is shown.

[0094] Figure 5 The structure shown illustrates the structure of the spatial domain optical convolution operator when the wavelengths of the signal beam and the control beam are different.

[0095] For example, by using two spatial light intensity modulators (SLMs) and parallel light transmitted or reflected through the SLMs, the spatial domain kernel and the spatial domain image can be modulated as a function of the optical amplitude.

[0096] The two modulation functions can be Fourier transformed by each lens (i.e., lens 1, lens 2). Specifically, the function that modulates the spatial domain kernel can be optically Fourier transformed by lens 1, and the function that modulates the spatial domain image can be optically Fourier transformed by lens 2.

[0097] In addition, the two optical paths can be adjusted to match the direction of the optical-optical modulator by using a device capable of combining light, such as a polarization beam splitter (PBS) or a dichroic mirror. At this time, the focal length of each lens can be selected by considering the two wavelengths of the control beam and the signal beam and the dimension of the Fourier domain.

[0098] In this regard, the correlation between the dimension of the Fourier domain, the wavelength, and the focal length of the lens can be expressed as the following Equation 6.

[0099] [Equation 6]

[0100]

[0101] In Equation 6, k x represents the spatial frequency, x i represents the coordinates of the Fourier plane, λ represents the wavelength of light, and f0 represents the focal length of the lens.

[0102] Therefore, in order to match the dimensions of the two Fourier transform functions on the plane during photo-optical modulation, the two wavelengths and the focal length of each lens can be optically designed to follow the relationship shown in Equation 7 below.

[0103] [Equation 7]

[0104]

[0105] In Equation 7, λ c represents the wavelength of the controlled beam, λ s represents the wavelength of the signal beam, f c represents the focal length of the lens (i.e., lens 1) that performs Fourier transform on the control beam, f s represents the focal length of the lens (ie, lens 2) that performs Fourier transform on the signal beam.

[0106] Regarding the spatial domain kernel and spatial domain image of the Fourier transform, an element-by-element product operation can be performed by an optical-optical modulator, the signal beam can undergo (optical) Fourier transform again through a lens (i.e., lens 3), and the final calculation result can be measured using a camera.

[0107] In this regard, the convolution operation result can be obtained by removing the second term in the above equation 3 from the measured light intensity and converting the measured light intensity value into an optical amplitude function. At this time, a polarization beam splitter, a dichroic mirror, or a bandpass filter can be used to prevent the control beam from entering the camera.

[0108] Figure 6 Another example of the structure of the spatial domain-based optical convolution operator considering the wavelengths of the signal beam and the control beam according to an embodiment of the present disclosure is shown.

[0109] Figure 6 The structure shown illustrates the structure of the spatial domain optical convolution operator when the wavelengths of the signal beam and the control beam are the same.

[0110] Reference Figure 6 For the spatial domain image and the spatial domain kernel, the two optical paths can be adjusted to match the directions of the light-optical modulators by using a device that can combine light, such as a polarization beam splitter (PBS) or a dichroic mirror.

[0111] Then, through lens 1, the spatial domain kernel and the spatial domain image can be optically Fourier transformed simultaneously.

[0112] Regarding the spatial domain kernel and spatial domain image of the Fourier transform, an element-by-element product operation can be performed by an optical-optical modulator, the signal beam can undergo (optical) Fourier transform again through a lens (i.e., lens 2), and the final calculation result can be measured using a camera.

[0113] In this regard, the convolution operation result can be obtained by removing the second term in the above equation 3 from the measured light intensity and converting the measured light intensity value into an optical amplitude function. At this time, the control beam can be prevented from entering the camera by using a polarization beam splitter, a dichroic mirror, or a bandpass filter.

[0114] Regarding the above Figure 5 and Figure 6 , the spatial light intensity modulator (SLM) that can be used in the optical convolution calculator can correspond to liquid crystal, digital micromirror device (DMD), etc.

[0115] If the incident light is transmitted or reflected and then travels perpendicular to the plane where the SLM is located, the optical convolution calculator can be designed to have the same Figure 5 or Figure 6 Same structure.

[0116] Conversely, if the light travels at an angle rather than perpendicular to the SLM plane due to diffraction (for example, if the SLM of the optical convolution computer is DMD-based), then the optical path may need to be corrected.

[0117] Figure 7 The structure of an optical convolution operator based on the spatial domain considering optical path correction according to an embodiment of the present disclosure is shown.

[0118] Reference Figure 7 , with respect to the control beam and the signal beam incident at a specific angle rather than perpendicular to the SLM plane, optical path correction through a wedge prism may be performed.

[0119] Thus, the control beam associated with the spatial domain kernel and the signal beam associated with the spatial domain image can be perpendicularly incident on each lens.

[0120] Figure 8 An operational flow chart of a method for performing an optical convolution operation according to an embodiment of the present disclosure is shown.

[0121] Reference Figure 8 , the optical convolution computing device may perform a first optical Fourier transform on the spatial domain image (S810).

[0122] In addition, the optical convolution operation device may perform a second optical Fourier transform on the spatial domain kernel (S820).

[0123] For example, the spatial domain image may be based on a function generated by applying spatial light intensity modulation to a signal beam, and the spatial domain kernel may be based on a function generated by applying spatial light intensity modulation to a control beam. That is, each spatial light intensity modulator (SLM) may use a signal beam or parallel light to modulate a spatial domain image or a spatial domain kernel as a function of an optical amplitude.

[0124] Thereafter, the optical convolution calculation device may perform an element-by-element product operation between a result of the first optical Fourier transform and a result of the second optical Fourier transform (S830).

[0125] For example, an element-wise product operation may be performed by an optical-optical modulator.

[0126] The optical convolution calculation device may calculate a convolution result by performing a third optical Fourier transform on the result of the above-mentioned element-by-element product operation (S840), and may obtain data based on the corresponding convolution result (S850).

[0127] For example, data may be obtained based on measurements of a camera that receives the convolution result as input. In this regard, an element capable of emitting light may be used to prevent a control beam associated with the spatial domain kernel from being input to the camera.

[0128] With respect to the foregoing process, the wavelength of the control light / beam associated with the spatial domain kernel and the wavelength of the signal light / beam associated with the spatial domain image may be selected based on the modulation characteristics of the optical-optical modulator.

[0129] At this time, when the wavelength of the control light / beam and the wavelength of the signal light / beam are different, a first optical Fourier transform may be performed by the first lens, and a second optical Fourier transform may be performed by the second lens. Here, the focal length of the first lens may be selected based on the wavelength of the signal light / beam and the dimension of the Fourier domain, and the focal length of the second lens may be selected based on the wavelength of the control light / beam and the dimension of the Fourier domain.

[0130] In contrast, when the wavelength of the control light / beam and the wavelength of the signal light / beam are the same, the first optical Fourier transform and the second optical Fourier transform may be performed by the same lens.

[0131] Fig. 9 is a block diagram illustrating a device according to an embodiment of the present disclosure.

[0132] Reference Fig. 9 , device 900 may represent a device that implements the method described in the present disclosure for performing a spatial domain-based optical convolution operation.

[0133] The device 900 may include at least one of a processor 910, a memory 920, a transceiver 930, an input interface device 940, and an output interface device 950. Each of the components may be connected to communicate with each other through a common bus 960. In addition, each of the components may be connected through a separate interface or a separate bus centered around the processor 910 instead of the common bus 960.

[0134] The processor 910 may be implemented in various types, such as an application processor (AP), a central processing unit (CPU), a graphics processing unit (GPU), etc., and may be any semiconductor device that executes commands stored in the memory 920. The processor 910 may execute program instructions stored in the memory 920. The processor 910 may be configured to implement the above-mentioned Figures 1 to 8 A method for performing an optical convolution operation based on the spatial domain is described.

[0135] And / or, the processor 910 may store program instructions for implementing at least one function of the corresponding module in the memory 920, and may control the execution of the program instructions based on the corresponding module. Figures 1 to 8 Describes the operation.

[0136] The memory 920 may include various types of volatile or non-volatile storage media. For example, the memory 920 may include a read-only memory (ROM) and a random access memory (RAM). In an embodiment of the present disclosure, the memory 920 may be located inside or outside the processor 910, and the memory 920 may be connected to the processor 910 by various known means.

[0137] The transceiver 930 may perform a function of transmitting and receiving data processed / to be processed by the processor 910 with an external device and / or an external system.

[0138] The input interface device 940 is configured to provide data to the processor 910 .

[0139] The output interface device 950 is configured to output data from the processor 910 .

[0140] As described above, in the present disclosure, the optical convolution calculator has the advantage of improving energy efficiency and calculation speed by using an optical calculator to calculate convolutions that account for a large proportion of the total calculation amount of the convolutional artificial neural network.

[0141] However, because the conventional optical convolution calculator will use the Fourier domain kernel, the number of learning variables for the artificial neural network increases, or an electronic computer will be used to perform additional Fourier transform operations. This may offset the advantages of the optical convolution calculator (such as high energy efficiency and fast calculation speed).

[0142] In contrast, the optical convolution operation device according to the present disclosure is a system that optically processes the convolution of the spatial domain kernel and the spatial domain image by optically Fourier transforming both of the spatial domain kernel and the spatial domain image and performing an element-wise product operation using an optical-optical modulator. In this case, by optically processing the Fourier transform process that must be performed in an electronic computer, energy efficiency can be improved and a fast calculation speed can be achieved. In addition, for the optical convolution operation device according to the present disclosure, when a passive optical-optical modulator that does not require a separate power supply is applied, the advantage can be maximized by performing the calculation in real time without using additional energy for the element-wise product calculation.

[0143] The components described in the example embodiments may be implemented by hardware components, including, for example, at least one digital signal processor (DSP), a processor, a controller, an application specific integrated circuit (ASIC), a programmable logic element (such as an FPGA), a GPU, other electronic devices, or a combination thereof. At least some of the functions or processes described in the example embodiments may be implemented by software, and the software may also be recorded on a recording medium. The components, functions, and processes described in the example embodiments may be implemented by a combination of hardware and software.

[0144] The method according to the example embodiment may be implemented as a program executable by a computer, and may be implemented as various recording media such as a magnetic storage medium, an optical reading medium, and a digital storage medium.

[0145] The various techniques described herein may be implemented as digital electronic circuitry, or as computer hardware, firmware, software, or a combination thereof. These techniques may be implemented as a computer program product, i.e., a computer program tangibly embodied in an information carrier (e.g., in a machine-readable storage device (e.g., a computer-readable medium) or in a propagated signal) for processing by a data processing apparatus (e.g., a programmable processor, a computer, or multiple computers), or to control the operation of a data processing apparatus.

[0146] The computer program(s) may be written in any form of programming language, including compiled or interpreted languages, and may be deployed in any form, including stand-alone programs or modules, components, subroutines, or other units suitable for a computing environment. A computer program may be deployed to be executed on one computer or on multiple computers at one site or distributed across multiple sites and interconnected by a communication network.

[0147] As an example, processors suitable for executing computer programs include both general-purpose microprocessors and special-purpose microprocessors, and any one or more processors of any kind of digital computer. Typically, the processor will receive instructions and data from a read-only memory or a random access memory or both. The elements of a computer may include at least one processor for executing instructions and one or more memory devices for storing instructions and data. Typically, the computer will also include or be coupled to receive data from one or more mass storage devices (e.g., magnetic disks, magneto-optical disks, or optical disks), send data to one or more mass storage devices, or perform both on one or more mass storage devices to store data. Examples of information carriers suitable for embodying computer program instructions and data include semiconductor memory devices, for example, magnetic media such as hard disks, floppy disks, and tapes, optical media such as compact disk read-only memories (CD-ROMs), digital video disks (DVDs), and magneto-optical media such as floppy disks, as well as read-only memories (ROMs), random access memories (RAMs), flash memories, erasable programmable ROMs (EPROMs), and electrically erasable programmable ROMs (EEPROMs), and any other known computer-readable media. The processor and memory may be supplemented by or integrated into a dedicated logic circuit.

[0148] The processor may run an operating system (OS) and one or more software applications running on the OS. The processor device may also access, store, manipulate, process, and create data in response to the execution of the software. For simplicity, the description of the processor device is used as a singular; however, those skilled in the art will appreciate that the processor device may include multiple processing elements and / or multiple types of processing elements. For example, the processor device may include multiple processors or a processor and a controller. In addition, different processing configurations are possible, such as parallel processors. In addition, non-transitory computer-readable media may be any available media that can be accessed by a computer, and may include both computer storage media and transmission media.

[0149] While this specification includes many specific implementation details, it should be understood that these details are not limitations of any inventions or of what may be claimed in the specification, but rather are descriptions of features of particular example embodiments.

[0150] Features described in the specification in the context of various example embodiments may be implemented as combinations in a single example embodiment. Conversely, various features described in the specification in the context of a single example embodiment may be implemented individually in multiple example embodiments or in appropriate sub-combinations. In addition, although features may function in a particular combination and are initially described as being claimed in that combination, in some cases, one or more features may be excluded from the claimed combination, and the claimed combination may be changed to a sub-combination or a modification of the sub-combination.

[0151] Similarly, although the operations are described in a particular order in the accompanying drawings, it should not be understood that the operations need to be performed in a particular order or sequence to obtain the desired results, or that all operations need to be performed. In certain cases, multitasking and parallel processing may be advantageous. In addition, it should not be understood that the various device components in the above-mentioned example embodiments need to be separated in all example embodiments, and it should be understood that the above-mentioned program components and devices can be integrated into a single software product or can be packaged in multiple software products.

[0152] It should be understood that the example embodiments disclosed herein are illustrative only and are not intended to limit the scope of the invention. It will be apparent to those skilled in the art that various modifications may be made to the example embodiments without departing from the spirit and scope of the claims and their equivalents.

[0153] Accordingly, the present disclosure is intended to embrace all other alternatives, modifications and variations that fall within the scope of the appended claims.

Claims

1. A method for performing a convolution operation, the method comprising: performing a first optical Fourier transform on the spatial domain image; performing a second optical Fourier transform on the spatial domain kernel; performing an element-wise product operation between a result of the first optical Fourier transform and a result of the second optical Fourier transform; computing a convolution result by performing a third optical Fourier transform on the result of the element-wise product operation; and The data is obtained based on the convolution result.

2. The method according to claim 1, in, The element-wise product operation is performed by the optical-optical modulator.

3. The method according to claim 2, in, The wavelength of the control beam associated with the spatial domain kernel and the wavelength of the signal beam associated with the spatial domain image are selected based on the modulation characteristics of the optical-optical modulator.

4. The method according to claim 3, in, In the case where the wavelength of the control beam and the wavelength of the signal beam are different, a first optical Fourier transform is performed by the first lens, and a second optical Fourier transform is performed by the second lens.

5. The method according to claim 4, in, The focal length of the first lens is selected based on the wavelength of the signal beam and the dimensionality of the Fourier domain, and The focal length of the second lens is selected based on the wavelength of the control beam and the dimensionality of the Fourier domain.

6. The method according to claim 3, in, In the case where the wavelength of the control light beam and the wavelength of the signal light beam are the same, the first optical Fourier transform and the second optical Fourier transform are performed by the same lens.

7. The method according to claim 1, in, The spatial domain image corresponds to a function generated by applying a spatial light intensity modulation to the signal beam, and Therein, the spatial domain kernel corresponds to a function generated by applying a spatial light intensity modulation to the control beam.

8. The method according to claim 1, in, The data is based on measurements made by a camera that receives the convolution result as input.

9. The method according to claim 8, in, An element capable of emitting light is used to block a control beam associated with the spatial domain kernel from being input to the camera.

10. A device for performing a convolution operation, the device comprising: processor and memory, The processor is configured as follows: performing a first optical Fourier transform on the spatial domain image; performing a second optical Fourier transform on the spatial domain kernel; performing an element-wise product operation between a result of the first optical Fourier transform and a result of the second optical Fourier transform; computing a convolution result by performing a third optical Fourier transform on the result of the element-wise product operation; and The data is obtained based on the convolution result.

11. The device according to claim 10, in, The element-wise product operation is performed by the optical-optical modulator.

12. The device according to claim 11, in, The wavelength of the control beam associated with the spatial domain kernel and the wavelength of the signal beam associated with the spatial domain image are selected based on the modulation characteristics of the optical-optical modulator.

13. The device according to claim 12, in, In the case where the wavelength of the control beam and the wavelength of the signal beam are different, a first optical Fourier transform is performed by the first lens, and a second optical Fourier transform is performed by the second lens.

14. The device according to claim 13, in, The focal length of the first lens is selected based on the wavelength of the signal beam and the dimensionality of the Fourier domain, and The focal length of the second lens is selected based on the wavelength of the control beam and the dimensionality of the Fourier domain.

15. The device according to claim 12, in, In the case where the wavelength of the control light beam and the wavelength of the signal light beam are the same, the first optical Fourier transform and the second optical Fourier transform are performed by the same lens.

16. The device according to claim 10, in, The spatial domain image corresponds to a function generated by applying a spatial light intensity modulation to the signal beam, and Therein, the spatial domain kernel corresponds to a function generated by applying a spatial light intensity modulation to the control beam.

17. The device according to claim 10, in, The data is based on measurements made by a camera that receives the convolution result as input.

18. The device according to claim 17, in, An element capable of emitting light is used to block a control beam associated with the spatial domain kernel from being input to the camera.

19. One or more non-transitory computer-readable media storing one or more instructions, in, The one or more instructions are executed by one or more processors and control a device for performing a convolution operation to perform the following operations: performing a first optical Fourier transform on the spatial domain image; performing a second optical Fourier transform on the spatial domain kernel; performing an element-wise product operation between a result of the first optical Fourier transform and a result of the second optical Fourier transform; computing a convolution result by performing a third optical Fourier transform on the result of the element-wise product operation; and The data is obtained based on the convolution result.

20. The computer readable medium of claim 19, in, The spatial domain image corresponds to a function generated by applying a spatial light intensity modulation to the signal beam, and Therein, the spatial domain kernel corresponds to a function generated by applying a spatial light intensity modulation to the control beam.

Images