A photon deconvolution acceleration system

Abstract

The application discloses a photonic deconvolution acceleration system, which is composed of a laser, a light source modulator, an optical beam splitter, an MZI, a phase shifter and an optical balanced detector to complete the deconvolution operation, replaces the deconvolution operation layer in the Spiking-FlowNet neural network architecture which occupies a large amount of computing power, and can greatly improve the computing speed and overcome the slow problem of the electric-based deconvolution operation. Meanwhile, the array formed by the MZI has the characteristics of strong reconfigurability, and the phase shifter can be used to change the transmission matrix of the MZI array, so that the deconvolution operation process of different channels can be completed. When the photonic deconvolution acceleration system is applied to optical flow estimation, an accelerated hybrid optoelectronic Spiking-FlowNet neural network architecture is designed, which fully utilizes the characteristics of parallelism, low power consumption and light speed calculation of optical calculation, and can complete the optical flow estimation more quickly and energy-efficiently while retaining the original accuracy of the Spiking-FlowNet neural network architecture.

Description

TECHNICAL FIELD

[0001] The present application relates to the field of image processing and computer vision, and particularly relates to a photon deconvolution acceleration system. BACKGROUND

[0002] Optical flow (OF) estimation is a technology for predicting the movement trend of an observed object based on the brightness change. The principle is to calculate the correlation between the pixel points in the adjacent frames of the image sequence based on the correlation between the pixel points, using the strong learning and approximation ability of the neural network. In recent decades, optical flow estimation has been a core research topic in the field of image processing and computer vision.

[0003] With the improvement of computer hardware and software level, optical flow estimation plays an increasingly important role in tracking, motion segmentation, navigation, image registration, distance warning and other application fields. Especially in high-speed motion scenes or low-light environments, due to the rapid change of the spatial object position or the extremely low light intensity, the traditional camera is almost in a state of failure, at this time, the camera that is more sensitive to environmental changes, event camera, is needed to provide the original scene materials for optical flow estimation.

[0004] Event camera, commonly known as bionic silicon retina, is a camera that can detect the logarithmic scale of brightness change asynchronously and independently through each pixel array unit (similar to retinal ganglion cells). Event camera optical flow estimation, that is, using the high time resolution (microsecond level) photos taken by the event camera and the optical flow estimation neural network architecture, can predict the movement change of the observed object, and plays an important role in protecting the life and property safety of users in the field of vehicle driving warning and monitoring discrimination.

[0005] In event camera optical flow estimation, a large number of neural network architectures (such as LiteFlowNet architecture and Spiking-FlowNet architecture) all introduce convolution and deconvolution to reduce or increase the dimension of the picture, occupying a large amount of computing power of the optical flow estimation architecture. LiteFlowNet architecture is a lightweight neural network compared with Spiking-FlowNet architecture, with small model size and relatively fast speed; the latter is a hybrid architecture of spiking neural network (SNN) and artificial neural network (ANN), with low energy consumption and pulse structure more in line with the characteristics of sparse input of event cameras. Both of the above two neural network architectures are electrical neural network architectures, which can only be carried on electronic chips. However, with the reduction of physical size, the "bottleneck effect" of electronic chips is becoming more and more obvious. Increasing the number of electronic computing units to speed up the optical flow estimation method is greatly limited. Optimizing the neural network architecture can only slightly increase the speed, with high cost, great difficulty and small increment.

[0006] Electronic neural networks are limited in computing speed due to the process limit of nm size of their core chips. The introduction of optical computing can increase the speed of operation. Optical computing has the ability of low power consumption, light speed processing and parallel processing. Therefore, by utilizing the characteristics of optical computing, a photonic deconvolution acceleration system is designed to replace the electrical deconvolution operation part, which can greatly reduce the time of optical flow estimation and better protect the safety of users, and has a profound significance in the field of vehicle driving warning.

[0007] There are mainly the following ways of optical computing. The first way is to use a spatial light modulator and a medium material (mask plate) to construct a diffractive optical neural network (DONN), which realizes all-optical machine learning, completes the classification of handwritten digit dataset and Fashion dataset, and can realize large-scale neural networks and complete relatively complex tasks. However, the actual structure of this way is too large, which is not conducive to integration, and the parameters of the medium material cannot be adjusted once the training is completed, which is true for both convolution operation part and full connection part, making its application in many fields greatly limited.

[0008] The second way is a full-optical neural network based on phase change material and cascaded micro-ring structure, which can directly input optical signals without photoelectric conversion processing. However, due to the characteristics of the phase change material, it cannot realize negative refractive index under the existing material system, the weight factor can only be positive, the fitting effect is poor, and the phase material has a short life and a short use cycle, and the response speed is not fast, so it can only complete simple tasks for the time being.

[0009] The third is a photonic neural network based on Mach-Zehnder interferometer (MZI). The theory can be traced back to the triangular decomposition algorithm proposed by Reck et al. in 1994, which proves that MZI can realize any rotation unitary matrix. According to the algorithm of singular value decomposition, any matrix can be represented as the product of a unitary matrix, a diagonal matrix and the transpose matrix of another unitary matrix, so that any matrix can be represented by using MZI, phase shifter and other devices. Compared with the diffraction photonic neural network and the photonic neural network built by phase change material, the optical computing method based on MZI has the characteristics of high integration and strong reconfigurability, and can quickly and accurately complete the matrix operation. However, at present, this optical computing method is mainly used to complete the all-optical convolution operation. At present, there is no independent system based on photonic that can directly complete the deconvolution operation. SUMMARY

[0010] The present application comprehensively considers the advantages and disadvantages of the prior art. Firstly, a photonic deconvolution acceleration system is proposed, which uses photonic devices such as laser, light source modulator, optical beam splitter, MZI, phase shifter and optical balance detector to complete the deconvolution operation. The deconvolution operation layer in the Spiking-FlowNet neural network architecture can be replaced, so that the calculation speed can be greatly improved, and the problem of slow deconvolution operation based on electricity can be overcome. At the same time, the array built by MZI has the characteristics of strong reconfigurability, and the phase shifter can be used to change the transmission matrix of the MZI array, so that it can complete the deconvolution operation process of different channels. Secondly, based on this structure, an application method of photonic deconvolution acceleration system for optical flow estimation is proposed, and an accelerated hybrid optoelectronic Spiking-FlowNet neural network architecture is designed. The hybrid optoelectronic neural network architecture fully utilizes the characteristics of parallelism, low power consumption and light speed calculation of optical computing, and can complete the optical flow estimation more quickly and energy-efficiently while preserving the original accuracy of the Spiking-FlowNet neural network architecture.

[0011] In order to achieve the above purpose, the present application provides the following technical scheme:

[0012] The present application provides a photonic deconvolution acceleration system, which is composed of a laser, a beam splitter array, a modulator array, a first MZI array, an attenuator module, a second MZI array, a phase shifter module, an optical balance detector module and a phase reference module, and completes the deconvolution operation. The beam splitter array is composed of a plurality of optical beam splitters. The first MZI array, the attenuator module and the second MZI array are connected in series, and represent the transpose V T , the diagonal matrix ∑ and the other positive definite matrix U in the singular value decomposition algorithm, respectively.

[0013] The deconvolution operation process is as follows:

[0014] First, an optical carrier is generated by a laser and input to a beam splitter array;

[0015] Then, the beam splitter array divides one optical signal into multiple paths and then inputs them into a modulator array;

[0016] Next, the modulator array simultaneously modulates multiple different signals according to a rule onto the optical carrier to become the input of the first MZI array path;

[0017] Then, the signal passes through the first MZI upper triangular structure to complete the matrix multiplication operation V T •I of the signal I and the positive definite matrix transpose V T •I, and then passes through the attenuator to complete the operation ∑•V T •I;

[0018] After that, the second MZI array is used to complete the operation, and the phase shifter module is used to deflect the operation result so that it can correctly output the intensity value of the light source. The output result is U•∑•V T •I;

[0019] Finally, the number of MZIs connected on each path and the waveguide line are supplemented to be the same as the last path. The output phase of the light source passing through the phase reference module is taken as the reference, and the output phase is specified as 0° phase. The balanced detector is used to detect the output phase of each path, and the results of each path are compared with the reference phase. If they are both 0°, it is a positive value, and if they are 180°, it is a negative value. The output is corrected.

[0020] Further, the frequency of the optical carrier is 193.1 THz, and the wavelength is 1550 nm.

[0021] Further, the beam splitter array divides one optical signal into 16 paths.

[0022] Further, the pre-processing algorithm of the signal is as follows:

[0023] The original image is first subjected to a zero-filling column expansion operation, that is, a blank pixel is added between every two pixel points of the original image, and two circles of blank pixel points are added around it;

[0024] A size frame is taken from the first row and the first column of the top left corner. The taken data is expanded by row into a strip of data, becoming the first signal;

[0025] The other multiple signals are taken with a size frame step of 1, and the remaining multiple different signals are taken in turn.

[0026] Further, take an image of 16x16 resolution as the original image, the deconvolution kernel size is 4x4, add a blank pixel in the middle of every two pixel points of the original image, and add two circles of blank pixel points around, to become an image data of 35x35 resolution; the beam splitter array divides one light signal into 16 paths, takes a size frame of 32x32, takes from the first row and first column in the top left corner, and the taken 32x32 size data is expanded into 1x1024 strip data by row, to become the first path signal; the other 15 path data continues to take values by sliding with a size frame step of 1, and the remaining 15 different signals are sequentially taken.

[0027] On the other hand, the application also provides an application of the above-mentioned photonic deconvolution acceleration system for optical flow estimation, wherein the matching photonic deconvolution acceleration system adopts an optimized Spiking-FlowNet optical flow estimation neural network architecture, which is a hybrid architecture of SNN and ANN, and includes an encoder layer, a residual layer and a decoder layer; the encoder layer is composed of four layers of spiking neural network neuron layers, input data is gradually input with time signals, convolution operations are completed by spiking neurons, dimension reduction functions are realized, all outputs are collected into corresponding output accumulators, and then are introduced into the residual layer; the residual layer is used to realize skip connection of signals and output to the decoder layer; the decoder accumulates the output of the previous layer and the output of the corresponding convolution layer in the encoder layer together as input, and completes the process of generating an intermediate predicted flow, and the process is sequentially repeated until the final optical flow estimation output is generated.

[0028] Further, the encoder layer in the neural network architecture is constructed as SNN, and the residual layer and the decoder layer maintain a neural network structure.

[0029] Compared with the prior art, the application has the following beneficial effects:

[0030] The application innovatively designs a photonic deconvolution acceleration system, successfully builds the photonic deconvolution acceleration system by using optoelectronic components such as optical beam splitters and optical modulators. The system only needs one laser and multiple electro-optical modulators to complete 16 different deconvolution operations on the whole image data. In the case of ensuring accuracy and saving cost as much as possible, the system utilizes the characteristics of optical parallel computing and light speed, greatly speeds up the deconvolution operation, solves the problem of slow deconvolution calculation speed based on traditional electronic operation. Moreover, the system introduces the idea of using the phase of light to represent positive and negative, and solves the problem of not being able to represent negative numbers in the light source and output.

[0031] The photonic deconvolution acceleration system provided by this invention is designed for optical flow estimation applications. It optimizes and improves the Spiking-FlowNet spiking neural network architecture, leveraging the multi-layered deconvolution characteristics and the sparse optical flow estimation signal. This neural network architecture is combined with a photonic deconvolution acceleration system. This hybrid optoelectronic architecture, utilizing the high speed of photonic computation, significantly accelerates the computation process and completes optical flow estimation much faster. Therefore, the hybrid optoelectronic neural network architecture incorporating the photonic deconvolution acceleration system achieves comparable accuracy to the original optical flow estimation but at a significantly faster speed. Attached Figure Description

[0032] To more clearly illustrate the technical solutions in the embodiments of this application or the prior art, the drawings used in the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments recorded in this invention. For those skilled in the art, other drawings can be obtained based on these drawings.

[0033] Figure 1 The flowchart of the photon deconvolution acceleration system provided by the present invention is shown.

[0034] Figure 2 This is a schematic diagram of the photon deconvolution input data preprocessing process provided by the present invention.

[0035] Figure 3 This is a schematic diagram of the optical flow estimation neural network provided by the present invention.

[0036] Figure 4 The data and results provided for this invention are illustrated below. In the figure, (a) is the original image, and (b) are 16 generated images.

[0037] Figure 5 The graph shows the variation of the loss function with algebra provided by this invention.

[0038] Figure 6 The optical flow estimation prediction diagram is provided for the four deconvolution layers in this invention. Detailed Implementation

[0039] To better understand this technical solution, the technical solution of the present invention will be clearly and completely described below with reference to the accompanying drawings of the embodiments of the present invention. Obviously, the described examples are only a part of the embodiments of the present invention, and not all of them. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art based on this application are within the scope of protection of the present invention.

[0040] This invention proposes a photonic deconvolution acceleration system, such as... Figure 1As shown, the system consists of a laser, a beam splitter array, a modulator array, a first MZI array, an attenuator module, a second MZI array, a phase shifter module, an optical balance detector module, and a phase reference module, and performs deconvolution operations. The beam splitter array is composed of several optical beam splitters. The first MZI array, the attenuator module, and the second MZI array are connected in series, representing the transpose V of the positive definite matrix in the singular value decomposition algorithm. T The diagonal matrix ∑ and another positive definite matrix U; the deconvolution operation process is as follows:

[0041] First, an optical carrier is generated by a laser and input to a beam splitter array. Then, the beam splitter array splits the single optical signal into multiple paths, which are subsequently input to a modulator array. Next, the modulator array simultaneously modulates these multiple different signals onto the optical carrier according to a set rule, becoming the input to multiple first MZI array paths. Finally, the signals pass through the upper triangular structure of the first MZI array, completing the transpose V of signal I and the positive definite matrix. T Matrix multiplication operation V T •I, then the calculation ∑•V is completed through an attenuator. T •I; Then, the calculation is completed through the second MZI array, and the calculation result is deflected by the phase shifter module so that it can correctly output the intensity value of the light source. The output result is U•∑•V T •I; Finally, when the number of MZI connected to each path and the waveguide are replenished to be the same as the last path, the output phase of the light source after passing through the phase reference module is taken as the standard, and the output phase is defined as 0° phase. The output phase of each path is detected by a balanced detector, and the result of each path is compared with the reference phase. If they are both 0°, it is a positive value; if they are 180°, it is a negative value. The output is then corrected.

[0042] For example, a laser generates an optical carrier wave with a frequency of 193.1 THz and a wavelength of 1550 nm, which is input into the system. Considering that the energy of one light source will be evenly distributed across 16 channels, the signal strength needs to be quadrupled from its original value. An optical beamsplitter array, composed of several beamsplitters, splits one optical signal into 16 channels, which are then input into a modulator array. The modulator array simultaneously modulates these 16 different signals onto the optical carrier wave according to existing rules, becoming the input to the 16 MZI array paths. To satisfy the requirement of representing arbitrary matrices, the MZI array is designed as a series connection of two upper triangular structures and one attenuator structure, representing the positive definite matrix U, the diagonal matrix ∑, and the transpose V of another positive definite matrix in the Singular Value Decomposition (SVD) algorithm, respectively. T The signal passes completely through the first MZI upper triangular structure, which can be considered as completing the transpose of the signal I and the positive definite matrix V. T The matrix multiplication operation is VT •I, through the attenuator part for ∑•V T •I, then through the second MZI array to complete the operation. The phase shifter part is to make a deflection to the calculation result after the MZI array part, so that it can correctly output the intensity value of the light source, and the output result is U•∑•V T •I. The last optical balance detector part and the phase reference module are the key to enable the entire deconvolution system to output negative values. The number of MZIs connected on each path is supplemented to be the same as that of the last path (the parameters of the supplemented MZIs are all set to 0), and the output phase of the light source through the phase reference module is used as the reference, which is defined as 0° phase. The output phases of each path are detected by the balance detector, and the results of each path are compared with the reference phase. If they are both 0°, it is a positive value, and if they are both 180°, it is a negative value. The output is corrected, so that the system can correctly output negative results.

[0043] Compared with the electronic-based deconvolution operation, the core of the innovation of the application is the signal preprocessing algorithm and the designed photonic deconvolution calculation structure based on the preprocessing algorithm, and the idea of using phase to represent the positive and negative values of input and output is introduced, so that the function of the system is more comprehensive.

[0044] Taking the value process of the first path signal as an example, the conversion process of the original image data of the light source coding part and the light source input data is shown in Figure 2 . A 16x16 resolution image is taken as the original image. The image needs to be zero-padded and column-extended, that is, a blank pixel is added between every two pixel points of the original image. Considering that the deconvolution kernel is 4x4 in size, two circles of blank pixel points should also be added around it to become a 35x35 resolution image data. Next, a 32x32 size frame is taken, starting from the first row and the first column in the top left corner. The taken 32x32 size data is expanded into 1x1024 strip data by row, becoming the first path signal. The other 15 path data is taken in turn by sliding with a step size of 1, and 15 different signals are taken in turn. Using the parallel computing characteristics of the MZI array, the sixteen signals are modulated to the input channels of the corresponding MZI array at the same time, so that 16 different deconvolution operations can be successfully completed.

[0045] On the other hand, the application also provides an application of the above-mentioned photonic deconvolution acceleration system for optical flow estimation, wherein the matching photonic deconvolution acceleration system adopts an optimized Spiking-FlowNet optical flow estimation neural network architecture, as shown in Figure 2As shown, the architecture includes an encoder layer, a residual layer, and a decoder layer. The encoder layer is composed of four layers of spiking neural network neuron layers, input data is gradually input with time signals, spiking neurons are fired or temporarily stored according to rules, and convolution operations are completed to realize dimension reduction functions. Then, after all continuous events pass, all outputs are collected into corresponding output accumulators, and then are imported into the residual layer. The residual layer is used to realize signal skip connection and output to the decoder layer, which can well solve the network degradation problem and make the information propagation more smooth. The decoder layer is like the reverse operation of the encoder layer, which adds the output of the last layer (the residual layer or the last deconvolution layer) and the output of the corresponding convolution layer in the encoder layer together as input, and completes the process of generating intermediate prediction flow. By analogy, the last optical flow estimation output is generated.

[0046] The architecture is a hybrid architecture of SNN (Spiking Neural Network) and ANN (Artificial Neural Network), which can adapt to different levels of SNN and ANN, can maintain the advantages of SNN in sparse event data processing, and can maintain the advantages of ANN in performance. Using deep SNN, with the increase of network depth, the pulse disappearance phenomenon often occurs, which may lead to the performance degradation of deep SNN. In addition, when estimating the accurate pixel network output, high numerical precision is required, that is, a regression task is required. Therefore, the architecture is only constructed as SNN in the encoder block, and the residual layer and the decoder layer maintain a neural network structure. When the photon deconvolution acceleration system is fused into the Spiking-FlowNet optical flow estimation neural network architecture, the deconvolution kernel trained in the optical flow estimation needs to be corresponded to the photon deconvolution structure. Because the photon deconvolution structure has 16 channels, 16 deconvolution operations can be performed at the same time, so 16 4x4 deconvolution kernels can be imported at the same time, and the corresponding parameter value can be solved by using the parameter corresponding evaluation algorithm in MZI.

[0047] According to the existing electro-optic modulator device, we can know that the frequency of the electro-optic modulator is 10GHz, in order to output accurately, a 50GHz balanced photodetector is used for sampling, that is, one symbol can be represented by 5 sampling points. In the process of deconvolution operation of a picture with a size of 16x16x128 to 32x32x16, the zero padding and column expansion processes of the two methods are the same and are not considered in time comparison. Using the deconvolution system for calculation, the selected signal length is 32x32=1024, and the channel number is 128, so 128 times of operation are required. The system can complete the calculation of 16 deconvolution kernels at a time, and the time of optical calculation in the photonic chip is ignored due to the order of magnitude difference. The time required for completing the calculation is 128x1024÷1010≈1.31x10 -5seconds. Using an electronic computer to calculate the same amount of calculation, a total of 128 x 16 x 32 x 32 x (16 times multiplication + 15 times addition) = 65011712 times, taking the computer 1 second to calculate 5 billion times, the time is 1.3 x 10 -2 seconds, thus it can be seen that using optical calculation can greatly speed up the deconvolution operation, and in an ideal case, the speed is increased by about 1000 times, greatly saving the calculation time.

[0048] According to the principle of the scheme of the present application, the scheme is applied to deconvolution calculation, and the verification method is to observe the calculation result by computer simulation. The practicability of the scheme is verified from two examples, which are respectively completing deconvolution calculation by using a photonic deconvolution acceleration system and application of the photonic deconvolution acceleration system to optical flow estimation.

[0049] Application Example 1: Completing Deconvolution Calculation by Using a Photonic Deconvolution Acceleration System

[0050] In the INTERCONNECT simulation, a 16 x 16 deconvolution structure as shown in Figure 1 is designed. Any one of the deconvolution kernels in the transmission matrix represented by the MZI array part is taken from the training result in the optical flow estimation neural network, and because of the large number of channels, the value of each deconvolution kernel is much smaller than 1, which conforms to the characteristics of MZI and other passive optical devices that do not increase additional energy, and has universal significance. The transmission matrix formed by the 16 convolution kernels is shown in Table 1.

[0051] Table 1 Transmission matrix of the photonic deconvolution acceleration system (x 10 -2 )

[0052] -1.65 2.77 -2.37 -3.09 4.77 -4.85 1.65 0.41 1.00 3.32 -2.27 -2.97 -1.93 -4.11 1.91 4.62 2.19 0.15 -1.21 6.03 2.50 -1.30 4.72, 1.23 2.10 -1.84 -5.02 1.73 0.30 4.25 4.25 1.02 0.93 -0.66 1.04 9.88 -1.69 0.03 1.14 -3.15 -5.14 -3.45 0.87 -0.35 -1.15 1.25 3.18 9.90 6.81 -9.08 1.60 -2.58 3.29 -4.66 -2.92 -1.03 -0.88 2.55 -2.73 0.52 2.81 -1.43 -0.26 4.78 2.18 4.07 -5.56 6.78 3.90 1.67 -0.44 -6.12 6.32 -0.82 1.78 -2.51 4.50 -3.38 1.43 4.93 0.71 -4.96 2.36 1.10 -1.21 -4.81 3.33 3.79 2.72 4.66 1.55 3.64 -5.31 4.93 -3.50 -2.44 0.37 1.91 2.14 -3.88 10.96 1.26 -1.40 4.61 3.98 -6.78 -8.19 0.34 10.61 0.28 -5.64 0.91 0.60 10.39 -0.51 -0.67 -4.00 1.32 -2.61 -1.23 -1.41 -5.26 2.85 2.34 -0.94 -0.31 3.86 -1.58 2.29 -6.27 0.58 -1.65 -0.80 2.13 -3.92 2.34 -6.64 -5.79 -2.69 -0.83 -3.50 3.68 -2.72 2.16 -10.79 -1.22 8.16 0.51 -7.65 2.40 -1.13 3.62 -1.80 -6.18 2.64 2.54 0.68 4.61 6.72 -1.85 -4.44 -2.23 6.31 6.34 1.85 -1.14 -0.38 8.17 0.40 1.39 4.06 1.10 5.95 3.53 -1.71 -2.93 4.59 1.69 -2.80 -1.69 3.26 -3.18 -1.53 -2.62 1.22 -0.22 -0.85 3.04 -0.95 0.19 2.20 0.48 2.60 3.68 0.77 0.06 5.23 -4.31 -3.17 -2.25 9.28 -5.42 -0.92 0.36 2.06 -0.58 0.49 4.32 -4.25 -0.58 -4.48 -2.01 0.34 -3.92 -0.22 -5.05 7.82 -5.51 -0.36 1.16 1.15 1.96 1.06 11.64 0.87 -2.11 4.74 4.00 -0.34 -7.04 0.36 0.00 3.06 -0.65 2.17 4.78 -3.39 3.58 4.37 5.47 -2.31 -2.57 -4.08 2.84 -2.32 0.73 0.29 5.08 -0.95 0.40 2.45 -0.01 -1.42 2.97 -0.26 4.31

[0053] The 16 x 16 original image as input is shown in Figure 4 (a). After the original image data is pre-encoded according to the method described in the scheme, the signal is modulated by using an electro-optical modulator and introduced into the deconvolution system. After the original image is deconvolved with 16 different deconvolution kernels, 16 outputs are generated, and the 16 outputs are integrated and plotted in Figure 4 (b) in the order from left to right and from top to bottom. From the above 16 generated images, it can be seen that the images after deconvolution operation still have most of the characteristics of the original image, and the output values are the same as the deconvolution calculation structure based on electronics, which proves the accuracy of the photonic deconvolution acceleration system.

[0054] Application Example 2: Application of the Photonic Deconvolution Acceleration System to Optical Flow Estimation

[0055] The Spiking-FlowNet neural network architecture is trained using the MVSEC dataset commonly used in the field of optical flow estimation. The dataset contains 28583 training set images and 2206 test set images, and a total of 100 generations are trained. The average loss function Mean-loss parameter of optical flow estimation is the sum of the multi-scale intensity loss of optical flow estimation and the smoothing loss multiplied by a fixed coefficient, and its variation with the generation number is shown in Figure 5 The part based on electronic deconvolution operation is replaced with the photon deconvolution acceleration system of the application, and using these deconvolution calculations, the optical flow at the next time can be estimated by the image data of the previous layer. Because there are 4 deconvolution calculations in the architecture, there are 4 predicted optical flow images, as shown in Figure 6 Using all the test set images, the value of the error evaluation index Mean-EPE of the hybrid architecture is 1.0667, which is not much different from the original architecture Mean-EPE value 1.018 without replacing the original architecture with the photon deconvolution system. Using the photon deconvolution system for acceleration, the original deconvolution calculation in the architecture can be accelerated by 1000 times, greatly saving the optical flow estimation time and improving the efficiency.

[0056] It should be noted that in this text, relational terms such as first and second are only used to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any such actual relationship or order between the entities or operations. Moreover, the terms "include", "contain" or any other variant thereof are intended to cover non-exclusive inclusion, so that the process, method, article or device including a series of elements not only includes those elements, but also includes other elements not explicitly listed or inherent to such process, method, article or device. Without more limitations, the element defined by the statement "including a…" does not exclude the presence of other identical elements in the process, method, article or device including the element.

[0057] Each embodiment in the specification is described in a related manner, and the same or similar parts between each embodiment can be referred to each other. Each embodiment focuses on the difference from other embodiments. In particular, for the device embodiment, the electronic device embodiment, the computer readable storage medium embodiment and the computer program product embodiment, because they are basically similar to the method embodiment, the description is relatively simple, and the relevant parts can be referred to the part of the method embodiment.

[0058] The above merely describes the preferred embodiments of the present application, but is not used to limit the protection scope of the present application. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present application shall be included in the protection scope of the present application.

Claims

1. A photonic deconvolution acceleration system, characterized in that, The system consists of a laser, a beam splitter array, a modulator array, a first MZI array, an attenuator module, a second MZI array, a phase shifter module, an optical balance detector module, and a phase reference module, and performs deconvolution operations. The beam splitter array is composed of several optical beam splitters. The first MZI array, the attenuator module, and the second MZI array are connected in series, representing the transpose V of the positive definite matrix in the singular value decomposition algorithm. T , a diagonal matrix ∑ and another positive definite matrix U; The deconvolution operation process is as follows: First, an optical carrier is generated by a laser and input into a beam splitter array; Then, the beam splitter array splits one optical signal into multiple paths, which are then input into the modulator array; Next, the modulator array simultaneously modulates multiple different signals onto the optical carrier according to a rule, becoming the input of multiple first MZI array paths; Then, the signal passes through the first MZI upper triangular structure, completing the transpose V of the signal I and the positive definite matrix. T Matrix multiplication operation V T •I, then the calculation ∑•V is completed through an attenuator. T •I; Next, the calculation is completed through the second MZI array, and then the calculation result is deflected by the phase shifter module to correctly output the intensity value of the light source. The output result is U•∑•V T •I; Finally, when the number of MZIs connected to each path and the waveguides are increased to be the same as the last path, the output phase of the light source after passing through the phase reference module is taken as the standard, and the output phase is defined as 0° phase. The output phase of each path is detected by a balanced detector, and the result of each path is compared with the reference phase. If they are both 0°, it is a positive value; if they are 180°, it is a negative value, and the output is corrected.

2. The photonic deconvolution acceleration system according to claim 1, characterized in that, The optical carrier frequency is 193.1 THz and the wavelength is 1550 nm.

3. The photonic deconvolution acceleration system according to claim 1, characterized in that, The beam splitter array splits one optical signal into 16 channels.

4. The photonic deconvolution acceleration system according to claim 1, characterized in that, During the deconvolution operation, the beam splitter array splits a single optical signal into multiple paths using a signal preprocessing algorithm. The signal preprocessing algorithm is as follows: The original image is first padded with zeros and expanded, that is, a blank pixel is added between every two pixels in the original image, and two circles of blank pixels are added around it. Take a size frame, starting from the first row and first column of the top left corner. Expand the extracted data into bar data by row, which becomes the first signal. The other multiple data streams continue to be retrieved by sliding the size box step by 1, and the remaining multiple different signals are retrieved in turn.

5. The photonic deconvolution acceleration system according to claim 4, characterized in that, A 16×16 resolution image is taken as the original image. The deconvolution kernel size is 4×4. A blank pixel is added between every two pixels in the original image, and two rings of blank pixels are added around it to become a 35×35 resolution image data. The beam splitter array divides one optical signal into 16 channels. A 32×32 size frame is taken, starting from the first row and first column of the top left corner. The extracted 32×32 size data is expanded into 1×1024 strip data by row, which becomes the first signal. The other 15 data continue to be extracted by sliding the size frame with a step size of 1, and the remaining 15 different signals are extracted in sequence.

Images