All-optical neural network device and system for speckle image classification and recovery

Abstract

The invention provides an all-optical neural network device and system for speckle image classification and recovery, and the training method comprises the steps: obtaining a training sample set which comprises an original image, a speckle image outputted by the original image through an optical fiber, and a classification label image corresponding to the original image; the training sample is input into the initial all-optical neural network, the training sample comprises a first convolution layer, a second convolution layer and a full connection layer, and the full connection layer comprises an image restoration area and an image classification area. And after optical convolution operation is carried out on the speckle image in the first convolution layer and the second convolution layer in sequence, the speckle image is transmitted to the full connection layer to combine convolution results, and a recovered image and a classified image are output. And constructing a composite loss function by using mean square errors and Pearson's correlation coefficients between the recovered image and the original image and between the classified image and the classified label image. And the loss is minimized to update the initial all-optical neural network parameters until training is completed, and a trained all-optical neural network is obtained. Noise and loss can be reduced, and calculation efficiency can be improved.

Description

Technical Field

[0001] The present invention relates to the field of optical computing technology, and in particular to an all-optical neural network device and system for speckle image classification and restoration. Background Art

[0002] As a transmission medium for optical communication, optical fiber has the characteristics of large information transmission capacity, small mass, and no interference from electromagnetic fields. It is widely used in various fields. In optical fiber communication, when image data is projected to the input end of the optical fiber, the input image is coupled into the optical fiber for propagation. After passing through the optical fiber channel, what is transmitted from the other end is a speckle image that cannot be decoded by the human eye, and the original state of the image is completely lost. Therefore, the classification of speckle images and their restoration to the original image have become hot research issues.

[0003] At present, the mainstream method is to use neural networks for speckle image classification and restoration. Convolutional networks with a large number of parameters can well learn speckle image data, extract features, and complete tasks. However, this method requires capturing and converting optical signals into electrical signals for processing, which affects the computational efficiency and has the defects of greater noise, greater delay, and greater loss. Summary of the invention

[0004] In view of this, an embodiment of the present invention provides an all-optical neural network device and system for speckle image classification and restoration, so as to eliminate or improve the defects of low computational efficiency, high noise, high delay and high loss in the prior art.

[0005] A first aspect of the present invention provides an all-optical neural network training method for speckle image classification and restoration, the method comprising the following steps:

[0006] Acquire a training sample set, wherein a group of training samples in the training sample set includes an original image, a speckle image of the original image output through an optical fiber, and a classification label image corresponding to the original image;

[0007] Inputting the training sample into an initial all-optical neural network; the initial all-optical neural network comprises a first convolutional layer, a second convolutional layer and a fully connected layer; the fully connected layer comprises an image restoration area and an image classification area;

[0008] After the speckle image is subjected to optical convolution operations in the first convolution layer and the second convolution layer in sequence, the speckle image is transmitted to the fully connected layer to combine the convolution results of the second convolution layer, reconstruct image features in the image restoration area to output a restored image, and identify image category information in the image classification area to output a classified image;

[0009] A composite loss function is constructed by using the mean square error and the Pearson correlation coefficient between the restored image and the original image and between the classified image and the classified label image; the composite loss function is minimized to update the parameters of the initial all-optical neural network until the training is completed, thereby obtaining a trained all-optical neural network.

[0010] In some embodiments of the present invention, the calculation formula for performing optical convolution operation on the speckle image in the first convolution layer and the second convolution layer is:

[0011]

[0012] Among them, U i is the input plane of the speckle image, U o is the output plane of the speckle image; F{V(x,y)·R(x,y)·G(x,y) is the convolution kernel; V(x,y) is the vortex phase, which is used to extract image edge features; R(x,y) is the random phase, which is used to learn the mapping rules in training; G(x,y) is the grating phase, which is used to move the output image on the output plane.

[0013] In some embodiments of the present invention, the calculation formula of the mean square error is:

[0014]

[0015] Wherein, O(i,j) is the restored image and the classified image, T(i,j) is the original image and the classified label image;

[0016] The calculation formula of the Pearson correlation coefficient is:

[0017]

[0018] Among them, O(i,j) is the restored image and the classified image, T(i,j) is the original image and the classified label image, is the average value of the restored image and the classified image, is the average value of the original image and the classification label image.

[0019] In some embodiments of the present invention, the calculation formula of the composite loss function is:

[0020] loss=MSE+(1-PCC 2 );

[0021] Wherein, MSE represents the mean square error, and PCC represents the Pearson correlation coefficient.

[0022] In some embodiments of the present invention, the method of minimizing the composite loss function to update the parameters of the initial all-optical neural network includes:

[0023] The loss value obtained by the composite loss function in each training is back-propagated, Fourier transformed and then gradient solved before reaching the second convolutional layer, and then propagated to the first convolutional layer for gradient solution; all parameters in this training are updated using the optimization algorithm; multiple rounds of training are performed until the preset training round is reached or until the loss value converges.

[0024] A second aspect of the present invention provides an all-optical neural network device for speckle image classification and restoration, wherein the parameters of the all-optical neural network device are obtained by the all-optical neural network training method for speckle image classification and restoration as described in any one of the above, and the all-optical neural network device comprises:

[0025] A first optical modulation element, configured to receive an optical signal carrying a speckle image and perform an optical convolution operation of a first convolution layer; the first convolution layer includes a plurality of parallel convolution kernels;

[0026] A second optical modulation element is used to receive the optical signal modulated and output by the first optical modulation element, and perform an optical convolution operation of a second convolution layer; the second convolution layer includes a plurality of parallel convolution kernels;

[0027] The third optical modulation element includes an image restoration area and an image classification area, and is used to receive the light signal modulated and output by the second optical modulation element, combine the convolution results transmitted by the second optical modulation element, output the restored image in the image restoration area, and output the classified image in the image classification area.

[0028] In some embodiments of the present invention, the first optical modulation element and the second optical modulation element use a single lens to implement an optical convolution operation, and the function expression of the lens is:

[0029]

[0030] in, represents the secondary lens phase, represents the coefficient of phase distribution, f represents the focal length parameter of the lens, and P(x, y) represents the convolution kernel in the Fourier domain.

[0031] In some embodiments of the present invention, the first optical modulation element, the second optical modulation element and the third optical modulation element are masks printed by a spatial light modulator or a photolithography machine.

[0032] In some embodiments of the present invention, the all-optical neural network device further comprises: a beam expander, a first beam splitter, a second beam splitter, a third beam splitter, a first polarizer and a second polarizer;

[0033] The all-optical neural network device receives an optical signal carrying a speckle image output by an optical fiber, and the optical signal passes through a beam expander, the first beam splitter, the first optical modulation element, the first beam splitter, the first polarizer, the second beam splitter, the second optical modulation element, the second beam splitter, the second polarizer, the third beam splitter, the third optical modulation element, and the third beam splitter in sequence.

[0034] A third aspect of the present invention provides a speckle image classification and restoration system based on an all-optical neural network, the system comprising:

[0035] An all-optical neural network device as described in any of the above items, used to receive a speckle image output by a target image through an optical fiber, and output a restored image and a classified image corresponding to the target image in the form of optical signals;

[0036] An image sensor is used to convert the optical signal output by the all-optical neural network device into an electrical signal and display a restored image and a classified image.

[0037] The beneficial effects of the present invention are at least:

[0038] The present invention provides an all-optical neural network device and system for speckle image classification and restoration, and the training method thereof is as follows: a training sample set is obtained, including an original image, a speckle image outputted by the original image through an optical fiber, and a classification label image corresponding to the original image. The training sample is inputted into an initial all-optical neural network, including a first convolution layer, a second convolution layer, and a fully connected layer, wherein the fully connected layer includes an image restoration region and an image classification region. After the speckle image is subjected to optical convolution operations in the first convolution layer and the second convolution layer in turn, it is transferred to the fully connected layer to combine the convolution results, and a restored image and a classified image are outputted. A composite loss function is constructed using the mean square error and Pearson correlation coefficient between the restored image and the original image and between the classified image and the classified label image. The loss is minimized to update the initial all-optical neural network parameters until the training is completed, and the trained all-optical neural network is obtained. The all-optical neural network for image restoration and classification obtained by the training method provided by the present invention can avoid the loss and noise of photoelectric conversion and improve the calculation rate. The structure of the all-optical neural network realizes multi-task parallelism through shared representation and joint optimization, improves the efficiency and performance of the model, and enhances the generalization ability and robustness of the model.

[0039] Additional advantages, purposes, and features of the present invention will be described in part in the following description, and will become apparent to those skilled in the art after studying the following, or may be learned from the practice of the present invention. The purposes and other advantages of the present invention may be achieved and obtained by the structures specifically indicated in the specification and the accompanying drawings.

[0040] Those skilled in the art will appreciate that the objectives and advantages that can be achieved with the present invention are not limited to the above specific description, and the above and other objectives that can be achieved by the present invention will be more clearly understood from the following detailed description. BRIEF DESCRIPTION OF THE DRAWINGS

[0041] The drawings described herein are used to provide a further understanding of the present invention, constitute a part of this application, and do not constitute a limitation of the present invention. The components in the drawings are not drawn to scale, but are only for illustrating the principles of the present invention. In order to facilitate the illustration and description of some parts of the present invention, the corresponding parts in the drawings may be enlarged, that is, they may become larger relative to other components in the exemplary device actually manufactured according to the present invention. In the drawings:

[0042] Figure 1 The flowchart of the all-optical neural network training method for speckle image classification and restoration in one embodiment of the present invention.

[0043] Figure 2 FIG. 4 is a structural diagram of an optical fiber communication system in another embodiment of the present invention.

[0044] Figure 3 2 is a structural diagram of an all-optical neural network in another embodiment of the present invention.

[0045] Figure 4 This is a flow chart of constructing an all-optical neural network system for speckle image classification and restoration in another embodiment of the present invention.

[0046] Figure 5 This is a phase mask result diagram in another embodiment of the present invention.

[0047] Figure 6 This is a structural diagram of a fiber transmission data and speckle image classification and recovery system in another embodiment of the present invention.

[0048] Figure 7 This is a diagram showing the classification and restoration effects of speckle images of optical fiber transmission data in another embodiment of the present invention. DETAILED DESCRIPTION

[0049] In order to make the purpose, technical solution and advantages of the present invention more clearly understood, the present invention is further described in detail below in conjunction with the embodiments and the accompanying drawings. Here, the illustrative embodiments of the present invention and their descriptions are used to explain the present invention, but are not intended to limit the present invention.

[0050] It should also be noted that, in order to avoid obscuring the present invention due to unnecessary details, only structures and / or processing steps closely related to the solutions according to the present invention are shown in the accompanying drawings, while other details that are not closely related to the present invention are omitted.

[0051] It should be emphasized that the term “include / comprises” when used herein refers to the presence of features, elements, steps or components, but does not exclude the presence or addition of one or more other features, elements, steps or components.

[0052] It should also be noted that, unless otherwise specified, the term “connection” herein may refer not only to a direct connection but also to an indirect connection involving an intermediate.

[0053] Hereinafter, embodiments of the present invention will be described with reference to the accompanying drawings. In the accompanying drawings, the same reference numerals represent the same or similar components, or the same or similar steps.

[0054] Optical Neural Networks (ONNs) are a type of neural network architecture that uses optical components for information processing and computing. Unlike traditional electronic neural networks, ONNs perform computing tasks through the properties of light waves (such as interference, diffraction, and beam modulation), and are particularly suitable for processing large-scale, high-speed data.

[0055] All-optical neural networks usually use optical elements such as lenses, mirrors, gratings, phase modulators, etc. to design convolutional layers and activation functions. The input image or data is converted into light signals optically, and the output is captured and decoded by light detectors. Feature extraction and other neural network operations are achieved by designing different optical phase patterns (such as vortex phase, random phase, etc.).

[0056] When image data is projected onto the input end of an optical fiber in an optical fiber communication system, the input image is coupled into the optical fiber for propagation, and what is transmitted from the other end through the optical fiber channel is a speckle image that cannot be decoded by the human eye, and the original state of the image is completely lost. The all-optical neural network device and system for speckle image classification and restoration provided by the present invention decodes the optical signal carrying image data output by the optical fiber communication system through the optical fiber and restores it to the original image, and classifies the image according to the classification label of the training sample.

[0057] An embodiment of the present invention provides an all-optical neural network device and system for speckle image classification and restoration. A first aspect of the embodiment provides an all-optical neural network training method for speckle image classification and restoration, such as Figure 1 As shown, the method includes the following steps S101 to S104:

[0058] Step S101: obtaining a training sample set, wherein a group of training samples in the training sample set includes an original image, a speckle image of the original image output through an optical fiber, and a classification label image corresponding to the original image.

[0059] Step S102: Input the training sample into the initial all-optical neural network. The initial all-optical neural network includes a first convolutional layer, a second convolutional layer and a fully connected layer. The fully connected layer includes an image restoration area and an image classification area.

[0060] Step S103: After the speckle image is subjected to optical convolution operations in the first convolution layer and the second convolution layer in sequence, it is passed to the fully connected layer to combine the convolution results of the second convolution layer, reconstruct the image features in the image restoration area to output the restored image, and identify the image category information in the image classification area to output the classified image.

[0061] Step S104: construct a composite loss function using the mean square error and Pearson correlation coefficient between the restored image and the original image and between the classified image and the classified label image. Minimize the composite loss function to update the parameters of the initial all-optical neural network until the training is completed, and obtain the trained all-optical neural network.

[0062] In some embodiments, the calculation formula for performing optical convolution operation on the speckle image in the first convolution layer and the second convolution layer is:

[0063]

[0064] Among them, U i is the input plane of the speckle image, U o is the output plane of the speckle image; F{V(x,y)·R(x,y)·G(x,y) is the convolution kernel; V(x,y) is the vortex phase, which is used to extract the edge features of the image; R(x,y) is the random phase, which is used to learn the mapping rules in training; G(x,y) is the grating phase, which is used to move the output image on the output plane.

[0065] In some embodiments, the mean square error is calculated as:

[0066]

[0067] Among them, O(i,j) is the restored image and the classified image, and T(i,j) is the original image and the classified label image.

[0068] The calculation formula of Pearson correlation coefficient is:

[0069]

[0070] Among them, O(i,j) is the restored image and the classified image, T(i,j) is the original image and the classified label image, is the average value of the restored image and the classified image, is the average of the original image and the classified label image.

[0071] That is, O(i, j) represents the output image of the all-optical neural network, and T(i, j) represents the target image.

[0072] In some embodiments, the compound loss function is calculated as:

[0073] loss=MSE+(1-PCC 2 );

[0074] Among them, MSE represents mean square error and PCC represents Pearson correlation coefficient.

[0075] In this composite loss function, the mean square error is the main error, and the Pearson correlation coefficient is used as a penalty term. The mean square error is usually used to represent the average of the squares of the differences between the predicted value and the actual value. For classification and recovery tasks, the lower the mean square error MSE value, the better. The Pearson correlation coefficient is used to measure the degree of linear correlation between two variables.

[0076] In some embodiments, a method of minimizing a composite loss function to update parameters of an initial all-optical neural network includes:

[0077] The loss value obtained by the composite loss function in each training is back-propagated, Fourier transformed and gradient solved before reaching the second convolutional layer, and then propagated to the first convolutional layer for gradient solution. All parameters in this training are updated using the optimization algorithm. Multiple rounds of training are performed until the preset training round is reached or until the loss value converges.

[0078] A second aspect of the embodiment provides an all-optical neural network device for speckle image classification and restoration, wherein parameters of the all-optical neural network device are obtained by any of the above-mentioned all-optical neural network training methods for speckle image classification and restoration, and the all-optical neural network device comprises:

[0079] The first optical modulation element is used to receive the optical signal carrying the speckle image and perform the optical convolution operation of the first convolution layer; the first convolution layer includes a plurality of parallel convolution kernels.

[0080] A second optical modulation element is used to receive the optical signal modulated and output by the first optical modulation element, and perform an optical convolution operation of a second convolution layer; the second convolution layer includes a plurality of parallel convolution kernels;

[0081] The third optical modulation element includes an image restoration area and an image classification area, and is used to receive the light signal modulated and output by the second optical modulation element, combine the convolution results transmitted by the second optical modulation element, output the restored image in the image restoration area, and output the classified image in the image classification area.

[0082] The parallel convolution kernel is specifically explained as follows: the convolution kernel used in the optical modulation element of the present invention is a collection of vortex phase, random phase, and grating phase. The vortex phase is used to extract the edge features of the image, and the random phase is used to discover and learn the image features. The characteristics of the grating phase enable a single convolution layer to achieve multiplexing, so that multiple convolution kernels on a convolution layer perform convolution operations in parallel, so the convolution layer of the optical modulation element includes multiple parallel convolution kernels.

[0083] In some embodiments of the present invention, the first optical modulation element and the second optical modulation element use a single lens to implement an optical convolution operation, and the function expression of the lens is:

[0084]

[0085] in, represents the secondary lens phase, represents the coefficient of phase distribution, f represents the focal length parameter of the lens, and P(x,y) represents the convolution kernel in the Fourier domain.

[0086] In some embodiments of the present invention, the first optical modulation element, the second optical modulation element and the third optical modulation element are spatial light modulators or masks printed by a photolithography machine.

[0087] In some embodiments of the present invention, the all-optical neural network device further comprises: a beam expander, a first beam splitter, a second beam splitter, a third beam splitter, a first polarizer, and a second polarizer;

[0088] The all-optical neural network device receives an optical signal carrying a speckle image output by an optical fiber, and the optical signal passes through a beam expander, a first beam splitter, a first optical modulation element, a first beam splitter, a first polarizer, a second beam splitter, a second optical modulation element, a second beam splitter, a second polarizer, a third beam splitter, a third optical modulation element, and a third beam splitter in sequence.

[0089] A third aspect of the embodiment provides a speckle image classification and restoration system based on an all-optical neural network, the system comprising:

[0090] An all-optical neural network device as described in any of the above items is used to receive a speckle image output by a target image through an optical fiber, and output a restored image and a classified image corresponding to the target image in the form of optical signals.

[0091] An image sensor is used to convert the optical signal output by the all-optical neural network device into an electrical signal and display the restored image and the classified image.

[0092] Another embodiment of the present invention provides an all-optical neural network and system for speckle image classification and restoration, and the implementation steps of the technical solution are as follows:

[0093] Step 1: Build an optical communication system to obtain the speckle image dataset. The structure of the optical fiber communication system is as follows: Figure 2 As shown in the figure, the light source passes through a half-wave lens, a polarizer, and a beam expander to reach the spatial light modulator, and the original image is loaded through the spatial light modulator. The light beam containing the original image information passes through the fiber coupler and the fiber device and reaches the CCD camera. At this time, the image collected by the CCD camera is the speckle image output by the optical fiber, which corresponds one-to-one to the input image of the spatial light modulator, and is saved and made into a speckle image data set.

[0094] Step 2: Build an all-optical neural network. An all-optical neural network is a network structure that uses optical principles to achieve deep learning functions. The network consists of two optical convolutional layers and one fully connected layer, such as Figure 3 As shown in Figure 2, the first two layers of the network are optical convolution layers, which are responsible for performing convolution operations on the input image to learn the features of the image.

[0095] The convolution layer of the all-optical neural network uses a single lens to perform optical convolution calculations on the original image. The calculation formula implemented by the convolution layer is:

[0096]

[0097] Among them U i is the input plane, U o is the output plane, and F{V(x,y)·R(x,y)·G(x,y) is the convolution kernel. V(x,y) is the vortex phase, which can extract image edge features. R(x,y) is the random phase, which can learn mapping rules during training. G(x,y) is the grating phase, which can move the output result image on the output plane. Therefore, the grating phase characteristics are used to achieve multiplexing in a single convolution layer.

[0098] The design of the convolution layer allows the convolution operation to be performed in parallel. The first optical convolution layer integrates 9 parallel convolution kernels. When the light beam passes through this layer, 9 different convolution results are generated, which are tiled on a 3x3 output plane. The second optical convolution layer further expands the integration level and designs 81 convolution kernels. This layer receives the 3x3 output of the first layer and performs a convolution operation, ultimately generating 81 convolution result images, which are tiled on a 9x9 output plane. These 81 convolution result images will undergo Fresnel propagation and enter the third layer, which is the fully connected layer.

[0099] The fully connected layer of the all-optical neural network combines the 81 different convolution result maps generated by the second convolution layer, and the output is the sum of the Fresnel diffraction results of all 81 feature maps for subsequent tasks. The expression of the combined convolution result of the fully connected layer is as follows:

[0100]

[0101] This expression represents the sum of all convolution kernels, (m,n) represents the coordinates of the convolution kernel feature map, and U3(u3,v3) represents the value at the position (u3,v3).

[0102] Step 3: Design an all-optical neural network for speckle image classification and restoration. The flowchart of constructing an all-optical neural network system for speckle image classification and restoration is as follows: Figure 4 shown.

[0103] Among them, the design principle of the all-optical neural network for speckle image classification and restoration is as follows:

[0104] In the all-optical neural network for speckle image classification and restoration, the first two optical convolution layers capture common features, and the fully connected layer processes specific features related to classification and restoration respectively. In this multi-task optical convolution network, the design of the input layer and the first two optical convolution layers are consistent, and they are jointly responsible for extracting common features from the input image. These features are then fed into the third layer of the network, the fully connected layer, where the network differentiates into two independent paths to process specific features of the classification and restoration tasks respectively. This design allows the model to share early feature extraction while customizing its post-processing for each task, thereby improving the efficiency and accuracy of the model in multi-task learning.

[0105] Specifically, the third fully connected layer has a total of 81 feature maps, tiled in the form of 9x9. The multi-task optical convolutional network divides the third fully connected layer into an upper half and a lower half. The upper half contains 36 feature maps, tiled in the form of 4x9, for classification tasks, focusing on identifying category information in the image. The lower half contains 45 feature maps, tiled in the form of 5x9, dedicated to reconstructing the image. This division of labor enables the network to handle classification and restoration tasks simultaneously without sacrificing the performance of either task.

[0106] Step 4: Train the all-optical neural network for speckle image classification and restoration.

[0107] In order to maintain consistency between tasks in a multi-task optical convolutional neural network, the classification task and the restoration task are uniformly transformed into the problem of finding the mapping rules between the input image and the target image.

[0108] For the classification task, a detector plane is designed, which is divided into 10 detection areas, each corresponding to a specific category. This design allows the network output to directly reflect the distribution of categories, where the detection area of ​​the target category is shown as high brightness, while the areas of other non-target categories are shown as low brightness. Such a design not only intuitively represents the category information, but also provides a clear optimization target for the network.

[0109] A composite loss function is constructed using the mean square error and Pearson correlation coefficient. The mean square error is expressed as:

[0110]

[0111] O(i,j) is the output image of the all-optical neural network, and T(i,j) is the target image. For classification and restoration tasks, the lower the MSE value, the better. The Pearson correlation coefficient can measure the linear correlation between two variables and is defined as:

[0112]

[0113] O(i,j) is the output graph of the all-optical neural network, T(i,j) is the true value, that is, the target image and classification label in the training sample set, and is the average value. The composite loss function takes into account the MSE and Pearson correlation coefficient. The calculation formula of the composite loss function is:

[0114] loss=MSE+(1-PCC 2 ).

[0115] After the output of the fully connected layer, the error calculation of the training image and the target image is performed. The error is back-propagated to the convolution layer. Since the convolution kernel is in the Fourier domain, the error also needs to be Fourier transformed to solve the gradient. Since the vortex phase and grating phase in the convolution kernel are fixed, the training parameters in the network are the random modulation phase R in the convolution layer and the fully connected layer.

[0116] Step 5: Construct a speckle image classification and restoration system for fiber optic transmission data.

[0117] The mask is made according to the parameters obtained by training in step 4. The phase mask result after training is shown in the figure below: Figure 5 As shown in , its effect can be verified through physical experiments. Figure 6As shown in the structure diagram, in step 1, the trained all-optical neural network is added to the middle part of the optical fiber communication system, and the trained phase mask is added to the all-optical neural network hardware to build a system for optical fiber transmission data and speckle image classification and recovery. Adjust the distance and position of each device to complete the speckle classification and recovery task in the optical domain. The effect is as follows Figure 7 shown.

[0118] Furthermore, the technical solution of the above embodiment is introduced by taking a specific device and specific data as an example. The specific operation steps of this embodiment include:

[0119] Step 1: Build a fiber optic communication system.

[0120] The light source of the optical fiber communication system is a 1550nm laser source, and the optical path is lens, polarizer, beam expander, spatial light modulator, polarizer, fiber coupler and optical fiber, CCD camera. Among them, the beam expander is built by a lens system. The original image loaded by the spatial light modulator comes from the handwritten digit set MNIST, 1000 images are selected for each number from 0 to 9, a total of 10,000 images. The optical fiber selected two types of multimode optical fiber and ring optical fiber for the experiment, and finally obtained two data sets of multimode optical fiber and ring optical fiber. Each data set includes 10,000 speckle images and 10,000 original images, a total of 10 categories.

[0121] Step 2: Build an all-optical neural network.

[0122] The all-optical neural network consists of two convolutional layers and one fully connected layer. The convolutional layer of the all-optical neural network uses a single lens to perform optical convolution calculations on the original image. The total lens function expression is as follows:

[0123]

[0124] The total lens function L(x,y) includes the secondary lens phase and the pupil function P(x,y) of the lens. The secondary lens phase is used to construct the optical convolution operation in the single-lens imaging system, and P(x,y) creates the convolution kernel in the Fourier domain. For the classification task, the pupil function P(x,y) of the lens is composed of the vortex phase V(x,y), the random modulation phase R(x,y) and the grating phase G(x,y).

[0125] The formula for the vortex phase is defined as:

[0126] V(x,y)=exp(ilθ),

[0127] Where i is an imaginary number, l is a parameter, and θ is the azimuth angle. l is [0, ±1, ±2, ±3, ±4].

[0128] The formula for random phase is defined as:

[0129] R(x,y)=exp[i·2π·a],

[0130] Where i is an imaginary number and a is a random number between [-0.2, 0.2]. The role of the random phase is to extract and learn features based on each pixel, so as to complete subsequent classification or recovery tasks. The random phase is a parameter for network back-propagation training and is also a key part of achieving all-optical processing.

[0131] The formula for the grating phase is defined as:

[0132] G m,n (x,y)=exp[i2π(mf a x+nf b y)],

[0133] Among them, f a and f b is the spatial frequency along the x and y axes, and its value is 0.0151 μm -1 The values ​​of m and n affect the shape of the grating phase. There are 9 types in total. This is the key to realizing parallel convolution kernels. After being processed by the grating phase convolution kernel, the image will be shifted in space, so that it can be transformed from a pattern on the input image to nine images on the output image.

[0134] Step 3: Train the multi-task all-optical neural network for speckle image classification and restoration.

[0135] When training the network, a composite loss function is used to obtain the error between the predicted image and the target image. The multi-task all-optical neural network will add up the errors to get the total error, and use the Adam optimizer for update modulation. The initial learning rate is set to 0.001. When the model convergence speed decreases, the learning rate decay strategy is adopted. Whenever the number of iterations is an integer multiple of 20, the learning rate decays to 50% of the original.

[0136] Step 4: Construct an all-optical neural network system for classification and restoration of optical fiber transmission data speckle images.

[0137] The speckle image classification and restoration system adds a beam expander, a beam splitter, and a mask printed by a spatial light modulator or a photolithography device to the optical fiber communication system as an optical modulation element of the all-optical neural network. The optical path is transmitted in the forward direction. At this time, the image input to the mask of the all-optical neural network is a speckle image. The image carried by the light beam after passing through the mask is the detector image or the restored image used for classification, which directly enters the CCD camera, and the image classification and restoration process of the entire optical domain can be realized. The CCD camera here can also be replaced by other image sensors, such as CMOS image sensors.

[0138] In summary, the present invention provides an all-optical neural network device and system for speckle image classification and restoration, and its training method is: obtain a training sample set, including an original image, a speckle image output by the original image through an optical fiber, and a classification label image corresponding to the original image. Input the training sample into the initial all-optical neural network, including a first convolutional layer, a second convolutional layer and a fully connected layer, and the fully connected layer includes an image restoration area and an image classification area. After the speckle image is subjected to optical convolution operations in the first convolutional layer and the second convolutional layer in turn, it is passed to the fully connected layer to combine the convolution results, and output a restored image and a classified image. A composite loss function is constructed using the mean square error and Pearson correlation coefficient between the restored image and the original image and between the classified image and the classified label image. Minimize the loss to update the initial all-optical neural network parameters until the training is completed, and obtain the trained all-optical neural network. The all-optical neural network for image restoration and classification trained by the training method provided by the present invention can avoid the loss and noise of photoelectric conversion and improve the calculation rate. During the training process, the mean square error and negative Pearson coefficient are combined as the loss function, which can simultaneously consider the size of the prediction error and the correlation between the predicted value and the true value, thereby improving the generalization ability of the all-optical neural network.

[0139] Corresponding to the above method, the present invention also provides an apparatus, which includes a computer device, the computer device includes a processor and a memory, the memory stores computer instructions, the processor is used to execute the computer instructions stored in the memory, and when the computer instructions are executed by the processor, the apparatus implements the steps of the method described above.

[0140] The embodiment of the present invention also provides a computer-readable storage medium on which a computer program is stored, and when the computer program is executed by a processor, the steps of the aforementioned edge computing server deployment method are implemented. The computer-readable storage medium can be a tangible storage medium, such as a random access memory (RAM), a memory, a read-only memory (ROM), an electrically programmable ROM, an electrically erasable programmable ROM, a register, a floppy disk, a hard disk, a removable storage disk, a CD-ROM, or any other form of storage medium known in the technical field.

[0141] It should be understood by those skilled in the art that the exemplary components, systems and methods described in conjunction with the embodiments disclosed herein can be implemented in hardware, software or a combination of the two. Whether it is performed in hardware or software depends on the specific application and design constraints of the technical solution. Professional and technical personnel can use different methods to implement the described functions for each specific application, but such implementation should not be considered to be beyond the scope of the present invention. When implemented in hardware, it can be, for example, an electronic circuit, an application specific integrated circuit (ASIC), appropriate firmware, a plug-in, a function card, etc. When implemented in software, the elements of the present invention are programs or code segments used to perform the required tasks. The program or code segment can be stored in a machine-readable medium, or transmitted on a transmission medium or a communication link via a data signal carried in a carrier.

[0142] It should be clear that the present invention is not limited to the specific configuration and processing described above and shown in the figures. For the sake of simplicity, a detailed description of the known method is omitted here. In the above embodiments, several specific steps are described and shown as examples. However, the method process of the present invention is not limited to the specific steps described and shown, and those skilled in the art can make various changes, modifications and additions, or change the order between the steps after understanding the spirit of the present invention.

[0143] In the present invention, features described and / or illustrated for one embodiment may be used in the same or similar manner in one or more other embodiments, and / or combined with features of other embodiments or replace features of other embodiments.

[0144] The above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. For those skilled in the art, the embodiments of the present invention may have various modifications and variations. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present invention shall be included in the protection scope of the present invention.

Claims

1. An all-optical neural network training method for speckle image classification and restoration, characterized in that: The method comprises the following steps: Acquire a training sample set, wherein a group of training samples in the training sample set includes an original image, a speckle image of the original image output through an optical fiber, and a classification label image corresponding to the original image; Inputting the training sample into an initial all-optical neural network; the initial all-optical neural network comprises a first convolutional layer, a second convolutional layer and a fully connected layer; the fully connected layer comprises an image restoration area and an image classification area; After the speckle image is subjected to optical convolution operations in the first convolution layer and the second convolution layer in sequence, the speckle image is transmitted to the fully connected layer to combine the convolution results of the second convolution layer, reconstruct image features in the image restoration area to output a restored image, and identify image category information in the image classification area to output a classified image; A composite loss function is constructed by using the mean square error and the Pearson correlation coefficient between the restored image and the original image and between the classified image and the classified label image; the composite loss function is minimized to update the parameters of the initial all-optical neural network until the training is completed, thereby obtaining a trained all-optical neural network.

2. The all-optical neural network training method for speckle image classification and restoration according to claim 1, characterized in that: The calculation formula for performing optical convolution operation on the speckle image in the first convolution layer and the second convolution layer is: Among them, U i is the input plane of the speckle image, U o is the output plane of the speckle image; F{V(x, y)·R(x, y)·G(x, y) is the convolution kernel; V(x, y) is the vortex phase, which is used to extract image edge features; R(x, y) is the random phase, which is used to learn the mapping rules in training; G(x, y) is the grating phase, which is used to move the output image on the output plane.

3. The all-optical neural network training method for speckle image classification and restoration according to claim 1, characterized in that: The calculation formula of the mean square error is: Wherein, O(i, j) is the restored image and the classified image, T(i, j) is the original image and the classified label image; The calculation formula of the Pearson correlation coefficient is: Wherein, O(i, j) is the restored image and the classified image, T(i, j) is the original image and the classified label image, is the average value of the restored image and the classified image, is the average value of the original image and the classification label image.

4. The all-optical neural network training method for speckle image classification and restoration according to claim 3, characterized in that: The calculation formula of the composite loss function is: loss=MSE+(1-PCC 2 ); Wherein, MSE represents the mean square error, and PCC represents the Pearson correlation coefficient.

5. The all-optical neural network training method for speckle image classification and restoration according to claim 1, characterized in that: The method of minimizing the composite loss function to update the parameters of the initial all-optical neural network includes: The loss value obtained by the composite loss function in each training is back-propagated, Fourier transformed and then gradient solved before reaching the second convolutional layer, and then propagated to the first convolutional layer for gradient solution; all parameters in this training are updated using the optimization algorithm; multiple rounds of training are performed until the preset training round is reached or until the loss value converges.

6. An all-optical neural network device for speckle image classification and restoration, characterized in that: The parameters of the all-optical neural network device are obtained by the all-optical neural network training method for speckle image classification and restoration according to any one of claims 1 to 5, and the all-optical neural network device comprises: A first optical modulation element, configured to receive an optical signal carrying a speckle image and perform an optical convolution operation of a first convolution layer; the first convolution layer includes a plurality of parallel convolution kernels; A second optical modulation element is used to receive the optical signal modulated and output by the first optical modulation element, and perform an optical convolution operation of a second convolution layer; the second convolution layer includes a plurality of parallel convolution kernels; The third optical modulation element includes an image restoration area and an image classification area, and is used to receive the light signal modulated and output by the second optical modulation element, combine the convolution results transmitted by the second optical modulation element, output the restored image in the image restoration area, and output the classified image in the image classification area.

7. The all-optical neural network device for speckle image classification and restoration according to claim 6, characterized in that: The first optical modulation element and the second optical modulation element use a single lens to implement an optical convolution operation, and the function expression of the lens is: in, represents the secondary lens phase, represents the coefficient of phase distribution, f represents the focal length parameter of the lens, and P(x, y) represents the convolution kernel in the Fourier domain.

8. The all-optical neural network device for speckle image classification and restoration according to claim 6, characterized in that: The first optical modulation element, the second optical modulation element and the third optical modulation element are masks printed by a spatial light modulator or a photolithography machine.

9. The all-optical neural network device for speckle image classification and restoration according to claim 6, characterized in that: The all-optical neural network device further includes: a beam expander, a first beam splitter, a second beam splitter, a third beam splitter, a first polarizer and a second polarizer; The all-optical neural network device receives an optical signal carrying a speckle image output by an optical fiber, and the optical signal passes through a beam expander, the first beam splitter, the first optical modulation element, the first beam splitter, the first polarizer, the second beam splitter, the second optical modulation element, the second beam splitter, the second polarizer, the third beam splitter, the third optical modulation element, and the third beam splitter in sequence.

10. A speckle image classification and restoration system based on an all-optical neural network, characterized in that: The system includes: The all-optical neural network device according to any one of claims 6 to 9, used to receive a speckle image output by a target image through an optical fiber, and output a restored image and a classified image corresponding to the target image in the form of optical signals; An image sensor is used to convert the optical signal output by the all-optical neural network device into an electrical signal and display a restored image and a classified image.

Images