Diffracted light neural network system and method for processing target in real time
Patent Information
- Application Number
- CN202411494068.X
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-10-24
- Publication Date
- 2025-03-07
Smart Images
Figure CN119578480A_ABST
Abstract
Description
Technical Field
[0001] The present application relates to the field of optical neural networks, and in particular to a diffraction optical neural network system for real-time target processing and an image processing method based on the diffraction optical neural network system for real-time target processing. Background Art
[0002] The traditional von Neumann architecture has power consumption and memory walls due to the separation of storage and computing, and Moore's Law has gradually slowed down. The diffraction optical neural network fully integrates the characteristics of high-speed optical communication, optical interconnection, optical integration, silicon-based optoelectronics, etc., and has the advantages of ultra-high speed, large bandwidth, and multi-dimensionality. It has broad application prospects in the fields of high-performance computing and artificial intelligence, and is a highly competitive solution to break through the limits of traditional microelectronic computing in the post-Moore era. At present, the diffraction optical neural network is mostly manufactured by 3D printing or photolithography technology based on the trained diffraction parameters. When the diffraction layer is manufactured, its physical structure is fixed, and there is no possibility of adjusting the diffraction layer structure to reuse it for different purposes. It cannot meet the different functions required in different scenarios, resulting in the problem of single function of the diffraction optical neural network. Summary of the invention
[0003] The present application provides a diffraction optical neural network system for real-time target processing and an image processing method based on the diffraction optical neural network system for real-time target processing, which can repeatedly perform diffraction modulation on light multiple times according to demand, reduce the limitation of the fixed structure on the number of diffraction, and thus enhance the availability of the system.
[0004] In a first aspect, the present application provides a diffraction optical neural network system for real-time target processing, including a light source module, a diffraction modulation module, and a control processing module;
[0005] The light source module emits monochromatic polarized light to the diffraction modulation module;
[0006] The control processing module is used to obtain a target image to be identified, and a plurality of groups of modulation parameters for diffraction modulation performed by the diffraction modulation module;
[0007] The diffraction modulation module is used to repeatedly perform diffraction modulation on the monochromatic polarized light based on the target image to obtain a processed image corresponding to the target image;
[0008] The diffraction modulation module includes a first modulation layer, a second modulation layer and a first acquisition module, and the diffraction modulation includes:
[0009] The control processing module loads the grayscale distribution map of the target image into the first modulation layer; and loads the current modulation parameters into the second modulation layer;
[0010] The first modulation layer modulates the received monochromatic polarized light according to the grayscale distribution diagram of the target image, and outputs a light intensity distribution that is consistent with the grayscale distribution. Figure 1 a first modulated optical signal caused by
[0011] The second modulation layer receives the first modulated optical signal, modulates the first modulated optical signal according to the current modulation parameter sent by the control processing module, and outputs a modulated second modulated optical signal;
[0012] The first acquisition module acquires the second modulated light signal to obtain a light intensity distribution diagram of the second modulated light signal, and transmits the light intensity distribution diagram of the second modulated light signal to the control processing module. The control processing module updates the target image to the light intensity distribution diagram and updates the modulation parameters to the next set to perform the next diffraction modulation.
[0013] According to the diffraction optical neural network system provided in this embodiment, the light source module emits monochromatic polarized light to the diffraction modulation module, and the diffraction modulation module modulates the monochromatic polarized light. The monochromatic polarized light is first modulated into an optical signal consistent with the grayscale distribution of the target image to be processed, that is, the first modulated optical signal, and then modulated again. The first modulated optical signal is modulated according to the modulation parameters of this modulation to obtain a second modulated optical signal, which is collected by the acquisition module as a light intensity distribution diagram. The light intensity distribution diagram is used as the target image for the next diffraction modulation by the control processing module, and the modulation parameters are updated to the next modulation parameters, and the second modulated optical signal is continuously subjected to the next diffraction modulation. The diffraction modulation module in this embodiment can repeatedly perform diffraction modulation according to actual needs, can be applied to various optical diffraction neural network structures, and has strong usability.
[0014] In an exemplary embodiment, the light source module includes:
[0015] Laser, which emits monochromatic polarized light;
[0016] A first half-wave plate, disposed between the laser and the first modulation layer, for modulating the polarization direction of the monochromatic polarized light into P polarized light;
[0017] The modulated monochromatic polarized light is incident on the first modulation layer.
[0018] In an exemplary embodiment, the light source module further includes:
[0019] A diverging lens and a collimating lens are sequentially arranged between the laser and the first half-wave plate;
[0020] The diverging lens diverges the monochromatic polarized light beam emitted by the laser;
[0021] The collimating lens collimates the divergent light beam, and the collimated light beam passes through the first half-wave plate;
[0022] a reflector, arranged on an outgoing light path of the first half-wave plate, and used for reflecting the monochromatic polarized light passing through the first half-wave plate to an aperture;
[0023] The aperture is used to adjust the beam size of the monochromatic polarized light.
[0024] In an exemplary embodiment, the diffraction modulation module further includes:
[0025] A second half-wave plate, disposed between the first modulation layer and the second modulation layer, for modulating the polarization direction of the first modulated optical signal output by the first modulation layer into P polarized light;
[0026] The modulated first modulated optical signal is incident on the second modulation layer.
[0027] In an exemplary embodiment, the first modulation layer includes a first PBS and a first SLM, and the second modulation layer includes a second PBS and a second SLM; the first SLM is a reflective amplitude-type SLM, and the second SLM is a reflective phase-type SLM; the first PBS transmits incident monochromatic polarized light to the first SLM, and the first SLM modulates the monochromatic polarized light based on a target image, and outputs a first modulated light signal whose light intensity distribution is consistent with the grayscale distribution;
[0028] The first modulated optical signal output by the first SLM is reflected by the first PBS to the second half-wave plate and converted into P polarized light by the second half-wave plate;
[0029] The first modulated optical signal converted into P polarized light is incident on the second PBS;
[0030] The second PBS transmits the first modulated optical signal to the second SLM, and the second SLM modulates the first modulated optical signal based on the modulation parameters sent by the control processing module, and outputs a modulated second modulated optical signal;
[0031] The second modulated optical signal is reflected and outputted by the second PBS.
[0032] In an exemplary embodiment, it further includes:
[0033] The second acquisition module acquires the target image in real time and transmits the target image to the control processing module.
[0034] In an exemplary embodiment, the diffraction modulation module further includes:
[0035] A converging lens is arranged on an outgoing optical path of the second modulated optical signal output by the diffraction modulation module;
[0036] The converging lens is used to converge the second modulated optical signal, and the converged second modulated optical signal is incident on the first acquisition module.
[0037] In an exemplary embodiment, the control processing module is also used to train the simulation model to obtain the phase modulation parameters of each layer of the network of the simulation model after training, and the phase modulation parameters of each layer of the network of the simulation model are the modulation parameters for each diffraction modulation.
[0038] In a second aspect, the present application provides an image processing method of a diffraction light neural network system based on the above-mentioned real-time processing target, comprising:
[0039] Acquire a target image and a modulation parameter of the target image;
[0040] The target image and the modulation parameters are input into a diffraction optical neural network system, and the diffraction optical neural network system repeatedly performs diffraction modulation on the target image according to the number of the modulation parameters to obtain a processed image of the diffraction modulation output.
[0041] The image processing method in this embodiment can process the target image in real time, and diffract the target image that needs to be classified, enhanced, etc. by using the diffracting optical neural network system. The diffracting optical neural network system can perform diffracting modulation of a corresponding number of times according to the number of layers of the modulation parameter, and finally obtain the processed image. The diffracting optical neural network system can be applicable to diffracting modulation of various times, which reduces the limitation of the structure on the diffraction number and can enhance the usability of the system.
[0042] In a third aspect, the present application provides an image processing device based on the above-mentioned diffraction light neural network system for real-time processing targets, comprising:
[0043] An image acquisition module, used to acquire a target image and modulation parameters of the target image;
[0044] The image diffraction processing module is used to input the target image and the modulation parameters into the diffraction optical neural network system, and the diffraction optical neural network system repeatedly performs diffraction modulation on the target image according to the number of layers of the modulation parameters to obtain a processed image of the diffraction modulation output.
[0045] In a fourth aspect, the present application provides an electronic device, the electronic device comprising a memory and one or more processors. The memory stores one or more computer programs, the computer programs comprising instructions, and when the instructions are executed by the processor, the electronic device can execute the image processing method of the diffraction optical neural network system based on real-time processing targets as in the second aspect.
[0046] In a fifth aspect, the present application provides a computer-readable storage medium having instructions stored therein. When the instructions are executed on an electronic device, the electronic device executes the image processing method of the diffraction light neural network system based on real-time processing targets as in the second aspect.
[0047] In a sixth aspect, the present application provides a computer program product, which, when executed on an electronic device, enables the electronic device to execute the image processing method of the diffraction optical neural network system based on real-time processing targets as described in the second aspect.
[0048] It can be understood that the beneficial effects that can be achieved by the electronic device, computer-readable storage medium, and computer program product provided above can refer to the beneficial effects in the first aspect or the second aspect, and will not be repeated here. BRIEF DESCRIPTION OF THE DRAWINGS
[0049] Figure 1 A schematic diagram of the structure of a diffraction optical neural network system for real-time target processing provided by an embodiment of the present application;
[0050] Figure 2 A schematic flow chart of an image processing method based on a diffraction light neural network system for real-time processing of targets provided in an embodiment of the present application;
[0051] Figure 3 A schematic diagram of the structure of an image processing device based on a diffraction light neural network system for real-time processing of targets provided in an embodiment of the present application;
[0052] Figure 4 A schematic diagram of the structure of an electronic device provided in an embodiment of the present application. DETAILED DESCRIPTION
[0053] In order to facilitate the clear description of the technical solutions of the embodiments of the present application, in the embodiments of the present application, the words "first", "second" and the like are used to distinguish the same items or similar items with substantially the same functions and effects. For example, the first chip and the second chip are only used to distinguish different chips, and their order is not limited. Those skilled in the art can understand that the words "first", "second" and the like do not limit the quantity and execution order, and the words "first", "second" and the like do not necessarily limit the difference. It should be noted that in the embodiments of the present application, the words "exemplary" or "for example" are used to indicate examples, illustrations or explanations. Any embodiment or design described as "exemplary" or "for example" in the present application should not be interpreted as being more preferred or more advantageous than other embodiments or design solutions. Specifically, the use of words such as "exemplary" or "for example" is intended to present related concepts in a specific way. In the embodiments of the present application, "at least one" refers to one or more, and "more" refers to two or more.
[0054] It should be noted that the “at…” in the embodiments of the present application may be the instant when a certain situation occurs, or may be a period of time after the occurrence of a certain situation, and the embodiments of the present application do not specifically limit this.
[0055] The implementation of this embodiment will be described in detail below with reference to the accompanying drawings.
[0056] The present embodiment provides a diffraction optical neural network system for real-time target processing. By way of example, the system can be applied to various electronic devices such as computers (PCs), tablet computers, virtual reality / augmented reality devices, wearable devices, industrial computers, and vehicle computers; it can also be applied to servers, cloud computing, server clusters, etc., and the present embodiment does not impose any special limitations on this.
[0057] Figure 1 A schematic diagram of the architecture of a diffraction optical neural network system for real-time processing of targets provided in an embodiment of the present application is shown.
[0058] like Figure 1 As shown, the diffraction optical neural network system may include a light source module 8, a diffraction modulation module 16, and a control processing module 17.
[0059] The light source module 8 is used to emit monochromatic P-polarized light. For example, the light source module 8 may be a laser.
[0060] The diffraction modulation module 16 is arranged on the outgoing light path of the light source module 8, and the monochromatic P polarized light emitted from the light source module 8 is incident on the diffraction modulation module 16. The diffraction modulation module 16 is used to perform diffraction modulation cyclically, and the number of cycles can be determined according to actual conditions. After the diffraction modulation is cyclically performed, a processed image of the target image can be output.
[0061] The control processing module 17 is used to obtain the target image to be identified in real time, and the modulation parameters of each diffraction modulation performed by the diffraction modulation module 16. The control processing module 17 can be a programmable controller, a processor, a computer, a tablet computer, an industrial computer, etc., but the present embodiment is not limited thereto.
[0062] Further, continue to refer to Figure 1 In an exemplary embodiment, the light source module 8 may specifically include a laser 1 and a first half-wave plate 5. The laser 1 is used to emit monochromatic linear polarized light; the first half-wave plate 5 is arranged between the laser and the first modulation layer, and is used to modulate the polarization direction of the monochromatic linear polarized light into P polarized light; the modulated monochromatic polarized light is incident on the first modulation layer.
[0063] The laser 1 is used to provide a continuous and stable light source for the entire system. The light it emits is monochromatic light and linearly polarized light with a fixed polarization direction.
[0064] The first half-wave plate 5 is used to change the polarization direction of the linearly polarized light output by the laser 1, and modulate the polarization state of the light beam incident on the SLM amplitude modulation array into P polarized light according to the use requirements of the spatial light modulator (SLM), so as to ensure that the reflective amplitude type SLM can work normally.
[0065] The light source module 8 may further include a laser 1, a beam expander lens group 4, a first half-wave plate 5, a reflector 6 and an aperture 7. The beam expander lens group 4 specifically includes a diverging lens 2 and a collimating lens 3.
[0066] A diverging lens 2 and a collimating lens 3 are sequentially arranged between the laser 1 and the first half-wave plate 5; the diverging lens 2 diverges the monochromatic polarized light beam emitted by the laser 1; the collimating lens 3 collimates the diverged light beam, and the collimated light beam passes through the first half-wave plate 5; a reflector 6 is arranged on the output light path of the first half-wave plate, and is used to reflect the monochromatic polarized light passing through the first half-wave plate to the aperture 7; the aperture 7 is used to adjust the size of the monochromatic polarized light beam.
[0067] The diverging lens 2 is used to diverge the monochromatic light beam emitted by the laser 1, so that the energy of the light beam is dispersed to avoid excessive power from damaging other components, while ensuring that the light beam irradiated on the reflective amplitude-type SLM 10 can fill the entire amplitude modulation array.
[0068] The collimating lens 3 is used to collimate and shape the monochromatic light beam diverged by the diverging lens 2, so that the light beam finally passes through the system after the collimating lens 3 as parallel light.
[0069] The beam expander group 4, which consists of a diverging lens 2 and a collimating lens 3, is used to diverge the monochromatic light beam emitted by the laser 1 and collimate the light beam so that the light beam finally passes through the system after the collimating lens 3 as parallel light, and disperses the energy of the light beam to avoid damage to other components due to excessive power, and ensures that the light beam irradiated on the reflective amplitude-type SLM 10 can fill the entire amplitude modulation array.
[0070] The reflector 6 is a high-power reflector, which is used to change the direction of light propagation in the system to avoid the light path being too long. At the same time, since the change in the angle between the reflected light and the incident light is equal to twice the angle of rotation of the mirror, adding a high-power reflector 6 in the light path makes it easier to adjust the light path to collimation.
[0071] The aperture 7 is used to limit the diameter of the beam after expansion, and the beam diameter can be adjusted according to the size of the target grayscale image, while filtering out unnecessary stray light.
[0072] The light source module 8 is used to provide the system with a continuous and stable monochromatic collimated light beam with an adjustable diameter and a specific polarization direction to meet the requirements for normal operation of the system.
[0073] The diffraction modulation module 16 specifically includes a first modulation layer, a second modulation layer, and a first acquisition module 15. The diffraction modulation process includes: the control processing module 17 loads the grayscale distribution map of the current target image to the first modulation layer; and loads the current modulation parameters to the second modulation layer; the first modulation layer modulates the received monochromatic P polarized light according to the grayscale distribution map of the target image, and outputs a light intensity distribution and the grayscale distribution. Figure 1 The first modulated light signal is generated by the first acquisition module 15, and the second modulation layer receives the first modulated light signal, modulates the first modulated light signal according to the current modulation parameters sent by the control processing module, and outputs the modulated second modulated light signal; the first acquisition module 15 acquires the second modulated light signal, obtains the light intensity distribution diagram of the second modulated light signal, and transmits the light intensity distribution diagram of the second modulated light signal to the control processing module, and the control processing module updates the target image to the light intensity distribution diagram, and updates the current modulation parameters to the next set of modulation parameters to perform the next diffraction modulation. Each diffraction modulation process includes the modulation of the first modulation layer and the modulation of the second modulation layer.
[0074] It can be understood that the diffraction modulation process in this embodiment can be repeated for multiple times, and a light intensity distribution diagram can be obtained each time diffraction modulation is performed. The light intensity distribution diagram is used as the target image for the next diffraction modulation, and the modulation parameters are updated to the next group, and the diffraction modulation process can be performed again. The control processing module 17 can store multiple groups of modulation parameters in advance, and load one group of modulation parameters to the second modulation layer each time according to the order of the multiple groups of modulation parameters, perform a diffraction modulation, and then update to the next group of modulation parameters, and then perform the next diffraction modulation, and so on, until all modulation parameters are processed, then the diffraction modulation process ends, and the final light intensity distribution diagram is output as the processed image.
[0075] For example, when the processing task of the system is image classification, the processed image is used to indicate the classification category of the target image, and the category corresponding to the area with the strongest light intensity in the processed image is the classification category of the target image.
[0076] Exemplarily, each modulation layer may include a polarizing beam splitter (PBS) and a spatial light modulator (SLM). Each diffraction layer includes two spatial light modulators. The amplitude-type SLM only modulates the amplitude and does not modulate the phase. Therefore, the amplitude-type SLM does not perform diffraction modulation, and all diffraction modulation is completed by the phase-type SLM.
[0077] The working principle of the PBS is as follows: two right-angle prisms made of two negative uniaxial crystals (ne < no) are glued along the inclined plane with a suitable adhesive (ne < n < no). The optical axes of the two crystals are parallel to the right-angle plane of light passing and the incident plane. At this time, the e-light is p-polarized light, and the o-light is s-polarized light (the component of the electric vector vibrating perpendicular to the incident plane is called the s-component. If a linearly polarized light has only the s-component, it is called s-polarized light).
[0078] The SLM is an optical device that can control the phase, amplitude, or polarization of a light beam and can be used to modulate the intensity, phase, polarization state, etc. of an incident light beam, thereby realizing optical information processing.
[0079] In this embodiment, a reflective amplitude-type SLM is adopted. The reflective amplitude-type SLM can incident p-polarized light and reflect s-polarized light. The reflective amplitude-type SLM is used to perform amplitude modulation on the incident p-polarized light. It is a sandwich structure composed of a photoconductive layer, a dielectric mirror, a liquid crystal layer, a transparent conductive electrode (ITO) of a glass substrate, etc., and then a two-dimensional array liquid crystal panel is composed of many basic independent units. The control processing module 17 can control the voltage through software to adjust the rotation direction of the liquid crystal molecules, realize the amplitude regulation of a single pixel point, and the reflected light of the modulation array is s-polarized light.
[0080] The modulation parameters may include the amplitude, light intensity, phase, polarization state, etc. of each pixel point of the SLM. This embodiment is not limited thereto.
[0081] The PBS in the first modulation layer is abbreviated as the first PBS, and the amplitude-type SLM is abbreviated as the first SLM. The PBS in the second modulation layer is abbreviated as the second PBS, and the phase-type SLM is abbreviated as the second SLM.
[0082] Continue to refer to Figure 1 , the first modulation layer includes the first PBS 9 and the first SLM 10, and the second modulation layer includes the second PBS 12 and the second SLM 13.
[0083] The control processing module 17 loads the pre-acquired target image onto the first SLM 10 in the first modulation layer. The monochromatic polarized light emitted by the light source module 8 is transmitted to the first SLM 10 by the first PBS 9 in the first modulation layer, and is modulated for the first time in the first SLM 10, so that the monochromatic polarized light is modulated into a light intensity distribution that is consistent with the grayscale distribution of the target image. Figure 1 After being reflected by the first PBS 9, the first modulated optical signal is further reflected by the first PBS 9 to the second modulation layer.
[0084] The control processing module 17 loads the stored first group of modulation parameters onto the second SLM 13 in the second modulation layer. The first PBS 9 in the first modulation layer reflects the first modulated optical signal, which is then transmitted to the second SLM 13 in the second modulation layer by the second PBS 12 in the second modulation layer. A second modulation is performed in the second SLM 13. The first modulated optical signal is modulated into a second modulated optical signal according to the modulation parameters loaded on the second SLM 13 in the second modulation layer. The second modulated optical signal is reflected by the second SLM 13 and then received by the first acquisition module 15.
[0085] The first PBS 9 and the second PBS 12 can both transmit light beams with polarization direction of P. When the polarization direction of the monochromatic polarized light incident on the first PBS 9 is P, the monochromatic polarized light can be transmitted through the first PBS 9 and incident on the first SLM 10 to be modulated by the first SLM 10.
[0086] Exemplarily, the diffraction modulation module 16 also includes: a second half-wave plate 11, arranged between the first modulation layer and the second modulation layer, for modulating the polarization direction of the first modulated light signal output by the first modulation layer to P polarized light; the modulated first modulated light signal is incident on the second modulation layer.
[0087] Since the reflected light of the reflective amplitude-type SLM is S-polarized light, according to the use requirements of the reflective phase-type SLM, the polarization state of the light beam incident on the phase modulation array needs to be modulated into P-polarized light. The second half-wave plate is used to change the polarization direction of the reflected light beam after the reflective amplitude-type SLM performs amplitude modulation on the light field, so as to ensure that the reflective phase-type SLM can work normally.
[0088] In this embodiment, the first PBS 9 transmits the incident monochromatic polarized light to the first SLM 10, and the first SLM 10 modulates the monochromatic polarized light based on the target image, and outputs a first modulated light signal whose light intensity distribution is consistent with the grayscale distribution; the first modulated light signal output by the first SLM 10 is reflected by the first PBS 9 to the second half-wave plate 11, and is converted into P-polarized light by the second half-wave plate 11; the first modulated light signal converted into P-polarized light is incident on the second PBS 12; the second PBS 12 transmits the first modulated light signal to the second SLM 13, and the second SLM 13 modulates the first modulated light signal based on the modulation parameters sent by the control processing module, and outputs the modulated second modulated light signal; the second modulated light signal is reflected and output by the second PBS 9.
[0089] The first acquisition module 15 is an image sensor, which is a complementary metal-oxide-semiconductor (CMOS) chip. The first acquisition module 15 is used to obtain the light field intensity after the reflective phase-type SLM in the second modulation layer performs phase modulation to form an intensity distribution diagram.
[0090] In this embodiment, multi-layer diffraction can be realized step by step by using CMOS and a control processing module. The number of cycles in actual use can be changed according to the accuracy of different layers during training, thereby changing the actual number of layers of the diffraction optical neural network, compressing the optical path, and having the characteristics of system miniaturization and high space utilization.
[0091] In an exemplary embodiment, the diffraction modulation module also includes: a converging lens 14, which is arranged on the output optical path of the second modulated optical signal output by the diffraction modulation module; the converging lens 14 is used to converge the second modulated optical signal, and the converged second modulated optical signal is incident on the first acquisition module 15.
[0092] The converging lens 14 is used to converge the light beam after the phase modulation of the reflective phase-type SLM 13, so that each pixel point on the reflective phase-type SLM 13 corresponds one-to-one with each pixel point on the first acquisition module 15. When the pixel size and filling rate of the detection array of the first acquisition module 15 are equal to the pixel size and filling rate of the phase modulation array of the reflective phase-type SLM 13, the converging lens 14 can be removed, otherwise it is necessary to use an image compression / filling algorithm to adjust the image of the second modulated light signal to a resolution equal to the phase modulation parameters.
[0093] Exemplarily, the diffraction optical neural network system may further include: a second acquisition module 18, which acquires a target image and transmits the target image to the control processing module.
[0094] The second acquisition module 18 is used to obtain the light field intensity of the target that needs to be processed by the diffractive optical neural network in real time, that is, the target image. In this embodiment, the second acquisition module can be used to obtain the target image in real time, saving the time of loading the target image onto the diffractive optical neural network after obtaining the target image, and has the characteristics of good real-time performance and simple processing.
[0095] Furthermore, the modulation parameters used for each diffraction modulation can be pre-stored in the control processing module 17. Exemplarily, the control processing module 17 is also used to train the simulation model, determine the phase modulation parameters of each layer of the network of the trained simulation model, and use the phase modulation parameters of each layer of the network as the modulation parameters for each diffraction modulation.
[0096] A simulation model is constructed by a simulation tool, and the structure of the model can be the same as the structure of the above-mentioned diffraction optical neural network system. The simulation model is constructed by mathematical means and can be run in an electronic device or processor, while the diffraction optical neural network system is an optical system and is constructed by optical elements. By training the simulation model, a trained model is obtained, thereby obtaining the phase modulation parameters of each layer of the network of the trained model, and then the phase modulation parameters of each layer of the network are saved in the control processing module 17 as the modulation parameters of each diffraction modulation, so that the control module 17 loads the corresponding modulation parameters into the diffraction modulation module 16 according to the current number of diffraction modulations.
[0097] The number of network layers of the simulation model can be adjusted according to the accuracy of the actual training. In this embodiment, the number of cycles in actual use can be changed according to the accuracy of different layers during training, thereby changing the actual number of layers of the diffraction optical neural network, compressing the optical path, and having the characteristics of system miniaturization and high space utilization.
[0098] Based on the above diffraction optical neural network system, this embodiment also provides an image processing method, which specifically includes:
[0099] S1, expand the continuous and stable monochromatic linearly polarized laser. Since SLM requires p-polarized light as input, the expanded light beam needs to be polarized and modulated to become p-polarized light. Then, the aperture 6 is used to adjust the beam diameter and filter out some stray light according to the size of the target or target surface to obtain continuous, stable, diameter-controllable monochromatic p-polarized light.
[0100] S2, according to actual usage requirements, the target image acquired in real time by the CMOS (second acquisition module 18) is loaded on the amplitude modulation array of the reflective amplitude type SLM 10, and the p-polarized light output by the light source module 8 is amplitude modulated so that the intensity of the reflected light field (i.e., the first modulated light signal) is distributed according to the grayscale value of the loaded grayscale image.
[0101] S3, since the reflected light of the reflective amplitude-type SLM 10 is s-polarized light, polarization modulation needs to be performed again between it and the reflective phase-type SLM 13, so that the amplitude-modulated light beam becomes p-polarized light again.
[0102] S4, according to the phase modulation parameters of the first layer network trained in the simulation model, the p-biased light beam after amplitude modulation is phase modulated, so that it can effectively simulate the effect of the target emitting a parallel light field through a single-layer diffraction optical neural network.
[0103] S5, the diffraction acquisition CMOS (i.e., the first acquisition module 15) transmits the acquired phase-modulated light intensity distribution diagram to the reflective amplitude-type SLM 10 through the control processing module 17, and then transmits the phase modulation parameters of the next layer of network trained in the simulation model to the reflective phase-type SLM 13, so that it can achieve the effect of the parallel light field modulated by the previous layer of network passing through the next layer of diffraction optical neural network.
[0104] S6, repeat S5 several times to achieve the effect that the parallel light field emitted by the target passes through the multi-layer diffraction neural network.
[0105] The specific process of S1 is as follows: the laser 1 emits a continuous and stable monochromatic linearly polarized laser beam which passes through a beam expander 4 composed of a diverging lens 2 and a collimating lens 3 to expand the beam diameter, wherein the diverging lens 2 makes the originally small-diameter parallel beam diverge, and the collimating lens 3 adjusts the diverged beam to parallel light. Since the normal operation of the SLM requires p-polarized light incident, the first half-wave plate 5 is required to perform polarization modulation on the expanded beam to make it p-polarized light, and then the aperture 7 is used to adjust the beam diameter according to the size of the target or target surface and filter out part of the stray light to obtain a continuous, stable, diameter-controllable monochromatic p-polarized light.
[0106] The specific process of S2 is as follows: the first PBS 9 performs polarization filtering on the outgoing light beam of the light source module 8, which can transmit p-polarized light and reflect s-polarized light, so that only p-polarized light is incident on the amplitude modulation array of the reflective amplitude-type SLM 10. At the same time, according to actual use requirements, the target image acquired in real time by the target acquisition CMOS 18 is loaded as a grayscale image on the amplitude modulation array of the reflective amplitude-type SLM 10 through the control processing module 17, so that the light field intensity of the reflected light is distributed according to the grayscale value of the loaded grayscale image. At this time, the reflected light field of the reflective amplitude-type SLM 10 is the input I of the first layer of the diffraction network. 1_in .
[0107] The specific process of S3 is as follows: the incident p-polarized light is reflected as s-polarized light after being amplitude modulated by the reflective amplitude-type SLM 10, and is reflected by the first PBS 9 in a direction perpendicular to the incident light path, and then polarization-modulated by the second half-wave plate 11 so that the amplitude-modulated light beam becomes p-polarized light again. At this time, the amplitude-modulated light beam can be transmitted through the second PBS 12 and incident on the phase modulation array of the rear reflective phase-type SLM 13.
[0108] The specific process of S4 is as follows: according to the physical model of the system (i.e., the diffraction optical neural network system), a simulation model is constructed using code to train the phase modulation parameters, and the trained phase modulation parameters of the first layer of the diffraction optical neural network are loaded on the reflective phase type SLM 13 through the control processing module 17, and the amplitude modulated light beam is phase modulated to achieve the effect of the first layer of the diffraction optical neural network. At this time, the reflected light field of the reflective phase type SLM 13 is the output I of the first layer of the diffraction network. 1_out .
[0109] The specific process of S5 is as follows:
[0110] S5.1, the p-polarized light incident on the reflective phase-type SLM 13 is reflected as s-polarized light after phase modulation, which is reflected by the second PBS 12 in a direction perpendicular to the original incident light path, and then converged by the converging lens 14 to the detection array of the diffraction collection CMOS 15. The detection array will obtain the phase-modulated light field I of the target image. 1_out The intensity distribution is transmitted to the control processing module (17)17.
[0111] S5.2, the control processing module 17 receives the diffraction collection CMOS 15 to obtain I 1_out The light field intensity is then transmitted to the reflective amplitude type SLM 10, so that it performs amplitude modulation on the parallel light beam input by the light source module 8 according to the acquired light field intensity. At this time, the reflected light field of the reflective amplitude type SLM 10 is I 1_out , that is, the input light field I of the second layer of diffraction optical neural network 2_in , which conforms to the fully connected structure of traditional diffractive optical neural networks.
[0112] S5.3, when the control processing module 17 updates the amplitude modulation image loaded on the reflective amplitude type SLM 10 to the diffraction acquisition CMOS 15 to obtain I 2_in After the light field intensity is adjusted, the phase modulation parameters of the trained second-layer diffraction optical neural network are loaded onto the phase modulation array of the reflective phase-type SLM 13. 2_in Phase modulation is performed, and the reflected light field of the reflective phase type SLM 13 after modulation is the output I of the second layer diffraction network 2_out .
[0113] The specific process of S6 is: repeat S5.1 and S5.2, 2_out The intensity is transmitted to the reflective amplitude type SLM 10 and the reflected light field is I 3_in , and then repeat S5.3, load the phase modulation parameters of the trained third layer diffraction optical neural network into the reflective phase type SLM 13, and complete the I 3_in The modulation of the reflected light field is I 3_out , and then repeat S5.1, S5.2, and S5.3 several times to realize the function of a multi-layer fully connected diffractive optical neural network.
[0114] According to the phase modulation parameters of the first layer network trained in the simulation model, the p-biased light beam after amplitude modulation is phase modulated so that it can achieve the effect of the parallel light field emitted by the target passing through a single-layer diffraction optical neural network.
[0115] In this embodiment, SLM is used to form an integrated diffraction optical neural network system, and all components are debugged and calibrated in advance and packaged into a whole, avoiding the trouble of repeatedly adjusting the position of components, and having the characteristics of convenience; and, by using SLM to replace the diffraction layer of the traditional 3D printed diffraction optical neural network, different trained phase modulation parameters can be loaded according to actual needs, and different needs of different goals can be achieved, so that the network has the characteristics of adjustable parameters and reusability. In addition, multi-layer diffraction can be realized step by step using CMOS and control processing modules, and the number of cycles in actual use can be changed according to the accuracy of different layers during training, thereby changing the actual number of layers of the diffraction optical neural network, compressing the optical path, and having the characteristics of system miniaturization and high space utilization.
[0116] Furthermore, this embodiment also provides an image processing method based on a diffraction light neural network system for real-time processing targets, such as Figure 2 As shown, specifically including:
[0117] S201: Acquire a target image and modulation parameters of the target image.
[0118] S202: Input the target image and the modulation parameters into the diffraction optical neural network system, and the diffraction optical neural network system cyclically performs diffraction modulation on the target image according to the number of layers of the modulation parameters to obtain a processed image of the diffraction modulation output.
[0119] The image processing method provided in this embodiment can be applied to a variety of scenarios. In an image classification scenario, the target image can be an image that needs to be classified, and the recognition result of the target image includes the category corresponding to the target image. In an image enhancement scenario, the target image can be an image that needs to be enhanced, and the processing result of the target image is the image after enhancement processing.
[0120] This image processing method is based on Figure 1 The method of the diffraction optical neural network system for real-time processing of targets, the specific processing process has been described in detail in the above-mentioned diffraction optical neural network system, and will not be repeated here.
[0121] In an exemplary embodiment, the repeatedly performing diffraction modulation on the target image through the diffraction optical neural network system includes: training to obtain a simulation model, determining the phase modulation parameters of each layer of the network of the simulation model, and using the phase modulation parameters of each layer of the network as the modulation parameters of each diffraction modulation.
[0122] This embodiment also provides an image processing device based on a diffractive optical neural network system for real-time processing targets, which can be used to execute the image processing method based on the diffractive optical neural network system for real-time processing targets. Figure 3 As shown, the image processing device 300 based on the diffraction light neural network system for real-time processing of targets specifically includes:
[0123] The image acquisition module 310 is used to acquire the target image and the modulation parameters of the target image; the image diffraction processing module 320 is used to input the target image and the modulation parameters into the diffraction optical neural network system, and the diffraction optical neural network system cyclically performs diffraction modulation on the target image according to the number of layers of the modulation parameters to obtain a processed image of the diffraction modulation output.
[0124] The specific details of each module or unit in the above-mentioned image processing device based on the diffraction light neural network system for real-time processing targets have been described in detail in the corresponding diffraction light neural network system, so they will not be repeated here.
[0125] The embodiment of the present application also provides an electronic device, Figure 4 A schematic diagram of the structure of an electronic device suitable for implementing the embodiments of the present disclosure is shown. Figure 4 The electronic device 600 shown is only an example and should not bring any limitation to the functions and scope of use of the embodiments of the present disclosure.
[0126] like Figure 4As shown, electronic device 600 includes a central processing unit (CPU) 601, which can perform various appropriate actions and processes according to a program stored in a read-only memory (ROM) 602 or a program loaded from a storage part 608 into a random access memory (RAM) 603. In RAM 603, various programs and data required for system operation are also stored. CPU 601, ROM 602 and RAM 603 are connected to each other via a bus 604. An input / output (I / O) interface 605 is also connected to bus 604.
[0127] The following components are connected to the I / O interface 605: an input section 606 including a keyboard, a mouse, etc.; an output section 607 including a cathode ray tube (CRT), a liquid crystal display (LCD), etc., and a speaker, etc.; a storage section 608 including a hard disk, etc.; and a communication section 609 including a network interface card such as a LAN card, a modem, etc. The communication section 609 performs communication processing via a network such as the Internet. A drive 610 is also connected to the I / O interface 605 as needed. A removable medium 611, such as a magnetic disk, an optical disk, a magneto-optical disk, a semiconductor memory, etc., is installed on the drive 610 as needed, so that a computer program read therefrom is installed into the storage section 608 as needed.
[0128] In particular, according to an embodiment of the present disclosure, the process described above with reference to the flowchart can be implemented as a computer software program. For example, an embodiment of the present disclosure includes a computer program product, which includes a computer program carried on a computer-readable storage medium, and the computer program includes a program code for executing the method shown in the flowchart. In such an embodiment, the computer program can be downloaded and installed from the network through the communication part 609, and / or installed from the removable medium 611. When the computer program is executed by the central processing unit (CPU) 601, the above-mentioned functions defined in the embodiment of the present application are executed.
[0129] For example, when the computer program is executed by the central processing unit (CPU) 601, it can be executed as follows: acquiring a target image and modulation parameters for the target image; inputting the target image and the modulation parameters into a diffraction optical neural network system, and the diffraction optical neural network system repeatedly performs diffraction modulation on the target image according to the number of the modulation parameters to obtain a processed image of the diffraction modulation output.
[0130] It should be noted that the computer-readable medium shown in the present disclosure may be a computer-readable signal medium or a computer-readable storage medium or any combination of the above two. The computer-readable storage medium may be, for example, but not limited to, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, device or device, or any combination of the above. More specific examples of computer-readable storage media may include, but are not limited to: an electrical connection with one or more wires, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the above. In the present disclosure, a computer-readable storage medium may be any tangible medium containing or storing a program that can be used by or in combination with an instruction execution system, device or device. In the present disclosure, a computer-readable signal medium may include a data signal propagated in a baseband or as part of a carrier wave, in which a computer-readable program code is carried. This propagated data signal may take a variety of forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination of the above. Computer-readable signal media may also be any computer-readable medium other than computer-readable storage media, which may send, propagate or transmit a program for use by or in conjunction with an instruction execution system, apparatus or device. The program code contained on the computer-readable medium may be transmitted using any appropriate medium, including but not limited to: wireless, wire, optical cable, RF, etc., or any suitable combination of the above.
[0131] The flow charts and block diagrams in the accompanying drawings illustrate the possible architecture, functions and operations of the systems, methods and computer program products according to various embodiments of the present disclosure. In this regard, each box in the flow chart or block diagram can represent a module, a program segment, or a part of a code, and the above-mentioned module, program segment, or a part of a code contains one or more executable instructions for realizing the specified logical function. It should also be noted that in some alternative implementations, the functions marked in the box can also occur in a different order from the order marked in the accompanying drawings. For example, two boxes represented in succession can actually be executed substantially in parallel, and they can sometimes be executed in the opposite order, depending on the functions involved. It should also be noted that each box in the block diagram or flow chart, and the combination of the boxes in the block diagram or flow chart can be implemented with a dedicated hardware-based system that performs a specified function or operation, or can be implemented with a combination of dedicated hardware and computer instructions.
[0132] The units involved in the embodiments described in the present disclosure may be implemented by software or hardware, and the units described may also be arranged in a processor. The names of these units do not, in some cases, constitute limitations on the units themselves.
[0133] As another aspect, the present application also provides a computer-readable medium, which may be included in the electronic device described in the above embodiment; or may exist independently without being assembled into the electronic device. The above computer-readable medium carries one or more programs, and the one or more programs include instructions, and when the instructions are executed by the electronic device, the electronic device implements the method described in the above embodiment.
[0134] It should be noted that, although several modules or units of the equipment for action execution are mentioned in the above detailed description, this division is not mandatory. In fact, according to the embodiments of the present application, the features and functions of two or more modules or units described above can be embodied in one module or unit. On the contrary, the features and functions of one module or unit described above can be further divided into being embodied by multiple modules or units.
[0135] The above contents are only specific implementation methods of the present application, but the protection scope of the present application is not limited thereto. Any changes or substitutions within the technical scope disclosed in the present application shall be included in the protection scope of the present application. Therefore, the protection scope of the present application shall be subject to the protection scope of the claims.
Claims
1. A diffraction optical neural network system for real-time processing of targets, characterized in that: It includes a light source module, a diffraction modulation module and a control processing module; The light source module emits monochromatic linear polarized light to the diffraction modulation module; The control processing module acquires a target image to be identified and a plurality of groups of modulation parameters for diffraction modulation performed by the diffraction modulation module; The diffraction modulation module cyclically performs diffraction modulation on the monochromatic linear polarized light based on the target image to obtain a processed image corresponding to the target image; The diffraction modulation module includes a first modulation layer, a second modulation layer and a first acquisition module, and the diffraction modulation includes: The control processing module loads the grayscale distribution map of the target image into the first modulation layer; and loads the current modulation parameters into the second modulation layer; The first modulation layer modulates the received monochromatic linear polarized light according to the grayscale distribution diagram of the target image, and outputs a first modulated light signal whose light intensity distribution is consistent with the grayscale distribution diagram; The second modulation layer receives the first modulated optical signal, modulates the first modulated optical signal according to the current modulation parameter sent by the control processing module, and outputs a modulated second modulated optical signal; The first acquisition module acquires the second modulated light signal to obtain a light intensity distribution diagram of the second modulated light signal, and transmits the light intensity distribution diagram of the second modulated light signal to the control processing module. The control processing module updates the target image to the light intensity distribution diagram of the second modulated light signal, and updates the modulation parameters to the next set to perform the next diffraction modulation.
2. The diffraction optical neural network system for real-time processing of targets according to claim 1, characterized in that: The light source module comprises: Laser, emitting monochromatic linearly polarized light; A first half-wave plate, disposed between the laser and the first modulation layer, for modulating the polarization direction of the monochromatic linear polarized light into P polarized light; The modulated monochromatic linearly polarized light is incident on the first modulation layer.
3. The diffraction optical neural network system for real-time target processing according to claim 2, characterized in that: The light source module further includes: A diverging lens and a collimating lens are sequentially arranged between the laser and the first half-wave plate; The diverging lens diverges the monochromatic polarized light beam emitted by the laser; The collimating lens collimates the divergent light beam, and the collimated light beam passes through the first half-wave plate; a reflector, arranged on an outgoing light path of the first half-wave plate, and used for reflecting the monochromatic polarized light passing through the first half-wave plate to an aperture; The aperture is used to adjust the beam size of the monochromatic polarized light.
4. The diffraction optical neural network system for real-time processing of targets according to claim 1, characterized in that: The diffraction modulation module also includes: A second half-wave plate, disposed between the first modulation layer and the second modulation layer, for modulating the polarization direction of the first modulated optical signal output by the first modulation layer into P polarized light; The modulated first modulated optical signal is incident on the second modulation layer.
5. The diffraction optical neural network system for real-time processing of targets according to claim 4, characterized in that: The first modulation layer includes a first PBS and a first SLM, and the second modulation layer includes a second PBS and a second SLM; the first SLM is a reflective amplitude-type SLM, and the second SLM is a reflective phase-type SLM; The first PBS transmits the incident monochromatic polarized light to the first SLM, and the first SLM modulates the monochromatic polarized light based on the target image and outputs a first modulated light signal whose light intensity distribution is consistent with the grayscale distribution; The first modulated optical signal output by the first SLM is reflected by the first PBS to the second half-wave plate and converted into P polarized light by the second half-wave plate; The first modulated optical signal converted into P polarized light is incident on the second PBS; The second PBS transmits the first modulated optical signal to the second SLM, and the second SLM modulates the first modulated optical signal based on the modulation parameters sent by the control processing module, and outputs a modulated second modulated optical signal; The second modulated optical signal is reflected and outputted by the second PBS.
6. The diffraction optical neural network system for real-time processing of targets according to claim 1, characterized in that: Also includes: The second acquisition module acquires the target image and transmits the target image to the control processing module.
7. The diffraction optical neural network system for real-time target processing according to claim 1, characterized in that: The diffraction modulation module also includes: A converging lens is arranged on an outgoing optical path of the second modulated optical signal output by the diffraction modulation module; The converging lens is used to converge the second modulated optical signal, and the converged second modulated optical signal is incident on the first acquisition module.
8. The diffraction optical neural network system for real-time target processing according to claim 1, characterized in that: The control processing module is also used to train the simulation model to obtain the phase modulation parameters of each layer of the network of the simulation model after training. The phase modulation parameters of each layer of the network of the simulation model are the modulation parameters for each diffraction modulation.
9. An image processing method based on a diffraction light neural network system for real-time processing of targets, characterized in that: include: Acquire a target image and a modulation parameter of the target image; The target image and the modulation parameters are input into a diffraction optical neural network system, and the diffraction optical neural network system cyclically performs diffraction modulation on the target image according to the number of layers of the modulation parameters to obtain a processed image of the diffraction modulation output.
10. The method according to claim 9, characterized in that Also includes: Phase modulation parameters are obtained through simulation model training, the phase modulation parameters of each layer of the network of the simulation model are determined, and the phase modulation parameters of each layer of the network are used as modulation parameters for each diffraction modulation.