Optical Input and Output Intelligent Computing and Sensing Integrated Model and Architecture

Abstract

The present disclosure relates to the field of optical computing technologies, and particularly to an optical input and optical output intelligent computing and sensing integrated model and architecture. The model includes: a plurality of optical field encoding modules, a plurality of high-efficiency computing modules, and an optical detection module. Among them, the plurality of optical field encoding modules are configured to extract key features in an input image based on high-dimensional information of the input image in a dynamic optical field scenario, and obtain optical field data corresponding to the input image; the plurality of high-efficiency computing modules are connected to the plurality of optical field encoding modules and are configured to calculate the optical field data of a target area to obtain an optical signal of a calculation result; the optical detection module is connected to the plurality of high-efficiency computing modules and is configured to output a target calculation result based on the optical signals of the plurality of calculation results. The present disclosure realizes the application of light in high-efficiency computing, reduces the dependence on energy consumption and resources, significantly improves the computing efficiency in a sensing manner, and promotes the development of intelligent computing technologies.

Description

Technical Field

[0001] The present disclosure relates to the field of optical computing technologies, and in particular, to an optical input and optical output intelligent computing and sensing integrated model and architecture. Background Art

[0002] With the rapid development of the fields of artificial intelligence and scientific computing, the complexity and scale of computing requirements are also increasing continuously. However, existing electronic computing technologies are limited by Moore's Law, and their performance is gradually approaching a saturation state, making it difficult to effectively meet the increasingly stringent requirements for computing power and power consumption of large-scale complex algorithms. Light has the advantages of high throughput and low latency during propagation, and by utilizing the dimensional information of the optical field, optical computing technology is regarded as an effective means to break through the existing computing bottleneck and improve the perception accuracy and decision reliability. Summary of the Invention

[0003] The present disclosure aims to at least partly solve one of the technical problems in the related art.

[0004] To this end, the first object of the present disclosure is to propose an optical input and optical output intelligent computing and sensing integrated model.

[0005] The second object of the present disclosure is to propose an optical input and optical output intelligent computing and sensing integrated architecture.

[0006] To achieve the above object, an embodiment of the first aspect of the present disclosure proposes an optical input and optical output intelligent computing and sensing integrated model, including: a plurality of optical field encoding modules, a plurality of high-efficiency computing modules, and an optical detection module, wherein,

[0007] The plurality of optical field encoding modules are configured to extract key features in the input image based on the high-dimensional information of the input image in the dynamic optical field scene, and obtain the optical field data corresponding to the input image;

[0008] The plurality of high-efficiency computing modules are connected to the plurality of optical field encoding modules, and are configured to calculate the optical field data of the target area to obtain an optical signal of the calculation result;

[0009] The optical detection module is connected to the plurality of high-efficiency computing modules, and is configured to output a target calculation result based on the optical signals of the plurality of calculation results.

[0010] Optionally, the optical field encoding module includes at least one of a spatial encoding module, an angular encoding module, and a spectral polarization encoding module, wherein,

[0011] The spatial encoding module is configured to perform spatial encoding on the high-dimensional information of the input image in the dynamic optical field scene, and map the optical fields at different spatial positions to different sub-pixel positions on the image plane;

[0012] The angle encoding module is configured to perform angle encoding through a reconfigurable metasurface at each sub-pixel of the image plane, and encode and map the angle information in the light field into different light intensity distribution characteristics within the sensor block corresponding to the sub-pixel;

[0013] The spectral polarization encoding module is configured to perform in-situ modulation on the light field spectrum and polarization dimensions in the different light intensity distribution characteristics, and perform set activation and summation operations in the spectral-polarization space to obtain the light field data corresponding to the input image.

[0014] Optionally, the spatial encoding module is specifically configured to perform spatial encoding on the high-dimensional information of the input image in the dynamic light field scene through a point-to-point mapping lens in each time frame, and map the light fields at different spatial positions to different sub-pixel positions on the image plane.

[0015] Optionally, the multiple light field encoding modules are connected in cascade.

[0016] Optionally, the efficient computing module includes a diffraction encoding module, an interference computing module, and a diffraction decoding module. The output of the diffraction encoding module is the input of the interference computing module, and the output of the interference computing module is the input of the diffraction decoding module. Among them,

[0017] The diffraction encoding module is configured to perform data encoding calculation on the light field data of the target area to obtain encoded data;

[0018] The interference computing module is configured to perform reconfigurable all-optical feature calculation on the encoded data through the feeding of calculation parameters to obtain feature data;

[0019] The diffraction decoding module is configured to decode the feature data to obtain the optical signal of the calculation result.

[0020] Optionally, each of the multiple efficient computing modules corresponds to a different target area in the input image.

[0021] To achieve the above object, an embodiment of the second aspect of the present disclosure proposes an optical input-optical output intelligent computing and sensing integrated architecture, including: at least one optical input-optical output intelligent computing and sensing integrated model shown in any one of the foregoing first aspects.

[0022] Optionally, the architecture includes: a plurality of light field encoding modules and a plurality of efficient computing modules.

[0023] Optionally, the connection manner between the multiple light field encoding modules is cascade.

[0024] Optionally, the high-efficiency computing module includes a diffraction encoding module, an interference calculation module, and a diffraction decoding module. The output of the diffraction encoding module is the input of the interference calculation module, and the output of the interference calculation module is the input of the diffraction decoding module.

[0025] In summary, the light-in light-out intelligent computing and sensing integrated model and architecture provided by the present disclosure preprocesses the light field through multiple light field encoding modules before sensing, and combines optoelectronic computing through multiple high-efficiency computing modules after sensing, thereby realizing the application of light in high-efficiency computing, reducing the dependence on energy consumption and resources, significantly improving the computing efficiency in a computing and sensing manner, and promoting the development of intelligent computing technology. At the same time, the present disclosure provides a new technical path for the field of intelligent sensing and computing, with broad application prospects, especially in fields with extremely high performance requirements such as unmanned systems, autonomous driving, ultrafast scientific imaging, and transient phenomenon analysis.

[0026] Additional aspects and advantages of the present disclosure will be given in part in the following description, become apparent in part from the following description, or be learned through the practice of the present disclosure. BRIEF DESCRIPTION OF THE DRAWINGS

[0027] The above and / or additional aspects and advantages of the present disclosure will become apparent and be readily understood from the following description of the embodiments in conjunction with the drawings, in which:

[0028] Figure 1 is a schematic structural diagram of a light-in light-out intelligent computing and sensing integrated model provided by an embodiment of the present disclosure;

[0029] Figure 2 is a schematic structural diagram of a high-efficiency computing module provided by an embodiment of the present disclosure. DETAILED DESCRIPTION OF THE EMBODIMENTS

[0030] Embodiments of the present disclosure will be described in detail below. Examples of the embodiments are shown in the drawings, where the same or similar reference numerals denote the same or similar elements or elements having the same or similar functions throughout. The embodiments described below with reference to the drawings are exemplary and are intended to explain the present disclosure and should not be construed as limiting the present disclosure.

[0031] With the rapid development of information technology, vision-based intelligent sensing and computing methods have become the mainstream at present. The prior art mainly relies on RGB cameras and depth cameras to achieve intelligent sensing and decision-making. However, with the increasing complexity of applications, the prior art faces limitations from basic principles such as physical laws and information theory, and it is difficult to further improve the sensing and decision-making capabilities of the system. Based on this, how to introduce more dimensional information of the light field into the sensing system has become an effective means to improve the sensing accuracy and decision-making reliability of intelligent systems.

[0032] At present, there are many challenges in both perception and computing in making full use of the multi-dimensional information contained in high-dimensional optical fields. At the perception end, the more dimensions there are in a high-dimensional optical field, the greater the amount of information. As the number of dimensions increases, the amount of information grows exponentially, leading to a sharp increase in the demand for information flux in the perception system. However, due to limitations in aspects such as hardware cost, size, precision, and manufacturing yield, the perception flux of existing systems is very limited, often requiring sacrificing spatial or temporal resolution and unable to fully perceive all the information in the high-dimensional optical field.

[0033] Moreover, at the computing end after perception, processing such a large amount of high-dimensional optical field information poses extremely high requirements for computing power, speed, and energy consumption, and traditional electronic computing technologies are difficult to meet this demand. Especially in scenarios that require real-time decision-making, traditional electronic computing architectures face bottlenecks in computing power, speed, and energy consumption and are difficult to meet the requirements of efficient and fast processing. As Moore's Law gradually fails and electronic computing performance approaches saturation, it is urgent to explore new intelligent perception system architectures to break through the limitations of existing technologies.

[0034] The following will elaborate on the present disclosure in detail with specific embodiments.

[0035] Figure 1 The structural schematic diagram of an optical input-optical output intelligent computing and sensing integrated model provided by an embodiment of the present disclosure. As Figure 1 shown, the optical input-optical output intelligent computing and sensing integrated model includes: a plurality of optical field encoding modules, a plurality of high-efficiency computing modules, and an optical detection module, where

[0036] The plurality of optical field encoding modules are used to extract key features in the input image based on the high-dimensional information of the input image in the dynamic optical field scene, and obtain the optical field data corresponding to the input image;

[0037] The plurality of high-efficiency computing modules are connected to the plurality of optical field encoding modules and are used to calculate the optical field data of the target area to obtain the optical signal of the calculation result;

[0038] The optical detection module is connected to the plurality of high-efficiency computing modules and is used to output the target calculation result based on the optical signals of the plurality of calculation results.

[0039] In an embodiment of the present disclosure, the plurality of optical field encoding modules are cascaded. Moreover, in an embodiment of the present disclosure, each of the above optical field encoding modules may include at least one of a spatial encoding module, an angular encoding module, and a spectral polarization encoding module. For example, in an embodiment of the present disclosure, the above model includes a first optical field encoding module, a second optical field encoding module, and a third optical field encoding module, where the first optical field encoding module includes a spatial encoding module, the second optical field encoding module includes an angular encoding module, and the third optical field encoding module includes a spectral polarization encoding module.

[0040] Among them, in an embodiment of the present disclosure, the above-mentioned spatial encoding module is used to encode the high-dimensional information input in the dynamic light field scene in the spatial dimension, and map the light fields at different spatial positions to different sub-pixel positions on the image plane; the angular encoding module is arranged at the image plane of the spatial encoding module, and is used to further encode the high-dimensional information after spatial encoding in the angular dimension at each sub-pixel of the image plane. The angular encoding is realized by a phase modulation reconfigurable metasurface, and the angular information in the light field is encoded and mapped into different light intensity distribution characteristics within the sensor block corresponding to the sub-pixel; the spectral polarization encoding module is arranged at the image plane of the angular encoding module and is attached above the planar array image sensor, and is used to perform in-situ modulation on the spectral and polarization dimensions in different light intensity distribution characteristics, and perform set activation and summation operations in the spectral-polarization space to obtain the light field data corresponding to the input image.

[0041] In addition, in an embodiment of the present disclosure, the above-mentioned spatial encoding module can specifically be used to spatially encode the high-dimensional information of the input image in the dynamic light field scene through a point-to-point mapping lens in each time frame, and map the light fields at different spatial positions to different sub-pixel positions on the image plane.

[0042] In an embodiment of the present disclosure, through the above-mentioned multiple light field encoding modules, the perception of light information in the angular, spectral, polarization, and time dimensions can be realized. By means of all-optical encoding and feature extraction, the data throughput of sampling is reduced, the perception ability of the sensor is matched, the throughput bottleneck of traditional perception is broken through in the way of computing perception, and the complexity of subsequent calculations and the data processing pressure are reduced.

[0043] In addition, in an embodiment of the present disclosure, after obtaining the light field data corresponding to the input image through the above-mentioned multiple light field encoding modules, through multiple high-efficiency computing modules, taking advantage of the optical signal, a general intelligent computing ability of 160 TOPS / W is realized, breaking through the efficiency bottleneck of traditional computing.

[0044] Specifically, in an embodiment of the present disclosure, the above-mentioned high-efficiency computing module may include a diffraction encoding module, an interference computing module, and a diffraction decoding module. The output of the diffraction encoding module is the input of the interference computing module, and the output of the interference computing module is the input of the diffraction decoding module.

[0045] Among them, in an embodiment of the present disclosure, the above-mentioned diffraction encoding module is used to perform data encoding calculation on the light field data of the target area to obtain encoded data; the interference computing module is used to perform reconfigurable all-optical feature calculation on the encoded data through the feeding of calculation parameters to obtain feature data; the diffraction decoding module is used to decode the feature data to obtain the optical signal of the calculation result.

[0046] Specifically, in an embodiment of the present disclosure, the basic unit in the interference calculation module is an MZI interferometer, which can achieve phase modulation, interference mixing, and output of two optical signals through a phase shifter, and realize a 2×2 reconfigurable matrix operation. At the same time, through cascaded MZI interferometers, the interconnected input and output realizes the multiplication calculation of any 8×8 unitary matrix and any 8×1 vector. Any unitary matrix can be realized by feeding the corresponding calculation parameters into each MZI for phase adjustment, so as to realize the reconfigurable all-optical feature calculation of the encoded data.

[0047] Moreover, in an embodiment of the present disclosure, each of the multiple high-efficiency calculation modules can correspond to different target regions in the input image. After obtaining the optical signals of the calculation results of different target regions through the multiple high-efficiency calculation modules, the optical detection module can output the target calculation result based on the optical signals of the multiple calculation results. Among them, the above-mentioned target calculation result can be the classification result to which the input image belongs, so that the above-mentioned light-in and light-out intelligent computing and sensing integrated model can be applied to various fields, such as unmanned systems, autonomous driving, and ultrafast science and other fields.

[0048] In summary, the light-in and light-out intelligent computing and sensing integrated model provided in this embodiment preprocesses the light field through multiple light field encoding modules before sensing, and combines optoelectronic computing through multiple high-efficiency calculation modules after sensing, thus realizing the application of light in high-efficiency computing, reducing the dependence on energy consumption and resources, significantly improving the computing efficiency in the way of sensing and computing, and promoting the development of intelligent computing technology. At the same time, the present disclosure provides a new technical path for the field of intelligent sensing and computing, and has broad application prospects, especially in fields with extremely high performance requirements such as unmanned systems, autonomous driving, ultrafast science imaging, and transient phenomenon analysis.

[0049] To implement the above embodiment, the present disclosure also proposes a light-in and light-out intelligent computing and sensing integrated architecture, which includes at least one light-in and light-out intelligent computing and sensing integrated model provided by the foregoing embodiment.

[0050] Optionally, the architecture includes: multiple light field encoding modules, multiple high-efficiency calculation modules, and an optical detection module. Among them, the connection method between the multiple light field encoding modules is cascaded.

[0051] Optionally, the high-efficiency calculation module includes a diffraction encoding module, an interference calculation module, and a diffraction decoding module. The output of the diffraction encoding module is the input of the interference calculation module, and the output of the interference calculation module is the input of the diffraction decoding module.

[0052] The collection, storage, use, processing, transmission, provision, and disclosure of the user's personal information involved in the present disclosure and other processing all comply with the provisions of relevant laws and regulations and do not violate public order and good customs.

[0053] It should be noted that personal information from users should be collected for legal and reasonable purposes and should not be shared or sold outside of such legal uses. In addition, such collection / sharing should be carried out after obtaining the informed consent of the users, including but not limited to notifying the users to read the user agreement / user notice before using the function and signing an agreement / authorization including authorizing the relevant user information. In addition, any necessary steps should be taken to safeguard and protect access to such personal information data and to ensure that others with access to the personal information data comply with their privacy policies and procedures.

[0054] The present disclosure anticipates embodiments that can provide users with the option to block the use or access of personal information data. That is, the present disclosure anticipates that hardware and / or software can be provided to prevent or block access to such personal information data. Once the personal information data is no longer needed, the risk can be minimized by restricting data collection and deleting the data. In addition, when applicable, personal identifiers are removed from such personal information to protect the privacy of the users.

[0055] In the technical solutions of the present disclosure, the acquisition, transmission, storage, use, processing, etc. of data all comply with the relevant provisions of national laws and regulations.

[0056] It should be noted that in the embodiments of the present disclosure, certain industry-existing solutions such as certain software, components, models, etc. may be mentioned. They should be considered exemplary, and their purpose is only to illustrate the feasibility in the implementation of the technical solutions of this application, but it does not mean that the applicant has already or necessarily used this solution.

[0057] In the descriptions of the foregoing embodiments, the descriptions referring to terms such as "one embodiment", "some embodiments", "example", "specific example", or "some examples" mean that the specific features, structures, materials, or characteristics described in connection with the embodiment or example are included in at least one embodiment or example of the present disclosure. In this specification, the schematic descriptions of the above terms do not necessarily refer to the same embodiment or example. Moreover, the specific features, structures, materials, or characteristics described can be combined in a suitable manner in any one or more embodiments or examples. In addition, without contradiction, those skilled in the art can combine and combine the different embodiments or examples described in this specification and the features of different embodiments or examples.

[0058] In addition, the terms "first" and "second" are only used for descriptive purposes and cannot be understood as indicating or implying relative importance or implicitly specifying the quantity of the indicated technical features. Thus, the features defined with "first" and "second" can explicitly or implicitly include at least one of such features. In the description of the present disclosure, the meaning of "a plurality" is at least two, such as two, three, etc., unless otherwise clearly and specifically defined.

[0059] Any process or method description, whether in a flowchart or otherwise described herein, can be understood to represent a module, segment, or portion of code including one or more executable instructions for implementing a customized logical function or process. The scope of the preferred embodiments of the present disclosure includes additional implementations where functions may be executed not in the order shown or discussed, including in a substantially simultaneous manner according to the involved functions or in a reverse order, which should be understood by those skilled in the art to which the embodiments of the present disclosure pertain.

[0060] The logic and / or steps represented in a flowchart or otherwise described herein, for example, can be considered a sequenced list of executable instructions for implementing a logical function and can be embodied in any computer-readable medium for use by or in connection with an instruction execution system, apparatus, or device, such as a computer-based system, a system including a processor, or other systems that can fetch and execute instructions from the instruction execution system, apparatus, or device. For the purposes of this specification, a "computer-readable medium" can be any device that can contain, store, communicate, propagate, or transport a program for use by or in connection with an instruction execution system, apparatus, or device. More specific examples (a non-exhaustive list) of the computer-readable medium include the following: an electrical connection having one or more wires (electronic device), a portable computer diskette (magnetic device), a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or Flash memory), an optical fiber device, and a portable compact disc read-only memory (CDROM). Additionally, the computer-readable medium can even be paper or other suitable medium on which a program can be printed, as the program can be obtained, for example, electronically by optically scanning the paper or other medium, followed by editing, interpretation, or other suitable processing as necessary, and then stored in a computer memory.

[0061] It should be understood that the various parts of the present disclosure can be implemented in hardware, software, firmware, or a combination thereof. In the above embodiments, multiple steps or methods can be implemented in software or firmware stored in a memory and executed by a suitable instruction execution system. For example, if implemented in hardware, as in another embodiment, any one or a combination of the following technologies well known in the art can be used: discrete logic circuits having logic gate circuits for implementing logical functions on data signals, application specific integrated circuits having appropriate combinational logic gate circuits, programmable gate arrays (PGA), field programmable gate arrays (FPGA), etc.

[0062] Those of ordinary skill in the art can understand that all or part of the steps carried out in implementing the above-described embodiment methods can be completed by a program instructing relevant hardware. The program can be stored in a computer-readable storage medium. When the program is executed, it includes one or a combination of the steps of the method embodiment.

[0063] In addition, in each of the various embodiments of the present disclosure, the functional units can be integrated in a processing module, or each unit can exist physically alone, or two or more units can be integrated in a module. The above-mentioned integrated module can be implemented in the form of hardware or in the form of a software functional module. When the integrated module is implemented in the form of a software functional module and sold or used as an independent product, it can also be stored in a computer-readable storage medium.

[0064] The above-mentioned storage medium can be a read-only memory, a magnetic disk, an optical disk, etc. Although the embodiments of the present disclosure have been shown and described above, it can be understood that the above embodiments are exemplary and should not be construed as limiting the present disclosure. Those of ordinary skill in the art can make changes, modifications, substitutions, and variations to the above embodiments within the scope of the present disclosure.

Claims

1. An intelligent computing and sensing integrated model with light input and light output, characterized in that, Including: A plurality of light field encoding modules, a plurality of high-efficiency computing modules, and a light detection module, wherein, The plurality of light field encoding modules are configured to extract key features in the input image based on the high-dimensional information of the input image in the dynamic light field scene, and obtain the light field data corresponding to the input image; The plurality of high-efficiency computing modules are connected to the plurality of light field encoding modules, and are configured to calculate the light field data of the target area to obtain an optical signal of the calculation result; The light detection module is connected to the plurality of high-efficiency computing modules, and is configured to output a target calculation result based on the optical signals of the plurality of calculation results; The light field encoding module includes a spatial encoding module, an angular encoding module, and a spectral polarization encoding module, wherein, The spatial encoding module is configured to perform spatial encoding on the high-dimensional information of the input image in the dynamic light field scene, and map the light fields at different spatial positions to different sub-pixel positions on the image plane; The angular encoding module is configured to pass through an angularly encoded reconfigurable metasurface at each sub-pixel on the image plane, and encode and map the angular information in the light field into different light intensity distribution characteristics within the sensor block corresponding to the sub-pixel; The spectral polarization encoding module is configured to perform in-situ modulation on the light field spectrum and polarization dimension in the different light intensity distribution characteristics, and perform set activation and summation operations in the spectral-polarization space to obtain the light field data corresponding to the input image.

2. The model according to claim 1, wherein Specifically, the spatial encoding module is configured to perform spatial encoding on the high-dimensional information of the input image in the dynamic light field scene through a point-to-point mapping lens in each time frame, and map the light fields at different spatial positions to different sub-pixel positions on the image plane.

3. The model according to claim 1, wherein The plurality of light field encoding modules are cascaded.

4. The model according to claim 1, wherein The high-efficiency computing module includes a diffraction encoding module, an interference computing module, and a diffraction decoding module. The output of the diffraction encoding module is the input of the interference computing module, and the output of the interference computing module is the input of the diffraction decoding module, wherein, The diffraction encoding module is configured to perform data encoding calculation on the light field data of the target area to obtain encoded data; The interference computing module is configured to perform reconfigurable all-optical feature calculation on the encoded data through the feeding of calculation parameters to obtain feature data; The diffraction decoding module is configured to decode the feature data to obtain an optical signal of the calculation result.

5. The model according to claim 1, characterized in that, Each high-efficiency computing module in the plurality of high-efficiency computing modules corresponds to a different target area in the input image.

Images