Light-in and light-out intelligent calculation, sensing and calculation integrated model and architecture

Abstract

The invention relates to the technical field of optical computing, in particular to a light-in-light-out intelligent computing, sensing and computing integrated model and architecture. The model comprises a plurality of light field coding modules, a plurality of efficient calculation modules and a light detection module, the plurality of light field coding modules are used for extracting key features in an input image based on high-dimensional information of the input image in a dynamic light field scene to obtain light field data corresponding to the input image; the plurality of efficient calculation modules are connected with the plurality of light field coding modules and are used for calculating the light field data of the target area to obtain an optical signal of a calculation result; and the optical detection module is connected with the plurality of efficient calculation modules and is used for outputting a target calculation result based on the optical signals of the plurality of calculation results. According to the invention, the application of light in efficient calculation is realized, the dependence on energy consumption and resources is reduced, the calculation efficiency is remarkably improved in a calculation sense mode, and the development of an intelligent calculation technology is promoted.

Description

Technical Field

[0001] The present disclosure relates to the field of optical computing technology, and in particular to a light-input and light-output intelligent computing and sensing integrated model and architecture. Background Art

[0002] With the rapid development of artificial intelligence and scientific computing, the complexity and scale of computing needs are also increasing. However, the existing electronic computing technology is limited by Moore's Law, and its performance is gradually approaching saturation, making it difficult to effectively cope with the increasingly stringent requirements of large-scale complex algorithms on computing power and power consumption. Light has the advantages of high throughput and low latency during propagation, and by using the dimensional information of the light field, optical computing technology is regarded as an effective means to break the existing computing bottleneck and improve perception accuracy and decision reliability. Summary of the invention

[0003] The present disclosure aims to solve one of the technical problems in the related art at least to some extent.

[0004] To this end, the first objective of the present disclosure is to propose an integrated light-input and light-output intelligent computing model.

[0005] The second objective of the present disclosure is to propose an integrated light-input and light-output intelligent computing and sensing architecture.

[0006] To achieve the above objectives, the first embodiment of the present disclosure proposes a light-in, light-out intelligent computing and sensing integrated model, including: multiple light field encoding modules and multiple efficient computing modules and light detection modules, wherein: The multiple light field encoding modules are used to extract key features in the input image based on high-dimensional information of the input image in the dynamic light field scene, and obtain light field data corresponding to the input image; The multiple efficient computing modules are connected to the multiple light field encoding modules, and are used to calculate the light field data of the target area to obtain optical signals of the calculation results; The optical detection module is connected to the multiple high-efficiency computing modules and is used to output a target computing result based on the optical signals of the multiple computing results.

[0007] Optionally, the light field encoding module includes at least one of a spatial encoding module, an angle encoding module and a spectral polarization encoding module, wherein: The spatial encoding module is used to perform spatial encoding on the high-dimensional information of the input image in the dynamic light field scene, and map the light fields at different spatial positions to different sub-pixel positions on the image plane; The angle encoding module is used to encode and map the angle information in the light field into different light intensity distribution characteristics in the sensor block corresponding to the sub-pixel through the angle encoding reconstructible metasurface at each sub-pixel of the image plane; The spectral polarization encoding module is used to in-situ modulate the light field spectrum and polarization dimension in the different light intensity distribution characteristics, perform set activation and summation operations in the spectrum-polarization space, and obtain light field data corresponding to the input image.

[0008] Optionally, the spatial encoding module is specifically used to spatially encode high-dimensional information of an input image in a dynamic light field scene through a point-to-point mapping lens in each time frame, and map light fields at different spatial positions to different sub-pixel positions on an image plane.

[0009] Optionally, the multiple light field encoding modules are cascade connected.

[0010] Optionally, the efficient calculation module includes a diffraction encoding module, an interference calculation module and a diffraction decoding module, the output of the diffraction encoding module is the input of the interference calculation module, and the output of the interference calculation module is the input of the diffraction decoding module, wherein: The diffraction coding module is used to perform data coding calculation on the light field data of the target area to obtain coded data; The interference calculation module is used to perform reconfigurable all-optical feature calculation on the coded data by inputting calculation parameters to obtain feature data; The diffraction decoding module is used to decode the characteristic data to obtain an optical signal of a calculation result.

[0011] Optionally, each of the multiple efficient computing modules corresponds to a different target area in the input image.

[0012] To achieve the above-mentioned purpose, the second aspect of the present disclosure proposes a light-input and light-output intelligent computing and sensing integrated architecture, including: at least one light-input and light-output intelligent computing and sensing integrated model shown in any one of the aforementioned first aspects.

[0013] Optionally, the architecture includes: a plurality of light field encoding modules and a plurality of efficient computing modules.

[0014] Optionally, the multiple light field encoding modules are connected in a cascade manner.

[0015] Optionally, the efficient calculation module includes a diffraction encoding module, an interference calculation module and a diffraction decoding module, the output of the diffraction encoding module is the input of the interference calculation module, and the output of the interference calculation module is the input of the diffraction decoding module.

[0016] In summary, the light-in, light-out intelligent computing, sensing and calculating integrated model and architecture provided by the present disclosure pre-processes the light field before perception through multiple light field encoding modules, and implements the combination of optoelectronic computing after perception through multiple high-efficiency computing modules, thereby realizing the application of light in efficient computing, reducing the dependence on energy consumption and resources, significantly improving computing efficiency in the form of computing and sensing, and promoting the development of intelligent computing technology. At the same time, the present disclosure provides a new technical path for the field of intelligent perception and computing, which has broad application prospects, especially in fields with extremely high performance requirements such as unmanned systems, autonomous driving, ultrafast scientific imaging and transient phenomenon analysis.

[0017] Additional aspects and advantages of the present disclosure will be given in part in the following description and in part will be obvious from the following description or learned through practice of the present disclosure. BRIEF DESCRIPTION OF THE DRAWINGS

[0018] The above and / or additional aspects and advantages of the present disclosure will become apparent and easily understood from the following description of the embodiments in conjunction with the accompanying drawings, in which: Figure 1 A structural diagram of a light-input and light-output intelligent computing and sensing-computing integrated model provided in an embodiment of the present disclosure; Figure 2 A schematic diagram of the structure of an efficient computing module provided in an embodiment of the present disclosure. DETAILED DESCRIPTION

[0019] Embodiments of the present disclosure are described in detail below, and examples of the embodiments are shown in the accompanying drawings, wherein the same or similar reference numerals throughout represent the same or similar elements or elements having the same or similar functions. The embodiments described below with reference to the accompanying drawings are exemplary and are intended to be used to explain the present disclosure, and should not be construed as limiting the present disclosure.

[0020] With the rapid development of information technology, vision-based intelligent perception and computing methods have become the current mainstream. Existing technologies mainly rely on RGB cameras and depth cameras to achieve intelligent perception and decision-making. However, as the complexity of applications increases, existing technologies face limitations from basic principles such as physical laws and information theory, making it difficult to further improve the perception and decision-making capabilities of the system. Based on this, how to introduce more dimensional information of the light field into the perception system has become an effective means to improve the perception accuracy and decision-making reliability of the intelligent system.

[0021] At present, there are many challenges in both perception and calculation to fully utilize the multi-dimensional information contained in high-dimensional light fields. On the perception side, the more dimensions a high-dimensional light field has, the greater the amount of information. As the dimensions increase, the amount of information grows exponentially, resulting in a sharp increase in the perception system's demand for information flux. However, due to limitations in hardware cost, size, accuracy, and manufacturing yield, existing systems have very limited perception flux, often requiring the sacrifice of spatial or temporal resolution, and are unable to fully perceive all the information in a high-dimensional light field.

[0022] Also, on the computing side after perception, processing such a huge amount of high-dimensional light field information places extremely high demands on computing power, speed, and energy consumption, and traditional electronic computing technology is difficult to meet this demand. Especially in scenarios that require real-time decision-making, traditional electronic computing architectures face bottlenecks in computing power, speed, and energy consumption, and are difficult to meet the requirements of efficient and fast processing. As Moore's Law gradually becomes ineffective, electronic computing performance is approaching saturation, and there is an urgent need to explore new intelligent perception system architectures to break through the limitations of existing technologies.

[0023] The present disclosure is described in detail below with reference to specific embodiments.

[0024] Figure 1 This is a schematic diagram of the structure of a light-input and light-output intelligent computing and sensing integrated model provided by an embodiment of the present disclosure. Figure 1 As shown, the light-in, light-out intelligent computing and sensing integrated model includes: multiple light field encoding modules and multiple high-efficiency computing modules and light detection modules, among which: Multiple light field encoding modules, used to extract key features in the input image based on high-dimensional information of the input image in the dynamic light field scene, and obtain light field data corresponding to the input image; A plurality of efficient computing modules are connected to a plurality of light field encoding modules, and are used to calculate the light field data of the target area to obtain an optical signal of the calculation result; The optical detection module is connected to a plurality of high-efficiency computing modules and is used to output target computing results based on optical signals of a plurality of computing results.

[0025] In one embodiment of the present disclosure, a plurality of light field encoding modules are cascade-connected. Also, in one embodiment of the present disclosure, each of the light field encoding modules may include at least one of a spatial encoding module, an angle encoding module, and a spectral polarization encoding module. For example, in one embodiment of the present disclosure, the model includes a first light field encoding module, a second light field encoding module, and a third light field encoding module, wherein the first light field encoding module includes a spatial encoding module, the second light field encoding module includes an angle encoding module, and the third light field encoding module includes a spectral polarization encoding module.

[0026] Among them, in one embodiment of the present disclosure, the above-mentioned spatial encoding module is used to encode the high-dimensional information input in the dynamic light field scene in the spatial dimension, and map the light fields at different spatial positions to different sub-pixel positions on the image plane; the angle encoding module is arranged at the image plane of the spatial encoding module, and is used to further encode the high-dimensional information after spatial encoding in the angle dimension at each sub-pixel on the image plane, and the angle encoding is realized by a phase-modulated reconfigurable metasurface, and the angle information encoding in the light field is mapped to different light intensity distribution characteristics in the sensor block corresponding to the sub-pixel; the spectral polarization encoding module is arranged at the image plane of the angle encoding module, attached above the planar array image sensor, and is used to perform in-situ modulation on the spectrum and polarization dimensions in different light intensity distribution characteristics, and perform set activation and summation operations in the spectral-polarization space to obtain light field data corresponding to the input image.

[0027] Furthermore, in one embodiment of the present disclosure, the above-mentioned spatial encoding module can be specifically used to spatially encode the high-dimensional information of the input image in the dynamic light field scene through a point-to-point mapping lens in each time frame, and map the light fields at different spatial positions to different sub-pixel positions on the image plane.

[0028] In one embodiment of the present disclosure, the above-mentioned multiple light field coding modules can be used to perceive the angle, spectrum, polarization, and time dimension information of light. The sampled data flux can be reduced by all-optical coding and feature extraction to match the perception capability of the sensor. The flux bottleneck of traditional perception can be broken through by computing perception, and the complexity of subsequent calculations and data processing pressure can be reduced.

[0029] Furthermore, in one embodiment of the present disclosure, after the light field data corresponding to the input image is obtained through the above-mentioned multiple light field encoding modules, a general intelligent computing capability of 160 TOPS / W can be achieved by using the advantages of optical signals through multiple high-efficiency computing modules, breaking through the efficiency bottleneck of traditional computing.

[0030] Specifically, in one embodiment of the present disclosure, the above-mentioned efficient computing module may include a diffraction encoding module, an interference computing module and a diffraction decoding module, the output of the diffraction encoding module is the input of the interference computing module, and the output of the interference computing module is the input of the diffraction decoding module.

[0031] Among them, in one embodiment of the present disclosure, the above-mentioned diffraction encoding module is used to perform data encoding calculation on the light field data of the target area to obtain encoded data; the interference calculation module is used to perform reconfigurable all-optical feature calculation on the encoded data by feeding in calculation parameters to obtain feature data; the diffraction decoding module is used to decode the feature data to obtain an optical signal of the calculation result.

[0032] Specifically, in one embodiment of the present disclosure, the basic unit in the interference calculation module is an MZI interferometer, which can realize phase modulation, interference mixing and output of two optical signals through a phase shifter, and realize 2*2 reconfigurable matrix operation. At the same time, through the cascaded MZI interferometers, the interconnected input and output realize the multiplication calculation of any 8*8 unitary matrix and any 8*1 vector, and any unitary matrix can be realized by feeding the corresponding calculation parameters into each MZI for phase modulation, thereby realizing reconfigurable all-optical feature calculation of the encoded data.

[0033] And, in one embodiment of the present disclosure, each of the above-mentioned multiple efficient computing modules can correspond to a different target area in the input image, and after obtaining the optical signals of the calculation results of different target areas through the multiple efficient computing modules, the target calculation results can be output through the optical detection module based on the optical signals of the multiple calculation results. Among them, the above-mentioned target calculation results can be the classification results to which the input image belongs, so that the above-mentioned light-in, light-out intelligent computing, sensing and computing integrated model can be applied to various fields, such as unmanned systems, autonomous driving, and ultrafast science.

[0034] In summary, the light-in, light-out intelligent computing, sensing and calculating integrated model provided by this embodiment uses multiple light field encoding modules to pre-process the light field before perception, and multiple high-efficiency computing modules to implement the combination of optoelectronic computing after perception, thereby realizing the application of light in efficient computing, reducing the dependence on energy consumption and resources, significantly improving computing efficiency in the form of computing and sensing, and promoting the development of intelligent computing technology. At the same time, the present disclosure provides a new technical path for the field of intelligent perception and computing, and has broad application prospects, especially in fields with extremely high performance requirements such as unmanned systems, autonomous driving, ultrafast scientific imaging and transient phenomenon analysis.

[0035] In order to implement the above embodiments, the present disclosure also proposes a light-input and light-output intelligent computing, sensing and calculating integrated architecture, which is a light-input and light-output intelligent computing, sensing and calculating integrated model provided by at least one of the above embodiments.

[0036] Optionally, the architecture includes: a plurality of light field encoding modules, a plurality of efficient computing modules and a light detection module, wherein the plurality of light field encoding modules are connected in a cascade manner.

[0037] Optionally, the efficient calculation module includes a diffraction encoding module, an interference calculation module and a diffraction decoding module, the output of the diffraction encoding module is the input of the interference calculation module, and the output of the interference calculation module is the input of the diffraction decoding module.

[0038] The collection, storage, use, processing, transmission, provision and disclosure of user personal information involved in this disclosure shall comply with the relevant laws and regulations and shall not violate public order and good morals.

[0039] It should be noted that personal information from users should be collected for legitimate and reasonable purposes and should not be shared or sold outside of these legitimate uses. In addition, such collection / sharing should be carried out after receiving the user's informed consent, including but not limited to notifying the user to read the user agreement / user notice and sign the agreement / authorization including authorization of relevant user information before the user uses the function. In addition, any necessary steps should be taken to protect and safeguard access to such personal information data and ensure that others who have access to personal information data comply with its privacy policy and procedures.

[0040] The present disclosure anticipates providing implementation schemes for users to selectively block the use or access of personal information data. That is, the present disclosure anticipates providing hardware and / or software to prevent or block access to such personal information data. Once the personal information data is no longer needed, risks can be minimized by limiting data collection and deleting the data. In addition, when applicable, such personal information is de-identified to protect the privacy of the user.

[0041] The acquisition, transmission, storage, use, and processing of data in the technical solution disclosed in this disclosure are in compliance with the relevant provisions of national laws and regulations.

[0042] It should be noted that in the embodiments of the present disclosure, certain software, components, models and other existing solutions in the industry may be mentioned, which should be regarded as exemplary. Their purpose is only to illustrate the feasibility of implementing the technical solution of the present application, but it does not mean that the applicant has or will necessarily use the solution.

[0043] In the description of the aforementioned embodiments, the description with reference to the terms "one embodiment", "some embodiments", "example", "specific example", or "some examples" etc. means that the specific features, structures, materials or characteristics described in conjunction with the embodiment or example are included in at least one embodiment or example of the present disclosure. In this specification, the schematic representations of the above terms do not necessarily refer to the same embodiment or example. Moreover, the specific features, structures, materials or characteristics described may be combined in any one or more embodiments or examples in a suitable manner. In addition, those skilled in the art may combine and combine the different embodiments or examples described in this specification and the features of the different embodiments or examples, unless they contradict each other.

[0044] In addition, the terms "first" and "second" are used for descriptive purposes only and should not be understood as indicating or implying relative importance or implicitly indicating the number of the indicated technical features. Thus, a feature defined as "first" or "second" may explicitly or implicitly include at least one of the features. In the description of the present disclosure, "plurality" means at least two, such as two, three, etc., unless otherwise clearly and specifically defined.

[0045] Any process or method description in a flowchart or otherwise described herein may be understood to represent a module, segment or portion of code that includes one or more executable instructions for implementing the steps of a custom logical function or process, and the scope of the preferred embodiments of the present disclosure includes additional implementations in which functions may not be performed in the order shown or discussed, including performing functions in a substantially simultaneous manner or in reverse order depending on the functions involved, which should be understood by technicians in the technical field to which the embodiments of the present disclosure belong.

[0046] The logic and / or steps represented in the flowchart or otherwise described herein, for example, can be considered as an ordered list of executable instructions for implementing logical functions, and can be embodied in any computer-readable medium for use by an instruction execution system, device or apparatus (such as a computer-based system, a system including a processor, or other system that can fetch instructions from an instruction execution system, device or apparatus and execute instructions), or in combination with these instruction execution systems, devices or apparatuses. For the purposes of this specification, "computer-readable medium" can be any device that can contain, store, communicate, propagate or transmit a program for use by an instruction execution system, device or apparatus, or in combination with these instruction execution systems, devices or apparatuses. More specific examples (non-exhaustive list) of computer-readable media include the following: an electrical connection with one or more wires (electronic device), a portable computer disk box (magnetic device), a random access memory (RAM), a read-only memory (ROM), an erasable and programmable read-only memory (EPROM or flash memory), an optical fiber device, and a portable compact disk read-only memory (CDROM). In addition, the computer-readable medium may even be paper or other suitable medium on which the program is printed, since the program may be obtained electronically, for example, by optically scanning the paper or other medium and then editing, interpreting or otherwise processing in a suitable manner if necessary, and then stored in a computer memory.

[0047] It should be understood that the various parts of the present disclosure can be implemented in hardware, software, firmware or a combination thereof. In the above-mentioned embodiments, multiple steps or methods can be implemented in software or firmware stored in a memory and executed by a suitable instruction execution system. For example, if implemented in hardware, as in another embodiment, it can be implemented by any one of the following technologies known in the art or their combination: a discrete logic circuit having a logic gate circuit for implementing a logic function for a data signal, a dedicated integrated circuit having a suitable combination of logic gate circuits, a programmable gate array (PGA), a field programmable gate array (FPGA), etc.

[0048] A person skilled in the art may understand that all or part of the steps in the method for implementing the above-mentioned embodiment may be completed by instructing related hardware through a program, and the program may be stored in a computer-readable storage medium, which, when executed, includes one or a combination of the steps of the method embodiment.

[0049] In addition, each functional unit in each embodiment of the present disclosure may be integrated into a processing module, or each unit may exist physically separately, or two or more units may be integrated into one module. The above-mentioned integrated module may be implemented in the form of hardware or in the form of a software functional module. If the integrated module is implemented in the form of a software functional module and sold or used as an independent product, it may also be stored in a computer-readable storage medium.

[0050] The storage medium mentioned above may be a read-only memory, a disk or an optical disk, etc. Although the embodiments of the present disclosure have been shown and described above, it can be understood that the above embodiments are exemplary and cannot be understood as limitations of the present disclosure. A person of ordinary skill in the art may change, modify, replace and modify the above embodiments within the scope of the present disclosure.

Claims

1. A light-input and light-output intelligent computing and sensing integrated model, characterized in that: include: Multiple light field encoding modules and multiple efficient computing modules and light detection modules, wherein: The multiple light field encoding modules are used to extract key features in the input image based on high-dimensional information of the input image in the dynamic light field scene, and obtain light field data corresponding to the input image; The multiple efficient computing modules are connected to the multiple light field encoding modules, and are used to calculate the light field data of the target area to obtain optical signals of the calculation results; The optical detection module is connected to the multiple high-efficiency computing modules and is used to output a target computing result based on the optical signals of the multiple computing results.

2. The model according to claim 1, characterized in that The light field encoding module includes at least one of a spatial encoding module, an angle encoding module and a spectral polarization encoding module, wherein: The spatial encoding module is used to perform spatial encoding on the high-dimensional information of the input image in the dynamic light field scene, and map the light fields at different spatial positions to different sub-pixel positions on the image plane; The angle encoding module is used to encode and map the angle information in the light field into different light intensity distribution characteristics in the sensor block corresponding to the sub-pixel through the angle encoding reconstructible metasurface at each sub-pixel of the image plane; The spectral polarization encoding module is used to in-situ modulate the light field spectrum and polarization dimension in the different light intensity distribution characteristics, perform set activation and summation operations in the spectrum-polarization space, and obtain light field data corresponding to the input image.

3. The model according to claim 2, characterized in that The spatial encoding module is specifically used to spatially encode the high-dimensional information of the input image in the dynamic light field scene through a point-to-point mapping lens in each time frame, and map the light fields at different spatial positions to different sub-pixel positions on the image plane.

4. The model according to claim 1, characterized in that The multiple light field encoding modules are cascade-connected.

5. The model according to claim 1, characterized in that The efficient calculation module includes a diffraction encoding module, an interference calculation module and a diffraction decoding module. The output of the diffraction encoding module is the input of the interference calculation module, and the output of the interference calculation module is the input of the diffraction decoding module. The diffraction coding module is used to perform data coding calculation on the light field data of the target area to obtain coded data; The interference calculation module is used to perform reconfigurable all-optical feature calculation on the coded data by inputting calculation parameters to obtain feature data; The diffraction decoding module is used to decode the characteristic data to obtain an optical signal of a calculation result.

6. The model according to claim 1, characterized in that Each of the multiple efficient computing modules corresponds to a different target area in the input image.

7. A light-input and light-output intelligent computing and sensing integrated architecture, characterized in that: include: At least one light-input and light-output intelligent computing model as described in any one of claims 1 to 6.

8. The architecture according to claim 7, characterized in that include: Multiple light field encoding modules, multiple efficient computing modules and light detection modules.

9. The architecture according to claim 8, characterized in that include: The multiple light field encoding modules are connected in a cascade manner.

10. The architecture according to claim 8, characterized in that The efficient calculation module includes a diffraction encoding module, an interference calculation module and a diffraction decoding module. The output of the diffraction encoding module is the input of the interference calculation module, and the output of the interference calculation module is the input of the diffraction decoding module.

Images