Sensing and calculating integrated system
Through the integrated sensing and computing system, information processing is performed in the optical domain, the problem of information redundancy during the imaging process is solved, and efficient image information perception and calculation mapping is achieved. It is suitable for application scenarios with high computing speed and data processing capabilities.
Patent Information
- Application Number
- CN202510477300.7
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2025-04-16
- Publication Date
- 2025-07-18
AI Technical Summary
The existing imaging technology has large information redundancy in optical imaging, array sampling, transmission, storage and calculation processes, resulting in redundancy in computing quantity and data processing time, and lacks optically effective redundant processing methods.
The integrated inductive computing system is adopted to realize information processing and calculation in the optical domain through intensity mask plate, photonic integrated chip and digital signal processing system, including convolution kernel matrix, nonlinear region array and linear region array, optical convolution and nonlinear operation are performed to reduce data redundancy.
Complete information processing in the optical domain, reduce data redundancy, and realize efficient image information perception and computing mapping. It is suitable for application scenarios with high computing speed and data processing capabilities, such as autonomous driving, industrial detection and intelligent robots.
Smart Images
Figure HDA0005361716200000011 
Figure HDA0005361716200000012 
Figure HDA0005361716200000021
Abstract
Description
Technical Field
[0001] The present invention relates to the field of semiconductor technologies, and in particular, to a sensing and computing integrated system. Background Art
[0002] Light is the means by which the vast majority of organisms, such as humans, perceive the external world. Since the world's first photograph, light has been used to record and form photographs or videos. Now, most imaging is based on lens imaging. After entering the digital age, uniform frame-by-frame sampling is used instead of the sampling method of film before. The light field itself has many dimensions, and each dimension can carry information. However, after imaging, the high-dimensional information is lost, and only two-dimensional information is sampled on the plane. With the development of technology, people hope to better retain high-dimensional information or information in specific dimensions during imaging. Traditional imaging pursues seeing what is there. New imaging methods, such as coded imaging, can encode the scene or during shooting, so that the high-dimensional information is retained after dimensionality reduction and the high-dimensional information is obtained again after decoding. Although coded imaging has been developed for decades, currently coded imaging is still mainly used to replace traditional imaging for clearer imaging applications and can obtain information that traditional imaging cannot obtain. With the rapid development of technologies such as artificial intelligence, the proportion of machines in human life and work is increasing. For visual perception, the vision for humans and the vision for machines are different. For humans, a larger viewing angle, clearer and richer details are needed. For machines (computational vision), these things are not needed. What they care about is the dimension and the characteristics of the information.
[0003] The new imaging technology can preserve the high-dimensional information of the light field in addition to the intensity information of the light field as preserved by traditional lens-based imaging, and is more suitable for transmitting information to computational vision. Most of the imaging steps are: optical imaging -> array sampling -> transmission, storage, calculation, output. Each step (imaging, array sampling, transmission, storage, calculation) of the processing process is independently optimized without joint optimization, resulting in a large amount of information redundancy. These problems have attracted attention. Leading enterprises such as Sony and Bosch have begun to research new optical sensing technologies - new machine vision, which can capture key information that the machine is concerned about, such as motion information. The advantage of this is that it can avoid collecting too much redundant information during the shooting process. However, this method still does not perform any optical processing, and the optics still only serves for imaging. For example, in image classification and video classification tasks, the amount of classification information obtained only accounts for one-thousandth or even one-hundred-thousandth of the original data volume. More than 99.999% of the information is wasted. However, all this information is first collected, the imaging action is completed, and after becoming digital signals, it is processed in the electrical domain. These data are continuously dimensioned up and down, and tasks are practiced through deep learning methods. Although it is in-memory computing / sensing-in-computing, or near-memory computing at the backend, the data volume is still huge, and almost most of it is redundant information. These information are energy and are computational consumption. The redundancy includes time redundancy and spatial redundancy. Currently, there are some redundancy processing methods, such as compression and encoding processing, but there is no good way to solve it optically.
[0004] The integrated sensing and computing system solution introduced in this patent combines the final task with the front-end optical acquisition, and does more things in the optical field. Before the light information enters the sensor, information is calculated and signal processed during the imaging process / propagation process. The purpose is to extract the truly useful information optically and transmit it to the subsequent computing unit, avoiding the waste of a lot of data storage capacity, computing power, and including the subsequent transmission bandwidth. The optical domain system can be changed according to different tasks, and the optical acquisition structure can be optimized based on visual tasks.
[0005] This patent provides an imaging system for performing simple calculations in the optical domain. The system solution proposed in this patent is to first pass the image light field information of the physical world through an intensity mask template, on which a convolution kernel matrix composed of different convolution kernels is distributed. After the light field passes through the intensity mask template, the convolution result with the convolution kernel will be achieved, and then it will enter the nonlinear region array for a nonlinear operation. The information after passing through the activation function will pass through a large multiplication and addition array and finally enter the detector array to become an electrical signal, and the result we need will be obtained after subsequent digital signal processing. The system solution proposed in this patent performs some simple signal processing calculation processes simultaneously with the imaging steps, uses photons as the basic carriers for information processing and transmission, and performs convolution, nonlinear, and linear multiplication and addition calculations through photon devices. Compared with traditional imaging and existing coded imaging technologies, it completely avoids the acquisition of excessive redundant information. During the imaging step, calculations can be synchronized according to visual tasks, and the optimization of imaging and calculation can be carried out simultaneously, greatly reducing the calculation amount and the time for data processing. It is a new implementation solution for high-performance visual imaging. Summary of the Invention
[0006] The purpose of the present invention is to provide a sensing and computing integrated system in view of the deficiencies of the prior art.
[0007] The purpose of the present invention is achieved through the following technical solutions: A sensing and computing integrated system, the system includes: an intensity mask template, a photon integrated chip, and a digital signal processing system; the photon integrated chip includes a nonlinear region array, a linear region array, and an output region.
[0008] Further, the intensity mask template is provided with different convolution kernels to form a convolution kernel matrix, and each convolution kernel is composed of a small matrix of NxN, where 1 ≤ N ≤ 3; the small matrices are air or a medium respectively.
[0009] Further, the nonlinear region array includes multiple nonlinear operation regions, and each nonlinear operation region includes a grating coupler, a first laser, a second laser, a first semiconductor optical amplifier, a second semiconductor optical amplifier, and a third semiconductor optical amplifier.
[0010] Further, the output end of the grating coupler is connected to the input end of the first 1×2 multimode interference coupler; the output ends of the first 1×2 multimode interference coupler are respectively connected to the input end of the first 2×1 multimode interference coupler and the input end of the first semiconductor optical amplifier; the output end of the first laser is respectively connected to the input end of the first semiconductor optical amplifier and the input end of the second semiconductor optical amplifier; the output end of the second laser is connected to the input end of the second semiconductor optical amplifier; the output end of the first semiconductor optical amplifier is respectively connected to the input end of the first 2×1 multimode interference coupler and the input end of the second 2×1 multimode interference coupler; the output end of the second semiconductor optical amplifier is connected to the input end of the second 2×1 multimode interference coupler; the output ends of the first 2×1 multimode interference coupler and the second 2×1 multimode interference coupler are respectively connected to the third semiconductor optical amplifier.
[0011] Further, the linear region array includes a plurality of linear operation regions, and each linear operation region is a weight coupling structure.
[0012] Further, the output region is an end-face coupling structure.
[0013] Further, it is used to output optical information after adjusting the optical field mode for subsequent coupling reception.
[0014] Further, the intensity mask is used to perform a convolution operation on the input first optical information to obtain second optical information and transmit it to the photonic integrated chip; the first optical information is the optical information radiated by an object in the physical world.
[0015] The photonic integrated chip is used to multiply the second optical information by the activation function using the grating coupler in the nonlinear operation region to complete a nonlinear operation and obtain a third optical signal; then the third optical information is coupled into the linear operation region, and the linear operation region performs a large-scale matrix linear multiplication and addition operation on the third optical information to obtain a fourth optical signal; the fourth optical signal is output to the air medium through the output region and then converted into a first electrical signal.
[0016] The digital signal processing system is used to receive the first electrical signal and obtain an image result after digital signal processing.
[0017] Further, the first laser and the second laser are used to control the operating state points of the first semiconductor optical amplifier and the second semiconductor optical amplifier: when there is no input optical signal, both the first semiconductor optical amplifier and the second semiconductor optical amplifier operate in the saturation region, and the output optical power is in the high-level state; when the intensity of the input optical signal is very high, both the first semiconductor optical amplifier and the second semiconductor optical amplifier operate in the transparent state region, and the output optical power is in the equivalent low-level state; when the input optical signal is at a relatively low power, the operating points of the first semiconductor optical amplifier and the second semiconductor optical amplifier are between the high-level state and the equivalent low-level state, and the gain effect is also between the two. Therefore, the output optical power shows a relatively sensitive linear relationship with the power of the input optical signal.
[0018] The beneficial effects of the present invention are as follows: The integrated sensing and computing system provided by the present invention can not only complete information processing in the optical domain, reduce data redundancy, but also optimize the optical acquisition structure according to different visual tasks, realizing efficient perception and computational mapping of physical world image information; this new imaging system provides an innovative solution for high-performance visual imaging, and is particularly suitable for application scenarios with high requirements for computing speed and data processing capabilities, such as autonomous driving, industrial inspection, intelligent robots, VR / AR and other fields. BRIEF DESCRIPTION OF THE DRAWINGS
[0019] Figure 1 is a flowchart of the operation of an integrated sensing and computing system;
[0020] Figure 2 is a structural diagram of an intensity mask template composed of convolution kernels formed by 3x3 small matrices;
[0021] Figure 3 is a structural diagram of a photonic integrated chip;
[0022] Figure 4 is a structural diagram of a non-linear operation region;
[0023] Figure 5 is a structural diagram of a linear operation region;
[0024] Figure 6 is a diagram of the operating states of the first semiconductor optical amplifier and the second semiconductor optical amplifier. Among them, Figure 6 (a) is a diagram of the operating states of the first semiconductor optical amplifier and the second semiconductor optical amplifier when there is no input optical signal, Figure 6 (b) is a diagram of the operating states of the first semiconductor optical amplifier and the second semiconductor optical amplifier when the intensity of the input optical signal is very high, Figure 6 (c) is a diagram of the operating states of the first semiconductor optical amplifier and the second semiconductor optical amplifier when the input optical signal is at a relatively low power. DETAILED DESCRIPTION OF THE EMBODIMENTS
[0025] In order to make the objectives, technical solutions and advantages of the present invention more clearly understood, the present invention will be further described in detail below in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present invention, rather than all embodiments. All other embodiments obtained by those of ordinary skill in the art based on the embodiments of the present invention without creative efforts fall within the protection scope of the present invention.
[0026] Embodiment 1
[0027] The present invention provides a sensing and computing integrated system, which includes: an intensity mask, a photonic integrated chip, and a digital signal processing system. As Figure 3 shown, the photonic integrated chip includes a non-linear region array, a linear region array, and an output region; the non-linear region array, all of its structures are monolithically integrated on a III-V compound semiconductor material substrate.
[0028] The intensity mask is provided with different convolution kernels to form a convolution kernel matrix, and each convolution kernel is composed of a small matrix of NxN, where 1≤N≤3; the small matrices are respectively air or dielectric. If the small matrix is air, it allows light to be transmitted; if the small matrix is dielectric, it allows non-transparent light to be transmitted. When a scene is far from the intensity mask, it is considered that the light travels in a straight line. When the scene is close to the intensity mask, the distance from the intensity mask to the receiver satisfies a certain relationship: the light is considered to travel in a straight line, and the intensity mask is approximately located at the middle focal position between the object and the chip, and a convolution result of a scene to the receiver will be achieved. As Figure 2 shown, a structural diagram of an intensity mask composed of convolution kernels formed by 3x3 small matrices is shown. This mask is a convolution kernel of a 3x3 matrix, with a total of 9 convolution kernels, and each convolution kernel is composed of a pile of small holes. When a scene is far from the intensity mask, it is considered that the light travels in a straight line. When the distance from the scene to the intensity mask and from the intensity mask to the receiving end satisfies a relationship, a convolution result of a scene to the receiving end will be achieved, thus realizing a lensless convolution operation. Doing multiple convolution kernels on a single intensity mask has a better recognition effect than a single-core one.
[0029] The grating coupler realizes the efficient coupling between the optical waveguide and the free-space optical wave by utilizing the diffraction and resonance effects of the grating. The grating coupler collects the scattered light that has passed through the intensity mask and couples it into the optical waveguide, and the optical information of the scene enters the photonic integrated chip, facilitating subsequent interaction with the devices on the photonic integrated chip to achieve different functions.
[0030] As Figure 4As shown, the non-linear region array includes a plurality of non-linear operation regions, and each non-linear operation region includes a grating coupler, a first laser, a second laser, a first semiconductor optical amplifier (SOA1), a second semiconductor optical amplifier (SOA2), and a third semiconductor optical amplifier.
[0031] The output end of the grating coupler is connected to the input end of a first 1×2 multimode interference coupler; the output ends of the first 1×2 multimode interference coupler are respectively connected to the input end of a first 2×1 multimode interference coupler and the input end of the first semiconductor optical amplifier; the output end of the first laser is respectively connected to the input end of the first semiconductor optical amplifier and the input end of the second semiconductor optical amplifier; the output end of the second laser is connected to the input end of the second semiconductor optical amplifier; the output end of the first semiconductor optical amplifier is respectively connected to the input end of the first 2×1 multimode interference coupler and the input end of a second 2×1 multimode interference coupler; the output end of the second semiconductor optical amplifier is connected to the input end of the second 2×1 multimode interference coupler; the output ends of the first 2×1 multimode interference coupler and the second 2×1 multimode interference coupler are respectively connected to the third semiconductor optical amplifier.
[0032] The working principle is as follows: Adjust the output optical power of the first laser to control the first semiconductor optical amplifier at a working point in the saturation state but relatively close to the transparent state, and adjust the output optical power of the second laser to control the second semiconductor optical amplifier at a point near the saturation state and the transparent state. As Figure 6 (a) shows, when there is no input optical signal, both the first semiconductor optical amplifier and the second semiconductor optical amplifier operate in the saturation region, and the output optical power is in the high-level state; as Figure 6 (b) shows, when the intensity of the input optical signal is very high, both the first semiconductor optical amplifier and the second semiconductor optical amplifier operate in the transparent state region, and the output optical power is in the equivalent low-level state; as Figure 6 (c) shows, when the input optical signal is at a relatively low power, the working points of the first semiconductor optical amplifier and the second semiconductor optical amplifier are between the high-level state and the equivalent low-level state, and the gain effect is also between the two. Therefore, the output optical power shows a relatively sensitive linear relationship with the power of the input optical signal. In summary, the Sigmoid function is realized, that is, the non-linear activation function is realized.
[0033] A suitable continuous-wave beam is injected from the first laser (LD1) into the first semiconductor optical amplifier (SOA1), causing the first semiconductor optical amplifier to operate at different gain levels slightly above the transparency region. At the same time, the continuous laser input signal of the second laser (LD2) forces the gain of the second semiconductor optical amplifier (SOA2) to approach the end point of its transparency region. Therefore, the differential gain between the two semiconductor optical amplifiers corresponds to the phase shift between the two semiconductor optical amplifier branches. Using this biasing scheme, the pulsed optical power output after the input optical signal with strong peak power passes through this structure is maintained at the same high level (logically '1', and the specific power value varies with different systems), and the pulsed optical power output after the input optical signal with low peak power passes through this structure is maintained at the same low level (logically '0', and the specific power value varies with different systems). The optical signals with intermediate peak powers are mapped one by one to achieve the sigmod activation function, that is, to map the input optical signal variable to a value between 0 and 1.
[0034] The linear region array includes a plurality of linear operation regions, and each linear operation region is a weight coupling structure.
[0035] The output region is an end-face coupling structure. The output region is used to adjust the optical field mode and then output the optical information for subsequent coupling and reception.
[0036] A working flow chart of a sensing and computing integrated system is as Figure 1 shown.
[0037] The intensity mask is used to perform a convolution operation on the input first optical information to obtain second optical information and transmit it to the photonic integration chip; the first optical information is the optical information radiated by an object in the physical world; the photonic integration chip is used to multiply the second optical information by the activation function using the grating coupler in the non-linear operation region to complete a non-linear operation and obtain a third optical signal; then the third optical information is coupled into the linear operation region, and the linear operation region performs a large-scale matrix linear multiplication and addition operation on the third optical information to obtain a fourth optical signal; the fourth optical signal is output to the air medium through the output region and then converted into a first electrical signal; the digital signal processing system is used to receive the first electrical signal and obtain an image result after digital signal processing.
[0038] The structural diagram of the linear operation region is as Figure 5 shown. The input optical signal vector is input from the left waveguide, and part of the light changes its propagation direction and propagates downward through the micro-ring structure. At the same time, the voltage value of the micro-ring is adjusted to assign different weights to the optical signal. Finally, all the signals will be accumulated in the output region to complete a multiplication and addition of the vector and output a value. Each output region outputs a value, forming a vector that is output to the subsequent digital signal processing system to obtain an image result.
[0039] Intensity mask: The convolution process is implemented through a special intensity mask. A convolution kernel matrix composed of different convolution kernels is designed on the mask. Each convolution kernel is composed of a small matrix of NxN. The smallest unit 1x1 on the small matrix is randomly air or medium to achieve the transmission or blocking of light. When the scene light field passes through the mask, it performs an optical convolution with the convolution kernel to achieve the extraction of scene information.
[0040] Nonlinear region array: This region array contains multiple nonlinear operation regions. Each nonlinear operation structure includes nonlinear optical elements such as grating couplers, lasers, and semiconductor amplifiers. The semiconductor amplifier-interferometer structure composed of a laser and a semiconductor amplifier can implement the function of an activation function and perform nonlinear operations on the passing optical signals. By changing the optical power injected into the two arms of the semiconductor optical amplifier, the saturation state of the semiconductor optical amplifier is adjusted, thereby adjusting the threshold of the nonlinear activation function. Using the deep saturation characteristic of the semiconductor optical amplifier, the amplitude of the optical pulse signal is compared with the nonlinear activation threshold, and a pulsed optical signal of 0 or 1 with an equalized amplitude is output, thus realizing the function of the activation function.
[0041] Linear region array: An array region for performing multiplication and addition operations. The optically processed information after nonlinear processing is subjected to multiplication and addition operations. Each optical signal after multiplication with the activation function will enter the linear region array respectively, and at the same time pass through the first group of microring modulators. While passing through, some optical signals are given different weight information by the microring modulators and transmitted into the waveguides in the north-south direction. The remaining optical signals will continue to pass through the second group of microring modulators along the east-west waveguides and repeat the above process.
[0042] Output region: The waveguides in the output region array are made with a spot size conversion structure, which is convenient for the optical signal when propagating into the air medium. Due to the different refractive indices of the chip and air, the divergence angle is too large, which is beneficial for subsequent coupling and receiving of the optical signal and reducing losses. The optical signal output by the Nth output waveguide is the sum of the optical signals after each nonlinear optical signal is multiplied by the Nth group of microring modulators.
[0043] The above are only the preferred embodiments of the present invention and are not intended to limit the present invention. Any modifications, equivalent replacements, improvements, etc. made within the spirit and principle of the present invention shall be included within the scope of protection of the present invention.
Claims
1. A sensing and computing integrated system, characterized in that, The system includes: an intensity mask, a photonic integrated chip, and a digital signal processing system; the photonic integrated chip includes a non-linear region array, a linear region array, and an output region.
2. The sensor-computation integrated system according to claim 1, wherein The intensity mask is provided with different convolution kernels to form a convolution kernel matrix, and each convolution kernel is composed of a small matrix of N×N, where 1≤N≤3; the small matrices are air or dielectric respectively.
3. The sensing and computing integrated system according to claim 1, characterized in that, The non-linear region array includes multiple non-linear operation regions, and each non-linear operation region includes a grating coupler, a first laser, a second laser, a first semiconductor optical amplifier, a second semiconductor optical amplifier, and a third semiconductor optical amplifier.
4. The sensing and computing integrated system according to claim 3, wherein, The output end of the grating coupler is connected to the input end of a first 1×2 multimode interference coupler; the output ends of the first 1×2 multimode interference coupler are respectively connected to the input end of a first 2×1 multimode interference coupler and the input end of the first semiconductor optical amplifier; the output end of the first laser is respectively connected to the input end of the first semiconductor optical amplifier and the input end of the second semiconductor optical amplifier; the output end of the second laser is connected to the input end of the second semiconductor optical amplifier; the output end of the first semiconductor optical amplifier is respectively connected to the input end of the first 2×1 multimode interference coupler and the input end of a second 2×1 multimode interference coupler; the output end of the second semiconductor optical amplifier is connected to the input end of the second 2×1 multimode interference coupler; the output ends of the first 2×1 multimode interference coupler and the second 2×1 multimode interference coupler are respectively connected to the third semiconductor optical amplifier.
5. The sensing and computing integrated system according to claim 1, characterized in that, The linear region array includes multiple linear operation regions, and each linear operation region is a weight coupling structure.
6. The computing-in-sensing system according to claim 1, wherein The output region is an end-face coupling structure.
7. The sensing and computing integrated system according to claim 6, wherein The output region is used to output optical information after adjusting the optical field mode for subsequent coupling and reception.
8. The sensor-computing integrated system according to claim 3, wherein The intensity mask is used to perform a convolution operation on the input first optical information to obtain second optical information and transmit it to the photonic integrated chip; the first optical information is the optical information radiated by an object in the physical world. The photonic integrated chip is used to multiply the second optical information by an activation function using the grating coupler in the non-linear operation region to complete a non-linear operation and obtain a third optical signal; then the third optical information is coupled into the linear operation region, and the linear operation region performs a large-scale matrix linear multiplication and addition operation on the third optical information to obtain a fourth optical signal; the fourth optical signal is output into the air medium through the output region and then converted into a first electrical signal. The digital signal processing system is used to receive the first electrical signal and obtain an image result after digital signal processing.
9. The sensing and computing integrated system according to claim 8, characterized in that, The first laser and the second laser are used to control the operating state points of the first semiconductor optical amplifier and the second semiconductor optical amplifier: when there is no input optical signal, both the first semiconductor optical amplifier and the second semiconductor optical amplifier operate in the saturation region, and the output optical power is in the high-level state; when the intensity of the input optical signal is very high, both the first semiconductor optical amplifier and the second semiconductor optical amplifier operate in the transparent state region, and the output optical power is in the equivalent low-level state; when the input optical signal is at a lower power, the operating points of the first semiconductor optical amplifier and the second semiconductor optical amplifier are between the high-level state and the equivalent low-level state, and the gain effect is also between the two. Therefore, the output optical power shows a relatively sensitive linear relationship with the power of the input optical signal.