Device and method for optically processing signals

Abstract

The invention relates to a device (100) for optically processing signals, comprising: - at least one image-capturing apparatus (10) for generating at least one image of an environment (200), wherein the at least one image has a plurality of pixels and at least one analog pixel signal (1) can be generated for each pixel; - means (71) for digitising the at least one analog pixel signal (1) into at least one digital signal (6); - at least one light-generating apparatus (30) for generating at least one optical signal (2); - at least one modulator (40) for modulating the at least one optical signal (2) on the basis of the at least one digital signal (6); - at least one optical processor (50) for processing the at least one modulated optical signal (3), wherein at least one processed optical signal (4) can be generated by means of the processing, wherein the at least one processed optical signal (4) represents at least one result value of the signal processing. The invention also relates to a method for optically processing signals.

Description

[0001] Description

[0002] Device and method for optical signal processing

[0003] The invention relates to a device and a method for optical signal processing.

[0004] For safe environmental analysis, e.g., for the purpose of automated driving, fast data processing is essential. In the automotive sector, environmental information is acquired using sensors, for example, with the help of radar, lidar, and camera systems. Neural networks, for example, are used to process the data acquired by sensors. These typically offer the option of processing camera images or data from other sensors at a processing rate of approximately 10 Hz. However, such a processing rate is insufficient for safe environmental analysis. In particular, the available frame rate of an image acquisition device can significantly exceed the achievable processing rate. Furthermore, the data load increases with increasing resolution of the camera images. A higher data load, in turn, has a negative impact on the achievable speed and accuracy of the processing or environmental analysis.

[0005] An artificial neural network is typically implemented in software, and the associated calculations are performed using a digital processor, for example. However, fast calculations are limited, in particular, by the processor or GPU speed. Furthermore, data processing, especially using software-implemented neural networks, is very energy-intensive.

[0006] In addition, data processing systems are known from the prior art in which a neural network is implemented on the hardware side, e.g. in the form of an optical neural network.

[0007] DE 102020 117 882 A1 discloses a device that integrates processing and scanning components into a single optical path. DE 102019 107 101 A1 discloses a method and device for determining the distance of objects.

[0008] WO 90 / 0 9641 A1 discloses an optical neural network.

[0009] However, the state of the art has the disadvantage that processing of detected environmental information is energy-intensive and does not allow for sufficient speed and accuracy.

[0010] The technical problem is therefore to create a device and a method for optical signal processing which enables energy-efficient processing of

[0011] Enable environmental information with sufficient speed and accuracy.

[0012] The solution to the technical problem is achieved by the subject matter having the features of the independent claims. Further advantageous embodiments of the invention are set forth in the subclaims.

[0013] A device for optical signal processing is proposed, comprising: at least one image capture device for generating at least one image of an environment, wherein the at least one image has a plurality of pixels and at least one analog pixel signal can be generated for each pixel,

[0014] Means for digitizing the at least one analog pixel signal into at least one digital signal, at least one light generating device for generating at least one optical signal, at least one modulator for modulating the at least one optical signal as a function of the at least one digital signal, at least one optical processor for processing the at least one modulated optical signal, wherein at least one processed optical signal can be generated by the processing, wherein the at least one processed optical signal represents at least one result value of the signal processing.

[0015] Furthermore, a method for optical signal processing is proposed, comprising the steps of: generating at least one image of an environment by means of at least one image capture device, wherein the image has a plurality of pixels and at least one analog pixel signal is generated for each pixel, digitizing the at least one analog pixel signal into at least one digital signal by means of a digitizing means,

[0016] Generating at least one optical signal by means of at least one light generating device,

[0017] Modulating the at least one optical signal as a function of the at least one digital signal by means of at least one modulator,

[0018] - Processing the at least one modulated optical signal by means of at least one optical processor, wherein the processing generates at least one processed optical signal, wherein the at least one processed optical signal represents at least one result value of the signal processing.

[0019] In particular, the same technical effects and advantages apply to the proposed method as are described with reference to the device in this disclosure and vice versa.

[0020] The proposed device and the proposed method are particularly designed for use in a vehicle.

[0021] A basic idea of ​​the invention is to process properties of analog and optical signals in such a way that the processed properties can represent a result value, such as a matrix entry of a matrix to be determined. Due to the effects and advantages of the invention explained below, a faster and more accurate matrix determination can be achieved. A matrix determined in this way can, in particular, contain environmental information that is required, for example, for the purpose of automated driving. This improves the speed and accuracy of processing environmental information, which in turn improves automated driving.

[0022] The image capture device can be designed, for example, as a photo or video camera, radar capture device, or light detection and ranging (LIDAR) capture device. The image capture device can comprise an image sensor for generating an image of an environment. The proposed method can thus be used to create an environment model. The image sensor can, for example, have N x M cells, wherein at least one pixel of the image can be assigned to each cell. A cell can also be referred to as a potential well. At least one analog pixel signal can be assigned to each matrix entry of the N x M pixel matrix formed in this way. The image sensor can, for example, be designed as a CCD sensor or CMOS sensor. For the purposes of this invention, a distinction is made between an analog signal and a digital signal, which has only discrete values, and an analog signal, which does not only have discrete values.

[0023] To generate the image, light from the environment can, for example, enter the image capture device. When light falls, at least one analog pixel signal can be generated, for example, in one or more cells of the image sensor. The analog pixel signal can, for example, be generated due to the photoelectric effect, whereby the incident light transfers its energy, for example, to the irradiated cell of the image sensor. In this way, for example, a voltage signal can be generated in a cell, which can correspond to the analog pixel signal. The analog pixel signal can therefore be an electrical signal, e.g. a voltage signal or a potential.

[0024] The analog pixel signal can have a property such as an amplitude that represents an optical property of the incident light. For example, the amplitude of the analog pixel signal can be proportional to an intensity of the incident light in a cell of the image sensor. The generated analog pixel signal can, for example, be stored in analog form in at least one memory cell of the image capture device. The at least one memory cell can, for example, be designed as a cell of the image sensor. In this case, the image sensor also represents a memory device for the analog signals. However, it is of course also possible for the image capture device to comprise a memory device for the analog pixel signals that is different from the image sensor. This memory device can comprise the at least one memory cell. In particular, the image capture device comprises at least one memory cell for each pixel of the image.A timestamp can also be assigned to each generated analog pixel signal. This allows for a chronological assignment of the generated pixel signals and / or the image. This facilitates processing.

[0025] The image can be composed of several pixels, e.g. N x M pixels, where N can be a number of pixels in a horizontal direction of the image and M can be a number of pixels in a vertical direction of the image. An Nth or Mth pixel can, for example, be assigned to an Nth or Mth cell of the image sensor. In particular, the image can have a resolution of at least 8 megapixels. The image can, in particular, depict an environment two-dimensionally, ie the image can be a two-dimensional image of the environment. This has the advantage that the image capture device can, for example, be designed as a camera and can therefore be cost-effective. The generated image can, for example, be composed of the analog pixel signals generated for each pixel.

[0026] The digitizing means can be an analog-to-digital converter. Furthermore, the digitizing means can comprise a readout device and / or one or more data lines. In order to digitize and / or read out the analog pixel signal, for example, a cell of the image sensor or the at least one memory cell of the image capture device can be controlled by means of the readout device and the analog pixel signal can be transmitted, for example, via a data line to the analog-to-digital converter. By controlling it, the readout device can, for example, establish an electrical connection to the cell or memory cell in order to tap off the cell-specific voltage / cell-specific potential. In particular, each pixel of the image or each memory cell of the image capture device can be specifically controlled. This has the advantage that the analog pixel signal can be read out and digitized for each pixel. Several or all pixels of the image sensor orMemory cells can be read and digitized simultaneously. This improves the speed of processing environmental information. The digital signal can represent the analog pixel signal in digital form. The digitized signal can be forwarded to the modulator. By digitizing, the device can, in particular, use a conventional image capture device, which typically has a means for digitizing an analog pixel signal.

[0027] The light generating device can, for example, comprise a laser or a super-emitter or be designed as such. The light generating device can comprise a control unit for controlling the generated optical signal. The control unit can, for example, comprise a photodiode, with which the control unit can determine an optical property of the generated optical signal. The determined optical property can be compared with a previously known value of the optical property. The light generating device can, for example, have a signal generator for adjusting a property of the optical signal, wherein the signal generator can adjust, in particular, an intensity, pulse width, pulse energy, frequency, phase, or amplitude of the optical signal as a property. By means of the control unit, for exampleensure that the optical signal does not exceed a previously known intensity value and, if necessary, adjust the actual value of an optical property of the optical signal to a desired value of an optical property, e.g. using the signal generator. In this way, for example, the quality of the generated optical signal can be checked and / or the optical processor can be protected from damage, e.g. due to excessive intensity. The light generating device can be connected to the modulator in terms of signal technology, for example via one or more conductor tracks for conducting the optical signal. In particular, the light generating device is not used to detect environmental information. This allows the energy requirement to be limited particularly effectively. The light generating device can also generate several optical signals in parallel, in particular simultaneously.These can have the same or different wavelengths. For example, optical signals can be generated with non-interfering wavelengths. Simultaneously generated signals with different wavelengths can also be referred to as channel-specific signals.

[0028] The optical signal can, for example, be superradiation and / or a pulsed beam. In particular, the optical signal can be coherent and / or non-coherent light. The optical signal is, in particular, modulatable, i.e., an optical property of the optical signal can be modified such that the optical signal becomes the carrier of at least one piece of information. This property can become a property of the read-out analog pixel signal or the digital signal.

[0029] The modulator may, for example, comprise or be configured as at least one Mach-Zehnder interferometer, and / or the modulator may, for example, comprise or be configured as an electroabsorption modulator. In particular, the modulator may comprise a plurality of modulators connected (optically) in parallel.

[0030] The optical signal can be modulated by the modulator such that an optical property of the optical signal represents at least one property of the analog pixel signal or digital signal. The modulation occurs as a function of the at least one digital signal. In particular, the digital signal can be converted into an analog signal by means of a digital-to-analog converter, with the modulation then occurring as a function of the digital signal converted into an analog signal.

[0031] For modulation, the modulator receives, for example, the at least one digital signal or converted analog signal and transmits, for example by means of electro-optical elements, at least one property of the signal, such as a (scaled or converted) amplitude value, to the at least one optical signal. In this way, the at least one optical signal becomes an information carrier of the analog pixel signal or the digital signal. The modulator or the parallel connection of modulators can, in particular, be digitally and / or optically multiplexable. The digital multiplexing capability of the modulator means that, for example, several digital signals can be received in parallel. The optical multiplexing capability of the modulator means that, for example, several optical signals can be modulated in parallel, e.g. depending on one or more analog pixel signals.If the light-generating device generates multiple optical signals, these can each be radiated to different modulators in the parallel circuit. This makes it possible for modulation to occur depending on multiple digital signals, especially in parallel.

[0032] By means of the readout means and / or the at least one modulator, one or more input signals can be provided for the optical processor as an input signal, but preferably a plurality of input signals. A plurality of input signals can be coupled simultaneously into optical input interfaces of the optical processor. In this way, in particular, an input signal vector with kx 1 input signals can be provided, which is then coupled into the optical processor. If, for example, a submatrix with uxv analog pixel signals is read out from the image using the readout means, the entries of this submatrix can form the entries of a kx 1 input vector. Here, u can be less than N and v can be less than M and can each be, for example, 3 or 9. In this way, image processing, for example smoothing, based on a kernel with uxv analog pixel signals can be carried out by the optical processor.

[0033] The optical processor can be designed, for example, as a semiconductor chip. The optical processor can have, for example, silicon dioxide, gallium arsenide and / or indium phosphide as material. The optical processor can in particular be integrated into a CMOS, SiN-CMOS, Bi-CMOS, or hybrid Bi-CMOS chip. This reduces the space required by the device. The optical processor can have conductor tracks, e.g., waveguides, for conducting the optical signal. The optical processor can have at least one attenuation element for attenuating an optical property of the optical signal and / or at least one coupling element for splitting the optical signal or for combining several optical signals. The optical processor can be designed to carry out arithmetic operations, such as multiplying or adding, for the purpose of signal processing. To carry out arithmetic operations, an optical signal can, for example, be fed into a coupling element, such asa multi-mode waveguide, wherein the optical signal is split in the coupling element, for example, into a first processed optical signal and a further processed optical signal, wherein the splitting can correspond to a computing operation. By means of the computing operation carried out, for example, a matrix entry of a matrix to be determined by means of the optical signal processing can be determined. A matrix entry determined in this way can, for example, correspond to the result value. In this way, matrices can, for example, be determined by means of the optical processor. The optical processor can, for example, comprise an optical neural network (ONN) or be designed as such. The optical processor can, in particular, be multiplex-capable. Due to the multiplex capability of the optical processor, for example, several optical signals can be processed in parallel. This increases the processing speed.

[0034] The ONN may be implemented as a deep learning neural network, such as a convolutional neural network, a recursive neural network, or another form of artificial neural network. The ONN may comprise a network of Mach-Zehnder interferometers, free-space elements, grating couplers, optical fiber bundles, and / or other optical components, which may be arranged in or form various layers of the ONN. In particular, the ONN relies on optical processing techniques to process the coupled light. The ONN may comprise at least one input, preferably a plurality of inputs, into which the optical signal(s) are coupled. The ONN may comprise first gratings that receive the light from the input(s) and control a direction of the light within the optical neural network with respect to at least one subsequent layer.The ONN may include couplers that receive the light from the first gratings and selectively transmit the light. The ONN may also include phase-shifting elements. The couplers and phase shifters may be arranged in alternating layers to process the light. The ONN may further include at least second gratings that receive the light from one of the alternating layers and provide the light in a processed form as an output embodying a result, in particular as a result value.

[0035] The ONN may also comprise optical waveguides, e.g., silicon photonic waveguides, arranged or formed in layers, in particular in layers of couplers and phase shifters, and in particular may form a network of Mach-Zehnder interferometers. A coupler layer may, for example, comprise coupled optical waveguides formed separately from one another and arranged parallel over a predetermined length with a spacing smaller than a predetermined coupling distance. This may serve to couple the light propagated in the optical waveguides, but also to split the light between subsequent branches. Thus, the couplers may also serve as beam splitters to form the network of Mach-Zehnder interferometers.

[0036] The processing of the modulated optical signal comprises, in particular, a passive change in an optical property of the modulated optical signal, e.g., by means of a material property of an attenuation element. Such a material property can be, for example, a degree of attenuation of the attenuation element. The attenuation therefore requires no additional energy. In particular, a layer of the optical processor can be designed as an attenuation element or comprise one or more attenuation elements. The attenuation element can, for example, correspond to a neuron of a neural network. By means of the attenuation element, for example, an intensity of the modulated optical signal can be attenuated as it passes through the attenuation element. The attenuation of the intensity of the optical signal can correspond to a passive change in an optical property of the modulated optical signal. The attenuated optical signal can correspond to a processed optical signal and, for example,have a damped intensity compared to the modulated optical signal. The damping factor of the damping element can be predetermined or configured, so that, based on the known damping factor, the processed optical signal can be specifically generated from the modulated optical signal. In this way, a predetermined weighting can be implemented in a neuron of a neural network, for example.

[0037] Processing the optical signal using an optical processor has the advantage that the propagation speed of the optical signal in the device is almost equal to the speed of light, thus significantly increasing the processing speed compared to conventional electronic or digital signal processing, for example, on a graphics processor. Furthermore, processing using an optical processor is significantly more energy-efficient than conventional digital signal processing, for example, on a graphics processor, because the optical signal can be processed passively and no active supply of energy is required for processing.

[0038] The result value can, for example, be represented by an optical property of the processed optical signal. An overall result of the optical signal processing can be generated from several result values. The result value can, for example, be a determined matrix entry of a matrix to be determined, whereby the overall result can correspond to the matrix to be determined. The optical property representing the result value can, for example, be an amplitude value of the processed optical signal or be represented by it. It is possible for a plurality of processed optical signals to be generated from one modulated optical signal, whereby each processed optical signal can represent an independent result value. In this way, for example, the resolution of the overall result can be increased.It is also possible for a processed optical signal to be generated from a plurality of, in particular parallel, modulated optical signals, wherein the processed optical signal can represent an independent result value for the plurality of modulated optical signals. In this way, for example, the information density of a single result value can be increased. This improves processing accuracy.

[0039] The proposed device and the proposed method thus advantageously provide energy-efficient signal processing with improved processing speed and accuracy.

[0040] Further advantages, features, and aspects of the invention will become apparent from the following description. The features mentioned in the claims and in the description can, in principle, be combined with one another in any desired manner.

[0041] In one embodiment, the device further comprises at least one optical memory for storing the at least one result value, wherein the at least one result value can be stored in the optical memory in the form of the at least one processed optical signal. The storage option improves user-friendliness. For example, several result values ​​can also be stored in the optical memory. In this way, the stored result values ​​can be combined sequentially, i.e., step by step, to form an overall result. The optical memory can, for example, be designed as a CMOS-based memory or comprise such a memory. Due to the optical storage, the optical properties are retained, i.e., there is no loss of information during storage. This increases the accuracy of the stored result value. The optical memory can, for example,one or more cells for storing one or more processed optical signals.

[0042] In one embodiment, the device further comprises at least one digital memory for storing the at least one result value, wherein the at least one optical signal representing the at least one result value can be converted into at least one digital result signal by means of at least one optical-digital converter, wherein the at least one digital result signal can be stored in the digital memory. The storage option improves user-friendliness. For example, a plurality of result values ​​can be stored in the digital memory. In this way, the stored result values ​​can be combined sequentially, i.e. step by step, to form an overall result. The digital memory can, for example, be designed as a RAM-based memory or comprise such a memory. The optical-digital converter can, for example, be designed as at least one photodiode or comprise such a memory.The signal from the photodiode can be digitized using the optical-to-digital converter. For each processed optical signal generated during processing, at least one digital result signal can be generated by means of the optical-to-digital converter. In particular, the optical-to-digital converter can have at least one photodiode for each processed optical signal generated. For example, the optical processor can output at least two, for example exactly four, processed optical signals, each representing a result value. In this case, the optical-to-digital converter can have at least two, for example four, photodiodes to convert each processed optical signal into an independent result signal. The overall result can, for example, be composed of several result signals. The digital memory can, for example, have one or more cells for storing the result signal.The digital memory can be 8-bit, 12-bit, 14-bit, 16-bit, or 32-bit capable, for example, to store the result signal with the appropriate color depth. This increases the user-friendliness and accuracy of optical signal processing. The result signal can represent the result value.

[0043] The digital result signal can be forwarded to a computing unit for further signal processing. The computing unit can be designed, for example, as a digital processor. The device can include the computing unit. The computing unit can be, for example, a digital processor.

[0044] In one embodiment, the device has means for reading out and / or transmitting the at least one result value, wherein the at least one modulator is additionally designed to modulate the optical signal generated by the light generating device as a function of the at least one result value. In this way, sequential signal processing can be implemented. The means for reading out and / or transmitting the result value can comprise a readout device and / or one or more data lines. To read out and / or transmit the result value, for example, a cell of the previously explained optical and / or digital memory can be controlled by means of the readout device and, for example, the stored result signal can be read out and transmitted to the at least one modulator via a data line. In particular, each cell of the optical or digital memory can be controlled specifically.For example, the previously explained analog result signal can be fed to the modulator as an input signal. The result signal can replace the digital signal as the input signal for the modulator. The optical signal can be modulated as a function of the result signal. For example, in a first signal processing sequence, a first optical signal can be processed in such a way that modulation with the analog pixel signal(s) generates a signal. Such a first result signal can, for example, be a smoothed digital signal. In a further sequence, a further optical signal can be modulated as a function of the first result signal, the first result signal being represented in the further modulated optical signal as a result of the modulation. Furthermore, the further modulated optical signal can be processed in the further sequence in such a way that a further result value or a further result signal is determined.In other words, the digital signal can be filtered sequentially, for example, through sequential processing. It is also possible for the processed optical signal to be converted into a digital signal, for example, using an optical-to-digital converter, and transmitted directly to the modulator, i.e., without memory.

[0045] In one embodiment, the optical signal generated by the light-generating device is embodied as a non-coherent optical signal. This increases cost efficiency. Non-coherent light is, in particular, more cost-effective to generate than, for example, coherent light. The non-coherent optical signal can be generated, for example, using the previously explained super-emitter.

[0046] In one embodiment, the at least one optical processor is designed to be reconfigurable. By reconfiguring the optical processor, in particular a type of processing can be changed. If processing takes place according to a first way before reconfiguration, processing after the reconfiguration can take place in a further way that is different from the first way. This improves applicability and user-friendliness. The reconfiguration can comprise at least one property of the optical processor being adaptable, for example to adapt a weighting of the neural network. For example, a material property of the optical processor or of a part of the processor can be adapted such that the optical signal is processed in a different way than before reconfiguration. The reconfiguration of the optical processor can take place by means of a control device.The device can comprise the control device. The control device can, for example, be designed as a microcontroller or comprise such a microcontroller. The control device can, for example, generate a control signal which specifically adapts an attenuation property of the optical processor or of a part of the processor. For example, an attenuation property of a phase change material can be adjusted by a control signal designed as an optical signal. An attenuation property of an electroabsorption modulator can also be adjusted by a control signal designed as a voltage signal. In these embodiments, the optical processor can therefore comprise phase change materials and / or electroabsorption modulators, each of which can effect controlled signal attenuation. For example, cells or attenuation elements of the optical processor can be formed by such phase change materials or electroabsorption modulators.In this way, the optical neural network can be reconfigured.

[0047] In one embodiment, the at least one optical processor has at least one attenuation element, wherein at least one degree of attenuation of the at least one attenuation element is adjustable, in particular for reconfiguring the optical processor. The attenuation element can be used to attenuate the intensity of an optical signal. By adjusting the degree of attenuation of the optical processor, the type of processing can be changed, in particular. This improves applicability and user-friendliness. The attenuation element can, for example, be formed from the phase-change material described above or also be designed as an electroabsorption modulator. The attenuation degree of the at least one attenuation element can be adjusted, for example, using the control device explained above.

[0048] In one embodiment, the at least one result value represents a result of image processing and / or classification. This can eliminate the need to perform image processing or classification of the image using conventional neural networks. This improves the efficiency of the processing. The image processing can, for example, determine an intensity value for one or more analog pixel signals. Thus, in particular, known image processing methods such as smoothing or segmentation can be implemented for the image. The classification can assign a specific class, for example an object from a previously known set of objects, to one or more analog pixel signals. The objects can, for example, be areas or objects depicted in the image, such as a road, a vehicle, or a passerby. The classification can, in particular, make it possible to use known classification methods such asobject recognition can be implemented. In one embodiment, the device is designed to generate at least one control command for the automated control of vehicle operation, in particular for assisted, partially automated, highly automated, fully automated or autonomous operation of the vehicle, depending on the at least one result value. For example, the result value can be generated during preprocessing of a vehicle environment detection, with control commands for the vehicle then being generated depending on the results of the preprocessing. Such preprocessing can, for example, be the detection of static or dynamic objects, e.g. other road users, in the environment of the vehicle.

[0049] Due to the improved processing of environmental information by means of the optical processor, the generated control command increases safety during automated control of a vehicle. The control command can be generated, for example, using the previously explained computing unit. In particular, a generation rate of the at least one control command can correspond to an image generation rate of the at least one image capture device. In particular, the device can be configured to generate at least thirty control commands per second. The control command can be, for example, a control command for steering, accelerating, and / or braking the vehicle.

[0050] A vehicle with a device according to one of the embodiments described in this disclosure is further described. The image capture device can be an environment capture device of the vehicle.

[0051] A method is further described which, after generating at least one result value, comprises the steps:

[0052] Generating at least one further optical signal by means of at least one light generating device,

[0053] Modulating the at least one optical signal as a function of the at least one result value by means of at least one modulator,

[0054] - Processing the at least one modulated optical signal by means of at least one optical processor, wherein the processing generates at least one further processed optical signal, wherein the at least one further processed optical signal represents at least one result value of the signal processing. If the at least one result value is stored, the method can comprise a step of reading out the at least one analog pixel signal using the means for reading out or a further means for reading out. Furthermore, the method can comprise reconfiguring the optical processor after generating the result value and before processing the modulated optical signal, which was generated by modulating the at least one optical signal as a function of the at least one result value.

[0055] The proposed method and the proposed device advantageously enable in particular: i. an increase in the signal processing speed, ii. signal processing with the functionality of deep optical neural networks, iii. signal processing at the speed of light, iv. signal processing with low optical power, in particular to carry out non-linear response functions, since furthermore in particular no multi-photon processes are involved and only material properties are used, v. direct reading of optical, analog signals for signal processing, vi. direct storage of optical, analog signals, vii. elimination of digitization and thus a saving of components, viii. an increase in the accuracy of the signal processing by eliminating the

[0056] Discretization, ix. space savings, since an entire ONN can be physically manufactured on a semiconductor, x. flexible manufacturing, since the proposed embodiments can be integrated on

[0057] Semiconductor chips with CMOS, SiN-CMOS, Bi-CMOS, hybrid Bi-CMOS processors based on, for example, photonic-electronic co-integrated chips.

[0058] The invention is explained in more detail using exemplary embodiments. The figures show:

[0059] Fig. 1 is a schematic block diagram of a device according to the invention,

[0060] Fig. 2 is a schematic block diagram of a device according to the invention in a further embodiment, and

[0061] Fig. 3 shows a schematic representation of an embodiment of a method. In the following, like reference numerals designate elements with the same or similar technical features.

[0062] Fig. 1 shows a schematic representation of an embodiment of a device 100 according to the invention. The device 100 comprises an image capture device 10 designed as a video camera. The image capture device 10 generates at least one image of an environment 200. In particular, the image capture device 10 can generate at least thirty images per second. The image capture device 10 comprises an image sensor 11 designed as a CMOS chip for generating the image. The image sensor 11 has a plurality of cells 12 (for the sake of clarity, only one cell 12 is identified by a reference numeral in Figs. 1 and 2). For the sake of clarity, three cells 12 are shown by way of example in Figs. 1 and 2. Each cell 12 of the image sensor 11 can be assigned to a pixel of the image. The image can, for example, have a resolution of approximately 8 megapixels, whereby the image sensor 11 can accordingly have 8 million cells.

[0063] If, for example, light (represented by wavy arrows in Figs. 1 and 2) from the environment 200 strikes the cells 12 of the image sensor 11, an analog pixel signal 1 (indicated by arrows in Figs. 1 and 2) is generated in each of the cells 12, e.g. due to the photoelectric effect. For the sake of clarity, only one analog pixel signal 1 is identified by a reference symbol in Figs. 1 and 2. A signal flow of analog signals is identified in Figs. 1 and 2 by solid lines or arrows. The analog pixel signal 1 can be a voltage signal and can represent, for example, a light intensity of the light falling on a cell 12. The generated analog pixel signals 1 can each be assigned to a pixel of the image, so that the image can be composed of the generated pixel signals 1.

[0064] For the purpose of optical signal processing, the cells 12 of the image sensor 11 can be individually read out and digitized with the aid of a digitizing means 71 designed as a readout device. By digitizing and reading out, the generated pixel signals 1 of an image can be forwarded, for example, via data lines to a modulator 40. By reading out, pixel signals can be provided in the form of a vector. This vector can, for example, comprise one data field per read-out cell 12. The modulator 40 can comprise a plurality, e.g. an array, of modulators, e.g. one modulator per data field of the read-out pixel vector. The digitizing and read-out can take place for several or all cells 12 in parallel, i.e. simultaneously. For this purpose, the means 71 can, for example, establish an electrical connection to several cells 12. The digital signal 6 can represent the analog pixel signal 1 in digital form.The device 100 further comprises a light generating device 30. The light generating device 30 has a light source 31 designed as a super-emitter, which generates a non-coherent optical signal 2. A signal flow of optical signals is indicated in Fig. 1 and 2 by dashed lines or arrows. The light generating device 30 can further have a signal generator 31 and a control unit 34 and a photodiode 33. By means of the signal generator 31, at least one optical property such as an intensity of the generated optical signal 2 can be set. By means of the photodiode 33, at least one optical property of the generated optical signal 2 can be determined. By means of the control unit 34, the determined optical property can be compared with a target value of the optical property and the optical signal 2 can be set to the target value by means of the signal generator 32.For this purpose, the control unit 34 can, for example, send a control signal 9 to the signal generator 31. This can ensure, for example, that an intensity value of the generated optical signal 2 is not greater than a previously known threshold value. It is further shown that the light-generating device 30 comprises a means 32 for changing polarization properties, e.g., a polarization filter, which adjusts the polarization properties of the generated light. However, this means 32 is an optional and not a mandatory component of the light-generating device 30.

[0065] A modulator 40 designed as a Mach-Zehnder interferometer, or a plurality of modulators, e.g., an array, modulates an optical signal 2 generated by a light-generating device 30 as a function of the digital signal 6. For this purpose, the signal generated by the light-generating device 30 can be divided, with each modulator receiving a partial signal. The light-generating device 30 can also generate multiple signals, with these signals then being radiated to different modulators.

[0066] In this way, for example, a property of the optical signal 2, such as an amplitude or intensity, can represent a corresponding property of the analog pixel signal 1 or the digital signal 6. The modulated optical signal 3 thus becomes the carrier of one or more pieces of information of the analog pixel signal 1 or the digital signal 6.

[0067] The modulated optical signal 3 is transmitted, for example, by means of a fiber optic cable (not shown) to an optical processor 50 and can be an input vector for the optical processor 50. The optical processor 50 comprises an optical neural network which, in Fig. 1 and 2, has, for example, three layers 51, 52, 53. Of course, the optical neural network can also have more or fewer than three layers 51, 52, 53. The optical neural network 50 is integrated into the optical processor 50 in terms of hardware. The layers 51, 52, 53 can each have three attenuation elements 54 formed from a phase change material (for the sake of clarity, only one attenuation element 54 is identified by a reference numeral). The attenuation elements 54 can, for example, represent neurons arranged in the layers 51, 52, 53.However, the number of layers 51, 52, 54 as well as the damping elements per layer 51, 52, 53 is purely exemplary and can be adapted depending on the application.

[0068] The modulated optical signal 3 is fed in its entirety to the attenuation elements 54 of the first layer 51 (input layer) and thus coupled into the optical processor. At least one optical property of the modulated optical signal 3 is passively modified by means of the attenuation elements 54. For example, the amplitude of the modulated optical signal 3 can be modified depending on the degree of attenuation of an attenuation element 54. The attenuation elements 54 of the first layer can have different degrees of attenuation. The modification of the optical property of the modulated optical signal 3 can correspond to processing of the modulated optical signal 3. In other words, the processing of the modulated optical signal 3 takes place depending on a material property of the optical processor 50 or a part or component thereof.This type of passive signal processing is more energy-efficient, especially compared to a software-implemented neural network, since no active processing of data is necessary, such as in a GPU.

[0069] Starting from the respective attenuation elements 54 of layer 51, a plurality of processed optical intermediate signals 4 can be split, for example, by means of coupling elements (not shown), and each transmitted to the attenuation elements 54 of a subsequent layer 52 (indicated by dashed arrows). Starting from the respective attenuation elements 54 of layer 52, further processed optical intermediate signals 4 can be split and transmitted to a further layer 53. Of course, it is also possible for two or more processed optical intermediate signals 4 to be combined, for example, by means of coupling elements. Starting from the last layer 53 (output layer), the finally processed optical signals 4 can be output by the optical processor 50 to an optical-to-analog converter 80.

[0070] Each of the output, processed optical signals 4 represents a result value of the optical neural network. For example, by processing the modulated optical signal 3 as described, an amplitude or other optical property of the processed optical signal 4 can be changed such that this changed amplitude (or other optical property) represents the result value. The result value can, for example, be a filtered amplitude value for the analog pixel signal 1.

[0071] The optical-to-digital converter 80 comprises a photodiode 81 for each processed optical signal 4, by means of which the respective processed optical signal 4 is converted into a digital result signal 5. The digital result signals 5 are identified by three arrows, whereby, for the sake of clarity, only one arrow is provided with a reference symbol. The conversion involves the converted digital result signal 5 also representing the result value, for example, in the form of a digital property. Furthermore, the optical-to-analog converter can also comprise an amplifier (not shown) that amplifies the analog signal(s).

[0072] The digital result signal 5 is transmitted, for example, via data lines to a digital memory 70. The digital memory 70 can be designed, for example, as a RAM and store each digital result signal 5 in a respective cell 74 (for the sake of clarity, only one cell 74 is identified by a reference symbol). The digital memory 70 can have a storage cell 74 for each transmitted digital result signal 5, for example, to store all digital result signals 5 that are generated for signal processing of an image. In other words, the digital memory 70 can be designed to store all result values ​​determined for an image.

[0073] An overall result can be compiled from the resulting values. For example, the resulting values ​​can be matrix entries of a matrix, with the overall result being a matrix composed of the matrix entries. The digital result signals 5 can be transmitted, for example, to a computing unit 72 embodied as a digital processor (represented by dash-dot arrows).

[0074] In the computing unit 72, the result signal 5 can be further processed, for example, in an artificial neural network (not shown). Depending on the result signal 5, a control command 8 for the automated control of a vehicle (not shown) can be generated. The control command 8 can be output, for example, to an actuating device (not shown) of the vehicle.

[0075] The result signal 5 can also be transmitted to a control device 73 designed as a microcontroller, which can comprise a CPU or GPU or be designed as such. This control device 73 can be used to create an environment model based on the processed optical signals 4. Depending on the result signal 5, for example, a control signal 9 for setting a degree of attenuation for at least one attenuation element 54 can be determined. For example, with the help of the control signal 9, a weighting to be set for processing with the optical neural network of the optical processor 50 can be set. The optical neural network can thus be reconfigured. The attenuation element 54 can, for example, have a phase-change material which, for example, changes an optical property as a result of the control signal 9, wherein the changed optical property corresponds to a set degree of attenuation.

[0076] Alternatively, the result signal can also be processed in the computing unit 72 (not shown). The processed digital signal can be transmitted to the control device 73. By processing the result signal 5 in the computing unit 72, the previously explained control signal 9, for example, can be determined even more precisely.

[0077] However, it is not mandatory for the control device 73 to generate the control signal 9 as a function of the digital signal 6. The control device 73 can also generate a control signal 9 for reconfiguring the optical neural network independently of the digital signal 9, e.g., after a predetermined period of time or when another criterion is met.

[0078] The device 100 further comprises a means 90 for reading out and transmitting the result signal 5. The means 90 can be designed as a readout unit and control the cells 74 with a readout signal 7 in order to read out one or more result signals 5, in particular in parallel. The means 90 can transmit the read-out result signal 5 to the modulator 40, where, for example, in a further processing sequence, an optical signal 2 is modulated as a function of the result signal 5. In this case, the result signal 5 replaces, for example, the digital signal 6. In this way, for example, different image processing methods can be implemented sequentially in several processing sequences.

[0079] Fig. 2 shows a schematic representation of a further embodiment of the device 100. In contrast to the embodiment shown in Fig. 1, in the further embodiment of the device 100, one or more optical signals 4 processed by means of an optical processor 50 are stored in an optical memory 60 designed as a CMOS chip. In this way, no conversion of the processed optical signals 4 has to take place to store the result values. It is possible for the device 100 to comprise an amplifier device (not shown) for amplifying the processed optical signals 4, in which case amplified processed optical signals 4 are then stored. The optical memory 60 has cells 61 for storing a plurality of processed optical signals 4.The cells 61 of the optical memory 60 can further be controlled by a means 90 for reading and transmitting a readout signal 7 such that the stored optical signal 4 is passed to an optical-to-digital converter 80 and can be converted into a digital result signal 5 with the aid of the optical-to-digital converter 80. The result signal 5 can be used, as already explained for Fig. 1, to modulate an optical signal 2 generated by a light generating device 30, for example, to implement sequential image processing.

[0080] Fig. 3 shows a schematic representation of an embodiment of a method for optical signal processing.

[0081] In a step S1, at least one image of an environment 200 is generated by means of at least one image capture device 10. The image has a plurality of pixels, and for each pixel, at least one analog pixel signal 1 is generated. The analog pixel signal 1 can be generated, for example, in a cell 12 of an image sensor 11 (see Fig. 1 and Fig. 2).

[0082] In a step S2, the at least one analog pixel signal 1 is read out using a digitizing means 71 and digitized into a digital pixel signal 6, e.g., from a cell 12 of an image sensor 11 (see Figs. 1 and 2). For this purpose, a readout signal 7 is generated by the digitizing means 71 and transmitted to the image sensor 11.

[0083] In a step S3, at least one optical signal 2 is generated by means of at least one light generating device 30. The optical signal 2 can, for example, be a non-coherent optical signal, which is generated by means of a light source 31 designed as a super-radiator and is transmitted via a fiber optic cable to at least one modulator 40 (see Figs. 1 and 2).

[0084] In a step S4, the at least one optical signal 2 is modulated by means of the modulator 40 as a function of the at least one digitized pixel signal 6. For this purpose, the digitized pixel signal 6 can be converted into an analog signal using a digital-to-analog converter (not shown). For example, an amplitude of the optical signal 2 can be modulated such that the amplitude of the modulated optical signal 3 represents an amplitude of the digital pixel signal 6 or the converted analog signal.

[0085] In a step S5, the at least one modulated optical signal 3 is processed by at least one optical processor 50. This processing generates at least one processed optical signal 4. The at least one processed optical signal 4 represents at least one result value of the signal processing. The result value can, for example, be a matrix entry of a matrix to be determined using the method.

[0086] List of reference symbols analog pixel signal optical signal modulated optical signal processed optical signal analog result signal digital (pixel) signal

[0087] Readout signal

[0088] control command

[0089] control signal

[0090] Image capture device

[0091] Image sensor

[0092] Cell of an image sensor

[0093] Means of reading

[0094] Light generating device

[0095] light source

[0096] Signal generator

[0097] Photodiode

[0098] control unit

[0099] Modulator optical processor

[0100] Layer of an optical neural network further layer of an optical neural network further layer of an optical neural network damping element optical memory

[0101] Optical storage cell digital storage

[0102] Means of digitization

[0103] Computing unit

[0104] Control device

[0105] Cell of the digital memory of optical-digital converters

[0106] Photodiode

[0107] Means for reading and / or transmitting

[0108] device

[0109] Vicinity

[0110] Step

[0111] Step

[0112] Step

[0113] Step

[0114] Step

Claims

Patent claims 1. Device (100) for optical signal processing, comprising: - at least one image capture device (10) for generating at least one image of an environment (200), wherein the at least one image has a plurality of pixels and at least one analog pixel signal (1) can be generated for each pixel, - means (71) for digitizing the at least one analog pixel signal (1) into at least one digital signal (6), - at least one light generating device (30) for generating at least one optical signal (2), - at least one modulator (40) for modulating the at least one optical signal (2) as a function of the at least one digital signal (6), - at least one optical processor (50) for processing the at least one modulated optical signal (3), wherein at least one processed optical signal (4) can be generated by the processing, wherein the at least one processed optical signal (4) represents at least one result value of the signal processing.

2. Device (100) according to claim 1, characterized in that the device (100) further comprises at least one optical memory (60) for storing the at least one result value, wherein the at least one result value can be stored in the optical memory (60) in the form of the at least one processed optical signal (4).

3. Device (100) according to claim 1 or 2, characterized in that the device (100) further comprises at least one digital memory (70) for storing the at least one result value, wherein the at least one optical signal (4) representing the at least one result value can be converted into at least one digital result signal (5) by means of at least one optical-digital converter (80), wherein the at least one digital result signal (5) can be stored in the digital memory (70).

4. Device (100) according to one of the preceding claims, characterized in that the device (100) has means (90) for digitizing the at least one result value, wherein the at least one modulator (40) is additionally designed to modulate the optical signal (2) generated by the light generating device (30) as a function of the at least one digitized result value.

5. Device (100) according to one of the preceding claims, characterized in that the optical signal (2) generated by means of the light generating device (30) is designed as a non-coherent optical signal.

6. Device (100) according to one of the preceding claims, characterized in that the at least one optical processor (50) is designed to be reconfigurable.

7. Device (100) according to one of the preceding claims, characterized in that the at least one optical processor (50) has at least one damping element (54), wherein at least one optical property of the at least one damping element (54) is adjustable.

8. Device (100) according to one of the preceding claims, characterized in that the at least one result value represents a result of image processing and / or classification.

9. Device (100) according to one of the preceding claims, characterized in that the device (100) is designed to generate at least one control command (8) for controlling vehicle operation as a function of the at least one result value.

10. A method for optical signal processing, comprising the steps: - generating (S1) at least one image of an environment (200) by means of at least one image capture device (10), wherein the image has a plurality of pixels and at least one analog pixel signal (1) is generated for each pixel, - digitizing (S2) the at least one analog pixel signal (1) into at least one digital signal (6) by means of a digitizing means (71), - generating (S3) at least one optical signal (2) by means of at least one light generating device (30), - modulating (S4) the at least one optical signal (2) as a function of the at least one digital signal (6) by means of at least one modulator (40), - Processing (S5) the at least one modulated optical signal (3) by means of at least one optical processor (50), wherein at least one processed optical signal (4) is generated by the processing (S5), wherein the at least one processed optical signal (4) represents at least one result value of the signal processing.

Images