A multi-modal physical neural network system and method based on gradient neurons

Abstract

This invention discloses a multimodal physical neural network system and method based on gradient neurons, relating to the field of multimodal information fusion technology. The system includes an input layer, a hidden layer, and an output layer. The input layer encodes information from at least three modalities as the number, frequency, and amplitude of electrical pulses, and synthesizes them into a single input electrical signal through time-division multiplexing. The hidden layer uses a two-stage semiconductor laser as its core. Under the sequential excitation of driving voltages encoding different modal information, the carriers in its gain region undergo nonlinear accumulation in two stages, thereby achieving superadditive fusion of multimodal information at the physical level and outputting a gradient pulse optical signal characterizing the fusion result. The output layer amplifies, converts, and processes the optical signal. This invention utilizes the physical dynamics characteristics of semiconductor lasers to directly simulate the superadditive fusion behavior of biological neurons at the hardware level, effectively improving the efficiency and quality of multimodal information fusion.

Description

Technical Field

[0001] This invention relates to the field of multimodal information fusion technology, and in particular to a multimodal physical neural network system and method based on gradient neurons. Background Technology

[0002] Multimodal information fusion technology aims to collaboratively process signals from different sensory channels (such as vision, hearing, and touch) to improve the accuracy, robustness, and decision-making efficiency of intelligent systems in understanding their environment. This technology has broad application prospects in fields such as autonomous driving, robot interaction, intelligent security, and medical diagnosis. In recent years, with the rapid development of artificial intelligence and edge computing, multimodal fusion has become one of the research hotspots in the fields of sensory systems and neuromorphic computing.

[0003] Currently, research on multimodal fusion mainly focuses on the integration and optimization at the algorithm level, such as extracting, aligning, and fusing features from different modalities using deep learning models. However, these methods typically rely on high-performance digital processing units and complex software architectures, resulting in problems such as high processing latency, high energy consumption, and strong hardware dependence. Especially in resource-constrained scenarios such as edge devices, it is often difficult to balance the real-time performance and energy efficiency of the algorithms.

[0004] In terms of hardware implementation, existing technologies mostly adopt the traditional architecture of sending sensor signals through analog-to-digital conversion to a central processing unit for post-processing, failing to fully utilize the dynamic characteristics of the physical devices themselves for front-end information fusion. Although some studies have attempted to achieve synchronous acquisition of multimodal signals through composite sensors such as photoelectric and piezoelectric sensors, the fusion process is still mainly based on simple linear superposition or weighted combination, lacking in-depth exploration of the nonlinear interactions and spatiotemporal correlations between cross-modal signals.

[0005] Biological nervous systems often exhibit a "hyperadditivity" effect when processing multimodal information, meaning that the intensity of the response generated by the synergistic interaction of multiple channels exceeds the sum of the independent responses of each channel. This effect stems from the plasticity of neurons and synapses and the interaction mechanisms between different sensory pathways, and is a crucial foundation for achieving efficient and adaptive information fusion. However, existing hardware systems struggle to physically simulate these biological characteristics, limiting their fusion capabilities and adaptability in dynamic environments.

[0006] In addition, traditional multimodal fusion systems are usually complex in structure, have many components, are difficult to integrate, and are limited by the inherent properties of optical and electronic devices, resulting in insufficient coding flexibility and difficulty in adapting to diverse information input and processing needs.

[0007] Therefore, how to design a multimodal information processing system that is simple in structure, highly energy-efficient, and capable of nonlinear fusion at the physical level and simulating biological hyperadditive behavior has become an urgent technical problem to be solved in this field. Summary of the Invention

[0008] To address this, embodiments of the present invention provide a multimodal physical neural network system and method based on gradient neurons, which solves the problems of complex structure, low energy efficiency, large processing delay, and inability to achieve nonlinear superadditive fusion at the physical hardware level in existing multimodal information fusion systems.

[0009] To address the aforementioned technical problems, embodiments of the present invention provide a multimodal physical neural network system based on gradient neurons, comprising:

[0010] An input layer is used to receive information from at least three modes and encode the information of each mode based on at least one dimension of the amplitude, frequency and number of pulses to generate a synthetic input electrical signal.

[0011] A hidden layer, connected to the input layer, includes at least one two-stage semiconductor laser, the two-stage semiconductor laser including a gain region and an electrically modulated saturable absorber; the hidden layer is used to receive the input electrical signal and convert it into a driving voltage applied to the semiconductor laser; wherein the driving voltage is configured to sequentially excite the semiconductor laser in a time-division multiplexing manner, such that the carrier concentration in the gain region is first accumulated based on voltage pulses encoded with first mode information and second mode information, and then a second accumulation is performed based on voltage pulses encoded with third mode information, thereby outputting a continuous regular pulse optical signal with gradient response characterizing multimodal information fusion;

[0012] The output layer, connected to the hidden layer, includes an optical amplifier and a photodetector for receiving and amplifying the continuous regular pulse optical signal and converting it into an electrical signal to obtain the fused multimodal features.

[0013] Preferably, the input layer includes:

[0014] An information encoder is used to map the information of the at least three modes into corresponding electrical signal parameters, respectively;

[0015] A waveform generator, connected to the information encoder, is used to generate the synthesized input electrical signal based on the electrical signal parameters.

[0016] Preferably, the information encoder includes:

[0017] The first encoding unit is used to encode the first modal information into the number of pulses in the input electrical signal;

[0018] The second encoding unit is used to encode the second modal information into the pulse repetition frequency or duty cycle in the input electrical signal;

[0019] The third encoding unit is used to encode the third modal information into the amplitude of a set pulse in the input electrical signal;

[0020] The signal synthesis unit is used to allocate timing to the outputs of the first coding unit, the second coding unit and the third coding unit, and synthesize them into the synthesized input electrical signal in a time-division multiplexing manner.

[0021] Preferably, in the input electrical signal, the amplitude of the voltage pulse generated based on the first mode information is greater than or equal to the amplitude of the voltage pulse generated based on the third mode information.

[0022] Preferably, the input layer further includes an electrical amplifier and a bias circuit; the electrical amplifier is used to amplify the input electrical signal; the bias circuit is used to superimpose a static bias voltage on the amplified input electrical signal to generate the driving voltage.

[0023] Preferably, the output layer further includes a programmable gate array connected to the photodetector for training and testing the fused multimodal features.

[0024] Preferably, the semiconductor laser is a distributed feedback laser, and the saturable absorber is modulated by the driving voltage to achieve the gradient response.

[0025] Preferably, the first modal information and the second modal information are tactile information and visual information, and the third modal information is auditory information.

[0026] This invention also provides a multimodal information fusion method based on gradient neurons, applied to the multimodal physical neural network system described above, the method comprising:

[0027] The input layer encodes input information from at least three modes to generate a synthesized input electrical signal.

[0028] The driving voltage, converted from the input electrical signal, is received by a two-stage semiconductor laser in the hidden layer. The driving voltage sequentially excites the semiconductor laser in a time-division multiplexing manner, so that the carrier concentration in its gain region is first accumulated based on voltage pulses encoded with first and second mode information, and then accumulated a second time based on voltage pulses encoded with third mode information, thereby outputting a continuous regular pulse optical signal with gradient response that integrates multi-mode information.

[0029] The continuous regular pulse light signal is photoelectrically converted and amplified by the output layer to obtain the fused multimodal feature signal.

[0030] Preferably, the method further includes: training the fused multimodal feature signal using a linear regression or ridge regression algorithm to complete the learning task of the neural network.

[0031] As can be seen from the above technical solutions, this invention application has the following beneficial effects:

[0032] (1) Traditional multimodal fusion relies on back-end digital algorithms to calculate discrete signals acquired by front-end sensors, which has inherent defects such as multiple data conversion layers, large processing delays, and high energy consumption. This invention creatively uses a two-stage semiconductor laser (gain region and saturable absorber) as the core processing unit (hidden layer), directly utilizing its physical dynamics of carrier accumulation and relaxation to process multimodal electrical signals. By designing a time-division multiplexed driving voltage timing sequence, the carrier concentration in the active region of the laser can respond to the information input of different modes in stages and nonlinearly, and the final optical pulse output itself is the physical representation of the fusion result. This "analog computing" paradigm completes the fusion during the signal conversion from electricity to light, eliminating the complex and energy-consuming digital and software algorithm processing links in the traditional architecture, significantly improving the speed and energy efficiency of information processing, and is particularly suitable for edge computing scenarios with stringent requirements for real-time performance and power consumption.

[0033] (2) Existing hardware fusion schemes are mostly based on linear weighting or simple superposition of signals, which cannot reflect the "1+1>2" enhancement effect (i.e., superadditivity) generated by multi-sensory collaboration in biological sensing systems. The core mechanism of this invention is that the change in carrier concentration in the laser gain region is a nonlinear process with a memory effect. The first and second modal information first induces the first accumulation of carriers, forming a "physical memory" state; subsequently, the input of the third modal information does not act independently, but induces a second accumulation on this basis. The final carrier concentration and the corresponding light output are nonlinear functions of the two excitations, rather than simple summation. This mechanism enables the system to generate a stronger response when faced with multimodal information that appears collaboratively than when each modality is input individually, thus achieving superadditive fusion natively at the hardware level. This makes the fused feature information more discriminative and robust, and can more effectively support subsequent recognition and decision-making tasks.

[0034] (3) Compared to traditional optoelectronic fusion systems that require separate optical modulators, beam combiners, and complex circuits, the core processing function of this invention is simultaneously completed by an integrated two-stage semiconductor laser: the electrical modulation of the saturable absorber realizes the gradient threshold response and information storage function of the "neuron," while the stimulated emission process of the entire laser completes information fusion and mapping to a higher-dimensional optical space. This "integrated" design greatly simplifies the system structure and reduces integration difficulty and manufacturing costs. At the same time, the input layer is extremely flexible in encoding multimodal information, and can freely map different information to multiple electrical dimensions such as the number of pulses, frequency (duty cycle), and amplitude, and combine them through timing design, breaking through the limitations of the inherent properties of a single physical device. This simple and flexible design makes the system not only easy to manufacture, but also easy to expand its processing capabilities by adding laser arrays (simulating neural networks) or adjusting the encoding strategy, adapting to more complex multimodal tasks. Attached Figure Description

[0035] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the drawings used in the embodiments will be briefly described below. Referring to the drawings will make the features and advantages of the present invention clearer. The drawings are illustrative and should not be construed as limiting the present invention in any way. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort. Wherein:

[0036] Figure 1 This is a schematic diagram of the overall structure of a multimodal physical neural network system based on gradient neurons provided by the present invention;

[0037] Figure 2 This is a detailed structural diagram of a multimodal physical neural network system based on gradient neurons provided by the present invention;

[0038] Figure 3 This is a schematic diagram of the information encoder structure in this invention;

[0039] Figure 4 This is a schematic diagram of the semiconductor laser in this invention;

[0040] Figure 5 This is a schematic diagram of the input and output signals of the semiconductor laser in this invention.

[0041] Explanation of reference numerals in the accompanying drawings: 1. Input layer; 11. Information encoder; 111. First encoding unit; 112. Second encoding unit; 113. Third encoding unit; 114. Signal synthesis unit; 12. Waveform generator; 13. Electrical amplifier; 14. Bias circuit; 2. Hidden layer; 21. Semiconductor laser; 211. Gain region; 212. Saturable absorber; 3. Output layer; 31. Optical amplifier; 32. Photodetector; 33. Programmable gate array. Detailed Implementation

[0042] To make the objectives, technical solutions, and advantages of the embodiments of the present invention clearer, the technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0043] Example 1: To address the problems of complex structure, low energy efficiency, large processing latency, and inability to achieve nonlinear superadditive fusion at the physical hardware level in existing multimodal information fusion systems. Please refer to... Figure 1 , Figure 1 The diagram shows the overall structure of the multimodal physical neural network system based on gradient neurons provided in this application. The system mainly consists of three parts: input layer 1, hidden layer 2, and output layer 3. Input layer 1 is responsible for encoding and synthesizing raw information from multiple modalities (e.g., visual, auditory, and tactile signals) into a single electrical signal. Hidden layer 2, as the core processing unit, receives this electrical signal, utilizes its internal physical dynamics characteristics to directly perform nonlinear fusion of multimodal information at the hardware level, and outputs an optical signal carrying fusion features. Output layer 3 is responsible for receiving and converting this optical signal, and extracting the fusion features that can be used by subsequent computing units (such as digital processors).

[0044] Furthermore, such as Figure 2 As shown, the detailed structure and operation of the input layer 1 of this invention are as follows:

[0045] The core function of input layer 1 is to encode and synthesize multimodal information into one-dimensional electrical signals. Specifically, it includes an information encoder 11, a waveform generator 12, an electrical amplifier 13, and a bias circuit 14.

[0046] The information encoder 11 is the execution unit of the encoding strategy. For example... Figure 3 As shown, it internally includes a first coding unit 111, a second coding unit 112, a third coding unit 113, and a signal synthesis unit 114. In a preferred embodiment of the present invention, a three-modal information fusion scheme is adopted:

[0047] First encoding unit 111: Used to process first modal information (e.g., a tactile information matrix). Its encoding strategy is to map the numerical value of the modal information to the number of electrical pulses. For example, for a 30×30 tactile matrix, after flattening it into a one-dimensional sequence, if the first value is 1, it is encoded as 1 pulse.

[0048] The second encoding unit 112 is used to process second modal information (e.g., a visual information matrix). Its encoding strategy is to map the numerical values ​​of the modal information to the repetition frequency or duty cycle of electrical pulses. For example, for a 28×28 visual matrix, if the first value after flattening is 0.1, it is encoded as a pulse train with a repetition frequency or duty cycle of 0.1.

[0049] The third encoding unit 113 is used to process third modal information (e.g., an auditory information matrix). Its encoding strategy is to map the numerical value of the modal information to the amplitude of a set pulse (e.g., the last pulse). For example, for a 41×40 auditory matrix, if the first value after flattening is 0.3, it is encoded as a pulse with an amplitude of 0.3.

[0050] The signal synthesis unit 114 receives parameters from three encoding units and uses time-division multiplexing (TDM) technology to combine them into a coherent one-dimensional electrical signal parameter sequence. Its workflow is as follows: a fixed time window is allocated to each processing unit (or "local region"); within this time window, the electrical signal parameters corresponding to the first mode (number of pulses), the second mode (pulse frequency), and the third mode (pulse amplitude) are arranged sequentially according to a preset timing sequence. This timing arrangement ensures that the information of different modes is separated on the time axis, but belongs to the same "local region" or related events.

[0051] The waveform generator 12 is connected to the information encoder 11. It receives the electrical signal parameter sequence output by the signal synthesis unit 114 and generates a corresponding, synthesized analog input electrical signal based on these parameters (including pulse number, frequency, amplitude, and timing). To effectively drive the subsequent semiconductor laser 21, this input electrical signal typically needs to be amplified. Therefore, the input layer 1 also includes an electrical amplifier 13 to increase the amplitude of the input electrical signal.

[0052] Furthermore, input layer 1 also includes a bias circuit 14. This circuit superimposes a stable static DC bias voltage onto the amplified input signal. This design is crucial, as it allows the input signal to effectively modulate the semiconductor laser 21 in hidden layer 2 with only a small amplitude variation (i.e., the AC component), while ensuring the laser operates at a suitable operating point to produce the desired dynamic response (e.g., pulsed output). The output of bias circuit 14 is the final driving voltage applied to hidden layer 2.

[0053] Furthermore, the core mechanism and working process of the hidden layer 2 of the present invention are as follows:

[0054] Hidden layer 2 is the innovative core of the entire system, and its main component is a two-stage semiconductor laser 21. For example... Figure 4 As shown, the laser is internally divided into two functional regions: a gain region 211 and a saturable absorber 212. The gain region 211 is responsible for providing optical amplification, while the saturable absorber 212 is a saturable absorption region whose absorption characteristics can be modulated by an externally applied electrical signal (i.e., driving voltage), which is the key to achieving "gradient response" and physical fusion.

[0055] The operation of hidden layer 2 simulates the integrated firing mechanism of biological neurons and innovatively achieves physical-level superadditive fusion:

[0056] 1. Information Storage Stage (First Accumulation): When the sequence of driving voltage pulses encoded with information of the first and second modes arrives at the semiconductor laser 21, these electrical pulses modulate the saturable absorber 212. Under the excitation of the pulses, charge carriers (electron-hole pairs) in the gain region 211 begin to accumulate, and their concentration rises to a specific level related to the number and frequency of pulses. This process is equivalent to "storing" the information of the first and second modes in the active region (gain region 211) of the laser in the form of a charge carrier concentration distribution.

[0057] 2. Information Fusion and Excitation Stage (Second Accumulation and Output): Immediately following, a driving voltage pulse (usually an amplitude-modulated pulse) encoded with third-mode information arrives. This pulse, based on the previously established carrier concentration, re-excites the gain region 211, triggering a second accumulation of carriers. Since this accumulation is based on the existing "memory" (the result of the first accumulation), the final carrier concentration is a nonlinear superposition of the two excitation processes, rather than a simple linear addition.

[0058] 3. Physical Fusion Output: The nonlinear change in carrier concentration directly determines the laser's output intensity. When the total carrier concentration exceeds the laser threshold, the laser outputs a high-intensity light pulse. Therefore, the intensity (or energy) of the final output continuous regular pulse light signal actually encodes the fusion result of all three modal information, and under appropriate parameter settings, its response intensity can be greater than the sum of the responses when each modality is excited individually, exhibiting a superadditive effect, thus simulating the multimodal perception fusion characteristics of the biological brain at the hardware level. Simultaneously, by adjusting the static voltage of the bias circuit 14, the laser's response threshold can be precisely controlled, causing the amplitude of its output light pulse to change gradient with the driving voltage, simulating the "gradient firing" characteristics of biological neurons.

[0059] Furthermore, the functions and training of output layer 3 of this invention are as follows:

[0060] Output layer 3 is responsible for receiving and transforming the fusion result from hidden layer 2, and performing subsequent processing. For example... Figure 2 As shown, output layer 3 includes:

[0061] Optical amplifier 31: Connected one-to-one with the output of each semiconductor laser 21, used to amplify weak, continuous, regular pulsed light signals, and improve the signal-to-noise ratio and detection sensitivity.

[0062] Photodetector 32: Connected to optical amplifier 31, its function is to convert the amplified optical pulse signal into a corresponding electrical signal. This electrical signal is the fused multimodal characteristic signal read from the physical system.

[0063] Programmable gate array 33 (e.g., FPGA): Connected to photodetector 32. The FPGA is responsible for high-speed acquisition of these feature electrical signals and can perform simple preprocessing (such as digitization and buffering). More importantly, the acquired fused feature data can be sent to a host computer or embedded processor for training and testing using lightweight machine learning algorithms such as linear regression and ridge regression, thereby completing specific classification, recognition, and other tasks and verifying the effectiveness of this hardware fusion system.

[0064] Example 2: This invention provides a multimodal information fusion method based on gradient neurons, applied to the multimodal physical neural network system described in Example 1 above. The method includes:

[0065] S1: The input information of at least three modes is encoded through the input layer to generate a synthesized input electrical signal;

[0066] S2: The driving voltage converted from the input electrical signal is received by a two-stage semiconductor laser in the hidden layer; wherein the driving voltage sequentially excites the semiconductor laser in a time-division multiplexing manner, so that the carrier concentration in its gain region is first accumulated based on voltage pulses encoded with first and second mode information, and then accumulated a second time based on voltage pulses encoded with third mode information, thereby outputting a continuous regular pulse optical signal with gradient response that integrates multi-mode information;

[0067] S3: The continuous regular pulse light signal is photoelectrically converted and amplified by the output layer to obtain the fused multimodal feature signal.

[0068] Further, in step S1, the information encoder 11 and waveform generator 12 of the input layer 1 are used to encode and time-division multiplex the original information of at least three modes to be processed, generate a synthesized input electrical signal, and form a driving voltage after amplification and biasing.

[0069] Further, in step S2, a driving voltage is injected into the two-segment semiconductor laser 21 of the hidden layer 2. The laser responds sequentially to the voltage timing sequence encoded with different modal information: firstly, the first accumulation (information storage) of charge carriers is completed based on the voltage pulses of the first and second modes; subsequently, the second accumulation is completed based on the voltage pulse of the third mode on the basis of the first accumulation. Through this nonlinear process, the superadditive fusion of multimodal information is realized at the physical level, and a gradient pulse light signal characterizing the fusion result is output.

[0070] Further, in step S3, the photodetector 32 of the output layer 3 converts the optical pulse signal into an electrical signal and acquires it. Finally, the acquired fused feature data is sent to a processing unit (such as a computer connected to an FPGA) and the network is trained and its performance tested using algorithms such as linear regression or ridge regression.

[0071] In one specific embodiment, hidden layer 2 employs a distributed feedback (DFB) laser with a saturable absorber 212 to simulate a gradient neuron. The dynamic behavior of this system can be described by the following set of rate equations:

[0072] ; ; ;

[0073] in: and These represent the carrier densities in the gain region 211 and the saturable absorber 212, respectively. This indicates the intensity of the light output from the laser; and These are the bias parameters (related to the driving voltage) for the gain region 211 and the saturable absorber 212, respectively. It is the differential absorption coefficient of saturated absorber 212; , , It is the relaxation rate of each variable; This represents the driving electrical signal (i.e. the signal from the bias circuit 14) injected through the saturable absorber 212 and encoded with multimodal information. This indicates spontaneously radiated noise.

[0074] By numerically solving this system of equations, simulations can be performed as follows: Figure 5 The relationship between the input (multimodal time-division multiplexed electrical signal) and the output (fused regular light pulse) shown verifies the fusion mechanism and superadditive effect of the present invention. In this embodiment, tactile and visual information can be used as the first and second modes, and auditory information as the third mode. These are injected into the saturable absorber of the laser through time-division multiplexing. The light output during the third mode period is sampled as the fusion feature for training, achieving good classification results on a standard dataset.

[0075] Those skilled in the art will understand that embodiments of this application can be provided as methods, systems, or computer program products. Therefore, this application can take the form of a completely hardware embodiment, a completely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, this application can take the form of a computer program product embodied on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0076] This application is described with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of this application. It will be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, special-purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, generate instructions for implementing the flowchart... Figure 1 One or more processes and / or boxes Figure 1 A device that provides the functions specified in one or more boxes.

[0077] These computer program instructions may also be stored in a computer-readable storage medium that can direct a computer or other programmable data processing device to function in a particular manner, such that the instructions stored in the computer-readable storage medium produce an article of manufacture including instruction means, which are implemented in a process Figure 1One or more processes and / or boxes Figure 1 The functions specified in one or more boxes. These computer program instructions may also be loaded onto a computer or other programmable data processing apparatus to cause a series of operational steps to be performed on the computer or other programmable apparatus to produce a computer-implemented process, thereby providing instructions that execute on the computer or other programmable apparatus for implementing the process. Figure 1 One or more processes and / or boxes Figure 1 The steps of the function specified in one or more boxes.

[0078] Obviously, the above embodiments are merely illustrative examples for clear explanation and are not intended to limit the implementation. Those skilled in the art will recognize that other variations or modifications can be made based on the above description. It is neither necessary nor possible to exhaustively list all possible implementations here. However, obvious variations or modifications derived therefrom are still within the scope of protection of this invention.

Claims

1. A multi-modal physical neural network system based on gradient neurons, characterized in that, include: An input layer is used to receive information from at least three modes and encode the information of each mode based on at least one dimension of pulse amplitude, frequency, and number to generate a synthesized input electrical signal; the input layer includes an information encoder and a waveform generator; the information encoder is used to map the information of the at least three modes to corresponding electrical signal parameters. The waveform generator is connected to the information encoder and is used to generate the synthesized input electrical signal according to the electrical signal parameters; The information encoder includes: The first encoding unit is used to encode the first modal information into the number of pulses in the input electrical signal; The second encoding unit is used to encode the second modal information into the pulse repetition frequency or duty cycle in the input electrical signal; The third encoding unit is used to encode the third modal information into the amplitude of a set pulse in the input electrical signal; A signal synthesis unit is used to allocate timing to the outputs of the first coding unit, the second coding unit and the third coding unit, and synthesize them into the synthesized input electrical signal in a time-division multiplexing manner; A hidden layer, connected to the input layer, includes at least one two-stage semiconductor laser, the two-stage semiconductor laser including a gain region and an electrically modulated saturable absorber; the hidden layer is used to receive the input electrical signal and convert it into a driving voltage applied to the semiconductor laser; wherein the driving voltage is configured to sequentially excite the semiconductor laser in a time-division multiplexing manner, such that the carrier concentration in the gain region is first accumulated based on voltage pulses encoded with first mode information and second mode information, and then a second accumulation is performed based on voltage pulses encoded with third mode information, thereby outputting a continuous regular pulse optical signal with gradient response characterizing multimodal information fusion; The output layer, connected to the hidden layer, includes an optical amplifier and a photodetector for receiving and amplifying the continuous regular pulse optical signal and converting it into an electrical signal to obtain the fused multimodal features.

2. The multimodal physical neural network system based on gradient neurons according to claim 1, characterized in that, In the input electrical signal, the amplitude of the voltage pulse generated based on the first mode information is greater than or equal to the amplitude of the voltage pulse generated based on the third mode information.

3. The multimodal physical neural network system based on gradient neurons according to claim 1, characterized in that, The input layer further includes an electrical amplifier and a bias circuit; the electrical amplifier is used to amplify the input electrical signal; the bias circuit is used to superimpose a static bias voltage on the amplified input electrical signal to generate the driving voltage.

4. The multimodal physical neural network system based on gradient neurons according to claim 1, characterized in that, The output layer also includes a programmable gate array connected to the photodetector for training and testing the fused multimodal features.

5. The multimodal physical neural network system based on gradient neurons according to claim 1, characterized in that, The semiconductor laser is a distributed feedback laser, and the saturable absorber is modulated by the driving voltage to achieve the gradient response.

6. The multimodal physical neural network system based on gradient neurons according to claim 1, characterized in that, The first modal information and the second modal information are tactile information and visual information, respectively, and the third modal information is auditory information.

7. A multimodal information fusion method based on gradient neurons, characterized in that, The method, applied to the multimodal physical neural network system as described in any one of claims 1 to 6, comprises: The input layer encodes input information from at least three modes to generate a synthesized input electrical signal. The driving voltage, converted from the input electrical signal, is received by a two-stage semiconductor laser in the hidden layer. The driving voltage sequentially excites the semiconductor laser in a time-division multiplexing manner, so that the carrier concentration in its gain region is first accumulated based on voltage pulses encoded with first and second mode information, and then accumulated a second time based on voltage pulses encoded with third mode information, thereby outputting a continuous regular pulse optical signal with gradient response that integrates multi-mode information. The continuous regular pulse light signal is photoelectrically converted and amplified by the output layer to obtain the fused multimodal feature signal.

8. The multimodal information fusion method based on gradient neurons according to claim 7, characterized in that, The method further includes: training the fused multimodal feature signal using linear regression or ridge regression algorithms to complete the learning task of the neural network.

Images