An audio-visual fusion neuron device and an audio-visual fusion target detection method

Abstract

The present invention discloses an audio-visual fusion neuron device and an audio-visual fusion target detection method, which belong to the field of intelligent recognition technology. The audio-visual fusion neuron device is a photodetector with a back-gate structure using ferroelectric semiconductor bismuth oxide selenium as the channel layer. Visual information is irradiated onto the channel in the form of light power to produce a photoconductivity effect; auditory information is converted into a voltage signal and transmitted to the gate electrode to induce ferroelectric polarization of the channel. The polarization direction effectively regulates the channel conductance, and further regulates the generation and transmission behavior of photogenerated carriers in the channel, thereby realizing nonlinear audio-visual fusion. Based on the superadditivity, anti-validity and spatiotemporal fusion principles of the audio-visual fusion, effective audio-visual fusion target recognition with enhanced reliability can be performed, especially the detection of continuous fuzzy targets, including but not limited to vehicle position detection and vehicle type recognition.

Description

Technical Field

[0001] The present invention relates to an audiovisual fusion detection method for blurred targets, and in particular to a method for realizing audiovisual fusion based on a ferroelectric semiconductor under the action of visual (light)-audio (electric) information and relying on the nonlinear response of the device inside a channel, and is used for strengthening target recognition of blurred audiovisual feature information, belonging to the field of sensing, storage and computing integration and intelligent recognition technology. Background Art

[0002] Vision and hearing are the two most important senses of organisms, accounting for more than 90% of the information that organisms obtain from the outside world. In the field of biology, audio-visual fusion is believed to effectively enhance the perception of external signals and improve the perception and detection capabilities of the outside world. Biological studies have shown that audio-visual fusion has superadditive, anti-validity and co-temporal co-domain. When the audio-visual information is relatively weak, the audio-visual fusion neurons can effectively integrate the audio-visual information, greatly enhancing the reliability of the information in the case of audio-visual fuzzy features. In the field of intelligent sensing devices, in order to achieve reliable detection of multimodal external information, sensors of various materials and structures are often integrated on the chip to achieve the perception and processing of multimodal information, but this solution often faces the problems of poor process compatibility, poor device stability, and serious signal crosstalk between different modes. In order to achieve reliable multimodal information perception and fusion, various fusion algorithms and hardware structures have emerged, but these methods generally use linear fusion methods, facing the problems of high power consumption, complex algorithms, and linear fusion rules violating the Bayesian principle. In order to solve these problems, a solution that relies on new mechanisms and new materials, can achieve nonlinear audio-visual fusion through device structure design, and can be used for remote fuzzy information recognition needs to be proposed urgently. Summary of the invention

[0003] In order to achieve the detection of blurred targets, the first purpose of the present invention is to provide an electronic device unit capable of realizing nonlinear audio-visual fusion. For this purpose, an audio-visual fusion neuron device based on high-mobility ferroelectric semiconductor bismuth oxide selenium is proposed, which realizes nonlinear audio-visual fusion response based on the regulation of photoconductivity by ferroelectric effect. Furthermore, the ferroelectric semiconductor audio-visual fusion neuron device of the present invention can effectively perform audio-visual fusion with enhanced reliability for continuous blurred target detection.

[0004] The technical solution of the present invention is as follows:

[0005] A neuron device for audio-visual fusion is a photodetector with a back-gate structure based on ferroelectric semiconductors, including a bismuth oxide selenide ferroelectric channel layer, source-drain (S / D) electrodes at both ends of the channel, and a gate dielectric layer and gate electrode below the channel. Visual information is irradiated onto the channel in the form of light power, generating a photoconductive effect; auditory information is converted into a voltage signal and transmitted to the gate electrode to induce the ferroelectric polarization of the ferroelectric channel. The polarization direction of the ferroelectric channel effectively regulates the channel conductance, and further regulates the generation and transmission behavior of photogenerated carriers in the channel, thereby achieving nonlinear audio-visual fusion.

[0006] In some embodiments of the present invention, heavily doped silicon is used as a substrate, and a gate dielectric layer and a bismuth oxide selenide iron channel layer are sequentially prepared on the substrate. Then, source and drain electrodes are prepared at both ends of the channel, and the heavily doped silicon substrate serves as a gate electrode to provide a gate voltage bias for the device.

[0007] Furthermore, the gate dielectric layer may be made of hafnium oxide (HfO 2 ), aluminum oxide, silicon oxide and other dielectric materials, the thickness is preferably in the range of 10-20 nm; the source and drain electrodes can be metal electrodes such as titanium gold, palladium gold, titanium palladium gold, etc., preferably Ti / Au metal stack; the thickness of the bismuth oxide selenide iron channel layer is in the range of 5-20 nm, and the thickness of the source and drain electrodes does not exceed 60 nm.

[0008] Based on the above audio-visual fusion neuron device, the present invention provides an audio-visual fusion target detection method, which is particularly beneficial for detecting continuously moving blurred targets, including:

[0009] First, the audio-visual fusion neuron device has extremely sensitive photoelectric information conversion capabilities, which can achieve a minimum of 60 μW / cm 2 The detection of optical power provides great convenience for the detection of remote blurred targets. The general image sensor is limited by the response sensitivity and the influence of noise, and it is difficult to detect dim light and blurred targets. However, based on the sensitive two-dimensional bismuth oxygen selenium selected by the present invention as the channel for photoelectric detection, it can effectively realize the resolution of remote blurred objects and improve the efficiency of signal transmission.

[0010] Secondly, the audio-visual fusion neuron device uses a new ferroelectric semiconductor bismuth oxide selenide as a channel. External sound information is converted into an electrical signal through a microphone and input into the device through the gate electrode of the device, further triggering the ferroelectric polarization flipping of the channel material. Since the two-dimensional ferroelectric semiconductor bismuth oxide selenide has a sensitive gate voltage regulation capability, it can effectively realize the conversion of external sound information. The channel resistance regulated by the gate electrode input effectively improves the sensitivity of sound detection. For the audio-visual fusion neuron device, its sound detection limit is 60 dB, which has good sound detection characteristics and can be effectively used for the detection, information collection and conversion of fuzzy sound targets of remote vehicles;

[0011] In addition, through the unique physical properties of the device, relying on the regulation of the light response by the ferroelectric polarization caused by sound, the synchronous detection of audiovisual information inside the device can be effectively realized. In the natural environment, multiple information can be used as the basis for cognitive detection of the outside world to effectively improve the ability to predict and judge the external environment. However, so far, there has never been a report on the perception of multiple modal information based on the nonlinear dynamic characteristics inside the device, and the reliable realization of information amplification and feature enhancement. The audio-visual fusion neuron device exhibits a very strong nonlinear integration and amplification effect when both visual and auditory information are weak. After being processed by the audio-visual fusion neuron device, the output signal characteristics are greatly enhanced compared to the component signals before fusion, and the detection ability for blurred targets is significantly improved.

[0012] In the audio-visual fusion target detection method of the present invention, the audio-visual fusion neuron device is used for front-end signal acquisition, transmission and preprocessing, and then the target is identified through a pre-trained artificial neural network. Taking the detection of continuously moving vehicles as an example, for the video stream data of a remote vehicle that also contains audio-visual information, the optical information is irradiated onto the channel of the device in the form of optical power, and based on the photoconductivity effect of the device, the conversion of optical information into electrical information is realized. The sound information is first converted into a corresponding voltage signal through a microphone module, the ground end of which is connected to the source of the device, and the output end is connected to the back gate of the device. When external sound information is transmitted, it is input into the back gate of the channel in the form of voltage, further inducing ferroelectric polarization of the channel. For ferroelectric semiconductors, different polarization states can affect the transmission of carriers in the channel and thus regulate photoconductivity. Based on this characteristic, the fusion of visual and auditory information can be realized inside the device, and the transformation of two-dimensional information into one-dimensional information can be realized, thereby compressing the amount of data. At the same time, based on the super-additivity of the device, it can achieve significant enhancement of features. Therefore, based on the hardware-based audio-visual fusion solution proposed in the present invention, it is possible to simultaneously achieve information dimensionality reduction and feature enhancement. By constructing a neuron array, the output of each device is detected at each moment, and then a set of image information obtained at different times is input into a network programmed with software such as Python for training and recognition, thereby completing the recognition of fuzzy vehicle types.

[0013] In some embodiments of the present invention, a plurality of dispersed audio-visual fusion neuronal devices are used as sensors. Since the distances between the detection target and different neuronal devices are different, the time to reach different neuronal devices and the intensity of sound information will have obvious differences. Based on the time fusion law in the device audio-visual fusion, different neuronal devices produce post-synaptic responses of different amplitudes, which can effectively realize the detection of azimuth and distance, thereby calibrating the azimuth of the detection target.

[0014] In other embodiments of the present invention, the audio-visual fusion neuron devices are arrayed to construct a hardware pulse emission module to convert continuous audio-visual signals into frequency-coded pulses, which are used to train artificial neural networks; then, the target is visually identified based on the trained artificial neural network.

[0015] In summary, the technical advantages of the present invention are as follows:

[0016] 1) For the first time, the nonlinear fusion of visual and auditory information was realized based on the ferroelectric semiconductor bismuth oxide selenide. The long-term memory characteristics of auditory information and the short-term memory characteristics of visual information of the bismuth oxide selenide audio-visual fusion neuron device were verified. The memory time of the ferroelectric semiconductor exceeded 10 years, reaching the extreme value of the ferroelectric semiconductor characteristics reported so far.

[0017] 2) For the first time, the superadditivity, anti-validity and spatiotemporal fusion principles of audiovisual fusion based on the ferroelectric semiconductor bismuth oxygen selenide were verified, and it was proposed that the fusion information processed by audiovisual fusion neurons has smaller variance and higher stability. The superadditivity factor of audiovisual fusion is higher than 2880%, and the data volatility is reduced from 69% of single vision and 22% of single hearing to 3% of audiovisual fusion.

[0018] 3) For the first time, vehicle position detection and vehicle type recognition are performed based on audio-visual fusion information. By building a hardware pulse emission module, continuous analog signals are converted into frequency-coded pulses, which are used to train pulse neural networks. The super-additivity and enhanced reliability achieved by audio-visual fusion achieve higher recognition accuracy than that of a single modality (single vision: 85.71%, single hearing 83.33%, audio-visual fusion: 100%). BRIEF DESCRIPTION OF THE DRAWINGS

[0019] Figure 1 It is a schematic diagram of the principle of realizing audio-visual fusion by the bismuth iron oxide selenide television and auditory fusion neuron in an embodiment of the present invention.

[0020] Figure 2The piezoelectric atomic force microscope (PFM) characterization results of the ferroelectricity of the channel material bismuth oxide selenide in the embodiment of the present invention, wherein (a) shows the change of phase of a two-dimensional bismuth oxide selenide crystal with a thickness of 4.8 nm under PFM characterization with voltage bias; (b) shows the change of the height of the two-dimensional bismuth oxide selenide crystal in the vertical direction under PFM characterization with voltage bias; (c) is a phase distribution diagram in the read-write back word test. During the read-write back word test, a 9 V voltage bias is first applied to the outer large square. Due to the polarization characteristics of the ferroelectric material, the electrical information can be stably stored. Then, a -9 V voltage bias is applied to the inner small square. At this time, the polarization direction begins to turn to the opposite direction, thereby forming a phase difference of 180°, indicating the existence of ferroelectricity; (d) is a height distribution diagram in the read-write back word test.

[0021] Figure 3 The results of the study on the electrical non-volatile behavior of the device based on the ferroelectric bismuth oxygen selenide channel in the embodiment of the present invention, wherein (a) is the gate bias voltage (V g ) Transfer characteristic curve during 200 cycles of scanning from -7 V to 6 V; (b) The switching ratio and storage window extracted from (a) change with the number of scans; (c) The channel current (I) induced by applying a single pulse voltage stimulus from 0.5 V to 3 V on the gate d ) changes with time, thus characterizing the long-term memory caused by the ferroelectric effect; (d) shows the distribution of the number of conductivity states obtained by repeatedly applying "set" Set (-1V) and "reset" Reset (1V) voltage pulses, and more than 264 symmetrically distributed conductivity states can be obtained.

[0022] Figure 4 The figures are the research results of the volatility behavior of the optical response of the device based on the ferroelectric bismuth oxygen selenide channel in the embodiment of the present invention, wherein (a) shows the change of the transfer characteristic curve with the light intensity under the condition of continuous light application; (b) shows the change of the channel current with time under the condition of a single light pulse.

[0023] Figure 5 Schematic diagram of the three basic rules of audio-visual fusion based on ferroelectric bismuth oxygen selenium: superadditivity, anti-validity, and time fusion principle, as well as the reliability-enhanced Bayesian integration rule, where (a) is a test representation diagram of superadditivity, time fusion rule, and anti-validity; (b) is a schematic diagram of the reliability-enhanced rule based on the Bayesian integration rule, where σ V , σ A , σ VA Represents the variance of visual (V), auditory (A), and audiovisual fusion (VA) response strength, μ V , μ A , μ VARepresent the mean of visual, auditory, and audiovisual fusion response intensities, respectively.

[0024] Figure 6 Schematic diagram of the fusion of audiovisual information of different intensities based on the ferroelectric bismuth oxygen selenium audiovisual fusion neuron in an embodiment of the present invention with superadditivity, wherein (a) shows the superadditivity of the audiovisual information when the light stimulation intensity is 200 μW / cm 2 , the visual (V) response, auditory (A) response and the current change over time when the sound intensity is 80dB and the audio-visual fusion (VA) stimulation is applied simultaneously; (b) shows the current change over time when the light stimulation intensity is 200 μW / cm 2 , visual response, auditory response and the change of current over time when the sound intensity is 90 dB; (c) shows the change of current over time when the light stimulation intensity is 200 μW / cm 2 , the visual response, auditory response and the current change over time when the sound intensity is 100 dB; (d) shows the current change over time when the light stimulation intensity is 200 μW / cm 2 , the visual response, auditory response and the current change over time obtained by applying visual and auditory fusion stimulation at a sound intensity of 110 dB; IF stands for integration factor.

[0025] Figure 7 This is a graph showing the results of the study on improving the reaction speed of the nervous system based on superadditivity in the embodiment of the present invention. The current excitation threshold is set to 10 μA, and the starting current is 4 μA. We tested the number of stimuli required to reach the threshold when visual, auditory, and visual and auditory fusion stimulations are applied, where (a) is 200 μW / cm 2 Light stimulation, (b) 90 dB sound stimulation, (c) 200 μW / cm 2 The relationship between the change of channel current and time under the co-stimulation of light and 90 dB sound.

[0026] Figure 8 These are the research results of superadditivity and anti-validity principle based on ferroelectric bismuth oxygen selenium audio-visual fusion neurons in the embodiments of the present invention, where IF is the integration factor.

[0027] Fig. 9 These are the research results of the time integration principle realized by the ferroelectric bismuth oxygen selenium audiovisual fusion neurons in the embodiments of the present invention, wherein (a) is the relationship between the change of the fusion current peak value and the time interval of the audiovisual fusion stimulation when the audiovisual fusion stimulation is applied simultaneously; (b) is a schematic diagram of reading the fusion response when the visual stimulation is applied first and then the auditory stimulation; (c) is a schematic diagram of reading the fusion response when the auditory stimulation is applied first and then the visual stimulation.

[0028] Fig.10 Schematic diagram of the principle of achieving superadditivity based on ferroelectric bismuth oxygen selenide audiovisual fusion neurons in an embodiment of the present invention, wherein (a) is a channel charge distribution diagram of the audiovisual fusion neuron in the absence of visual and auditory stimulation; (b) is a channel charge distribution diagram of the audiovisual fusion neuron when auditory stimulation is applied alone; (c) is a channel charge distribution diagram of the audiovisual fusion neuron when visual stimulation is applied alone; (d) is a channel charge distribution diagram of the audiovisual fusion neuron when visual stimulation and auditory stimulation are applied simultaneously.

[0029] Fig.11 The signal reliability estimation results of the ferroelectric bismuth oxygen selenium audio-visual fusion neurons acting on visual and auditory information separately in the embodiment of the present invention are shown in Figure 1, where (a) is a distribution result diagram of the statistical response after 150 stimuli are repeatedly applied in the range of sound intensity from 80 dB to 110 dB; (b) is a distribution result diagram of the distribution result in (a) fitted with a Gaussian distribution, and the ratio of the variance to the mean is calculated as the distribution deviation, and its distribution result diagram is studied as the sound intensity increases; (c) is a distribution result diagram of the applied light intensity of 68 μW / cm 2 to 2870 μW / cm 2 (d) is a graph showing the statistical distribution of responses after 150 stimuli are repeatedly applied within the range; (c) is a graph showing the distribution of the results in the figure with Gaussian distribution fitting, and the ratio of the variance to the mean is calculated as the distribution dispersion, and its distribution results as the light intensity increases are studied.

[0030] Fig.12 The results of the research on the reliability enhancement Bayesian integration rule based on the ferroelectric bismuth oxygen selenium audio-visual fusion neuron in the embodiment of the present invention are shown in Figure 1, where (a) is the value of the reliability enhancement Bayesian integration rule under the condition of visual intensity of 200 μW / cm 2 , (a) the response distribution results of visual (V) stimulation alone, auditory (A) stimulation alone, and visual and auditory (VA) stimulation at a sound intensity of 80 dB; (b) the response distribution results of visual (V) stimulation alone, auditory (A) stimulation alone, and visual and auditory (VA) stimulation at a sound intensity of 2870 μW / cm 2 , the response distribution results of visual stimulation alone, auditory stimulation alone, and visual and auditory stimulation at the sound intensity of 80 dB; (c) is the response distribution results of visual stimulation at the sound intensity of 200 μW / cm 2 , the response distribution results of visual stimulation alone, auditory stimulation alone, and visual and auditory stimulation at the sound intensity of 110 dB; (d) is the response distribution results of visual stimulation at the sound intensity of 2870 μW / cm 2 , the response distribution results of visual stimulation alone, auditory stimulation alone, and visual and auditory stimulation at the same time when the sound intensity is 110 dB; (σ / μ) V 、(σ / μ) A、(σ / μ) VA They represent the coefficient of fluctuation (CoV) of the visual, auditory, and audiovisual fusion response intensities, respectively. The CoV value is defined as the ratio of the variance σ to the mean μ.

[0031] Fig.13 The module structure and implementation principle diagram of the pulse emission circuit constructed based on the ferroelectric bismuth oxygen selenium audio-visual fusion neuron in the embodiment of the present invention, wherein (a) is a schematic diagram of the processing flow of the external signal in the described embodiment, and the video stream containing visual and auditory information is sequentially input into the audio-visual fusion neuron array, and then the corresponding analog signal is converted into a pulse signal through the pulse emission module, and then the pulse signals output by all devices are summed, and the pulse distribution is subjected to frequency statistical distribution; (b) is a schematic diagram of the structure of the described pulse emission circuit, which includes three parts: an STM32 microcontroller, a level converter (Level Shifter) and a current-voltage converter (IV Converter), wherein the STM32 microcontroller mainly plays the role of signal collection and conversion, the level converter mainly raises the level of the collected current signal and controls the output voltage value to the required range, the current-voltage converter mainly converts the read current signal into a voltage signal for subsequent reading and processing, and the pulse conversion is to read the current value through hardware and compare it with the set current threshold at the software level. Once the threshold is reached, a 3V pulse will be emitted for subsequent training and input.

[0032] Fig.14 Figure 1 is a graph showing the pulse conversion results of the pulse emission module designed in the embodiment of the present invention for individual visual information, auditory information, and fused information, wherein (a) is a graph showing the distribution of the current reaching the emission threshold over time when a single visual stimulus is applied; (b) is a graph showing the distribution of the current reaching the emission threshold over time when a single auditory stimulus is applied; (c) is a graph showing the distribution of the current reaching the emission threshold over time when both visual and auditory stimuli are applied; and (d) is a graph showing the distribution of the current reaching the emission threshold over time when both visual and auditory stimuli are applied. Fig.13 The circuit diagram shown in (e) is a pulse distribution diagram obtained by summing the pulse output conversion results of all devices in the array when visual stimulation is applied alone; (e) Fig.13 The circuit diagram shown in (a) is a pulse distribution diagram obtained by summing the pulse output conversion results of all devices in the array when auditory stimulation is applied alone; (f) is a pulse distribution diagram according to Fig.13 The circuit diagram shown in is a pulse distribution diagram over time obtained by summing the pulse output conversion results of all devices in the array when visual and auditory stimulation are applied simultaneously.

[0033] Fig.15The invention relates to a pulse emission module designed in an embodiment of the present invention. The invention relates to a pulse emission module designed to estimate the number of visual (V), auditory (A) and audiovisual fusion (VA) information of an audiovisual fusion neuron array constructed under three conditions of far, medium and near detection target distances. (a) is the distribution result of visual, auditory and fusion information obtained by estimating the number of visual fusion neuron array constructed under far-end conditions; (b) is the distribution result of visual, auditory and fusion information obtained by estimating the number of visual fusion neuron array constructed under medium-distance conditions; (c) is the distribution result of visual, auditory and fusion information obtained by estimating the number of visual fusion neuron array constructed under near-end conditions.

[0034] Fig.16 The present invention provides a schematic diagram and result diagram of the realization of position recognition and distance detection of distant vehicles based on two symmetrically structured bismuth oxygen selenium audio-visual fusion neurons in an embodiment of the present invention, wherein (a) is a schematic diagram of the principle of position and distance detection based on the audio-visual fusion neurons; (b) is a schematic diagram of the applied audio-visual stimulation and the obtained postsynaptic response; (c) is the change of the postsynaptic current distribution over time obtained by the device detecting a signal source within the range of 50 to 500 cm; and (d) is a result diagram of the position and distance detection obtained based on the actual measurement results of the device.

[0035] Fig.17 Schematic diagram of the array structure and identification method for identifying the type of blurred vehicles in the distance based on the constructed bismuth iron oxide selenide television and auditory fusion neurons in an embodiment of the present invention.

[0036] Fig.18 Schematic diagram of the visualization results of visual information, auditory information and fusion information obtained by detecting a distant blurred vehicle based on the constructed bismuth iron oxide selenide television-auditory fusion neuron in an embodiment of the present invention, wherein (a) is a visual information result diagram obtained by detecting a distant vehicle; (b) is a result diagram of the frequency distribution changing over time after performing Mel frequency transformation on the auditory information obtained by detecting the distant vehicle; and (c) is an information result diagram obtained by detecting the distant vehicle based on the audio-visual fusion response.

[0037] Fig.19The figure is a schematic diagram of the results of the subsequent artificial neural network identifying the type of remote vehicles based on the fusion information obtained by the iron-video audio-visual fusion neuron processing, wherein (a) shows the change of the accuracy of the audio-visual fusion algorithm, the visual network, the auditory network and the audio-visual fusion hardware solution proposed in the present invention in identifying the type of remote fuzzy vehicles with the training nodes; (b) shows the distribution of the accuracy of the audio-visual fusion algorithm, the visual network, the auditory network and the audio-visual fusion hardware solution proposed in the present invention in identifying the type of remote fuzzy vehicles, as well as the resistance to noise. DETAILED DESCRIPTION

[0038] In order to make the purpose, technical scheme and advantages of the present invention clearer, the structure, performance, audio-visual fusion principle and specific application of the ferroelectric semiconductor audio-visual fusion neuron of the present invention are described in detail below in conjunction with the accompanying drawings. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of them.

[0039] See also Figure 1 The ferroelectric semiconductor audio-visual fusion neuron in this embodiment is composed of a ferroelectric semiconductor channel Bi 2 O 2 Se, source and drain electrodes, gate dielectric layer, and back gate electrode, wherein the layered two-dimensional bismuth oxygen selenium is transferred to Si / HfO 2 On the substrate, the heavily doped silicon substrate acts as the back gate electrode, and HfO 2 It acts as a gate dielectric, and the layered two-dimensional bismuth oxide selenide acts as a channel layer for charge transfer, with source and drain electrodes located at both ends of the channel. External information is input into the ferroelectric semiconductor audio-visual fusion neuron device array in the form of a video stream, wherein the visual information is irradiated on the device channel in the form of light wavelength and power, and is converted into a channel current through the grating and photoconductivity effect; the auditory information is converted into electrical information through an integrated microphone and input into the device gate, and triggers the device ferroelectric polarization flip, thereby regulating the device's optical response and electrical response, thereby realizing the nonlinearity of the optical-electrical response based on the ferroelectric effect inside the device.

[0040] In order to verify the ferroelectricity of the studied semiconductor material, piezoelectric atomic force microscopy (PFM) was performed on the two-dimensional bismuth oxygen selenide. The research results are as follows Figure 2 As shown in the figure, during the ±10 V voltage cyclic scanning process, both the amplitude diagram and the phase diagram of the PFM show obvious hysteresis windows, and the corresponding coercive voltage is ±7 V. In addition, during the electrical writing process, the channel material exhibits obvious non-volatile characteristics, indicating the existence of channel ferroelectricity at room temperature.

[0041] In order to study the changes in electrical properties caused by ferroelectricity, the electrical transfer characteristic curve of the device was scanned cyclically. The research results are as follows: Figure 3As shown in (a) and (b), during the repeated cyclic scanning of 200 cycles, the transfer characteristic curve of the device basically does not drift significantly, and the switching ratio of the device is maintained at 10 6 The memory window reaches 3.5 V. To study the nonvolatile storage time, an electrical pulse of 0 to 3 V is applied to the back gate electrode of the audio-visual fusion neuron. Figure 3 As shown in (c), the channel current memory time is as long as 10 years, indicating the stable and reliable long-term storage capability of the device. In order to study the multi-conductance state of the device, 264 1 V electric pulses were applied continuously, as shown in Figure 3 As shown in (d), the channel exhibits 264 conductance states with obvious changes, indicating that the audio-visual fusion neuron in this embodiment has good storage capacity.

[0042] To study the light response characteristics of the audiovisual fusion neurons, the 2 ) conditions, the curve transfer characteristics were cyclically scanned, and the threshold voltage of the curve drifted significantly, indicating that the grating effect and photoconductivity effect effectively mediated the light response process of the device (such as Figure 4 Then, a single light pulse was applied to the device, and the neuron showed obvious volatile behavior, with a relaxation time between 20 and 100 s (as shown in Figure 4 (as shown in (b)).

[0043] Figure 5 Schematic diagram of the key rules of audio-visual fusion studied in this embodiment. Among them, superadditivity refers to the fact that the fusion response of the device is greater than the linear sum of the responses caused by the separate effects of vision and hearing when vision (V) and hearing (A) act simultaneously; the temporal fusion principle refers to the strongest fusion response when vision and hearing reach the device at the same time. As the interval between audio-visual stimuli increases, the fusion response weakens and the integration effect weakens; the anti-validity rule refers to the phenomenon that the fusion effect weakens as the intensity of a certain component stimulus increases, that is, the fusion effect presents the strongest enhancement effect when the audio-visual information is relatively weak; and the Bayesian integration rule refers to the fact that the signal after fusion has lower data fluctuations and stronger reliability than the signal before fusion.

[0044] Figure 6 This is the actual result of the anti-validity test of this embodiment. In this example, the optical information input intensity is fixed and the auditory information intensity is gradually increased. By defining the integration factor (IF) as the ratio of the device's audiovisual fusion response to the linear sum of the visual and auditory separate responses, the results show that as the auditory information intensity increases, the integration factor gradually decreases, verifying the anti-validity law of audiovisual fusion.

[0045] Figure 7This is the result of regulating channel conductance to reach the excitation threshold by changing the number of pulses of input visual and auditory information under a fixed response threshold. The results show that for separate visual and auditory stimulation, 20 and 5 pulse stimulations are required respectively to reach the neuronal firing threshold, while for the case of audio-visual fusion, only 1 pulse is needed to reach the firing threshold, indicating that audio-visual fusion can effectively improve the response sensitivity and response speed of neurons.

[0046] Figure 8 is the change of integration factor under different visual and auditory stimulation intensities. When the visual and auditory stimulation intensities are relatively low, the integration factor reaches the maximum value. With the increase of visual and auditory stimulation intensities, the integration factor shows a decreasing trend, indicating the inverse validity law of the audiovisual fusion effect. At the same time, within the research range, the integration factors are all values ​​greater than 1, indicating that the superadditivity law of audiovisual fusion is effectively verified in this embodiment.

[0047] Fig. 9 The diagram and result diagram of the verification of the temporal fusion principle of audio-visual fusion are shown in Figure 1. The results show that when audio-visual stimulation acts on the channel at the same time, the current response value is the highest, indicating that the integration effect is the strongest at this time. As the time interval between audio-visual stimulation increases, the fusion response decreases, indicating that the temporal fusion principle is effectively verified in this embodiment.

[0048] Fig.10 : is a schematic diagram of superadditivity based on ferroelectricity in this embodiment. When no audiovisual stimulation is applied to the device, the device is in an intrinsic state, and the channel is in a polarization downward state. When auditory stimulation is applied, the polarization state of the channel is partially reversed, and a part of it is polarized upward. When visual stimulation is applied alone, due to the photoconductivity effect and grating effect of the channel, the generated electrons and holes are transmitted in two directions respectively due to the polarization direction of the ferroelectric. At this time, the electron movement is regulated by the ferroelectric polarization field and the external electric field at the same time. In the intrinsic state, the two force fields have opposite effects on the electron movement, thereby showing the effect of inhibiting transmission. When audiovisual information is applied at the same time, some electrons with partially reversed polarization are forced to be in the same direction, which greatly promotes the separation and transmission of electrons and holes, thereby triggering superadditivity.

[0049] In order to verify the effect of audio-visual fusion on information reliability, the device described in this embodiment was subjected to repeated visual and auditory stimulation for 100 consecutive times, and the response distribution of the audio-visual fusion neurons was statistically analyzed, as shown in FIG. Fig.11As shown in the figure, for individual visual and auditory stimuli, when the stimulus intensity is low, the response shows greater volatility, and the data has a larger coefficient of variation (CoV). As the stimulus intensity increases, the CoV value decreases. As for the fused information, whether it is strong vision and strong hearing, strong vision and weak hearing, weak vision and weak hearing, or weak vision and strong hearing, the fused information has a smaller CoV value (where the CoV value is defined as the ratio of variance to mean, CoV = σ / μ), indicating that the information after audio-visual fusion has stronger reliability (such as Fig.12 as shown).

[0050] Fig.13 The figure is a functional diagram of the pulse conversion circuit constructed in this embodiment. External information enters the 3×3 device array in the form of a video stream. Due to the difference in the brightness of each pixel, each device has a different integration factor, which is defined as IF1~IF9. The output of each audio-visual fusion neuron is connected to the pulse conversion circuit. After reaching the set current threshold, the pulse conversion circuit outputs a voltage pulse with an amplitude of 1V. The output of the nine neurons is summed to obtain the statistical distribution of the neural pulse, which is used as the input of the subsequent artificial neural network.

[0051] Fig.14 The pulse conversion results of this embodiment are shown. For the more distant fuzzy information, since the visual and auditory stimulation intensities are relatively low, the neurons only fire pulses within a short period of time when the stimulation is applied. However, for the fused signal, due to the superadditivity of the device, when the same threshold is set, the neurons are continuously excited within the research time range, generating continuously fired pulses, indicating that the recognition ability for distant fuzzy targets is greatly enhanced.

[0052] In order to study the distribution of responses, the audio-visual information of three situations, far distance, medium distance and near distance, was fused and recognized. Fig.15 The signal frequency distribution after pulse conversion is shown. When the far-end visual and auditory information is relatively vague, the fused information has the strongest amplification effect, and the fusion factor reaches 2880%. As the detection target approaches, the fusion factor decreases. When it reaches the closest end, it can be accurately identified by vision or hearing alone. At this time, both visual and auditory pulses are continuously emitted, and the fusion factor is 23.4%.

[0053] Based on the above-mentioned audio-visual fusion neural technology, the vehicle type and vehicle orientation can be accurately identified. The identification process and principle are explained below.

[0054] For orientation recognition tasks, such as Fig.16As shown, two audio-visual fusion neurons with the same structure are placed at the positions of neuron (S) 1 and neuron (S) 2 as sensors, with an interval of 2 l Since the visual information arrives quickly, it can be considered that the two devices respond at the same time. However, due to the different distances between the detection target and the two neurons, the time to reach the two neurons and the intensity of the sound information will be significantly different. Based on the audio-visual fusion time fusion law of the aforementioned study, the two neurons produce postsynaptic responses of different amplitudes. By using the ratio of the postsynaptic response amplitudes of neuron (S) 2 to neuron (S) 1, the azimuth and distance detection can be effectively achieved, thereby calibrating the azimuth of the detection target.

[0055] In order to realize the recognition of vehicle types, the audiovisual fusion neuron in this embodiment is used to perform front-end nonlinear fusion of visual and auditory information, and then the fusion graphic visualization recognition is performed based on the artificial neural network. The recognition process is as follows: Fig.17 The visualization results of the original visual information, auditory information, and audio-visual fusion information of device array recognition are shown in Fig.18 As shown in the figure, the artificial neural network is trained based on this information. The implementation results show that the fused information has a higher recognition accuracy. For the fused information, the accuracy can be as high as 100%. For the separate visual and auditory networks, the recognition accuracy of the artificial neural network is 85.7% and 88.9% respectively. It can be seen that the audio-visual fusion effect reliably and effectively enhances the reliability of information and significantly improves the recognition accuracy of subsequent algorithms ( Fig.19 ). In addition, the audio-visual fusion technology based on the Bayesian integration rule has stronger noise tolerance. Compared with algorithm fusion, simple visual and auditory recognition, the recognition accuracy of audio-visual device-level fusion in noise conditions is as high as 99.8%, which effectively verifies the reliability of the audio-visual fusion rule.

[0056] The embodiments described above are not intended to limit the present invention. Any person skilled in the art may make various changes and modifications without departing from the spirit and scope of the present invention. Therefore, the protection scope of the present invention is defined by the scope of the claims.

Claims

1. A target detection method based on audio-visual fusion, characterized in that: The audio-visual fusion neuron device is used for front-end signal acquisition, transmission and preprocessing, and then the target is recognized through a pre-trained artificial neural network; wherein the audio-visual fusion neuron device is a back-gate photodetector, including a bismuth oxide selenide iron electric channel layer, source and drain electrodes at both ends of the channel, and a gate dielectric layer and a gate electrode below the channel; visual information is irradiated onto the channel in the form of light power to produce a photoconductivity effect; auditory information is converted into a voltage signal and transmitted to the gate electrode to induce ferroelectric polarization of the channel, and the polarization direction of the channel regulates the channel conductivity, and further regulates the generation and transmission behavior of photogenerated carriers in the channel, thereby realizing nonlinear audio-visual fusion.

2. The audio-visual fusion target detection method according to claim 1, characterized in that: The thickness of the bismuth oxide selenide iron channel layer is 5-20 nm.

3. The target detection method of audio-visual fusion according to claim 1, characterized in that: The audio-visual fusion neuron device uses a heavily doped silicon substrate as a gate electrode; the gate dielectric layer is hafnium oxide, aluminum oxide or silicon oxide with a thickness of 10-20 nm; and the source and drain electrodes are metal electrodes.

4. The audio-visual fusion target detection method according to claim 1, characterized in that: The target's optical information is irradiated onto the device's channel in the form of light power; the target's sound information is converted into an electrical signal through a microphone and applied to the device's gate electrode in the form of voltage.

5. The audio-visual fusion target detection method according to claim 1, characterized in that: The target position detection is realized by using a plurality of dispersedly arranged audio-visual fusion neuron devices as sensors.

6. The audio-visual fusion target detection method according to claim 1, characterized in that: The audio-visual fusion neuron devices are used to form an array, and a hardware pulse emission module is constructed to convert continuous audio-visual signals into frequency-coded pulses, which are used to train artificial neural networks; then, the target is visually identified based on the trained artificial neural network.

7. The audio-visual fusion target detection method according to claim 1, characterized in that: The detection target is a continuously moving fuzzy target.

8. The audio-visual fusion target detection method according to claim 7, characterized in that: The detection target is a moving vehicle.

Images