A multifunctional photoelectric memristor device array and its preparation method

Abstract

The present invention relates to the field of intelligent sensing technology, and in particular, provides a multifunctional photoelectric memristor device array and a preparation method thereof. The photoelectric memristor device array is divided into two areas, one area is used to sense, store and process visual signals, and the other area is used to sense, store and process voice signals; the photoelectric memristor device array is externally connected to a neural network, wherein the neural network is a back propagation neural network; the processed visual signal and voice signal are fused and input into the neural network to complete the final recognition work, and the photoelectric memristor device array reduces time delay and energy consumption, simplifies system design, and improves work efficiency.

Description

Technical Field

[0001] The present invention relates to the field of intelligent sensing technology, and in particular to a multifunctional photoelectric memristor device array and a preparation method thereof. Background Art

[0002] With the advent of the information age, the development of technologies such as artificial intelligence has put forward higher requirements in terms of improving the working efficiency of sensor systems and reducing power consumption. In traditional sensor systems, the modules that perceive, store and process information are separated from each other. This separate architecture inevitably requires the sensor system to transmit a large amount of data between different modules when performing tasks. For example, the sensor module needs to first collect analog signals and convert them into digital signals, and then transmit them to the storage and processing modules in turn. The transmission of large amounts of data leads to serious energy consumption and time delays. In addition, in traditional sensor systems, a single sensor can usually only perceive information in a single mode. In order to achieve multimodal perception functions, multiple sensors usually have to be used to capture information in different modes separately, which leads to complex hardware design and increases energy consumption.

[0003] Compared with traditional sensor systems, the human perception system can not only process multi-sensory signals simultaneously, such as the fusion processing of visual and auditory signals, but also store information in situ and perform real-time pre-processing such as feature extraction on the information, and only transmit the extracted data to the brain for further analysis after eliminating redundant data, which has the characteristics of integrated sensing, storage and computing. The ability of multi-sensory signal fusion processing and integrated sensing, storage and computing enables the human brain to respond quickly in a changing environment with extremely high efficiency and extremely low power consumption.

[0004] Inspired by the human perception system, researchers have designed new brain-inspired sensing systems to mimic the multi-sensory information fusion processing and sense-storage-computation integration capabilities of the human perception system. For example, neuromorphic visual sensors can integrate the perception and preprocessing of light signals into a single device, realize in-sensor computing, eliminate the need for data transmission, and improve information processing efficiency. On the other hand, some studies have reported neuromorphic devices that integrate multimodal perception functions into a single device, but such research is still in its infancy. Sensor hardware that can integrate multimodal perception, signal preprocessing, and multi-level storage functions in a single device is still very rare. Summary of the invention

[0005] In view of this, the present invention provides a multifunctional photoelectric memristor device array and a preparation method thereof, so as to reduce time delay and energy consumption, simplify system design and improve work efficiency.

[0006] In the first aspect, the present invention provides a multifunctional photoelectric memristor device array, which is divided into two areas, one area is used to sense, store and process visual signals, and the other area is used to sense, store and process voice signals; the photoelectric memristor device array is externally connected to a neural network, wherein the neural network is a back propagation neural network; the processed visual signal and voice signal are fused and input into the neural network to complete the final recognition work.

[0007] In a second aspect, the present invention provides a method for preparing a multifunctional photoelectric memristor device array, wherein the photoelectric memristor device array comprises, from bottom to top, a substrate layer NSTO and a top electrode layer ITO;

[0008] The substrate layer is NSTO oxide semiconductor. The laser direct writing technology is used to define the pattern of the memristor device on the substrate layer, and then the pulsed laser deposition technology PLD is used to deposit transparent indium tin oxide ITO on it.

[0009] Optionally, 25 ITO / NSTO heterostructure optoelectronic synapse devices are prepared on the substrate layer NSTO to form a sensing array.

[0010] Optionally, a substrate layer NSTO is first provided, and then a pulsed laser deposition technology PLD is used to deposit a 100 nm to 500 nm thick ITO transparent electrode on the NSTO substrate, and the above steps are repeated to prepare a 5×5 photoelectric memristor device array.

[0011] Optionally, photoelectric memristor devices are prepared on the substrate layer at a distance of 240 μm to form an array.

[0012] Optionally, since the surface of the substrate layer NSTO contains oxygen vacancies, by applying an electric field or light pulse stimulation to the photoelectric memristor device, electrons will be captured or decaptured, thereby achieving resistance switching by regulating the Schottky barrier at the ITO / NSTO interface; after removing the electric field or light pulse, the captured or released electrons will be released or captured again by the oxygen vacancies, and the conductivity will spontaneously decay, thereby obtaining a photoelectric memristor device array that produces resistance switching and dynamically changing conductivity under the stimulation of both light and electrical signals.

[0013] In the technical solution provided by the present invention, the photoelectric memristor device array is divided into two areas, one area is used to sense, store and process visual signals, and the other area is used to sense, store and process voice signals; the photoelectric memristor device array is externally connected to a neural network, wherein the neural network is a back-propagation neural network; the processed visual signal and voice signal are fused and input into the neural network to complete the final recognition work, and the photoelectric memristor device array reduces time delay and energy consumption, simplifies system design, and improves work efficiency. BRIEF DESCRIPTION OF THE DRAWINGS

[0014] In order to more clearly illustrate the technical solutions of the embodiments of the present invention, the drawings required for use in the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0015] Figure 1 A schematic diagram of a multifunctional optoelectronic memristor device array provided by an embodiment of the present invention;

[0016] Figure 2 An array schematic diagram of a multifunctional optoelectronic memristor device array provided by an embodiment of the present invention;

[0017] Figure 3 A current-voltage curve diagram provided by an embodiment of the present invention;

[0018] Figure 4 A conductivity-voltage curve diagram provided by an embodiment of the present invention;

[0019] Figure 5 A schematic diagram of the retention characteristics of different resistance states of a photoelectric memristor device array over time provided by an embodiment of the present invention;

[0020] Figure 6 A schematic diagram of the response of the photoelectric memristor device array to light pulses of different wavelengths provided by an embodiment of the present invention;

[0021] Figure 7 A schematic diagram of a multimodal emotion sensor designed with a photoelectric memristor device array as the core provided in an embodiment of the present invention. DETAILED DESCRIPTION

[0022] In order to make the purpose, technical solution and advantages of the embodiments of the present invention clearer, the technical solution in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0023] It should be clear that the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0024] The terms used in the embodiments of the present invention are only for the purpose of describing specific embodiments, and are not intended to limit the present invention. The singular forms "a", "said" and "the" used in the embodiments of the present invention are also intended to include plural forms, unless the context clearly indicates other meanings.

[0025] It should be understood that the term "and / or" used in this article is only a description of the association relationship of associated objects, indicating that there can be three relationships. For example, A and / or B can represent: A exists alone, A and B exist at the same time, and B exists alone. In addition, the character " / " in this article generally indicates that the associated objects before and after are in an "or" relationship.

[0026] The word "if" as used herein may be interpreted as "at the time of" or "when" or "in response to determining" or "in response to detecting", depending on the context. Similarly, the phrases "if it is determined" or "if (stated condition or event) is detected" may be interpreted as "when it is determined" or "in response to determining" or "when detecting (stated condition or event)" or "in response to detecting (stated condition or event)", depending on the context.

[0027] Figure 1 A schematic diagram of a multifunctional optoelectronic memristor device array provided by an embodiment of the present invention, such as Figure 1 As shown, the photoelectric memristor device array is divided into two areas, one area is used to sense, store and process visual signals, and the other area is used to sense, store and process voice signals; the photoelectric memristor device array is externally connected to a neural network, wherein the neural network is a back propagation neural network; the processed visual signal and voice signal are fused and input into the neural network to complete the final recognition work.

[0028] The present invention provides a method for preparing a multifunctional photoelectric memristor device array, wherein the photoelectric memristor device array comprises, from bottom to top, a substrate layer NSTO and a top electrode layer ITO;

[0029] The substrate layer is NSTO oxide semiconductor. The laser direct writing technology is used to define the pattern of the memristor device on the substrate layer, and then the pulsed laser deposition technology PLD is used to deposit transparent indium tin oxide ITO on it.

[0030] In the embodiment of the present invention, 25 ITO / NSTO heterostructure optoelectronic synapse devices are prepared on the substrate layer NSTO to form a sensor array.

[0031] In the embodiment of the present invention, a substrate layer NSTO is first provided, and then a pulsed laser deposition technology PLD is used to deposit a 100nm-500nm thick ITO transparent electrode on the NSTO substrate, and the above steps are repeated to prepare a 5×5 photoelectric memristor device array.

[0032] In the embodiment of the present invention, Figure 2 As shown, photoelectric memristor devices are prepared on the substrate layer with a distance of 240 μm to form an array.

[0033] In the embodiment of the present invention, since the surface of the substrate layer NSTO contains oxygen vacancies, electrons will be captured or decaptured by applying an electric field or light pulse stimulation to the photoelectric memristor device, thereby achieving resistance change by regulating the Schottky barrier at the ITO / NSTO interface; after removing the electric field or light pulse, the captured or released electrons will be released or captured again by the oxygen vacancies, and the conductivity will spontaneously decay, thereby obtaining a photoelectric memristor device array that produces resistance change and dynamically changing conductivity under the stimulation of both light and electrical signals.

[0034] In the embodiment of the present invention, Figure 3 As shown in Figure 2, the photoelectric memristor device produces a resistive switching behavior under the stimulation of an electric pulse, that is, the photoelectric memristor device can be modulated by an electric pulse to multiple conductance states, such as Figure 4 The conductivity retention characteristics, such as Figure 5 As shown in the figure, after the electric pulse is removed, the conductivity relaxes spontaneously, but it does not relax to the initial conductivity state. The conductivity of the photoelectric memristor device shows semi-volatile characteristics, that is, volatility and non-volatility coexist. The photoelectric memristor device exhibits the characteristics of multi-level storage states. Among them, the volatile conductivity can be used to sense and process electrical signals, and the non-volatile conductivity can be used to store signals.

[0035] In the embodiment of the present invention, Figure 6 As shown in the figure, the photoelectric memristor device can also be modulated to different conductance states by a single optical pulse of 405nm, 660nm and 880nm with a width of 10s. The conductance state under optical pulse modulation is also volatile and non-volatile. Specifically, the device conductance relaxes after the optical pulse is removed, but cannot relax back to the initial state, indicating that the resistance change of the photoelectric memristor device under optical modulation is also volatile and non-volatile. Among them, the volatile conductance can be used to sense and process the optical signals in the environment, and the non-volatile conductance can be used to store signals.

[0036] After forming an array, the photoelectric memristor device can be used as brain-inspired sensor hardware to realize multimodal sensing, storage and computing functions. Taking the example of realizing emotion recognition tasks based on the fusion perception and processing of image and voice signals, under the joint action of light and electrical pulses, the array hardware can realize the parallel perception, processing and storage of multimodal signals.

[0037] Facial expressions and voice intonation are the two main ways that humans express emotions. Relying on only one expression can easily lead to misjudgment, while combining the two expressions is more accurate than recognizing only one. Figure 7A schematic diagram of a multimodal emotion sensor designed with a photoelectric memristor device array as the core provided in an embodiment of the present invention, the sensor includes an input layer, a sensing layer and an output layer. The input layer is the expression picture signal and the voice signal. The sensing layer is the photoelectric memristor device array prepared by the present invention, and is also the core of the entire emotion perception system. It utilizes the nonlinearity of the device and the short-term dynamic response characteristics of the device to light and electrical pulses to respectively realize feature extraction and simplified representation of image and sound signals, that is, to realize encoding (perception and preprocessing) of image and sound signals. The output layer is a neural network. Here we use the response of the photoelectric memristor array to light pulses to perceive and process visual information, and use its response to electrical pulses to perceive and process auditory information. As Figure 7 As shown in , facial expression images are encoded by light pulses and speech signals are encoded by electrical pulses, as shown in Figure 7 As shown in , the perception layer is designed to process facial image expressions and sound signals simultaneously. In the initial state, all ITO / NSTO devices in the sensing layer are set to the same conductance state by a negative voltage pulse of -2.2V for 40.00μS. During the learning process, the perception layer receives facial expression and sound information and converts these inputs into a simplified feature space through its short-term dynamic nonlinearity. Then, these final device conductances containing visual and auditory information are input to the output layer of the sensing system to perform the recognition task.

[0038] Facial expression images, including smiling and sad faces, are represented by pixel images. Each row of the image is encoded by a series of light pulses, where the pixel area corresponds to the light pulse (wavelength 660nm, duration 4s, interval 5s), and white corresponds to no pulse. The image is converted into a sequence of light pulses, which is used to stimulate the devices in the sensing layer to generate photocurrent, which is converted into device conductance values ​​to replace the original facial expression information. Subsequently, the collection of device conductance states in the sensing layer is input to the output layer, which is a simulated neural network based on the back-propagation algorithm for facial expression recognition.

[0039] For the sound signal, two sounds were used: an excited sound and a frustrated sound. The original sound signal was recorded by the experimenter, saying "oh" in an excited voice and a frustrated voice, corresponding to the happy and sad states, respectively. The process of converting the sound signal into a voltage pulse is as follows: first, the original audio waveform is converted into a spectrogram, and then the magnitude of the frequency is converted into a voltage value after scaling and translation. These voltages are input into Figure 7 In the sensing layer device shown in the figure, the current value corresponding to the sequential voltage pulse is collected (the current value is used to represent the conductivity state), and it is used to replace the original voice signal and input into the output layer, finally achieving 100% recognition accuracy of the voice signal.

[0040] Finally, an “AND” operation is performed based on the emotion recognition results of the facial expression image and the sound signal. The system will only make an emotional judgment when the two results are consistent, effectively avoiding misjudgments caused by single-modal perception. This scenario takes advantage of the unique advantages of our device, such as multimodal perception fusion, in-sensor computing power, and dynamic conductivity characteristics, to achieve efficient and high-precision processing within a simplified hardware architecture.

[0041] In the technical solution provided by the present invention, the photoelectric memristor device array is divided into two areas, one area is used to sense, store and process visual signals, and the other area is used to sense, store and process voice signals; the photoelectric memristor device array is externally connected to a neural network, wherein the neural network is a back-propagation neural network; the processed visual signal and voice signal are fused and input into the neural network to complete the final recognition work, and the photoelectric memristor device array reduces time delay and energy consumption, simplifies system design, and improves work efficiency.

[0042] The above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. Any modifications, equivalent substitutions, improvements, etc. made within the spirit and principles of the present invention should be included in the scope of protection of the present invention.

Claims

1. A multifunctional photoelectric memristor device array, characterized in that: The photoelectric memristor device array is divided into two areas, one area is used to sense, store and process visual signals, and the other area is used to sense, store and process voice signals; the photoelectric memristor device array is externally connected to a neural network, wherein the neural network is a back propagation neural network; the processed visual signal and voice signal are fused and input into the neural network to complete the final recognition work.

2. A method for preparing a multifunctional photoelectric memristor device array, characterized in that: The photoelectric memristor device array comprises a substrate layer NSTO and a top electrode layer ITO from bottom to top; The substrate layer is NSTO oxide semiconductor. The laser direct writing technology is used to define the pattern of the memristor device on the substrate layer, and then the pulsed laser deposition technology PLD is used to deposit transparent indium tin oxide ITO on it.

3. The method according to claim 2, characterized in that 25 ITO / NSTO heterostructure optoelectronic synapse devices were prepared on the substrate layer NSTO to form a sensing array.

4. The method according to claim 2, characterized in that: First, a substrate layer NSTO is provided, and then a pulsed laser deposition technology PLD is used to deposit a 100nm-500nm thick ITO transparent electrode on the NSTO substrate, and the above steps are repeated to prepare a 5×5 photoelectric memristor device array.

5. The method according to claim 4, characterized in that Photoelectric memristor devices are prepared on the substrate layer at a distance of 240 μm to form an array.

6. The method according to claim 4, characterized in that Since the surface of the substrate layer NSTO contains oxygen vacancies, electrons will be captured or decaptured by applying an electric field or light pulse stimulation to the photoelectric memristor device, thereby achieving resistance change by regulating the Schottky barrier at the ITO / NSTO interface; after removing the electric field or light pulse, the captured or released electrons will be released or captured again by the oxygen vacancies, and the conductivity will spontaneously decay, thereby obtaining a photoelectric memristor device array that produces resistance change and dynamically changing conductivity under the stimulation of both light and electrical signals.

Images