A multimodal hybrid memristor neural network edge computing terminal system and method

Abstract

This invention discloses a multimodal hybrid memristor neural network edge computing terminal system and method, belonging to the field of artificial intelligence technology. This system addresses the shortcomings of existing edge computing in terms of high energy efficiency and real-time multimodal data processing. It integrates storage and computation through memristors, reducing the bottleneck caused by the separation of storage and computation in traditional computing structures. Utilizing the adaptive conductivity and Hebbian plasticity of the biomimetic memristor neural network structure, it achieves multimodal data fusion and adaptive learning, adapting to complex and ever-changing application environments. The core module can directly input the voltage analog matrix collected by the multimodal sensor module into the memristor neural network in the edge computing module via the servo light source training module. Then, through memristor neural networks of different dimensions, it fuses and classifies the multimodal sensor data, generating reliable decisions to guide the actions of the execution module, making it suitable for efficient, low-power intelligent edge applications.

Description

Technical Field

[0001] This invention relates to the field of artificial intelligence technology, specifically to a multimodal hybrid memristor neural network edge computing terminal system and method. Background Technology

[0002] In the field of artificial intelligence, especially for the training and inference processes of large-scale neural networks and deep learning models, the rapid advancement of computing technology is facing severe challenges in terms of computing power and energy efficiency. The traditional von Neumann architecture, due to its inherent separation of computation and storage, has given rise to the so-called "memory wall" problem. This bottleneck not only seriously hinders the further improvement of system computing power, but also leads to a sharp increase in power consumption, becoming a key factor restricting the efficient operation of large-scale artificial intelligence models.

[0003] To address this challenge, memristors emerged as a revolutionary electronic component. Their unique feature lies in their ability to simultaneously perform storage and computation functions, achieving integrated data storage and processing in the same physical location, thus completely circumventing the limitations imposed by the separation of computation and storage. Compared to traditional computing architectures, memristors can directly execute core matrix operations in neural networks, providing powerful support for large-scale parallel computing tasks. This design not only significantly reduces data transmission frequency and energy consumption but also substantially improves computational efficiency at the hardware level, opening up entirely new paths for artificial intelligence applications that pursue high-efficiency computing and low-power consumption.

[0004] Against this backdrop, HP Labs innovatively proposed the Crossbar architecture based on memristors. This architecture cleverly employs a cross-grid layout, precisely placing memristors at the intersection nodes of the grid, achieving seamless parallel processing of computation and storage within the same physical space. However, it is worth noting that the current Crossbar architecture still uses a traditional control strategy, with its weight assignment process still relying on traditional computers for decision-making. While this compromise improves computational efficiency to some extent, it fails to fully exploit the potential of memristors' in-memory computing capabilities, especially in terms of energy consumption optimization, where significant room for improvement remains.

[0005] On the other hand, the multimodal mixture problem, as a complex challenge in the field of computing systems and models, requires the effective fusion and comprehensive analysis of data from multiple modalities such as vision, hearing, temperature, and vibration to reveal the intrinsic connections and mutual influences between the modalities, thereby making more accurate and robust decisions or predictions. This problem has wide application value in many fields such as intelligent edge computing, machine learning, and artificial intelligence. In particular, in scenarios such as image and speech fusion and multimodal data processing in sensor networks, a single modality often cannot fully and accurately express complex information, making the introduction of multimodal mixture technology particularly important.

[0006] Currently, multimodal fusion methods mainly fall into three categories: feature-level fusion, decision-level fusion, and data-level fusion. Each category exhibits different advantages and limitations in information preservation, noise processing, and modal coupling, making them suitable for various application scenarios. However, with the increasing diversity of modal types and the continuous improvement in task complexity, existing methods still fall short in addressing challenges related to heterogeneous data compatibility, adaptability, and fusion efficiency. They struggle to meet the demands for real-time performance and efficient computation in complex environments, necessitating the emergence of a more advanced and comprehensive multimodal fusion technology. Summary of the Invention

[0007] The purpose of this invention is to provide a multimodal hybrid memristor neural network edge computing terminal system and method to address the shortcomings of existing edge computing in terms of high energy efficiency and real-time multimodal data processing.

[0008] The present invention solves the above-mentioned technical problems through the following technical solution:

[0009] A multimodal hybrid memristor neural network edge computing terminal system includes, along the signal processing direction, a multimodal sensor module, a servo light source training module, an edge computing module, an embedded processor, and an execution module. The terminal system also includes peripheral circuits, a cloud platform big data center, and a human-computer interaction module. The servo light source training module, the edge computing module, and the embedded processor constitute the core module. The core module also includes a wireless communication module. The cloud platform big data center and the embedded processor are bidirectionally interconnected through the wireless communication module.

[0010] The peripheral circuits are connected to the servo light source training module, the edge computing module, and the embedded processor to support the operation of the system terminal; the multimodal sensor module is used to acquire multimodal voltage analog matrices.

[0011] The edge computing module is a multimodal hybrid memristor neural network, which consists of several parallel memristor neural networks. The number of memristor neural networks corresponds one-to-one with the number of modal inputs. The human-computer interaction module includes a touch screen, a keyboard matrix, and a visualization module, which are interconnected with the embedded processor to realize system result visualization and human-computer interaction functions.

[0012] A further improvement of the present invention is that the peripheral circuit specifically includes a signal processing module, a multi-level power management module, and a high-speed storage module. The signal processing module is used to filter and reduce noise and sample and hold the system input signal; the multi-level power management module is used to provide power supply at different voltage levels; and the high-speed storage module is used to store the data during the operation of the terminal system.

[0013] A further improvement of this invention is that the memristor neural network is one or more of a one-dimensional memristor neural network, a two-dimensional memristor neural network, or a multi-dimensional memristor neural network, used to support the training requirements of modal data of different dimensions.

[0014] A further improvement of the present invention is that: the one-dimensional memristor neural network includes a plurality of memristor neurons, a negative driving source, a directional driving source and a positive driving source;

[0015] The memristor neuron includes an input second-order photosensitive memristor group and a bias second-order photosensitive memristor group. The input second-order photosensitive memristor group consists of a first input memristor switch, a first positive second-order photosensitive memristor, a second input memristor switch, and a second positive second-order photosensitive memristor connected in sequence.

[0016] The biased second-order photosensitive memristor group consists of a first biased memristor switch, a first reverse-connected second-order photosensitive memristor, a second biased memristor switch, and a second reverse-connected second-order photosensitive memristor connected in sequence.

[0017] The positive drive source is connected to the first input memristor switch and the first bias memristor switch respectively; the negative drive source is connected to the second positive second-order photosensitive memristor and the second reverse second-order photosensitive memristor respectively; the connection point between the first positive second-order photosensitive memristor and the second input memristor switch and the connection point between the first reverse second-order photosensitive memristor and the second bias memristor switch are connected to the directional drive source, the directional drive source is connected to the output bus, and the output node of the output bus is the knowledge output point of the one-dimensional memristor neural network.

[0018] A further improvement of this invention is that: the two-dimensional memristor neural network includes a plurality of memristor neurons, a negative driving source, a directional driving source, and a positive driving source; the memristor neurons include a first second-order photosensitive memristor, a second second-order photosensitive memristor, a third second-order photosensitive memristor, and a fourth second-order photosensitive memristor.

[0019] The positive terminals of the first and second order photoresistors and the third and second order photoresistors are connected to a positive driving source, and the negative terminals of the first and second order photoresistors and the third and second order photoresistors are connected to a directional driving source.

[0020] The positive terminals of the second-order photoresistor and the fourth-order photoresistor are connected to a directional driving source, and the negative terminals of the second-order photoresistor and the fourth-order photoresistor are connected to a negative driving source.

[0021] Memristor neurons are connected to directional driving sources in a self-similar I-shaped structure via switching switches. The directional driving sources are connected to the output bus, and the output nodes of the output bus are the knowledge output points of the two-dimensional memristor neural network.

[0022] A further improvement of this invention is that the multidimensional memristor neural network is composed of several parallel two-dimensional memristor neural networks.

[0023] The knowledge output points of each two-dimensional memristor neural network are connected in parallel on the output bus, and the output nodes of the output bus are the knowledge output points of the multi-dimensional memristor neural network.

[0024] A further improvement of the present invention is that the servo light source training module includes a light source matrix, wherein the point light sources in the light source matrix correspond one-to-one with the second-order photosensitive memristors in the multimodal hybrid memristor neural network, and is used to adjust the illumination intensity and duration of the point light sources according to the multimodal voltage simulation matrix to realize the initial training of the multimodal hybrid memristor neural network.

[0025] A multimodal hybrid memristor neural network edge computing method, employing the aforementioned multimodal hybrid memristor neural network edge computing terminal system, includes the following steps:

[0026] Step 1: Based on the modalities involved in the actual problem, determine the number and dimensions of the modal inputs of the multimodal sensor module, and determine the number and dimensions of the memristor neural network based on the number and dimensions of the modal inputs; obtain the modal type of the acquired signal, and acquire the multimodal voltage simulation matrix;

[0027] Step 2: Filter and denoise the multimode voltage simulation matrix and perform sample-and-hold operations to obtain the processed multimode voltage simulation matrix;

[0028] Step 3: The servo light source training module adjusts the illumination intensity and illumination time of the point light source according to the processed multimodal voltage simulation matrix, stimulating the memristor neural network to perform initial assignment and realize bionic learning.

[0029] Step 4: By controlling the state of each memristor neural network, read the training results of a single network for a single mode to obtain the independent output knowledge points of the memristor neural network under each mode; based on current driving and Heb rule, adaptively adjust the weights of the memristor neural network to realize the self-coordinated update of the memristor neural network under each mode, simulating bionic forgetting.

[0030] Step 5: By controlling the hybrid switching of the multimodal hybrid memristor neural network, read the hybrid output knowledge points after multimodal fusion; adaptively adjust the weights of the multimodal hybrid memristor neural network based on current driving and Heb rule to achieve adaptive fusion of multimodal information; generate personalized decision commands based on the confidence matrix formed by the independent output knowledge points and the hybrid output knowledge points.

[0031] Step 6: Pass the decision command to the execution module to guide the specific movement of the execution module, and end the main process.

[0032] A further improvement of the present invention is that, after performing step three or step four, the following step is also included:

[0033] The terminal system stores runtime data via peripheral circuits; it uploads the process and results data of the terminal system to the cloud platform's big data center for historical data accumulation and analysis to support training strategy updates and parameter optimization, enabling remote maintenance and intelligent management; it displays the process and results data of the terminal system through a human-computer interaction module; and users control and view the terminal system's training parameters and visualizations through the human-computer interaction module.

[0034] A further improvement of the present invention is that, after step six is ​​completed, the following steps are also included: uploading the process and result data of the terminal system operation to the cloud platform big data center for historical data accumulation and analysis to support training strategy updates and parameter optimization, and to realize remote maintenance and intelligent management; and displaying the process and result data of the terminal system operation through the human-computer interaction module.

[0035] Compared with the prior art, the positive and progressive effects of the present invention are as follows:

[0036] This invention provides a multimodal hybrid memristor neural network edge computing terminal system that addresses the shortcomings of existing edge computing in terms of high energy efficiency and real-time multimodal data processing. By integrating storage and computation through a multimodal hybrid memristor neural network, it reduces data transmission bottlenecks caused by the separation of storage and computation in traditional computing architectures, thereby improving the overall energy efficiency and computational efficiency of the system. Utilizing the adaptive conductivity and Hebbian plasticity of the biomimetic memristor neural network structure, the system can dynamically adjust network parameters to achieve multimodal data fusion and adaptive learning, enhancing the system's flexibility and adaptability. The integrated design of the core modules allows the system to directly input data collected by the multimodal sensor module into the memristor neural network in the edge computing module, simplifying the data processing flow and improving the system's real-time performance. Connection to the cloud platform's big data center enables the system to remotely store and analyze historical data, support training strategy updates and parameter optimization, and achieve remote maintenance and intelligent management. The human-computer interaction module provides a user-friendly interface, allowing users to easily control and view the system's training parameters and visualizations.

[0037] Furthermore, the introduction of peripheral circuit modules enables functions such as filtering and noise reduction of system input signals, sample and hold, and data storage, thereby further improving the stability and reliability of the system.

[0038] Furthermore, the system employs a multimodal hybrid memristor neural network, which can simultaneously process data from multiple modalities and fuse and classify data through memristor neural networks of different dimensions to generate reliable decisions. This enables the system to be applicable to complex and ever-changing application environments and improves the system's intelligence level.

[0039] The multimodal hybrid memristor neural network edge computing method provided by this invention reduces data transmission, optimizes the computing structure, and utilizes the in-memory computing characteristics of memristors. As a result, the system can maintain high performance while reducing power consumption and improving the energy efficiency and real-time performance of edge computing. Attached Figure Description

[0040] The accompanying drawings are provided to further understand the invention and constitute a part of this invention. The illustrative embodiments of the invention and their descriptions are used to explain the invention and do not constitute an improper limitation of the invention.

[0041] Figure 1 This is a schematic diagram of the structure of the present invention;

[0042] Figure 2 This is a schematic diagram of the structure of a multimodal hybrid memristor neural network;

[0043] Figure 3 This is a schematic diagram of the structure of a one-dimensional memristor neural network;

[0044] Figure 4 This is a schematic diagram of the structure of a two-dimensional memristor neural network;

[0045] Figure 5 This is a schematic diagram of the structure of a multidimensional memristor neural network;

[0046] Figure 6 This is a schematic diagram of the process of the present invention;

[0047] The module includes: Core Module-10, Embedded Processor-11, Servo Light Source Training Module-12, Edge Computing Module-13, Wireless Communication Module-14, Peripheral Circuits-20, Multi-level Power Management Module-21, Signal Processing Module-22, High-speed Storage Module-23, Cloud Platform Big Data Center-30, Human-Computer Interaction Module-40, Touch Display Screen-41, Keyboard Matrix-42, Visualization Module-43, Multimodal Sensor Module-50, Execution Module-60, Positive Drive Source-70, Directional Drive Source-80, Negative Drive Source-90, Memristor Neural Network-100, Modal Input-101, Positive Memristor Neural Network-102, and Positive Memristor Neural Network. Network independent switch-103, Network positive bias part and negative memristor neural network-104, Negative memristor neural network independent switch-105, Knowledge output point-106, One-dimensional memristor neural network-110, Positive connection second-order photosensitive memristor-111, Input memristor switch-112, Reverse connection second-order photosensitive memristor-113, Bias memristor switch-114, Two-dimensional memristor neural network-120, Memristor neuron-121, First second-order photosensitive memristor-122, Second second-order photosensitive memristor-123, Third second-order photosensitive memristor-124, Fourth second-order photosensitive memristor-125, Switch-on / off switch-126, Multidimensional memristor neural network-130. Detailed Implementation

[0048] To make the objectives, technical solutions, and advantages of the embodiments of the present invention clearer, the technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. The components of the embodiments of the present invention described and shown in the accompanying drawings can generally be arranged and designed in various different configurations.

[0049] Therefore, the following detailed description of the embodiments of the invention provided in the accompanying drawings is not intended to limit the scope of the claimed invention, but merely to illustrate selected embodiments of the invention. All other embodiments obtained by those skilled in the art based on the embodiments of the invention without inventive effort are within the scope of protection of the invention.

[0050] It should be noted that similar reference numerals and letters denote similar items in the following drawings, and therefore, once an item is defined in one drawing, it does not need to be further defined or explained in subsequent drawings.

[0051] In the description of the embodiments of the present invention, it should be noted that if terms such as "upper," "lower," "horizontal," or "inner" indicate the orientation or positional relationship based on the orientation or positional relationship shown in the accompanying drawings, or the orientation or positional relationship commonly used when the product of the invention is in use, they are only for the convenience of describing the present invention and simplifying the description, and do not indicate or imply that the device or element referred to must have a specific orientation, or be constructed and operated in a specific orientation, and therefore should not be construed as a limitation of the present invention. Furthermore, terms such as "first" and "second" are only used to distinguish descriptions and should not be construed as indicating or implying relative importance.

[0052] Furthermore, it should be noted that, unless otherwise explicitly specified and limited, the terms "set," "install," "connect," and "link" should be interpreted broadly. For example, they can refer to a fixed connection, a detachable connection, or an integral connection; they can refer to a mechanical connection or an electrical connection; they can refer to a direct connection or an indirect connection through an intermediate medium; and they can refer to the internal connection of two components. Those skilled in the art can understand the specific meaning of the above terms in this invention according to the specific circumstances.

[0053] The present invention will be further described in detail below with reference to the accompanying drawings and specific embodiments. This is an explanation of the present invention and not a limitation thereof.

[0054] This invention provides a multimodal hybrid memristor neural network edge computing terminal system, which, along the signal processing direction, sequentially includes a multimodal sensor module 50, a servo light source training module 12, an edge computing module 13, an embedded processor 11, and an execution module 60. The terminal system also includes peripheral circuits 20, a cloud platform big data center 30, and a human-computer interaction module 40. The servo light source training module 12, the edge computing module 13, and the embedded processor 11 constitute a core module 10. The core module 10 also includes a wireless communication module 14. The cloud platform big data center 30 and the embedded processor 11 are bidirectionally interconnected through the wireless communication module 14.

[0055] The peripheral circuit 20 is connected to the servo light source training module 12, the edge computing module 13 and the embedded processor 11 respectively to support the operation of the system terminal; the multimodal sensor module 50 is used to acquire multimodal voltage analog matrices;

[0056] The edge computing module 13 is a multimodal hybrid memristor neural network, which consists of several parallel memristor neural networks 100. The number of memristor neural networks 100 corresponds one-to-one with the number of modal inputs 101. The human-computer interaction module 40 includes a touch screen 41, a keyboard matrix 42, and a visualization module 43, which are interconnected with the embedded processor 11 to realize system result visualization and human-computer interaction functions.

[0057] Specifically, the peripheral circuit 20 includes a signal processing module 22, a multi-level power management module 21, and a high-speed storage module 23. The signal processing module 22 is used to filter and reduce noise and sample and hold the system input signal; the multi-level power management module 21 is used to provide power supply at different voltage levels; and the high-speed storage module 23 is used to store the data during the operation of the terminal system.

[0058] Specifically, the memristor neural network 100 is one or more of a one-dimensional memristor neural network 110, a two-dimensional memristor neural network 120, or a multi-dimensional memristor neural network 130, used to support the training requirements of modal data of different dimensions.

[0059] Specifically, the one-dimensional memristor neural network 110 includes a number of memristor neurons 121, negative driving sources 90, directional driving sources 80, and positive driving sources 70.

[0060] The memristor neuron 121 includes an input second-order photosensitive memristor group and a bias second-order photosensitive memristor group. The input second-order photosensitive memristor group consists of a first input memristor switch 112, a first positive second-order photosensitive memristor 111, a second input memristor switch 112, and a second positive second-order photosensitive memristor 111 connected in sequence.

[0061] The biased second-order photosensitive memristor group consists of a first biased memristor switch 114, a first reverse-connected second-order photosensitive memristor 113, a second biased memristor switch 114, and a second reverse-connected second-order photosensitive memristor 113 connected in sequence.

[0062] The positive drive source 70 is connected to the first input memristor switch 112 and the first bias memristor switch 114 respectively; the negative drive source 90 is connected to the second positive second-order photosensitive memristor 111 and the second reverse second-order photosensitive memristor 113 respectively; the connection point between the first positive second-order photosensitive memristor 111 and the second input memristor switch 112 and the connection point between the first reverse second-order photosensitive memristor 113 and the second bias memristor switch 114 are connected to the directional drive source 80, the directional drive source 80 is connected to the output bus, and the output node of the output bus is the knowledge output point 106 of the one-dimensional memristor neural network 110.

[0063] Specifically, the two-dimensional memristor neural network 120 includes a plurality of memristor neurons 121, a negative driving source 90, a directional driving source 80, and a positive driving source 70. The memristor neurons 121 include a first second-order photosensitive memristor 122, a second second-order photosensitive memristor 123, a third second-order photosensitive memristor 124, and a fourth second-order photosensitive memristor 125.

[0064] The positive terminals of the first and second order photoresistors 122 and the third and second order photoresistors 124 are connected to the positive driving source 70, and the negative terminals of the first and second order photoresistors 122 and the third and second order photoresistors 124 are connected to the directional driving source 80.

[0065] The positive terminals of the second-order photoresistor 123 and the fourth-order photoresistor 125 are connected to the directional driving source 80, and the negative terminals of the second-order photoresistor 123 and the fourth-order photoresistor 125 are connected to the negative driving source 90.

[0066] Memristor neurons 121 are connected to directional driving sources 80 in a self-similar I-shaped structure via switching switches 126. The directional driving sources 80 are connected to the output bus, and the output nodes of the output bus are the knowledge output points 106 of the two-dimensional memristor neural network 120.

[0067] Specifically, the multidimensional memristor neural network 130 consists of several parallel two-dimensional memristor neural networks 120.

[0068] The knowledge output points 106 of each two-dimensional memristor neural network 120 are connected in parallel on the output bus, and the output nodes of the output bus are the knowledge output points 106 of the multi-dimensional memristor neural network 130.

[0069] Specifically, the servo light source training module 12 includes a light source matrix, in which point light sources correspond one-to-one with second-order photosensitive memristors in the multimodal hybrid memristor neural network. This module is used to adjust the illumination intensity and duration of the point light sources according to the multimodal voltage simulation matrix, thereby achieving the initial training of the multimodal hybrid memristor neural network.

[0070] The terminal system provided by this invention aims to address the shortcomings of existing edge computing in terms of high energy efficiency and real-time multimodal data processing. It integrates storage and computation through memristors, reducing the bottleneck caused by the separation of storage and computation in traditional computing structures. Utilizing the adaptive conductivity and Hebbian plasticity of the biomimetic memristor neural network structure, it achieves the fusion and adaptive learning of multimodal data, adapting to complex and ever-changing application environments. The edge computing terminal system consists of a core module, a peripheral circuit module, a cloud platform big data center, a human-computer interaction module, a multimodal sensor module, and an execution module. The core module can directly input the voltage analog matrix collected by the multimodal sensor module into the memristor neural network in the edge computing module via the servo light source training module. Then, the multimodal sensor data is fused and classified through memristor neural networks of different dimensions to generate reliable decisions and guide the actions of the execution module. It is suitable for efficient and low-power intelligent edge applications.

[0071] See Figure 6 Based on the same inventive concept, this invention provides a multimodal hybrid memristor neural network edge computing method, employing the aforementioned multimodal hybrid memristor neural network edge computing terminal system, including the following steps:

[0072] Step 1: Based on the modalities involved in the actual problem, determine the number and dimensions of the modal inputs 101 of the multimodal sensor module 50, and determine the number and dimensions of the memristor neural network 100 based on the number and dimensions of the modal inputs 101; obtain the modal type of the acquired signal, and acquire the multimodal voltage analog matrix;

[0073] Step 2: Filter and denoise the multimode voltage simulation matrix and perform sample-and-hold operations to obtain the processed multimode voltage simulation matrix;

[0074] Step 3: The servo light source training module 12 adjusts the illumination intensity and illumination time of the point light source according to the processed multimodal voltage simulation matrix, stimulating the memristor neural network 100 to perform initial assignment and realize bionic learning.

[0075] Step 4: By controlling the state of each memristor neural network 100, read the training results of a single network for a single mode to obtain the independent output knowledge points of the memristor neural network 100 under each mode; based on current driving and Heb rule, adaptively adjust the weights of the memristor neural network 100 to achieve self-coordinated updating of the memristor neural network 100 under each mode, simulating bionic forgetting.

[0076] Step 5: By controlling the hybrid switching of the multimodal hybrid memristor neural network, read the hybrid output knowledge points after multimodal fusion; adaptively adjust the weights of the multimodal hybrid memristor neural network based on current driving and Heb rule to achieve adaptive fusion of multimodal information; generate personalized decision commands based on the confidence matrix formed by the independent output knowledge points and the hybrid output knowledge points.

[0077] Step 6: Transmit the decision command to the execution module 60 to guide the specific movement of the execution module 60, and end the main process.

[0078] In a specific embodiment of the present invention, step three is specifically as follows: adjusting the illumination intensity and illumination time of the point light source according to the processed multimodal voltage simulation matrix, stimulating the corresponding second-order photosensitive memristor to change the conductance value, thereby realizing the input of the processed multimodal voltage simulation into the memristor neural network 100 to achieve biomimetic learning.

[0079] In a specific embodiment of the present invention, step four specifically involves: by controlling the on / off states of the positive driving source 70, the directional driving source 80, and the negative driving source 90, and the switching of the memristor neuron 121, reading the training results of a single network for a single modality, and obtaining the independent output knowledge points of the memristor neural network 100 under each modality; by stimulating the second-order photosensitive memristor with current driving, and adaptively adjusting the conductance value based on the Heb rule, the self-coordinated update of the network is realized, simulating bionic forgetting.

[0080] In a specific embodiment of the present invention, step five is as follows: by controlling the hybrid switching of the multimodal hybrid memristor neural network, the hybrid output knowledge points after multimodal fusion are read; by stimulating the memristor neurons 121 with current driving, the conductance value is adaptively adjusted based on the Heb rule to realize the adaptive fusion of multimodal information; and based on the confidence matrix formed by the independent output knowledge points and the hybrid output knowledge points, personalized decision commands are generated.

[0081] This method employs a parallel and scalable architecture, with each memristor neural network independently corresponding to a specific modality of input data. The number of network modules is dynamically adjusted according to the modal requirements of the actual problem. Initial training of each memristor neural network module is performed using a servo light source module, directly inputting simplified analog voltage signals from peripheral circuits into the network to achieve biomimetic learning. During training, specific memristor neurons in the network are dynamically connected via electric drive control, training results are read, and weights are automatically adjusted according to the Heb rule to achieve biomimetic memory and forgetting, thus possessing adaptive learning capabilities. In multimodal information processing, the system generates a confidence voltage matrix that fuses multimodal outputs by controlling the independent connections and coupling operations of the memristor neural network modules. This matrix not only reflects the independent signal characteristics of each modality but also adaptively enhances the coupling between modalities, achieving efficient integration of multimodal information and ultimately achieving efficient edge computing capabilities for complex tasks.

[0082] Specifically, after completing step three or four, the following steps are also included:

[0083] The peripheral circuit 20 stores the data of the terminal system during operation; the process and result data of the terminal system operation are uploaded to the cloud platform big data center 30 for historical data accumulation and analysis to support training strategy updates and parameter optimization, and realize remote maintenance and intelligent management; the process and result data of the terminal system operation are displayed through the human-computer interaction module 40; users control and view the training parameters and visualization content of the terminal system through the human-computer interaction module 40.

[0084] Specifically, after step six is ​​completed, the following steps are also included: uploading the process and result data of the terminal system operation to the cloud platform big data center 30 for historical data accumulation and analysis to support training strategy updates and parameter optimization, and to realize remote maintenance and intelligent management; displaying the process and result data of the terminal system operation through the human-computer interaction module 40.

[0085] See Figure 1 A multimodal hybrid memristor neural network edge computing terminal system is disclosed. The core module 10 is the nerve center of the entire terminal system. Specifically, the edge computing module 13 uses a memristor neural network 100 as the computing core. The servo light source module 12 controls the illumination intensity and illumination time of each light source in the light source matrix according to the input of the voltage analog quantity, thereby realizing the training of the second-order photosensitive memristor matrix. The embedded processor 11 controls the opening and closing of the memristor neural network and its memristor neurons, judges the generated confidence matrix, and finally forms a decision command.

[0086] The core module 10 includes a servo light source training module 12, an edge computing module 13, an embedded processor 11, and a wireless communication module 14. It possesses online training and autonomous learning capabilities, and can update the memristor conductance value in real time based on data transmitted from the afferent nerve fibers, achieving efficient edge computing and data processing. Decision results are transmitted to the actuator via the efferent nerve fibers, which serve as the data transmission medium. The peripheral circuit module 20 supports the stable operation of the memristor neural network, including a multi-level power management module 21 to meet the driving requirements of different voltage conditions for each module within the system; a signal processing module 22 for simple filtering, noise reduction, and sample-and-hold of the signals input to the core module; a high-speed storage module 23 for saving important results and process data during system operation, along with level conversion and signal isolation circuits; and a cloud platform big data center 30 for remote data storage, analysis, and model optimization of the terminal system. Through the cloud platform big data center 30, historical data can be accumulated and analyzed to provide functions such as training strategy updates and parameter optimization, enabling remote maintenance and intelligent management of edge computing terminals. The human-computer interaction module 40 includes a display screen 41, a keyboard control panel, and other result visualization modules such as LEDs, facilitating user operation and monitoring, and enabling intuitive interaction with functions such as data acquisition, model parameter adjustment, and task target correction. The multimodal sensor module 50 and the execution module 60 can be customized according to different target tasks. The multimodal sensor module 50, as a neural system receptor structure, can mix multiple types of sensors and input the results into the core module in the form of a multi-dimensional voltage analog quantity matrix. The execution module 60, as a neural system effector structure, can execute specific tasks according to the decision commands output by the core module 10, such as realizing functions such as fine grasping and posture adjustment, with high responsiveness and execution accuracy.

[0087] The multimodal hybrid memristor neural network, as the core of the edge computing module 13, is structured as a memristor neural network 100 composed of several memgroup neural networks 100 connected in parallel, controlled by three driving sources. Each memgroup neural network 100 corresponds to a specific modality of data in the actual problem. In multimodal hybrid processing, each memristor neural network is first independently trained on its specific modality of data, and the training result serves as the independent knowledge output point of each memristor neural network element. Subsequently, the independently trained memristor neural networks are combined and activated. See also Figure 2In practical applications, each memristor neural network 100 is equivalent to a positive and negative half structure composed of upper and lower biases. It can be viewed as a network consisting of a positive bias portion (formed by a positive memristor neural network 102 and independent switches 103) and a negative bias portion (formed by a negative memristor neural network 104 and independent switches 105). The output is considered symmetrical between the positive and negative driving sources. When positive and negative voltages are conducting, the magnitude and direction of the output voltage of the directional driving source are jointly determined by the biases of the positive and negative half structures. During multimodal fusion, the weights of each mode are determined by the size of the corresponding memristor neural network 100. Based on the structural characteristics of the memristor neural network 100, when the networks are all conducting, the memristor neurons between different memristor neural networks will adaptively adjust their weights according to the Heblin rule, thus forming a coupling relationship between the training results of multiple independent modes. This makes multimodal mixing no longer a simple weighted summation, but a multimodal fusion with a real coupling effect, resulting in a mixed output of knowledge points. Finally, the system combines the knowledge points independently output by the memristor neural network 100 with the knowledge points output in combination to generate a decision confidence matrix. The embedded processor 11 then performs decision analysis to generate control commands that guide the execution module.

[0088] The Memristor Neural Network 100 is constructed from memristor neurons based on second-order photosensitive memristors, simulating the functional structure of neurons in biological neural networks. By applying appropriate external optical and electrical feedback, it achieves an adaptive learning mechanism based on the Hebbian rule in biological neural networks, exhibiting positive and negative Hebbian plasticity at the circuit level. To enhance the network's adaptability, various topologies are designed to support data inputs of different dimensions and forms, and it is compatible with multiple types of sensor signal sources. The memristor neurons in the network are arranged in a self-similar structure, and the bias state of the network is dynamically adjusted by regulating the conductance of the photosensitive memristor to adapt to changes in external information, achieving effective processing of complex multimodal data.

[0089] See Figures 3-5This invention designs a one-dimensional memristor neural network 110, a two-dimensional memristor neural network 120, and a multi-dimensional memristor neural network 130 for voltage simulation matrices with different modal inputs. The basic building blocks are memristor neurons constructed based on second-order photosensitive memristors. The working principle of a second-order photosensitive memristor is based on the core characteristic of memristors: its conductance changes with the application of voltage or current. Unlike ordinary memristors, the conductance state of a second-order photosensitive memristor is also affected by the dynamic adjustment of light intensity, making its "second-order" characteristic dependent not only on external voltage but also on light stimulation. In the one-dimensional, two-dimensional, and multi-dimensional memristor neural networks, the conductance value of each second-order photosensitive memristor corresponds one-to-one with the feature points of the physical structure in the input data, enabling direct reception of voltage simulation matrix data collected by sensors without additional encoding. The network is trained using a biomimetic learning and forgetting mechanism based on Hebbus rules. The entire process more closely resembles the real training mode of biological neural networks, significantly reducing power consumption and making it more efficient and energy-saving than traditional machine learning methods.

[0090] See Figure 3 The core unit of a one-dimensional memristor neural network (110) is the memristor neuron. Each neuron consists of two second-order photosensitive memristors of the same polarity, either forward or reverse, connected in series, forming an input memristor group or a bias memristor group. The network's output principle is based on the positive half structure between the positive driving source and the directional driving source, and the negative half structure between the directional driving source and the negative driving source. By applying symmetrical positive and negative voltages, due to the difference in conductance between different memristor groups, a result voltage corresponding to the bias state is generated on the directional driving source. After multiple memristor neurons are connected in series, the combination of the result voltages of each neuron under different biases constitutes the overall computational output of the memristor neural network for the input data.

[0091] See Figure 4 , Figure 5 Both the two-dimensional memristor neural network 120 and the multi-dimensional memristor neural network 130 are based on an I-shaped structure design. The basic unit of the two-dimensional memristor neural network 120 consists of four second-order photosensitive memristors connected in an I-shaped structure, which are then combined using self-similar I-shaped structures. The multi-dimensional memristor neural network 130 is composed of multiple stacked layers of two-dimensional memristor neural networks, with the specific number of layers depending on the dimension of the input data. The core working principle of the two-dimensional and multi-dimensional memristor neural networks is similar to that of the one-dimensional network, both relying on the voltage bias caused by the difference in conductance of different memristors as the network output. The difference lies in the more complex structure of the basic memristor neurons and the connections between neurons, in order to adapt to higher-dimensional data input.

[0092] The above content is only for explaining the technical idea of ​​the present invention and cannot be used to limit the protection scope of the present invention. Any changes made on the basis of the technical solution in accordance with the technical idea proposed by the present invention shall fall within the protection scope of the claims of the present invention.

Claims

1. A multimodal hybrid memristor neural network edge computing terminal system, characterized in that, Along the signal processing direction, the system includes a multimodal sensor module (50), a servo light source training module (12), an edge computing module (13), an embedded processor (11), and an execution module (60). The terminal system also includes peripheral circuits (20), a cloud platform big data center (30), and a human-computer interaction module (40). The servo light source training module (12), the edge computing module (13), and the embedded processor (11) constitute the core module (10). The core module (10) also includes a wireless communication module (14). The cloud platform big data center (30) and the embedded processor (11) are bidirectionally connected through the wireless communication module (14). The peripheral circuit (20) is connected to the servo light source training module (12), the edge computing module (13) and the embedded processor (11) respectively to support the operation of the system terminal; the multimodal sensor module (50) is used to collect multimodal voltage simulation matrix; The edge computing module (13) is a multimodal hybrid memristor neural network, which consists of several parallel memristor neural networks (100). The number of memristor neural networks (100) corresponds one-to-one with the number of modal inputs (101). The human-computer interaction module (40) includes a touch screen (41), a keyboard matrix (42), and a visualization module (43), which are interconnected with the embedded processor (11) to realize system result visualization and human-computer interaction functions. The memristor neural network (100) is one or more of a one-dimensional memristor neural network (110), a two-dimensional memristor neural network (120), or a multi-dimensional memristor neural network (130), used to support the training requirements of modal data of different dimensions; The one-dimensional memristor neural network (110) includes a plurality of memristor neurons (121), negative driving sources (90), directional driving sources (80) and positive driving sources (70). The memristor neuron (121) includes an input second-order photosensitive memristor group and a bias second-order photosensitive memristor group. The input second-order photosensitive memristor group consists of a first input memristor switch (112), a first positive second-order photosensitive memristor (111), a second input memristor switch (112), and a second positive second-order photosensitive memristor (111) connected in sequence. The biased second-order photosensitive memristor group consists of a first biased memristor switch (114), a first reverse-connected second-order photosensitive memristor (113), a second biased memristor switch (114), and a second reverse-connected second-order photosensitive memristor (113) connected in sequence. The positive drive source (70) is connected to the first input memristor switch (112) and the first bias memristor switch (114) respectively; the negative drive source (90) is connected to the second positive second-order photosensitive memristor (111) and the second reverse second-order photosensitive memristor (113) respectively; the connection point between the first positive second-order photosensitive memristor (111) and the second input memristor switch (112) and the connection point between the first reverse second-order photosensitive memristor (113) and the second bias memristor switch (114) are connected to the directional drive source (80), the directional drive source (80) is connected to the output bus, and the output node of the output bus is the knowledge output point (106) of the one-dimensional memristor neural network (110).

2. The multimodal hybrid memristor neural network edge computing terminal system according to claim 1, characterized in that, The peripheral circuit (20) specifically includes a signal processing module (22), a multi-level power management module (21), and a high-speed storage module (23). The signal processing module (22) is used to filter and reduce noise and sample and hold the system input signal; the multi-level power management module (21) is used to provide power supply at different voltage levels; and the high-speed storage module (23) is used to store the data during the operation of the terminal system.

3. The multimodal hybrid memristor neural network edge computing terminal system according to claim 1, characterized in that, The two-dimensional memristor neural network (120) includes a plurality of memristor neurons (121), negative driving sources (90), directional driving sources (80) and positive driving sources (70). The memristor neurons (121) include a first second-order photosensitive memristor (122), a second second-order photosensitive memristor (123), a third second-order photosensitive memristor (124) and a fourth second-order photosensitive memristor (125). The positive terminals of the first second-order photoresistor (122) and the third second-order photoresistor (124) are connected to the positive driving source (70), and the negative terminals of the first second-order photoresistor (122) and the third second-order photoresistor (124) are connected to the directional driving source (80). The positive terminals of the second second-order photoresistor (123) and the fourth second-order photoresistor (125) are connected to the directional driving source (80), and the negative terminals of the second second-order photoresistor (123) and the fourth second-order photoresistor (125) are connected to the negative driving source (90). Memristor neurons (121) are connected to directional driving sources (80) in a self-similar I-shaped structure via switching switches (126). The directional driving sources (80) are connected to the output bus, and the output nodes of the output bus are the knowledge output points (106) of the two-dimensional memristor neural network (120).

4. The multimodal hybrid memristor neural network edge computing terminal system according to claim 3, characterized in that, The multidimensional memristor neural network (130) consists of several parallel two-dimensional memristor neural networks (120). The knowledge output points (106) of each two-dimensional memristor neural network (120) are connected in parallel on the output bus, and the output nodes of the output bus are the knowledge output points (106) of the multi-dimensional memristor neural network (130).

5. A multimodal hybrid memristor neural network edge computing terminal system according to any one of claims 1 to 4, characterized in that, The servo light source training module (12) includes a light source matrix. The point light sources in the light source matrix correspond one-to-one with the second-order photosensitive memristors in the multimodal hybrid memristor neural network. It is used to adjust the illumination intensity and duration of the point light sources according to the multimodal voltage simulation matrix to realize the initial training of the multimodal hybrid memristor neural network.

6. A multimodal hybrid memristor neural network edge computing method, characterized in that, The multimodal hybrid memristor neural network edge computing terminal system according to claim 1 includes the following steps: Step 1: Based on the modalities involved in the actual problem, determine the number and dimensions of the modal inputs (101) of the multimodal sensor module (50), and determine the number and dimensions of the memristor neural network (100) based on the number and dimensions of the modal inputs (101); obtain the modal type of the acquired signal, and acquire the multimodal voltage simulation matrix; Step 2: Filter and denoise the multimode voltage simulation matrix and perform sample-and-hold operations to obtain the processed multimode voltage simulation matrix; Step 3: The servo light source training module (12) adjusts the illumination intensity and irradiation time of the point light source according to the processed multimodal voltage simulation matrix, stimulates the memristor neural network (100) to perform initial assignment, and realizes bionic learning; Step 4: By controlling the state of each memristor neural network (100), read the training results of a single network for a single mode, and obtain the independent output knowledge points of the memristor neural network (100) under each mode; based on current driving and Heb rule, adaptively adjust the weights of the memristor neural network (100) to realize the self-coordinated update of the memristor neural network (100) under each mode, and simulate bionic forgetting. Step 5: By controlling the hybrid switching of the multimodal hybrid memristor neural network, read the hybrid output knowledge points after multimodal fusion; adaptively adjust the weights of the multimodal hybrid memristor neural network based on current driving and Heb rule to achieve adaptive fusion of multimodal information; generate personalized decision commands based on the confidence matrix formed by the independent output knowledge points and the hybrid output knowledge points. Step 6: Pass the decision command to the execution module (60) to guide the specific movement of the execution module (60) and end the main process.

7. The multimodal hybrid memristor neural network edge computing method according to claim 6, characterized in that, After completing step three or four, the following steps are also included: The peripheral circuit (20) stores the data of the terminal system during operation; the process and result data of the terminal system operation are uploaded to the cloud platform big data center (30) for historical data accumulation and analysis to support training strategy updates and parameter optimization, and realize remote maintenance and intelligent management; the process and result data of the terminal system operation are displayed through the human-computer interaction module (40); the user controls and views the training parameters and visualization content of the terminal system through the human-computer interaction module (40).

8. The multimodal hybrid memristor neural network edge computing method according to claim 6, characterized in that, When step six is ​​completed, the following steps are also included: uploading the process and result data of the terminal system operation to the cloud platform big data center (30) for historical data accumulation and analysis to support training strategy updates and parameter optimization, and to realize remote maintenance and intelligent management; displaying the process and result data of the terminal system operation through the human-computer interaction module (40).

Images