Multi-mode hybrid memristor neural network edge computing terminal system and method

Abstract

The invention discloses a multi-mode hybrid memristor neural network edge computing terminal system and method, and belongs to the technical field of artificial intelligence. According to the system, the defects of existing edge calculation in the aspects of high energy efficiency and real-time multi-modal data processing are overcome, fusion of storage and calculation is achieved through the memristor, and the bottleneck generated by separation of storage and calculation in a traditional calculation structure is reduced; fusion and self-adaptive learning of multi-modal data are realized by utilizing the self-adaptive conductivity characteristic and the Hertz plasticity of the bionic memristor neural network structure, and the method adapts to a complex and changeable application environment; the core module can directly input a voltage analog quantity matrix collected by the multi-modal sensor module into the memristor neural network in the edge calculation module through the servo light source training module, and then fusion and classification are carried out on multi-modal sensor data through the memristor neural networks of different dimensions, and a reliable decision is generated. The method is suitable for intelligent edge applications with high efficiency and low power consumption.

Description

Technical Field

[0001] The present invention relates to the field of artificial intelligence technology, and in particular to a multimodal hybrid memristor neural network edge computing terminal system and method. Background Art

[0002] In the field of artificial intelligence, especially for the training and reasoning of large-scale neural networks and deep learning models, the rapid progress of computing technology is facing severe challenges in computing power and energy efficiency. The traditional von Neumann architecture, due to its inherent separation of computing and storage design, has caused the so-called "storage wall" problem. This bottleneck not only seriously hinders the further improvement of system computing power, but also leads to a sharp increase in power consumption, becoming a key factor restricting the efficient operation of large-scale artificial intelligence models.

[0003] To solve this problem, memristors, a revolutionary electronic component, came into being. Their uniqueness lies in their ability to carry out storage and computing functions at the same time, realizing the integrated operation of data storage and processing at the same physical location, thus completely avoiding the various limitations brought about by the separation of computing and storage. Compared with traditional computing architectures, memristors can directly perform core matrix operations in neural networks, providing strong support for large-scale parallel computing tasks. This design not only greatly reduces the frequency and energy consumption of data transmission, but also significantly improves computing efficiency at the hardware level, opening up a new path for artificial intelligence applications that pursue efficient computing and low power consumption.

[0004] In this context, HP Labs innovatively proposed the Crossbar architecture based on memristors. This architecture cleverly adopts a cross-grid layout and precisely places memristors at the cross nodes of the grid, achieving seamless parallel processing of computing and storage in the same physical space. However, it is worth noting that the current Crossbar architecture still uses the traditional model in terms of control strategy, and its weight assignment process still relies on traditional computers for decision-making. Although this compromise solution has improved computing efficiency to a certain extent, it has failed to fully tap the potential of the integrated storage and computing characteristics of memristors, especially in terms of energy consumption optimization, and there is still much room for improvement and improvement.

[0005] On the other hand, as a complex challenge in the field of computing systems and models, the multimodal mixing problem requires effective fusion and comprehensive analysis when processing data from multiple modalities such as vision, hearing, temperature, vibration, etc., in order to reveal the intrinsic connection and mutual influence between the modalities, so as to make more accurate and robust decisions or predictions. This problem has a wide range of application value in many fields such as intelligent edge computing, machine learning and artificial intelligence. In particular, in scenarios such as image and speech fusion and multimodal data processing in sensor networks, a single modality is often difficult to fully and accurately express complex information, and the introduction of multimodal mixing technology is particularly important.

[0006] At present, multimodal hybrid methods mainly include three categories: feature-level fusion, decision-level fusion and data-level fusion. They each show different advantages and limitations in information retention, noise processing and modal coupling, and are suitable for different application scenarios. However, with the increasing variety of modal types and the continuous improvement of task complexity, existing methods still seem to be unable to cope with the challenges of heterogeneous data compatibility, adaptability and fusion efficiency. It is difficult to meet the real-time and efficient computing requirements in complex environments. A more advanced and comprehensive multimodal hybrid technology is urgently needed. Summary of the invention

[0007] The purpose of the present invention is to provide a multimodal hybrid memristor neural network edge computing terminal system and method to address the deficiencies of edge computing in the prior art in terms of high energy efficiency and real-time multimodal data processing.

[0008] The present invention solves the above technical problems through the following technical solutions:

[0009] A multimodal hybrid memristor neural network edge computing terminal system, which includes a multimodal sensor module, a servo light source training module, an edge computing module, an embedded processor and an execution module in sequence along the signal processing direction. The terminal system also includes a peripheral circuit, a cloud platform big data center and a human-computer interaction module. The servo light source training module, the edge computing module and the embedded processor constitute a core module. The core module also includes a wireless communication module. The cloud platform big data center and the embedded processor are bidirectionally interactively connected through the wireless communication module.

[0010] The peripheral circuits are respectively connected to the servo light source training module, the edge computing module and the embedded processor to support the operation of the system terminal; the multi-modal sensor module is used to collect the multi-modal voltage simulation matrix;

[0011] The edge computing module is a multimodal hybrid memristor neural network, which is a number of parallel-arranged memristor neural networks, and the number of memristor neural networks corresponds one-to-one to the number of modal inputs; the human-computer interaction module includes a touch display, a keyboard matrix, and a visualization module, which are interactively connected to the embedded processor respectively to realize system result visualization and human-computer interaction functions.

[0012] A further improvement of the present invention is that the peripheral circuit specifically includes a signal processing module, a multi-level power management module and a high-speed storage module. The signal processing module is used to implement filtering, noise reduction and sampling and holding of the system input signal; the multi-level power management module is used to provide power supply at different voltage levels; and the high-speed storage module is used to store data when the terminal system is running.

[0013] A further improvement of the present invention is that the memristor neural network is one or more of a one-dimensional memristor neural network, a two-dimensional memristor neural network or a multi-dimensional memristor neural network, which is used to support the training requirements of modal data of different dimensions.

[0014] A further improvement of the present invention is that: the one-dimensional memristor neural network comprises a plurality of memristor neurons, a negative driving source, a directional driving source and a positive driving source;

[0015] The memristor neuron includes an input second-order photosensitive memristor group and a bias second-order photosensitive memristor group, wherein the input second-order photosensitive memristor group is a first input memristor switch, a first positive second-order photosensitive memristor, a second input memristor switch and a second positive second-order photosensitive memristor connected in sequence;

[0016] The biased second-order photosensitive memristor group is a first biased memristor switch, a first reverse-connected second-order photosensitive memristor, a second biased memristor switch and a second reverse-connected second-order photosensitive memristor connected in sequence;

[0017] The positive driving source is respectively connected to the first input memristor switch and the first bias memristor switch; the negative driving source is respectively connected to the second positive second-order photosensitive memristor and the second reverse second-order photosensitive memristor; the connection point between the first positive second-order photosensitive memristor and the second input memristor switch and the connection point between the first reverse second-order photosensitive memristor and the second bias memristor switch are connected to the directional driving source, the directional driving source is connected to the output bus, and the output node of the output bus is the knowledge output point of the one-dimensional memristor neural network.

[0018] A further improvement of the present invention is that the two-dimensional memristor neural network includes a plurality of memristor neurons, a negative driving source, a directional driving source and a positive driving source, the memristor neurons include a first second-order photosensitive memristor, a second second-order photosensitive memristor, a third second-order photosensitive memristor and a fourth second-order photosensitive memristor,

[0019] The positive ends of the first second-order photosensitive memristor and the third second-order photosensitive memristor are connected to a positive driving source, and the negative ends of the first second-order photosensitive memristor and the third second-order photosensitive memristor are connected to a directional driving source;

[0020] The positive ends of the second second-order photosensitive memristor and the fourth second-order photosensitive memristor are connected to a directional driving source, and the negative ends of the second second-order photosensitive memristor and the fourth second-order photosensitive memristor are connected to a negative driving source;

[0021] The memristor neurons are connected to a directional driving source through a switching switch in a self-similar I-shaped structure, wherein the directional driving source is connected to an output bus, and the output node of the output bus is a knowledge output point of the two-dimensional memristor neural network.

[0022] A further improvement of the present invention is that the multidimensional memristor neural network is a plurality of parallel two-dimensional memristor neural networks.

[0023] The knowledge output point of each two-dimensional memristor neural network is connected in parallel on an output bus, and the output nodes of the output bus are the knowledge output points of the multi-dimensional memristor neural network.

[0024] A further improvement of the present invention is that the servo light source training module includes a light source matrix, and the point light sources in the light source matrix correspond one-to-one to the second-order photosensitive memristors in the multimodal hybrid memristor neural network, which is used to adjust the brightness and duration of the point light sources according to the multimodal voltage simulation matrix to achieve initial training of the multimodal hybrid memristor neural network.

[0025] A multimodal hybrid memristor neural network edge computing method, using the above-mentioned multimodal hybrid memristor neural network edge computing terminal system, includes the following steps:

[0026] Step 1: According to the modality involved in the actual problem, determine the number and dimension of the modal input of the multimodal sensor module, and determine the number and dimension of the memristor neural network according to the number and dimension of the modal input; obtain the modal type of the collected signal, and collect the multimodal voltage simulation matrix;

[0027] Step 2: filtering, denoising and sampling and holding the multi-modal voltage simulation matrix to obtain a processed multi-modal voltage simulation matrix;

[0028] Step 3: The servo light source training module adjusts the light intensity and irradiation time of the point light source according to the processed multi-modal voltage simulation matrix, stimulates the memristor neural network to perform initial assignment, and realizes bionic learning;

[0029] Step 4: By controlling the state of each memristor neural network to read the single network training results for a single mode, the independent output knowledge points of the memristor neural network in each mode are obtained; based on current drive and Hebbian rule, the weights of the memristor neural network are adaptively adjusted to achieve self-coordinated update of the memristor neural network in each mode, simulating bionic forgetting;

[0030] Step 5: Read the mixed output knowledge points after multimodal fusion by controlling the mixed switching of the multimodal hybrid memristor neural network; adaptively adjust the weights of the multimodal hybrid memristor neural network based on current drive and Hebbian rule to achieve adaptive fusion of multimodal information; construct a confidence matrix based on independent output knowledge points and mixed output knowledge points to generate personalized decision commands;

[0031] Step 6: Pass the decision command to the execution module to guide the specific movement form of the execution module and end the main process.

[0032] A further improvement of the present invention is that after executing step 3 or step 4, the following steps are also included:

[0033] The data of the terminal system during operation is stored through peripheral circuits; the process and result data of the terminal system operation are uploaded to the cloud platform big data center for historical data accumulation and analysis to support training strategy updates and parameter optimization, and to achieve remote maintenance and intelligent management; the process and result data of the terminal system operation are displayed through the human-computer interaction module; users control and view the training parameters and visualization content of the terminal system through the human-computer interaction module.

[0034] A further improvement of the present invention is that when step six is ​​executed, the following steps are also included: uploading the process and result data of the terminal system operation to the cloud platform big data center, accumulating and analyzing historical data to support training strategy updates and parameter optimization, and realizing remote maintenance and intelligent management; displaying the process and result data of the terminal system operation through the human-computer interaction module.

[0035] Compared with the prior art, the positive and progressive effects of the present invention are:

[0036] The multimodal hybrid memristor neural network edge computing terminal system provided by the present invention solves the shortcomings of existing edge computing in high energy efficiency and real-time multimodal data processing. The fusion of storage and computing is realized through the multimodal hybrid memristor neural network, which reduces the data transmission bottleneck caused by the separation of storage and computing in the traditional computing architecture, thereby improving the overall energy efficiency and computing efficiency of the system; using the adaptive conductivity characteristics and Hebbian plasticity of the bionic memristor neural network structure, the system can dynamically adjust the network parameters, realize the fusion and adaptive learning of multimodal data, and enhance the flexibility and adaptability of the system; the integrated design of the core module enables the system to directly input the data collected by the multimodal sensor module into the memristor neural network in the edge computing module, simplifying the data processing process and improving the real-time performance of the system; the connection of the cloud platform big data center enables the system to remotely store and analyze historical data, support training strategy updates and parameter optimization, and realize remote maintenance and intelligent management. The human-computer interaction module provides a friendly user interface, allowing users to easily control and view the training parameters and visualization content of the system.

[0037] Furthermore, the introduction of peripheral circuit modules realizes functions such as filtering and noise reduction, sampling and holding, and data storage of system input signals, further improving the stability and reliability of the system.

[0038] Furthermore, the system adopts a multimodal hybrid memristor neural network, which can process data from multiple modalities simultaneously, and fuse and classify them through memristor neural networks of different dimensions to generate reliable decisions, making the system suitable for complex and changing application environments and improving the intelligence level of the system.

[0039] The multimodal hybrid memristor neural network edge computing method provided by the present invention reduces data transmission, optimizes the computing structure, and utilizes the storage and computing integrated characteristics of the memristor. The system can reduce power consumption while maintaining high performance, thereby improving the energy efficiency and real-time performance of edge computing. BRIEF DESCRIPTION OF THE DRAWINGS

[0040] The drawings in the specification are used to provide further understanding of the present invention and constitute a part of the present invention. The schematic embodiments of the present invention and their descriptions are used to explain the present invention and do not constitute improper limitations on the present invention.

[0041] Figure 1 It is a structural schematic diagram of the present invention;

[0042] Figure 2 It is a schematic diagram of the structure of a multimodal hybrid memristor neural network;

[0043] Figure 3 It is a schematic diagram of the structure of a one-dimensional memristor neural network;

[0044] Figure 4 It is a schematic diagram of the structure of a two-dimensional memristor neural network;

[0045] Figure 5 It is a schematic diagram of the structure of a multi-dimensional memristor neural network;

[0046] Figure 6 It is a schematic diagram of the process of the present invention;

[0047] Among them, core module-10, embedded processor-11, servo light source training module-12, edge computing module-13, wireless communication module-14, peripheral circuit-20, multi-level power management module-21, signal processing module-22, high-speed storage module-23, cloud platform big data center-30, human-computer interaction module-40, touch display screen-41, keyboard matrix-42, visualization module-43, multimodal sensor module-50, execution module-60, positive drive source-70, directional drive source-80, negative drive source-90, memristor neural network-100, modal input-101, positive memristor neural network-102, positive memristor neural network Network independent switch-103, network positive bias part and negative memristor neural network-104, negative memristor neural network independent switch-105, knowledge output point-106, one-dimensional memristor neural network-110, positively connected second-order photosensitive memristor-111, input memristor switch-112, reversely connected second-order photosensitive memristor-113, bias memristor switch-114, two-dimensional memristor neural network-120, memristor neuron-121, first second-order photosensitive memristor-122, second second-order photosensitive memristor-123, third second-order photosensitive memristor-124, fourth second-order photosensitive memristor-125, switching switch-126, multi-dimensional memristor neural network-130. DETAILED DESCRIPTION

[0048] In order to make the purpose, technical solutions and advantages of the embodiments of the present invention clearer, the technical solutions in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are part of the embodiments of the present invention, not all of the embodiments. Generally, the components of the embodiments of the present invention described and shown in the drawings here can be arranged and designed in various different configurations.

[0049] Therefore, the following detailed description of the embodiments of the present invention provided in the accompanying drawings is not intended to limit the scope of the invention claimed for protection, but merely represents selected embodiments of the present invention. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0050] It should be noted that similar reference numerals and letters denote similar items in the following drawings, and therefore, once an item is defined in one drawing, further definition and explanation thereof is not required in subsequent drawings.

[0051] In the description of the embodiments of the present invention, it should be noted that if the terms "upper", "lower", "horizontal", "inner", etc. indicate an orientation or positional relationship based on the orientation or positional relationship shown in the drawings, or the orientation or positional relationship in which the product of the invention is usually placed when in use, it is only for the convenience of describing the present invention and simplifying the description, and does not indicate or imply that the device or element referred to must have a specific orientation, be constructed and operated in a specific orientation, and therefore cannot be understood as a limitation on the present invention. In addition, the terms "first", "second", etc. are only used to distinguish the description, and cannot be understood as indicating or implying relative importance.

[0052] In addition, it should be noted that, unless otherwise clearly specified and limited, the terms "set", "install", "connect", and "connect" should be understood in a broad sense, for example, it can be a fixed connection, a detachable connection, or an integral connection; it can be a mechanical connection or an electrical connection; it can be a direct connection, or it can be indirectly connected through an intermediate medium, or it can be the internal connection of two components. For ordinary technicians in this field, the specific meanings of the above terms in the present invention can be understood according to specific circumstances.

[0053] The present invention is further described in detail below in conjunction with the accompanying drawings and specific embodiments, which are intended to explain the present invention rather than to limit it.

[0054] The present invention provides a multimodal hybrid memristor neural network edge computing terminal system, which includes a multimodal sensor module 50, a servo light source training module 12, an edge computing module 13, an embedded processor 11 and an execution module 60 in sequence along the signal processing direction. The terminal system also includes a peripheral circuit 20, a cloud platform big data center 30 and a human-computer interaction module 40. The servo light source training module 12, the edge computing module 13 and the embedded processor 11 constitute a core module 10. The core module 10 also includes a wireless communication module 14. The cloud platform big data center 30 and the embedded processor 11 are bidirectionally interactively connected through the wireless communication module 14.

[0055] The peripheral circuit 20 is respectively connected to the servo light source training module 12, the edge computing module 13 and the embedded processor 11, and is used to support the operation of the system terminal; the multi-modal sensor module 50 is used to collect the multi-modal voltage simulation matrix;

[0056] The edge computing module 13 is a multimodal hybrid memristor neural network, which is a plurality of parallel-arranged memristor neural networks 100, and the number of memristor neural networks 100 corresponds one-to-one to the number of modal inputs 101; the human-computer interaction module 40 includes a touch display screen 41, a keyboard matrix 42 and a visualization module 43, which are respectively interactively connected to the embedded processor 11 to realize system result visualization and human-computer interaction functions.

[0057] Specifically, the peripheral circuit 20 includes a signal processing module 22, a multi-level power management module 21 and a high-speed storage module 23. The signal processing module 22 is used to filter, reduce noise and sample and hold the system input signal; the multi-level power management module 21 is used to provide power supply at different voltage levels; and the high-speed storage module 23 is used to store data when the terminal system is running.

[0058] Specifically, the memristor neural network 100 is one or more of a one-dimensional memristor neural network 110, a two-dimensional memristor neural network 120 or a multi-dimensional memristor neural network 130, which is used to support the training requirements of modal data of different dimensions.

[0059] Specifically, the one-dimensional memristor neural network 110 includes a plurality of memristor neurons 121, a negative driving source 90, a directional driving source 80 and a positive driving source 70;

[0060] The memristor neuron 121 includes an input second-order photosensitive memristor group and a bias second-order photosensitive memristor group, wherein the input second-order photosensitive memristor group is a first input memristor switch 112, a first positive second-order photosensitive memristor 111, a second input memristor switch 112 and a second positive second-order photosensitive memristor 111 connected in sequence;

[0061] The biased second-order photosensitive memristor group is a first biased memristor switch 114, a first reverse-connected second-order photosensitive memristor 113, a second biased memristor switch 114 and a second reverse-connected second-order photosensitive memristor 113 connected in sequence;

[0062] The positive driving source 70 is respectively connected to the first input memristor switch 112 and the first bias memristor switch 114; the negative driving source 90 is respectively connected to the second positive second-order photosensitive memristor 111 and the second reverse second-order photosensitive memristor 113; the connection point between the first positive second-order photosensitive memristor 111 and the second input memristor switch 112 and the connection point between the first reverse second-order photosensitive memristor 113 and the second bias memristor switch 114 are connected to the directional driving source 80, and the directional driving source 80 is connected to the output bus, and the output node of the output bus is the knowledge output point 106 of the one-dimensional memristor neural network 110.

[0063] Specifically, the two-dimensional memristor neural network 120 includes a plurality of memristor neurons 121, a negative driving source 90, a directional driving source 80 and a positive driving source 70, the memristor neuron 121 includes a first second-order photosensitive memristor 122, a second second-order photosensitive memristor 123, a third second-order photosensitive memristor 124 and a fourth second-order photosensitive memristor 125,

[0064] The positive terminals of the first second-order photosensitive memristor 122 and the third second-order photosensitive memristor 124 are connected to the positive driving source 70 , and the negative terminals of the first second-order photosensitive memristor 122 and the third second-order photosensitive memristor 124 are connected to the directional driving source 80 ;

[0065] The positive terminals of the second second-order photosensitive memristor 123 and the fourth second-order photosensitive memristor 125 are connected to the directional driving source 80, and the negative terminals of the second second-order photosensitive memristor 123 and the fourth second-order photosensitive memristor 125 are connected to the negative driving source 90;

[0066] The memristor neurons 121 are connected to the directional driving source 80 in a self-similar I-shaped structure via the switching switch 126 , wherein the directional driving source 80 is connected to the output bus, and the output node of the output bus is the knowledge output point 106 of the two-dimensional memristor neural network 120 .

[0067] Specifically, the multi-dimensional memristor neural network 130 is a plurality of parallel-arranged two-dimensional memristor neural networks 120.

[0068] The knowledge output points 106 of each two-dimensional memristor neural network 120 are connected in parallel on an output bus, and the output nodes of the output bus are the knowledge output points 106 of the multi-dimensional memristor neural network 130 .

[0069] Specifically, the servo light source training module 12 includes a light source matrix, in which the point light sources in the light source matrix correspond one-to-one to the second-order photosensitive memristors in the multimodal hybrid memristor neural network, and are used to adjust the brightness and duration of the point light sources according to the multimodal voltage simulation matrix to achieve initial training of the multimodal hybrid memristor neural network.

[0070] The terminal system provided by the present invention aims to solve the shortcomings of existing edge computing in high-energy efficiency and real-time multimodal data processing. It realizes the fusion of storage and computing through memristors, reducing the bottleneck caused by the separation of storage and computing in traditional computing structures; utilizes the adaptive conductivity characteristics and Hebbian plasticity of the bionic memristor neural network structure to realize the fusion and adaptive learning of multimodal data, and adapt to complex and changeable application environments; the edge computing terminal system consists of a core module, a peripheral circuit module, a cloud platform big data center, a human-computer interaction module, a multimodal sensor module and an execution module. The core module can directly input the voltage analog matrix collected by the multimodal sensor module into the memristor neural network in the edge computing module through the servo light source training module, and then fuse and classify the multimodal sensor data through memristor neural networks of different dimensions, generate reliable decisions, and guide the actions of the execution module, which is suitable for efficient and low-power intelligent edge applications.

[0071] See also Figure 6 Based on the same inventive concept, the present invention provides a multimodal hybrid memristor neural network edge computing method, using the above-mentioned multimodal hybrid memristor neural network edge computing terminal system, including the following steps:

[0072] Step 1: According to the modality involved in the actual problem, determine the number and dimension of the modal input 101 of the multimodal sensor module 50, and determine the number and dimension of the memristor neural network 100 according to the number and dimension of the modal input 101; obtain the modal type of the collected signal, and collect the multimodal voltage simulation matrix;

[0073] Step 2: filtering, denoising and sampling and holding the multi-modal voltage simulation matrix to obtain a processed multi-modal voltage simulation matrix;

[0074] Step 3: The servo light source training module 12 adjusts the light intensity and irradiation time of the point light source according to the processed multi-modal voltage simulation matrix, stimulates the memristor neural network 100 to perform initial assignment, and realizes bionic learning;

[0075] Step 4: by controlling the state of each memristor neural network 100 to read the single network training result for a single mode, the independent output knowledge points of the memristor neural network 100 in each mode are obtained; based on the current drive and Hebbian rule, the weights of the memristor neural network 100 are adaptively adjusted to achieve self-coordinated update of the memristor neural network 100 in each mode, simulating bionic forgetting;

[0076] Step 5: Read the mixed output knowledge points after multimodal fusion by controlling the mixed switching of the multimodal hybrid memristor neural network; adaptively adjust the weights of the multimodal hybrid memristor neural network based on current drive and Hebbian rule to achieve adaptive fusion of multimodal information; construct a confidence matrix based on independent output knowledge points and mixed output knowledge points to generate personalized decision commands;

[0077] Step 6: pass the decision command to the execution module 60 to guide the execution module 60 to perform a specific movement, and end the main process.

[0078] In a specific embodiment of the present invention, step three is specifically as follows: adjusting the light intensity and irradiation time of the point light source according to the processed multimodal voltage simulation matrix, stimulating the corresponding second-order photosensitive memristor to change the conductance value, and realizing the input of the processed multimodal voltage simulation into the memristor neural network 100 to realize bionic learning.

[0079] In a specific embodiment of the present invention, step four is specifically as follows: by controlling the on / off states of the positive driving source 70, the directional driving source 80 and the negative driving source 90 and the switching of the memristor neuron 121, the single network training results for a single mode are read to obtain independent output knowledge points of the memristor neural network 100 under each mode; by stimulating the second-order photosensitive memristor through current drive, the conductance value is adaptively adjusted based on the Hebbian rule, so as to realize self-coordinated updating of the network and simulate bionic forgetting.

[0080] In a specific embodiment of the present invention, step five is specifically as follows: by controlling the mixed switching of the multimodal hybrid memristor neural network, the mixed output knowledge points after multimodal fusion are read; by stimulating the memristor neurons 121 through current drive, the conductance value is adaptively adjusted based on the Hebbian rule to achieve adaptive fusion of multimodal information; a confidence matrix is ​​constructed based on the independent output knowledge points and the mixed output knowledge points to generate personalized decision commands.

[0081] This method adopts a parallel and scalable architecture, each memristor neural network independently corresponds to a modal input data, and the number of network modules is dynamically adjusted according to the modal requirements in the actual problem; each memristor neural network module is initially trained through a servo light source module, and the analog voltage signal simplified by the peripheral circuit is directly input into the network to realize bionic learning; during the training process, the specific memristor neurons in the network are dynamically connected through electric drive control, the training results are read, and the weights are automatically adjusted according to the Hebbian rule to realize bionic memory and forgetting, thereby having adaptive learning capabilities; in the process of multimodal information processing, the system generates a confidence voltage matrix that integrates multimodal outputs by controlling the independent connection and coupling operations of the memristor neural network modules; this matrix not only reflects the independent signal characteristics of each mode, but also adaptively enhances the coupling between modes to achieve efficient integration of multimodal information, and ultimately achieves efficient edge computing functions for complex tasks.

[0082] Specifically, after executing step 3 or step 4, the following steps are also included:

[0083] The data of the terminal system during operation is stored through the peripheral circuit 20; the process and result data of the terminal system operation are uploaded to the cloud platform big data center 30 to accumulate and analyze historical data to support training strategy updates and parameter optimization, and realize remote maintenance and intelligent management; the process and result data of the terminal system operation are displayed through the human-computer interaction module 40; the user controls and views the training parameters and visualization content of the terminal system through the human-computer interaction module 40.

[0084] Specifically, when step six is ​​executed, the following steps are also included: uploading the process and result data of the terminal system operation to the cloud platform big data center 30, accumulating and analyzing historical data to support training strategy updates and parameter optimization, and realizing remote maintenance and intelligent management; displaying the process and result data of the terminal system operation through the human-computer interaction module 40.

[0085] See also Figure 1 A multimodal hybrid memristor neural network edge computing terminal system, the core module 10 is the nerve center of the entire terminal system, specifically, the edge computing module 13 uses the memristor neural network 100 as the computing core, the servo light source module 12 controls the light intensity and illumination time of each point light source in the light source matrix according to the input of the voltage analog quantity, thereby realizing the training of the second-order photosensitive memristor matrix, and the embedded processor 11 controls the switching of the memristor neural network and its memristor neurons, and judges the generated confidence matrix to finally form a decision command.

[0086] The core module 10 includes a servo light source training module 12, an edge computing module 13, an embedded processor 11, and a wireless communication module 14. It has online training and autonomous learning capabilities, and can update the memristor conductance value in real time based on the data transmitted by the afferent nerve fibers, thereby achieving efficient edge computing and data processing, and transmitting the decision results to the actuator through the efferent nerve fibers. The efferent and afferent nerve fibers are actually media for data transmission; the peripheral circuit module 20 is used to support the stable operation of the memristor neural network, including a multi-level power management module 21, which is used to meet the driving requirements of different voltage conditions of each module in the system; a signal processing module 22, which is used to perform simple filtering, noise reduction and sampling and holding on the signal input into the core module; a high-speed storage module 23, which is used to save important results and process data during system operation, and has level conversion and signal isolation circuits; a cloud platform big data center 30 is used for remote data storage, analysis and model optimization of terminal systems Through the cloud platform big data center 30, historical data can be accumulated and analyzed to provide functions such as training strategy update and parameter optimization, so as to realize remote maintenance and intelligent management of edge computing terminals; the human-computer interaction module 40 includes a display screen 41, a keyboard control panel and other result visualization modules such as LEDs, which are convenient for users to operate and monitor, and realize intuitive interaction of functions such as data acquisition, model parameter adjustment, and task target correction; the multimodal sensor module 50 and the execution module 60 can be customized according to different target tasks. The multimodal sensor module 50, as a nervous system receptor structure, can mix multiple types of sensors and input the results into the core module in the form of a multi-dimensional voltage analog matrix; the execution module 60, as a nervous system effector structure, can execute specific tasks according to the decision commands output by the core module 10, such as realizing functions such as fine grasping and posture adjustment, with high responsiveness and execution accuracy.

[0087] The multimodal hybrid memristor neural network is the core of the edge computing module 13. Its structure is a memristor neural network 100 composed of a number of memristor neural networks 100 connected in parallel and controlled by three driving sources. Each memristor neural network 100 corresponds to a modal data in the actual problem. In the multimodal hybrid processing, each memristor neural network is first trained independently for its specific modal data, and the training results are used as independent knowledge output points of each memristor neural network element. Subsequently, the independently trained memristor neural network is combined and turned on. See Figure 2, each memristor neural network 100 is equivalent to the positive and negative half structures composed of upper and lower biases in practical applications, which can be regarded as the positive bias part of the network composed of the positive memristor neural network 102 and the positive memristor neural network independent switch 103 and the negative bias part of the network composed of the negative memristor neural network 104 and the negative memristor neural network independent switch 105. The output result is regarded as symmetrical between the positive and negative driving sources. When the positive and negative voltages are turned on, the output voltage size and direction of the directional driving source are jointly determined by the bias conditions of the positive and negative half structures. During multi-modal fusion, the weight of each mode is determined by the scale of the corresponding memristor neural network 100. Based on the structural characteristics of the memristor neural network 100, when the network is turned on together, the memristor neurons between different memristor neural networks will also adaptively adjust the weights according to the Hebbian rule, thereby forming a coupling relationship between the training results of multiple independent modes. This makes multi-modal mixing no longer a simple weighted superposition, but a multi-modal fusion with a real coupling effect, and obtains knowledge points of mixed output. Finally, the system combines the knowledge points independently output by the memristor neural network 100 with the knowledge points of the mixed output to generate a decision confidence matrix, and the embedded processor 11 performs decision analysis to generate control commands to guide the execution module.

[0088] The memristor neural network 100 is composed of memristor neurons based on second-order photosensitive memristors, which are used to simulate the functional structure of neurons in biological neural networks. By applying appropriate external optical feedback and electrical feedback, an adaptive learning mechanism based on the Hebbian rule in biological neural networks is realized, and positive and negative Hebbian plasticity is possessed at the circuit level. In order to enhance the adaptability of the network, a variety of topological structures are designed for the network, which supports data input of different dimensions and forms and is compatible with various types of sensor signal sources. The memristor neurons in the network are laid out using a self-similar structure, and the bias state of the network is dynamically adjusted by adjusting the conductance value of the photosensitive memristor to adapt to changes in external information and achieve effective processing of complex multimodal data.

[0089] See also Figures 3 to 5, the present invention designs a one-dimensional memristor neural network 110, a two-dimensional memristor neural network 120 and a multi-dimensional memristor neural network 130 for voltage analog matrices of modal inputs of different dimensions, and its basic constituent unit is a memristor neuron constructed based on a second-order photosensitive memristor. The working principle of the second-order photosensitive memristor is based on the core characteristics of the memristor, that is, its conductance value changes with the application of voltage or current. Unlike ordinary memristors, the conductance state of the second-order photosensitive memristor is also affected by the dynamic regulation of light intensity, which makes its "second-order" characteristics not only dependent on external voltage, but also on light stimulation. In the one-dimensional, two-dimensional and multi-dimensional memristor neural networks, the conductance value of each second-order photosensitive memristor corresponds to the characteristic points of the physical structure in the input data one by one, and can directly receive the voltage analog matrix data collected by the sensor without additional encoding. The network is trained through a bionic learning and forgetting mechanism based on the Hebbian rule. The whole process is closer to the real training mode of biological neural networks, significantly reduces power consumption, and is more efficient and energy-saving than traditional machine learning methods.

[0090] See also Figure 3 The core unit of the one-dimensional memristor neural network 110 structure is a memristor neuron, each of which is composed of two forward or reverse second-order photosensitive memristors with the same polarity connected in series to form an input memristor group or a bias memristor group. The output principle of the network is based on the positive half structure between the positive driving source and the directional driving source, and the negative half structure between the directional driving source and the negative driving source. By applying symmetrical positive and negative voltages, due to the difference in conductance of different memristor groups, a result voltage corresponding to the bias state is generated on the directional driving source. After multiple memristor neurons are connected in series, the result voltage combination of each neuron under different biases constitutes the overall calculation output of the memristor neural network for the input data.

[0091] See also Figure 4 , Figure 5 , the two-dimensional memristor neural network 120 and the multi-dimensional memristor neural network 130 are both designed based on an I-shaped structure. The basic unit of the two-dimensional memristor neural network 120 is composed of four second-order photosensitive memristors connected in an I-shaped structure and combined in a self-similar I-shaped structure. The multi-dimensional memristor neural network 130 is composed of multiple two-dimensional memristor neural networks stacked together, and the specific number of layers depends on the dimension of the input data. The core working principles of two-dimensional and multi-dimensional memristor neural networks are similar to those of one-dimensional networks, and both rely on the voltage bias caused by the difference in conductance of different memristors as the output of the network. The difference is that the basic memristor neuron structure and the connection method between neurons are more complex to adapt to higher-dimensional data input.

[0092] The above contents are only for explaining the technical idea of ​​the present invention and cannot be used to limit the protection scope of the present invention. Any changes made on the basis of the technical solution in accordance with the technical idea proposed by the present invention shall fall within the protection scope of the claims of the present invention.

Claims

1. A multi-modal hybrid memristor neural network edge computing terminal system, characterized in that: Along the signal processing direction, it includes a multimodal sensor module (50), a servo light source training module (12), an edge computing module (13), an embedded processor (11) and an execution module (60) in sequence. The terminal system also includes a peripheral circuit (20), a cloud platform big data center (30) and a human-computer interaction module (40). The servo light source training module (12), the edge computing module (13) and the embedded processor (11) constitute a core module (10). The core module (10) also includes a wireless communication module (14). The cloud platform big data center (30) and the embedded processor (11) are bidirectionally interactively connected via the wireless communication module (14). The peripheral circuit (20) is respectively connected to the servo light source training module (12), the edge computing module (13) and the embedded processor (11) to support the operation of the system terminal; the multi-modal sensor module (50) is used to collect a multi-modal voltage simulation matrix; The edge computing module (13) is a multimodal hybrid memristor neural network, wherein the multimodal hybrid memristor neural network is a plurality of parallel-arranged memristor neural networks (100), wherein the number of the memristor neural networks (100) corresponds one-to-one to the number of modal inputs (101); the human-computer interaction module (40) comprises a touch display screen (41), a keyboard matrix (42), and a visualization module (43), which are respectively interactively connected to the embedded processor (11) to realize system result visualization and human-computer interaction functions.

2. A multimodal hybrid memristor neural network edge computing terminal system according to claim 1, characterized in that: The peripheral circuit (20) specifically comprises a signal processing module (22), a multi-level power management module (21) and a high-speed storage module (23); the signal processing module (22) is used to implement filtering, noise reduction and sampling and holding of system input signals; the multi-level power management module (21) is used to provide power supply at different voltage levels; and the high-speed storage module (23) is used to store data when the terminal system is running.

3. A multimodal hybrid memristor neural network edge computing terminal system according to claim 1, characterized in that: The memristor neural network (100) is one or more of a one-dimensional memristor neural network (110), a two-dimensional memristor neural network (120) or a multi-dimensional memristor neural network (130), and is used to support training requirements for modal data of different dimensions.

4. A multi-modal hybrid memristor neural network edge computing terminal system according to claim 3, characterized in that: The one-dimensional memristor neural network (110) comprises a plurality of memristor neurons (121), a negative driving source (90), a directional driving source (80) and a positive driving source (70); The memristor neuron (121) comprises an input second-order photosensitive memristor group and a bias second-order photosensitive memristor group, wherein the input second-order photosensitive memristor group is a first input memristor switch (112), a first positive-connected second-order photosensitive memristor (111), a second input memristor switch (112), and a second positive-connected second-order photosensitive memristor (111) connected in sequence; The biased second-order photosensitive memristor group comprises a first biased memristor switch (114), a first reverse-connected second-order photosensitive memristor (113), a second biased memristor switch (114), and a second reverse-connected second-order photosensitive memristor (113) connected in sequence; The positive driving source (70) is respectively connected to the first input memristor switch (112) and the first biasing memristor switch (114); the negative driving source (90) is respectively connected to the second positive-connected second-order photosensitive memristor (111) and the second reverse-connected second-order photosensitive memristor (113); the connection point between the first positive-connected second-order photosensitive memristor (111) and the second input memristor switch (112) and the connection point between the first reverse-connected second-order photosensitive memristor (113) and the second biasing memristor switch (114) are connected to the directional driving source (80); the directional driving source (80) is connected to an output bus, and the output node of the output bus is a knowledge output point (106) of the one-dimensional memristor neural network (110).

5. A multimodal hybrid memristor neural network edge computing terminal system according to claim 3, characterized in that: The two-dimensional memristor neural network (120) comprises a plurality of memristor neurons (121), a negative driving source (90), a directional driving source (80) and a positive driving source (70); the memristor neuron (121) comprises a first second-order photosensitive memristor (122), a second second-order photosensitive memristor (123), a third second-order photosensitive memristor (124) and a fourth second-order photosensitive memristor (125); The positive ends of the first second-order photosensitive memristor (122) and the third second-order photosensitive memristor (124) are connected to a positive driving source (70), and the negative ends of the first second-order photosensitive memristor (122) and the third second-order photosensitive memristor (124) are connected to a directional driving source (80); The positive ends of the second second-order photosensitive memristor (123) and the fourth second-order photosensitive memristor (125) are connected to a directional driving source (80), and the negative ends of the second second-order photosensitive memristor (123) and the fourth second-order photosensitive memristor (125) are connected to a negative driving source (90); The memristor neurons (121) are connected to a directional drive source (80) in a self-similar I-shaped structure via a switching switch (126), wherein the directional drive source (80) is connected to an output bus, and the output node of the output bus is a knowledge output point (106) of the two-dimensional memristor neural network (120).

6. A multi-modal hybrid memristor neural network edge computing terminal system according to claim 5, characterized in that: The multidimensional memristor neural network (130) is a plurality of two-dimensional memristor neural networks (120) arranged in parallel; The knowledge output points (106) of each two-dimensional memristor neural network (120) are connected in parallel on an output bus, and the output nodes of the output bus are the knowledge output points (106) of the multi-dimensional memristor neural network (130).

7. A multimodal hybrid memristor neural network edge computing terminal system according to any one of claims 4 to 6, characterized in that: The servo light source training module (12) comprises a light source matrix, wherein the point light sources in the light source matrix correspond one-to-one to the second-order photosensitive memristors in the multimodal hybrid memristor neural network, and is used to adjust the light brightness and duration of the point light sources according to the multimodal voltage simulation matrix, thereby realizing initial training of the multimodal hybrid memristor neural network.

8. A multimodal hybrid memristor neural network edge computing method, characterized in that: The multimodal hybrid memristor neural network edge computing terminal system according to claim 1 comprises the following steps: Step 1: According to the modality involved in the actual problem, the number and dimension of the modal input (101) of the multimodal sensor module (50) are determined, and the number and dimension of the memristor neural network (100) are determined according to the number and dimension of the modal input (101); the modal type of the collected signal is obtained, and a multimodal voltage simulation matrix is ​​collected; Step 2: filtering, denoising and sampling and holding the multi-modal voltage simulation matrix to obtain a processed multi-modal voltage simulation matrix; Step 3: The servo light source training module (12) adjusts the light intensity and illumination time of the point light source according to the processed multi-modal voltage simulation matrix, stimulates the memristor neural network (100) to perform initial assignment, and realizes bionic learning; Step 4: by controlling the state of each memristor neural network (100) to read the single network training result for a single mode, the independent output knowledge points of the memristor neural network (100) in each mode are obtained; based on current drive and Hebbian rule, the weight of the memristor neural network (100) is adaptively adjusted to achieve self-coordinated update of the memristor neural network (100) in each mode, so as to simulate bionic forgetting; Step 5: Read the mixed output knowledge points after multimodal fusion by controlling the mixed switching of the multimodal hybrid memristor neural network; adaptively adjust the weights of the multimodal hybrid memristor neural network based on current drive and Hebbian rule to achieve adaptive fusion of multimodal information; construct a confidence matrix based on independent output knowledge points and mixed output knowledge points to generate personalized decision commands; Step 6: The decision command is transmitted to the execution module (60) to instruct the execution module (60) on a specific movement form, thereby ending the main process.

9. A multimodal hybrid memristor neural network edge computing method according to claim 8, characterized in that: When step 3 or step 4 is completed, the following steps are also included: The data of the terminal system during operation is stored through the peripheral circuit (20); the process and result data of the terminal system operation are uploaded to the cloud platform big data center (30) to accumulate and analyze historical data to support training strategy update and parameter optimization, and realize remote maintenance and intelligent management; the process and result data of the terminal system operation are displayed through the human-computer interaction module (40); and the user controls and views the training parameters and visualization content of the terminal system through the human-computer interaction module (40).

10. A multimodal hybrid memristor neural network edge computing method according to claim 8, characterized in that: When step six is ​​executed, the following steps are also included: uploading the process and result data of the terminal system operation to the cloud platform big data center (30) to accumulate and analyze historical data to support training strategy updates and parameter optimization, and realize remote maintenance and intelligent management; and displaying the process and result data of the terminal system operation through the human-computer interaction module (40).

Images