An edge computing device based on multimodal brain-like active perception

Abstract

The present invention relates to an edge computing device based on multimodal brain-inspired active perception, comprising: a perception neuron module for acquiring perception information of different modalities and pre-processing and encoding the perception information into pulse sequences with a predetermined format; a transfer neuron module for parallel storage and processing of the pulse sequences of different modalities, extracting feature vectors of the corresponding modality; and a brain-inspired multimodal processing module for selecting and allocating the feature vectors of different modalities, performing information fusion and scene understanding processing, and obtaining decision-making information. Compared with existing technologies, the present invention has the advantages of high computational efficiency and high accuracy of multimodal information fusion, thereby achieving more reliable scene understanding.

Description

Technical Field

[0001] The present invention belongs to the field of AI computing and relates to a brain-like intelligent terminal, and in particular to an edge computing device based on multimodal brain-like active perception. Background Art

[0002] Computing power, algorithms, and data are the three essential elements of artificial intelligence. In recent years, traditional von Neumann computing systems and single-task, single-modality AI algorithms have been unable to keep up with the growing speed and processing demands of massive multimodal data. Some scholars have proposed finding improved models based on brain science research, and brain-inspired intelligence has become a key development direction for artificial intelligence. Although much remains to be uncovered about the structure and function of the human brain, the basic units of the human nervous system, which combine information storage and computational reasoning, hold promise as a technological path to breaking the constraints of the von Neumann architecture. In recent years, the convergence of brain science with computing technology, artificial intelligence, nanomaterials, cognitive psychology, and other disciplines has laid the theoretical and technological foundation for the development of brain-inspired intelligent computing systems. In particular, breakthroughs in key devices and technologies, such as memristors and neuromorphic computing, have made it possible to simulate the structure and processing mechanisms of the brain's nervous system.

[0003] However, existing brain-like systems still have problems with low efficiency and high energy consumption in multimodal information fusion and perception computing. Summary of the Invention

[0004] The purpose of the present invention is to overcome the defects of the above-mentioned prior art and to provide an edge computing device based on multimodal brain-like active perception, which solves the problems of low efficiency and high energy consumption of multimodal information fusion and perception computing of smart terminals.

[0005] The purpose of the present invention can be achieved by the following technical solutions:

[0006] An edge computing device based on multimodal brain-like active perception, comprising:

[0007] A perception neuron module is used to obtain perception information of different modalities and pre-process and encode the perception information into a pulse sequence with a set format;

[0008] A transfer neural module, configured to store and process the pulse sequences of different modalities in parallel and extract feature vectors of the corresponding modalities;

[0009] The brain-like multimodal processing module is used to select and allocate the feature vectors of different modalities, and perform information fusion and scene understanding processing to obtain decision-making information.

[0010] Furthermore, the perception neuron module includes a connected sensor chip and a preprocessing unit, the sensor chip includes at least two of a light wave sensor, a microwave sensor, an acoustic wave sensor, and an analog electronic dedicated sensor, the preprocessing unit includes a first unit connected to the microwave sensor, the acoustic wave sensor or the analog electronic dedicated sensor respectively and a second unit connected to the light wave sensor, the first unit collects a pulse sequence of the corresponding sensor chip based on a memristor, and the second unit collects light wave data of the light wave sensor and obtains a corresponding pulse sequence after sequence conversion.

[0011] Furthermore, the first unit correspondingly connected to the analog electronic dedicated sensor further performs: uniformly adjusting the frequency of the collected pulse sequence.

[0012] Furthermore, the transfer neural module includes multiple storage and processing units for different modalities arranged in parallel, each storage and processing unit includes a connected memristor array and a brain-like chip, wherein:

[0013] Furthermore, the memristor array is used to store the acquired pulse sequence;

[0014] Furthermore, the brain-like chip simulates the computational mechanism of neurons in different functional areas of the brain in response to stimulation, extracts features from the received pulse sequences, and obtains feature vectors of corresponding modalities.

[0015] Furthermore, a signal transmission structure is provided between the brain-like chips, which is composed of multiple simulated neurons. When the rear neuron receives stimulation from the front neuron, if the potential exceeds the threshold, the rear neuron sends an action potential in the form of a pulse to realize information transmission.

[0016] Furthermore, the brain-like multi-mode processing module includes a multiplexer and multiple memristor-based perception fusion processing channels, wherein:

[0017] The multiplexer is used to select and allocate the acquired feature vectors of different modalities according to data requirements, and select different perception fusion processing channels for data processing;

[0018] Each of the perception fusion processing channels processes feature vectors of different modalities based on a parallel competitive feedback mechanism to obtain a final fusion output result, and performs scene understanding based on the final fusion output result. The perception fusion processing channel is a neuron circuit.

[0019] Furthermore, the selection and deployment is implemented by an active selection mechanism based on a state machine, and the state machine is implemented based on a pulse neural network.

[0020] Furthermore, the parallel competition feedback mechanism is constructed based on a continuous attractor network.

[0021] Furthermore, the parallel competition feedback mechanism is specifically as follows:

[0022] Multiple neuronal circuits processing different modalities are processed synchronously in a parallel structure. Once the membrane potential of a spiking neuron in a neuronal circuit corresponding to a certain modality reaches the threshold, the output of the neuronal circuit is fed back as the most significant signal to the input of other neuronal circuits to guide the output of other neuronal circuits.

[0023] Furthermore, each of the neuron circuits is linked and communicated in real time during the process of processing different modal information.

[0024] Compared with the prior art, the present invention has the following beneficial effects:

[0025] 1. The present invention refers to the mechanism of information processing in the brain and designs an edge computing device that includes a perception neuron module, a transmission neuron module, and a brain-like multimodal processing module. From a brain-like bionic perspective, it solves the problems of low efficiency and high energy consumption of multimodal information fusion and perception computing in smart terminals.

[0026] 2. The present invention realizes information selection and processing based on the brain's attention selection mechanism for multimodal information, thereby improving the efficiency and accuracy of multimodal information fusion.

[0027] 3. The present invention utilizes the neuromorphic memristor principle and distributed storage technology to realize the simulation transmission process of neural signals and the design of the data bus, converting multimodal information into a unified format with higher efficiency and transmitting it on the bus network.

[0028] 4. The brain-like chip of the present invention maps different brain-like areas for information processing and realizes information exchange between different chips through the chip network. Multiple brain-like chip processors constitute a bionic brain to improve the accuracy of information processing.

[0029] 5. The present invention realizes storage and computing integration through memristors, is not limited by the von Neumann bottleneck in traditional computer architecture, and greatly improves the energy efficiency of computing. BRIEF DESCRIPTION OF THE DRAWINGS

[0030] Figure 1 It is a structural schematic diagram of the present invention;

[0031] Figure 2 This is a simplified diagram of the working principle of the perception neuron module of the present invention;

[0032] Figure 3 This is a simplified diagram of the working principle of the active selection mechanism based on the state machine of the present invention;

[0033] Figure 4 This is a simplified diagram of the working principle of the parallel competition feedback mechanism based on the continuous attractor network of the present invention. DETAILED DESCRIPTION

[0034] The present invention is described in detail below with reference to the accompanying drawings and specific embodiments. This embodiment is implemented based on the technical solution of the present invention, and provides a detailed implementation method and specific operation process, but the protection scope of the present invention is not limited to the following embodiments.

[0035] like Figure 1 As shown, this embodiment provides an edge computing device based on multimodal brain-like active perception, including a perception neuron module 1, a transfer neuron module 2, and a brain-like multimodal processing module 3. The perception neuron module 1 is used to obtain perception information of different modalities and pre-process and encode the perception information into a pulse sequence with a set format; the transfer neuron module 2 is used to store and process the pulse sequences of different modalities in parallel and extract the feature vectors of the corresponding modalities; and the brain-like multimodal processing module 3 is used to select and allocate the feature vectors of different modalities, perform information fusion and scene understanding processing, and obtain decision information. After obtaining the decision information, remote interactive operations can also be performed.

[0036] The perception neuron module 1 obtains different perception information, performs pre-processing encoding for subsequent storage and processing, and then stores, transcodes, and transmits it to the brain-like multimodal processing module 3 through the transfer neuron module 2, and uses the pulse neural network to perform multimodal information fusion and understanding analysis, and finally realizes the multimodal brain-like active perception task of integrated sensing, storage and computing. In this embodiment, the perception neuron module includes a connected sensor chip and a pre-processing unit, the sensor chip includes at least two of a light wave sensor, a microwave sensor, an acoustic wave sensor, and an analog electronic dedicated sensor, the pre-processing unit includes a first unit connected to the microwave sensor, the acoustic wave sensor or the analog electronic dedicated sensor, and a second unit connected to the light wave sensor, the first unit collects the pulse sequence of the corresponding sensor chip based on the neuromorphic memristor, and the second unit collects the light wave data of the light wave sensor and obtains the corresponding pulse sequence after sequence conversion.

[0037] The information contents of different modalities perceived by the perception neuron module 1 are different, and after being converted into pulses, they are not in the same dimension. Therefore, the modal information needs to be unified through the encoder. Specifically, Figure 2 As shown, different sensor chips in the perception neuron module 1 acquire data of different modalities, and uniformly pre-process and encode different types of multi-source heterogeneous data into a unified format pulse sequence. Data collected by light wave sensors, etc. that cannot be directly collected in the form of pulses, are transmitted to the second unit after data collection, and the multi-modal information is further converted into a pulse sequence through conversion to obtain input. Specifically:

[0038] The optical wave sensor data needs to be converted into a sequence. The image is divided according to the required accuracy, and the characteristics of the block are distinguished by high and low voltage. If there is a 16*16 block, a pulse sequence period of 256 is required to represent it.

[0039] The acoustic wave, microwave and analog electronic information are directly obtained as pulse sequences through the acquisition unit based on the memristor. Among them, the data with time characteristics obtained by the analog electronic dedicated sensor needs to be converted into a unified period T according to the set period. a ,follow The pulse frequency is uniformly adjusted for unified processing by the multimodal fusion module.

[0040] In a specific embodiment, the perception neuron module may include multiple dedicated AI sensor chips for collecting data of different modalities, including light wave sensing chips, microwave sensing chips, sound wave sensing chips, analog electronic dedicated sensor chips, etc., and realize distributed storage through memristors.

[0041] The transfer neural module is a dedicated transfer neural bus composed of memristors and encoders, designed to encode sensor information from different modalities and coordinate communication links. The innovative bus architecture of neural signal transmission simulates neuronal transmission. Memristors and a sensor fusion array receive incoming stimuli and generate pulse trains. Information is then transferred between different modal processing links via a signal transmission structure. The signal transmission structure consists of simulated neurons, with memristors simulating synaptic function. Capacitors, memristors, and other components form the basic neuronal structure. Synapses dynamically receive pulse trains from the axons of upstream neurons and transmit them to the cell body. The cell body dynamically receives current and ultimately emits asynchronous pulse signals, simultaneously inducing a refractory period and potential reset. When a synapse receives an action potential from the preceding neuron, it generates a postsynaptic potential, which stimulates the membrane of the following neuron. If the potential exceeds a threshold, the postsynaptic neuron triggers, emitting an action potential in the form of a pulse.

[0042] In one specific embodiment, the transfer neural module includes multiple parallel storage and processing units for different modalities. Each storage and processing unit includes a connected memristor array and a brain-inspired chip. The memristor array is used to store acquired pulse trains. The brain-inspired chip simulates the computational mechanism of neurons in different functional areas of the brain in response to stimulation, extracting features from the received pulse trains to obtain feature vectors for the corresponding modality. After receiving the incoming pulse trains through the synaptic structure constructed by the neuromorphic memristors, the brain-inspired chip simulates the different responses of neurons in different cell groups to the specialized properties of the stimulation, obtaining the more significant feature vectors for that modality.

[0043] Information transmission between different brain-inspired chips is carried out using the aforementioned signal transmission structure. Specifically, during the processing process, different modal information is exchanged within each processing module. Different brain-inspired chips communicate through the chip network through pulses, with the pulse signals encoded in a unified format: chip address + transmission content.

[0044] After the brain-like multimodal processing module obtains the processed data of each modality, it uses the parallel competitive feedback mechanism constructed based on the continuous attractor network and the multimodal information fusion decision module to process and calculate the multimodal information, completing information fusion and scene understanding such as data fusion, feature fusion, and semantic fusion based on the pulse neural network.

[0045] In a specific embodiment, the brain-like multi-modal processing module includes a multiplexer and multiple memristor-based perception fusion processing channels, wherein the multiplexer is used to select and allocate the acquired feature vectors of different modalities according to data requirements, and select different perception fusion processing channels for data processing; each of the perception fusion processing channels processes the feature vectors of different modalities based on a parallel competitive feedback mechanism to obtain a final fusion output result, and performs scene understanding based on the final fusion output result, and the perception fusion processing channel is a neuronal circuit.

[0046] Each modality perception neuron D1, D2, D3...D n After the raw sensor data is processed, each modal pulse sequence is sent through the transmission neuron. The brain actively selects the modal input data based on the state machine according to the data requirements and processes the data in different channels. The state machine is implemented based on the pulse neural network. After the channel processing of the active selection mechanism, the modal coding sequence enters the brain-like multi-modal processing unit for fusion and understanding. Figure 3 As shown, the selection and allocation is implemented using an active selection mechanism based on a state machine, which is implemented based on a spiking neural network. Different requirements may arise based on data characteristics, such as a greater emphasis on visual information or analog information. Based on these different requirements, the active selection mechanism selects different data processing channels for processing.

[0047] The brain-like multimodal processing module is used to perform data fusion, feature fusion, and semantic fusion on the feature vectors of different input modalities, and to understand the scene and make the final decision based on the fused data. Multimodal fusion includes group encoding based on the incoming data and multimodal fusion based on Bayesian optimization information integration to integrate and unify the input different modal information. The parallel competition feedback mechanism in feature extraction multimodal information fusion is constructed based on the continuous attractor network CANN, such as Figure 4 As shown, specifically including:

[0048] Referring to the attention selection mechanism in the human brain, when multimodal information is input simultaneously, the most significant modality is extracted as the dominant one, which guides the processing of other modal information. In this device, the neuronal circuits that process different modalities are parallel and synchronously processed. Once the membrane potential of a spiking neuron in a certain modal pathway reaches the threshold, the remaining spiking neurons in the circuit will be reset to the resting potential, and the most significant signal will be returned to the input to guide the processing of other modal information. That is, once the most significant feature is sufficient to support the result judgment, the feedback signal will be used to guide the output of other routes in the brain-like system. At the same time, there is real-time connection and communication between different modalities during the modal processing process.

[0049] Modal output M i =p(O i ,R j ,I ai ,I bi ,…,I xi ), where M i is the final output of mode i after undergoing the processing unit, O i To process the initial input of array mode i, R j is the feedback input of the most significant mode, I ni It is the real-time impact of mode n on mode i during the processing.

[0050] In other embodiments, the brain-inspired multimodal processing module also has a pulse learning function, which can learn based on historical processing data of different modalities to further improve fusion accuracy and scene understanding accuracy.

[0051] The construction of the above-mentioned device is realized through memristors. In addition to realizing the storage and computing functions, the storage function of the memristor array is realized by the memristor array during the pulse learning process. The characteristic of memristors is storage and computing in an integrated manner, which is closer to the concept of neurons. Because the amount of data stored by a single memristor is limited, the memristors are grouped according to the data size and stacked in a specific structure to store a unit of data. Each group of memristors stacks several memristor array grain layers, and the peripheral circuit performs data refresh, retention, update, and clearing operations to realize distributed storage of data, effectively solving the von Neumann bottleneck.

[0052] It will be understood by those skilled in the art that the embodiments of the present invention may be provided as methods, systems, or computer program products. Therefore, the present invention may take the form of a complete hardware embodiment, a complete software embodiment, or an embodiment combining software and hardware. Furthermore, the present invention may take the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code. The solutions in the embodiments of the present invention may be implemented in various computer languages, for example, the object-oriented programming language Java and the interpreted scripting language JavaScript.

[0053] The present invention is described with reference to flowcharts and / or block diagrams of methods, devices (systems), and computer program products according to embodiments of the present invention. It should be understood that each process and / or block in the flowcharts and / or block diagrams, as well as combinations of processes and / or blocks in the flowcharts and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing device to produce a machine, so that the instructions executed by the processor of the computer or other programmable data processing device generate instructions for implementing the processes in the flowcharts and / or block diagrams. Figure 1 a process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.

[0054] It should be understood that the terms used in this manner are interchangeable where appropriate for the embodiments of the present application described herein. In addition, the terms "comprises" and "comprising" and any variations thereof are intended to cover non-exclusive inclusions, e.g., a process, method, system, product, or apparatus that includes a series of steps or elements is not necessarily limited to those steps or elements explicitly listed, but may include other steps or elements not explicitly listed or inherent to such process, method, product, or apparatus.

[0055] The above describes the preferred embodiments of the present invention. It should be understood that the present invention is not limited to the above-mentioned specific embodiments, and the devices and structures that are not described in detail should be understood to be implemented in a common manner in the art; any technician familiar with the art can use the above-mentioned disclosed methods and technical contents to make many possible changes and modifications to the technical solutions of the present invention without departing from the scope of the technical solutions of the present invention, or modify them into equivalent embodiments of equivalent changes, which does not affect the essential content of the present invention. Therefore, any simple modifications, equivalent changes and modifications made to the above embodiments based on the technical essence of the present invention that do not depart from the content of the technical solutions of the present invention still fall within the scope of protection of the technical solutions of the present invention.

Claims

1. An edge computing device based on multimodal brain-like active perception, characterized in that: include: A perception neuron module is used to obtain perception information of different modalities and pre-process and encode the perception information into a pulse sequence with a set format; A transfer neural module, configured to store and process the pulse sequences of different modalities in parallel and extract feature vectors of the corresponding modalities; A brain-inspired multimodal processing module, which is used to select and allocate the feature vectors of different modalities, perform information fusion and scene understanding processing, and obtain decision-making information; The brain-like multi-mode processing module includes a multiplexer and multiple memristor-based perception fusion processing channels, wherein: The multiplexer is used to select and allocate the acquired feature vectors of different modalities according to data requirements, and select different perception fusion processing channels for data processing; Each of the perception fusion processing channels processes feature vectors of different modalities based on a parallel competitive feedback mechanism to obtain a final fusion output result, and performs scene understanding based on the final fusion output result, wherein the perception fusion processing channel is a neuron circuit; The parallel competition feedback mechanism is specifically as follows: Multiple neuronal circuits processing different modalities are processed synchronously in a parallel structure. Once the membrane potential of a spiking neuron in a neuronal circuit corresponding to a certain modality reaches the threshold, the output of the neuronal circuit is fed back as the most significant signal to the input of other neuronal circuits to guide the output of other neuronal circuits.

2. The edge computing device based on multimodal brain-like active perception according to claim 1, characterized in that: The perception neuron module includes a connected sensor chip and a preprocessing unit, the sensor chip includes at least two of a light wave sensor, a microwave sensor, an acoustic wave sensor, and an analog electronic dedicated sensor, the preprocessing unit includes a first unit connected to the microwave sensor, the acoustic wave sensor, or the analog electronic dedicated sensor, and a second unit connected to the light wave sensor, the first unit collects a pulse sequence of the corresponding sensor chip based on a memristor, and the second unit collects light wave data of the light wave sensor and obtains a corresponding pulse sequence through sequence conversion.

3. The edge computing device based on multimodal brain-like active perception according to claim 2, characterized in that: The first unit correspondingly connected to the analog electronic dedicated sensor further performs: uniformly adjusting the frequency of the collected pulse sequence.

4. The edge computing device based on multimodal brain-like active perception according to claim 1, characterized in that: The transfer neural module includes multiple storage and processing units arranged in parallel for different modalities, each storage and processing unit includes a connected memristor array and a brain-like chip, wherein: The memristor array is used to store the acquired pulse sequence; The brain-like chip simulates the computational mechanism of neurons in different functional areas of the brain in response to stimulation, extracts features from received pulse sequences, and obtains feature vectors of corresponding modalities.

5. The edge computing device based on multimodal brain-like active perception according to claim 4 is characterized in that: A signal transmission structure is set between the brain-like chips. The signal transmission structure is composed of multiple simulated neurons. When the rear neuron receives stimulation from the front neuron, if the potential exceeds the threshold, the rear neuron will emit an action potential in the form of a pulse to realize information transmission.

6. The edge computing device based on multimodal brain-like active perception according to claim 1, characterized in that: The selection and deployment is implemented by an active selection mechanism based on a state machine, and the state machine is implemented based on a pulse neural network.

7. The edge computing device based on multimodal brain-like active perception according to claim 1, characterized in that: The parallel competition feedback mechanism is constructed based on a continuous attractor network.

8. The edge computing device based on multimodal brain-like active perception according to claim 1, characterized in that: The neuron circuits are linked and communicated in real time during the process of processing different modal information.

Images