Neuronal models, topological structures, information processing methods and retinal neurons

Abstract

The present invention discloses a neuron model, a neuron topology, an information processing method and a retinal neuron topology. The neuron model comprises: at least two independent neural network units, each of which is used to process signals of different data formats; wherein, among the neural network units of the neuron model, one neural network unit is selected. The technical solution of the embodiment of the present invention optimizes the structure of the existing neuron model, realizes the mixed processing of signals of multiple data formats in the process of simulating neuron functions, and meets the growing demand for the construction of personalized and convenient neuron topology structures.

Description

Technical Field

[0001] The embodiments of the present invention relate to artificial intelligence technology, and in particular to a neuron model, a neuron topology structure, an information processing method and a retinal neuron topology structure. Background Art

[0002] Artificial intelligence is a discipline that studies how to use computers to simulate certain human thought processes and intelligent behaviors (such as learning, reasoning, thinking or planning). It mainly includes the principles of how computers achieve intelligence, how to create computers that are similar to human brain intelligence, and how to enable computers to achieve higher-level applications.

[0003] With the continuous development of artificial intelligence technology, various neural network units have emerged to simulate neurons in the human brain, such as ANN (Artificial Neural Network) units or SNN (Spiking Neuron Networks) units.

[0004] In the prior art, a certain type of neural network unit can only process signals in a certain data format. When the human brain processes a complex processing function, such as retinal imaging, it may involve mixed processing of signals in multiple data formats. The various neural network units provided by the prior art cannot cope with the above complex data processing requirements. Summary of the invention

[0005] The embodiments of the present invention provide a neuron model, a neuron topology structure, an information processing method and a retinal neuron topology structure, so as to realize mixed processing of signals in various data formats in the process of simulating complex neuron functions.

[0006] In a first aspect, an embodiment of the present invention provides a neuron model, the neuron model comprising: at least two independent neural network units, each of the neural network units being used to process signals in different data formats;

[0007] Among them, among the neural network units of the neuron model, one neural network unit is selected.

[0008] In a second aspect, an embodiment of the present invention further provides a neuron topology structure, the neuron topology structure comprising: a first neuron model, a second neuron model and a data format conversion device;

[0009] The first neuron model and the second neuron model are connected via the data format conversion device, and different types of neural network units are respectively selected in the first neuron model and the second neuron model;

[0010] The data format conversion device is used to convert a signal in a first data format adapted by the first neuron model into a signal in a second data format adapted by the second neuron model;

[0011] The first neuron model and the second neuron model are the neuron models described in any embodiment of the present invention.

[0012] In a third aspect, an embodiment of the present invention further provides an information processing method, which is applied to a neural network, wherein the neural network is obtained by networking a plurality of neural models as described in any embodiment of the present invention in accordance with a set connection mode, and the information processing method comprises:

[0013] Receiving a network input signal input into the neural network, wherein the network input signal includes routing information, and the routing information includes: a routing neuron model identifier and a neural network unit identifier selected in the routing neuron model;

[0014] Through each of the neuron models, routing selection information is parsed in the currently received target input signal; when it is determined that the target input signal needs to be routed based on the routing selection information, the neural network unit matching the routing selection information is selected to process the target input signal and output the processing result.

[0015] In a fourth aspect, an embodiment of the present invention further provides a retinal neuron topology structure, wherein the topology structure comprises: at least one retinal cell simulation layer, each of the retinal cell simulation layers comprises a plurality of neuron models as described in any embodiment of the present invention;

[0016] The topological structure is used to process multiple input single color image signals and multiple grayscale description signals through at least one retinal cell simulation layer to obtain color reconstruction signals and optical flow reconstruction signals.

[0017] The technical solution of the embodiment of the present invention provides a new type of neuron model, which includes at least two independent neural network units, each of which is used to process signals of different data formats, and among the neural network units of the neuron model, only one neural network unit is selected. By applying the above-mentioned neuron model, a large-scale neuron topology structure that meets the neuron processing function can be constructed by means of selection control, which solves the technical defect that the prior art cannot simulate neurons to perform mixed processing of heterogeneous information, proposes a new form of heterogeneous fusion neural network hybrid operation, optimizes the structure of the existing neuron model, and realizes mixed processing of signals of multiple data formats in the process of simulating complex neuron functions, meeting people's growing demand for the construction of personalized and convenient neuron topology structures. BRIEF DESCRIPTION OF THE DRAWINGS

[0018] Figure 1 is a schematic diagram of a neuron model in Embodiment 1 of the present invention;

[0019] Figure 2 is a schematic diagram of a neuron model in Embodiment 2 of the present invention;

[0020] Figure 3 is a schematic diagram of a neuron model in Embodiment 3 of the present invention;

[0021] Figure 4a is a schematic diagram of a neuron topology structure in Embodiment 4 of the present invention;

[0022] Figure 4b It is a schematic diagram of the structure of a pulse neural network unit applicable to the fourth embodiment of the present invention;

[0023] Figure 5a is a flowchart of an implementation of an information processing method in Embodiment 5 of the present invention;

[0024] Figure 5b is a schematic diagram of a neural network applicable to the fifth embodiment of the present invention;

[0025] Figure 6a is a schematic diagram of a retinal neuron topological structure in Embodiment 6 of the present invention;

[0026] Figure 6b It is a schematic diagram of a specific retinal neuron topological structure in Example 6 of the present invention. DETAILED DESCRIPTION

[0027] The present invention will be further described in detail below in conjunction with the accompanying drawings and embodiments. It is to be understood that the specific embodiments described herein are only used to explain the present invention, rather than to limit the present invention. It should also be noted that, for ease of description, only parts related to the present invention, rather than all structures, are shown in the accompanying drawings.

[0028] It should also be noted that, for ease of description, only the part relevant to the present invention but not all content is shown in the accompanying drawings. It should be mentioned before discussing exemplary embodiments in more detail that some exemplary embodiments are described as processing or methods depicted as flow charts. Although the flow chart describes each operation (or step) as sequential processing, many operations therein can be implemented in parallel, concurrently or simultaneously. In addition, the order of each operation can be rearranged. When its operation is completed, the processing can be terminated, but it can also have additional steps not included in the accompanying drawings. The processing can correspond to methods, functions, procedures, subroutines, subprograms, etc.

[0029] Embodiment 1

[0030] Figure 1 A schematic diagram of a neuron model provided in Example 1 of the present invention is shown in FIG. Figure 1 As shown, the neuron model includes at least two independent neural network units (in Figure 1 Shown in the figure are neural network unit 1, neural network unit 2, ..., neural network unit n).

[0031] Each of the neural network units is used to process signals of different data formats. Among the neural network units of the neuron model, one neural network unit is selected.

[0032] In this embodiment, the neural network unit specifically refers to a minimum computing unit for simulating the neuron computing function. The neural network unit can be implemented by software and / or hardware. In a certain neural network unit, a signal of a certain data format can be processed. The data format may include: analog signals, pulse signals, and digital level signals. In this neuron model, different neural network units are used to process signals of different data formats.

[0033] Generally speaking, the calculation accuracy or signal encoding method corresponding to signals in different data formats are different. For example, if neural network unit A is used to process analog signals, then the neural network unit A can process and transmit higher precision signals, but because the neural network unit A cannot encode analog signals, the form of computing tasks that the neural network unit A can handle will be relatively single, and the application scenarios are limited; if neural network unit B is used to process pulse signals, then the neural network unit B can be used to process signals with rich coding forms, such as time domain coding or space-time domain coding. Furthermore, the application scenarios of the neural network unit B are relatively wide, but the signal accuracy is not high.

[0034] As mentioned above, when constructing a neural network of a certain type, the prior art generally only uses a combination of neural network units that process signals of a certain data format. Therefore, the neural network can only process signals of a certain data format. In this embodiment, the inventor creatively proposed a new structure of a neuron model. In this neuron model, multiple independent neural network units are included at the same time. The multiple neural network units are used to process signals of different data formats respectively. When used in practice, the data format of the signal can be processed according to actual needs. In this neuron model, the matching neural network unit is selected. By using the neuron topology structure obtained by combining the above-mentioned multiple neural network units, mixed processing of signals of different data formats can be achieved.

[0035] Optionally, the neuron model may have multiple input terminals and multiple matching output terminals, and different input and output terminals may be selected to select different neural network units; or, the neuron model may have a single input terminal and a single output terminal, and a selection switch inside the neuron model, through which the selected neural network unit inside the neuron model can be selected and connected to the single input terminal and the single output terminal.

[0036] Alternatively, different output terminals may be configured according to different neural network units selected by the neuron model or different signal processing requirements.

[0037] For example, if the neuron model selects a neural network unit for processing analog signals, the output end of the neural network unit can be directly used as the output end of the neuron model; if the neuron model selects a neural network unit for processing pulse signals, and only a single pulse signal processed by the neuron model is needed subsequently, the output end of the neural network unit can be directly used as the output end of the neuron model; or, if the neuron model selects a neural network unit for processing pulse signals, and only multiple pulse signals processed by the neuron model are needed subsequently, all (or multiple) output ends of the last hidden layer of the neural network unit can be selected as the output end of the neuron model, etc. This embodiment does not limit this.

[0038] The technical solution of the embodiment of the present invention provides a new type of neuron model, which includes at least two independent neural network units, each of which is used to process signals of different data formats, and among the neural network units of the neuron model, only one neural network unit is selected. By applying the above-mentioned neuron model, a large-scale neuron topology structure that meets the neuron processing function can be constructed by means of selection control, which solves the technical defect that the prior art cannot simulate neurons to perform mixed processing of heterogeneous information, proposes a new form of heterogeneous fusion neural network hybrid operation, optimizes the structure of the existing neuron model, and realizes mixed processing of signals of multiple data formats in the process of simulating complex neuron functions, meeting people's growing demand for the construction of personalized and convenient neuron topology structures.

[0039] On the basis of the above embodiments, the neural network unit in the neuron model may include: a first neural network unit and a second neural network unit;

[0040] The first neural network unit is used to process and output analog signals; the second neural network unit is used to process and output pulse signals.

[0041] In this embodiment, by setting a first neural network unit for processing analog signals and a second neural network unit for processing pulse signals in the neural model, a heterogeneous fusion neural network capable of mixed processing of analog signals and pulse signals can be constructed by uniformly using the neural model provided by the embodiment of the present invention, so that the heterogeneous fusion neural network has greater signal accuracy when processing analog signals and more encoding options when processing pulse signals.

[0042] Based on the above embodiments, the first neural network unit may be an artificial neural network (ANN) unit, and the second neural network unit may be a spiking neural network (SNN) unit.

[0043] Among them, ANN units directly process analog information and transmit high-precision activations. They show strong capabilities in scenarios that require high-precision calculations, but involve huge computing operations, resulting in high power consumption and large delays. SNN units memorize historical time information through intrinsic neuronal dynamics and input code information into digital spike sequences to achieve event-driven computing. The input data usually contains time information, and the data stream is sparse. Therefore, SNN units have richer information encoding methods than ANN units, such as time domain coding, spatiotemporal domain coding, group coding, Bayesian coding, time lag coding, sparse coding, etc. Due to the ability to perform spatiotemporal integrated coding, SNN units have great potential in processing complex spatiotemporal information and multimodal information tasks. However, the discontinuity of SNN units, the complexity of spatiotemporal coding, and the uncertainty of network structure make it difficult to mathematically complete the description of the entire network and to construct an effective and general supervised learning algorithm. Therefore, SNN units usually have limited computational complexity and low energy consumption, but low accuracy. It can be seen that the advantages of ANN units and SNN units complement each other and are suitable for different application scenarios. Furthermore, by using the neuron model provided in the embodiment of the present invention, it is possible to adapt to application scenarios that comprehensively utilize the advantages of ANN units and SNN units.

[0044] Embodiment 2

[0045] Figure 2 A schematic diagram of a neuron model provided in the second embodiment of the present invention. In this embodiment, the neural network unit in the neuron model is concretized as a first neural network unit and a second neural network unit, and a gating device is also included in the neuron model. Figure 2 As shown, the neuron model includes: a gating device 210, a first neural network unit 220 and a second neural network unit 230. The gating device 210 is connected to the first neural network unit 220 and the second neural network unit 230 respectively.

[0046] The gating device 210 is used to select the first neural network unit 210 or the second neural network unit 230 in the neuron model.

[0047] In this embodiment, the neuron model may have only a single input terminal and a single output terminal. Inside the neuron model, the single input terminal is connected to the input terminal of the gating device 210, and the output terminal of the gating device 210 is simultaneously connected to the input terminals of the first neural network unit 220 and the second neural network unit 230, and the output terminals of the first neural network unit 220 and the second neural network unit 230 are respectively connected to the single output terminal.

[0048] The gating device may be a manual device (e.g., a dip switch or a touch switch) or a programmable device (e.g., a programmable switch or a multiple-choice data selector, a two-choice data selector in this embodiment), etc., and this embodiment does not limit this. By using the gating device, the path selected by the gating device can be controlled by manual control or controller control, so that the neuron model selects the desired neural network unit.

[0049] The technical solution of the embodiment of the present invention simplifies the external connection mode of the neuron model by setting a gating device inside the neuron model, and can be interconnected with other neuron models only through a single input terminal and a single output terminal. Furthermore, various desired neuron topological structures can be flexibly obtained by simply controlling the gating devices in the interconnected neuron models.

[0050] On the basis of the above embodiments, the gating device may be a program-controlled device, and the neuron model further comprises: a gating control device, the gating control device being connected to the gating device;

[0051] The gating control device is used to determine the target neural network unit to be gated, and send a gating control instruction matching the target neural network unit to the gating device;

[0052] The gating device is specifically used to enable the target neural network unit in the neuron model according to the gating control instruction.

[0053] In this optional implementation, a program-controlled method can be used to control the gating device in the neuron model to further improve the intelligence and versatility of the entire neuron model. Specifically, in this embodiment, the gating control device in the neuron model can be used to perform gating control on the gating device. Optionally, the gating control device can be a control chip, and the gating controller can determine the target neural network unit that currently needs to be gated based on the control instructions input by the user (received by wire and / or wirelessly), or directly based on the input signal input into the neuron model, and then can determine the gating control instruction that matches the target neural network unit based on the preset correspondence, that is, the correspondence between the neural network unit that needs to be gated and the gating control instruction, and send the gating control instruction to the gating device.

[0054] Optionally, the neuron model may further include: a wireless transceiver module, which is connected to the gating control device, and the gating control device may determine the target neural network unit based on the control instructions received by the wireless transceiver module or the input signal input into the neuron model.

[0055] The advantage of this setting is that through simple program control operations, the neuron topology structure constructed by multiple neuron models can be dynamically adjusted, further expanding the use scenarios of the neuron model.

[0056] Embodiment 3

[0057] Figure 3 A schematic diagram of a neuron model provided in Embodiment 3 of the present invention is shown in FIG. Figure 3 As shown, in this embodiment, the gating control device is connected to the input end of the neuron model.

[0058] Accordingly, the gating control device is used to parse the routing information included in the input signal, and determine the target neural network unit to be gated according to the routing information; and send a gating control instruction matching the target neural network unit to the gating device. Furthermore, the gating device is specifically used to gate the target neural network unit in the neuron model according to the gating control instruction, and send the input signal to the target neural network unit (that is, the first neural network unit or the second neural network unit) accordingly.

[0059] In this embodiment, multiple neuron models (equivalent to transmission nodes) can be used to form a network to obtain a neuron network, and routing information is added to each input signal sent to the neuron network. The routing information specifies the order of the neuron models that each input signal needs to pass through, and the target neural network unit that needs to be selected for each neuron model passed through. Furthermore, when a neuron model receives an input signal, the target neural network unit to be selected can be determined simply and conveniently by parsing the routing information included in the input signal.

[0060] Through the above settings, based on the pre-built standard neural network, a variety of different types of neuron topologies can be flexibly implemented by simply setting different routing information, further expanding the use scenarios and versatility of the neuron model.

[0061] Embodiment 4

[0062] Figure 4a is a schematic diagram of a neuron topology structure in the fourth embodiment of the present invention, such as Figure 4a As shown, the neuron topology structure includes a first neuron model 410 , a data format conversion device 420 and a second neuron model 430 .

[0063] The first neuron model 410 and the second neuron model 430 are connected via a data format conversion device 420, and different types of neural network units are respectively selected in the first neuron model 410 and the second neuron model 430;

[0064] The data format conversion device 420 is used to convert a signal in a first data format adapted by the first neuron model 410 into a signal in a second data format adapted by the second neuron model 430;

[0065] The first neuron model 410 and the second neuron model 430 are neuron models as described in any embodiment of the present invention.

[0066] In this embodiment, a neuron topology structure applicable to the neuron model provided by each embodiment of the present invention is specifically shown, and in the neuron topology structure, an implementation method for mixed processing of signals in multiple data formats is provided. In order to give full play to the advantages of different types of neural network units in the neuron topology structure, different types of neural network units are selected in the first neuron model 410 and the second neuron model 430 respectively.

[0067] Optionally, neural network unit A is selected in the first neuron model 410, and the neural network unit A is used to process signals in a first data format; and neural network unit B is selected in the second neuron model 430, and the neural network unit B is used to process signals in a second data format.

[0068] In order to realize the mixed processing of the signal of the first data format and the signal of the second data format, in this embodiment, a data format conversion device 420 is introduced to realize the conversion of signals of different data formats. Among them, different types of data format conversion devices 420 can be selected according to the specific forms of the first data format and the second data format, for example, a pulse signal to analog signal converter or an analog signal to pulse signal converter, etc.; or different types of data format conversion units can be pre-configured in a unified data format conversion device 420, and according to the specific forms of the first data format and the second data format, a matching data format conversion unit can be selected, etc., and this embodiment does not limit this.

[0069] The technical solution of the embodiment of the present invention introduces a data format conversion device, which can ensure that different neural network units can process signals of adapted data formats under the premise that different types of neural network units are respectively selected in adjacent neuron models. A new form of heterogeneous fusion neural network hybrid operation is proposed, which effectively realizes the mixed processing of signals of multiple data formats.

[0070] In an optional implementation of this embodiment, the spiking neural network unit may be selected in the first neuron model, and the artificial neural network unit may be selected in the second neuron model;

[0071] The input end of the data format conversion device is connected to the output end of the first neuron model, and the output end of the data format conversion device is connected to the input end of the second neuron model;

[0072] The data format conversion device can be specifically used to: accumulate the pulse signals output by the first neuron model in time-sharing for a set time length to obtain an analog signal, and send the analog signal to the second neuron model.

[0073] In this optional implementation, the data format conversion device needs to convert the pulse signal output by the pulse neural network unit in the first neuron model into an analog signal adapted by the artificial neural network unit in the second neuron model. The specific implementation method is to accumulate the pulse signals output by the first neuron model in time-sharing for a set time length to obtain the analog signal.

[0074] Although the above setting can simply and conveniently obtain multiple pulse signals, it requires a certain amount of waiting time, which will increase the calculation delay of the entire neuron topology structure and have poor timeliness.

[0075] In another optional implementation of this embodiment, the pulse neural network unit can be selected in the first neuron model, and the artificial neural network unit can be selected in the second neuron model.

[0076] The input end of the data format conversion device is connected to each output end in the last hidden layer of the first neuron model, and the output end of the data format conversion device is connected to the input end of the second neuron model;

[0077] The data format conversion device is specifically used to accumulate the pulse signals output by the output terminals in the last hidden layer of the first neuron model at the same time to obtain an analog signal, and send the analog signal to the second neuron model.

[0078] In this optional embodiment, if Figure 4b As shown, the inventors analyzed the structure of the spiking neural network unit and found that although the output layer of the spiking neural network unit can only output a single pulse signal, the last hidden layer can output multiple pulse signals so that the output layer can finally obtain the single pulse signal. Therefore, in this optional embodiment, the output terminals ( Figure 4b The output 1, output 2, output 3, output 4 and output 5 of the example are all led out of the first neuron model, and then the data format conversion device can no longer be connected to the output end of the first neuron model, but can be connected to each output end in the last hidden layer of the pulse neural network unit. Accordingly, the data format conversion device can obtain multiple pulse signals at the same time, and then quickly accumulate the required analog signals to provide them to the artificial neural network unit of the second neuron model, so as to further reduce the calculation delay of the entire neuron topology.

[0079] On the basis of the above embodiments, the artificial neural network unit can be selected in the first neuron model, and the pulse neural network unit can be selected in the second neuron model;

[0080] The input end of the data format conversion device is connected to the output end of the first neuron model, and the output end of the data format conversion device is connected to the input end of the second neuron model;

[0081] The data format conversion device is specifically used to: sample the analog signal output by the first neuron model according to a set sampling rule, and send the sampled pulse signal to the second neuron model.

[0082] Through the above settings, the data format conversion device needs to convert the analog signal output by the artificial neural network unit in the first neuron model into a pulse signal adapted by the pulse neural network unit in the second neuron model, so as to further ensure the mixed processing of multiple data format signals.

[0083] Embodiment 5

[0084] Figure 5a The flowchart of an information processing method in Embodiment 5 of the present invention is shown. This embodiment can be applied to a neural network obtained by networking multiple neural models according to any embodiment of the present invention in a set connection mode. The method of this embodiment specifically includes:

[0085] S510: Receive a network input signal input into the neural network.

[0086] in, Figure 5b Schematic diagram of a neural network applicable to Embodiment 5 of the present invention. Different neural models in the neural network can transmit signals in pairs by direct connection or relay. Among them, a data format conversion device can be connected between at least two directly connected neural models in the neural network to locally form a neural topology structure as described in any embodiment of the present invention.

[0087] The network input signal includes routing selection information, and the routing selection information includes: a routing neuron model identifier and an identifier of a neural network unit selected in the routing neuron model.

[0088] Specifically, the routing selection information includes multiple routing neuron model identifiers constructed in sequence, that is, a routing neuron model identifier sequence, which indicates which neuron models (that is, routing neuron models) the network input signal needs to be sequentially input into for signal processing, and further indicates through the neural network unit identifier which neural network unit of each routing neuron model needs to be selected to process the network input signal.

[0089] Among them, after processing the network input signal, each routing neuron model can identify and output the current signal processing position in the routing neuron model identification sequence, so that the neuron model that receives the processed network input signal can quickly determine the next-hop neuron model of the network input signal and the selected neural network unit in the next-hop neuron model.

[0090] S520: Analyze the currently received target input signal through each neuron model to obtain routing information.

[0091] After the network input signal is input into the neural network, each neural model may receive the network input signal (or the network input signal that has been processed by other neural models). The above two signals can be collectively referred to as target input signals.

[0092] Correspondingly, when a certain neuron model receives the target input signal, it can parse the routing information from the target input signal, and then obtain the neuron model of the next hop and the neural network unit selected in the neuron model of the next hop. A neuron model identifier can be set for each neuron model, so that each neuron model can determine whether the neuron model of the next hop is itself after parsing the routing neuron model identifier included in the routing information.

[0093] S530. Through each of the neuron models, when it is determined that the target input signal needs to be routed according to the routing selection information, the neural network unit matching the routing selection information is selected to process the target input signal and output the processing result.

[0094] In a specific example, the neuron model 10 (neuron model identifier) ​​in the neural network currently receives the target input signal, and after parsing the routing selection information included in the target input signal through the selection control device, it is determined that the next-hop neuron model is the neuron model 10. At the same time, the neural network unit selected in the neuron model 10 is a pulse neural network unit.

[0095] Correspondingly, the gating control device in the neuron model 10 can send a gating instruction to the gating device to gating the pulse neural network unit, so that the pulse neural network unit in the neuron model 10 processes the currently received target input signal. In addition, after determining that the processing is completed, the gating control device can add the processed identifier of the neuron model 10 to the routing selection information in the processing result.

[0096] The technical solution of the embodiment of the present invention can establish a hybrid communication mechanism for signals of different data formats through a flexible and reconfigurable routing strategy, enriching the networking form of the neural network and meeting people's flexible neural networking needs. At the same time, relying on the reconfigurable routing strategy, the mutual conversion between different codecs (data format signals) can be realized, so that the signals to be transmitted can be freely transmitted between different levels, different neural models and different network models such as a single neural model, a neural model group, a neural model group, a large-scale neural network, etc.

[0097] Embodiment 6

[0098] Figure 6a FIG. 4 shows a retinal neuron topology structure in Embodiment 6 of the present invention. Figure 6a As shown, the topological structure includes: at least one retinal cell simulation layer, each of which includes a plurality of neuron models as described in any embodiment of the present invention.

[0099] The topological structure is used to process multiple input single color image signals and multiple grayscale description signals through at least one retinal cell simulation layer to obtain color reconstruction signals and optical flow reconstruction signals.

[0100] Optionally, the topological structure may be subdivided into two processing pathways, one processing pathway A for generating a color reconstruction signal, and the other processing pathway B for generating an optical flow reconstruction signal. Optionally, in the at least one retinal cell simulation layer, one or more neuron models in each layer belong to the processing pathway A, and the remaining one or more neuron models in each layer belong to the processing pathway B.

[0101] Optionally, in processing path A and processing path B, different neuron models in each retinal cell simulation layer may be used non-repeatedly to generate the final color reconstruction signal and optical flow reconstruction signal, or one or more neuron models may be shared in at least one retinal cell simulation layer. This embodiment does not impose any restrictions on this.

[0102] Optionally, all single color image signals may be input separately into processing channel A, and all grayscale description signals may be input separately into processing channel B; or, all single color image signals may be input into processing channel A while several of the single color image signals may be input into processing channel B for mixed calculation; or, all grayscale description signals may be input into processing channel B while several of the grayscale description signals may be input into processing channel A for mixed calculation, etc.

[0103] By way of example and not limitation, Figure 6aIn the topological structure shown in , along the input to output direction (i.e., the signal transmission direction), retinal cell simulation layer 1, retinal cell simulation layer 2, retinal cell simulation layer 3 and retinal cell simulation layer 4 are included. Each retinal cell simulation layer includes multiple neuron models. For example, retinal cell simulation layer 1 includes neuron model 1-neuron model 4. The input of this topological structure (the signal input into the retinal cell simulation layer 1) is a plurality of single color image signals and a plurality of grayscale description signals, and the output of this topological structure (the signal output by the retinal cell simulation layer 4) is a color reconstruction signal and an optical flow reconstruction signal.

[0104] In this embodiment, the single color image signal may specifically be three primary color image signals, namely, an R color image signal, a B color image signal, and a G color image signal.

[0105] Among them, considering that the human eye retina mainly includes two types of visual perception cells, cones and rods. Cones are sensitive to absolute light intensity information and color information, so they have high image restoration accuracy, but slow speed. On the contrary, rods cannot perceive color and absolute light intensity information, and they mainly perceive the change in light intensity information, so they have high speed and a large dynamic range.

[0106] At present, traditional methods only model single retinal cells or a small number of cell groups, and lack theoretical models for large-scale retinal modeling and simulation. The inventors found through research that the combination of artificial neural network units can achieve high-quality color image signal reconstruction based on frequency coding, while the combination of pulse neural network units can achieve event-driven high-speed optical flow signal reconstruction based on time coding. Since the technical solution of the embodiment of the present invention has constructed a universal neuron model that can select artificial neural networks or pulse neural networks, the neuron model can be used as the smallest unit to construct a retinal neuron topology that can simultaneously reconstruct color image signals and optical flow signals.

[0107] By constructing the above retinal neuron topology, a unified visual perception paradigm of heterogeneous fusion of pulse neural network units and artificial neural network units can be obtained to combine the advantages of the above two neural network units to obtain better performance and efficiency when processing complex systems. This hybrid solution is suitable for edge sensor applications, automotive applications, drone applications, robots, etc., as well as occasions that require high accuracy, low latency, and high energy efficiency at the same time.

[0108] In this embodiment, the retinal cell simulation layer is used to simulate the real cell layers in the retina, such as the outer plexiform layer, inner nuclear layer, inner plexiform layer, and ganglion cell layer. At the same time, multiple neuron models in different retinal cell simulation layers are used to simulate the signal processing process of neurons in real cell layers. Optionally, retinal cell simulation layer 1 can be used to simulate the outer plexiform layer, retinal cell simulation layer 2 can be used to simulate the inner nuclear layer, retinal cell simulation layer 3 can be used to simulate the inner plexiform layer, and retinal cell simulation layer 4 can be used to simulate the ganglion cell layer.

[0109] Based on the above embodiments, the number of retinal cell simulation layers included in the topological structure is multiple ( Figure 6a The four retinal cell simulation layers shown are examples only).

[0110] The neuron model in the retinal cell simulation layer is connected to the neuron models in other retinal cell simulation layers by at least one of forward direct connection, reverse direct connection, forward cross-layer connection and reverse cross-layer connection, wherein the forward direction is the transmission direction of the signal from input to output.

[0111] like Figure 6a As shown, neuron model 1 in retinal cell simulation layer 1 is directly connected to neuron model 5 in retinal cell simulation layer 2, neuron model 13 in retinal cell simulation layer 4 is directly connected to neuron model 10 in retinal cell simulation layer 3, neuron model 6 in retinal cell simulation layer 2 is directly connected to neuron model 13 in retinal cell simulation layer 4, and neuron model 13 in retinal cell simulation layer 4 is directly connected to neuron model 5 in retinal cell simulation layer 2. Based on the hierarchical processing, the cross-layer connection design method is adopted to realize efficient multi-scale receptive field (Receptive Field) feature extraction, especially the feature extraction of mixed receptive fields.

[0112] On the basis of the above embodiments, the neural network units adapted to each neuron model in the retinal neuron topology structure are trained by unsupervised learning.

[0113] Based on the above embodiments, the neural network units selected by each neuron model in the retinal neuron topology structure are trained using at least one of a back propagation algorithm, a winner-takes-all algorithm, and an spike-timing-dependent plasticity algorithm.

[0114] Based on the above embodiments, the grayscale description signal may include: a grayscale image signal, or an optical flow signal.

[0115] The optical flow signal refers to the real-time light intensity change. The real-time light intensity change specifically refers to the light intensity change of a pixel in a color image at a certain moment, or can also be expressed as a relative gray value (brightness value) change, which represents the change between the current brightness value of the pixel and the historical brightness value at a previous moment.

[0116] Based on the above embodiments, at least one neuron model included in each of the retinal cell simulation layers together constitutes a first retinal simulation pathway; at least one remaining neuron model included in each of the retinal cell simulation layers together constitutes a second retinal simulation pathway;

[0117] The first retina simulation pathway is used to fuse multiple input single color image signals and at least one grayscale description signal to obtain a color reconstruction signal;

[0118] The second retina simulation pathway is used to process the input multiple grayscale description signals to obtain an optical flow reconstruction signal.

[0119] Optionally, each neuron model in the first retinal simulation pathway selects a first neural network unit; each neuron model in the second retinal simulation pathway selects a second neural network unit.

[0120] Furthermore, the first neural network unit may be an artificial neural network (ANN) unit, and the second neural network unit may be a spiking neural network (SNN) unit.

[0121] Optionally, each ANN unit in the first retinal simulation pathway can be used to simulate the parasol cell pathway in the human eye retina to achieve high-quality color image signal reconstruction based on frequency coding. In the first retinal simulation pathway, the cone cells, cone horizontal cells, bipolar cells, amacrine cells and parasol ganglion cells in the retina are mainly simulated; each SNN unit in the second retinal simulation pathway can be used to simulate the dwarf cell pathway in the human eye retina to achieve event-driven high-speed optical flow signals based on time coding. In this pathway, the rod cells, rod horizontal cells, bipolar cells, amacrine cells and dwarf ganglion cells in the retina are mainly simulated.

[0122] On the basis of the above embodiments, the signal input end of the first retina simulation path is used to input the voltage signal of the light intensity of the light signal collected by the color image sensing circuit in the dual-modal vision sensor, and the current signal of the light intensity variation of the light signal collected by the light intensity variation sensing circuit in the dual-modal vision sensor;

[0123] The signal input end of the second retina simulation path is used to input a current signal of the light intensity variation of the light signal collected by the light intensity variation sensing circuit in the dual-modal vision sensor.

[0124] In an optional implementation of the embodiment of the present invention, the dual-modal visual sensor may specifically include: a first sensing circuit (also referred to as a light intensity variation sensing circuit) and a second sensing circuit (also referred to as a color image sensing circuit);

[0125] A first sensing circuit is used to extract a light signal of a first set wavelength band from the target light signal, and output a current signal representing a light intensity change of the light signal of the first set wavelength band;

[0126] The second sensing circuit is used to extract the optical signal of the second set wavelength band from the target optical signal, and output a voltage signal representing the light intensity of the optical signal of the second set wavelength band.

[0127] Optionally, the first sensing circuit includes a first excitatory photosensitive unit and a first inhibitory photosensitive unit, and the first excitatory photosensitive unit and the first inhibitory photosensitive unit are both used to extract a light signal of a first set wavelength band in the target light signal, and convert the light signal of the first set wavelength band into a current signal;

[0128] The first sensing circuit is further configured to output a current signal representing a change in light intensity of the light signal of the first set wavelength band according to a difference between current signals converted by the first excitatory photosensitive unit and the first inhibitory photosensitive unit.

[0129] Optionally, the second sensing circuit includes at least one second photosensitive unit, and the second photosensitive unit is used to extract a light signal of a second set wavelength band from the target light signal, and convert the light signal of the second set wavelength band into a current signal;

[0130] The second sensing circuit is further configured to output a voltage signal representing the light intensity of the light signal in the second set wavelength band according to the current signal converted by the second photosensitive unit.

[0131] Through the above-mentioned setting, the above-mentioned voltage-current dual-modal bionic vision sensor (dual-modal vision sensor) can be used to simultaneously obtain high-speed spatial gradient signals (rods, ganglion cells and horizontal cells) and low-speed color signals (cones) like the retina of the human eye, and then through the retinal neuron topology structure as described in the embodiment of the present invention, it is possible to simulate the color reconstruction signal and the optical flow reconstruction signal obtained by reconstructing the voltage signal of the light intensity of the light signal and the current signal of the light intensity change of the light signal based on the human eye.

[0132] On the basis of the above embodiments, the grayscale description signal input into the first retinal simulation pathway may be a subset of the grayscale description signal input into the second retinal simulation pathway.

[0133] On the basis of the above embodiments, the topological structure may further include: an input signal simulation layer; the input signal simulation layer includes at least one neuron model as described in any embodiment of the present invention;

[0134] The output end of the input signal simulation layer is connected to the input ends of the first retina simulation pathway and the second retina simulation pathway respectively;

[0135] The input signal simulation layer is used to process the input light simulation signal to obtain multiple single color image signals and multiple grayscale description signals; send the multiple single color image signals and at least one grayscale description signal to the input end of the first retinal simulation pathway, and send the multiple grayscale description signals to the input end of the second retinal simulation pathway.

[0136] In this embodiment, considering that the rod-cone layer in the human eye retina can generate multiple single color image signals and multiple grayscale description signals according to the light signal, therefore, in addition to using the dual-modal vision sensor to obtain the input signal required by the retinal neuron topology structure, the rod-cone layer can also be directly simulated through the input signal simulation layer to directly simulate the input signal required by the retinal neuron topology structure.

[0137] It needs to be emphasized again that no matter what method is used to obtain the multiple single color image signals and the multiple grayscale description signals, the signal input to the first retinal simulation pathway (used to simulate the parasol cell pathway) needs to include at least one grayscale description signal, and through the mixed calculation of the grayscale description signal in the first retinal simulation pathway and the multiple single color image signals, a color reconstructed signal can be finally obtained.

[0138] Among them, Figure 6b A schematic diagram showing a specific retinal neuron topological structure in Embodiment 6 of the present invention is shown.

[0139] like Figure 6b As shown in the figure, the hierarchical network framework used in the retinal neuron topology corresponds to the simplified multi-layer retinal structure: rod and cone layer, outer retinal layer, inner retinal layer, inner retinal layer and ganglion cell layer. For the input of visual signals, the simulation network includes both the bottom-up feedforward process and the top-down feedback process.

[0140] For the feedforward process, heterogeneously fused ANN units and SNN units can be used to simulate various neural dynamic phenomena in the retina. ANN units can be mainly used to simulate the parasol cell pathway to achieve high-quality color image signal reconstruction based on frequency coding. In this pathway, cones, cone horizontal cells, bipolar cells, amacrine cells and parasol ganglion cells in the retina are mainly simulated; SNN units can be used to simulate the dwarf cell pathway to achieve event-driven high-speed optical flow signals based on time coding. In this pathway, rods, rod horizontal cells, bipolar cells, amacrine cells and dwarf ganglion cells in the retina are mainly simulated. At the same time, in order to simulate more than 40 different receptive field types of amacrine cells, on the basis of hierarchical processing, a cross-layer connection (forward and reverse) design method is adopted to achieve efficient feature extraction of multi-scale mixed receptive fields.

[0141] For the feedback process, the gap junction network of the inner nuclear layer horizontal cells can be simulated using the STDP (Spike Timing Dependent Plasticity) learning rule based on local learning, and the attention mechanism of the human eye can be simulated using an unsupervised learning algorithm based on the WTA (Winner Take All) rule. The horizontal cells aggregate the intensity of the received photoreceptor cell signals and measure the average brightness of the light on the retina in a certain area. The feedback inhibition signal adjusts the output signal of the photoreceptor cells to an appropriate level, so that the signal received by the bipolar cells is neither too small to be submerged in the noise of the neural pathway, nor too large to oversaturate the neural pathway, greatly improving the adaptive ability and dynamic range of the model.

[0142] At the same time, if Figure 6b As shown, after the input is processed by the six neuron models of the rod and cone layers, the first three (in order from left to right) neuron models output multiple single color image signals to the parasol cell pathway in the exopolysaccharide layer, and the last three neuron models output multiple grayscale description signals to the pygmy cell pathway in the exopolysaccharide layer, among which the grayscale description signal output by the fourth neuron model is also provided to the parasol cell pathway at the same time.

[0143] Note that the above are only preferred embodiments of the present invention and the technical principles used. Those skilled in the art will understand that the present invention is not limited to the specific embodiments described herein, and that various obvious changes, readjustments and substitutions can be made by those skilled in the art without departing from the scope of protection of the present invention. Therefore, although the present invention has been described in more detail through the above embodiments, the present invention is not limited to the above embodiments, and may include more other equivalent embodiments without departing from the concept of the present invention, and the scope of the present invention is determined by the scope of the appended claims.

Claims

1. An information processing method, applied to a neural network, wherein the neural network is obtained by networking a plurality of neural models in a set connection mode, characterized in that: include: Receiving a network input signal input into the neural network, wherein the network input signal includes routing information, and the routing information includes: a routing neuron model identifier and a neural network unit identifier selected in the routing neuron model; By means of each of the neuron models, the routing information is parsed in the currently received target input signal; when it is determined that the target input signal needs to be routed according to the routing information, the neural network unit matching the routing information is selected to process the target input signal, and the processing result is output; The neuron model comprises: at least two independent neural network units, each of which is used to process signals in different data formats; wherein, among the neural network units of the neuron model, one neural network unit is selected; Furthermore, the network input signal includes: a plurality of single color image signals and a plurality of grayscale description signals; and the processing result includes: a color reconstruction signal and an optical flow reconstruction signal.

2. A retinal neuron topology, characterized in that: The topological structure includes: a plurality of retinal cell simulation layers, each of which includes a plurality of neuron models; the neuron model includes: at least two independent neural network units, each of which is used to process signals of different data formats; wherein, among the neural network units of the neuron model, one neural network unit is selected; The topological structure is used to process multiple input single color image signals and multiple grayscale description signals through at least one retinal cell simulation layer to obtain color reconstruction signals and optical flow reconstruction signals; Among them, the connection methods between the neuron model in the retinal cell simulation layer and the neuron models in other retinal cell simulation layers include: forward direct connection, reverse direct connection, forward cross-layer connection and reverse cross-layer connection, among which the forward direction is the transmission direction of the signal from input to output.

3. The retinal neuron topological structure according to claim 2, characterized in that: At least one neuron model included in each of the retinal cell simulation layers together constitutes a first retinal simulation pathway; at least one remaining neuron model included in each of the retinal cell simulation layers together constitutes a second retinal simulation pathway; The first retina simulation pathway is used to fuse multiple input single color image signals and at least one grayscale description signal to obtain a color reconstruction signal; The second retina simulation pathway is used to process the input multiple grayscale description signals to obtain an optical flow reconstruction signal.

4. The retinal neuron topological structure according to claim 3, characterized in that: Each neuron model in the first retinal simulation pathway selects the first neural network unit; each neuron model in the second retinal simulation pathway selects the second neural network unit.

5. The retinal neuron topological structure according to claim 3, characterized in that: The grayscale description signal includes: a grayscale image signal, or an optical flow signal; and / or, The grayscale description signal input into the first retina simulation pathway is a subset of the grayscale description signal input into the second retina simulation pathway.

6. The retinal neuron topological structure according to claim 2, characterized in that: The neural network units adapted to the neuron models in the retinal neuron topology are obtained by training in an unsupervised learning manner; and / or The neural network units selected by each of the neuron models in the retinal neuron topology structure are trained using at least one of a back-propagation algorithm, a winner-takes-all algorithm, and an spike-timing-dependent plasticity algorithm.

7. The retinal neuron topological structure according to any one of claims 3 to 5, characterized in that: The signal input end of the first retinal simulation path is used to input a voltage signal of the light intensity of the light signal collected by the color image sensing circuit in the dual-modal vision sensor, and a current signal of the light intensity variation of the light signal collected by the light intensity variation sensing circuit in the dual-modal vision sensor; The signal input end of the second retina simulation path is used to input a current signal of the light intensity variation of the light signal collected by the light intensity variation sensing circuit in the dual-modal vision sensor.

8. The retinal neuron topological structure according to any one of claims 3 to 5, characterized in that: The topological structure further comprises: an input signal simulation layer; the input signal simulation layer comprises at least one neuron model; The output end of the input signal simulation layer is connected to the input ends of the first retina simulation pathway and the second retina simulation pathway respectively; The input signal simulation layer is used to process the input light simulation signal to obtain multiple single color image signals and multiple grayscale description signals; send the multiple single color image signals and at least one grayscale description signal to the input end of the first retinal simulation pathway, and send the multiple grayscale description signals to the input end of the second retinal simulation pathway.

9. The retinal neuron topological structure according to claim 2, characterized in that: Also includes: A gating device, wherein the gating device is connected to each of the neural network units respectively; The gating device is used to select a neural network unit in the neuron model.

10. The retinal neuron topological structure according to claim 9, characterized in that: The gating device is a program-controlled device, and the neuron model further comprises: a gating control device, the gating control device being connected to the gating device; The gating control device is used to determine the target neural network unit to be gated, and send a gating control instruction matching the target neural network unit to the gating device; The gating device is specifically used to enable the target neural network unit in the neuron model according to the gating control instruction.

11. The retinal neuron topological structure according to claim 10, characterized in that: The gating control device is connected to the input end of the neuron model; The gating control device is used to parse the routing selection information included in the input signal and determine the target neural network unit to be gated based on the routing selection information.

12. The retinal neuron topological structure according to claim 11, characterized in that: The neural network unit in the neuron model includes: a first neural network unit and a second neural network unit; The first neural network unit is used to process and output the analog signal; The second neural network unit is used to process and output the pulse signal.

13. The retinal neuron topological structure according to claim 4, characterized in that: The first neural network unit is an artificial neural network unit, and the second neural network unit is a pulse neural network unit.

14. The retinal neuron topological structure according to claim 2, characterized in that: The retinal neuron topology structure includes: a first neuron model, a second neuron model and a data format conversion device; The first neuron model and the second neuron model are connected via the data format conversion device, and different types of neural network units are respectively selected in the first neuron model and the second neuron model; The data format conversion device is used to convert a signal in a first data format adapted by the first neuron model into a signal in a second data format adapted by the second neuron model.

15. The retinal neuron topological structure according to claim 14, characterized in that: The spiking neural network unit is selected in the first neuron model, and the artificial neural network unit is selected in the second neuron model; The input end of the data format conversion device is connected to the output end of the first neuron model, and the output end of the data format conversion device is connected to the input end of the second neuron model; The data format conversion device is specifically used to: accumulate the pulse signals output by the first neuron model in time-sharing for a set time length to obtain an analog signal, and send the analog signal to the second neuron model.

16. The retinal neuron topological structure according to claim 14, characterized in that: The spiking neural network unit is selected in the first neuron model, and the artificial neural network unit is selected in the second neuron model; The input end of the data format conversion device is connected to each output end in the last hidden layer of the first neuron model, and the output end of the data format conversion device is connected to the input end of the second neuron model; The data format conversion device is specifically used to accumulate the pulse signals output by the output terminals in the last hidden layer of the first neuron model at the same time to obtain an analog signal, and send the analog signal to the second neuron model.

17. The retinal neuron topological structure according to claim 14, characterized in that: An artificial neural network unit is selected in the first neuron model, and a pulse neural network unit is selected in the second neuron model; The input end of the data format conversion device is connected to the output end of the first neuron model, and the output end of the data format conversion device is connected to the input end of the second neuron model; The data format conversion device is specifically used to: sample the analog signal output by the first neuron model according to a set sampling rule, and send the sampled pulse signal to the second neuron model.

Images