Perception and motion control method based on modularized in-vitro biological neural network

Abstract

The invention relates to a perception and motion control method based on a modularized in-vitro biological neural network, and relates to the technical field of biological neural network calculation, and the method comprises the following steps: 1) coding the image information of an object to be grabbed; 2) encoding the motion information of the multi-joint dexterous hand; 3) constructing a modularized in-vitro biological neural network, training the modularized in-vitro biological neural network by using a repeated stimulation mode, and establishing a neural activity decoding model; and 4) inputting the encoded image and motion information into the trained in-vitro biological neural network, analyzing the activity mode of the biological neural network by using a neural activity decoding model, and mapping the output result of the decoding model into dexterous hand motion control. According to the method, the modularized biological neural network can sense complex and natural environment information, and complex motion control of the dexterous hand is achieved by outputting a control instruction.

Description

Technical Field

[0001] This invention relates to the field of biological neural network computing technology, specifically a perception and motion control method based on a modular in vitro biological neural network. Background Technology

[0002] Artificial neural networks have developed rapidly in recent years, achieving remarkable results in fields such as computer vision, natural language processing, and robot control. However, they still face challenges in terms of resource consumption and dynamic environment adaptation. In contrast, biological neural networks transmit electrical signals through synaptic connections, exhibiting significant advantages such as high energy efficiency, parallel computing, and real-time learning.

[0003] Research has found that directly cultivating biological neural networks on high-density microelectrode arrays (MEAs) enables high-resolution measurement of multi-channel synchronous electrical stimulation input and network discharge signals, providing technical support for studying the spontaneous discharge, information processing, and learning behavior of biological neural networks. Based on this, researchers have proposed directly utilizing the information processing and learning capabilities of biological neural networks to complete computational tasks such as perception, cognition, and decision-making.

[0004] Vision is a crucial pathway for living systems to perceive their external environment, and image recognition is a fundamental basis for environmental perception, target detection, and advanced decision-making. Similarly, motion is a vital means for living systems to interact with the physical world. Since in vitro biological neural networks lack the ability to interact with the external environment on their own, they require training through neural signal encoding and decoding methods to generate specific functions. Therefore, developing encoding methods suitable for image and motion information is of great significance for expanding the application potential of biological neural networks. Furthermore, based on rich information encoding, utilizing biological neural networks to achieve complex motion control can effectively leverage their computational potential.

[0005] In existing technologies, linear encoding, binary encoding, or spatial location encoding methods are typically used to encode one-dimensional sensor signals into pulse stimulation sequences for biological neural networks. However, the environmental information that in vitro biological neural networks can perceive is limited. Patent CN115810138A proposes arranging stimulation electrodes in the spatial positions of the letters L, O, I, and X, based on in vitro cultured neurons recognizing four different letters. However, the images that can be generated using electrode spatial arrangement are relatively simple and have poor scalability; in vitro biological neural networks still cannot perceive complex and near-natural environmental image information. Due to the limited information that biological neural networks can perceive, existing work has only been able to achieve simple control tasks such as obstacle avoidance for small vehicles.

[0006] Delayed phase coding is a classic method for converting images into pulse sequences. However, traditional delayed phase coding produces pulse sequences with excessively high firing frequencies, which, if directly used as stimulation sequences for biological neural networks, can cause neural activity to enter a refractory period. Furthermore, there are currently no methods for encoding multi-degree-of-freedom cooperative continuous motion stimulation for in vitro biological neural networks, resulting in a limited range of control modes and degrees of freedom that can be output by these networks. Summary of the Invention

[0007] The purpose of this invention is to provide a perception and motion control method based on a modular in vitro biological neural network, which enables the in vitro biological neural network to perceive complex image and motion information and make control decisions based on the perceived information, thereby expanding the computing power of the in vitro biological neural network and overcoming the shortcomings of the above-mentioned in vitro biological neural network electrical stimulation encoding and motion control methods.

[0008] The technical solution adopted by the present invention to achieve the above objectives is as follows:

[0009] A perception and motion control method based on a modular in vitro biological neural network includes the following steps:

[0010] 1) Encode the image information of the object to be captured;

[0011] 2) Encode the motion information of multi-joint dexterous hands;

[0012] 3) Construct a modular in vitro biological neural network capable of generating neural activity, train it using repeated stimulation, and establish a neural activity decoding model.

[0013] 4) Input the encoded image and motion information into the trained in vitro biological neural network, use the neural activity decoding model to analyze the activity pattern of the biological neural network, and map the output of the decoding model into dexterous hand motion control.

[0014] Step 1) includes the following steps:

[0015] 1.1) Collect image information of the object to be captured, extract features from it using a convolutional neural network, and normalize the extracted features;

[0016] 1.2) The pixel units that make up the feature map are generated with specific ignition times through the defined pulse ignition function. Each feature map represents a receptive field. The ignition times of the pixel units in the same receptive field are arranged into a time series. By adding a time offset, a time difference is generated in the pulse ignition time.

[0017] 1.3) Use the SMO function to align and compress time series with time differences to generate a series of pulse sequences.

[0018] The pulse ignition function Specifically:

[0019]

[0020] in, Ignition time, The value of a pixel unit in the feature map after normalization. The maximum duration of the pulse stimulation. To adjust the parameters;

[0021] The time difference generated by the pulse ignition time is:

[0022]

[0023]

[0024] in, The position coefficient of the pixel. To sense the size of the field, This is the time offset. This is an ignition time sequence with a time difference.

[0025] Step 1.3) specifically refers to:

[0026] Pixel units within the same receptive field are configured with an SMO function having the same initial phase. The SMO function adjusts pulses at different positions in the time series to the nearest peak of the oscillation function, obtaining the neuronal stimulation nodes corresponding to each receptive field and compressing them into a series of pulse sequences. Specifically, the SMO function is:

[0027]

[0028] in, It is the amplitude of the subthreshold membrane oscillation. It is the number of oscillation periods. This is the initial phase value, which is usually set to 0.

[0029] Step 2) includes the following steps:

[0030] 2.1) Collect the initial angle information of each joint of the dexterous hand;

[0031] 2.2) Based on the object to be grasped, confirm the grasping mode planning, obtain the multi-joint angle time series of the grasping task, and discretize it;

[0032] 2.3) The spatiotemporal stimulus sequence of motion information is jointly encoded using binary encoding and multi-site spatial encoding.

[0033] Step 2.2) specifically refers to:

[0034] Connect the five initial joint angles of the dexterous hand to the target joint angles respectively, and find several intermediate points evenly to generate five discrete time series of dexterous hand joint angles.

[0035] Step 2.3) specifically refers to:

[0036] In a multi-joint angle time series, the angle of a joint at a certain moment is a value within the range of 0° to 180°, and the angles of multiple joints form a multi-dimensional vector. A time-coding method is used to represent the joint angles of each dimension using six-bit binary numbers. In the resulting sequence of 0s and 1s, 0 indicates that no stimulus is applied at the current moment, and 1 indicates that a stimulus is applied. A spatial coding method is used to map the different joint angle information to stimulation input electrodes at different spatial locations to obtain the joint information at a certain moment. Five pulse sequences were generated after spatiotemporal co-coding, and five stimulation electrodes were selected accordingly.

[0037] The modular in vitro biological neural network is divided into an image sensing area, a motion sensing area, and a motion output area, and is cultured on a high-density microelectrode array with tens of thousands of acquisition channels and dozens of stimulation channels.

[0038] The neural activity decoding model performs the following steps:

[0039] (1) Obtain the discharge signal of an in vitro biological neural network after a period of time following the application of stimulation;

[0040] (2) Extract features from the discharge signal, divide the signal segment of each recording electrode into several equal time windows, calculate the number of neuron discharges in each time window and combine them into a feature sequence, and splice all the feature sequences into a feature vector;

[0041] (3) Use principal component analysis to reduce the dimensionality of the eigenvectors and decode them using classification and regression algorithms.

[0042] Step 4) specifically involves:

[0043] The image category labels output by the neural activity decoding model are mapped to the hand gestures of the dexterous hand, and the hand gestures are used to control the grasping mode of the dexterous hand.

[0044] The joint angles output by the neural activity decoding model are mapped to the joint motion information of the five fingers of a dexterous hand, and the joint motion information is used to perform fine control on the finger joint angles of the dexterous hand.

[0045] The present invention has the following beneficial effects and advantages:

[0046] 1. This invention proposes a spatiotemporal stimulation sequence encoding method for image and motion information, which transforms two-dimensional image information and multi-joint angle information into spatiotemporal electrical stimulation sequences that can induce specific neural activities in an in vitro biological neural network.

[0047] 2. This invention enables biological neural networks to learn and memorize different spatiotemporal electrical stimulation sequences through repeated stimulation training. Combined with neural activity feature extraction methods, it verifies that the training and encoding stimulation methods can induce stable, separable, and predictable neural activity patterns.

[0048] 3. This invention utilizes logistic regression classification models and multiple linear regression models to decode the in vitro biological neural network activities under different image and multi-joint motion information stimuli into discrete and continuous motion control information, respectively, and maps them to the motion control of a dexterous hand. Based on the in vitro biological neural network, low-power integrated perception-control computing is realized. Attached Figure Description

[0049] Figure 1 Method framework diagram in this embodiment of the invention;

[0050] Figure 2 Image information and motion information coding framework diagram;

[0051] Among them, (a) feature extraction of the image of the object to be grasped; (b) delayed phase coding; (c) phase alignment; (d) compression operation; (e) spatiotemporal stimulus sequence of image information; (f) joint angle planning of the object to be grasped; (g) binary time series coding of joint angles; (h) multi-site spatial coding; and (i) spatiotemporal stimulus sequence of motion information.

[0052] Figure 3 Schematic diagram of modular in vitro biological neural networks and neural activity discharge characteristics;

[0053] Among them, (a) modular in vitro biological neural network; (b) discharge characteristics of biological neural network under different spatiotemporal stimulation sequences;

[0054] Figure 4 A schematic diagram showing the classification accuracy of a modular biological neural network for images, the regression prediction coefficient for joint angles, and the number of times the neural network fires at different training cycles under intraday / cross-day training.

[0055] Among them, (a) the classification accuracy of the modular biological neural network for images and the regression prediction coefficient for joint angles under training within one day / across days; (b) the number of induced discharges of the neural network under repeated stimulation.

[0056] Figure 5 Comparison of neural network connection weights and functional connection topology before and after training;

[0057] Among them, (a) the correlation matrix represents the connection weights between representative electrodes before and after training; and (b) the functional connection topology of the neural network before and after training. Detailed Implementation

[0058] To enable those skilled in the art to better understand the present invention, the solutions in the embodiments of the present invention will be fully described below with reference to the accompanying drawings. The described embodiments are only some, not all, of the embodiments of the present invention. Other embodiments obtained by those skilled in the art based on the embodiments of the present invention without creative effort are all within the scope of protection of the present invention.

[0059] The following describes in detail, in four parts, a method for integrated perception and motion control of a dexterous hand based on a modular in vitro biological neural network, according to an embodiment of the present invention. Figure 1 As shown, the four parts include: image information encoding, motion information encoding, modular biological neural network training, and dexterous hand perception-motion integrated control.

[0060] Image information encoding includes: image information of the object to be captured 101, pre-trained convolutional neural network 102, image feature extraction and normalization 103, delayed phase encoding 104, time series alignment and compression 105, spatiotemporal stimulus sequence of image information 106.

[0061] The image information of the object to be grasped is collected by a vision sensor, including images of common everyday items such as apples, water cups, and cars.

[0062] Image feature extraction is performed using convolutional neural networks (CNNs) to generate feature maps with fewer pixels.

[0063] The designed CNN model contains three convolutional layers and three pooling layers. The number of convolutional kernels in each convolutional layer is 12, 14 and 8 respectively. The kernel size is 5×5, the stride is 1, zero padding is used, and the activation function is ReLU. The pooling layers use 2×2 max pooling to reduce the feature map size.

[0064] When pre-training the convolutional neural network, three categories (apple, car, cup) from the ETH-80 dataset were selected, with 100 different images for each category, totaling 300 images as the training set. The learning rate was set to 0.0005, the Adam optimizer was chosen, and cross-entropy was used as the loss function. The trained CNN was used as a fixed feature extractor to process the images, generating eight 5×5 feature maps. After normalization, the feature maps were input into the delayed phase coding module (…). Figure 2 (a)).

[0065] The pixel units that make up the feature map generate specific ignition times through a defined pulse ignition function:

[0066]

[0067] in, Ignition time, The value of a pixel unit in the feature map after normalization. The maximum duration of the pulse stimulation. To adjust the parameters, each feature map represents a receptive field (RF), and the firing times of pixel units within the same receptive field are arranged into a time series. Figure 2 (b) Then, by adding a time offset, a time difference is created in the pulse ignition time, which is calculated as follows:

[0068]

[0069]

[0070] in, It is the position coefficient of the pixel. The size of the receptive field. In traditional methods, each pixel unit in the receptive field has a different sub-threshold membrane oscillation (SMO) function. However, to reduce the number of pulses, this embodiment sets the pixel units in the same receptive field to have the same SMO function with the same initial phase.

[0071]

[0072] in, It is the amplitude of the subthreshold membrane oscillation. It is the number of oscillation periods. This is the initial phase value, usually set to 0. The SMO function adjusts pulses at different positions to the nearest peak in the oscillation function. Figure 2 (c)). The phase-aligned time series are finally integrated into the neuronal stimulation nodes corresponding to each receptive field and compressed into a series of pulse sequences. Figure 2 (d)). Finally, the original image, after improved delayed phase coding, yielded eight pulse sequences, corresponding to the eight selected stimulation electrodes ( Figure 2 (e) The stimulation duration of each pulse sequence is 2 s, with a minimum interval of 200 ms between adjacent pulses to ensure that the in vitro biological neural network can recover to a resting state before the next stimulation. The selection rules for the stimulation electrodes are described in detail in the modular in vitro biological neural network training section.

[0073] The motion information encoding includes: dexterous hand multi-joint angle information 107, multi-joint angle planning for grasping 108, multi-joint angle sequence discretization 109, binary time series encoding 110, multi-site spatial encoding 111, and motion information spatiotemporal stimulus sequence 112.

[0074] The dexterous hand multi-joint angle information refers to the initial angle information of each joint of the dexterous hand, which is collected by the motor encoder.

[0075] Based on the object to be grasped, the grasping mode is determined and the grasping task is planned using a multi-joint angle time series. This multi-joint angle time series is then discretized. In this embodiment, the initial angles of the five finger joints (thumb, index finger, middle finger, ring finger, and little finger) of the dexterous hand are [90°, 90°, 90°, 90°, 90°]. When grasping an apple, a water cup, and a toy car, the target joint angles of the dexterous hand are [15°, 57°, 36°, 51°, 39°], [33°, 63°, 33°, 171°, 129°], and [3°, 15°, 168°, 165°, 159°], respectively. The five initial joint angles of the dexterous hand are connected to the target joint angles, and eight intermediate points are evenly selected to generate five discrete time series of dexterous hand joint angles. Figure 2 (f)).

[0076] The spatiotemporal stimulus sequence of motion information is jointly encoded using binary encoding and multi-site spatial encoding. The angle of a joint at a certain moment is a value within the range of 0° to 180°, and multiple joint angles form a multidimensional vector. Then, a time-coding method is used to represent the joint angles of each dimension using six-bit binary numbers. In the resulting sequence of 0s and 1s, 0 indicates that no stimulus is applied at the current moment, and 1 indicates that a stimulus is applied. In this way, the multi-joint angle information of the dexterous hand is converted into multiple pulse stimulation sequences. Figure 2 (g) In this case, for every 1 change in the value of the binary code, the joint angle changes by 3°.

[0077] A spatial coding method is used to map different joint angle information to stimulation input electrodes at different spatial locations. Figure 2 (h)). Finally, the joint information at that moment. After spatiotemporal joint encoding, five pulse sequences were obtained, corresponding to five selected stimulation electrodes. Figure 2 (i) Each pulse sequence has a duration of 1200 ms, with a minimum interval of 200 ms between adjacent pulses to ensure that the in vitro biological neural network can recover to a resting state before the next stimulation. The selection rules for stimulation electrodes are described in detail in the modular in vitro biological neural network training section.

[0078] Modular in vitro biological neural network training includes: in vitro culture of modular biological neural networks 113, biological neural network perception and motor area division 114, image and motion perception area electrical stimulation 115, intraday and cross-day repetitive stimulation training 116, motor area neural activity feature extraction 117, establishment of classification and regression model 118, neural activity decoding model 119.

[0079] The in vitro cultured modular biological neural network 113 was cultured on a high-density microelectrode array with 26,400 acquisition channels and 32 stimulation channels; a modular in vitro biological neural network with an image sensing region, a motion sensing region, and a motion output region was constructed using polydimethylsiloxane (PDMS). Figure 3 (a)).

[0080] This embodiment uses the MaxOne high-throughput microelectrode array system to record and stimulate in vitro biological neural networks. This system can simultaneously record the firing activity of 1024 channels of in vitro biological neural networks in a single operation. A bandpass filter (300~3000Hz) is used to filter the raw voltage signal; a neuron firing is identified when the filtered voltage exceeds a threshold of 5.5 times the standard deviation of the background noise.

[0081] Based on the peak voltage of spontaneous discharge detected by each electrode, 1024 electrodes with higher discharge peak values ​​in the motion output region were selected as recording electrodes for the experiment. The 8 electrodes with the highest discharge peak values ​​in the image perception region and the 5 electrodes with the highest discharge peak values ​​in the motion perception region were used as stimulation electrodes to ensure optimal stimulation efficiency. A bidirectional voltage pulse with an amplitude of 500mV and a phase width of 500µs (positive then negative) was used as the basic unit to construct the pulse stimulation sequence and as the stimulation input signal for the in vitro biological neural network.

[0082] This embodiment employs a repetitive stimulation method to train the biological neural network. For spatiotemporal stimulation of image information, three types of images were encoded in the experiment, and the corresponding pulse sequence for each type of image was repeatedly stimulated to the biological neural network 50 times. According to the improved delayed phase encoding method, each image stimulus lasted for 2 seconds, with a 15-second interval between stimuli to ensure that the neural network recovered to a resting state. To enable the decoding model to distinguish between spontaneous discharge and image-evoked discharge, after completing the image stimulation, 50 2-second spontaneous discharge activity segments were recorded in the absence of neural network input.

[0083] For the spatiotemporal stimulation of motion information, five pulse stimulation sequences generated from three randomly selected points in the grasping motion process were cyclically stimulated 30 times. According to the encoding method, each stimulus lasted 1200ms, and a 5s interval was set between stimuli to ensure that the neural network returned to a resting state.

[0084] In the 2-second evoked discharge signal segment recorded during electrical stimulation, the discharge signals detected within 10 ms after stimulation were discarded to avoid the influence of stimulation artifacts. Then, feature extraction was performed. The signal segment from each recording electrode was divided into 10 equal-length windows (200 ms each), and the number of neuronal firings within each time window was calculated and combined into a feature sequence. The feature sequences extracted from all recording electrodes were concatenated into a feature vector. To avoid excessively high dimensionality of the feature vector leading to model overfitting, principal component analysis (PCA) was used to reduce the dimensionality of the feature vector before it was input into logistic regression and multiple linear regression algorithms for decoding, ultimately yielding a neural activity decoding model.

[0085] The dexterous hand perception-motion integrated control includes: acquiring images of the object to be grasped and joint angle planning 120, image and motion information encoding 121, image and motion perception area electrical stimulation 122, extraction of neural activity features in the motion output area 123, neural activity decoding model 124, output image category and joint angle 125, mapping to dexterous hand grasping mode and joint angle 126.

[0086] Among them, the image and motion sensing area electrical stimulation 122, neural activity feature extraction 123, and neural activity decoding model 124 are the same as the image and motion sensing area electrical stimulation 115, motion output area neural activity feature extraction 117, and neural activity decoding model 119 in the aforementioned modular in vitro biological neural network training.

[0087] The output image category labels are apple, car, water cup, and no action, which are mapped to dexterous hand clenching fist, two-finger pinch, three-finger pinch, and still hand gestures, realizing the control of dexterous hand grasping mode by in vitro biological neural network.

[0088] The output joint angles are a sequence of joint angles, which are mapped to the joint movement information of the five fingers of a dexterous hand, enabling the in vitro biological neural network to precisely control the joint angles of the fingers of a dexterous hand.

[0089] By utilizing PCA for dimensionality reduction analysis of high-dimensional neural activity data, this embodiment further reveals the distribution characteristics of firing patterns induced by different spatiotemporal stimulus sequences in the feature space. For example... Figure 3 As shown, spontaneous discharge and stimulus-induced neural responses are significantly separated in the PC1 direction, verifying the activation effect of external stimuli on biological neural networks; in the PC2 and PC3 planes, the three stimulus sequences induce discharge patterns that form independent clusters, demonstrating the separability of different types of neural responses.

[0090] This embodiment verifies the ability of an in vitro biological neural network to recognize and learn spatiotemporal joint stimulation patterns by evaluating its image classification accuracy and joint angle fitting coefficient within a day / across days. Figure 4 (a) As shown in the left figure, in this embodiment, feature vectors were extracted from three types of image-induced discharges and spontaneous discharge data from the neural network in three stimulation phases (1-15 times, 16-30 times, and 31-45 times) to form 10 categories. The parameters of the logistic regression model in the decoding layer were trained, and the image classification accuracy of the neural network was tested using five discharge data points after the three phases. The results show that, under neuronal involvement, with the increase in the number of stimulations, the classification accuracy improved from 88.75% ± 9.35% in the first phase to 93.75% ± 5.42% in the second phase, and further to 95.42% ± 4.97% in the third phase, with a total improvement of 6.67% in the classification performance of the biological neural network. This demonstrates that with the accumulation of training iterations, the in vitro biological neural network can gradually adapt to external inputs and improve its ability to recognize specific information.

[0091] This embodiment also tested the regression effect of in vitro biological neural networks under continuous multi-day stimulation training. For example... Figure 4 (a) As shown in the right figure, the linear regression model was trained using the first 25 of 30 stimulation-induced discharge data over three days as the training set, and the data from the last 5 rounds of stimulation-induced discharge as the test set. The final regression scores were 0.68±0.19, 0.73±0.11, and 0.80±0.06, respectively, representing a performance improvement of approximately 17.65%. This demonstrates that the in vitro biological neural network exhibits learning behavior regardless of whether stimulation training is conducted within a single day or across multiple days.

[0092] The in vitro biological neural network training was divided into three training phases: 1-15 times, 16-30 times, and 31-45 times. The number of induced discharges of the neural network in each phase was counted. Figure 4 As shown in (b), the number of discharges induced in different training phases did not change significantly with the increase in the number of stimulations. The discharge rate in each phase remained stable, indicating that the pulse stimulation sequence obtained by the designed encoding method can effectively maintain normal neural discharge dynamics in the in vitro biological neural network under long-term and repeated stimulation, without obvious decay or overactivation.

[0093] This embodiment uses 1024 recording electrodes to detect spontaneous discharge signals of a biological neural network over a duration of 5 minutes. The Spike Time Tiling Coefficient (STTC) is used to calculate the correlation between firing sequences recorded by different electrodes, thereby assessing the functional connectivity strength between neural network nodes. The firing sequences detected by two specific electrodes are shown in the figure. and The formula for calculating STTC correlation is as follows:

[0094]

[0095] in, Indicates the issuance sequence All distribution times before and after The proportion of the total duration after time windows are superimposed to the total recording duration of the release sequence. Indicates in sequence Of all the distribution times, those falling within the sequence Before and after any of the distributions The number of items issued within the time window accounts for a portion of the sequence. The proportion of the total number of distributions , The meaning is similar. Related time window ( The value is set to 10ms.

[0096] This embodiment employs a channel selection method based on K-means clustering, which effectively preserves the spatial distribution characteristics of neural signals while reducing computational complexity. Finally, 40 channels closest to each cluster center were selected as representatives. An adjacency matrix is ​​used to display the STTC coefficients (e.g., forty channels) of significant functional connections (edges) between representative electrode channels (nodes) before and after training. Figure 5 (a) and, combined with the physical location of the electrode channels, generate a functional connectivity diagram before and after network training (e.g., Figure 5 (b)).

[0097] Network connectivity characteristics are quantitatively analyzed by calculating network performance metrics such as node degree, node strength, and edge weight. Node degree represents the number of edges connecting each node to other nodes. Edge weight represents the connection strength between two nodes. Node strength represents the sum of the edge weights of each node. Their calculation formulas are as follows:

[0098] Node degree:

[0099]

[0100] in, A node in the adjacency matrix and nodes Does a connected binary quantity exist? It represents the total number of nodes in the network.

[0101] Edge weight:

[0102]

[0103] in, It is a node and nodes The STTC values ​​between.

[0104] Node strength:

[0105]

[0106] To delve into the dynamic changes within the internal structure of biological neural networks, this embodiment employs a consensus clustering method to modularize the neural network, decomposing it into a set of independent functional modules. The connection patterns of nodes within and between modules are measured based on two parameters: the within-module degree z-score and the inter-module participation coefficient. The formula for calculating these parameters is as follows:

[0107] Intra-module connectivity:

[0108]

[0109] in, For nodes Its modules The number of connections to other nodes in the process; yes On all nodes The average value, yes middle The standard deviation of the standard deviation. Intra-module connectivity measures the relative connectivity of a node within its own module, and is used to assess the importance of a node within the module.

[0110] Inter-module participation coefficient:

[0111]

[0112] in, It is a node To module The number of connections to the nodes in the data. It is a node Total degree, This represents the total number of network modules. If the connections of nodes are evenly distributed across all modules, the inter-module participation coefficient is close to 1; if all connections are within their own modules, the participation coefficient is close to 0. These metrics are visualized as box plots to show changes in the distribution of the data before and after training. (This is achieved through independent samples.) t To test and evaluate the significant differences in characteristic values ​​before and after training, P A value <0.05 is considered statistically significant.

[0113] This embodiment uses statistical analysis of network characteristic indicators of an external neural network to further verify the optimization effect of training on the network topology, revealing the changing trends of network functional connections. For example... Figure 5 As shown, after training, node degree and edge weight significantly increased from 28.20±11.82 and 0.23±0.09 to 32.49±8.55 (P<0.0001) and 0.25±0.10 (P<0.001), respectively, indicating that the training process not only increased the number of connections but also enhanced the strength of those connections. Node strength increased from 6.66±4.12 to 8.08±4.08 (P<0.0001), reflecting a significant improvement in the information transmission capability of network nodes. Intra-module connectivity measures the degree of deviation of a node's degree from the average degree of nodes within its module. A higher intra-module connectivity indicates that the node is closer to the "central node" within the module. Inter-module participation coefficient measures the distribution of a node's connections across different modules. When this value is close to 1, it indicates that the node's connections are evenly distributed across all modules, and the node acts as a bridge between different modules. When it is close to 0, it indicates that almost all of the node's connections are concentrated within its own module, with fewer connections to other modules. Under the experimental conditions of this embodiment, the inter-module participation coefficient of the in vitro biological neural network after training significantly increased from 0.80±0.14 to 0.84±0.10 (P<0.0001), while the intra-module connectivity did not change significantly. This indicates that the connections between nodes within the same module are relatively uniform, and the internal structure of the module remains stable after experiencing stimuli. Simultaneously, the connections between network nodes and other modules increase, and these connections are more evenly distributed among different modules.

[0114] The foregoing has described a specific embodiment of the present invention in detail. It should be understood that those skilled in the art can make numerous modifications and variations based on the concept of the present invention without creative effort. Therefore, all technical solutions that can be obtained by those skilled in the art based on the concept of the present invention through logical analysis, reasoning, or limited experimentation on the basis of existing technology should be within the scope of protection defined by the claims.

Claims

1. A perception and motion control method based on a modular in vitro biological neural network, characterized in that, Includes the following steps: 1) Encode the image information of the object to be captured; 2) Encode the motion information of multi-joint dexterous hands; 3) Construct a modular in vitro biological neural network capable of generating neural activity, train it using repeated stimulation, and establish a neural activity decoding model. 4) Input the encoded image and motion information into the trained in vitro biological neural network, use the neural activity decoding model to analyze the activity pattern of the biological neural network, and map the output of the decoding model into dexterous hand motion control.

2. The perception and motion control method based on a modular in vitro biological neural network according to claim 1, characterized in that, Step 1) includes the following steps: 1.1) Collect image information of the object to be captured, extract features from it using a convolutional neural network, and normalize the extracted features; 1.2) The pixel units that make up the feature map are generated with specific ignition times through the defined pulse ignition function. Each feature map represents a receptive field. The ignition times of the pixel units in the same receptive field are arranged into a time series. By adding a time offset, a time difference is generated in the pulse ignition time. 1.3) Use the SMO function to align and compress time series with time differences to generate a series of pulse sequences.

3. The perception and motion control method based on a modular in vitro biological neural network according to claim 2, characterized in that, The pulse ignition function Specifically: ； in, Ignition time, The value of a pixel unit in the feature map after normalization. The maximum duration of the pulse stimulation. To adjust the parameters; The time difference generated by the pulse ignition time is: ； ； in, The position coefficient of the pixel. To sense the size of the field, This is the time offset. This is an ignition time sequence with a time difference.

4. The perception and motion control method based on a modular in vitro biological neural network according to claim 2, characterized in that, Step 1.3) specifically refers to: Pixel units within the same receptive field are configured with an SMO function having the same initial phase. The SMO function adjusts pulses at different positions in the time series to the nearest peak of the oscillation function, obtaining the neuronal stimulation nodes corresponding to each receptive field and compressing them into a series of pulse sequences. Specifically, the SMO function is: ； in, It is the amplitude of the subthreshold membrane oscillation. It is the number of oscillation periods. This is the initial phase value, which is usually set to 0.

5. The perception and motion control method based on a modular in vitro biological neural network according to claim 1, characterized in that, Step 2) includes the following steps: 2.1) Collect the initial angle information of each joint of the dexterous hand; 2.2) Based on the object to be grasped, confirm the grasping mode planning, obtain the multi-joint angle time series of the grasping task, and discretize it; 2.3) The spatiotemporal stimulus sequence of motion information is jointly encoded using binary encoding and multi-site spatial encoding.

6. The perception and motion control method based on a modular in vitro biological neural network according to claim 5, characterized in that, Step 2.2) specifically refers to: Connect the five initial joint angles of the dexterous hand to the target joint angles respectively, and find several intermediate points evenly to generate five discrete time series of dexterous hand joint angles.

7. The perception and motion control method based on a modular in vitro biological neural network according to claim 5, characterized in that, Step 2.3) specifically refers to: In a multi-joint angle time series, the angle of a joint at a certain moment is a value within the range of 0° to 180°, and the angles of multiple joints form a multi-dimensional vector. A time-coding method is used to represent the joint angles of each dimension using six-bit binary numbers. In the resulting sequence of 0s and 1s, 0 indicates that no stimulus is applied at the current moment, and 1 indicates that a stimulus is applied. A spatial coding method is used to map the different joint angle information to stimulation input electrodes at different spatial locations to obtain the joint information at a certain moment. Five pulse sequences were generated after spatiotemporal co-coding, and five stimulation electrodes were selected accordingly.

8. The perception and motion control method based on a modular in vitro biological neural network according to claim 1, characterized in that, The modular in vitro biological neural network is divided into an image sensing area, a motion sensing area, and a motion output area, and is cultured on a high-density microelectrode array with tens of thousands of acquisition channels and dozens of stimulation channels.

9. The perception and motion control method based on a modular in vitro biological neural network according to claim 1, characterized in that, The neural activity decoding model performs the following steps: (1) Obtain the discharge signal of an in vitro biological neural network after a period of time following the application of stimulation; (2) Extract features from the discharge signal, divide the signal segment of each recording electrode into several equal time windows, calculate the number of neuron discharges in each time window and combine them into a feature sequence, and splice all the feature sequences into a feature vector; (3) Use principal component analysis to reduce the dimensionality of the eigenvectors and decode them using classification and regression algorithms.

10. The perception and motion control method based on a modular in vitro biological neural network according to claim 1, characterized in that, Step 4) specifically involves: The image category labels output by the neural activity decoding model are mapped to the hand gestures of the dexterous hand, and the hand gestures are used to control the grasping mode of the dexterous hand. The joint angles output by the neural activity decoding model are mapped to the joint motion information of the five fingers of a dexterous hand, and the joint motion information is used to perform fine control on the finger joint angles of the dexterous hand.

Images