Bio-robot future action prediction method fusing visual image and neural signal

Abstract

A bio-robot future action prediction method fusing visual images and neural signals. The method uses historical behavior data of a bio-robot carried by a video and auto-encoded data of brain neural signals of the bio-robot to predict future motion behaviors of the bio-robot. The method models motion intentions of the bio-robot by means of a visual image sequence, and fuses multi-modal intrinsic brain neural signals of the bio-robot, so as to construct a reversible neural network to predict future actions and motion videos of the bio-robot, and enhance the generalization ability of the model by combing the historical data and an adaptive update policy, thus finally achieving refined prediction of future motion behaviors of the bio-robot.

Description

A method for predicting future actions of biorobots by integrating visual images and neural signals Technical Field

[0001] The present invention belongs to the fields of brain neuroscience and computer vision, and relates to a method for predicting the future actions of a biological robot by fusing visual images and neural signals. Background Art

[0002] With the rapid development of bioengineering and neuroscience, the theoretical research and practical application of implantable brain-computer interfaces for controlling biological behavior have gradually matured. Initial efforts to roughly control the behavior of biorobots have been achieved using common biological agents, such as rats and pigeons. However, due to the complex physiological structure of biorobots and the unpredictability of brain activity, the research on technologies to predict the movement intentions of biorobots and achieve precise control of the behavior of biological agents presents numerous challenges.

[0003] Invention patent CN202210950240.2 proposes multiple models to adaptively decode brain neural signals. However, the model used in this method is a linear model. Although multiple linear models are used, its nonlinear expression ability is limited, making it difficult to accurately decode complex neural signals. Invention patent CN202010852583.6 proposes a neural signal decoding method based on a deep convolutional generative adversarial network. Compared with linear models, this invention has stronger nonlinear feature extraction capabilities, but this method has difficulty in fine-grained analysis of animal behavior and difficulty fusing multimodal data to further extract effective information from neural signals.

[0004] Summary of the Invention

[0005] To overcome the shortcomings of existing technologies and effectively predict biorobot motion, this paper designs a behavior prediction method using visual images and brain neural signals, addressing the high uncertainty of biorobot behavioral responses to complex environments and the difficulty in predicting their own brain activity. This method uses historical behavioral data from videos and autoencoded data from the biorobot's brain neural signals to predict the biorobot's future motion. By modeling the animal's motion intentions through visual image sequences and integrating multimodal intrinsic brain neural signals, a reversible neural network is constructed to predict the organism's future movements and motion videos. This method, combined with historical data and an adaptive update strategy, enhances the model's generalization capabilities, ultimately enabling refined predictions of the organism's future motion behavior.

[0006] The specific technical solutions of the present invention are:

[0007] A method for predicting the future actions of a biorobot by integrating visual images and neural signals, with the following steps:

[0008] Step 1: Use a camera to acquire the environmental background image and the visual image sequence of the bio-robot, use the brain-computer interface to collect biological neural signals, and then divide the visual image sequence and neural signal data into two parts according to the acquisition time. The details are as follows:

[0009] Before placing the bio-robot, shoot the site background to obtain the environmental background image I B ; Place the biorobot in the data collection site, use the camera to collect the visual image sequence of the biorobot's movement in real time, and use the brain-computer interface to obtain the neural signal. The collected visual image sequence is recorded as I, and the collected neural signal is recorded as X; the neural signal and visual image are divided into two parts in chronological order, and are recorded as I and X. b , I n 、X b 、X n , where I b and I n Respectively represent the visual image data from 2t seconds to t seconds from the current time and the visual image data from t seconds to the current time, as the former data, X b and X n They respectively represent the neural signal data from 2t seconds to t seconds from the current time and the neural signal data from t seconds to the current time as the latter part of the data.

[0010] Step 2: Preprocess the visual image to extract the foreground and background, decouple the organism from the environment, obtain the motion component and the environmental component, and extract multi-scale features of the two time periods before and after through the backbone feature extraction network. The details are as follows:

[0011] The visual image sequence I is split into video frames, and the difference between the video frames and the background of the scene is used to estimate the motion status of the biological robot, so as to roughly separate the foreground and background and obtain the biological robot motion image; the backbone feature extraction network is used Input background image and biological robot motion image to extract multi-scale environmental features s from 2t seconds to t seconds from the current time. b and multi-scale motion features m b , and the multi-scale environmental features s from the current time t seconds to the current time n and multi-scale motion features m n .

[0012] Step 3: Preprocess and denoise the neural signal, and use the autoencoding method to autoencode the neural signal to obtain the high-dimensional neural signal features of the two time periods before and after. The details are as follows:

[0013] Using Neural Networks Denoise the neural signal X and then use the feature extraction network Encode the features of neural signals to obtain high-dimensional neural signal features Among them, the neural network N d Through unsupervised loss function Training is performed, where the TV (Total Variation) term is used as a regularization term to constrain the smoothness of the neural signal, and λ is the coefficient of the regularization term; feature extraction network By introducing a decoder and using the Mask Auto Encoder (MAE) method for training; using a feature extraction network Get the high-dimensional neural signal features from 2t seconds to t seconds from the current time And the high-dimensional neural signal features from the current time t seconds to the current time

[0014] Step 4: Divide by the shortest acquisition time, concatenate the multi-scale features of the first part in step 2 and the high-dimensional neural signal features of the corresponding time period in step 3 as the first part of the underlying semantic input, use the reversible neural network to decode the multi-scale features of the first part and the first part of the underlying semantic input, and predict the behavior classification probability and motion visual image sequence of the biological robot in the future unit time based on the time period. The details are as follows:

[0015] Cascade the neural signal representation from 2t seconds to t seconds from the current time Multi-scale environmental features b and multi-scale motion features m b , as the underlying semantic input in Represents the connection operation; uses a reversible neural network decoder For the underlying semantic input z b Decode and obtain the probability distribution of the behavior classification of the biological robot from the current time t seconds to the current time and motion visual image sequences Right now

[0016] Step 5: Divide by acquisition time, repeat the process of step 4 for the multi-scale features of the latter part in step 2 and the high-dimensional neural signal features of the corresponding time period in step 3, and predict the behavior classification probability and motion visual image sequence of the biological robot in the future unit time of this time period based on the time period; use the action recognition method to identify the action category of the latter part, use the inverse process of the reversible neural network to input the image sequence and action behavior of the latter part, and estimate the high-dimensional neural signal features of the former part. The details are as follows:

[0017] Cascade neural signal representation from the current time t seconds to the current time Multi-scale environmental features n and multi-scale motion features m n , as the underlying semantic input in Represents the connection operation; uses a reversible neural network decoder For the underlying semantic input z n Decode and obtain the probability distribution of the behavior classification of the biological robot in the next t seconds and motion visual image sequences Right now Use the action recognition model as the behavior classifier to obtain the action behavior y of the biological robot in the first t seconds n , using the inverse process of the reversible neural network Input the image features from 2t seconds to t seconds from the current time and the visual image sequence and action behavior from t seconds to the current time, and estimate the high-dimensional neural signal features from 2t seconds to t seconds from the current time.

[0018] Step 6: Construct a loss function based on the error between the inference value and the true value, and adaptively update the parameter weights of the reversible neural network. The details are as follows:

[0019] use With z b 、y n and with I n The difference between them updates the reversible neural network, and the updated loss function is in Represents a reversible neural network The determinant of represents the cross entropy loss between the action prediction label and the true value label. The network parameters are updated in small steps using gradient descent and stochastic gradient descent methods. λ1, λ2, λ3, and λ4 represent the coefficients between different loss terms.

[0020] Step 7: Repeat steps 2, 3, and 5 using the updated network weights to obtain the behavior classification probability and motion visual image sequence of the bio-robot in the future unit time, thereby estimating the bio-robot's movement intention and predicting its refined motion behavior. The details are as follows:

[0021] Repeat steps 2, 3, and 5, using I n 、X n Predict biological motion videos within t seconds and action behavior classification Realize the estimation of biological movement intention and the prediction of biological refined movement behavior.

[0022] In computer vision, this invention overcomes the problem of inaccurate prediction of future biological behavior caused by estimating motion influencing factors only through visual images, and in neuroscience, it proposes a new idea of ​​neural network encoding and decoding to obtain refined and deep features that are sufficient to characterize the biological movement intention.

[0023] The beneficial effects of the present invention are as follows: 1) Visual signals and neural signals complement each other, and can be used to predict the future movement state of an organism, and can also further refine the prediction of the organism's movement intention; 2) The visual prediction branch uses a reversible network to reversely constrain the neural signal features, and uses data fitting to extract effective features, eliminating complex and tedious manual feature engineering; 3) Since the visual prediction branch is bidirectional, it can use historical data to bidirectionally standardize the neural network feature extraction module and the visual prediction module to achieve situations where the network is not applicable due to individual changes or environmental changes. BRIEF DESCRIPTION OF THE DRAWINGS

[0024] FIG1 is a flow chart of a method for predicting future actions of a biorobot by integrating visual images and neural signals.

[0025] Figure 2 is the overall architecture diagram of the prediction network. DETAILED DESCRIPTION

[0026] The specific implementation of the present invention is further described below in conjunction with the accompanying drawings and technical solutions.

[0027] The present invention designs a method for predicting the future actions of a biorobot by integrating visual images and neural signals. The overall process is shown in Figures 1 and 2.

[0028] In this example, the experimental site used Hikvision cameras to record biological behavior at a 30-frame rate and a video resolution of 640x480. Plexon neural signal acquisition equipment was used via an invasive brain-computer interface to collect Spike action potentials during rat movement. The Spike neural signal sampling frequency was 30,000 Hz, with 16 signal channels. To ensure real-time performance, an NVIDIA 3090 GPU was used to implement real-time neural network updates and future state inference. The overall data flow of the prediction network is shown in Figure 2.

[0029] The specific prediction steps are as follows:

[0030] 1. Before placing the bio-robot, photograph the site background to obtain the environmental background image I BThe biological robot is placed in the data collection scene, and the camera is used to collect the visual image sequence of the biological robot's movement in real time and the neural signal is obtained using the brain-computer interface. The collected visual image sequence is recorded as I, and the collected neural signal is recorded as X. The neural signal and visual image are divided into two parts in chronological order, and are recorded as I and X. b , I n 、X b 、X n , where I b , I n They represent the image data from 1 second to 0.5 seconds from the current time and the image data from 0.5 seconds from the current time to the current time, respectively. b 、X n Respectively represent the neural signal data from 1 second to 0.5 seconds from the current time and the neural signal data from 0.5 seconds from the current time to the current time;

[0031] 2. Split I into video frames and use the difference between the video frames and the background to estimate the motion of the organism to obtain a motion image. Use the residual neural network ResNet50 as the backbone feature extraction network Input the environmental background image and biological motion image to extract multi-scale environmental features s from 1 second to 0.5 seconds from the current time. b and multi-scale motion features m b , and the multi-scale environmental features s from 0.5 seconds to the current time n and multi-scale motion features m n ;

[0032] 3. Using Neural Networks Denoise the neural signal X and then use the feature extraction network Encode features to obtain high-dimensional neural signal features Among them, the neural network N d Through unsupervised loss function Training is performed, where the TV (Total Variation) term is used as a regularization term to constrain the smoothness of the neural signal, and λ is the coefficient of the regularization term, which is taken as 0.5 here; feature extraction network By introducing a decoder and using the Mask Auto Encoder (MAE) method for training. Neural Network It consists of two layers of 16-channel one-dimensional convolution with a convolution kernel size of 5, and a feature extraction network It consists of four layers of one-dimensional convolution and an additional pooling layer. The number of input channels is 16, 128, 256, and 512 respectively, and the number of output channels is 128, 256, 512, and 1024 respectively. The convolution stride is 2. The pooling layer is not used during masked autoencoder training. During inference, the average pooling operation is used to compress the data content to 1024×1. To facilitate subsequent feature cascade operations, the data dimension is expanded to 1024×80×60 using a padding operation. This network is used to obtain high-dimensional neural signal features from 1 second to 0.5 seconds from the current time. And the high-dimensional neural signal features from 0.5 seconds to the current time

[0033] 4. Cascade neural signal representations from 1 second to 0.5 seconds from the current time Multi-scale environmental features b and multi-scale motion features m b , as the underlying semantic input in Represents the connection operation. Using reversible neural network For the underlying semantic input z b Decode and obtain the probability distribution of biological behavior classification from 0.5 seconds to the current time and motion visual image sequences Right now in The common Unet decoder structure is added with a fully connected layer as the classification head of the classifier, and a reversible network is constructed by the standardized flow model to replace the convolution operation in the Unet and classifier;

[0034] 5. Cascade the neural signal representation from 0.5 seconds to the current time Multi-scale environmental features n and multi-scale motion features m n , as the underlying semantic input Using reversible neural networks For the underlying semantic input z n Decode and obtain the probability distribution of biological behavior classification within the next 0.5 seconds and motion visual image sequences Right now Use the action recognition model TSM (Temporal Shift Module for Efficient Video Understanding) as the behavior classifier to obtain the action behavior y of the biological robot from the previous 0.5 seconds to the current time n . Using the inverse process of a reversible neural network Input image features from 1 second to 0.5 seconds from the current time and image sequences and action behaviors from 0.5 seconds to the current time, and estimate high-dimensional neural signal features from 1 second to 0.5 seconds from the current time.

[0035] 6. Utilize With z b 、y n and with I n The difference between them updates the reversible neural network, and the updated loss function is in Represents a reversible neural network The determinant of represents the cross entropy loss between the action prediction label and the true value label. The network parameters are updated in small steps using gradient descent and stochastic gradient descent. λ1, λ2, λ3, and λ4 represent the coefficients between different loss terms, and their values ​​are 0.1, 0.1, 0.1, and 1 respectively.

[0036] 7. Repeat steps 2, 3, and 5, using I n 、X n Predict biological motion videos within t seconds and action behavior classification Realize the estimation of biological movement intention and the prediction of biological refined movement behavior.

Claims

1. A method for predicting future actions of a biological robot by integrating visual images and neural signals, characterized in that: Here are the steps: Step 1: Use a camera to obtain the environmental background image and the visual image sequence of the biological robot, use the brain-computer interface to collect biological neural signals, and then divide the visual image sequence and neural signal data into two parts according to the acquisition time; the details are as follows: Before placing the bio-robot, take a photo of the site background to obtain the environmental background image I B ; Place the bio-robot in the data collection site, use the camera to collect the visual image sequence of the bio-robot's movement in real time, and use the brain-computer interface to obtain the neural signal. The collected visual image sequence is recorded as I, and the collected neural signal is recorded as X; the neural signal and the visual image are divided into two parts in time order, which are recorded as I and X. b ,I n , X b , X n , where I b and I n Respectively represent the visual image data from 2t seconds to t seconds from the current time and the visual image data from t seconds to the current time, as the former data, X b and X n Respectively represent the neural signal data from 2t seconds to t seconds from the current time and the neural signal data from t seconds to the current time as the latter part of the data; Step 2: Preprocess the visual image, extract the foreground and background, decouple the organism from the environment, obtain the motion component and the environmental component, and extract the multi-scale features of the two time periods before and after through the backbone feature extraction network; The details are as follows: The visual image sequence I is split into video frames, and the difference between the video frames and the background of the venue is used to estimate the motion status of the biological robot, so as to roughly separate the foreground and background and obtain the biological robot motion image; the backbone feature extraction network is used Input background image and biological robot motion image to extract multi-scale environmental features s from 2t seconds to t seconds from the current time. b and multi-scale motion features m b , and the multi-scale environmental features s from the current time t seconds to the current time n and multi-scale motion features m n ; Step 3: Preprocess and denoise the neural signal, and use the autoencoding method to autoencode the neural signal to obtain the high-dimensional neural signal features of the two time periods before and after; the details are as follows: Using Neural Networks De-noise the neural signal X, and then use the feature extraction network Encode the features of neural signals to obtain high-dimensional neural signal features Among them, the neural network N d Through the unsupervised loss function Training, including TV The term is used as a regular term to constrain the smoothness of the neural signal, and λ is the coefficient of the regular term; feature extraction network Train by introducing a decoder and using mask self-encoding; use a feature extraction network Get the high-dimensional neural signal features from 2t seconds to t seconds from the current time And the high-dimensional neural signal features from the current time t seconds to the current time Step 4: Divide by the short acquisition time, cascade the multi-scale features of the first part in step 2 and the high-dimensional neural signal features of the corresponding time period in step 3 as the first part of the underlying semantic input, use the reversible neural network to decode the multi-scale features of the first part and the first part of the underlying semantic input, and predict the behavior classification probability and motion visual image sequence of the biological robot in the future unit time based on the time period; the details are as follows: Cascade the neural signal representation from 2t seconds to t seconds from the current time Multi-scale environmental features b and multi-scale motion features m b , as the underlying semantic input in Represents the concatenation operation; uses a reversible neural network decoder For the underlying semantic input z b Decode and obtain the probability distribution of the behavior classification of the biological robot from the current time t seconds to the current time and motion visual image sequences Right now Step 5: Divide by acquisition time, repeat step 4 for the multi-scale features of the latter part in step 2 and the high-dimensional neural signal features of the corresponding time period in step 3, predict the behavior classification probability and motion visual image sequence of the biological robot in the future unit time of the time period based on the time period; and use the action recognition method to identify the action category of the latter part, use the inverse process of the reversible neural network to input the image sequence and action behavior of the latter part, and estimate the high-dimensional neural signal features of the former part; the details are as follows: Cascade the neural signal representation from the current time t seconds to the current time Multi-scale environmental features n and multi-scale motion features m n , as the underlying semantic input in Represents the concatenation operation; uses a reversible neural network decoder For the underlying semantic input z n Decode and obtain the probability distribution of the behavior classification of the biological robot in the next t seconds and motion visual image sequences Right now Use the action recognition model as a behavior classifier to obtain the action behavior y of the biological robot in the previous t seconds n , using the inverse process of a reversible neural network Input the image features from 2t seconds to t seconds from the current time and the visual image sequence and action behavior from t seconds to the current time, and estimate the high-dimensional neural signal features from 2t seconds to t seconds from the current time. Step 6: Construct a loss function based on the error between the inference value and the true value, and adaptively update the parameter weights of the reversible neural network; the details are as follows: use With z b ,y n and with I n The difference between updates the reversible neural network, and the updated loss function is in Represents a reversible neural network The determinant of represents the cross entropy loss between the action prediction label and the true value label. The network parameters are updated in small steps using the gradient descent and stochastic gradient descent methods. λ1, λ2, λ3, and λ4 represent the coefficients between different loss terms. Step 7: Repeat steps 2, 3, and 5 using the updated network weights to obtain the behavior classification probability and motion visual image sequence of the biological robot in the future unit time, and realize the estimation of the biological robot's movement intention and the prediction of the biological robot's refined motion behavior; the details are as follows: Repeat steps 2, 3, and 5, using I n , X n Predict biological motion video within t seconds in the future and action behavior classification Realize the estimation of biological movement intention and the prediction of biological refined movement behavior.

2. A method for predicting future actions of a biological robot by integrating visual images and neural signals according to claim 1, characterized in that: The backbone feature extraction network The residual neural network ResNet50 is used.

3. The method for predicting future actions of a biological robot by integrating visual images and neural signals according to claim 1, characterized in that: The neural network The feature extraction network consists of two layers of 16-channel one-dimensional convolution with a convolution kernel size of 5. It consists of four layers of one-dimensional convolution and additional pooling layers.

4. The method for predicting future actions of a biological robot by integrating visual images and neural signals according to claim 1, characterized in that: The reversible neural network decoder A fully connected layer is added to the Unet decoder structure as the classification head of the classifier, and a reversible network is constructed by the standardized flow model to replace the convolution operations in Unet and the classifier.

5. The method for predicting future actions of a biological robot by integrating visual images and neural signals according to claim 1, characterized in that: The action recognition model adopts TSM as the action classifier.

Images