A behavior analysis method, system, device and medium based on multi-modal data

Abstract

The application discloses a behavior analysis method, system, device and medium based on multi-modal data. The method comprises the following steps: extracting a behavior label corresponding to each frame of image in animal behavior video data, and constructing a behavior label sequence; aligning a neuron calcium activity time sequence in a standardized neuron calcium activity time sequence dataset with the behavior label sequence, and obtaining an aligned structured data table; taking the aligned neuron calcium activity time sequence as a feature of a node, and constructing a weighted graph structure; training a cascaded graph convolution network, a graph sampling aggregation network and a graph attention network according to the weighted graph structure and the aligned behavior label, and obtaining a trained graph convolution network, a trained graph sampling aggregation network and a trained graph attention network, so as to perform behavior prediction on a neuron calcium ion fluorescence signal value to be predicted, and obtain a behavior prediction result. The application can improve the behavior analysis accuracy.

Description

Technical Field

[0001] This application relates to the field of data analysis technology, and in particular to a method, system, device and medium for behavior analysis based on multimodal data. Background Technology

[0002] With breakthroughs in neuroscience technologies such as two-photon calcium imaging, researchers are now able to acquire large-scale neuronal activity data at single-cell resolution in deep brain regions of animals. This type of large-scale, neuronal-level, multimodal brain science data (including high-frame-rate fluorescence imaging videos, time series of single-neuron calcium signals, and synchronously recorded videos of animal behavior) is experiencing explosive growth.

[0003] However, existing behavior analysis methods suffer from multi-scale heterogeneity and high-dimensional spatiotemporal complexity of data, modal isolation of analysis paradigms, and feature extraction bottlenecks in complex dynamic networks. This makes it difficult for existing technologies to efficiently and accurately decode the intricate dependencies between neuronal population activity signals and their downstream behavioral labels from massive amounts of data, resulting in relatively low accuracy in behavior analysis. Summary of the Invention

[0004] This application aims to propose a behavior analysis method, system, device, and medium based on multimodal data, which can efficiently and accurately decode the intricate dependencies between neuronal population activity signals and their downstream behavioral labels from massive amounts of data, thereby improving the accuracy of behavior analysis.

[0005] In a first aspect, embodiments of this application provide a behavior analysis method based on multimodal data, the method comprising:

[0006] Acquire standardized time-series datasets of neuronal calcium activity, animal behavioral video data, and neuronal calcium ion fluorescence signal values ​​to be predicted;

[0007] Extract the behavior tags corresponding to each frame of the animal behavior video data, and construct a behavior tag sequence;

[0008] Align the neuronal calcium activity time series in the standardized neuronal calcium activity time series dataset with the behavior label sequence to obtain an aligned structured data table, which includes the aligned behavior labels and the aligned neuronal calcium activity time series.

[0009] The aligned neuronal calcium activity time series is used as the feature of the node to construct a weighted graph structure;

[0010] Based on the weighted graph structure and the aligned behavior labels, a cascaded graph convolutional network, graph sampling aggregation network, and graph attention network are trained to obtain a trained graph convolutional network, a trained graph sampling aggregation network, and a trained graph attention network. These networks are then used to predict the behavior of the neuron calcium ion fluorescence signal value to be predicted, thus obtaining the behavior prediction result.

[0011] Compared with the prior art, the first aspect of this application has the following beneficial effects:

[0012] This method acquires a standardized neuronal calcium activity time-series dataset, animal behavior video data, and the neuronal calcium ion fluorescence signal value to be predicted. It extracts behavior labels corresponding to each frame of the animal behavior video data to construct a behavior label sequence. The standardized neuronal calcium activity time-series dataset is aligned with the behavior label sequence to obtain an aligned structured data table, which includes the aligned behavior labels and the aligned neuronal calcium activity time-series. The aligned neuronal calcium activity time-series is used as the feature of the nodes to construct a weighted graph structure. Based on the weighted graph structure and the aligned behavior labels, a cascaded graph convolutional network, graph sampling aggregation network, and graph attention network are trained to obtain a trained graph convolutional network, a trained graph sampling aggregation network, and a trained graph attention network. These trained graph convolutional networks, graph sampling aggregation networks, and graph attention networks are then used to predict the behavior of the neuronal calcium ion fluorescence signal value to be predicted, thus obtaining the behavior prediction result. Thus, by aligning the time series of neuronal calcium activity with the behavioral label sequence, the barrier between microscopic neuronal activity and macroscopic behavioral labels is broken down, providing a novel computational paradigm with biological interpretability for decoding complex neuro-behavioral relationships. Furthermore, by using the aligned neuronal calcium activity time series as node features to construct a weighted graph structure, "temporal correlation" is effectively mapped to "spatial topological connectivity," thereby laying a solid data foundation for subsequent graph deep learning to obtain an accurate predictive network model. Finally, using the trained network model for behavior prediction, the complex dependencies between neuronal population activity signals and their downstream behavioral labels can be decoded efficiently and accurately from massive amounts of data, thereby improving the accuracy of behavior analysis.

[0013] In some implementations, obtaining a standardized time-series dataset of neuronal calcium activity includes:

[0014] Calculate the calcium activity intensity of each neuron at each time step to construct a time series dataset of neuronal calcium activity, wherein the calcium activity intensity is the relative fluorescence intensity of each neuron at each time step;

[0015] Smooth and denoise the calcium activity intensity data in the neuronal calcium activity time series dataset to obtain the denoised neuronal calcium activity time series dataset.

[0016] The calcium activity intensity data in the denoised neuronal calcium activity time series dataset is normalized to obtain a standardized neuronal calcium activity time series dataset.

[0017] In some implementations, constructing a weighted graph structure by using the aligned neuronal calcium activity time series as features of nodes includes:

[0018] Construct an edge between two adjacent nodes;

[0019] Based on the characteristics of the nodes, the correlation coefficient between two adjacent nodes is calculated, and the correlation coefficient is used as the weight value of the edge between the two adjacent nodes.

[0020] Based on the weight values, nodes, and edges, construct a weighted graph structure.

[0021] In some implementations, calculating the correlation coefficient between two adjacent nodes based on the characteristics of the nodes includes:

[0022] Based on the characteristics of the nodes, obtain the calcium activity intensity of each node at time t, and obtain the average calcium activity intensity of each node throughout the entire time series.

[0023] The correlation coefficient between two adjacent nodes is calculated based on the calcium activity intensity and the average calcium activity intensity.

[0024] In some implementations, the step of training a cascaded graph convolutional network, a graph sampling aggregation network, and a graph attention network based on the weighted graph structure and the aligned behavior labels to obtain a trained graph convolutional network, a trained graph sampling aggregation network, and a trained graph attention network includes:

[0025] The weight values ​​of the edges in the weighted graph structure are compared with a preset threshold to construct a weighted adjacency matrix;

[0026] Based on the time series of neuron calcium activity corresponding to each node in the weighted graph structure, a node feature matrix is ​​constructed.

[0027] Based on the weighted adjacency matrix, the node feature matrix, and the aligned behavior labels, a cascaded graph convolutional network, a graph sampling aggregation network, and a graph attention network are trained to obtain the trained graph convolutional network, the trained graph sampling aggregation network, and the trained graph attention network. The weighted graph structure is represented by the weighted adjacency matrix and the node feature matrix.

[0028] In some embodiments, the step of performing behavioral prediction on the calcium ion fluorescence signal value of the neuron to be predicted using the trained graph convolutional network, the trained graph sampling aggregation network, and the trained graph attention network to obtain the behavioral prediction result includes:

[0029] The predicted neuron calcium ion fluorescence signal value is input into the trained graph convolutional network to obtain the first output result;

[0030] The first output result is input into the trained graph sampling aggregation network to obtain the second output result;

[0031] The second output result is input into the trained graph attention network to obtain the third output result;

[0032] Based on the third output result, behavior prediction is performed to obtain the behavior prediction result.

[0033] In some implementations, the step of performing behavior prediction based on the third output result to obtain a behavior prediction result includes:

[0034] The third output result is aggregated into a single graph-level representation by the readout function;

[0035] The single graph-level representation is input into a multilayer perceptron classifier for behavior prediction, and the behavior prediction result is obtained.

[0036] Secondly, embodiments of this application also provide a behavior analysis system based on multimodal data, the system comprising:

[0037] The data acquisition unit is used to acquire standardized time-series datasets of neuronal calcium activity, animal behavior video data, and neuronal calcium ion fluorescence signal values ​​to be predicted.

[0038] The tag extraction unit is used to extract the behavior tags corresponding to each frame of the animal behavior video data and construct a behavior tag sequence.

[0039] A data alignment unit is used to align the neuronal calcium activity time series in the standardized neuronal calcium activity time series dataset with the behavior label sequence to obtain an aligned structured data table, wherein the structured data table includes aligned behavior labels and aligned neuronal calcium activity time series;

[0040] A weighted graph construction unit is used to construct a weighted graph structure by using the aligned neuronal calcium activity time series as the features of the nodes.

[0041] The behavior prediction unit is used to train a cascaded graph convolutional network, graph sampling aggregation network, and graph attention network based on the weighted graph structure and the aligned behavior labels, so as to obtain a trained graph convolutional network, a trained graph sampling aggregation network, and a trained graph attention network, so as to perform behavior prediction on the calcium ion fluorescence signal value of the neuron to be predicted through the trained graph convolutional network, the trained graph sampling aggregation network, and the trained graph attention network, and obtain a behavior prediction result.

[0042] Thirdly, embodiments of this application also provide an electronic device, including at least one control processor and a memory for communicatively connecting to the at least one control processor; the memory stores instructions executable by the at least one control processor, the instructions being executed by the at least one control processor to enable the at least one control processor to perform a behavior analysis method based on multimodal data as described above.

[0043] Fourthly, embodiments of this application also provide a computer-readable storage medium storing computer-executable instructions for causing a computer to execute a behavior analysis method based on multimodal data as described above.

[0044] It is understood that the beneficial effects of the second to fourth aspects compared with the related technologies are the same as the beneficial effects of the first aspect compared with the related technologies. Please refer to the relevant description in the first aspect above, which will not be repeated here. Attached Figure Description

[0045] The above and / or additional aspects and advantages of this application will become apparent and readily understood from the description of the embodiments taken in conjunction with the following drawings, in which:

[0046] Figure 1 This is a flowchart illustrating an embodiment of the behavior analysis method based on multimodal data provided in this application;

[0047] Figure 2 This is a schematic diagram of the weighted graph structure construction process in the best embodiment of the behavior analysis method based on multimodal data provided in this application;

[0048] Figure 3 This is a schematic diagram of the behavior prediction process in the best embodiment of the behavior analysis method based on multimodal data provided in this application;

[0049] Figure 4 This is a schematic diagram of the structure of an embodiment of the behavior analysis system based on multimodal data provided in this application;

[0050] Figure 5This is a schematic diagram of the structure of an embodiment of the electronic device provided in this application. Detailed Implementation

[0051] The embodiments of this application are described in detail below. Examples of these embodiments are shown in the accompanying drawings, wherein the same or similar reference numerals denote the same or similar elements or elements having the same or similar functions throughout. The embodiments described below with reference to the accompanying drawings are exemplary and are only used to explain this application, and should not be construed as limiting this application.

[0052] In the description of this application, the use of terms such as "first," "second," etc., is for the purpose of distinguishing technical features only and should not be construed as indicating or implying relative importance or implicitly indicating the number of technical features indicated or the order of the technical features indicated.

[0053] In the description of this application, it should be understood that the orientation descriptions, such as up, down, etc., are based on the orientation or positional relationship shown in the accompanying drawings, and are only for the convenience of describing this application and simplifying the description, and do not indicate or imply that the device or element referred to must have a specific orientation, or be constructed and operated in a specific orientation, and therefore should not be construed as a limitation of this application.

[0054] In the description of this application, it should be noted that, unless otherwise explicitly defined, terms such as "setup," "installation," and "connection" should be interpreted broadly, and those skilled in the art can reasonably determine the specific meaning of the above terms in this application in conjunction with the specific content of the technical solution.

[0055] First, let's analyze some of the terms used in this application:

[0056] Multimodal data refers to multiple data streams from different sources with different structures or types. In this embodiment, it includes high frame rate fluorescence imaging video, single neuron calcium signal time series, and synchronously recorded animal behavior video.

[0057] Graph Neural Network (GNN): A deep learning model specifically designed for processing graph-structured data.

[0058] Graph Convolutional Network (GCN): A type of GNN layer that learns node representations by aggregating features from the direct neighbors of a node.

[0059] GraphSAGE is a type of GNN layer that learns node representations by sampling and aggregating the neighborhood of nodes, and has good inductive ability.

[0060] Graph Attention Network (GAT) is a type of GNN layer that uses an attention mechanism to assign different weights to a node's neighbors.

[0061] Elevated Plus-Maze (EPM): A standard experimental setup used to study anxiety behavior in rodents such as mice.

[0062] Receptive field: In neuroscience, this specifically refers to the area in a neuron that responds to external stimuli. For example, a neuron in the visual cortex may only respond to a light spot or line at a specific location in the visual field; this area is the neuron's receptive field. In this embodiment, it specifically refers to the effective stimulation area of ​​a single neuron in visual space, determined by analyzing two-photon calcium imaging data.

[0063] The Elevated Plus Maze Experiment (EPM) is a widely used research paradigm in rodent behavior studies to assess anxiety-like behaviors in animals. The apparatus consists of a plus-shaped maze, including two opposing open arms and two opposing closed arms, elevated off the ground. In this embodiment, it specifically refers to a standardized behavioral testing scenario used to simultaneously record neural activity data of experimental animals under specific behavioral states.

[0064] Two-photon calcium imaging is an advanced in vivo optical microscopy technique that utilizes the principle of two-photon excitation to achieve deep-penetrating, high-resolution fluorescence imaging of neurons expressing calcium ion indicator proteins. When neurons fire and generate action potentials, the intracellular calcium ion concentration increases, leading to changes in fluorescence intensity. In this embodiment, it specifically refers to a core data acquisition method for simultaneously monitoring the dynamic activity of hundreds or thousands of neurons in a specific region deep within the brain.

[0065] Calcium Fluorescent Indicator Imaging (CFIA) is a technique that uses an optical microscope to record indirect signals of neural activity by employing a protein (i.e., a calcium ion fluorescent indicator) that is highly sensitive to changes in intracellular calcium ion concentration, encoded by genes or labeled with chemical dyes. When neurons are excited, the influx of calcium ions causes them to bind to the indicator, resulting in changes in fluorescence intensity or wavelength. In this embodiment, it specifically refers to the raw fluorescence signal source acquired by a two-photon microscope, reflecting the dynamics of the electrical activity of a neuronal population.

[0066] Due to the multi-scale heterogeneity and high-dimensional spatiotemporal complexity of data, the modal isolation of analytical paradigms, and the bottleneck of feature extraction in complex dynamic networks, existing behavioral analysis methods are unable to efficiently and accurately decode the intricate dependencies between neuronal population activity signals and their downstream behavioral labels from massive amounts of data, resulting in relatively low accuracy in behavioral analysis.

[0067] To address the problems existing in the prior art, this application proposes a behavior analysis method, system, device, and medium based on multimodal data.

[0068] Reference Figure 1 This application provides a flowchart illustrating a behavior analysis method based on multimodal data. This behavior analysis method is applied to electronic devices, such as servers or mobile terminals. Figure 1 As shown, this behavior analysis method based on multimodal data may include the following steps:

[0069] Step S101: Obtain standardized time-series datasets of neuronal calcium activity, animal behavior video data, and neuronal calcium ion fluorescence signal values ​​to be predicted;

[0070] Step S102: Extract the behavior labels corresponding to each frame of the animal behavior video data and construct a behavior label sequence;

[0071] Step S103: Align the neuronal calcium activity time series in the standardized neuronal calcium activity time series dataset with the behavior label sequence to obtain an aligned structured data table. The structured data table includes the aligned behavior labels and the aligned neuronal calcium activity time series.

[0072] Step S104: Use the aligned neuronal calcium activity time series as the features of the nodes to construct a weighted graph structure;

[0073] Step S105: Based on the weighted graph structure and aligned behavior labels, train the cascaded graph convolutional network, graph sampling aggregation network, and graph attention network to obtain the trained graph convolutional network, trained graph sampling aggregation network, and trained graph attention network. Then, use the trained graph convolutional network, trained graph sampling aggregation network, and trained graph attention network to predict the behavior of the neurons to be predicted by the calcium ion fluorescence signal values, and obtain the behavior prediction results.

[0074] In this embodiment, a standardized neuronal calcium activity time-series dataset, animal behavior video data, and the neuronal calcium ion fluorescence signal value to be predicted are acquired. Behavioral labels corresponding to each frame of the animal behavior video data are extracted to construct a behavioral label sequence. The neuronal calcium activity time-series in the standardized dataset is aligned with the behavioral label sequence to obtain an aligned structured data table, which includes aligned behavioral labels and aligned neuronal calcium activity time-series data. The aligned neuronal calcium activity time-series data is used as the features of nodes to construct a weighted graph structure. Based on the weighted graph structure and the aligned behavioral labels, a cascaded graph convolutional network, graph sampling aggregation network, and graph attention network are trained to obtain a trained graph convolutional network, a trained graph sampling aggregation network, and a trained graph attention network. These networks are then used to predict the behavior of the neuronal calcium ion fluorescence signal value to be predicted, thus obtaining the behavior prediction result. Thus, by aligning the time series of neuronal calcium activity with the behavioral label sequence, the barrier between microscopic neuronal activity and macroscopic behavioral labels is broken down, providing a novel computational paradigm with biological interpretability for decoding complex neuro-behavioral relationships. Furthermore, by using the aligned neuronal calcium activity time series as node features to construct a weighted graph structure, "temporal correlation" is effectively mapped to "spatial topological connectivity," thereby laying a solid data foundation for subsequent graph deep learning to obtain an accurate predictive network model. Finally, using the trained network model for behavior prediction, the complex dependencies between neuronal population activity signals and their downstream behavioral labels can be decoded efficiently and accurately from massive amounts of data, thereby improving the accuracy of behavior analysis.

[0075] The neuronal calcium ion fluorescence signal value to be predicted can be the obtained relative fluorescence intensity (i.e. calcium activity intensity), and the relative fluorescence intensity to be predicted is used as the neuronal calcium ion fluorescence signal value to be predicted.

[0076] The aforementioned acquisition of standardized neuronal calcium activity time-series datasets and animal behavior video data can be standardized neuronal calcium activity time-series datasets and animal behavior video data acquired within the same time period.

[0077] The above-mentioned extraction of behavioral labels corresponding to each frame of animal behavior video data can be achieved by parsing the animal behavior video data and extracting the behavioral labels corresponding to the frame-level images (for example, in the elevated cross maze EPM experiment, behavioral states are classified into categories such as 'Teetering' [shaking / anxiety], 'Rest' [stillness], and 'Exploring' [exploration]), resulting in a discrete sequence of behavioral labels (i.e., temporal data of behavioral labels).

[0078] The above-mentioned alignment of neuronal calcium activity time series with behavioral label sequences in a standardized neuronal calcium activity time series dataset can be achieved by aligning the neuronal calcium activity time series with behavioral label sequences in the order of time in the sequences.

[0079] In some implementations, a standardized time-series dataset of neuronal calcium activity is obtained, including:

[0080] Calculate the calcium activity intensity of each neuron at each time step, construct a time series dataset of neuronal calcium activity, and the calcium activity intensity is the relative fluorescence intensity of each neuron at each time step;

[0081] Smooth and denoise the calcium activity intensity data in the neuronal calcium activity time series dataset to obtain the denoised neuronal calcium activity time series dataset.

[0082] The calcium activity intensity data in the denoised neuronal calcium activity time series dataset is normalized to obtain a standardized neuronal calcium activity time series dataset.

[0083] In this embodiment, by smoothing, denoising, and normalizing the calcium activity intensity data in the neuronal calcium activity time series dataset, it is possible to ensure that the feature vectors of different neurons have a uniform scale distribution, thereby eliminating the influence of individual baseline differences on the subsequent learning weights of the graph network.

[0084] The above-mentioned smoothing and denoising of calcium activity intensity data in the neuronal calcium activity time series dataset can be achieved by using a smoothing algorithm to smooth and denoise the calcium activity intensity data in the neuronal calcium activity time series dataset.

[0085] In some implementations, the aligned neuronal calcium activity time series are used as features of the nodes to construct a weighted graph structure, including:

[0086] Construct an edge between two adjacent nodes;

[0087] Based on the characteristics of the nodes, calculate the correlation coefficient between two adjacent nodes, and use the correlation coefficient as the weight value of the edge between the two adjacent nodes.

[0088] Construct a weighted graph structure based on weight values, nodes, and edges.

[0089] In this embodiment, by using the correlation coefficient as the weight value of the edge between two adjacent nodes, a weighted graph structure is constructed based on the weight value, nodes, and edges. The constructed weighted graph structure includes the shared temporal dynamics and close functional connections between neurons, laying a solid data foundation for subsequent graph deep learning (including graph convolutional networks, graph sampling aggregation networks, and graph attention networks) to obtain a network model with accurate predictions.

[0090] The above calculation of the correlation coefficient between two adjacent nodes can be done by calculating the Pearson correlation coefficient between two adjacent nodes.

[0091] In some implementations, the correlation coefficient between two adjacent nodes is calculated based on the characteristics of the nodes, including:

[0092] Based on the characteristics of the nodes, obtain the calcium activity intensity of each node at time t, and obtain the average calcium activity intensity of each node throughout the entire time series.

[0093] The correlation coefficient between two adjacent nodes is calculated based on the calcium activity intensity and the average calcium activity intensity.

[0094] In this embodiment, by calculating the correlation coefficient between two adjacent nodes based on calcium activity intensity and average calcium activity intensity, the shared temporal dynamics and close functional connections between neurons can be obtained, laying a solid data foundation for subsequent graph deep learning and obtaining a network model with accurate predictions.

[0095] The above-mentioned calcium activity intensity can be relative fluorescence intensity.

[0096] The above method for obtaining the average calcium activity intensity of each node in the entire time series can be achieved by summing the calcium activity intensity at each moment in the entire time series and then averaging the sums to obtain the average calcium activity intensity of each node in the entire time series.

[0097] In some implementations, based on the weighted graph structure and aligned behavior labels, a cascaded graph convolutional network, a graph sampling aggregation network, and a graph attention network are trained to obtain the trained graph convolutional network, the trained graph sampling aggregation network, and the trained graph attention network, including:

[0098] The weight values ​​of the edges in the weighted graph structure are compared with a preset threshold to construct a weighted adjacency matrix;

[0099] Construct a node feature matrix based on the time series of neuron calcium activity corresponding to each node in the weighted graph structure;

[0100] Based on the weighted adjacency matrix, node feature matrix, and aligned behavior labels, a cascaded graph convolutional network, graph sampling aggregation network, and graph attention network are trained to obtain the trained graph convolutional network, trained graph sampling aggregation network, and trained graph attention network. The weighted graph structure is represented by the weighted adjacency matrix and node feature matrix.

[0101] In this embodiment, the weight values ​​of edges in the weighted graph structure are compared with preset thresholds to construct a weighted adjacency matrix. This can filter weakly correlated connections caused by noise and reduce the computational complexity of subsequent graph convolution operations (including graph convolutional networks, graph sampling aggregation networks, and graph attention networks), thereby improving the efficiency of later behavior prediction. The cascaded graph convolutional networks, graph sampling aggregation networks, and graph attention networks are not simply a stack of modules, but rather based on a biomimetic cognitive logic of "from local to global, from structure to semantics." This achieves a progressive evolution and enhancement of neural signal features from low-order structural information to high-order behavioral semantics, significantly improving the model's decoding accuracy and generalization ability for complex downstream tasks (such as the classification of specific anxiety behaviors in experimental animals on EPM).

[0102] The above method compares the weight values ​​of edges in the weighted graph structure with a preset threshold to construct a weighted adjacency matrix. This can be achieved by representing the row and column values ​​with weight values ​​greater than or equal to the preset threshold with the weight values, and representing the row and column values ​​with weight values ​​less than the preset threshold with zero, thus constructing a weighted adjacency matrix.

[0103] The above-mentioned preset threshold can be adjusted according to the actual situation, and this embodiment does not impose specific limitations.

[0104] The above-mentioned training of a cascaded graph convolutional network, graph sampling aggregation network, and graph attention network based on the weighted adjacency matrix, node feature matrix, and aligned behavior labels yields a trained graph convolutional network, a trained graph sampling aggregation network, and a trained graph attention network. This can be achieved using a loss function known to those skilled in the art (e.g., the cross-entropy loss function), with the weighted adjacency matrix, node feature matrix, and aligned behavior labels as the input dataset for training the cascaded graph convolutional network, graph sampling aggregation network, and graph attention network.

[0105] In some implementations, behavior prediction is performed on the calcium ion fluorescence signal values ​​of the neurons to be predicted using a trained graph convolutional network, a trained graph sampling aggregation network, and a trained graph attention network, resulting in behavior prediction results, including:

[0106] The calcium ion fluorescence signal value of the neuron to be predicted is input into the trained graph convolutional network to obtain the first output result;

[0107] The first output result is input into the trained graph sampling aggregation network to obtain the second output result;

[0108] The second output is input into the trained graph attention network to obtain the third output.

[0109] Behavior prediction is performed based on the third output result to obtain the behavior prediction result.

[0110] In this embodiment, by using a trained graph convolutional network, a trained graph sampling aggregation network, and a trained graph attention network to predict the behavior of the neuron calcium ion fluorescence signal value to be predicted, it is possible to efficiently and accurately decode the intricate dependencies between neuronal population activity signals and their downstream behavioral labels from massive amounts of data, thereby improving the accuracy of behavior analysis.

[0111] In some implementations, behavior prediction is performed based on the third output result to obtain the behavior prediction result, including:

[0112] The third output is aggregated into a single graph-level representation by reading out the function;

[0113] A single graph-level representation is input into a multilayer perceptron classifier for behavior prediction, and the behavior prediction result is obtained.

[0114] In this embodiment, the third output result is aggregated into a single graph-level representation by the readout function. The single graph-level representation is then input into a multilayer perceptron classifier for behavior prediction. This enables high-precision behavior prediction by utilizing the most discriminative features learned from multimodal data.

[0115] The readout function described above is mainly used to aggregate the features of each node in the graph into a feature representation of the entire graph.

[0116] To facilitate understanding by those skilled in the art, a set of preferred embodiments is provided below:

[0117] With breakthroughs in neuroscience technologies such as two-photon calcium imaging, researchers are now able to acquire large-scale neuronal activity data at single-cell resolution in deep animal brain regions. This type of large-scale, neuronal-level, multimodal brain science data (including high-frame-rate fluorescence imaging videos, time-series of single-neuron calcium signals, and simultaneously recorded animal behavioral videos) is experiencing explosive growth. However, efficiently and accurately decoding the intricate dependencies between neuronal population activity signals and their downstream behavioral labels from massive amounts of data still faces significant challenges, primarily manifested in the following three technical bottlenecks:

[0118] (1) Multi-scale heterogeneity and high-dimensional spatiotemporal heterogeneity of data: Neuroscience data contains distinctly different data streams: microscopic neuronal firing patterns (high frequency, discrete), mesoscopic calcium ion concentration changes (continuous, noisy), and macroscopic animal behavioral characteristics (posture, movement trajectory). These data are not only modally diverse, but also differ greatly in time scale and data dimension, making it difficult for traditional single data processing models to effectively integrate such complex heterogeneous information.

[0119] (2) Limitations of Unimodal Analytical Paradigms: Current mainstream processing methods tend to analyze neural activity and behavioral data separately, lacking effective means to unify heterogeneous data streams under the same analytical framework. This "modal isolation" approach ignores the potential high correlation between different data streams (e.g., the functional connection between the synchronous firing activity of specific neuronal clusters and specific anxiety behaviors of experimental animals in an elevated cruciate maze), thus hindering a comprehensive understanding of the intrinsic connection between neural circuits and behavioral phenotypes.

[0120] (3) Inefficiency in Mining Complex Network Topologies: Although Graph Neural Networks (GNNs) perform well in processing non-Euclidean data, existing standard GNN models (such as single GCN or GAT models) often exhibit performance limitations when directly applied to such noisy, dynamic, and biologically-dependent neural data. They struggle to simultaneously capture multi-scale spatiotemporal features in neural networks, ranging from local topology to global dependencies, resulting in a failure to balance high accuracy and computational efficiency in behavior prediction tasks.

[0121] To address the aforementioned technical problems, this embodiment provides an innovative graph-based perspective multimodal data analysis and prediction scheme. This embodiment goes beyond simple model application, constructing a complete end-to-end processing system from raw data to behavioral decisions. The specific technical solution of this embodiment includes the following:

[0122] I. Data Preprocessing and Graph Construction.

[0123] The first stage of this embodiment is data preprocessing and graph construction. This stage aims to establish a standardized data processing pipeline to transform raw, heterogeneous, multimodal brain science data (including microscopic neuronal calcium ion fluorescence signal values ​​(i.e., relative fluorescence intensity) and macroscopic animal behavioral videos) into high-quality graph-structured data that can be directly processed by graph neural network (GNN) models. The overall processing flow is as follows: Figure 2 As shown. The specific implementation steps are as follows:

[0124] 1. Multimodal data input and neural signal optimization. First, the system receives two core input data streams: a standardized time-series dataset of neuronal calcium activity. , Includes the relative fluorescence intensity (i.e., calcium activity intensity) of each neuron over time. ,in This represents the change in fluorescence intensity (i.e., the fluorescence intensity at the current moment minus the baseline intensity). This represents baseline fluorescence intensity (background fluorescence value at rest). Animal behavioral video data was also recorded simultaneously. .

[0125] against In order to extract the real neuronal firing patterns from the noisy raw records, this embodiment performs the following key data preprocessing operations: (1) Signal Smoothing Denoising: The extracted relative fluorescence intensity data is smoothed using a smoothing algorithm to reduce the influence of high-frequency measurement noise and improve the signal-to-noise ratio; (2) Feature Amplitude Normalization: The relative fluorescence intensity of each neuron after denoising is normalized (e.g., Z-score normalization) to ensure that the feature vectors of different neurons have a uniform scale distribution, eliminate the influence of individual baseline differences on the subsequent learning weights of the graph network, and thus obtain a standardized neuronal calcium activity time series dataset.

[0126] 2. Behavioral label extraction and strict time alignment. For those from... The behavioral action videos are analyzed to extract the behavioral labels corresponding to the frame-level images (for example, in the elevated cross maze EPM experiment, the behavioral state is classified into categories such as 'Teetering' [shaking / anxiety], 'Rest' [stillness], and 'Exploring' [exploration]), resulting in discrete behavioral label sequences (i.e. behavioral label time series data).

[0127] Subsequently, a strict data alignment procedure is performed. At a defined uniform sampling frequency (e.g., synchronized with the imaging frame rate), continuous time-series data of neuronal calcium activity are... Precise matching of discrete behavioral label sequences with timestamps generates a spatiotemporally strictly aligned structured data table.

[0128] 3. Construction of the neuronal functional network graph. In this step, this embodiment maps the above structured data table into a graph topology (i.e., a weighted graph structure). (1) Node definition: Each observed neuron is defined as a node in the weighted graph. (2) Node characteristics: Each node initial feature vector Defined as its aligned full-time calcium activity time series ,in The length of the calcium activity time series. This makes the node features directly contain the dynamic temporal information of the neurons. (3) Edge and connectivity definition: Edges in a weighted graph Represents the functional connectivity between neurons. Edge weights. Through compute nodes and nodes Calcium activity time series and The Pearson correlation coefficient is defined by the absolute value of the correlation coefficient between the two variables.

[0129] ;

[0130] in, This represents the Pearson correlation coefficient. Neuron In time The intensity of calcium activity, Neuron The average calcium activity intensity over the entire time series, Represents neurons In time The intensity of calcium activity, Represents neurons Average calcium activity intensity over the entire time series.

[0131] It should be noted that in this embodiment, a node represents a neuron, and the edge weights between nodes are obtained using the Pearson correlation coefficient.

[0132] This embodiment uses absolute values ​​to calculate the Pearson correlation coefficient because, in neural circuits, significant positive correlations (synchronous excitation) and significant negative correlations (antagonistic inhibition) both represent shared temporal dynamics and close functional connections between neurons.

[0133] 4. Sparsification and Matrix Output. To filter weakly correlated connections caused by noise and reduce the computational complexity of subsequent graph convolution operations, this embodiment sets a preset threshold. (Threshold). The adjacency relationship of a weighted graph structure is defined as follows:

[0134] ;

[0135] ;

[0136] in, Indicates the first Line 1 Column node characteristics, This represents the total number of neurons (i.e., the number of nodes in the weighted graph structure). Given the length of the time series (i.e., the total number of time steps), the matrix has... Each column corresponds to a sampling time step.

[0137] That is, only when the correlation strength between two neurons reaches or exceeds Only at the node and nodes A weighted edge is established between them; otherwise, they are considered to have no direct functional connection. This stage ultimately outputs two core matrices, which serve as inputs to the next stage of the GNN model:

[0138] Weighted adjacency matrix : Characterizes the spatial topology and functional connectivity strength of a neuronal population.

[0139] Node feature matrix : Represents the dynamic activity patterns of all neurons over time.

[0140] II. Hierarchical Graph Feature Aggregation Network.

[0141] The core innovation of this embodiment lies in constructing a deep hierarchical feature evolution architecture. For example... Figure 3As shown, this architecture is based on the biomimetic cognitive logic of "local topological anchoring—wide-area inductive generalization—key feature refinement." The GNN model organically cascades Graph Convolutional Network (GCN), Graph Sampling Aggregation Network (GraphSAGE), and Graph Attention Network (GAT). This design aims to simulate the complex decoding process of the brain when processing information, from microscopic neurons to macroscopic behavioral decisions, achieving progressive abstraction and enhancement of multi-scale neuronal interaction features. The specific implementation process is as follows:

[0142] 1. Initial Layer: Local Topology Anchoring Based on Graph Convolutional Network (GCN) First, the weighted graph structure constructed in the first stage... The input is fed into the GCN module, which serves as the initial layer of the encoder, to obtain the first output. The main task of this layer is to establish a structured baseline for the features. Utilizing the approximation idea of ​​spectral graph convolution, the GCN layer forces the aggregation of the direct one-hop (1-hop) neighbor features of nodes, thereby effectively capturing the basic topological dependencies of neurons within local micro-loops. Its feature evolution process is shown in the following formula:

[0143] ;

[0144] in, Indicates the first The output result of the layer, i.e., the first layer The feature matrix of the layer, when hour, , This represents a renormalized adjacency matrix with added self-loops, designed to preserve the original feature information of the nodes themselves. yes The degree matrix is ​​used to perform Laplacian normalization on the features to prevent numerical explosion; It is the learnable linear transformation weight matrix of this layer; It is a non-linear activation function (such as the ReLU activation function). Through this step, the model completes the initial embedding of the neuron population topology, laying a robust spatial foundation for subsequent feature propagation.

[0145] 2. Intermediate Layer: Field-of-View Expansion & Inductive Aggregation based on GraphSAGE network; the initial embedding output by the GCN layer. It is passed to the GraphSAGE layer to obtain the second output result. It is the characteristic matrix A specific line in the table. It represents a single node. In the The feature vector of the layer.

[0146] To address the high noise and non-Euclidean nature of neuroscience data, this embodiment utilizes GraphSAGE's inductive sampling mechanism to overcome the limitations of full-graph convolution. This layer samples deeper or broader neighborhood nodes and fuses neighborhood information using an aggregator, effectively expanding the receptive field. This mechanism not only enhances the model's ability to capture long-distance neuronal dependencies but also smooths local noise through mean or max pooling operations, significantly improving the robustness of feature representation. The feature update rule for this layer is shown in the following formula:

[0147] ;

[0148] in, Represents a single node In the The eigenvectors of a layer are the eigenmatrix. The first in OK. Representative node The sampled neighbor set; This indicates an aggregation operation (preferably Mean, Sum, or Max-pooling functions in this embodiment) used to extract statistical features within the neighborhood; and These steps are used to perform weighted transformations on the features themselves and the features aggregated in their neighborhood. This step marks the shift of features from "structured" to "semantic".

[0149] 3. Final Layer: Saliency Attention Refinement based on Graph Attention Network (GAT), and wide-area aggregated feature representation output from the GraphSAGE layer. The data is passed to the GAT layer, resulting in the third output. To address the flawed assumption that "all neighbors are equally important," this layer introduces a multi-head self-attention mechanism. This mechanism simulates the "selective attention" characteristic of biological neural systems, dynamically calculating the influence weights (i.e., attention coefficients) of different neighbors on the central node based on the context of node features. This automatically suppresses irrelevant background noise connections and highlights "core neurons" closely related to specific downstream behaviors. Its neighboring nodes The original attention energy between The calculation is as follows:

[0150] ;

[0151] in, Represents a node The input feature vector of the current layer, Representing neighboring nodes The input feature vector of the current layer, It is the characteristic transformation matrix. It is an attention vector. This represents the vector concatenation operation. Subsequently, the attention energy is normalized using the Softmax function to obtain the final attention weights. :

[0152] ;

[0153] in, Represents a node The neighborhood group, Represents a node and nodes The energy of attention between them.

[0154] Ultimately, the node Higher-order feature representation By using a weighted linear combination of features from all neighbors, semantic-level feature reconstruction is achieved.

[0155] ;

[0156] This hierarchical dynamic weighting mechanism endows the model with strong interpretability, enabling it to accurately identify the key neural circuits that trigger specific behaviors and complete the final mapping from raw signals to high-level behavioral semantics.

[0157] III. Prediction and Classification.

[0158] The third stage of this embodiment is prediction and classification. In this stage, the model inputs the graph information aggregated by a multilayer graph neural network (GNN) (i.e., the third output) into a multilayer perceptron (MLP) classifier. This classifier ultimately outputs a predicted label for the animal's behavioral state. This end-to-end learning paradigm utilizes the most discriminative features from multimodal data to achieve high-precision behavior prediction.

[0159] Complete model architecture (such as) Figure 3 As shown, graph information aggregated through a multilayer graph neural network (GNN) is input into a multilayer perceptron (MLP) classifier. Specifically, this is done through a readout function. The final node features of the last GNN layer are aggregated into a single graph-level representation. Then, it is fed into the MLP classifier:

[0160] ;

[0161] in, Represents a set of nodes. Represents a single node In the The feature vector of the layer (i.e., the third output result).

[0162] The MLP classifier maps this representation to the final predicted behavioral state, using the most discriminative features learned from multimodal data to achieve high-precision behavioral prediction, thus obtaining the behavioral prediction result. for:

[0163] .

[0164] Compared with the prior art, the technical solution of this embodiment has the following advantages:

[0165] 1. A novel, end-to-end graph-based multimodal fusion analysis method is proposed. This embodiment innovatively transforms the multimodal heterogeneous data problem in neuroscience into a graph structure data mining problem. By constructing a unified analysis framework, this embodiment successfully breaks down the barrier between microscopic neuronal activity and macroscopic behavioral labels, providing a new, biologically interpretable computational paradigm for decoding complex neuro-behavioral associations.

[0166] 2. A rigorous heterogeneous data alignment and graph-structured computational process was designed. Addressing the challenge of heterogeneous multimodal data streams, this embodiment employs a rigorous computational pipeline. This pipeline achieves strict temporal alignment between neural signals and behavioral labels. Building upon this, this embodiment calculates the Pearson correlation coefficient between neurons and applies threshold filtering to transform multi-channel time-series signals into a dynamic topological graph structure. This effectively maps "temporal correlation" to "spatial topological connectivity," laying a solid data foundation for subsequent graph deep learning.

[0167] 3. A hierarchical hybrid graph neural network prediction model based on the "deep feature evolution" mechanism is constructed. To overcome the feature extraction bottleneck of single graph neural network models when processing high-noise and dynamic neural data, this embodiment proposes a deep hierarchical feature evolution architecture. This model is not a simple stacking of modules, but rather a three-level interconnected feature processing flow designed based on the biomimetic cognitive logic of "from local to global, from structure to semantics":

[0168] (1) Local Topology Anchoring: The first layer uses a graph convolutional network (GCN) as a structural encoder to forcibly aggregate the direct neighborhood information of neurons and establish the basic topological embedding of neurons in microcircuits, providing a robust structural benchmark for subsequent processing.

[0169] (2) Inductive Field Expansion: The intermediate layer introduces the inductive sampling mechanism of GraphSAGE network, which effectively overcomes the limitations of full graph computation. By expanding the receptive field and aggregating multi-hop neighborhood information, it enhances the model's ability to capture long-distance neuron dependencies and its noise robustness.

[0170] (3) Saliency Attention Refinement: Top-level ensemble graph attention network (GAT) simulates the "selective attention" characteristic of biological nervous systems, dynamically calculates the contribution weights of different neuronal connections, automatically suppresses background noise and highlights "core neurons" that have a decisive influence on specific downstream behaviors.

[0171] This hierarchical design enables the progressive evolution and enhancement of neural signal features from low-order structural information to high-order behavioral semantics, significantly improving the model's decoding accuracy and generalization ability for complex downstream tasks (such as the classification of specific anxiety behaviors in experimental animals on EPM).

[0172] Experimental Validation and Results: Experimental results based on a self-built brain calcium ion fluorescence indicator imaging dataset show that the model proposed in this embodiment achieves an average accuracy of 91.1% in the behavior prediction task, significantly outperforming baseline models using GCN, GraphSAGE, or GAT alone. Furthermore, thanks to its efficient architecture design, this model maintains high accuracy while also demonstrating significant advantages in computational efficiency (training time), achieving an optimal balance between accuracy and efficiency.

[0173] Table 1 shows the experimental results of comparing various models.

[0174]

[0175] 1. The model in this embodiment significantly improves prediction accuracy. On a multimodal dataset of simultaneous recordings of experimental animals' behavior in the elevated cross maze (EPM) and brain calcium ion fluorescence indicator imaging, the hybrid GNN model proposed in this embodiment (i.e., the model in this embodiment) performed excellently, achieving an average accuracy of 91.1% and an F1-score of 0.90. This is significantly better than baseline models used individually, such as Graph Convolutional Network (GCN) (73.2%), GraphSAGE (83.5%), and Graph Attention Network (GAT) (90.8%).

[0176] 2. This embodiment improves computational efficiency. Although this embodiment uses a more complex hybrid architecture, its computational efficiency (training time) is 1.5 seconds / epoch, which is better than GAT (2.5 seconds / epoch) and GraphSAGE (1.8 seconds / epoch), achieving the best balance between accuracy and efficiency.

[0177] 3. The model in this embodiment is robust and converges quickly. Experimental results show that the training loss of the model in this embodiment decreases steadily and rapidly, and the validation accuracy increases rapidly and stabilizes at around 91%, indicating that the model has excellent generalization ability and no obvious overfitting.

[0178] 4. The model in this embodiment has a strong ability to identify biological features: Confusion matrix analysis shows that the model in this embodiment can successfully learn the unique neural features of different behavioral categories (for example, the recall rate of “teetering” behavior associated with high anxiety exceeds 98%), and even for categories with indistinct features and easy confusion (such as “rest”), its performance is far superior to the baseline model.

[0179] Reference Figure 4 This application also provides a behavior analysis system based on multimodal data, which includes a data acquisition unit 401, a label extraction unit 402, a data alignment unit 403, a weighted graph construction unit 404, and a behavior prediction unit 405, wherein:

[0180] The data acquisition unit 401 is used to acquire a standardized time-series dataset of neuronal calcium activity, animal behavior video data, and neuronal calcium ion fluorescence signal values ​​to be predicted.

[0181] The tag extraction unit 402 is used to extract the behavior tags corresponding to each frame of the animal behavior video data and construct a behavior tag sequence.

[0182] Data alignment unit 403 is used to align the neuronal calcium activity time series in the standardized neuronal calcium activity time series dataset with the behavior label sequence to obtain an aligned structured data table. The structured data table includes the aligned behavior labels and the aligned neuronal calcium activity time series.

[0183] Weighted graph construction unit 404 is used to construct a weighted graph structure by using the aligned neuronal calcium activity time series as the features of the nodes.

[0184] The behavior prediction unit 405 is used to train a cascaded graph convolutional network, graph sampling aggregation network, and graph attention network based on the weighted graph structure and aligned behavior labels, so as to obtain a trained graph convolutional network, a trained graph sampling aggregation network, and a trained graph attention network, so as to predict the behavior of the neuron to be predicted by using the trained graph convolutional network, the trained graph sampling aggregation network, and the trained graph attention network, and obtain the behavior prediction result.

[0185] It should be noted that since the behavior analysis system based on multimodal data in this embodiment and the behavior analysis method based on multimodal data described above are based on the same inventive concept, the corresponding content in the method embodiment is also applicable to this system embodiment, and will not be described in detail here.

[0186] Reference Figure 5 This application also provides an electronic device, which includes:

[0187] At least one memory;

[0188] At least one processor;

[0189] At least one program;

[0190] The program is stored in memory, and the processor executes at least one program to implement the behavior analysis method based on multimodal data described above in this disclosure.

[0191] This electronic device can be any smart terminal, including mobile phones, tablets, personal digital assistants (PDAs), and in-vehicle computers.

[0192] The electronic devices according to embodiments of this application will now be described in detail.

[0193] The processor 1600 can be implemented using a general-purpose central processing unit (CPU), microprocessor, application-specific integrated circuit (ASIC), or one or more integrated circuits, and is used to execute relevant programs to implement the technical solutions provided in the embodiments of this disclosure.

[0194] The memory 1700 can be implemented as a read-only memory (ROM), static storage device, dynamic storage device, or random access memory (RAM). The memory 1700 can store the operating system and other applications. When the technical solutions provided in the embodiments of this specification are implemented through software or firmware, the relevant program code is stored in the memory 1700 and is called and executed by the processor 1600 to execute the behavior analysis method based on multimodal data of the embodiments of this disclosure.

[0195] The input / output interface 1800 is used to implement information input and output.

[0196] The communication interface 1900 is used to enable communication and interaction between this device and other devices. Communication can be achieved through wired means (such as USB, Ethernet cable, etc.) or wireless means (such as mobile network, WIFI, Bluetooth, etc.).

[0197] Bus 2000 transmits information between various components of the device (e.g., processor 1600, memory 1700, input / output interface 1800, and communication interface 1900);

[0198] The processor 1600, memory 1700, input / output interface 1800 and communication interface 1900 are connected to each other within the device via bus 2000.

[0199] This disclosure also provides a storage medium, which is a computer-readable storage medium storing computer-executable instructions for causing a computer to execute the above-described behavior analysis method based on multimodal data.

[0200] Memory, as a non-transitory computer-readable storage medium, can be used to store non-transitory software programs and non-transitory computer-executable programs. Furthermore, memory may include high-speed random access memory, and may also include non-transitory memory, such as at least one disk storage device, flash memory device, or other non-transitory solid-state storage device. In some embodiments, memory may optionally include memory remotely located relative to the processor, and these remote memories can be connected to the processor via a network. Examples of such networks include, but are not limited to, the Internet, intranets, local area networks, mobile communication networks, and combinations thereof.

[0201] The embodiments described in this disclosure are for the purpose of more clearly illustrating the technical solutions of this disclosure and do not constitute a limitation on the technical solutions provided by this disclosure. As those skilled in the art will know, with the evolution of technology and the emergence of new application scenarios, the technical solutions provided by this disclosure are also applicable to similar technical problems.

[0202] Those skilled in the art will understand that the technical solutions shown in the figures do not constitute a limitation on the embodiments of this disclosure, and may include more or fewer steps than shown, or combine certain steps, or different steps.

[0203] The device embodiments described above are merely illustrative. The units described as separate components may or may not be physically separate; that is, they may be located in one place or distributed across multiple network units. Some or all of the modules can be selected to achieve the purpose of this embodiment according to actual needs.

[0204] Those skilled in the art will understand that all or some of the steps in the methods disclosed above, as well as the functional modules / units in the systems and devices, can be implemented as software, firmware, hardware, or suitable combinations thereof.

[0205] The terms “first,” “second,” “third,” “fourth,” etc. (if present) in the specification and accompanying drawings of this application are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. It should be understood that such data can be interchanged where appropriate so that the embodiments of this application described herein can be implemented in orders other than those illustrated or described herein. Furthermore, the terms “comprising” and “having,” and any variations thereof, are intended to cover a non-exclusive inclusion; for example, a process, method, system, product, or apparatus that comprises a series of steps or units is not necessarily limited to those steps or units explicitly listed, but may include other steps or units not explicitly listed or inherent to such processes, methods, products, or apparatus.

[0206] It should be understood that in this application, "at least one (item)" means one or more, and "more than" means two or more. "And / or" is used to describe the relationship between related objects, indicating that three relationships can exist. For example, "A and / or B" can represent three cases: only A exists, only B exists, and both A and B exist simultaneously, where A and B can be singular or plural. The character " / " generally indicates that the preceding and following related objects are in an "or" relationship. "At least one (item) of the following" or similar expressions refer to any combination of these items, including any combination of single or plural items. For example, at least one (item) of a, b, or c can represent: a, b, c, "a and b", "a and c", "b and c", or "a and b and c", where a, b, and c can be single or multiple.

[0207] In the several embodiments provided in this application, it should be understood that the disclosed apparatus and methods can be implemented in other ways. For example, the apparatus embodiments described above are merely illustrative; for instance, the division of units is only a logical functional division, and in actual implementation, there may be other division methods. For example, multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed. Furthermore, the coupling or direct coupling or communication connection shown or discussed may be through some interfaces; the indirect coupling or communication connection between apparatuses or units may be electrical, mechanical, or other forms.

[0208] The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the units can be selected to achieve the purpose of this embodiment according to actual needs.

[0209] Furthermore, the functional units in the various embodiments of this application can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The integrated unit can be implemented in hardware or as a software functional unit.

[0210] If the integrated unit is implemented as a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, or all or part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes multiple instructions to cause an electronic device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods of the various embodiments of this application. The aforementioned storage medium includes various media capable of storing programs, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks. The embodiments of this application have been described in detail above with reference to the accompanying drawings, but this application is not limited to the above embodiments. Various changes can be made within the scope of knowledge possessed by those skilled in the art without departing from the spirit of this application.

[0211] The embodiments of this application have been described in detail above with reference to the accompanying drawings. However, this application is not limited to the above embodiments. Within the scope of knowledge possessed by those skilled in the art, various changes can be made without departing from the spirit of this application.

Claims

1. A method of behavior analysis based on multi-modal data, characterized in that, The method comprises: acquiring a standardized neuron calcium activity time series data set, animal behavior video data, and a neuron calcium ion fluorescence signal value to be predicted; extracting a behavior label corresponding to each frame of image in the animal behavior video data to construct a behavior label sequence; aligning the neuron calcium activity time series in the standardized neuron calcium activity time series data set with the behavior label sequence to obtain an aligned structured data table, wherein the structured data table comprises an aligned behavior label and an aligned neuron calcium activity time series; taking the aligned neuron calcium activity time series as a feature of a node to construct a weighted graph structure, comprising: constructing an edge between two adjacent nodes; calculating a correlation coefficient between the two adjacent nodes according to the feature of the node, and taking the correlation coefficient as a weight value of the edge between the two adjacent nodes; constructing a weighted graph structure according to the weight value, the node and the edge; training a cascaded graph convolution network, a graph sampling aggregation network and a graph attention network according to the weighted graph structure and the aligned behavior label to obtain a trained graph convolution network, a trained graph sampling aggregation network and a trained graph attention network, so as to perform behavior prediction on the neuron calcium ion fluorescence signal value to be predicted through the trained graph convolution network, the trained graph sampling aggregation network and the trained graph attention network to obtain a behavior prediction result, comprising: comparing the weight value of the edge in the weighted graph structure with a preset threshold to construct a weighted adjacency matrix; constructing a node feature matrix according to the neuron calcium activity time series corresponding to each node in the weighted graph structure; training a cascaded graph convolution network, a graph sampling aggregation network and a graph attention network according to the weighted adjacency matrix, the node feature matrix and the aligned behavior label to obtain a trained graph convolution network, a trained graph sampling aggregation network and a trained graph attention network, wherein the weighted graph structure is represented by the weighted adjacency matrix and the node feature matrix; inputting the neuron calcium ion fluorescence signal value to be predicted into the trained graph convolution network to obtain a first output result; inputting the first output result into the trained graph sampling aggregation network to obtain a second output result; inputting the second output result into the trained graph attention network to obtain a third output result; performing behavior prediction according to the third output result to obtain a behavior prediction result.

2. The method of claim 1, wherein, The acquisition of the standardized neuron calcium activity time series data set comprises: calculating the calcium activity intensity of each neuron at each time to construct a neuron calcium activity time series data set, wherein the calcium activity intensity is the relative fluorescence intensity of each neuron at each time; performing smoothing and denoising on the calcium activity intensity data in the neuron calcium activity time series data set to obtain a denoised neuron calcium activity time series data set; performing normalization processing on the calcium activity intensity data in the denoised neuron calcium activity time series data set to obtain a standardized neuron calcium activity time series data set. 3.The behavior analysis method based on multi-modal data according to claim 1, characterized in that, The calculating the correlation coefficient between the two adjacent nodes according to the characteristics of the nodes comprises: According to the characteristics of the nodes, the calcium activity intensity of each node at time t is obtained, and the average calcium activity intensity of each node in the entire time sequence is obtained; According to the calcium activity intensity and the average calcium activity intensity, the correlation coefficient between the two adjacent nodes is calculated. 4.The behavior analysis method based on multi-modal data according to claim 1, characterized in that, The behavior prediction according to the third output result comprises: The third output result is aggregated into a single graph-level representation through a readout function; The single graph-level representation is input into a multi-layer perception classifier for behavior prediction to obtain a behavior prediction result.

5. A behavior analysis system based on multi-modal data, characterized in that, The system comprises: A data acquisition unit configured to acquire a standardized neuron calcium activity time sequence dataset, animal behavior video data, and a neuron calcium ion fluorescence signal value to be predicted; A label extraction unit configured to extract a behavior label corresponding to each frame of image in the animal behavior video data to construct a behavior label sequence; A data alignment unit configured to align the neuron calcium activity time sequence in the standardized neuron calcium activity time sequence dataset with the behavior label sequence to obtain an aligned structured data table, wherein the structured data table comprises an aligned behavior label and an aligned neuron calcium activity time sequence; A weighted graph construction unit configured to construct a weighted graph structure by taking the aligned neuron calcium activity time sequence as the characteristics of the nodes, comprising: constructing an edge between two adjacent nodes; calculating a correlation coefficient between the two adjacent nodes according to the characteristics of the nodes, and taking the correlation coefficient as a weight value of the edge between the two adjacent nodes; constructing a weighted graph structure according to the weight value, nodes, and edges; A behavior prediction unit configured to train a cascaded graph convolution network, graph sampling aggregation network, and graph attention network according to the weighted graph structure and the aligned behavior label to obtain a trained graph convolution network, a trained graph sampling aggregation network, and a trained graph attention network, so as to perform behavior prediction on the neuron calcium ion fluorescence signal value to be predicted through the trained graph convolution network, the trained graph sampling aggregation network, and the trained graph attention network to obtain a behavior prediction result, comprising: comparing the weight value of the edge in the weighted graph structure with a preset threshold to construct a weighted adjacency matrix; constructing a node feature matrix according to the neuron calcium activity time sequence corresponding to each node in the weighted graph structure; training a cascaded graph convolution network, graph sampling aggregation network, and graph attention network according to the weighted adjacency matrix, the node feature matrix, and the aligned behavior label to obtain a trained graph convolution network, a trained graph sampling aggregation network, and a trained graph attention network, wherein the weighted graph structure is represented by the weighted adjacency matrix and the node feature matrix; inputting the neuron calcium ion fluorescence signal value to be predicted into the trained graph convolution network to obtain a first output result; inputting the first output result into the trained graph sampling aggregation network to obtain a second output result; inputting the second output result into the trained graph attention network to obtain a third output result; performing behavior prediction according to the third output result to obtain a behavior prediction result.

6. An electronic device, comprising: The method comprises: a control processor and a memory connected to the control processor; the memory stores instructions executable by the control processor, and the instructions are executed by the control processor to enable the control processor to perform the behavior analysis method based on multi-modal data according to any one of claims 1 to 4.

7. A computer readable storage medium characterized in that, The computer readable storage medium stores computer executable instructions for causing a computer to perform the behavior analysis method based on multi-modal data according to any one of claims 1 to 4.

Images