Object behavior analysis method and device, equipment and storage medium

Abstract

The embodiment of the invention provides an object behavior analysis method and device, equipment and a storage medium, and belongs to the technical field of animal behavior analysis. The method comprises the following steps: acquiring depth video data, an original neuron calcium signal sequence and an original heart rate signal sequence of a target object; performing behavior prediction on the target object according to the depth video data to obtain an object behavior sequence; the object behavior sequence represents the behavior mode change of the target object. Preprocessing the object behavior sequence to obtain a target behavior sequence, preprocessing the original neuron calcium signal sequence to obtain a target neuron calcium signal sequence, and preprocessing the original heart rate signal sequence to obtain a target heart rate signal sequence. According to the target behavior sequence, the target neuron calcium signal sequence and the target heart rate signal sequence, behavior analysis is conducted on the target object, and behavior analysis information is obtained. The problem that behavior analysis is carried out only through information expressed by a single object spontaneous behavior, and the analysis result is inaccurate is solved.

Description

Technical Field

[0001] The present application relates to the technical field of animal behavior analysis, and in particular to an object behavior analysis method, device, equipment and storage medium. Background Art

[0002] The functions of the nervous system are expressed through behaviors, and systematic and comprehensive object behavioral analysis is extremely important for revealing brain functions. In addition, behavioral analysis has great application potential in basic research in neuroscience and clinical research on major neuropsychiatric diseases. In related technologies, object behavior analysis is mostly based on the information expressed by a single object's spontaneous behavior. For example, the object can be a mouse, and the behavior analysis is performed based on the information expressed by basic physical quantities such as the movement speed, acceleration, position, and movement distance of the mouse's spontaneous behavior. Based on a single behavior, it is impossible to identify the deeper information contained in the object's spontaneous behavior, and it is impossible to fully perform object behavior analysis, resulting in inaccurate results of object behavior analysis. Therefore, how to accurately perform object behavior analysis has become a technical problem that needs to be solved urgently. Summary of the invention

[0003] The main purpose of the embodiments of the present application is to provide an object behavior analysis method, device, equipment and storage medium, aiming to accurately perform object behavior analysis.

[0004] To achieve the above object, a first aspect of an embodiment of the present application provides an object behavior analysis method, the method comprising:

[0005] Collecting original multimodal data of the target object; wherein the original multimodal data includes depth video data, original neuronal calcium signal sequence and original heart rate signal sequence;

[0006] Predicting the behavior of the target object according to the depth video data to obtain an object behavior sequence; wherein the object behavior sequence represents a change in the behavior pattern of the target object;

[0007] Preprocessing the object behavior sequence to obtain a target behavior sequence;

[0008] Preprocessing the original neuron calcium signal sequence to obtain a target neuron calcium signal sequence;

[0009] Preprocessing the original heart rate signal sequence to obtain a target heart rate signal sequence;

[0010] The target object is subjected to a behavior analysis according to the target behavior sequence, the target neuron calcium signal sequence and the target heart rate signal sequence to obtain behavior analysis information.

[0011] In some embodiments, performing behavior analysis on the target object according to the target behavior sequence, the target neuron calcium signal sequence, and the target heart rate signal sequence to obtain behavior analysis information includes:

[0012] Performing correlation evaluation according to the target behavior sequence, the target neuron calcium signal sequence, and the target heart rate signal sequence to obtain correlation evaluation data;

[0013] A behavior analysis is performed on the target object according to the correlation evaluation data to obtain the behavior analysis information.

[0014] In some embodiments, the depth video data includes at least two frames of depth image data, and the step of performing behavior prediction on the target object according to the depth video data to obtain an object behavior sequence includes:

[0015] Performing behavior prediction on the target object according to each of the depth image data to obtain an object behavior pattern;

[0016] Acquire behavior duration and behavior conversion mode according to a plurality of said object behavior modes;

[0017] The object behavior sequence is constructed according to the object behavior pattern, the behavior duration and the behavior conversion pattern.

[0018] In some embodiments, the step of performing behavior prediction on the target object according to the depth image data to obtain an object behavior pattern includes:

[0019] Performing target detection on the target object according to the depth image data to obtain depth behavior data;

[0020] Extracting features from the deep behavior data to obtain object behavior features;

[0021] Performing dimensionality reduction processing on the object behavior feature based on a preset target dimension to obtain a target behavior feature;

[0022] Behavior pattern recognition is performed according to the target behavior characteristics to obtain the object behavior pattern.

[0023] In some embodiments, performing target detection on the target object according to the depth image data to obtain depth behavior data includes:

[0024] Acquire a preset amount of the depth image data to obtain a depth image data set; wherein the depth image data in the depth image data set is the target image data;

[0025] Obtaining depth values ​​of all pixels in the target image data and performing average calculation to obtain an average depth value;

[0026] Extracting a target depth value from the average depth value;

[0027] Calculate the difference between each depth value of the depth image data and the target depth value to obtain a plurality of target difference values;

[0028] The depth value of the depth image data is set according to the multiple target difference values ​​to obtain the depth behavior data.

[0029] In some embodiments, the preprocessing includes at least one of the following: time alignment processing, outlier detection and correction processing, detrending processing and standardization processing.

[0030] In some embodiments, if the preprocessing is the outlier detection and correction processing, the preprocessing of the original neuron calcium signal sequence to obtain the target neuron calcium signal sequence includes:

[0031] Acquiring data distribution information of the original neuronal calcium signal sequence;

[0032] Filtering a target correction rule from preset candidate correction rules according to the data distribution information;

[0033] The original neuron calcium signal sequence is corrected according to the target correction rule to obtain the target neuron calcium signal sequence.

[0034] To achieve the above-mentioned purpose, a second aspect of an embodiment of the present application provides an object behavior analysis device, the device comprising:

[0035] An acquisition module, used to acquire original multimodal data of a target object; wherein the original multimodal data includes: depth video data, original neuron calcium signal sequence and original heart rate signal sequence;

[0036] A prediction module, configured to predict the behavior of the target object according to the depth video data to obtain an object behavior sequence; wherein the object behavior sequence represents a change in a behavior pattern of the target object;

[0037] A first preprocessing module, used for preprocessing the object behavior sequence to obtain a target behavior sequence;

[0038] A second preprocessing module is used to preprocess the original neuron calcium signal sequence to obtain a target neuron calcium signal sequence;

[0039] A third preprocessing module, used for preprocessing the original heart rate signal sequence to obtain a target heart rate signal sequence;

[0040] The analysis module is used to perform behavior analysis on the target object according to the target behavior sequence, the target neuron calcium signal sequence and the target heart rate signal sequence to obtain object behavior analysis information.

[0041] To achieve the above-mentioned purpose, the third aspect of an embodiment of the present application proposes an electronic device, which includes a memory and a processor, wherein the memory stores a computer program, and when the processor executes the computer program, the method described in the first aspect is implemented.

[0042] To achieve the above-mentioned purpose, the fourth aspect of an embodiment of the present application proposes a computer-readable storage medium, wherein the computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the method described in the first aspect is implemented.

[0043] The object behavior analysis method, device, equipment and storage medium proposed in the present application collect the original multimodal data of the target object; wherein the original multimodal data includes depth video data, original neuronal calcium signal sequence and original heart rate signal sequence. The behavior of the target object is predicted according to the depth video data to obtain the object behavior sequence; wherein the object behavior sequence characterizes the change in the behavior pattern of the target object. The object behavior sequence is preprocessed to obtain the target behavior sequence, the original neuronal calcium signal sequence is preprocessed to obtain the target neuronal calcium signal sequence, and the original heart rate signal sequence is preprocessed to obtain the target heart rate signal sequence. The target object is analyzed for behavior according to the target behavior sequence, the target neuronal calcium signal sequence and the target heart rate signal sequence to obtain behavior analysis information. The problem of inaccurate analysis results caused by only analyzing behavior based on the information expressed by the spontaneous behavior of a single object is solved. BRIEF DESCRIPTION OF THE DRAWINGS

[0044] Figure 1 is a system architecture diagram of the object behavior analysis method provided in an embodiment of the present application;

[0045] Figure 2 is a schematic diagram of an object behavior analysis method provided in an embodiment of the present application;

[0046] Figure 3 yes Figure 1 Flow chart of step S104 in FIG.

[0047] Figure 4 yes Figure 1 Flow chart of step S106 in FIG.

[0048] Figure 5 yes Figure 1 Flow chart of step S102 in FIG.

[0049] Figure 6is a schematic diagram of an object behavior sequence provided in an embodiment of the present application;

[0050] Figure 7 yes Figure 5 Flow chart of step S501 in FIG.

[0051] Figure 8 yes Figure 7 Flow chart of step S701 in FIG.

[0052] Fig. 9 is a schematic diagram of the structure of an object behavior analysis device provided in an embodiment of the present application;

[0053] Fig.10 It is a schematic diagram of the hardware structure of the electronic device provided in the embodiment of the present application. DETAILED DESCRIPTION

[0054] In order to make the purpose, technical solution and advantages of the present application more clearly understood, the present application is further described in detail below in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present application and are not used to limit the present application.

[0055] It should be noted that, although the functional modules are divided in the device schematic diagram and the logical order is shown in the flowchart, in some cases, the steps shown or described may be performed in a different order than the module division in the device or the order in the flowchart. The terms "first", "second", etc. in the specification, claims and the above drawings are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence.

[0056] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as those commonly understood by those skilled in the art to which this application belongs. The terms used herein are only for the purpose of describing the embodiments of this application and are not intended to limit this application.

[0057] In the related art, object behavior analysis is mostly based on the information expressed by a single object's spontaneous behavior. For example, the object can be a mouse, and the behavior analysis is performed based on the information expressed by basic physical quantities such as the movement speed, acceleration, position, and movement distance of the mouse's spontaneous behavior. Based on a single behavior, it is impossible to identify the deeper information contained in the object's spontaneous behavior, and it is impossible to fully perform object behavior analysis, resulting in inaccurate results of object behavior analysis.

[0058] Based on this, the embodiments of the present application provide an object behavior analysis method, apparatus, device and storage medium, which are intended to collect the depth video data, original neuron calcium signal sequence and original heart rate signal sequence of the target object. And based on the depth video data, the behavior of the target object is predicted to obtain the object behavior sequence. The object behavior sequence, the original neuron calcium signal sequence and the original heart rate signal sequence are preprocessed respectively, and the behavior of the target object is analyzed according to the preprocessed target behavior sequence, the target heart rate signal sequence and the target heart rate signal sequence to obtain behavior analysis information. It is achieved by joint analysis of biological signals of multiple modalities, so as to obtain more accurate object behavior analysis results.

[0059] The object behavior analysis method, apparatus, device, and storage medium provided in the embodiments of the present application are specifically described through the following embodiments. First, the object behavior analysis method in the embodiments of the present application is described.

[0060] The embodiments of the present application can acquire and process relevant data based on artificial intelligence technology. Artificial Intelligence (AI) is the theory, method, technology and application system that uses digital computers or machines controlled by digital computers to simulate, extend and expand human intelligence, perceive the environment, acquire knowledge and use knowledge to obtain the best results.

[0061] AI basic technologies generally include sensors, dedicated AI chips, cloud computing, distributed storage, big data processing technology, operation / interaction systems, mechatronics, etc. AI software technologies mainly include computer vision technology, robotics technology, biometrics technology, speech processing technology, natural language processing technology, and machine learning / deep learning.

[0062] The object behavior analysis method provided in the embodiment of the present application can be applied to the terminal, can also be applied to the server side, and can also be software running in the terminal or the server side. In some embodiments, the terminal can be a smart phone, a tablet computer, a laptop computer, a desktop computer, etc.; the server side can be configured as an independent physical server, or a server cluster or distributed system composed of multiple physical servers, and can also be configured as a cloud server that provides basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communications, middleware services, domain name services, security services, CDN, and big data and artificial intelligence platforms; the software can be an application that implements the object behavior analysis method, etc., but is not limited to the above forms.

[0063] The present application can be used in many general or special computer system environments or configurations. For example: personal computers, server computers, handheld or portable devices, tablet devices, multiprocessor systems, microprocessor-based systems, set-top boxes, programmable consumer electronics, network PCs, minicomputers, mainframe computers, distributed computing environments including any of the above systems or devices, etc. The present application can be described in the general context of computer-executable instructions executed by a computer, such as program modules. Generally, program modules include routines, programs, objects, components, data structures, etc. that perform specific tasks or implement specific abstract data types. The present application can also be practiced in distributed computing environments, in which tasks are performed by remote processing devices connected through a communication network. In a distributed computing environment, program modules can be located in local and remote computer storage media including storage devices.

[0064] Please refer to Figure 1 , Figure 1 is an optional flow chart of the object behavior analysis method provided in the embodiment of the present application. Figure 1 The method may include but is not limited to steps S101 to S106.

[0065] Step S101, collecting original multimodal data of the target object; wherein the original multimodal data includes depth video data, original neuron calcium signal sequence and original heart rate signal sequence;

[0066] Step S102, predicting the behavior of the target object according to the depth video data to obtain an object behavior sequence; wherein the object behavior sequence represents a change in the behavior pattern of the target object;

[0067] Step S103, preprocessing the object behavior sequence to obtain a target behavior sequence;

[0068] Step S104, preprocessing the original neuron calcium signal sequence to obtain a target neuron calcium signal sequence;

[0069] Step S105, preprocessing the original heart rate signal sequence to obtain a target heart rate signal sequence;

[0070] Step S106, performing behavior analysis on the target object according to the target behavior sequence, the target neuron calcium signal sequence and the target heart rate signal sequence to obtain behavior analysis information.

[0071] In step S101 of some embodiments, see Figure 2The target object is a mouse. The mouse is placed in a circular open field experimental device, and the depth camera is placed in the center of the open field. The depth camera is used to collect the depth signal of a single mouse in the open field to obtain depth video data. At the same time, the heart rate signal and neuronal calcium signal of the mouse are collected.

[0072] It should be noted that when the range captured by the depth camera is larger than the range of the open field, it is necessary to specify the mine range in the depth video data as the area of ​​interest. The subsequent calculation of the depth video data is only for the area of ​​interest, and the pixel values ​​outside the area of ​​interest are set to 0.

[0073] In one example, the original neuronal calcium signal sequence is synchronously collected by calcium ion imaging technology. Calcium ion imaging technology is a type of optogenetic technology that actually detects changes in calcium ion concentration in neurons. The calcium concentration changes are converted into fluorescent signals, thereby converting cell electrical activity into light signals that can be recorded and observed. The detection method is completed using a calcium ion indicator that can sense calcium ion concentration.

[0074] Specifically, a certain amount of GCaMP virus is injected into a specific brain area of ​​the mouse brain. The virus is specifically expressed in certain types of neurons, such as cck neurons. After a period of injection, cck neurons will express a marker carrying a fluorescent protein on the cell membrane. The marker can emit fluorescence under laser irradiation of a specific wavelength. An optical fiber ceramic ferrule is implanted at the same location where the GCaMP virus is injected to transmit excitation light and collect emission light.

[0075] The excitation light transmitted by the fluorescence excitation system reaches a specific brain area of ​​the mouse brain after passing through the optical fiber ceramic ferrule, thereby stimulating the fluorescence signal. The excited fluorescence signal can be collected by the end of the optical fiber implanted in the mouse brain. The collected fluorescence signal is converted into an electrical signal by the sensor, and then transmitted to the fluorescence collection system through the data acquisition card to obtain the original neuronal calcium signal. Multiple original neuronal calcium signals constitute the original neuronal calcium signal sequence. The original neuronal calcium signal sequence is collected to achieve the purpose of real-time observation of the activity of a group of neurons in the studied brain area. In one example, the original neuronal calcium signal sequence is smoothed by sliding window averaging to remove high-frequency noise in the original neuronal calcium signal sequence.

[0076] In one example, the heart rate of a mouse is synchronously recorded by a heart rate recording device to obtain an original heart rate signal, and multiple original heart rate signals form an original heart rate signal sequence, wherein the sampling frequency is 15 Hz.

[0077] In step S102 of some embodiments, the depth video data includes multiple frames of depth image data, and each frame of the image corresponds to a behavior pattern of the mouse. Among them, the behavior patterns include moving forward, sniffing and moving, raising the head, standing, rushing forward, moving and jumping, etc. The mouse behavior is predicted for each frame of the image to obtain the behavior pattern of each frame of the image, and multiple behavior patterns constitute the object behavior sequence of the mouse. Among them, the object behavior sequence characterizes the change of the behavior pattern of the target object, that is, the object behavior sequence includes the duration of each behavior pattern, the proportion of each behavior pattern and the conversion relationship between the behavior patterns.

[0078] In steps S103 to S105 of some embodiments, since the acquisition mode and data format of the original multimodal data are usually mismatched with each other, in order to perform joint analysis on the original multimodal data, the original multimodal data needs to be preprocessed.

[0079] In an example, the preprocessing includes at least one of the following: time alignment processing, outlier detection and correction processing, detrending processing, and standardization processing.

[0080] Specifically, although the devices for collecting deep video data, raw neuronal calcium signal sequences, and raw heart rate signal sequences start synchronously, the time when valid signals are collected is not synchronous. Therefore, it is necessary to perform time alignment processing on the object behavior sequence, raw neuronal calcium signal sequence, and raw heart rate signal sequence. The actual operation can be to import all the original multimodal data into the multimodal synchronization analysis software, record the trigger time point of each signal through the event channel of each signal, and realize the temporal alignment of the original multimodal data based on the trigger time point. For example, according to the trigger time point of the first behavior pattern, the first raw neuronal calcium signal, and the first raw heart rate signal in the object behavior sequence, the first behavior pattern, the first raw neuronal calcium signal, and the first raw heart rate signal are shifted so that the first behavior pattern, the first raw neuronal calcium signal, and the first raw heart rate signal are aligned in time, and so on.

[0081] Since the sensor will produce a long-term trend due to photobleaching or other environmental reasons during the long-term recording process, and the time series analysis of physiological signals only focuses on short-term random fluctuations, it is necessary to detrend the original neuronal calcium signal sequence and the original heart rate signal sequence to make the signal more stable and easier to analyze. The time series after detrending does not contain the trend component, and only the random fluctuation part is retained. According to the linear or nonlinear type of the data in the sequence, the difference method or polynomial fitting method can be selected for detrending. For example, a second-order polynomial fitting model can be used for baseline correction to remove the long-term trend in the original neuronal calcium signal sequence.

[0082] It should be noted that the original neuron calcium signal sequence and the original heart rate signal sequence need to be subjected to abnormal value detection and correction processing. The following is an explanation of the steps of performing normal value detection and correction processing on the original neuron calcium signal sequence, and step S104 includes but is not limited to steps S301 to S303:

[0083] Step S301, obtaining data distribution information of the original neuron calcium signal sequence;

[0084] Step S302, selecting a target correction rule from preset candidate correction rules according to data distribution information;

[0085] Step S303, correcting the original neuron calcium signal sequence according to the target correction rule to obtain a target neuron calcium signal sequence.

[0086] In step S301 of some embodiments, a QQ graph is drawn according to the original neuronal calcium signal sequence, so that the data distribution information of the original neuronal calcium signal sequence can be obtained. Among them, the QQ graph (Quantile-Quantile Plot, quantile-quantile graph) is a method for comparing two probability distributions. The most common use is to compare the quantile of a set of sample data with the quantile of a known theoretical distribution (such as normal distribution, exponential distribution, etc.) to evaluate whether the sample data is close to the theoretical distribution. In one example, the quantile of the original neuronal calcium signal sequence is compared with the quantile of the normal distribution.

[0087] In steps S302 to S303 of some embodiments, the candidate correction rules include setting a threshold to remove extreme values ​​and using a logarithmic transformation or a square root transformation. If the data distribution information in the QQ graph represents the data tail offset, indicating that there are extreme values ​​or outliers in the signal, then the selected target correction rule is to set a threshold to remove extreme values. If the data distribution information represents that the data are skewed, then the selected target correction rule is to use a logarithmic transformation or a square root transformation to convert the original neuronal calcium signal sequence into a form closer to a normal distribution. In one example, the average value can also be used to fill in the gaps in the original neuronal calcium signal sequence. Thereby, the outlier detection and correction processing of the original neuronal calcium signal sequence is achieved to obtain the target neuronal calcium signal sequence.

[0088] It should be noted that the original heart rate signal sequence can also be processed for outlier detection and correction in the above manner.

[0089] In steps S301 to S303 shown in the embodiment of the present application, abnormal value detection and correction processing are performed on the original neuronal calcium signal sequence to make the behavior analysis information of the target object more accurate.

[0090] In one example, the standardization process includes sampling point alignment. Due to the limitation of the hardware equipment for collecting the original multimodal data, the sampling frequencies are different. Therefore, it is necessary to perform sampling point alignment processing on the object behavior sequence, the original neuronal calcium signal sequence, and the original heart rate signal sequence. The nearest neighbor interpolation algorithm can be used for sampling point alignment. For example, the sampling frequency of the original neuronal calcium signal is 30Hz, and the sampling frequency of the original heart rate signal is 15Hz. In the 10s data, the original neuronal calcium signal has 300 data points, and the original heart rate signal has 150 data points. The length of the original heart rate signal is less than the original neuronal calcium signal. To align the two, it is necessary to perform nearest neighbor interpolation on the original heart rate signal sequence, take the time axis of the original neuronal calcium signal sequence as the common time axis, insert an approximate value between each data point of the original heart rate signal (obtain the approximate value through the nearest neighbor interpolation algorithm), and extend it to the same length as the original neuronal calcium signal. The object behavior sequence is also sampled and aligned in the above manner.

[0091] The standardization process also includes z-value transformation, which is performed on the original neuronal calcium signal sequence and the original heart rate signal sequence. The z-value transformation is specifically the current value minus the mean value, and then divided by the standard deviation. Through the z-transformation, the discrete time domain signal can be converted into a continuous function on the complex plane, which is convenient for analysis and processing.

[0092] In some embodiments, step S106 may also include but is not limited to steps S401 to S402:

[0093] Step S401, performing correlation evaluation according to the target behavior sequence, the target neuron calcium signal sequence and the target heart rate signal sequence to obtain correlation evaluation data;

[0094] Step S402: Performing behavior analysis on the target object according to the correlation evaluation data to obtain behavior analysis information.

[0095] In step S401 to step S402 of this embodiment, the correlation and correlation strength between the target behavior sequence, the target neuron calcium signal sequence and the target heart rate signal sequence are understood through correlation evaluation data. Whether two or more modalities need to be analyzed simultaneously is determined based on the correlation. If the correlation is strong, a joint analysis is required.

[0096] The process of correlation evaluation can be to use a certain behavior pattern in the target behavior sequence as an event, and calculate the correlation between the target neuron calcium signal and the target heart rate signal when the event occurs. All events of this type in an experiment are calculated once to obtain a statistical value of the correlation of the event. The correlation evaluation data includes the correlation statistical values ​​of all events.

[0097] For example, if the correlation statistic of an event is a strong positive correlation, then the behavioral analysis information ultimately obtained through the statistic is that the behavior pattern is related to the heart rate and the calcium signal of a specific neuron, and when the behavior pattern occurs, the activity of the neuron (quantified by the neuron calcium signal) and the periphery (quantified by the heart rate signal) show a specific trend change with a strong correlation. Through this embodiment, a joint analysis process of multiple modal data is realized, and a more accurate and intuitive analysis result is obtained.

[0098] Steps S101 to S106 shown in the embodiment of the present application are performed by collecting the original multimodal data of the target object; wherein the original multimodal data includes depth video data, original neuronal calcium signal sequence and original heart rate signal sequence. The behavior of the target object is predicted based on the depth video data to obtain an object behavior sequence; wherein the object behavior sequence characterizes the change in the behavior pattern of the target object. The object behavior sequence is preprocessed to obtain a target behavior sequence, the original neuronal calcium signal sequence is preprocessed to obtain a target neuronal calcium signal sequence, and the original heart rate signal sequence is preprocessed to obtain a target heart rate signal sequence. The target object is behaviorally analyzed based on the target behavior sequence, the target neuronal calcium signal sequence and the target heart rate signal sequence to obtain behavior analysis information. This solves the problem of inaccurate analysis results caused by only analyzing the behavior based on the information expressed by the spontaneous behavior of a single object.

[0099] See also Figure 5 In some embodiments, the depth video data includes at least two frames of depth image data, and step S102 may include but is not limited to steps S501 to S503:

[0100] Step S501, predicting the behavior of the target object according to each depth image data to obtain the object behavior pattern;

[0101] Step S502, obtaining behavior duration and behavior conversion mode according to multiple object behavior patterns;

[0102] Step S503: construct an object behavior sequence according to the object behavior pattern, behavior duration and behavior conversion pattern.

[0103] In step S501 of some embodiments, behavior prediction is performed on the target object in each depth image data to obtain the target object behavior. In one example, a behavior prediction model can be pre-trained, the depth image data is input into the behavior prediction model, and the object behavior pattern is output.

[0104] In steps S502 to S503 of some embodiments, the behavior duration and behavior conversion mode are obtained according to a plurality of object behavior modes. In one example, if the duration of one frame of image is 42 milliseconds, the object behavior mode corresponding to the first frame of image is forward, the object behavior mode corresponding to the second frame of image is also forward, the object behavior mode corresponding to the third frame of image is sniffing and moving, the object behavior mode corresponding to the fourth frame of image is standing..., then the duration of the forward behavior is 84 milliseconds, the behavior conversion mode is from forward to sniffing and moving, the duration of the sniffing and moving behavior is 42 milliseconds, the behavior conversion mode is from sniffing and moving to standing..., and so on, a complete object behavior sequence can be obtained. That is, an object behavior sequence can be constructed according to the object behavior mode, the behavior duration and the behavior conversion mode.

[0105] See also Figure 6 , Figure 6 The object behavior sequence of three mice. Different colors represent different object behavior patterns, and the length of the strip corresponding to the color represents the duration of the behavior. If red represents sniffing and moving, green represents standing, and blue represents moving forward, then the object behavior sequence of Mouse1 includes the behavior conversion pattern from sniffing and moving to standing, and then from standing to moving forward, etc.

[0106] In steps S501 to S503 shown in this embodiment, after acquiring multiple object behavior patterns, behavior duration and behavior conversion patterns are acquired according to the multiple object behavior patterns, thereby constructing an object behavior sequence, and performing preprocessing based on the object behavior sequence to obtain a target behavior sequence.

[0107] See also Figure 7 In some embodiments, step S501 may include but is not limited to steps S701 to S704:

[0108] Step S701, performing target detection on the target object according to the depth image data to obtain depth behavior data;

[0109] Step S702, extracting features from the deep behavior data to obtain object behavior features;

[0110] Step S703, performing dimensionality reduction processing on the object behavior feature based on a preset target dimension to obtain a target behavior feature;

[0111] Step S704, performing behavior pattern recognition according to the target behavior characteristics to obtain the object behavior pattern.

[0112] In step S701 of some embodiments, a target detection model may be pre-trained, and the depth image data may be input into the target detection model for target detection to obtain depth behavior data, wherein the depth behavior data is data of only the target object after background noise in the depth image data is removed through target detection.

[0113] In steps S702 to S703 of some embodiments, feature extraction is performed on the deep behavior data, and the extracted features include the depth data of the target object and physical quantities such as the speed, height, width, length, and position of the target object. The principal component analysis calculation is performed on the features of the target object, and the conclusion shows that more than 90% of the explained variance can be achieved using features of 14 dimensions. Therefore, the target dimension can be set to 14 dimensions, and the object behavior features can be reduced to 14 dimensions to obtain the target behavior features. Through dimensionality reduction processing, higher data dimensions can be ignored and the computing cost can be reduced.

[0114] In step S704 of some embodiments, a hidden Markov model capable of performing behavior pattern recognition is pre-trained, target behavior features are input into the hidden Markov model for behavior pattern recognition, and the target behavior pattern is output.

[0115] In the steps S701 to S704 shown in this embodiment, after performing target detection, dimensionality reduction processing and behavior pattern recognition on the depth image data, an object behavior pattern is obtained, so that an object behavior sequence is constructed based on multiple object behavior patterns.

[0116] See also Figure 8 In some embodiments, step S701 may include but is not limited to steps S801 to S805:

[0117] Step S801, obtaining a preset amount of depth image data to obtain a depth image data set; wherein the depth image data in the depth image data set is target image data;

[0118] Step S802, obtaining the depth values ​​of all pixels in the target image data and performing average calculation to obtain an average depth value;

[0119] Step S803, extracting a target depth value from the average depth value;

[0120] Step S804, performing difference calculation between each depth value of the depth image data and the target depth value to obtain a plurality of target difference values;

[0121] Step S805 , setting depth values ​​for the depth image data according to the multiple target difference values ​​to obtain depth behavior data.

[0122] In step S801 of some embodiments, the preset number is less than or equal to the total number of frames of the depth video data. In one example, the preset number can be set to 20% of the number of frames. For example, if the total number of frames of the depth video data is 100 frames, the preset number is 20. A preset number of depth image data can be selected from the depth video data according to a preset rule to obtain a depth image data set. Among them, the preset rule can be to select image frames in front, behind or in the middle of the depth video data. For example. When the total number of frames is 100 frames and the preset number is 20, the depth image data of the first 20 frames in the depth video data are used as the depth image data set. Or select depth image data from the depth video data at intervals of a certain number of frames until a preset number of depth image data is selected as the depth image data set. Among them, the depth image data in the depth image data set is the target image data.

[0123] In some embodiments, in step S802 to step S803, the depth values ​​of all pixels in the target image data are averaged to obtain an average depth value. The average depth value of each target image data is obtained. All average depth values ​​are sorted, and the average depth value at the median position is used as the target depth value. In one example, if there are two average depth values ​​at the median position, the average of the two average depth values ​​is taken as the target depth value.

[0124] In steps S804 to S805 of some embodiments, each depth value of the depth image data is subtracted from the target depth value to obtain multiple target difference values. Pixels with target difference values ​​greater than 0 are set to 1, and pixels with target difference values ​​less than or equal to 0 are set to 0 to obtain depth behavior data. It should be noted that the depth behavior data of all depth image data is obtained in the above manner.

[0125] In steps S801 to S805 shown in this embodiment, the average depth value of a preset number of depth image data is obtained, and the target depth value is selected from the average depth value, and the depth value of each pixel in the depth image data is set based on the target depth value to obtain the depth behavior data. Compared with obtaining the depth behavior data through the target detection model, the amount of calculation is smaller, and the target can be effectively extracted, which is simple and highly interpretable.

[0126] See also Fig. 9 The present application also provides an object behavior analysis device, which can implement the above object behavior analysis method, and the device includes:

[0127] The acquisition module 901 is used to acquire the original multimodal data of the target object; wherein the original multimodal data includes: depth video data, original neuron calcium signal sequence and original heart rate signal sequence;

[0128] A prediction module 902 is used to predict the behavior of the target object according to the depth video data to obtain an object behavior sequence; wherein the object behavior sequence represents the change of the behavior pattern of the target object;

[0129] A first preprocessing module 903 is used to preprocess the object behavior sequence to obtain a target behavior sequence;

[0130] A second preprocessing module 904 is used to preprocess the original neuron calcium signal sequence to obtain a target neuron calcium signal sequence;

[0131] The third preprocessing module 905 is used to preprocess the original heart rate signal sequence to obtain a target heart rate signal sequence;

[0132] The analysis module 906 is used to perform behavior analysis on the target object according to the target behavior sequence, the target neuron calcium signal sequence and the target heart rate signal sequence to obtain object behavior analysis information.

[0133] The specific implementation of the object behavior analysis device is basically the same as the specific implementation of the above-mentioned object behavior analysis method, and will not be repeated here.

[0134] The embodiment of the present application also provides an electronic device, the electronic device includes a memory and a processor, the memory stores a computer program, and the processor implements the object behavior analysis method when executing the computer program. The electronic device can be any intelligent terminal including a tablet computer, a car computer, etc.

[0135] See also Fig.10 , Fig.10 The hardware structure of an electronic device of another embodiment is illustrated, and the electronic device includes:

[0136] The processor 1001 may be implemented by a general-purpose CPU (Central Processing Unit), a microprocessor, an application-specific integrated circuit (ASIC), or one or more integrated circuits, and is used to execute relevant programs to implement the technical solutions provided in the embodiments of the present application.

[0137] The memory 1002 can be implemented in the form of a read-only memory (ROM), a static storage device, a dynamic storage device, or a random access memory (RAM). The memory 1002 can store an operating system and other application programs. When the technical solution provided in the embodiment of this specification is implemented by software or firmware, the relevant program code is stored in the memory 1002, and the processor 1001 calls and executes the object behavior analysis method of the embodiment of this application;

[0138] Input / output interface 1003, used to implement information input and output;

[0139] The communication interface 1004 is used to realize the communication interaction between the device and other devices. The communication can be realized through a wired manner (such as USB, network cable, etc.) or a wireless manner (such as mobile network, WI FI, Bluetooth, etc.);

[0140] A bus 1005 , which transmits information between various components of the device (e.g., the processor 1001 , the memory 1002 , the input / output interface 1003 , and the communication interface 1004 );

[0141] The processor 1001 , the memory 1002 , the input / output interface 1003 and the communication interface 1004 are connected to each other in communication within the device via the bus 1005 .

[0142] An embodiment of the present application further provides a computer-readable storage medium, which stores a computer program. When the computer program is executed by a processor, the object behavior analysis method described above is implemented.

[0143] The memory, as a non-transient computer-readable storage medium, can be used to store non-transient software programs and non-transient computer executable programs. In addition, the memory may include a high-speed random access memory, and may also include a non-transient memory, such as at least one disk storage device, a flash memory device, or other non-transient solid-state storage device. In some embodiments, the memory may optionally include a memory remotely disposed relative to the processor, and these remote memories may be connected to the processor via a network. Examples of the above-mentioned network include, but are not limited to, the Internet, an intranet, a local area network, a mobile communication network, and combinations thereof.

[0144] The embodiments described in the embodiments of the present application are intended to more clearly illustrate the technical solutions of the embodiments of the present application and do not constitute a limitation on the technical solutions provided in the embodiments of the present application. Those skilled in the art will appreciate that with the evolution of technology and the emergence of new application scenarios, the technical solutions provided in the embodiments of the present application are also applicable to similar technical problems.

[0145] Those skilled in the art will appreciate that the technical solutions shown in the figures do not constitute a limitation on the embodiments of the present application, and may include more or fewer steps than shown in the figures, or a combination of certain steps, or different steps.

[0146] The device embodiments described above are merely illustrative, and the units described as separate components may or may not be physically separated, that is, they may be located in one place or distributed on multiple network units. Some or all of the modules may be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0147] Those skilled in the art will appreciate that all or some of the steps in the methods disclosed above, and the functional modules / units in the systems and devices may be implemented as software, firmware, hardware, or a suitable combination thereof.

[0148] The terms "first", "second", "third", "fourth", etc. (if any) in the specification of the present application and the above-mentioned drawings are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence. It should be understood that the data used in this way can be interchangeable where appropriate, so that the embodiments of the present application described herein can be implemented in an order other than those illustrated or described herein. In addition, the terms "including" and "having" and any of their variations are intended to cover non-exclusive inclusions, for example, a process, method, system, product or device comprising a series of steps or units is not necessarily limited to those steps or units clearly listed, but may include other steps or units that are not clearly listed or inherent to these processes, methods, products or devices.

[0149] It should be understood that in the present application, "at least one (item)" means one or more, and "plurality" means two or more. "And / or" is used to describe the association relationship of associated objects, indicating that three relationships may exist. For example, "A and / or B" can mean: only A exists, only B exists, and A and B exist at the same time, where A and B can be singular or plural. The character " / " generally indicates that the objects associated before and after are in an "or" relationship. "At least one of the following" or similar expressions refers to any combination of these items, including any combination of single or plural items. For example, at least one of a, b or c can mean: a, b, c, "a and b", "a and c", "b and c", or "a and b and c", where a, b, c can be single or multiple.

[0150] In the several embodiments provided in the present application, it should be understood that the disclosed devices and methods can be implemented in other ways. For example, the device embodiments described above are only schematic. For example, the division of the above units is only a logical function division. There may be other division methods in actual implementation, such as multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. In addition, the mutual coupling or direct coupling or communication connection shown or discussed can be through some interfaces, indirect coupling or communication connection of devices or units, which can be electrical, mechanical or other forms.

[0151] The units described above as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed on multiple network units. Some or all of the units may be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0152] In addition, each functional unit in each embodiment of the present application may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit. The above-mentioned integrated unit may be implemented in the form of hardware or in the form of software functional units.

[0153] If the integrated unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present application is essentially or the part that contributes to the prior art or all or part of the technical solution can be embodied in the form of a software product, and the computer software product is stored in a storage medium, including multiple instructions to enable a computer device (which can be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods of various embodiments of the present application. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (Read-Only Memory, referred to as ROM), random access memory (Random Access Memory, referred to as RAM), disk or optical disk and other media that can store programs.

[0154] The preferred embodiments of the present invention are described above with reference to the accompanying drawings, but the scope of the rights of the present invention is not limited thereto. Any modification, equivalent substitution and improvement made by a person skilled in the art without departing from the scope and essence of the present invention should be within the scope of the rights of the present invention.

Claims

1. A method for analyzing object behavior, characterized in that: The method comprises: Collecting original multimodal data of the target object; wherein the original multimodal data includes depth video data, original neuronal calcium signal sequence and original heart rate signal sequence; Predicting the behavior of the target object according to the depth video data to obtain an object behavior sequence; wherein the object behavior sequence represents a change in the behavior pattern of the target object; Preprocessing the object behavior sequence to obtain a target behavior sequence; Preprocessing the original neuron calcium signal sequence to obtain a target neuron calcium signal sequence; Preprocessing the original heart rate signal sequence to obtain a target heart rate signal sequence; The target object is subjected to a behavior analysis according to the target behavior sequence, the target neuron calcium signal sequence and the target heart rate signal sequence to obtain behavior analysis information.

2. The method according to claim 1, characterized in that The step of performing behavior analysis on the target object according to the target behavior sequence, the target neuron calcium signal sequence and the target heart rate signal sequence to obtain behavior analysis information includes: Performing correlation evaluation according to the target behavior sequence, the target neuron calcium signal sequence, and the target heart rate signal sequence to obtain correlation evaluation data; A behavior analysis is performed on the target object according to the correlation evaluation data to obtain the behavior analysis information.

3. The method according to claim 1, characterized in that The depth video data includes at least two frames of depth image data, and the behavior prediction of the target object according to the depth video data to obtain the object behavior sequence includes: Performing behavior prediction on the target object according to each of the depth image data to obtain an object behavior pattern; Acquire behavior duration and behavior conversion mode according to a plurality of said object behavior modes; The object behavior sequence is constructed according to the object behavior pattern, the behavior duration and the behavior conversion pattern.

4. The method according to claim 3, characterized in that The step of performing behavior prediction on the target object according to the depth image data to obtain an object behavior pattern includes: Performing target detection on the target object according to the depth image data to obtain depth behavior data; Extracting features from the deep behavior data to obtain object behavior features; Performing dimensionality reduction processing on the object behavior feature based on a preset target dimension to obtain a target behavior feature; Behavior pattern recognition is performed according to the target behavior characteristics to obtain the object behavior pattern.

5. The method according to claim 4, characterized in that The performing target detection on the target object according to the depth image data to obtain depth behavior data includes: Acquire a preset amount of the depth image data to obtain a depth image data set; wherein the depth image data in the depth image data set is the target image data; Obtaining depth values ​​of all pixels in the target image data and performing average calculation to obtain an average depth value; Extracting a target depth value from the average depth value; Calculate the difference between each depth value of the depth image data and the target depth value to obtain a plurality of target difference values; The depth value of the depth image data is set according to the multiple target difference values ​​to obtain the depth behavior data.

6. The method according to claim 1, characterized in that The preprocessing includes at least one of the following: time alignment processing, outlier detection and correction processing, detrending processing and standardization processing.

7. The method according to claim 6, characterized in that If the preprocessing is the abnormal value detection and correction processing, the preprocessing of the original neuron calcium signal sequence to obtain the target neuron calcium signal sequence includes: Acquiring data distribution information of the original neuronal calcium signal sequence; Filtering a target correction rule from preset candidate correction rules according to the data distribution information; The original neuron calcium signal sequence is corrected according to the target correction rule to obtain the target neuron calcium signal sequence.

8. An object behavior analysis device, characterized in that: The device comprises: An acquisition module, used to acquire original multimodal data of a target object; wherein the original multimodal data includes: depth video data, original neuron calcium signal sequence and original heart rate signal sequence; A prediction module, configured to predict the behavior of the target object according to the depth video data to obtain an object behavior sequence; wherein the object behavior sequence represents a change in a behavior pattern of the target object; A first preprocessing module, used for preprocessing the object behavior sequence to obtain a target behavior sequence; A second preprocessing module is used to preprocess the original neuron calcium signal sequence to obtain a target neuron calcium signal sequence; A third preprocessing module, used for preprocessing the original heart rate signal sequence to obtain a target heart rate signal sequence; The analysis module is used to perform behavior analysis on the target object according to the target behavior sequence, the target neuron calcium signal sequence and the target heart rate signal sequence to obtain object behavior analysis information.

9. An electronic device, characterized in that: The electronic device includes a memory and a processor, the memory stores a computer program, and the processor implements the object behavior analysis method according to any one of claims 1 to 7 when executing the computer program.

10. A computer-readable storage medium storing a computer program, characterized in that: When the computer program is executed by a processor, the object behavior analysis method according to any one of claims 1 to 7 is implemented.

Images