Animal behavior frame-level time sequence segmentation method fusing skeleton and visual information

Abstract

The invention provides an animal behavior frame-level time sequence segmentation method fusing skeleton and visual information, and relates to the field of image processing, and the method comprises the steps: obtaining an original video of an animal behavior; constructing a refined behavior phenotype model; the refined behavior phenotype model comprises a feature extraction module, an initial time sequence modeling and prediction module, a multi-modal fusion module and a multi-stage refining module; training a refined behavior phenotype model through the original video; acquiring a to-be-analyzed animal behavior video; and inputting a to-be-analyzed animal behavior video into the refined behavior phenotype model to obtain a behavior analysis result. The tool for performing refined behavioral phenotype analysis on the experimental animal provided by the invention can provide data support for related mechanism research of neurological and mental diseases and drug effect evaluation.

Description

Technical Field

[0001] This application relates to the field of image processing, and in particular to a frame-level temporal segmentation method for animal behavior that integrates skeleton and visual information. Background Technology

[0002] Animal behavior is a comprehensive external manifestation of its higher central nervous system functions, psychological state, and motor coordination ability. Quantitative analysis of animal behavior is a core research method in fields such as neuroscience, psychology, and drug development.

[0003] Traditional methods of animal behavior analysis heavily rely on manual recording and scoring by human observers. This process is not only time-consuming, labor-intensive, and costly, but also limited by the observer's subjective judgment and experience, making it difficult to guarantee the objectivity, repeatability, and accuracy of the results. Furthermore, traditional methods typically only quantify some rough behavioral indicators, such as the time an animal spends in a specific area or the number of times it completes a task. This results in a significant loss of behavioral data and fails to capture the dynamic and structural characteristics of behavior.

[0004] With the rise of computational behavioral science, automated behavioral analysis technology has developed, but significant technical bottlenecks still exist: (1) The analysis granularity is coarse, lacking the ability to resolve temporal structure. Most existing automated behavior analysis methods focus on behavior recognition or spatiotemporal behavior detection. Behavior recognition tasks are usually performed on pre-edited short video clips to determine which behavior category the clip belongs to. Spatiotemporal behavior detection tasks locate sparse keyframes or time windows where behaviors occur in unedited long videos. Neither of these paradigms can meet the need for the finest-grained (i.e., frame-by-frame) analysis of continuous behavior streams. They can answer "what behavior happened in a certain time period", but cannot accurately answer "how long did the behavior last", "when did it start and when did it end", and "how did different behaviors transition?" This lack of ability to resolve the temporal structure of behavior greatly limits the exploration of deep dynamic features such as behavior duration, transition patterns, and rhythms.

[0005] (2) Reliance on Single Modal Information, Insufficient Robustness and Information Integrity: Many existing methods rely on a single information source. For example, methods based purely on RGB video, while capturing rich visual appearance information, are highly susceptible to environmental noise such as changes in lighting, cluttered backgrounds, changes in viewing angle, and camera shake. Furthermore, when different behaviors appear similar but have key differences in body posture, RGB information alone is insufficient for effective differentiation. On the other hand, methods based purely on skeleton data analyze behavior by tracking key body points, exhibiting natural robustness to changes in lighting and background. However, skeleton data is essentially a highly abstract representation of animal morphology, resulting in the loss of a significant amount of crucial information, such as subtle limb movements, interactions with objects, facial expressions, and dynamic changes in surface texture. More importantly, the performance ceiling of these methods is limited by the accuracy of the upstream pose estimation algorithm. For animals with dense fur and varied morphologies, pose estimation itself is a highly challenging task, and the noise it introduces directly affects the accuracy of downstream behavior analysis.

[0006] In summary, a clear technological gap exists in the current field of animal behavior analysis: the lack of an effective method for frame-level temporal action segmentation of continuous, unedited animal behavior videos. This task aims to assign a precise behavioral label to each frame of the video, thereby parsing a continuous stream of behavioral videos into an equally continuous stream of semantic labels. Solving this technical challenge can significantly improve the temporal resolution and data richness of behavior analysis, and help researchers analyze the temporal patterns and structural features of behavioral occurrences, providing strong support for revealing the underlying neural mechanisms and conducting quantitative research on disease-related behavioral characteristics. However, no research has yet systematically designed and validated a technical solution specifically for this task that can effectively integrate multimodal information to overcome their respective limitations. Summary of the Invention

[0007] The purpose of this invention is to fill the technical gap in frame-level temporal action segmentation of continuous, unedited animal behavior videos, and to provide a frame-level temporal segmentation method for animal behavior that integrates skeleton and visual information.

[0008] The above-mentioned objective of this application is achieved through the following technical solution: S1: Obtain raw video of animal behavior; S2: Construct a refined behavioral phenotype model; the refined behavioral phenotype model includes: a feature extraction module, an initial temporal modeling and prediction module, a multimodal fusion module, and a multi-stage refinement module; S3: Train a refined behavioral phenotypic model using the original video; S4: Obtain videos of the animal's behavior to be analyzed; input the videos of the animal's behavior to be analyzed into a refined behavioral phenotypic model to obtain the behavioral analysis results.

[0009] Optionally, step S1 includes: a feature extraction module, an initial temporal modeling and prediction module, a multimodal fusion module, and a multi-stage refinement module connected in sequence; The original video contains three complementary data streams: an RGB frame sequence, an optical flow sequence, and a two-dimensional keypoint coordinate sequence. The feature extraction module is used to process the original video to obtain a spatiotemporal feature sequence. Spatiotemporal feature sequences and body kinematic feature sequence ; The initial temporal modeling and prediction module is used to perform preliminary temporal dependency modeling on the three extracted sequences to obtain the initial prediction of the skeleton flow. Initial prediction of visual flow ; The multimodal fusion module is used to fuse the initial high-dimensional features from the initial temporal modeling and prediction modules to obtain fused features. ; The multi-stage refinement module is used to refine the predictions of skeleton flow and visual flow in stages. Its first stage is applied to the initial prediction. and Subsequent stages then apply the refinement results of the previous stage, and after each stage, a dual-stream fusion interaction is performed to ultimately generate the category probability output of the refined behavioral phenotype model.

[0010] Optionally, the feature extraction module includes: The RGB frame sequence of the original video Input a 3D convolutional neural network to obtain a spatiotemporal feature sequence that simultaneously represents spatial appearance, color, texture, and temporal context information. ; Indicates the first Frame image; H, W, and 3 represent the image height, width, and number of channels, respectively; It is the dimension of the feature vector; By using an optical flow estimation algorithm, pixel-level dense displacement fields are calculated from consecutive frames of the original video, generating an optical flow sequence. The optical flow sequence is input into a three-dimensional convolutional neural network to extract a spatiotemporal feature sequence centered on motion. ; Each frame of the original video is processed by a pre-trained pose estimation algorithm model to detect and locate J predefined body key points, resulting in a two-dimensional key point coordinate sequence. A set of higher-order kinematic features is derived from the two-dimensional keypoint coordinate sequence; these features include normalized keypoint coordinates, keypoint velocities, and keypoint accelerations; the higher-order kinematic features are then concatenated to obtain a body kinematic feature sequence. Where J is the number of key points. It is the dimension of the derived features for each key point.

[0011] Optionally, the initial temporal modeling and prediction module includes: a skeleton flow processing network stacked by ST-GCN modules and a visual flow processing network stacked by multiple TCN modules; the ST-GCN module includes: a spatial graph convolutional layer and a temporal convolutional layer; the TCN module includes: two parallel dilated convolutional layers with symmetric dilation factors, batch normalization, ReLU activation function and residual connection mechanism; Body kinematic feature sequences are processed through a skeleton flow processing network composed of stacked ST-GCN modules. Resulting in the frame-level category probability sequence The details are as follows: The spatial graph convolution operation is defined as follows: for a node at time t... Its input features are The output of the graph convolution is:

[0012] in yes The set of neighboring nodes, It is a learnable weighted function, derived from labels. Index, tag Describes a subset of neighbors. It is a normalization term; After spatial aggregation, a diffusion-time convolution is applied to model the temporal dynamics. Finally, a residual connection is added after the input and activation features. This process is formalized as follows:

[0013] in This refers to the output characteristics of a single ST-GCN module. This represents a 2D diffusing convolution operation. It is a bias term. It is the weight matrix of a 2D diffuse convolutional filter with a kernel size of (k,1), representing convolution along the time axis of each keypoint; The entire ST-GCN module processes body kinematic feature sequences. Then, after a spatial pooling layer, the initial high-dimensional features of the skeleton flow are obtained. C is the number of channels for the feature; The initial high-dimensional features are processed through a single 1×1 convolution to generate a frame-level class probability sequence. , It is the number of behavior categories; Frame-level category probability sequences are generated using a visual stream processing network. Specifically, it includes: RGB feature sequence and optical flow characteristic sequence By concatenating along the feature channel dimension, a visual feature sequence is obtained. ; The forward propagation process for each TCN module is as follows:

[0014] in It is the output characteristic of a single TCN module. This represents a 1D diffusion convolution operation. It is a bias term. It is the weight matrix of a 1D diffuse convolutional filter with kernel size k, which is convolved along the time dimension; The entire TCN module processes visual feature sequences. Then, the initial high-dimensional features of the visual flow are obtained. The initial high-dimensional features of the visual stream are processed through a single 1×1 convolution layer to generate a frame-level category probability sequence. , It is the number of behavior categories.

[0015] Optionally, step S4 includes: The multimodal fusion module is used for the initial high-dimensional features of the two branches. and Perform a fusion operation to obtain fusion features. Specifically, it includes: Two initial high-dimensional features and Connect and stitch along the feature channel dimension; to ; The channel dimensions are remapped to C using a 1×1 convolutional layer to generate the final fused features. .

[0016] Optionally, the multi-stage refinement module includes S refinement stages; The initial segmentation prediction includes: initial prediction of the skeleton flow. Initial prediction of visual flow ; The multi-stage refinement module iteratively optimizes the initial segmentation prediction to generate the prediction result for that stage. This process is defined as follows:

[0017] in This is the output of stage s. It is the output of the previous stage. It is an independent TCN network; At the end of each refinement stage, a fusion module is introduced to combine the high-dimensional features generated by the two parallel branches. and By performing fusion, fusion characteristics are obtained. Based on the fusion features, the prediction results for this stage are generated. The prediction results from the final refinement stage are used as the class probability output of the refined behavioral phenotype model.

[0018] Optionally, the refined behavioral phenotype model employs a boundary-aware composite loss function, including: classification loss, smoothing loss, and boundary regression loss; The composite loss function can be expressed as:

[0019]

[0020] in, For classification loss; To smooth out the loss, For boundary regression loss; It is the loss of a single refinement stage. It is the total loss across all S stages. and For loss weights; The specific loss of boundary regression is as follows: Real boundary signals are generated by identifying the start and end frames of each action segment. ; using a one-dimensional Gaussian filter Smoothing is performed to create a smoother target distribution around the boundaries; Derive the prediction boundary signal from the class probabilities frame by frame. This includes: the probability of a boundary existing at time step t is modeled as 1 minus the dot product of the class probability distributions of adjacent frames:

[0021] in It is the probability of predicting behavior category c at time step t; The boundary regression loss is derived by calculating the binary cross-entropy between the predicted boundary signal and the smoothed true boundary signal.

[0022] An electronic device includes a processor, a memory, a user interface, and a network interface. The memory is used to store instructions, the user interface and the network interface are used to communicate with other devices, and the processor is used to execute the instructions stored in the memory to enable the electronic device to perform a frame-level temporal segmentation method for animal behavior that integrates skeletal and visual information.

[0023] A computer-readable storage medium storing instructions that, when executed, perform a frame-level temporal segmentation method for animal behavior that fuses skeleton and visual information.

[0024] The beneficial effects of the technical solution provided in this application are: 1. Achieved high temporal resolution behavior segmentation. This invention overcomes the limitations of existing technologies, which can only perform behavior recognition on pre-edited short video clips or sparse behavior detection in long videos. It enables dense temporal segmentation of continuous, unedited animal behavior video streams frame by frame, parsing the continuous behavior stream into an equally continuous semantic tag stream. This leap from point-like, segmented analysis to continuous parsing solves the problems of coarse-grained analysis and inability to capture the complete temporal structure of behavior in existing technologies, making it possible to accurately quantify deep dynamic features such as behavior duration, occurrence order, transition patterns, and intrinsic rhythms.

[0025] 2. By employing a multimodal fusion strategy, the completeness of behavioral representation and the robustness of analysis are improved. This invention designs a deep fusion architecture that collaboratively utilizes three information modalities from different sources but with complementary functions: skeleton pose, visual appearance (RGB), and dense motion (optical flow). Skeletal flow provides body structure and kinematic information that is insensitive to visual noise such as lighting and background; visual flow captures fine visual cues missing from skeleton information, such as interactions with objects and dynamic texture changes; optical flow explicitly models pixel-level fine motion. This multimodal synergy constructs a more comprehensive and robust behavioral representation than any single modality, overcoming the inherent information bottlenecks and poor anti-interference capabilities of single-modality methods.

[0026] 3. Through a unique network architecture and loss function design, the temporal coherence and boundary accuracy of the segmentation results are effectively guaranteed. This invention introduces two strategies: a multi-stage iterative refinement architecture and a boundary-aware loss function. The multi-stage refinement architecture can systematically correct local errors in the initial prediction and smooth the prediction results, effectively preventing the over-segmentation and result fragmentation problems common in existing technologies. Simultaneously, the specially designed boundary-aware loss function can explicitly guide the model to learn and accurately locate the start and end boundaries of behavioral segments. This effectively solves the problems of ambiguous behavioral boundaries and inconsistent temporal sequence of segmentation results, ensuring high accuracy and usability of the output results. Attached Figure Description

[0027] The present application will be further described below with reference to the accompanying drawings and embodiments. In the accompanying drawings: Figure 1 This is a network structure diagram in an embodiment of this application; Figure 2 This is a schematic diagram of the ST-GCN in the embodiments of this application; Figure 3 This is a schematic diagram of the electronic device structure in the embodiments of this application. Detailed Implementation

[0028] To provide a clearer understanding of the technical features, objectives, and effects of this application, the specific embodiments of this application will now be described in detail with reference to the accompanying drawings.

[0029] The embodiments of this application provide a frame-level temporal segmentation method for animal behavior that integrates skeleton and visual information.

[0030] Please refer to Figure 1 , Figure 1 This is a network structure diagram of a frame-level temporal segmentation method for animal behavior that integrates skeleton and visual information, as described in an embodiment of this application, including: S1: Obtain raw video of animal behavior; S2: Construct a refined behavioral phenotype model; the refined behavioral phenotype model includes: a feature extraction module, an initial temporal modeling and prediction module, a multimodal fusion module, and a multi-stage refinement module; S3: Train a refined behavioral phenotypic model using the original video; S4: Obtain videos of the animal's behavior to be analyzed; input the videos of the animal's behavior to be analyzed into a refined behavioral phenotypic model to obtain the behavioral analysis results.

[0031] As one example, the overall structure of a deep learning network architecture is as follows: Figure 1As shown in Figure 1, the proposed network first extracts skeleton and visual features through a feature extraction module, and then feeds them into a multimodal fusion behavior segmentation network for analysis. The proposed network processes information from three different modalities to build a comprehensive understanding of animal behavior. These modalities are raw visual appearance (RGB), short-term motion cues (optical flow), and dynamic body posture (skeleton data). Each modality is processed by a dedicated feature extractor to generate rich representations suitable for subsequent temporal analysis. Figure 1 shows the network structure. Figure a illustrates the overall architecture of the model, which employs a dual-stream design to process skeleton and visual features (including RGB and optical flow features) in parallel. After initial encoding, the dual-stream features enter n cascaded fusion and optimization stages, ultimately outputting frame-level behavior predictions. Figure b shows the structure of the ST-GCN network, which consists of multiple stacked ST-GCN Blocks used to process skeleton data. Figure c shows the structure of the SS-TCN-2 network, used for initial prediction encoding of the Visual branch. It consists of multiple TCN Blocks, each containing two parallel dilated convolutional layers with increasing 2... i and decreasing by 2 L-i The dilation factor is designed to capture multi-scale temporal dependencies simultaneously. Figure d shows the structure of the SS-TCN-1 network, which consists of dilated convolutional layers with increasing dilation factors for subsequent optimization stages. Figure e shows the structure of the fusion module, which achieves feature dimensionality reduction and information interaction for the skeleton and RGB branches through feature concatenation and 1x1 convolutional layers.

[0032] Specifically, the technical solution of this application has high practical application value, providing an automated analysis tool for animal behavior research and related interdisciplinary studies. This invention can serve as an objective, automated behavioral phenotypic analysis platform, applicable to fields such as neuroscience, psychology, and drug development. It can automatically quantify subtle changes in behavioral patterns induced by disease, drugs, or environmental changes, which are often key indicators for assessing the physiological and psychological states of animals. This provides strong technical support for exploring and identifying behavioral indicators related to specific states in animal models, improving the efficiency, objectivity, and depth of related research.

[0033] Step S1 includes: a feature extraction module, an initial temporal modeling and prediction module, a multimodal fusion module, and a multi-stage refinement module connected in sequence; The original video contains three complementary data streams: an RGB frame sequence, an optical flow sequence, and a two-dimensional keypoint coordinate sequence. The feature extraction module is used to process the original video to obtain a spatiotemporal feature sequence. Spatiotemporal feature sequences and body kinematic feature sequence ; The initial temporal modeling and prediction module is used to perform preliminary temporal dependency modeling on the three extracted sequences to obtain the initial prediction of the skeleton flow. Initial prediction of visual flow ; The multimodal fusion module is used to fuse the initial high-dimensional features from the initial temporal modeling and prediction modules to obtain fused features. ; The multi-stage refinement module is used to refine the predictions of skeleton flow and visual flow in stages. Its first stage is applied to the initial prediction. and Subsequent stages then apply the refinement results of the previous stage, and after each stage, a dual-stream fusion interaction is performed to ultimately generate the category probability output of the refined behavioral phenotype model.

[0034] The feature extraction module includes: The RGB frame sequence of the original video Input a 3D convolutional neural network to obtain a spatiotemporal feature sequence that simultaneously represents spatial appearance, color, texture, and temporal context information. ; Indicates the first Frame image; H, W, and 3 represent the image height, width, and number of channels, respectively; It is the dimension of the feature vector.

[0035] This application provides an embodiment as follows: the three-dimensional convolutional neural network can employ an inflated 3D convolutional network (I3D). This network learns and extracts spatiotemporal feature sequences that can simultaneously represent spatial appearance, color, texture, and temporal context information by performing convolution operations in the temporal and spatial dimensions.

[0036] By using an optical flow estimation algorithm, pixel-level dense displacement fields are calculated from consecutive frames of the original video, generating an optical flow sequence. The optical flow sequence is input into a three-dimensional convolutional neural network to extract a spatiotemporal feature sequence centered on motion. ; This application provides an embodiment as follows: an optical flow estimation algorithm, such as RAFT (Recurrent All-Pairs Field Transforms), is used to process consecutive frames in the original video sequence to calculate pixel-level dense displacement fields and generate an optical flow sequence. Then, the optical flow sequence is input into an I3D network with the same structure as the RGB flow to extract the spatiotemporal feature sequence centered on motion. This step explicitly models the motion information of objects, which helps to distinguish behaviors that are similar in appearance but different in motion characteristics (such as speed and direction).

[0037] Each frame of the original video is processed by a pre-trained pose estimation algorithm model to detect and locate J predefined body key points, resulting in a two-dimensional key point coordinate sequence. A set of higher-order kinematic features is derived from the two-dimensional keypoint coordinate sequence; these features include normalized keypoint coordinates, keypoint velocities, and keypoint accelerations; the higher-order kinematic features are then concatenated to obtain a body kinematic feature sequence. Where J is the number of key points. It is the dimension of the derived features for each key point.

[0038] This application provides an embodiment as follows: each frame of a video is processed by a pre-trained pose estimation algorithm model (e.g., YOLOv8m-Pose) to detect and locate J predefined body key points (e.g., 17 anatomical key points for non-human primates) to obtain a two-dimensional key point coordinate sequence.

[0039] In a preferred embodiment, to provide richer and more discriminative dynamic information than the original coordinates, a set of higher-order kinematic features is derived from the coordinate sequence. This feature set includes: (1) normalized keypoint coordinates: eliminating the influence of changes in the animal's position within the cage; (2) keypoint velocities: obtained by calculating the difference in keypoint positions between adjacent frames (i.e., inter-frame displacement); and (3) keypoint accelerations: obtained by calculating the difference between adjacent velocity vectors. These features are then concatenated to form the final body kinematic feature sequence. Where J is the number of key points. It is the dimension of the derived features for each keypoint. Adding velocity and acceleration information can provide the model with explicit temporal dynamic cues about pose changes, enhancing its ability to discriminate behavior.

[0040] The initial temporal modeling and prediction module includes: a skeleton flow processing network consisting of stacked ST-GCN modules and a visual flow processing network consisting of stacked TCN modules; the ST-GCN module includes: spatial graph convolutional layers and temporal convolutional layers; the TCN module includes: two parallel dilated convolutional layers with symmetric dilation factors, batch normalization, ReLU activation function and residual connection mechanism. Body kinematic feature sequences are processed through a skeleton flow processing network composed of stacked ST-GCN modules. Resulting in the frame-level category probability sequence The details are as follows: The spatial graph convolution operation is defined as follows: for a node at time t... Its input features are The output of the graph convolution is:

[0041] in yes The set of neighboring nodes, It is a learnable weighted function, derived from labels. Index, tag Describes a subset of neighbors. It is a normalization term; In one specific embodiment of this application, the label Representing a subset of neighbors, based on a spatial partitioning strategy, such as... Figure 2 The diagram shows the self, centrifugal, and centripetal groups. Figure a provides a visual overview of the spacetime graph; Figure b illustrates the spatial partitioning strategy. This strategy divides nodes into three subsets based on their distance from the self-selected root node (green): the node itself (blue), the node closest to the root node (red), and the node farthest from the root node (yellow).

[0042] After spatial aggregation, a diffusion-time convolution is applied to model the temporal dynamics. Finally, a residual connection is added after the input and activation features. This process is formalized as follows:

[0043] in This refers to the output characteristics of a single ST-GCN module. This represents a 2D diffusing convolution operation. It is a bias term. It is the weight matrix of a 2D diffuse convolutional filter with a kernel size of (k,1), representing convolution along the time axis of each keypoint; The entire ST-GCN module processes body kinematic feature sequences. Then, after a spatial pooling layer, the initial high-dimensional features of the skeleton flow are obtained. C is the number of channels for the feature; The initial high-dimensional features are processed through a single 1×1 convolution to generate a frame-level class probability sequence. , It is the number of behavior categories; This application provides an embodiment as follows, which uses body skeletal feature sequences. Input a network consisting of multiple stacked Spatio-Temporal Graph Convolutional Network (ST-GCN) modules, such as... Figure 1As shown in (b). In this network, the J keypoints of each frame are considered as nodes of a graph, and the natural connections between body parts (such as bones) are considered as edges of the graph. Each ST-GCN module first passes through a spatial graph convolutional layer to aggregate the information of connected joints in the spatial dimension to model the cooperative motion relationship between body parts; then it passes through a temporal convolutional layer to model the motion trajectory of each keypoint in the temporal dimension. By stacking multiple such modules, the network can learn the complex structured dynamic patterns of skeleton pose in spacetime. Figure 2 is a schematic diagram of ST-GCN. Figure a shows a visual overview of the spatiotemporal graph; Figure b introduces the spatial partitioning strategy. This strategy divides nodes into three subsets based on their distance from the self-selected root node (green): the node itself (blue), the node closest to the root node (red), and the node farthest from the root node (yellow).

[0044] Frame-level category probability sequences are generated using a visual stream processing network. Specifically, it includes: RGB feature sequence and optical flow characteristic sequence By concatenating along the feature channel dimension, a visual feature sequence is obtained. ; In a preferred embodiment, the visual feature sequence is input into a network consisting of multiple stacked Temporal Convolutional Network (TCN) modules. Specifically, a network is employed as follows: Figure 1 (c) shows the SS-TCN-2 network architecture. This network consists of multiple TCN-2 modules, each containing two parallel dilated convolutional layers with symmetric dilation factors, followed by batch normalization (BN) and ReLU activation functions. A residual connection mechanism is introduced into the network, whereby the module input is added to its output after the convolution and activation operations.

[0045] The forward propagation process for each TCN module is as follows:

[0046] in It is the output characteristic of a single TCN module. This represents a 1D diffusion convolution operation. It is a bias term. It is the weight matrix of a 1D diffuse convolutional filter with kernel size k, which is convolved along the time dimension; The entire TCN module processes visual feature sequences. Then, the initial high-dimensional features of the visual flow are obtained. The initial high-dimensional features of the visual stream are processed through a single 1×1 convolution layer to generate a frame-level category probability sequence. , It is the number of behavior categories.

[0047] Step S4 includes: The multimodal fusion module is used for the initial high-dimensional features of the two branches. and Perform a fusion operation to obtain fusion features. Specifically, it includes: Two initial high-dimensional features and Connect and stitch along the feature channel dimension; to ; The channel dimensions are remapped to C using a 1×1 convolutional layer to generate the final fused features. .

[0048] As one embodiment, the fusion operation is as follows: Figure 1 As shown in (e). This operation allows the network to learn linear combinations of spliced ​​features, effectively learning how to best integrate information from different modalities.

[0049] The multi-stage refinement module includes S refinement stages; The initial segmentation prediction includes: initial prediction of the skeleton flow. Initial prediction of visual flow ; The multi-stage refinement module iteratively optimizes the initial segmentation prediction to generate the prediction result for that stage. This process is defined as follows:

[0050] in This is the output of stage s. It is the output of the previous stage. It is an independent TCN network; At the end of each refinement stage, a fusion module is introduced to combine the high-dimensional features generated by the two parallel branches. and By performing fusion, fusion characteristics are obtained. Based on the fusion features, the prediction results for this stage are generated. The prediction results from the final refinement stage are used as the class probability output of the refined behavioral phenotype model.

[0051] As one example, after the initial fusion, the network enters a series of S refinement stages, aiming to progressively optimize the initial segmentation predictions through iteration. The skeleton stream and visual stream each use separate multi-stage refinement branches, but both branches employ the same SS-TCN architecture, such as... Figure 1As shown in (d), the initial prediction of the skeleton flow. Initial prediction of visual flow generated by ST-GCN-based network modules Generated by a TCN-based network module. Each stage Both directly affect the Softmax activation value of the previous stage to generate more accurate predictions.

[0052] To continuously utilize multimodal information during the refinement process, a fusion module is introduced at the end of each stage s to combine the high-dimensional features generated by the two parallel branches. and By fusion The system then generates predictions for this stage based on the fused features. The predictions from the final refinement stage will serve as the final output of the overall framework.

[0053] The refined behavioral phenotype model employs a boundary-aware composite loss function, including: classification loss, smoothing loss, and boundary regression loss; The composite loss function can be expressed as:

[0054]

[0055] in, For classification loss; To smooth out the loss, For boundary regression loss; It is the loss of a single refinement stage. It is the total loss across all S stages. and For loss weights; The specific loss of boundary regression is as follows: Real boundary signals are generated by identifying the start and end frames of each action segment. ; using a one-dimensional Gaussian filter Smoothing is performed to create a smoother target distribution around the boundaries; Derive the prediction boundary signal from the class probabilities frame by frame. This includes: the probability of a boundary existing at time step t is modeled as 1 minus the dot product of the class probability distributions of adjacent frames:

[0056] in It is the probability of predicting behavior category c at time step t; The boundary regression loss is derived by calculating the binary cross-entropy between the predicted boundary signal and the smoothed true boundary signal.

[0057] This application provides an embodiment as follows: In order to effectively train the entire network, the present invention employs a composite loss function and calculates the loss (i.e., deep supervision) at each refinement stage to ensure that the network can learn meaningful representations in the early stages.

[0058] Classification loss: The standard cross-entropy loss function is used to penalize errors in behavior class prediction on each frame. It is defined as follows:

[0059] in It is classification loss. It's a real label. The predicted label represents the predicted score for behavior category c at time t.

[0060] Smoothing Loss: To reduce oversegmentation, a smoothing loss is introduced, which penalizes drastic changes in prediction probabilities between adjacent frames. A truncated mean squared error is used to achieve the smoothing loss.

[0061] This application also discloses an electronic device. (See reference...) Figure 3 , Figure 3 This is a schematic diagram of the structure of an electronic device disclosed in an embodiment of this application. The electronic device 500 may include: at least one processor 501, at least one network interface 504, a user interface 503, a memory 505, and at least one communication bus 502.

[0062] The communication bus 502 is used to enable communication between these components.

[0063] The user interface 503 may include a display screen, and optionally, the user interface 503 may also include a standard wired interface or a wireless interface.

[0064] The network interface 504 may optionally include a standard wired interface or a wireless interface (such as a Wi-Fi interface).

[0065] This application also discloses a computer-readable storage medium storing multiple instructions adapted for loading by a processor to execute the above-described method for frame-level temporal segmentation of animal behavior that integrates skeleton and visual information.

[0066] The above are merely exemplary embodiments of this disclosure and should not be construed as limiting the scope of this disclosure. Any equivalent changes and modifications made in accordance with the teachings of this disclosure shall still fall within the scope of this disclosure.

[0067] This application is intended to cover any variations, uses, or adaptations of this disclosure that follow the general principles of this disclosure and include common knowledge or customary techniques in the art not described in this disclosure. The specification and embodiments are to be considered exemplary only, and the scope and spirit of this disclosure are defined by the claims.

Claims

1. A frame-level temporal segmentation method for animal behavior that integrates skeletal and visual information, characterized in that, The method includes the following steps: S1: Obtain raw video of animal behavior; S2: Construct a refined behavioral phenotype model; the refined behavioral phenotype model includes: a feature extraction module, an initial temporal modeling and prediction module, a multimodal fusion module, and a multi-stage refinement module; S3: Train a refined behavioral phenotypic model using the original video; S4: Obtain videos of the animal's behavior to be analyzed; input the videos of the animal's behavior to be analyzed into a refined behavioral phenotypic model to obtain the behavioral analysis results.

2. The animal behavior frame-level temporal segmentation method fusing skeleton and visual information as described in claim 1, characterized in that, Step S1 includes: a feature extraction module, an initial temporal modeling and prediction module, a multimodal fusion module, and a multi-stage refinement module connected in sequence; The original video contains three complementary data streams: an RGB frame sequence, an optical flow sequence, and a two-dimensional keypoint coordinate sequence. The feature extraction module is used to process the original video to obtain a spatiotemporal feature sequence. Spatiotemporal feature sequences and body kinematic feature sequence ; The initial temporal modeling and prediction module is used to perform preliminary temporal dependency modeling on the three extracted sequences to obtain the initial prediction of the skeleton flow. Initial prediction of visual flow ; The multimodal fusion module is used to fuse the initial high-dimensional features from the initial temporal modeling and prediction modules to obtain fused features. ; The multi-stage refinement module is used to refine the predictions of skeleton flow and visual flow in stages. Its first stage acts on the initial prediction. and Subsequent stages then apply to the refinement results of the previous stage, and after each stage, a dual-stream fusion interaction is performed to ultimately generate the category probability output of the refined behavioral phenotype model.

3. The animal behavior frame-level temporal segmentation method fusing skeleton and visual information as described in claim 2, characterized in that, The feature extraction module includes: The RGB frame sequence of the original video Input a 3D convolutional neural network to obtain a spatiotemporal feature sequence that simultaneously represents spatial appearance, color, texture, and temporal context information. ; Indicates the first Frame image; H, W, and 3 represent the image height, width, and number of channels, respectively; It is the dimension of the feature vector; By using an optical flow estimation algorithm, pixel-level dense displacement fields are calculated from consecutive frames of the original video, generating an optical flow sequence. The optical flow sequence is input into a three-dimensional convolutional neural network to extract spatiotemporal feature sequences centered on motion. ; Each frame of the original video is processed by a pre-trained pose estimation algorithm model to detect and locate J predefined body key points, resulting in a two-dimensional key point coordinate sequence. A set of higher-order kinematic features is derived from the two-dimensional keypoint coordinate sequence; these features include normalized keypoint coordinates, keypoint velocities, and keypoint accelerations; the higher-order kinematic features are then concatenated to obtain a body kinematic feature sequence. Where J is the number of key points. It is the dimension of the derived features for each key point.

4. The animal behavior frame-level temporal segmentation method fusing skeleton and visual information as described in claim 2, characterized in that, The initial temporal modeling and prediction module includes: a skeleton flow processing network consisting of stacked ST-GCN modules and a visual flow processing network consisting of stacked TCN modules; the ST-GCN module includes: spatial graph convolutional layers and temporal convolutional layers; the TCN module includes: two parallel dilated convolutional layers with symmetric dilation factors, batch normalization, ReLU activation function and residual connection mechanism. Body kinematic feature sequences are processed through a skeleton flow processing network composed of stacked ST-GCN modules. Resulting in the frame-level category probability sequence The details are as follows: The spatial graph convolution operation is defined as follows: for a node at time t... Its input features are The output of the graph convolution is: in yes The set of neighboring nodes, It is a learnable weighted function, derived from labels. Index, tag Describes a subset of neighbors. It is a normalization term; After spatial aggregation, a diffusion-time convolution is applied to model the temporal dynamics. Finally, a residual connection is added after the input and activation features. This process is formalized as follows: in This refers to the output characteristics of a single ST-GCN module. This represents a 2D diffusing convolution operation. It is a bias term. It is the weight matrix of a 2D diffuse convolutional filter with a kernel size of (k,1), representing convolution along the time axis of each keypoint; The entire ST-GCN module processes body kinematic feature sequences. Then, after a spatial pooling layer, the initial high-dimensional features of the skeleton flow are obtained. C is the number of channels for the feature; The initial high-dimensional features are processed through a single 1×1 convolution to generate a frame-level class probability sequence. , It is the number of behavior categories; Frame-level category probability sequences are generated using a visual stream processing network. Specifically, it includes: RGB feature sequence and optical flow characteristic sequence By concatenating along the feature channel dimension, a visual feature sequence is obtained. ; The forward propagation process for each TCN module is as follows: in It is the output characteristic of a single TCN module. This represents a 1D diffusion convolution operation. It is a bias term. It is the weight matrix of a 1D diffuse convolutional filter with kernel size k, which is convolved along the time dimension; The entire TCN module processes visual feature sequences. Then, the initial high-dimensional features of the visual flow are obtained. The initial high-dimensional features of the visual stream are processed through a single 1×1 convolution layer to generate a frame-level category probability sequence. , It is the number of behavior categories.

5. The animal behavior frame-level temporal segmentation method fusing skeleton and visual information as described in claim 4, characterized in that, Step S4 includes: The multimodal fusion module is used to process the initial high-dimensional features of the two branches. and Perform a fusion operation to obtain fusion features. Specifically, it includes: Two initial high-dimensional features and Connect and stitch along the feature channel dimension; to ; The channel dimensions are remapped to C using a 1×1 convolutional layer to generate the final fused features. .

6. The animal behavior frame-level temporal segmentation method fusing skeleton and visual information as described in claim 5, characterized in that, The multi-stage refinement module includes S refinement stages; The initial segmentation prediction includes: initial prediction of the skeleton flow. Initial prediction of visual flow ; The multi-stage refinement module iteratively optimizes the initial segmentation prediction to generate the prediction result for that stage. This process is defined as follows: in This is the output of stage s. It is the output of the previous stage. It is an independent TCN network; At the end of each refinement stage, a fusion module is introduced to combine the high-dimensional features generated by the two parallel branches. and By performing fusion, fusion characteristics are obtained. Based on the fusion features, the prediction results for this stage are generated. The prediction results of the final refinement stage are used as the class probability output of the refined behavioral phenotype model.

7. The animal behavior frame-level temporal segmentation method fusing skeleton and visual information as described in claim 6, characterized in that, The refined behavioral phenotype model employs a boundary-aware composite loss function, including: classification loss, smoothing loss, and boundary regression loss; The composite loss function can be expressed as: in, For classification loss; To smooth out the loss, For boundary regression loss; It is the loss of a single refinement stage. It is the total loss across all S stages. and For loss weights; The specific loss of boundary regression is as follows: Real boundary signals are generated by identifying the start and end frames of each action segment. ; using a one-dimensional Gaussian filter Smoothing is performed to create a smoother target distribution around the boundaries; Derive the prediction boundary signal from the class probabilities frame by frame. This includes: the probability of a boundary existing at time step t is modeled as 1 minus the dot product of the class probability distributions of adjacent frames: in It is the probability of predicting behavior category c at time step t; The boundary regression loss is derived by calculating the binary cross-entropy between the predicted boundary signal and the smoothed true boundary signal.

8. An electronic device, characterized in that, The device includes a processor, a memory, a user interface, and a network interface. The memory is used to store instructions, the user interface and the network interface are used to communicate with other devices, and the processor is used to execute the instructions stored in the memory to cause the electronic device to perform the method as described in any one of claims 1-7.

9. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores instructions that, when executed by a computer, perform the method as described in any one of claims 1-7.

Images