A drosophila behavior recognition method based on hyperbolic space visual and language alignment

Abstract

The application provides a fruit fly behavior recognition method based on hyperbolic space vision and language alignment, relates to the technical field of artificial intelligence and behavior recognition, and the application obtains description information and a training sample set of M behavior categories, trains a fruit fly behavior recognition model based on description information and a training sample set of N behavior categories, constructs a hyperbolic semantic reference benchmark according to description information of M-N behavior categories except the N behavior categories, and integrates the benchmark into a semantic space of the fruit fly behavior recognition model to obtain a trained fruit fly behavior recognition model; a fruit fly two-dimensional key point skeleton sequence to be detected is input into the trained fruit fly behavior recognition model, and a target behavior category corresponding to the fruit fly two-dimensional key point skeleton sequence to be detected is output. The model trained by the application can depict the hierarchical structure and nonlinear dynamics of behavior, and can improve the generalization ability to new behavior categories, thereby realizing cross-domain behavior transfer recognition in a zero sample scene.

Description

Technical Field

[0001] This invention relates to the fields of artificial intelligence and behavior recognition technology, and in particular to a method for fruit fly behavior recognition based on hyperbolic space vision and language alignment. Background Technology

[0002] In life science research, fruit flies, as a classic model organism, play a crucial role in revealing gene function, neural mechanisms, and environmental adaptation patterns through behavioral analysis. Especially under special environments such as microgravity, changes in fruit fly behavior patterns provide a key entry point for studying organisms' responses to extreme environments. However, traditional fruit fly behavioral research relies on manual observation and classification, which is not only inefficient but also struggles to capture fine-grained movement characteristics. Therefore, automated behavior recognition technology is urgently needed to meet the demands of high-throughput experiments.

[0003] In existing technologies, one type of method is based on graph neural networks (GCN) to model fruit fly skeleton sequences and achieves behavior classification by capturing the spatial topology and temporal dynamics between key points. For example, spatiotemporal graph convolutional networks are used to extract skeleton motion features, which has achieved a certain recognition accuracy on closed datasets. Another type of method introduces vision-language models (such as CLIP) to enhance semantic understanding by aligning text labels with visual features, and shows cross-modal generalization potential in tasks such as human action recognition.

[0004] However, the aforementioned technologies have significant limitations when dealing with the complex behavior analysis of small organisms such as fruit flies: existing methods mostly rely on Euclidean space for feature representation, making it difficult to characterize the inherent hierarchical structure of fruit fly behavior (such as the fine-grained action hierarchy involving multiple parts such as the head and legs in "grooming") and nonlinear dynamics, resulting in insufficient generalization ability for new behavior categories (such as specific actions in microgravity environments). At the same time, the alignment of visual and linguistic modalities only stays at the label level, lacking deep correlation modeling of anatomical structure and semantic description, and cannot achieve behavior transfer recognition in zero-shot scenarios. This becomes a key technical bottleneck in scenarios with scarce data and open categories, such as space experiments. Summary of the Invention

[0005] This invention provides a fruit fly behavior recognition method based on hyperbolic space vision and language alignment. The trained behavior recognition model can characterize the hierarchical structure and nonlinear dynamics of behavior, and can improve the generalization ability to new behavior categories, thereby realizing cross-domain behavior transfer recognition in zero-shot scenarios.

[0006] To achieve the above objectives, the embodiments of the present invention adopt the following technical solutions:

[0007] Firstly, a method for fruit fly behavior recognition based on hyperbolic spatial visual and linguistic alignment is provided. The method includes: acquiring descriptive information and a training sample set for M behavior categories, where the descriptive information includes global semantic description and fine-grained description at the body part level; the training sample set includes multiple two-dimensional keypoint skeleton sequences of fruit flies corresponding to N behavior categories out of the M behavior categories, where M is greater than N, and M and N are positive integers; where N behavior categories are known behavior categories, and MN behavior categories other than N behavior categories are unseen behavior categories; training a fruit fly behavior recognition model based on the descriptive information of the N behavior categories and the training sample set; constructing a hyperbolic semantic reference benchmark based on the descriptive information of the MN behavior categories other than N behavior categories, and integrating it into the semantic space of the fruit fly behavior recognition model to obtain a trained fruit fly behavior recognition model; acquiring a two-dimensional keypoint skeleton sequence of fruit flies to be detected; inputting the two-dimensional keypoint skeleton sequence of fruit flies to be detected into the trained fruit fly behavior recognition model, and outputting the target behavior category corresponding to the two-dimensional keypoint skeleton sequence of fruit flies to be detected, where the target behavior category is one of the M behavior categories.

[0008] The beneficial effects of this invention are as follows: The method provided by this invention clearly divides "seen behavior categories (N categories)" and "unseen behavior categories (MN categories)," constructing a framework of "training a model based on N categories of samples + constructing a hyperbolic semantic reference benchmark based on MN category descriptive information." It eliminates the need to provide labeled samples for unseen categories, integrating their semantic information into the model's semantic space solely through natural language descriptions. This design fundamentally solves the core pain points of traditional methods, namely "closed categories and reliance on labeling," enabling the trained model to directly identify M behavior categories (including unseen categories). It perfectly adapts to scenarios with scarce data and open categories, such as space experiments, achieving full coverage of behavior recognition in zero-sample scenarios and providing a feasible solution for real-time analysis of fruit fly behavior in orbit.

[0009] In one possible implementation of the first aspect, the fruit fly behavior recognition model includes a feature extraction module, a hyperbolic embedding module, and a semantic alignment module. The feature extraction module is used to determine a temporal-spatial skeleton map comprising multiple sub-regions based on multiple fruit fly two-dimensional keypoint skeleton sequences. These sub-regions include the head, torso, wings, legs, back, and abdomen. The feature extraction module is also used to extract the dynamic motion features of each fruit fly two-dimensional keypoint skeleton sequence through a spatiotemporal graph convolutional network, obtaining first feature information corresponding to each fruit fly two-dimensional keypoint skeleton sequence. The first feature information includes global skeleton features and local features of each sub-region. The hyperbolic embedding module is used to project the first feature information from Euclidean space to hyperbolic space, obtaining second feature information for each fruit fly two-dimensional keypoint skeleton sequence. The feature information includes a global skeleton hyperbolic representation and a local hyperbolic representation of each sub-region. The semantic alignment module is used to convert the description information of M behavior categories into a global semantic vector corresponding to the global semantic description and a part-level semantic vector corresponding to each sub-region of the fine-grained description at the body part level. The semantic alignment module is also used to construct a hyperbolic semantic reference benchmark for the semantic vectors of the MN behavior categories and integrate them into the unified hyperbolic semantic space of the fruit fly behavior recognition model. The semantic alignment module is also used to align the global skeleton hyperbolic representation and the global semantic vector included in the second feature information of the same behavior category with the second feature information of the same behavior category in the unified hyperbolic semantic space, and to align the local hyperbolic representation and the part-level semantic vector of each sub-region included in the second feature information of the same behavior category to obtain the alignment result, and to train the model based on the alignment result.

[0010] The method provided by this invention divides the anatomical structure into six sub-regions through a feature extraction module, and simultaneously captures global and local dynamics using spatiotemporal graph convolution, enabling precise differentiation of fine-grained behaviors such as grooming and eating. The hyperbolic embedding module projects features into hyperbolic space, adapting to the hierarchical structure of the behavior and overcoming the limitations of Euclidean space representation. The semantic alignment module not only achieves visual-linguistic alignment of known categories but also constructs semantic vectors of unseen categories as hyperbolic reference benchmarks, providing semantic anchors for zero-shot recognition. The collaboration of these three modules significantly improves the reliability of recognition in complex scenarios.

[0011] In one possible implementation of the first aspect, before obtaining the descriptive information of the M behavior categories and the training sample set, the method further includes: generating descriptive information corresponding to each of the M behavior categories based on the large language model.

[0012] The method provided by this invention generates descriptive information for M behavioral categories using a large language model. On the one hand, this significantly improves the efficiency of descriptive information generation, avoiding the tedious process of manually writing descriptions for each category, making it particularly suitable for scenarios with a large M. On the other hand, the large language model can automatically generate standardized and complete global semantic descriptions and fine-grained descriptions at the body part level based on the biological characteristics of fruit fly behavior, ensuring semantic consistency and fine-grained coverage of the descriptive information. This lays the foundation for the subsequent semantic alignment module to convert the language descriptions into accurate semantic vectors, thereby improving the accuracy of visual-language modality alignment and indirectly optimizing the model's behavior recognition performance.

[0013] In one possible implementation of the first aspect, the feature extraction module is specifically used for: for each two-dimensional keypoint skeleton sequence of a fruit fly corresponding to N behavioral categories, determining a temporal-spatial skeleton map of each two-dimensional keypoint skeleton sequence of a fruit fly based on the anatomical structure of the fruit fly. The temporal-spatial skeleton map includes multiple sub-regions, each sub-region corresponding to the head, trunk, wings, legs, back, and abdomen, respectively; extracting features from each sub-region using a spatiotemporal graph convolutional network to obtain local feature information corresponding to each sub-region; performing temporal average pooling on the local feature information of each sub-region to aggregate temporal dimension features to obtain the local embedding of each sub-region; and concatenating the local embeddings of each sub-region to obtain the first feature information.

[0014] The two-dimensional keypoint skeleton sequence x of Drosophila is:

[0015] ;

[0016] Where T is the number of frames, V is the number of keypoints per fruit fly individual, and M is the number of fruit fly individuals.

[0017] The method provided by this invention first constructs a temporal-spatial skeleton map containing sub-regions such as the head and torso based on the anatomical structure of the fruit fly using a feature extraction module, ensuring that the motion features of each part can be modeled independently. Then, a spatiotemporal graph convolutional network is used to selectively extract local feature information for each sub-region, capturing the dynamic motion patterns at the part level. Subsequently, temporal average pooling is used to aggregate temporal dimension features, avoiding the randomness of single-frame features and obtaining stable sub-region local embeddings. Finally, the local embeddings are concatenated to form the first feature information containing both global and local information. This process achieves end-to-end optimization of "spatial part segmentation - spatiotemporal feature extraction - temporal dimension aggregation," accurately capturing the local fine-grained dynamics and global motion patterns of fruit fly behavior. This provides high-quality, high-discriminatory feature input for the subsequent hyperbolic embedding module, significantly improving the model's accuracy in recognizing fine-grained behaviors.

[0018] In one possible implementation of the first aspect, the hyperbolic space is a Poincaré sphere model B. nThe hyperbolic embedding module is specifically used to: project the first feature information from Euclidean space to hyperbolic space based on a preset hyperbolic projection layer to obtain the second feature information corresponding to each fruit fly two-dimensional key point skeleton sequence, wherein the second feature information is a hyperbolic representation set used for hierarchical semantic modeling;

[0019] The hyperbolic representation of the set is:

[0020] {z skel ,z part1 ,z part2 ,...,z partP};

[0021] Among them, z skel For the global skeleton hyperbolic representation, z partP Let P be the local hyperbolic representation of the P-th sub-region, where P is the number of sub-regions;

[0022] The preset hyperbolic projection layer z is:

[0023] ;

[0024] Where, z∈B n W z R is a learnable preset linear transformation, where R is the radius of curvature of the hyperbolic space.

[0025] The method provided by this invention defines the hyperbolic space as a Poincaré sphere model and projects the first feature information from Euclidean space to the hyperbolic space through a preset hyperbolic projection layer. On the one hand, the geometric characteristics of the Poincaré sphere model are naturally adapted to hierarchical structure modeling, enabling a more compact representation of the hierarchical relationship between the global skeleton and local sub-regions in a lower dimension, thus solving the feature redundancy problem in Euclidean space. On the other hand, the hyperbolic representation set clearly distinguishes between the global skeleton hyperbolic representation and the local hyperbolic representation of the sub-regions, and the preset hyperbolic projection layer adapts to the distribution pattern of fruit fly behavioral features through learnable linear transformations and radii of curvature, ensuring that the projected hyperbolic features retain the discriminative power of the original features. This design enables the feature representation to possess both hierarchical modeling capabilities and high discriminative power, providing a suitable feature foundation for the subsequent semantic alignment module to achieve accurate alignment in the hyperbolic space, further enhancing the model's ability to characterize complex behaviors.

[0026] In one possible implementation of the first aspect, the semantic alignment module is specifically used for: inputting the global semantic description into the text encoder for the description information of each behavior category among the M behavior categories, and outputting the global semantic vector of each behavior category; inputting the fine-grained description at the body part level into the text encoder according to the corresponding sub-regions, and outputting the part-level semantic vector corresponding to each sub-region; performing norm pruning on the global semantic vector and the part-level semantic vector to constrain the vector norm within a preset radius r to adapt to the geometric characteristics of the hyperbolic space; in the unified hyperbolic semantic space, for N behavior categories, performing global-level alignment between the global skeleton hyperbolic representation contained in the second feature information of the same behavior category and the global semantic vector, and calculating the hyperbolic space normalized cosine similarity of the global-level alignment; performing part-level alignment between the local hyperbolic representation of each sub-region of the same behavior category among the N behavior categories and the part-level semantic vector of the corresponding sub-region, and calculating the hyperbolic space normalized cosine similarity of the part-level alignment of each sub-region respectively; and using the global semantic vector and the part-level semantic vector as hyperbolic semantic reference benchmarks in the hyperbolic semantic space for the MN behavior categories.

[0027] The method provided by this invention first decomposes the behavior description into global and part-level components, generating corresponding semantic vectors that match the hierarchical structure of visual features; then, it adapts the semantic vectors to hyperbolic space constraints through norm pruning to avoid geometric conflicts; it performs alignment at different granularities at the global and part levels for seen categories, calculates hyperbolic similarity, and strengthens fine-grained differentiation; finally, it uses the semantic vectors of unseen categories as hyperbolic reference benchmarks to provide anchor points for zero-shot identification, thereby significantly improving the reliability and accuracy of identifying unseen behavior categories.

[0028] In one possible implementation of the first aspect, when the semantic alignment module trains the model based on the alignment results, the semantic alignment module is specifically used to: calculate a global contrast loss based on the hyperbolic space normalized cosine similarity of global alignment, with the goal of maximizing the global similarity of samples of the same behavior category and minimizing the global similarity of samples of different behavior categories; for each sub-region, calculate a part-level contrast loss based on the hyperbolic space normalized cosine similarity of part-level alignment of each sub-region, with the goal of maximizing the similarity between the local hyperbolic representation of each sub-region in samples of the same behavior category and the corresponding part-level semantic vector, and minimizing the similarity between the local hyperbolic representation of each sub-region in samples of different behavior categories and the corresponding part-level semantic vector; weighted summation of the global contrast loss and the part-level contrast loss of each sub-region to obtain the total loss function; and iteratively training the fruit fly behavior recognition model with the goal of minimizing the total loss function to obtain the trained fruit fly behavior recognition model.

[0029] The method provided in this invention optimizes model training through a hierarchical loss design: First, a global contrastive loss is calculated based on global alignment similarity to ensure the model learns the semantic differences of overall behavior; then, a part-level contrastive loss is calculated for each sub-region to focus on optimizing the association between local part actions and language descriptions; finally, the two types of losses are weighted and summed to obtain the total loss function, achieving collaborative optimization at different granularities of global and local levels. This design avoids the problem of local part features being masked by global features, enabling the model to simultaneously grasp the overall behavioral patterns and the details of local part actions. It can more accurately distinguish semantically similar behaviors but different local actions, significantly improving the model's recognition accuracy and generalization ability for complex behaviors, especially in zero-shot scenarios where the discrimination of unseen behaviors is more reliable.

[0030] In one possible implementation of the first aspect, the method further includes: obtaining a two-dimensional keypoint skeleton sequence of the fruit fly to be detected; inputting the two-dimensional keypoint skeleton sequence of the fruit fly to be detected into a trained fruit fly behavior recognition model, and outputting the behavior category corresponding to the two-dimensional keypoint skeleton sequence of the fruit fly to be detected.

[0031] The method provided by this invention obtains the two-dimensional keypoint skeleton sequence of the fruit fly to be detected, and directly inputs it into a trained model to output the behavior category. Combined with the zero-shot generalization capability already possessed by the model during training, this prediction process does not require additional sample labeling for new behavior categories, enabling automated, zero-shot behavior recognition. This process significantly reduces manual intervention, improves the efficiency of fruit fly behavior analysis, and is particularly suitable for scenarios where data is scarce and rapid acquisition of behavior results is required under experimental conditions such as microgravity, meeting the high-throughput analysis needs of life science experiments.

[0032] In one possible implementation of the first aspect, M behavioral categories include individual behavior, group behavior, and special behavior in microgravity, and N behavioral categories include individual behavior and group behavior. Individual behavior includes crawling, stillness, feeding, and grooming, while group behavior includes courtship, mating, and touching. Special behavior in microgravity includes floating, tumbling, and winged crawling.

[0033] The method provided by this invention explicitly defines M behavioral categories, including individual behaviors (crawling, stillness, etc.), group behaviors (courtship, mating, etc.), and special behaviors under microgravity (floating, tumbling, etc.). Furthermore, the known categories corresponding to the N training samples are individual and group behaviors, while the unseen categories are special behaviors under microgravity. This division allows the model to specifically learn common individual and group behavioral characteristics on the ground during training, clearly defining the training objective. On the other hand, treating special behaviors under microgravity as unseen precisely matches the actual need for scarce microgravity behavioral data in space experiments. It allows for direct evaluation of the model's zero-sample generalization ability for new behavioral categories under microgravity scenarios, solving the technical bottleneck of microgravity behavior recognition in space experiments and providing accurate and reliable technical support for analyzing the behavioral responses of fruit flies to extreme environments in space biology research.

[0034] Secondly, this invention provides a fruit fly behavior recognition system based on hyperbolic spatial vision and language alignment. The system includes: an acquisition unit for acquiring descriptive information of M behavior categories and a training sample set; the descriptive information includes global semantic description and fine-grained description at the body part level; the training sample set includes multiple two-dimensional keypoint skeleton sequences of fruit flies corresponding to N behavior categories out of the M behavior categories, where M is greater than N, and M and N are positive integers; wherein the N behavior categories are known behavior categories, and the remaining MN behavior categories are unseen behavior categories; and a training unit for training based on the N behavior categories... The descriptive information and training sample set are used to train the fruit fly behavior recognition model. A hyperbolic semantic reference benchmark is constructed based on the descriptive information of MN behavior categories other than N behavior categories, and integrated into the semantic space of the fruit fly behavior recognition model to obtain the trained fruit fly behavior recognition model. The acquisition unit is also used to acquire the two-dimensional keypoint skeleton sequence of the fruit fly to be detected. The detection unit is used to input the two-dimensional keypoint skeleton sequence of the fruit fly to be detected into the trained fruit fly behavior recognition model and output the target behavior category corresponding to the two-dimensional keypoint skeleton sequence of the fruit fly to be detected. The target behavior category is one of the M behavior categories.

[0035] Thirdly, an electronic device is provided, the electronic device including a memory and one or more processors; the memory is coupled to the processors; wherein the memory stores computer program code, the computer program code including computer instructions, which, when executed by the processor, cause the electronic device to perform the method as described in any implementation of the first aspect.

[0036] Fourthly, a computer-readable storage medium is provided, including computer instructions that, when executed on an electronic device, cause the electronic device to perform a method as described in any implementation of the first aspect.

[0037] Fifthly, a computer program product is provided that, when run on a computer, causes the computer to perform the method in any implementation of the first aspect.

[0038] Understandably, the beneficial effects achieved by the system of the second aspect, the electronic device of the third aspect, the computer-readable storage medium of the fourth aspect, and the computer program product of the fifth aspect provided above can be referred to with reference to the beneficial effects of the first aspect and any of its possible design embodiments, which will not be repeated here. Attached Figure Description

[0039] Figure 1 This is a schematic diagram of the structure of an electronic device provided in an embodiment of the present invention;

[0040] Figure 2 A flowchart illustrating a fruit fly behavior recognition method based on hyperbolic spatial vision and language alignment, provided in an embodiment of the present invention;

[0041] Figure 3 This is a schematic diagram of another fruit fly behavior recognition model provided in an embodiment of the present invention;

[0042] Figure 4 A schematic diagram illustrating the process of a fruit fly behavior recognition method based on hyperbolic spatial vision and language alignment provided in an embodiment of the present invention;

[0043] Figure 5 This is a schematic diagram of the structure of an identification system provided in an embodiment of the present invention. Detailed Implementation

[0044] The technical solutions of the embodiments of the present invention will be described below with reference to the accompanying drawings. In the description of the present invention, unless otherwise stated, " / " indicates that the objects before and after are in an "or" relationship. For example, A / B can represent A or B. The "or" in the present invention is merely a description of the relationship between the related objects, indicating that three relationships can exist. For example, A or B can represent: A alone, A and B simultaneously, and B alone. A and B can be singular or plural. Furthermore, in the description of the present invention, unless otherwise stated, "multiple" refers to two or more. "At least one of the following" or similar expressions refer to any combination of these items, including any combination of single or plural items.

[0045] Furthermore, to facilitate a clear description of the technical solutions of the embodiments of the present invention, the terms "first" and "second" are used in the embodiments of the present invention to distinguish identical or similar items with essentially the same function and effect. Those skilled in the art will understand that the terms "first" and "second" do not limit the quantity or execution order, and that the terms "first" and "second" are not necessarily different.

[0046] In this embodiment of the invention, the terms "exemplary" or "for example" are used to indicate that something is an example, illustration, or description. Any embodiment or design described as "exemplary" or "for example" in this embodiment of the invention should not be construed as superior or more advantageous than other embodiments or designs. Specifically, the use of terms such as "exemplary" or "for example" is intended to present the relevant concepts in a concrete manner for ease of understanding.

[0047] In life science research, fruit flies, as a classic model organism, play a crucial role in revealing gene function, neural mechanisms, and environmental adaptation patterns through behavioral analysis. Especially under special environments such as microgravity, changes in fruit fly behavior patterns provide a key entry point for studying organisms' responses to extreme environments. However, traditional fruit fly behavioral research relies on manual observation and classification, which is not only inefficient but also struggles to capture fine-grained movement characteristics. Therefore, automated behavior recognition technology is urgently needed to meet the demands of high-throughput experiments.

[0048] In existing technologies, one type of method is based on graph neural networks (GCN) to model fruit fly skeleton sequences and achieves behavior classification by capturing the spatial topology and temporal dynamics between key points. For example, spatiotemporal graph convolutional networks are used to extract skeleton motion features, which has achieved a certain recognition accuracy on closed datasets. Another type of method introduces vision-language models (such as CLIP) to enhance semantic understanding by aligning text labels with visual features, and shows cross-modal generalization potential in tasks such as human action recognition.

[0049] However, the aforementioned technologies have significant limitations when dealing with the complex behavior analysis of small organisms such as fruit flies: existing methods mostly rely on Euclidean space for feature representation, making it difficult to characterize the inherent hierarchical structure of fruit fly behavior (such as the fine-grained action hierarchy involving multiple parts such as the head and legs in "grooming") and nonlinear dynamics, resulting in insufficient generalization ability for new behavior categories (such as specific actions in microgravity environments). At the same time, the alignment of visual and linguistic modalities only stays at the label level, lacking deep correlation modeling of anatomical structure and semantic description, and cannot achieve behavior transfer recognition in zero-shot scenarios. This becomes a key technical bottleneck in scenarios with scarce data and open categories, such as space experiments.

[0050] In view of this, embodiments of the present invention provide a fruit fly behavior recognition method based on hyperbolic space visual and linguistic alignment. The method includes: acquiring descriptive information of M behavior categories and a training sample set, wherein the descriptive information includes global semantic description and fine-grained description at the body part level, and the training sample set includes multiple two-dimensional keypoint skeleton sequences of fruit flies corresponding to N behavior categories among the M behavior categories, wherein M is greater than N, and M and N are positive integers, wherein the N behavior categories are known behavior categories, and the MN behavior categories other than the N behavior categories are unseen behavior categories; training a fruit fly behavior recognition model based on the descriptive information of the N behavior categories and the training sample set, and constructing a hyperbolic semantic reference benchmark based on the descriptive information of the MN behavior categories other than the N behavior categories, and integrating it into the semantic space of the fruit fly behavior recognition model to obtain a trained fruit fly behavior recognition model; acquiring a two-dimensional keypoint skeleton sequence of fruit flies to be detected; inputting the two-dimensional keypoint skeleton sequence of fruit flies to be detected into the trained fruit fly behavior recognition model, and outputting the target behavior category corresponding to the two-dimensional keypoint skeleton sequence of fruit flies to be detected, wherein the target behavior category is one of the M behavior categories.

[0051] The method provided by this invention clearly distinguishes between "seen behavior categories (N categories)" and "unseen behavior categories (MN categories)," constructing a framework of "model training based on N categories of samples + hyperbolic semantic reference benchmark construction based on MN category descriptive information." This eliminates the need for labeled samples for unseen categories, integrating their semantic information into the model's semantic space solely through natural language descriptions. This design fundamentally solves the core pain points of traditional methods—"closed categories and reliance on labeling"—enabling the trained model to directly identify M behavior categories (including unseen categories). It perfectly adapts to scenarios with scarce data and open categories, such as space experiments, achieving full coverage of behavior recognition in zero-sample scenarios and providing a feasible solution for real-time analysis of fruit fly behavior in orbit.

[0052] In some embodiments, the fruit fly behavior recognition method based on hyperbolic space vision and language alignment provided in this invention can be executed by a fruit fly behavior recognition system 100 based on hyperbolic space vision and language alignment (hereinafter referred to as recognition system 100).

[0053] As an example, the identification system 100 can be any electronic device 200 with data processing capabilities, such as a general-purpose computer, personal computer, laptop computer, switch, or tablet computer. The specific implementation of the identification system 100 is not limited here.

[0054] Figure 1 A schematic diagram of the hardware structure of an electronic device provided in an embodiment of the present invention is shown. The electronic device 200 includes a processor 210, a memory 220, and a communication interface 230.

[0055] Processor 210 may include one or more processing cores. Processor 210 connects to various parts within electronic device 200 using various interfaces and lines, and performs various functions and processes data of electronic device 200 by running or executing instructions, programs, code sets, or instruction sets stored in memory 220, and by calling data stored in memory 220. Optionally, processor 210 may be implemented using at least one of the following hardware forms: Central Processing Unit (CPU), Graphics Processing Unit (GPU), Digital Signal Processing (DSP), Field-Programmable Gate Array (FPGA), and Programmable Logic Array (PLA).

[0056] The memory 220 may include random access memory (RAI) or read-only memory (ROI). Optionally, the memory 220 may include non-transitory computer-readable storage ledger. The memory 220 may be used to store instructions, programs, code, code sets, or instruction sets. The memory 220 may include a stored program area. The stored program area may store instructions for implementing an operating system, instructions for implementing at least one function (such as data acquisition, model training, and behavior recognition), and instructions for implementing the various method embodiments described above.

[0057] Communication interface 230 is used to communicate with other devices, equipment or communication networks, such as data storage devices, image processing devices or Ethernet, wireless access network (RAN), wireless local area network (WLAN), etc.

[0058] In terms of physical implementation, the aforementioned devices (such as processor 210, memory 220, and communication interface 230) can each be devices within the same device (such as a laptop computer). Alternatively, at least two of these devices can be located within the same device, i.e., as different devices within the same device, similar to the deployment of devices or components in a distributed system.

[0059] It is understood that the structure illustrated in this embodiment does not constitute a specific limitation on the electronic device 200. In other embodiments of the present invention, the electronic device 200 may include more or fewer components than illustrated, or combine some components, or split some components, or have different component arrangements. The illustrated components may be implemented in hardware, software, or a combination of software and hardware.

[0060] The following description, in conjunction with the accompanying drawings, illustrates a fruit fly behavior recognition method based on hyperbolic spatial vision and language alignment provided by an embodiment of the present invention.

[0061] Figure 2 This is a flowchart illustrating a fruit fly behavior recognition method based on hyperbolic space vision and language alignment, provided as an embodiment of the present invention. Optionally, this method can be... Figure 1 The illustrated electronic device 200 performs this operation. The method may include the following steps:

[0062] S1. Obtain description information and training sample set for M behavior categories. The description information includes global semantic description and fine-grained description at the body part level.

[0063] The training sample set includes multiple two-dimensional keypoint skeleton sequences of fruit flies corresponding to N behavior categories out of M behavior categories, where M is greater than N and M and N are positive integers. Among them, N behavior categories are seen behavior categories, and MN behavior categories other than N behavior categories are unseen behavior categories.

[0064] In one possible implementation, there are M behavior categories including individual behavior, group behavior, and special behaviors in microgravity, and N behavior categories including individual behavior and group behavior. Individual behavior includes crawling, remaining still, feeding, and grooming, while group behavior includes courtship, mating, and touching. Special behaviors in microgravity include floating, tumbling, and wing-spreading crawling.

[0065] Based on the above examples, individual behavior and group behavior are observed behavior categories, while special behaviors in microgravity environments are unobserved behavior categories.

[0066] Specifically, crawling is defined as the movement of fruit flies along a solid surface by alternating the movement of their forelegs, midlegs, and hindlegs, exhibiting continuous or intermittent horizontal movement. Stillness is defined as the fruit fly maintaining a static posture with all legs on the ground and no obvious body or displacement movement. Feeding is defined as the fruit fly using its mouthparts to contact and absorb nutrients, often accompanied by fixed forelegs and slight head movements. Grooming is defined as the fruit fly using its forelegs, midlegs, or hind legs to clean its body parts (such as the head, wings, or abdomen), exhibiting repetitive rubbing or wiping motions. Courtship is defined as male fruit flies attempting to attract females through following, wing flapping, and other actions, accompanied by directional following and orientational body postures. Mating is defined as the male fruit fly mounting the female's back, maintaining a stable mating posture, and completing the mating behavior. Touching is defined as physical contact between two or more individual fruit flies, which may be exploratory, pushing, or aimless physical contact. Floating is defined as the fruit fly floating in the air in a microgravity environment without being attached to a surface, accompanied by very small spontaneous movements. Tumbling is defined as the fruit fly's body rotating continuously or discontinuously around its own axis, possibly triggered by microgravity disturbances. Winged crawling is defined as the fruit fly walking on the ground while keeping its wings partially or fully extended; it is usually an atypical form of movement influenced by changes in gravity.

[0067] It should be noted that the above behavior categories are merely illustrative examples, and the embodiments of the present invention do not impose any particular limitation on the specific types of behavior categories.

[0068] The method provided by this invention explicitly defines M behavioral categories, including individual behaviors (crawling, stillness, etc.), group behaviors (courtship, mating, etc.), and special behaviors under microgravity (floating, tumbling, etc.). Furthermore, the known categories corresponding to the N training samples are individual and group behaviors, while the unseen categories are special behaviors under microgravity. This division allows the model to specifically learn common individual and group behavioral characteristics on the ground during training, clearly defining the training objective. On the other hand, treating special behaviors under microgravity as unseen precisely matches the actual need for scarce microgravity behavioral data in space experiments. It allows for direct evaluation of the model's zero-sample generalization ability for new behavioral categories under microgravity scenarios, solving the technical bottleneck of microgravity behavior recognition in space experiments and providing accurate and reliable technical support for analyzing the behavioral responses of fruit flies to extreme environments in space biology research.

[0069] In some embodiments, prior to S1 above, the method provided by the present invention further includes: generating descriptive information corresponding to each of the M behavior categories based on a large language model.

[0070] Furthermore, for each type of behavior, this invention employs a large language model (such as ChatGPT4o) to automatically generate two types of linguistic description information. Firstly, a global semantic description of the behavior is used to summarize the overall action intent. Secondly, a fine-grained description with joint / part awareness is used to precisely describe the action details of different body parts.

[0071] For example, taking courtship behavior as an example, the generated global description is that during courtship in fruit flies, the male chases the female by vibrating his wings. The part-level descriptions include: Head: Facing the female, slightly nodding. Back: Slightly swaying from side to side to attract the mate. Abdomen: Rhythmically swaying. Tail: Remaining stable. Wings: Fully extended on one side and vibrating continuously. Legs: Raising the legs in an attracting motion.

[0072] The method provided by this invention generates descriptive information for M behavioral categories using a large language model. On the one hand, this significantly improves the efficiency of descriptive information generation, avoiding the tedious process of manually writing descriptions for each category, making it particularly suitable for scenarios with a large M. On the other hand, the large language model can automatically generate standardized and complete global semantic descriptions and fine-grained descriptions at the body part level based on the biological characteristics of fruit fly behavior, ensuring semantic consistency and fine-grained coverage of the descriptive information. This lays the foundation for the subsequent semantic alignment module to convert the language descriptions into accurate semantic vectors, thereby improving the accuracy of visual-language modality alignment and indirectly optimizing the model's behavior recognition performance.

[0073] S2. The fruit fly behavior recognition model is trained based on the description information of N behavior categories and the training sample set. A hyperbolic semantic reference benchmark is constructed based on the description information of X behavior categories other than N behavior categories, and integrated into the semantic space of the fruit fly behavior recognition model to obtain the trained fruit fly behavior recognition model.

[0074] Where X = MN.

[0075] Specifically, a hyperbolic semantic reference benchmark is constructed based on the descriptive information of MN behavior categories in addition to the N behavior categories. This can also be understood as follows: first, a text encoder converts these descriptions into global semantic vectors and corresponding sub-region-level semantic vectors, respectively. Then, norm pruning is performed on these vectors to ensure that the vector norms conform to the geometric constraints of the hyperbolic space. Finally, these processed vectors are used as the hyperbolic semantic reference benchmark and embedded into the unified hyperbolic semantic space of the fruit fly behavior recognition model, forming a complete semantic network together with the semantic vectors of the seen classes.

[0076] In this way, the fruit fly behavior recognition model can not only identify N seen behavior categories, but also MN unseen behavior categories. In summary, the constructed hyperbolic semantic reference benchmark is equivalent to setting clear coordinates for each unseen category in the semantic space. During the training phase, the model has learned the correspondence between "visual skeleton features (such as wing movements and limb postures) and semantic descriptions" through samples of seen behavior categories. During inference, it is only necessary to perform similarity matching between the hyperbolic skeleton features of the fruit fly to be detected and the semantic benchmarks of these unseen behavior categories. Based on the mastered correspondence rules, it can determine whether the behavior to be detected conforms to the semantic features of a certain unseen behavior category, and can complete the recognition without labeled samples of X types of behavior.

[0077] In some embodiments, see Figure 3 The fruit fly behavior recognition model 300 includes a feature extraction module 310, a hyperbolic embedding module 320, and a semantic alignment module 330. The feature extraction module 310 is used to determine a temporal-spatial skeleton map including multiple sub-regions based on multiple fruit fly two-dimensional keypoint skeleton sequences. The multiple sub-regions include the head, trunk, wings, legs, back, and abdomen. The feature extraction module 310 is also used to extract the dynamic motion features of each fruit fly two-dimensional keypoint skeleton sequence through a spatiotemporal graph convolutional network to obtain the first feature information corresponding to each fruit fly two-dimensional keypoint skeleton sequence. The first feature information includes global skeleton features and local features of each sub-region.

[0078] In one possible implementation, the feature extraction module 310 is specifically used for: for each Drosophila two-dimensional keypoint skeleton sequence, determining a temporal-spatial skeleton map of each Drosophila two-dimensional keypoint skeleton sequence based on the Drosophila anatomical structure, the temporal-spatial skeleton map including multiple sub-regions, each sub-region corresponding to the head, trunk, wings, legs, back, and abdomen respectively; extracting features from each sub-region through a spatiotemporal graph convolutional network to obtain the local feature information corresponding to each sub-region; performing temporal average pooling on the local feature information of each sub-region to aggregate the temporal dimension features to obtain the local embedding of each sub-region; and concatenating the local embeddings of each sub-region to obtain the first feature information.

[0079] The two-dimensional keypoint skeleton sequence x of Drosophila is:

[0080] ;

[0081] Where T is the number of frames, V is the number of keypoints per fruit fly individual, and M is the number of fruit fly individuals.

[0082] In one example, the feature extraction module 310 first uses the two-dimensional keypoint skeleton sequence extracted from a video of fruit fly behavior as input to construct a temporal-spatial skeleton map. To enhance the ability to identify actions in different parts of the body, the skeleton map is divided into multiple sub-regions according to anatomical structure (such as head, torso, wings, legs, etc.), and feature modeling is performed separately for each. A spatio-temporal graph convolutional network (Spatial-TemporalGCN) is used to extract dynamic motion features from the global skeleton and each sub-region to obtain an intermediate representation tensor (first feature information) characterizing individual behavior.

[0083] Specifically, in order to effectively capture the fine-grained, high-frequency behavioral dynamics of fruit flies under microgravity, this invention proposes a location-aware hyperbolic graph convolutional network (PH-GCN). This method performs spatiotemporal modeling on a two-dimensional skeleton sequence and enhances the geometric representation of features through hyperbolic spatial projection.

[0084] Let the input skeleton sequence be x∈R 2×T×V×M Where T represents the number of frames, V represents the number of keypoints per individual, and M represents the number of fruit flies. After normalization and preprocessing, the sequence is input into a multi-layer graph convolutional network (GCN), which consists of spatial graph convolution and multi-scale temporal convolution. The network acts on a skeleton graph with keypoints as nodes and anatomical structures as edges to extract motion features in the spatial and temporal dimensions.

[0085] To incorporate anatomical priors and enhance local discrimination capabilities, this invention divides the skeleton into P functional regions based on keypoint topology, including the head, back, abdomen, tail, wings, and legs. Each region uses an independent GCN encoder. Extract the corresponding local motion representation:

[0086] ;

[0087] Where Xp represents the input feature sequence of the p-th region. The local features are aggregated using time-averaged pooling.

[0088] ;

[0089] Finally, all local parts are embedded and spliced ​​together to form a global skeleton representation: ;

[0090] The method provided by this invention first constructs a temporal-spatial skeleton map containing sub-regions such as the head and torso based on the anatomical structure of the fruit fly using a feature extraction module, ensuring that the motion features of each part can be modeled independently. Then, a spatiotemporal graph convolutional network is used to selectively extract local feature information for each sub-region, capturing the dynamic motion patterns at the part level. Subsequently, temporal average pooling is used to aggregate temporal dimension features, avoiding the randomness of single-frame features and obtaining stable sub-region local embeddings. Finally, the local embeddings are concatenated to form the first feature information containing both global and local information. This process achieves end-to-end optimization of "spatial part segmentation - spatiotemporal feature extraction - temporal dimension aggregation," accurately capturing the local fine-grained dynamics and global motion patterns of fruit fly behavior. This provides high-quality, high-discriminatory feature input for the subsequent hyperbolic embedding module, significantly improving the model's accuracy in recognizing fine-grained behaviors.

[0091] Furthermore, the hyperbolic embedding module 320 is used to project the first feature information from Euclidean space to hyperbolic space to obtain the second feature information of each fruit fly two-dimensional keypoint skeleton sequence. The second feature information includes a global skeleton hyperbolic representation and a local hyperbolic representation of each sub-region.

[0092] In one possible implementation, the hyperbolic space is represented by the Poincaré sphere model B. n The hyperbolic embedding module is specifically used to: project the first feature information from Euclidean space to hyperbolic space based on a preset hyperbolic projection layer to obtain the second feature information corresponding to each fruit fly two-dimensional key point skeleton sequence, wherein the second feature information is a hyperbolic representation set used for hierarchical semantic modeling;

[0093] The hyperbolic representation of the set is:

[0094] {z skel ,z part1 ,z part2 ,...,z partP};

[0095] Among them, z skel For the global skeleton hyperbolic representation, z partP Let P be the local hyperbolic representation of the P-th sub-region, where P is the number of sub-regions;

[0096] The preset hyperbolic projection layer z is:

[0097] ;

[0098] Where, z∈B n W z R is a learnable preset linear transformation, where R is the radius of curvature of the hyperbolic space.

[0099] In one example, the hyperbolic embedding module 320, targeting the nonlinear, hierarchical structural features of fruit fly behavior, introduces a hyperbolic geometric embedding method. This method projects the skeleton features from Euclidean space to a hyperbolic space such as a Poincaré sphere, constructing curvature-aware, highly expressive features. This module effectively captures the implicit hierarchical structure of behavior, enhances the ability to model complex sequence patterns, and obtains a structure-aware behavioral representation (secondary feature information).

[0100] Specifically, traditional Euclidean space modeling assumes that the geometric relationships between features are flat and that feature distances are uniformly distributed. This has limitations when modeling behavioral semantics with hierarchical structures and fine-grained differences. For example, Euclidean space often fails to maintain sufficient semantic separability when distinguishing subtle actions such as "grooming" and "eating."

[0101] To this end, this invention maps skeleton features to a hyperbolic space to better preserve the semantic tree structure and hierarchical relationships. Specifically, it employs the Poincaré sphere model B. n And define a learnable hyperbolic projection layer: Among them, W z It is a learnable linear transformation, where R represents the radius of curvature. This projection is applied to the global skeleton embedding and all local embeddings, ultimately yielding the following set of hyperbolic representations: ;

[0102] By embedding behavioral features into hyperbolic manifolds, this method not only models subtle differences but also preserves hierarchical semantic relationships between behavioral categories, thereby improving alignment with linguistic semantics and enhancing generalization ability for unknown categories or cross-domain tasks. These embedded features are then fed into a multimodal alignment module for language-guided comparative learning under zero-shot and generalized zero-shot settings.

[0103] The method provided by this invention defines the hyperbolic space as a Poincaré sphere model and projects the first feature information from Euclidean space to the hyperbolic space through a preset hyperbolic projection layer. On the one hand, the geometric characteristics of the Poincaré sphere model are naturally adapted to hierarchical structure modeling, enabling a more compact representation of the hierarchical relationship between the global skeleton and local sub-regions in a lower dimension, thus solving the feature redundancy problem in Euclidean space. On the other hand, the hyperbolic representation set clearly distinguishes between the global skeleton hyperbolic representation and the local hyperbolic representation of the sub-regions, and the preset hyperbolic projection layer adapts to the distribution pattern of fruit fly behavioral features through learnable linear transformations and radii of curvature, ensuring that the projected hyperbolic features retain the discriminative power of the original features. This design enables the feature representation to possess both hierarchical modeling capabilities and high discriminative power, providing a suitable feature foundation for the subsequent semantic alignment module to achieve accurate alignment in the hyperbolic space, further enhancing the model's ability to characterize complex behaviors.

[0104] In some embodiments, the semantic alignment module 330 is used to convert the description information of M behavior categories into a global semantic vector corresponding to the global semantic description and a part-level semantic vector corresponding to each sub-region of the fine-grained description of the body part. The semantic alignment module is also used to construct the semantic vectors of the MN behavior categories into a hyperbolic semantic reference benchmark and integrate them into the unified hyperbolic semantic space of the fruit fly behavior recognition model. The semantic alignment module is also used to align the global skeleton hyperbolic representation and the global semantic vector included in the second feature information of the same behavior category with the second feature information of the same behavior category in the unified hyperbolic semantic space, and to align the local hyperbolic representation and the part-level semantic vector of each sub-region included in the second feature information of the same behavior category to obtain the alignment result, and to train the model based on the alignment result.

[0105] In one possible implementation, the semantic alignment module is specifically used for: inputting the global semantic description into the text encoder for the description information of each behavior category among M behavior categories, and outputting the global semantic vector of each behavior category; inputting the fine-grained description at the body part level into the text encoder according to the corresponding sub-regions, and outputting the part-level semantic vector corresponding to each sub-region; performing norm pruning on the global semantic vector and the part-level semantic vector to constrain the vector norm within a preset radius r to adapt to the geometric characteristics of the hyperbolic space; in the unified hyperbolic semantic space, for N behavior categories, performing global-level alignment between the global skeleton hyperbolic representation contained in the second feature information of the same behavior category and the global semantic vector, and calculating the hyperbolic space normalized cosine similarity of the global-level alignment; performing part-level alignment between the local hyperbolic representation of each sub-region of the same behavior category among the N behavior categories and the part-level semantic vector of the corresponding sub-region, and calculating the hyperbolic space normalized cosine similarity of the part-level alignment of each sub-region respectively; and using the global semantic vector and the part-level semantic vector as the hyperbolic semantic reference benchmark in the hyperbolic semantic space for MN behavior categories.

[0106] The method provided by this invention first decomposes the behavior description into global and part-level components, generating corresponding semantic vectors that match the hierarchical structure of visual features; then, it adapts the semantic vectors to hyperbolic space constraints through norm pruning to avoid geometric conflicts; it performs alignment at different granularities at the global and part levels for seen categories, calculates hyperbolic similarity, and strengthens fine-grained differentiation; finally, it uses the semantic vectors of unseen categories as hyperbolic reference benchmarks to provide anchor points for zero-shot identification, thereby significantly improving the reliability and accuracy of identifying unseen behavior categories.

[0107] In some embodiments, when the semantic alignment module trains the model based on the alignment results, the semantic alignment module is specifically used to: calculate a global contrast loss based on the hyperbolic space normalized cosine similarity of global alignment, with the goal of maximizing the global similarity of samples of the same behavior category and minimizing the global similarity of samples of different behavior categories; for each sub-region, calculate a part-level contrast loss based on the hyperbolic space normalized cosine similarity of part-level alignment of each sub-region, with the goal of maximizing the similarity between the local hyperbolic representation of each sub-region in samples of the same behavior category and the corresponding part-level semantic vector, and minimizing the similarity between the local hyperbolic representation of each sub-region in samples of different behavior categories and the corresponding part-level semantic vector; perform a weighted summation of the global contrast loss and the part-level contrast loss of each sub-region to obtain a total loss function; and iteratively train the fruit fly behavior recognition model with the goal of minimizing the total loss function to obtain a trained fruit fly behavior recognition model.

[0108] In one example, the semantic alignment module 330 automatically generates linguistic descriptions of behavior categories using a large language model (such as GPT), including overall behavioral semantic descriptions and fine-grained action descriptions of body parts. Then, a pre-trained visual-language model (such as CLIP) is used to embed the behavioral descriptions into semantic vector representations. Subsequently, the skeleton embedding and the linguistic embedding are aligned in a unified hyperbolic semantic space. The model is trained through multi-level contrastive learning objectives, enabling it to learn structure-semantic correspondences and thus achieve semantic generalization ability for unseen categories.

[0109] Specifically, in order to achieve effective alignment between skeleton temporal features and behavioral semantics, the semantic alignment module 330 proposed in this invention is a language-guided contrastive learning framework. This framework jointly utilizes global and part-level natural language descriptions to map skeleton sequences and language information to a shared semantic space, and performs alignment optimization through contrastive learning objectives.

[0110] All language descriptions will be uniformly formatted as "a cropped video of [description content]" and embedded using the pre-trained CLIP text encoder .ftext. For each behavior category y, the following set of language embeddings will be obtained:

[0111] ;

[0112] Where P represents the number of anatomical sites and m is the embedding dimension.

[0113] Furthermore, the visual-language contrastive learning mechanism of the semantic alignment module 330 provided in this embodiment of the invention is as follows:

[0114] Assuming the global skeleton embedding after hyperbolic projection is Each part is embedded as This invention utilizes shared linear transformation Map it to the language embedding space:

[0115] ;

[0116] Subsequently, a symmetric contrastive learning objective is employed to match the skeleton embedding with the corresponding language description. Given the normalized cosine similarity function sim(·,·) and the temperature coefficient... The alignment probability is calculated as follows:

[0117] ;

[0118] To enhance the modeling ability for multiple positive samples and avoid overfitting to difficult negative samples, this invention uses KL divergence as the contrastive loss function:

[0119] ;

[0120] Where y is the one-hot vector corresponding to the positive match.

[0121] The final total loss function comprehensively considers the comparative learning objectives at both the global level and the level of each part:

[0122] ;

[0123] Where, α p With α global These are weighting coefficients, which can be set to a fixed value or learned automatically during training.

[0124] By simultaneously modeling the alignment relationship between skeleton features and linguistic semantics at multiple granular levels, the multimodal contrastive learning framework proposed in this invention can effectively capture micro-action semantics, enhance the interpretability and generalization ability of behavior recognition models, and is especially suitable for zero-shot and cross-domain behavior recognition scenarios.

[0125] The method provided in this invention optimizes model training through a hierarchical loss design: First, a global contrastive loss is calculated based on global alignment similarity to ensure the model learns the semantic differences of overall behavior; then, a part-level contrastive loss is calculated for each sub-region to focus on optimizing the association between local part actions and language descriptions; finally, the two types of losses are weighted and summed to obtain the total loss function, achieving collaborative optimization at different granularities of global and local levels. This design avoids the problem of local part features being masked by global features, enabling the model to simultaneously grasp the overall behavioral patterns and the details of local part actions. It can more accurately distinguish semantically similar behaviors but different local actions, significantly improving the model's recognition accuracy and generalization ability for complex behaviors, especially in zero-shot scenarios where the discrimination of unseen behaviors is more reliable.

[0126] S3. Obtain the two-dimensional key point skeleton sequence of the fruit fly to be detected.

[0127] S4. Input the two-dimensional keypoint skeleton sequence of the fruit fly to be detected into the trained fruit fly behavior recognition model, and output the target behavior category corresponding to the two-dimensional keypoint skeleton sequence of the fruit fly to be detected. The target behavior category is one of M behavior categories.

[0128] The following example illustrates the process of identifying (predicting) the target behavior category corresponding to a two-dimensional keypoint skeleton sequence of a fruit fly. For each candidate behavior category y, a global description and a fine-grained description at the part level are first generated using a language encoding module, and then embedded into a shared semantic space. For the input skeleton sequence x, this method extracts its global skeleton embedding Z. skel Embedded in various parts And calculate the similarity with the language features of the corresponding category y. The specific category score is defined as follows:

[0129] ;

[0130] Where sim(⋅,⋅) represents the normalized cosine similarity calculated in hyperbolic space, and α p With α global These are the weighting coefficients at the part level and the global level, respectively. Finally, the model outputs the behavior category with the highest score as the prediction result (the behavior category corresponding to the detected fruit fly 2D keypoint skeleton sequence):

[0131] ;

[0132] The method provided by this invention obtains the two-dimensional keypoint skeleton sequence of the fruit fly to be detected, and directly inputs it into the trained model to output the behavior category. Combined with the zero-shot generalization capability already possessed by the model during training, this prediction process does not require additional sample labeling for new behavior categories, enabling automated, zero-shot behavior recognition. This process significantly reduces manual intervention and improves the efficiency of fruit fly behavior analysis, especially suitable for scenarios where data is scarce and rapid acquisition of behavior results is required under experimental conditions such as microgravity, meeting the high-throughput analysis needs of life science experiments. In other words, this invention constructs a unified semantic space during the training phase through a semantic alignment mechanism between skeleton representation and linguistic description, thereby enabling the recognition of behaviors of unseen categories during the prediction phase, achieving a unified prediction process for zero-shot recognition (ZSL) and generalized zero-shot recognition (GZSL).

[0133] In other words, the method provided by this invention clearly distinguishes between "seen behavior categories (N categories)" and "unseen behavior categories (MN categories)," constructing a framework of "training a model based on N categories of samples + building a hyperbolic semantic reference baseline based on MN category descriptive information." This eliminates the need for labeled samples for unseen categories, integrating their semantic information into the model's semantic space solely through natural language descriptions. This design fundamentally solves the technical problems of traditional methods relying on annotations and being unable to identify zero-sample behaviors. The trained model can directly identify M behavior categories (including unseen categories), perfectly adapting to scenarios with scarce data and open categories, such as space experiments. It achieves full coverage of behavior recognition in zero-sample scenarios, providing a feasible solution for real-time analysis of in-orbit fruit fly behavior.

[0134] As described above, the method provided in this embodiment of the invention, through the integration of the three modules, realizes a unified structure-semantic alignment behavior modeling framework. This framework not only supports traditional supervised recognition tasks but also possesses strong generalization capabilities for zero-sample classes and cross-scenario tasks. The entire system is end-to-end optimized, exhibiting good scalability and deployment capabilities, and is suitable for intelligent analysis of fruit fly behavior under complex experimental conditions such as on the ground or in space.

[0135] In one example, the recognition system 100 completes the method provided in this embodiment of the invention by performing the following steps: In the data preprocessing stage, key point sequences of individual fruit fly skeletons are extracted to construct a space-time map; the skeleton map is input to the GCN module to obtain skeleton embedding features; hyperbolic space mapping is performed on the skeleton features to obtain a more discriminative representation; simultaneously, predefined or GPT-generated behavioral description text is input into a language encoder (such as CLIP text pyramid) to extract semantic vectors; contrastive loss (such as KL divergence or MarginLoss) is introduced during training to align the skeleton with the text embedding; in the testing stage, the input skeleton sequence and a predefined behavioral description set are compared for similarity in the aligned semantic space to achieve the recognition of new behavioral categories.

[0136] To facilitate understanding of this solution, the following specific example will be used to explain the beneficial effects of the method provided by this invention.

[0137] For details, see Figure 4 , Figure 4This is a schematic diagram of a fruit fly behavior recognition method based on hyperbolic spatial vision and language alignment provided in an embodiment of the present invention. First, the feature extraction module 310 determines a temporal-spatial skeleton map including multiple sub-regions based on multiple fruit fly two-dimensional keypoint skeleton sequences. The feature extraction module 310 extracts the dynamic motion features of each fruit fly two-dimensional keypoint skeleton sequence through a spatiotemporal graph convolutional network to obtain the first feature information corresponding to each fruit fly two-dimensional keypoint skeleton sequence. The first feature information includes global skeleton features and local features of each sub-region. Among them, the fruit fly two-dimensional keypoint skeleton sequences of N behavior categories come from the ground environment, and the N behavior categories are seen behavior categories.

[0138] The hyperbolic embedding module 320 projects the first feature information from Euclidean space to hyperbolic space to obtain the second feature information of the skeleton sequence of each two-dimensional key point of the fruit fly. The second feature information includes the global skeleton hyperbolic representation and the local hyperbolic representation of each sub-region. The semantic alignment module converts the description information of M behavior categories into the global semantic vector corresponding to the global semantic description and the part-level semantic vector of each sub-region corresponding to the fine-grained description of the body part.

[0139] The semantic alignment module 330 constructs the semantic vectors of MN behavior categories into a hyperbolic semantic reference and integrates them into the unified hyperbolic semantic space of the fruit fly behavior recognition model. Within the unified hyperbolic semantic space, the semantic alignment module 330 aligns the global skeleton hyperbolic representation of the second feature information of the same behavior category with the global semantic vector, and aligns the local hyperbolic representation of each sub-region and the part-level semantic vector of the second feature information of the same behavior category, obtaining the alignment result. Based on the alignment result, the model is trained to obtain the trained fruit fly behavior recognition model. Here, MN behavior categories are unseen behavior categories.

[0140] Finally, the two-dimensional keypoint skeleton sequence of the fruit fly to be detected from the microgravity environment in orbit is input into the trained fruit fly behavior recognition model. At this time, the trained fruit fly behavior recognition model can perform zero-shot recognition (it can recognize both MN unseen behavior categories and N seen behavior categories), and output the target behavior category corresponding to the two-dimensional keypoint skeleton sequence of the fruit fly to be detected. The target behavior category is one of the M behavior categories.

[0141] Combination Figure 4 For example, the identification system 100 evaluated the method on a self-built fruit fly behavior dataset containing precisely labeled two-dimensional skeleton sequences, covering two experimental environments: a ground environment and an in-orbit microgravity environment. The space experimental data came from the life ecology experiment cabinet on the space station and is data obtained from real in-orbit experiments.

[0142] It should be noted that the experimental data construction process included: simultaneous high-resolution video acquisition, expert behavior category annotation, cascaded processing of multi-fruit fly pose estimation, and biological validation. The final dataset contains 10 behavior categories, divided into three types: 4 individual behaviors, 3 interactive behaviors (group behaviors), and 3 microgravity-specific behaviors. Each sequence contains T frames, with 26 keypoints annotated for each fruit fly in each frame. Furthermore, during model training and testing, ground-based samples were used for training, while in-orbit samples were used for model testing. The testing tasks were divided into three evaluation settings: intra-domain evaluation, cross-domain evaluation, and zero-shot / generalized zero-shot evaluation.

[0143] Specifically, the recognition performance of behavior categories is evaluated by the classification accuracy of Top 1-Top 5.

[0144] The formula for calculating accuracy is as follows:

[0145] ;

[0146] in, This represents the first k predicted labels for sample i. This is an indicator function.

[0147] In a zero-shot learning (ZSL) setup, we test on a test set Dtest(y) where the class is unseen. u We evaluate model performance on a mixed test set (y) that includes both seen and unseen classes, considering the significant domain bias and class imbalance issues in small animal behavior in the Generalized Zero-Shot Learning (GZSL) setting. s ∪y u The metrics reported are the overall accuracy across all domains, rather than individual accuracy rates. These metrics comprehensively measure the model's ability to recognize fine-grained behaviors in both semantic and domain-specific generalization scenarios.

[0148] Furthermore, each skeleton sequence was normalized to a fixed frame length T=64. Spatial modeling was performed using a part-aware graph convolutional network (GCN) layer, followed by temporal convolution, and the features were projected into a Poincaré sphere with curvature c to achieve hyperbolic embedding. Textual features were encoded using a pre-trained CLIPViT-B / 32 model.

[0149] The entire framework is trained using the Adam optimizer with a learning rate of 1×10⁻⁶. -4 The total number of training epochs was 100, and early stopping was used to prevent overfitting. All experiments were implemented in the PyTorch framework.

[0150] The proposed method (named ZS-Drosophila) was compared with several mainstream skeleton-based action recognition methods, including CTR-GCN, HD-GCN, TD-GCN-gesture, and GAP. All methods were tested under four defined evaluation settings.

[0151] The task is defined as follows:

[0152] The skeleton-based fruit fly behavior recognition problem is formally defined as follows: Given a sequence of pose key points extracted from video frames, the goal is to achieve behavior classification and recognition, including unseen categories, without the need for additional labeled samples for new behavior categories, and to be adaptable to domain shifts between different experimental domains (such as ground and space).

[0153] Let the training dataset be denoted as D={x} i ,y i ,a i}, where i = 1, 2, 3...N, and each sample x i ∈R 2×T×V×M This represents a two-dimensional skeleton sequence containing T frames, V keypoints per frame, and a total of M fruit fly individuals. The corresponding behavior label is y. i ∈Y (where Y is the label space), semantically described as a i ∈A (A is a set of semantic descriptions), such as category names or natural language prompts generated by large language models (such as GPT-4o), and used after expert review (this invention uses GPT-4o to automatically generate semantic descriptions, which are then polished by experts).

[0154] It should be understood that natural language prompts generated by large language models (such as GPT-4o) can also be used directly without polishing, and the embodiments of the present invention do not impose any special restrictions on this.

[0155] The entire label space is divided into non-overlapping known categories y s Unseen category y u , that is, y s ∩ The dataset covers two domains: the ground domain (D...). g ) and spatial domain (D s Based on the above settings, we define four evaluation protocols covering supervised identification and zero-shot identification capabilities in two domains:

[0156] 1. Same-domain assessment (Ground domain D) g )

[0157] In this setup, both training and testing samples are derived from ground-based experimental data to evaluate the model's performance under no-domain offset conditions:

[0158] Supervised recognition capability: In the known category test set D test (y s An evaluation will be conducted on the above.

[0159] Zero-shot recognition capability: On the unseen class test set D test The model was evaluated on (yu) and two metrics were used to measure its zero-shot generalization ability:

[0160] ZSL (Zero-Shot Learning): Only available in unseen categories. u Classify within the scope;

[0161] GZSL (Generalized Zero-Shot Learning): In all categories... s ∪y u Classification is performed within the specified range, and the test samples still come from y. u .

[0162] 2. Cross-domain assessment (spatial domain D) s )

[0163] In this setup, training data comes from the ground domain, and test data comes from in-orbit experiments. The model's cross-domain generalization ability is evaluated: supervised generalization ability: on the known class test set D in the spatial domain. test (D s ,y s The evaluation is conducted on a test set D, testing the model's cross-domain generalization ability from ground to space; zero-shot generalization ability: on the unseen class test set D in the spatial domain. test (Ds,y u The evaluation includes two settings: ZSL and GZSL.

[0164] Throughout the training process, the model can only access training samples D belonging to the seen categories in the ground domain. train (D g ,y s To simulate the limited label resources in real-world experiments, the model needs to learn a shared latent space that aligns the skeleton representation with the language embedding features F (details below) to achieve unified structure-semantic modeling and cross-class reasoning in behavior recognition tasks.

[0165] Referring to Table 1, the table below shows the recognition performance results within the same domain (ground data), covering both seen and unseen behavior categories. On the left side of the table, we evaluate the supervised recognition accuracy for seen categories. The proposed ZS-Drosophila achieves the highest Top-1 accuracy (63.21%), surpassing all baseline models, including the previous best method GAP (56.33%), demonstrating that the method provided by this invention still possesses superior modeling capabilities even in fully supervised scenarios. On the right side of the table, we evaluate the model's zero-shot generalization ability. Existing methods such as CTR-GCN, HD-GCN, and TD-GCN-gesture do not support zero-shot inference, while GAP's ZSL and GZSL accuracies are 26.35% and 22.09%, respectively. In contrast, ZS-Drosophila achieves a significant improvement, with a ZSL accuracy of 30.23% and a GZSL accuracy of 25.77%. These results highlight the model's strong semantic alignment capabilities and its effectiveness in generalizing to new behavior categories within the same domain.

[0166] Table 1

[0167]

[0168] Note: "--" indicates that the method does not support zero-sample or generalized zero-sample evaluation, and the same applies to the following table.

[0169] Referring to Table 2, the following table presents the experimental results for cross-domain behavior recognition, where the model was trained on terrestrial data and tested on spatial domain samples. The left column reports the Top-1 / Top-5 accuracy of the seen category under supervised settings to evaluate the model's robustness to domain shifts; the right column provides the recognition performance of the unseen category under zero-shot (ZSL) and generalized zero-shot (GZSL) settings, reflecting the model's generalization ability in the absence of direct supervision.

[0170] Among the baseline methods, CTR-GCN, HD-GCN, and TD-GCN-gesture exhibit limited cross-domain generalization capabilities, with Top-1 accuracies all below 15%, and none support zero-shot identification of unseen classes. The best-performing baseline method, GAP, achieves a Top-1 accuracy of 13.66% in supervised evaluation and 18.35% under the ZSL setting, but its accuracy drops to 14.22% under the GZSL setting due to significant influence from domain bias.

[0171] In comparison, the proposed model significantly outperforms all baseline methods across all metrics. In cross-domain tests with known categories, its Top-1 and Top-5 accuracies reach 19.48% and 78.49%, respectively, demonstrating stronger domain robustness. In zero-shot recognition, ZS-Drosophila achieves a ZSL accuracy of 21.00%, a 2.65% improvement over GAP; it also maintains a competitive performance of 12.16% under the GZSL setting, despite significant domain and label bias. These results demonstrate the effectiveness of our method in semantic alignment and hyperbolic modeling, simultaneously enhancing the model's cross-domain transferability and unseen behavior recognition capabilities.

[0172] Table 2

[0173]

[0174] Furthermore, in the ablation experiments conducted based on the method provided by this invention and several mainstream skeleton-based action recognition methods, the ablation results are as follows:

[0175] We conducted ablation experiments on the Valinna model related to hyperbolic space projection to investigate the impact of different hyperparameter configurations on recognition performance. Specifically, we examined the curvature parameter c, the sphere embedding dimension (BallDim), whether or not a Riemannian mapping (log mapping, denoted as Riem. in the table) was used, the CLIP feature norm clipping radius r, and the positional variation of the projection layer. All experimental results are shown in the table below:

[0176] Table 3

[0177]

[0178] Referring to Table 3, the experimental results show that reducing the curvature parameter c from 0.7 to 0.3 improves the Top-1 accuracy from 56.17% to 60.03%, indicating that a hyperbolic space with higher curvature is beneficial for modeling complex structural relationships. However, introducing the Riemann mapping leads to a significant performance decrease, demonstrating its instability in hyperbolic optimization. Furthermore, increasing the norm clipping radius of the CLIP text features from 2.3 to 3.0 improves the Top-5 accuracy to 92.59%, indicating that a more relaxed norm constraint helps improve semantic alignment.

[0179] To further evaluate the body structure segmentation strategy and the actual contribution of the hyperbolic projection module (HP), we conducted joint ablation experiments on a representative subset, and the results are shown in the table below.

[0180] First, we compared the body partitioning strategies of 4 and 6 partitions. Without introducing the HP module, the Top-1 accuracy of the 6-partition configuration was 50.00%, significantly better than the 47.62% of the 4-partition configuration, indicating that finer-grained anatomical partitioning helps improve spatial modeling capabilities.

[0181] Furthermore, referring to Table 4, after adding the HP module to the 6-partition base, the Top-1 accuracy was further improved to 51.19%, indicating that the introduction of hyperbolic space helps to more effectively model non-Euclidean geometric relationships, thereby improving the performance of behavior recognition.

[0182] Table 4

[0183]

[0184] In summary, hyperbolic space projection plays a crucial role in feature modeling and semantic alignment, especially when combined with finer-grained body structure segmentation strategies, which can significantly improve the performance and generalization ability of fruit fly behavior recognition models.

[0185] In summary, the method provided by the embodiments of this invention has enhanced behavioral semantic modeling capabilities: it enhances the abstract expression and semantic generalization of behavior through linguistic description; it also has improved recognition accuracy and generalization ability, achieving significantly better performance than existing methods in cross-domain recognition and unseen class recognition tasks; finally, it has higher modeling efficiency and compressed expression capability: it models complex behavioral topological structures based on hyperbolic space, which is more expressive than Euclidean space. It can also be understood that the method provided by this invention can be applied to various scenarios, such as behavioral analysis and health assessment in space life science experiments; experimental animal behavior research and basic neuroscience research; small-scale biological behavior recognition tasks in multimodal intelligent systems; and behavior recognition and semantic understanding in unlabeled scenarios.

[0186] It should be noted that, in addition to the verified posture tracking of nematodes, fruit flies, and zebrafish, this invention can be extended to model animals in common biological experiments and neuroscience experiments such as mice and macaques, as well as large wild animals such as horses and tigers, and even humans. Based on this, it can support the analysis of the motion trajectory of each key point and statistically analyze key information such as motion speed, direction, acceleration, and oscillation frequency. Furthermore, based on the above results, it can further analyze the behavioral categories of model animals using unsupervised methods such as clustering and key point-based behavior recognition methods, conducting behavioral spectrum studies and providing necessary and important observational information for research in neuroscience, animal behavior, and life sciences. This invention does not impose any particular limitations on these aspects.

[0187] The foregoing mainly describes the solutions of the embodiments of the present invention from a methodological perspective. It is understood that, in order to achieve the above-mentioned functions, the identification system 100 includes at least one of the hardware structures and software modules corresponding to the execution of each function. Those skilled in the art should readily recognize that, in conjunction with the units and algorithm steps of the various examples described in the embodiments disclosed herein, the embodiments of the present invention can be implemented in hardware or a combination of hardware and computer software. Whether a function is executed in hardware or by computer software driving hardware depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of the embodiments of the present invention.

[0188] In this embodiment of the invention, the identification system 100 can be divided into functional units according to the above method example. For example, the identification system 100 can be divided into functional units corresponding to various functions, or two or more functions can be integrated into one processing unit. The integrated unit can be implemented in hardware or as a software functional unit. It should be noted that the unit division in this embodiment of the invention is illustrative and only represents one logical functional division; other division methods may be used in actual implementation.

[0189] For example, Figure 5 This diagram illustrates the hardware structure of a recognition system according to an embodiment of the present invention. The recognition system 100 includes: an acquisition unit 110, configured to acquire descriptive information of M behavioral categories and a training sample set, wherein the descriptive information includes global semantic description and fine-grained description at the body part level, and the training sample set includes multiple two-dimensional keypoint skeleton sequences of fruit flies corresponding to N behavioral categories out of the M behavioral categories, where M is greater than N, and M and N are positive integers; wherein the N behavioral categories are known behavioral categories, and the remaining MN behavioral categories are unseen behavioral categories; and a training unit 120, configured to recognize fruit fly behaviors based on the descriptive information of the N behavioral categories and the training sample set. The model is trained, and a hyperbolic semantic reference baseline is constructed based on the descriptive information of MN behavior categories other than N behavior categories. This reference baseline is then integrated into the semantic space of the fruit fly behavior recognition model to obtain the trained fruit fly behavior recognition model. The acquisition unit 110 is also used to acquire the two-dimensional keypoint skeleton sequence of the fruit fly to be detected. The detection unit 130 is used to input the two-dimensional keypoint skeleton sequence of the fruit fly to be detected into the trained fruit fly behavior recognition model and output the target behavior category corresponding to the two-dimensional keypoint skeleton sequence of the fruit fly to be detected. The target behavior category is one of the M behavior categories.

[0190] It should be understood that specific descriptions of the above-mentioned optional methods can be found in the foregoing method embodiments, and will not be repeated here. Furthermore, explanations of any of the identification systems 100 provided above, as well as descriptions of their beneficial effects, can be found in the corresponding method embodiments described above, and will not be repeated here.

[0191] This invention also provides a computer-readable storage medium storing at least one computer instruction, which is loaded and executed by a processor to implement the methods of the various embodiments described above. Explanations of the relevant content and descriptions of the beneficial effects of any of the computer-readable storage media provided above can be found in the corresponding embodiments described above, and will not be repeated here.

[0192] This invention also provides a chip. This chip integrates a control circuit for implementing the functions of the aforementioned identification system 100 and one or more ports. Optionally, the functions supported by this chip are as described above and will not be repeated here.

[0193] Those skilled in the art will understand that the program for implementing all or part of the steps of the above embodiments, which can be executed by a program instructing related hardware, can be stored in a computer-readable storage medium. The storage medium mentioned above can be a read-only memory, a random access memory, etc. The processing unit or processor mentioned above can be a central processing unit, a general-purpose processor, an application-specific integrated circuit (ASIC), a microprocessor (DSP), a field-programmable gate array (FPGA), or other programmable logic devices, transistor logic devices, hardware components, or any combination thereof.

[0194] It should be noted that the devices for storing computer instructions or computer programs provided in the embodiments of the present invention, such as, but not limited to, the aforementioned memory, computer-readable storage medium, and communication chip, are all non-transitory. Those skilled in the art should recognize that the functions described in the embodiments of the present invention in one or more of the above examples can be implemented using hardware, software, firmware, or any combination thereof. When implemented using software, these functions can be stored in a computer-readable storage medium or transmitted as one or more instructions or code on a computer-readable storage medium. Computer-readable storage media include computer storage media and communication media, wherein communication media include any medium that facilitates the transmission of computer programs from one place to another. Storage media can be any available medium accessible to general-purpose or special-purpose computers.

[0195] Although embodiments of the present invention have been shown and described above, it is understood that the above embodiments are exemplary and should not be construed as limiting the present invention. Those skilled in the art can make changes, modifications, substitutions and variations to the above embodiments within the scope of the present invention.

Claims

1. A method for fruit fly behavior recognition based on hyperbolic space visual and linguistic alignment, characterized in that, The method includes: Obtain description information and training sample set for M behavior categories. The description information includes global semantic description and fine-grained description at the body part level. The training sample set includes multiple two-dimensional keypoint skeleton sequences of fruit flies corresponding to N behavior categories among the M behavior categories, where M is greater than N and M and N are positive integers. The N behavior categories are known behavior categories, and the MN behavior categories other than the N behavior categories are unseen behavior categories. The fruit fly behavior recognition model is trained based on the description information of N behavior categories and the training sample set. A hyperbolic semantic reference benchmark is constructed based on the description information of MN behavior categories other than the N behavior categories and integrated into the semantic space of the fruit fly behavior recognition model to obtain the trained fruit fly behavior recognition model. Obtain the two-dimensional keypoint skeleton sequence of the fruit fly to be detected; The two-dimensional keypoint skeleton sequence of the fruit fly to be detected is input into the trained fruit fly behavior recognition model, and the target behavior category corresponding to the two-dimensional keypoint skeleton sequence of the fruit fly to be detected is output. The target behavior category is one of the M behavior categories. The fruit fly behavior recognition model includes a feature extraction module, a hyperbolic embedding module, and a semantic alignment module. The feature extraction module determines a temporal-spatial skeleton map comprising multiple sub-regions based on the multiple fruit fly two-dimensional keypoint skeleton sequences. These sub-regions include the head, torso, wings, legs, back, and abdomen. Dynamic motion features of each fruit fly two-dimensional keypoint skeleton sequence are extracted using a spatiotemporal graph convolutional network to obtain first feature information corresponding to each fruit fly two-dimensional keypoint skeleton sequence. The first feature information includes global skeleton features and local features of each sub-region. The hyperbolic embedding module projects the first feature information from Euclidean space to hyperbolic space to obtain second feature information for each fruit fly two-dimensional keypoint skeleton sequence. The second feature information includes a global hyperbolic skeleton representation and a local hyperbolic representation of each sub-region. The semantic alignment module is used to... The descriptive information of M behavior categories is converted into a global semantic vector corresponding to the global semantic description and a part-level semantic vector corresponding to each sub-region of the fine-grained description at the body part level. Norm clipping is performed on the global semantic vector and the part-level semantic vector to constrain the vector norm within a preset radius r to adapt to the geometric characteristics of hyperbolic space. For MN behavior categories, the global semantic vector and the part-level semantic vector are used as hyperbolic semantic reference benchmarks in the hyperbolic semantic space and integrated into the unified hyperbolic semantic space of the fruit fly behavior recognition model. In the unified hyperbolic semantic space, the global skeleton hyperbolic representation included in the second feature information of the same behavior category is aligned with the global semantic vector, and the local hyperbolic representation of each sub-region included in the second feature information of the same behavior category is aligned with the part-level semantic vector to obtain the alignment result. The model is trained based on the alignment result.

2. The method according to claim 1, characterized in that, Before obtaining the descriptive information of the M behavior categories and the training sample set, the method further includes: Based on the large language model, generate descriptive information for each of the M behavior categories.

3. The method according to claim 2, characterized in that, The feature extraction module is specifically used for: For each two-dimensional keypoint skeleton sequence of fruit flies corresponding to N behavioral categories, a time-space skeleton map of each two-dimensional keypoint skeleton sequence of fruit flies is determined based on the anatomical structure of fruit flies. The time-space skeleton map includes multiple sub-regions, and each sub-region corresponds to the head, trunk, wings, legs, back and abdomen, respectively. The spatiotemporal graph convolutional network is used to extract features from each sub-region to obtain the local feature information corresponding to each sub-region. Temporal average pooling is performed on the local feature information of each sub-region to aggregate the temporal dimension features, resulting in the local embedding of each sub-region; the local embeddings of each sub-region are then concatenated to obtain the first feature information. The two-dimensional keypoint skeleton sequence x of Drosophila is: ； Where T is the number of frames, V is the number of keypoints per fruit fly individual, and M is the number of fruit fly individuals.

4. The method according to claim 3, characterized in that, The hyperbolic space is Poincaré sphere model B. n The hyperbolic embedding module is specifically used for: Based on a preset hyperbolic projection layer, the first feature information is projected from Euclidean space to hyperbolic space to obtain the second feature information corresponding to each fruit fly two-dimensional key point skeleton sequence, wherein the second feature information is a hyperbolic representation set used for hierarchical semantic modeling. The hyperbolic representation set is as follows: {With skel ,With part1 ,With part2 ,...,With partP }; Among them, z skel For the global skeleton hyperbolic representation, z partP Let P be the local hyperbolic representation of the P-th sub-region, where P is the number of sub-regions; The preset hyperbolic projection layer z is: ； Where, z∈B n W z R is a learnable preset linear transformation, where R is the radius of curvature of the hyperbolic space.

5. The method according to claim 4, characterized in that, The semantic alignment module is specifically used for: For the description information of each behavior category among the M behavior categories, the global semantic description is input into the text encoder, and the global semantic vector of each behavior category is output; the fine-grained description at the body part level is input into the text encoder according to the corresponding sub-regions, and the part-level semantic vector corresponding to each sub-region is output. In the unified hyperbolic semantic space, for N behavior categories, the global skeleton hyperbolic representation contained in the second feature information of the same behavior category is globally aligned with the global semantic vector, and the hyperbolic space normalized cosine similarity of the global alignment is calculated; the local hyperbolic representation of each sub-region of the same behavior category in the N behavior categories is aligned with the part-level semantic vector of the corresponding sub-region, and the hyperbolic space normalized cosine similarity of the part-level alignment of each sub-region is calculated respectively.

6. The method according to claim 5, characterized in that, When the semantic alignment module trains the model based on the alignment results, the semantic alignment module is specifically used for: Based on hyperbolic space normalized cosine similarity with global alignment, the global contrast loss is calculated with the goal of maximizing the global similarity of samples of the same behavior category and minimizing the global similarity of samples of different behavior categories. For each sub-region, based on the hyperbolic space normalized cosine similarity of the part-level alignment of each sub-region, the part-level contrast loss of each sub-region is calculated with the goal of maximizing the similarity between the local hyperbolic representation of each sub-region and the corresponding part-level semantic vector in the same behavior category sample and minimizing the similarity between the local hyperbolic representation of each sub-region and the corresponding part-level semantic vector in different behavior category samples. The total loss function is obtained by weighted summing the global contrast loss with the part-level contrast loss of each sub-region. With the goal of minimizing the total loss function, the fruit fly behavior recognition model is iteratively trained to obtain the trained fruit fly behavior recognition model.

7. The method according to claim 6, characterized in that, The M behavioral categories include individual behavior, group behavior, and special behaviors in microgravity environments. The N behavioral categories include individual behavior and group behavior. Individual behavior includes crawling, remaining still, feeding, and grooming. Group behavior includes courtship, mating, and touching. Special behaviors in microgravity environments include floating, tumbling, and crawling with outstretched wings.

8. A fruit fly behavior recognition system based on hyperbolic spatial vision and language alignment, characterized in that, The system includes: The acquisition unit is used to acquire description information of M behavior categories and training sample set. The description information includes global semantic description and fine-grained description at the body part level. The training sample set includes multiple two-dimensional keypoint skeleton sequences of fruit flies corresponding to N behavior categories among the M behavior categories, where M is greater than N and M and N are positive integers. The N behavior categories are seen behavior categories, and the MN behavior categories other than the N behavior categories are unseen behavior categories. The training unit is used to train the fruit fly behavior recognition model based on the description information of N behavior categories and the training sample set, and to construct a hyperbolic semantic reference benchmark based on the description information of MN behavior categories other than the N behavior categories, and integrate it into the semantic space of the fruit fly behavior recognition model to obtain the trained fruit fly behavior recognition model. The acquisition unit is also used to acquire the two-dimensional key point skeleton sequence of the fruit fly to be detected; The detection unit is used to input the two-dimensional keypoint skeleton sequence of the fruit fly to be detected into the trained fruit fly behavior recognition model, and output the target behavior category corresponding to the two-dimensional keypoint skeleton sequence of the fruit fly to be detected, wherein the target behavior category is one of the M behavior categories. The fruit fly behavior recognition model includes a feature extraction module, a hyperbolic embedding module, and a semantic alignment module. The feature extraction module determines a temporal-spatial skeleton map comprising multiple sub-regions based on the multiple fruit fly two-dimensional keypoint skeleton sequences. These sub-regions include the head, torso, wings, legs, back, and abdomen. Dynamic motion features of each fruit fly two-dimensional keypoint skeleton sequence are extracted using a spatiotemporal graph convolutional network to obtain first feature information corresponding to each fruit fly two-dimensional keypoint skeleton sequence. The first feature information includes global skeleton features and local features of each sub-region. The hyperbolic embedding module projects the first feature information from Euclidean space to hyperbolic space to obtain second feature information for each fruit fly two-dimensional keypoint skeleton sequence. The second feature information includes a global hyperbolic skeleton representation and a local hyperbolic representation of each sub-region. The semantic alignment module is used to... The descriptive information of M behavior categories is converted into a global semantic vector corresponding to the global semantic description and a part-level semantic vector corresponding to each sub-region of the fine-grained description at the body part level. Norm clipping is performed on the global semantic vector and the part-level semantic vector to constrain the vector norm within a preset radius r to adapt to the geometric characteristics of hyperbolic space. For MN behavior categories, the global semantic vector and the part-level semantic vector are used as hyperbolic semantic reference benchmarks in the hyperbolic semantic space and integrated into the unified hyperbolic semantic space of the fruit fly behavior recognition model. In the unified hyperbolic semantic space, the global skeleton hyperbolic representation included in the second feature information of the same behavior category is aligned with the global semantic vector, and the local hyperbolic representation of each sub-region included in the second feature information of the same behavior category is aligned with the part-level semantic vector to obtain the alignment result. The model is trained based on the alignment result.

9. An electronic device, characterized in that, include: processor; Memory used to store the processor's executable instructions; The processor is configured to execute the instructions to implement the fruit fly behavior recognition method based on hyperbolic space vision and language alignment as described in any one of claims 1-7.

Images