Zebrafish multi-target tracking method and system based on spatial science experiment video

Abstract

The application provides a zebrafish multi-target tracking method and system based on space science experiment video, which first acquires a zebrafish space science experiment video. Then, motion information is extracted based on the zebrafish space science experiment video, and the motion information is used to represent the motion state of the pixel points in the zebrafish space science experiment video. Next, multi-modal feature fusion is performed based on the motion information and appearance information to determine the multi-modal fusion features. Finally, target detection and tracking are performed according to the multi-modal fusion features to determine the target motion trajectory information corresponding to each target zebrafish. This method extracts motion feature information representing dynamic changes through adaptive threshold-based motion modeling, and then uses heterogeneous graphs to model the interaction and fusion relationship between appearance and motion features at different levels, enhancing the robustness of target detection, data association and trajectory prediction. In this way, the stability and accuracy of multi-target tracking of zebrafish can be effectively improved.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of target tracking, and in particular to a zebrafish multi-target tracking method and system based on space science experiment video. BACKGROUND

[0002] With the rapid development of space station technology, as a space laboratory, the space station has unique advantages such as covering the whole field of space science, strong on-orbit support capability, supporting human participation and up-down transportation, and can carry out large-scale multidisciplinary space science research. At present, the life and ecological experiment cabinet carried by the space station can obtain zebrafish experiment observation video (i.e. space science experiment video), which provides an important data basis for studying the influence of space environment on biological behavior and physiological response. Under this background, it is urgent to use multi-target tracking technology to realize accurate positioning, identity maintenance and motion trajectory generation of multiple zebrafish targets under the condition of given space science experiment video input, so as to provide reliable technical support for in-depth analysis.

[0003] Generally, multi-target tracking technology can be divided into two types of schemes: tracking after detection and joint detection and tracking. Among them, the core of tracking after detection is to decouple the tracking task into two stages of target detection and data association, and to improve the overall performance by optimizing the detection accuracy and association strategy independently. For example: SORT first realizes the separation of detection and association, predicts the motion state of the target based on Kalman filter and matches the detection box and the trajectory by combining the Hungarian algorithm. DeepSORT introduces appearance feature matching based on SORT, which improves the robustness of complex scenes. MOTDT improves the association accuracy by multi-cue fusion (such as appearance feature, motion feature). IOU ByteTrack only uses the intersection over union of the detection box for association, which is simple and efficient but sensitive to target motion speed and occlusion. Joint detection and tracking converts the detector into a tracker and integrates the two tasks in the same framework. For example: Tracktor uses the Faster R-CNN regression head to directly predict the next frame position of the target. FairMOT uses anchor-free detection based on CenterNet and jointly learns detection and re-identification features. TransTrack is based on the Transformer architecture, which uses self-attention mechanism to realize the joint modeling of detection and tracking.

[0004] However, the above-mentioned related multi-target tracking method is mostly designed for general public data sets, and has significant adaptability bottleneck when applied to space station zebrafish video analysis, which is difficult to meet the precise tracking demand. The specific technical problems include: first, the background interference suppression is insufficient. The strong reflection and diffuse reflection phenomenon caused by the limited light source in the space experiment environment causes multiple reflections of zebrafish, which easily leads to false matching and target splitting, seriously affecting the continuity and accuracy of tracking. Second, the ability to cope with violent movement and occlusion is limited. Zebrafish in microgravity environment shows stress response such as high-speed swimming, irregular turning and group gathering, which leads to frequent occlusion and overlap between targets, increases the difficulty of target segmentation and identity maintenance, and the joint detection and tracking method relies on the hypothesis of high overlap of inter-frame bounding boxes, which is not true in the case of fast target displacement. The pre-detection and then association method has the defects of discarding image appearance features or high feature extraction cost, and splitting the detection and tracking process. Third, the adaptability to environmental dynamic changes is poor. Bubbles generated in the water system during the on-orbit experiment period will adhere to the observation window or float in the field of view, forming a dynamic occlusion area. The existing method cannot effectively cope with the influence of such dynamic environmental changes on tracking effect. It can be seen that the related multi-target tracking method cannot stably and accurately realize multi-target tracking of zebrafish, and thus it is difficult to meet the scientific demand of trajectory precision and long-term identity consistency for space station zebrafish behavior analysis. SUMMARY

[0005] The technical problem to be solved by the present application is the problem of unstable and accurate multi-target tracking of zebrafish.

[0006] To solve the above technical problems, the present application provides a zebrafish multi-target tracking method and system based on space science experiment video, which specifically adopts the following technical solutions:

[0007] In a first aspect, the present application provides a zebrafish multi-target tracking method based on space science experiment video, which comprises: first, acquiring a zebrafish space science experiment video, which is used to represent the movement process of multiple target zebrafish in the microgravity environment of the space station. Then, extracting movement information based on the zebrafish space science experiment video, which is used to represent the motion state of the pixel points in the zebrafish space science experiment video. Next, multi-modal feature fusion is performed based on the movement information and appearance information to determine the multi-modal fusion feature, and the appearance information is the video frame image of the zebrafish space science experiment video. Finally, target detection and tracking are performed according to the multi-modal fusion feature to determine the target motion trajectory information corresponding to the multiple target zebrafish respectively.

[0008] The method extracts motion feature information representing dynamic changes through adaptive threshold-based motion modeling, and then uses a heterogeneous graph network to model the interaction and fusion relationship between appearance and motion features at each level, thereby enhancing the robustness of target detection, data association, and trajectory prediction. According to the multi-modal fusion features, detection and displacement prediction, trajectory matching, and confidence fusion are integrated to realize integrated target detection and tracking, and finally determine the target motion trajectory information corresponding to each of the multiple target zebrafish. In this way, the method can effectively improve the stability and accuracy of multi-target tracking of zebrafish.

[0009] In combination with the first aspect, in an alternative implementation, the above motion information extraction based on the zebrafish space science experiment video includes: first, motion modeling is performed according to the zebrafish space science experiment video to determine the first estimated displacement of each pixel point in the video frame image of the zebrafish space science experiment video, and the first estimated displacement is in the Cartesian coordinate system. Then, polar coordinate conversion is performed on the first estimated displacement of each pixel point to obtain the second estimated displacement of each pixel point, and the second estimated displacement is used to represent the motion amplitude and motion direction of the pixel point between two adjacent video frame images. Finally, adaptive threshold filtering is performed based on the second estimated displacement of each pixel point to determine the motion information corresponding to the pixel point.

[0010] In combination with the first aspect, in an alternative implementation, the above motion modeling according to the zebrafish space science experiment video to determine the first estimated displacement of each pixel point in the video frame image of the zebrafish space science experiment video includes: first, the image intensity of each pixel point in the first neighborhood is determined according to the zebrafish space science experiment video. Then, the first estimated displacement corresponding to each pixel point is determined by a minimum error estimation algorithm based on the image intensity of each pixel point in the first neighborhood.

[0011] In combination with the first aspect, in an alternative implementation, the above adaptive threshold filtering based on the second estimated displacement of each pixel point to determine the motion information corresponding to the pixel point includes: first, the motion amplitude mean and motion amplitude standard deviation of the pixel points in the first neighborhood are determined according to the motion amplitude in the second estimated displacement of each pixel point. Then, the first filtering threshold and the second filtering threshold of the pixel points in the first neighborhood are determined according to the motion amplitude mean and motion amplitude standard deviation of the pixel points in the first neighborhood. Wherein, the second filtering threshold is greater than the first filtering threshold, and the expression of the first filtering threshold is:

[0012] ;

[0013] Wherein, represents the first filtering threshold, represents the motion amplitude mean, represents the motion amplitude standard deviation, represents the first experience control value. The expression of the second filtering threshold is:

[0014] ;

[0015] wherein, represents the second filtering threshold, represents the second experience control value. Finally, according to the motion amplitude corresponding to the pixel point and the motion state represented by the pixel point according to the first filtering threshold and the second filtering threshold in the first neighborhood in which the pixel point is located, the motion information corresponding to the pixel point is determined, and the expression of the motion information is:

[0016] ;

[0017] wherein, represents the motion information corresponding to the pixel point, represents the motion amplitude corresponding to the pixel point, in the case of , the pixel point represents a static background; in the case of , the pixel point represents a target zebrafish with an uncertain motion state; in the case of , the pixel point represents a target zebrafish in a motion state.

[0018] In combination with the first aspect, in an alternative implementation, the above multi-modal feature fusion based on the motion information and the appearance information to determine the multi-modal fusion feature includes: first, respectively extracting features from the motion information and the appearance information to obtain a two-dimensional motion feature vector and a two-dimensional appearance feature vector. Then, a heterogeneous graph is constructed based on the two-dimensional motion feature vector and the two-dimensional appearance feature vector, the heterogeneous graph including a node set and an edge set, and the expression of the heterogeneous graph is:

[0019] ;

[0020] ;

[0021] ;

[0022] wherein, represents the heterogeneous graph corresponding to the kth video frame image, represents the node set corresponding to the kth video frame image, represents the appearance feature vector corresponding to the kth video frame image, represents the motion feature vector corresponding to the kth video frame image, ​indicates motion trajectory information determined based on appearance features corresponding to the image of the k-1th frame of video frames, indicates motion trajectory information determined based on motion features corresponding to the image of the k-1th frame of video frames, indicates a time edge, for connecting and , and connecting and ; indicates a space edge, indicates a heterogeneous edge, for connecting and . Secondly, based on the heterogeneous graph, attention encoding is performed through a time attention mechanism, a space attention mechanism and a heterogeneous attention mechanism to obtain aggregated encoding features. Finally, the aggregated encoding features are deformable decoded through a deformable attention mechanism to obtain multi-modal fusion features.

[0023] In combination with the first aspect, in an alternative implementation, the target detection and tracking based on the multi-modal fusion features to determine target motion trajectory information corresponding to each of the plurality of target zebrafish includes: first, based on the multi-modal fusion features, a convolutional layer is used to process to obtain center heat map information, bounding box coordinates and refinement offset values, respectively. The center heat map information is used to represent the confidence of the presence of a target zebrafish in the video frame image. Then, a target tracking query vector is used to predict the predicted displacement of the target zebrafish. The target tracking query vector includes: identity information of the target zebrafish, a current frame appearance feature vector, a current frame motion feature vector, a historical appearance feature vector, a historical motion feature vector and historical motion trajectory information. Next, the predicted position coordinate value of the target zebrafish is determined according to the predicted displacement. Secondly, based on the center heat map information, the bounding box coordinates, the refinement offset values and the target tracking query vector, the predicted position coordinate value is associated and matched with the historical motion trajectory information to determine an associated matching result. Finally, the target motion trajectory information corresponding to each of the plurality of target zebrafish is determined based on the associated matching result.

[0024] In combination with the first aspect, in an alternative implementation, the expression of the center heat map information is:

[0025] ;

[0026] wherein, indicates the center heat map information corresponding to the kth frame of video frame images, indicates the multi-modal fusion features, indicates a convolutional layer, indicates a Sigmoid activation function, indicates a ReLU activation function. The expression of the bounding box coordinates is:

[0027] ;

[0028] wherein, represents the bounding box coordinates corresponding to the k-th frame of video frame image. The expression of the refined offset value is:

[0029] ;

[0030] wherein, represents the refined offset value corresponding to the k-th frame of video frame image, represents the Tanh activation function. The expression of the predicted displacement is:

[0031] ;

[0032] wherein, represents the predicted displacement, represents the target tracking query vector, represents the first learning weight, represents the second learning weight, represents the first bias term, represents the second bias term. The expression of the predicted position coordinate value is:

[0033] ;

[0034] wherein, represents the predicted position coordinate value of the target zebrafish in the k-th frame of video frame image, represents the predicted position coordinate value of the target zebrafish in the k-1-th frame of video frame image, , represents the Gaussian noise.

[0035] In combination with the first aspect, in an alternative implementation manner, the above-mentioned association matching result is a cross-frame affinity matrix, the cross-frame affinity matrix is used to represent the matching probability of the predicted position coordinate value and the historical motion trajectory information, and the expression of the cross-frame affinity matrix is:

[0036] ;

[0037] wherein, represents the cross-frame affinity matrix, represents the matching score of the predicted position coordinate value of the i-th target zebrafish and the j-th historical motion trajectory information, The target motion trajectory information corresponding to each of the plurality of target zebrafish based on the association matching result is determined based on the center heat map information, the bounding box coordinates, the refined offset value and the target tracking query vector. Specifically, the target motion trajectory information is determined based on the following steps. First, a target pairing combination with the maximum matching probability is determined by using the Hungarian algorithm or the KM algorithm. The target pairing combination includes a set of predicted position coordinate values and corresponding historical motion trajectory information. Then, when the matching probability of the target pairing combination is greater than a matching threshold, the predicted position coordinate values and the corresponding historical motion trajectory information in the target pairing combination are associated and updated to determine the target motion trajectory information.

[0038] In combination with the first aspect, in an alternative implementation, the center heat map information includes an original detection confidence. The target motion trajectory information corresponding to each of the plurality of target zebrafish is determined based on the target detection and tracking based on the multi-modal fusion feature, and further includes refining the original detection confidence based on the center heat map information to obtain a refined detection confidence. The expression of the refined detection confidence is as follows:

[0039] ;

[0040] ;

[0041] wherein, represents the refined detection confidence corresponding to the i-th target zebrafish in the k-th video frame image, represents the original detection confidence corresponding to the i-th target zebrafish in the k-th video frame image, is a fusion intensity coefficient, represents a harmonic mean of the modal-specific confidence corresponding to the i-th target zebrafish in the k-th video frame image, represents the confidence corresponding to the appearance modal in the k-th video frame image, represents the confidence corresponding to the motion modal in the k-th video frame image, represents a minimum value.

[0042] In a second aspect, the present application provides a zebrafish multi-target tracking system based on space science experiment video, comprising: an acquisition module, a motion information extraction module, a multi-modal feature fusion module, and a target detection and tracking module. The acquisition module can be used to acquire a zebrafish space science experiment video, which is used to represent the motion process of multiple target zebrafish in a space station microgravity environment. The motion information extraction module can be used to extract motion information based on the zebrafish space science experiment video, which is used to represent the motion state of a pixel point in the zebrafish space science experiment video. The multi-modal feature fusion module can be used to perform multi-modal feature fusion based on the motion information and appearance information, determine multi-modal fusion features, and the appearance information is a video frame image of the zebrafish space science experiment video. The target detection and tracking module can be used to perform target detection and tracking according to the multi-modal fusion features, and determine target motion trajectory information corresponding to the multiple target zebrafish respectively.

[0043] In a third aspect, the present application provides an electronic device, comprising: a memory, one or more processors; the memory is coupled with the processor; wherein the memory stores computer program code, the computer program code comprises computer instructions, when the computer instructions are executed by the processor, the electronic device executes the method provided by the first aspect and any one of the optional implementation manners thereof.

[0044] In a fourth aspect, the present application provides a computer readable storage medium, comprising computer instructions, when the computer instructions run on the electronic device, the electronic device executes the method provided by the first aspect and any one of the optional implementation manners thereof.

[0045] It can be understood that the beneficial effects that can be achieved by the zebrafish multi-target tracking system based on space science experiment video provided by the second aspect, the electronic device of the third aspect, and the computer readable storage medium of the fourth aspect can refer to the beneficial effects in the first aspect and any one of the possible design manners, which will not be described here. BRIEF DESCRIPTION OF DRAWINGS

[0046] Figure 1 A principle schematic diagram of the zebrafish multi-target tracking method based on space science experiment video provided by the embodiments of the present application;

[0047] Figure 2 A flowchart of the zebrafish multi-target tracking method based on space science experiment video provided by the embodiments of the present application;

[0048] Figure 3 A structural relationship schematic diagram of a heterogeneous graph provided by the embodiments of the present application;

[0049] Figure 4A structural schematic diagram of a zebrafish multi-target tracking system based on a space science experiment video is provided for an embodiment of the present application. DETAILED DESCRIPTION

[0050] The embodiments will be described in detail below with reference to examples thereof as illustrated in the accompanying drawings. In the following description, the same numbers are used to denote the same elements throughout the different drawings. The embodiments described in the following examples do not represent all the implementations consistent with the present application. They are merely examples of systems and methods consistent with some aspects of the present application as detailed in the claims.

[0051] With the rapid development of space station technology, as a space laboratory, the space station has unique advantages such as covering the whole discipline field of space science, strong on-orbit support capability, supporting human participation and up-down transportation, and can carry out large-scale multidisciplinary space science research. At present, the life ecological experiment cabinet carried by the space station can obtain zebrafish experimental observation video (i.e. space science experiment video), which provides an important data basis for studying the influence of space environment on the behavior and physiological response of organisms. In this context, it is urgent to use multi-target tracking technology to realize accurate positioning, identity maintenance and motion trajectory generation of multiple zebrafish targets under the condition of given space science experiment video input, so as to provide reliable technical support for in-depth analysis.

[0052] Generally, multi-target tracking technology can be divided into two types of schemes: tracking after detection and joint detection and tracking. Among them, the core of tracking after detection is to decouple the tracking task into two stages of target detection and data association, and to improve the overall performance by optimizing the detection accuracy and association strategy independently. For example, SORT first separates detection and association, laying the foundation for the two-stage framework, predicting the motion state of the target based on Kalman filtering, and matching the detection frame and the trajectory based on the Hungarian algorithm, which is lightweight and efficient. DeepSORT introduces appearance feature matching based on SORT, solves the problem of target occlusion and similar appearance interference, and combines motion model and appearance feature to improve the robustness of complex scenes. MOTDT combines detection results and tracking prediction, and improves the association accuracy through multi-cue fusion (such as appearance feature and motion feature). IOU ByteTrack only uses the IoU (intersection over union) of the detection frame for association, which is simple and efficient, but sensitive to target motion speed and occlusion. This kind of method has two shortcomings. First, the data association discards the image appearance feature or needs to calculate the expensive feature extractor. Second, it breaks the integration process of detection and tracking.

[0053] Joint detection and tracking converts detectors into trackers and integrates both tasks in the same framework. For example, Tracktor directly predicts the location of the target in the next frame using the regression head of the detector Faster R-CNN, without additional association steps. FairMOT is based on CenterNet, which uses an anchor-free detection method, and jointly learns detection and re-identification features, achieving a good balance between speed and accuracy. TransTrack is based on the Transformer architecture and uses self-attention mechanisms to jointly model detection and tracking. These methods use the tracking box as a region proposal for the detector, which requires the assumption that the bounding box has a large overlap between frames, which is not true in scenarios where the target moves quickly.

[0054] For example, related technology one proposes a MOTDT method based on a detection-then-association framework, which combines detection and tracking results into candidates and selects the optimal candidate based on a deep neural network to deal with unreliable detection interference in tracking. A hierarchical data association strategy is further designed to improve tracking performance. The overall idea and basic process of this method includes real-time target classification, trajectory confidence scoring, appearance feature representation, and hierarchical data association. Related technology two proposes a FairMOT method based on a joint detection and tracking framework, which converts multi-target tracking into a pixel-level key point estimation and identity classification problem on a high-resolution feature map. Through a single-stage end-to-end architecture and feature alignment design, it solves the speed bottleneck and identity switching problem of traditional two-stage methods. The overall idea and basic process of this method includes high-resolution feature extraction, detection branch, re-identification branch, and boundary box temporal association.

[0055] However, most of the above-mentioned related multi-target tracking methods are designed for general public datasets and are suitable for multi-target tracking tasks in general scenarios. However, when applied to the analysis of zebrafish space science experiment videos in space station environments, these methods face significant adaptability bottlenecks. On the one hand, general methods are difficult to cope with a series of unique and complex challenges in this task; on the other hand, they also fail to effectively tap into and utilize the unique advantages of zebrafish space science experiment videos at the data level. When applied to the analysis of zebrafish space science experiment videos in space stations, there are significant adaptability bottlenecks, making it difficult to meet the precise tracking requirements, and the specific technical problems include:

[0056] (1) Severe complex background interference: Due to the limited arrangement of light sources in space experiment environments, strong reflection and diffuse reflection phenomena often occur, causing multiple reflections of zebrafish individuals on the container wall or water surface. These reflections are highly similar to the real target, causing false matches and target splitting in the target detection and tracking process, severely affecting tracking continuity and accuracy.

[0057] (2) Special behavior patterns in microgravity environment: Under the condition of space microgravity, zebrafish exhibit strong stress response in the early stage of the experiment, showing frequent high-speed swimming, irregular turning and group gathering behavior. This highly dynamic and intensive interaction mode of movement easily leads to frequent occlusion and overlap between targets, increasing the difficulty of target segmentation and identity maintenance.

[0058] (3) Significant dynamic changes in on-orbit experimental environment: As the experimental period progresses, bubbles are gradually generated and increased in the aquatic system, and some bubbles adhere to the observation window or float in the field of view, forming dynamic occlusion areas. This not only reduces the visibility of the target, but also may cause false positives or false negatives, further affecting the robustness of the tracking algorithm.

[0059] It can be seen that the related multi-target tracking method cannot stably and accurately realize multi-target tracking of zebrafish, and thus it is difficult to meet the scientific needs of trajectory accuracy and long-term identity consistency for zebrafish behavior analysis in space stations.

[0060] In order to solve the above problems, the embodiment of the present application provides a zebrafish multi-target tracking method and system based on space science experiment video. The method and system can be applied to zebrafish observation experiment videos collected in a stable scene under a fixed field of view in a microgravity environment, for example, space science experiment videos collected in zebrafish observation experiments in space stations. The method first extracts motion feature information representing dynamic changes based on adaptive threshold motion modeling, and then uses a heterogeneous graph network to model the interaction and fusion relationship of appearance and motion features at each level, enhancing the robustness of target detection, data association and trajectory prediction.

[0061] Specifically, Figure 1 The principle diagram of the zebrafish multi-target tracking method based on space science experiment video provided by the embodiment of the present application is shown in Figure 1 As shown, the method first extracts motion information based on zebrafish space science experiment video through motion modeling, polar coordinate conversion and adaptive threshold filtering. Then, based on the motion information and appearance information, multi-modal feature fusion is realized through feature extraction, heterogeneous graph construction, attention encoding and deformable decoding in turn, to determine the multi-modal fusion features. Finally, integrated target detection and tracking are realized through detection and displacement prediction, trajectory matching and confidence fusion according to the multi-modal fusion features, and finally the target motion trajectory information corresponding to each target zebrafish is determined. The method can effectively improve the stability and accuracy of multi-target tracking of zebrafish.

[0062] The scheme provided by the embodiment of the present application will be described below with reference to the accompanying drawings.

[0063] Specifically, Figure 2A flowchart of a zebrafish multi-target tracking method based on a space science experiment video is shown in FIG. 1, which is provided in the embodiments of the present application. The zebrafish multi-target tracking method based on a space science experiment video provided in the embodiments of the present application includes the following steps S101-S104. Figure 2

[0064] S101, acquiring a zebrafish space science experiment video.

[0065] In the embodiments of the present application, the zebrafish space science experiment video can be an experimental observation video collected during zebrafish observation experiments in a life ecological experiment cabinet carried on a space station. The zebrafish space science experiment video can be used to represent the movement process of multiple target zebrafish in the microgravity environment of the space station. Specifically, the zebrafish space science experiment video is composed of continuous multiple video frame images.

[0066] S102, extracting motion information based on the zebrafish space science experiment video.

[0067] In the embodiments of the present application, the motion information of the zebrafish space science experiment video acquired in S101 is first extracted, which can be used to represent the motion state of the pixel points in the zebrafish space science experiment video. In this way, the motion pattern of the zebrafish in the zebrafish space science experiment video and noise artifacts are effectively separated based on the motion information, and complementary semantic enhancement is provided for the appearance features.

[0068] In some embodiments, motion modeling, polar coordinate conversion and adaptive threshold filtering can be used to extract reliable motion information for subsequent multi-modal fusion. Specifically, S102 can include the following steps S1021-S1023.

[0069] S1021, motion modeling is performed according to the zebrafish space science experiment video to determine a first estimated displacement corresponding to each pixel point in the video frame image of the zebrafish space science experiment video. The first estimated displacement is in a Cartesian coordinate system.

[0070] In some embodiments, S1021 specifically includes:

[0071] First, the image intensity of each pixel point in the first neighborhood is determined according to the zebrafish space science experiment video.

[0072] Then, the first estimated displacement corresponding to each pixel point is determined by a minimum error estimation algorithm according to the image intensity of each pixel point in the first neighborhood.

[0073] ​Specifically, the local motion modeling is performed on the video of the zebra fish space science experiment, aiming to model how the image intensity (brightness) in a preset neighborhood around a pixel point in the video frame image of the zebra fish space science experiment changes with the spatial position. Since the movement of the target zebra fish in the small neighborhood of the pixel point in the video frame image is relatively smooth, a quadratic function can be used to describe the movement. Therefore, the local approximation of the image intensity function can be performed by using the second-order Taylor expansion. The second-order Taylor can capture the curvature information (convex, concave, edge direction, etc.) and accurately describe the local structure.

[0074] For example, for the pixel position in the video frame image, the expression of the image intensity is:

[0075] ;

[0076] wherein, represents the local offset relative to the center pixel point in a preset neighborhood, that is, the intensity distribution in the small neighborhood with the center pixel point as the center. For example, the center point is set as the origin (0, 0) in the embodiment of the application, and are the displacements relative to the center origin (0, 0). In the formula, c is the intensity value of the video frame image at the center point (0, 0) (that is, the constant term).

[0077] is the second-order derivative matrix (for the partial derivative of and ). is the second-order partial derivative of the image intensity function with respect to , indicating the intensity change in the x direction, and so on. is the mixed second-order partial derivative of the image intensity function with respect to x and y, indicating the intensity change in the x direction. The second-order derivative matrix can be used to capture the curvature information of the local region of the image, such as convex, concave, edge direction, etc., so as to accurately describe the local structure. Specifically, the elements of the second-order derivative matrix have a clear numerical meaning, which can be large or small, positive or negative. The larger the value is, the more intense the curvature change in the direction is. For example, the larger the value is, the more intense the intensity curvature change in the x direction is. The positive value usually corresponds to “convex” (such as bright spots), and the negative value may correspond to “concave” (such as dark spots). When the matrix as a whole is positive (combined with eigenvalue analysis), it can represent that the local region has a convex feature; if the sign is complex, it may correspond to structures such as edges or corners.

[0078] The gradient vector represents the rate of change of image intensity in the x and y directions. The gradient vector describes the trend of image intensity change along the x and y directions within a local region; its magnitude indicates the drasticness of the change (the larger the gradient, the more pronounced the intensity change); and its direction points in the direction of the fastest increase in intensity.

[0079] The aforementioned second-order derivative matrix, being a symmetric matrix, possesses two eigenvalues ​​(λ1 and λ2). λ1 and λ2 are values ​​obtained by solving the matrix's characteristic equation, reflecting the matrix's ability to "scale" in different directions. Larger eigenvalues ​​indicate more significant curvature changes in that direction. Eigenvalues ​​with the same sign (both positive or both negative) may correspond to convex or concave structures (such as bright spots or dark spots); opposite signs indicate drastic curvature changes in two orthogonal directions, corresponding to "corner points." For example, when the two eigenvalues ​​are large and have opposite signs, it indicates that the local region exhibits strong convexity and concavity in two vertical directions, consistent with the structural characteristics of a corner point. Thus, the image intensity variation within a predetermined neighborhood can be effectively approximated using the aforementioned expression for image intensity.

[0080] Furthermore, considering the temporal variation, assuming that at time t, the image intensity within the preset neighborhood is described by the aforementioned quadratic model, and at time "t+∆t", translational motion may exist within this preset neighborhood. Since the intensity variation of the same physical point in video frames at different times is mainly affected by noise, illumination changes, occlusion, etc., a minimum error estimation method can be used, that is, the first estimated displacement can be expressed in the form of minimizing the error. :

[0081] ;

[0082] in, This represents the regularization coefficient, which prevents overfitting and improves stability.

[0083] S1022. Perform polar coordinate transformation on the first estimated displacement corresponding to each pixel to obtain the second estimated displacement corresponding to each pixel.

[0084] Specifically, the first estimated displacement determined in S1021 Using Cartesian coordinates makes it difficult to directly analyze the intensity and direction of movement of the zebrafish. Polar coordinates, however, clearly separate two key attributes: motion amplitude (representing the magnitude of displacement of the pixel, i.e., velocity), used to determine whether significant movement has occurred; and motion direction, used to analyze movement trends, such as upward or leftward.

[0085] Therefore, the first estimated displacement corresponding to each pixel point can be converted into polar coordinates to obtain a second estimated displacement corresponding to each pixel point. The second estimated displacement is used to represent the motion amplitude and motion direction of the pixel point between two adjacent video frame images. For example, the expression of the second estimated displacement (D2) can be:

[0086]

[0087] S1023, performing adaptive threshold filtering on the second estimated displacement corresponding to each pixel point to determine the motion information corresponding to the pixel point.

[0088] Next, the second estimated displacement corresponding to each pixel point is filtered based on the adaptive threshold, i.e., classified to determine the motion type represented by the pixel point. For example, the motion type can include: suppressing static background, target zebrafish in an uncertain motion state, and target zebrafish in a determined motion state. Wherein, the adaptive threshold filtering means that different thresholds are used in different preset neighborhoods, rather than using a fixed threshold globally (for the entire video frame image).

[0089] In some embodiments, the specific implementation method of performing adaptive threshold filtering in S1023 to determine the motion information corresponding to the pixel point can include the following steps S10231-S10233:

[0090] S10231, determining the mean value and standard deviation of the motion amplitude of the pixel point in the first neighborhood according to the motion amplitude in the second estimated displacement corresponding to each pixel point.

[0091] S10232, determining the first filtering threshold and the second filtering threshold of the pixel point in the first neighborhood according to the mean value and standard deviation of the motion amplitude of the pixel point in the first neighborhood.

[0092] Wherein, the second filtering threshold is greater than the first filtering threshold. The first filtering threshold (i.e., the low threshold) can be used to preliminarily distinguish between small motion and potential motion, and the second filtering threshold (i.e., the high threshold) can be used to distinguish between certain motion, i.e., to distinguish between uncertain motion state and determined motion state.

[0093] Specifically, the expression of the first filtering threshold is:

[0094]

[0095] Wherein, D1 represents the first filtering threshold, μ represents the mean value of the motion amplitude, σ represents the standard deviation of the motion amplitude, and a represents the first empirical control value.

[0096] ​​​The expression for the second filtering threshold is:

[0097] ;

[0098] in, This indicates the second filtering threshold. This represents the second empirical control value.

[0099] For example, the first empirical control value It can be 1.0, the second empirical control value. It can be 2.5. First empirical control value. Second empirical control value Presets can be made according to actual application needs; this application does not impose specific limitations on this.

[0100] S10233. Based on the motion amplitude corresponding to the pixel and the first and second filtering thresholds within the first neighborhood of the pixel, classify the motion state represented by the pixel to determine the motion information corresponding to the pixel. The expression for the motion information is:

[0101] ;

[0102] in, Represents pixels Corresponding motion information Represents pixels The corresponding amplitude of motion; in In the case of pixels Indicates suppression of static background; in In the case of pixels This represents a target zebrafish in an uncertain state of motion; in In the case of pixels This indicates that the target zebrafish is in motion.

[0103] S103. Perform multimodal feature fusion based on motion information and appearance information to determine the multimodal fusion features.

[0104] Next, to effectively integrate the appearance and motion information of the target zebrafish in the space science experiment video, a multimodal feature fusion method based on a multimodal heterogeneous graph architecture can be adopted. This method determines the multimodal fusion features based on motion and appearance information through: feature extraction, heterogeneous graph modeling, attention encoding, and deformable decoding. The appearance information refers to video frame images from the zebrafish space science experiment video.

[0105] In some embodiments, specifically, S103 may include the following steps S1031-S1034:

[0106] S1031, respectively, feature extraction is carried out on motion information and appearance information, and two-dimensional motion feature vectors and two-dimensional appearance feature vectors are obtained.

[0107] In an implementation manner, the motion information is motion modal , and the appearance information is RGB video frame of appearance modal. Based on the above motion information and appearance information, feature extraction can be performed through a PVTv2 backbone network, which is a multi-scale output visual Transformer architecture, and feature maps are output at four down-sampling rates r e {4, 8, 16, 32}, forming a feature pyramid feature , wherein k is a time step (when (appearance) front frame), a (appearance) represents an appearance modal (i.e. the modal corresponding to the appearance information), m (motion) represents a motion modal (i.e. the modal corresponding to the motion information), and r is a spatial scale (i.e. resolution). Wherein, the size of each feature decreases with the increase of r.

[0108] In order to convert the three-dimensional feature map at each scale into a set of two-dimensional query vector sets to construct the initial nodes that can match the graph neural network, the stretching processing and linear projection are adopted in the embodiment of the application to flatten the three-dimensional matrix into two-dimensional features by the following formula, that is, two-dimensional motion feature vectors and two-dimensional appearance feature vectors are obtained.

[0109] ;

[0110] , wherein, represents a two-dimensional motion feature vector or a two-dimensional appearance feature vector, represents a flattening function, is a learnable projection weight, represents a multi-scale position encoding to explicitly add spatial position information.

[0111] S1032, constructing a heterogeneous graph based on the two-dimensional motion feature vectors and the two-dimensional appearance feature vectors, the heterogeneous graph including a node set and an edge set.

[0112] Specifically, Figure 3 is a structural relationship diagram of the heterogeneous graph provided by the embodiment of the application, as shown in Figure 3 , the expression of the heterogeneous graph is:

[0113] ;

[0114] ;

[0115] ;

[0116] , wherein, denotes a heterogeneous graph corresponding to the k-th frame of video frame image, denotes a node set corresponding to the k-th frame of video frame image, denotes an appearance feature vector corresponding to the k-th frame of video frame image, denotes a motion feature vector corresponding to the k-th frame of video frame image, denotes motion trajectory information determined based on the appearance feature corresponding to the k-1-th frame of video frame image, denotes motion trajectory information determined based on the motion feature corresponding to the k-1-th frame of video frame image, denotes a temporal edge, is used to connect and , and connect and ; denotes a spatial edge, denotes a heterogeneous edge, is used to connect and .

[0117] S1033, attention encoding is performed based on the heterogeneous graph through a temporal attention mechanism, a spatial attention mechanism and a heterogeneous attention mechanism, to obtain aggregated encoding features.

[0118] In the attention encoding stage, a context-aware query representation is generated based on the heterogeneous graph constructed in S1032 through an attention mechanism, specifically including: a temporal attention mechanism, a spatial attention mechanism and a heterogeneous attention mechanism.

[0119] Specifically, the temporal attention mechanism aims to model the temporal consistency within the modal, and the historical trajectory is used to enhance the current query, with the appearance modal encoding features as an example, which can be represented as:

[0120] ;

[0121] In the formula, indicates that the current feature is combined with the historical feature to enhance the memory. , , is a learnable parameter matrix of the time attention mechanism, is a scaling factor, is a feature dimension, which is used to prevent the gradient from being unstable due to the result of the dot product being too large. denotes the temporal dimension. Similarly, the motion modal information can be processed to obtain the motion modal encoding features .

[0122] The spatial attention mechanism aims to achieve cross-modal spatial alignment to solve the problem of incomplete overlap between appearance regions and motion regions. Specifically, a relative position bias Modeling spatial structure:

[0123] ;

[0124] wherein, represents the spatial attention feature of the motion modality to the appearance modality, can be used as an appearance query, can provide a key value for the motion modality, guiding the appearance to focus on the motion active area. Similarly, the reverse operation is: . , , is a learnable parameter matrix of the spatial attention mechanism, and s represents the spatial dimension. , represents the relative coordinate offset between the encoded nodes i, j in the heterogeneous graph.

[0125] The heterogeneous attention mechanism is used to uniformly integrate information of all modalities and types to generate a unified representation. The bidirectional enhanced features are spliced as queries to aggregate information from the motion modality and the appearance modality, obtaining aggregated encoding features :

[0126] ;

[0127] wherein, , , correspond to queries, keys and values of different modalities (for example: appearance modality and motion modality), , , is a learnable parameter matrix of the heterogeneous attention mechanism, and h represents heterogeneity.

[0128] In this way, the attention encoding through the above temporal attention mechanism, spatial attention mechanism and heterogeneous attention mechanism can effectively learn rich cross-modal representations while preserving modality-specific features to accurately determine the aggregated encoding features.

[0129] S1034, performing deformable decoding on the aggregated encoding features through a deformable attention mechanism to obtain a multi-modal fusion feature.

[0130] Further, based on the deformable attention mechanism, the aggregated encoding features determined in S1033 can be deformable decoded to obtain a multi-modal fusion feature.

[0131] Specifically, the deformable attention mechanism aims to efficiently sample key regions from the aggregated encoding features to generate the multi-modal fusion features for tracking. Specifically, the multi-modal fusion features are determined by dynamic sampling through the deformable attention mechanism :

[0132] ;

[0133] wherein Head is the number of attention heads, which can be set to 8, for example. is the number of sampling points for each attention head, which can be set to 4, for example. is a learnable normalized reference point coordinate, with a value range of 0-1, is a learnable offset, S represents bilinear interpolation sampling, is a learnable parameter matrix.

[0134] In this way, the robust integration of appearance features and motion features can be achieved through the above S1031-S1034, thereby providing an important feature basis for accurate and context-aware tracking of target zebrafish in a complex spatial environment.

[0135] S104, target detection and tracking are performed according to the multi-modal fusion features to determine target motion trajectory information corresponding to a plurality of target zebrafish respectively.

[0136] Finally, target detection and tracking can be performed according to the multi-modal fusion features determined in S103 to determine target motion trajectory information. The present application adopts an end-to-end trainable architecture to integrate target detection and tracking into one, thereby realizing real-time, accurate and robust tracking of target zebrafish in a spatial environment. Detection and tracking share multi-modal features, detection output provides initial target hypotheses, and tracking queries maintain inter-frame temporal continuity.

[0137] In some embodiments, specifically, S104 can include the following steps S1041-S1045:

[0138] S1041, center heat map information, bounding box coordinates and refining offset values are respectively obtained by processing the multi-modal fusion features through a convolution layer.

[0139] In the detection stage, the multi-modal fusion features are first reshaped and processed through a series of convolution layers to generate center heat map information, bounding box coordinates and refining offset values. The center heat map information is used to represent the confidence of the presence of target zebrafish in the video frame image.

[0140] In some embodiments, the expression of the above center heat map information is:

[0141] ;

[0142] wherein represents the center heat map information corresponding to the k-th frame of video frame image, represents the multi-modal fusion feature, represents the convolution layer, represents the Sigmoid activation function, represents the ReLU activation function.

[0143] The expression of the above bounding box coordinates is:

[0144] ;

[0145] wherein, represents the bounding box coordinates corresponding to the k-th frame of video frame image.

[0146] The expression of the above fine-tuning offset value is:

[0147] ;

[0148] wherein, represents the fine-tuning offset value corresponding to the k-th frame of video frame image, represents the Tanh activation function.

[0149] S1042, predicting the predicted displacement of the target zebrafish based on the target tracking query vector.

[0150] Then, in the target tracking stage, the motion trajectory of the existing target is predicted, and the predicted displacement of the target zebrafish is predicted by learning the target tracking query vector. The target tracking query vector is a core variable for maintaining the inter-frame time sequence continuity of the target, and its essence is a learnable feature vector. Specifically, the target tracking query vector includes: identity information of the target zebrafish, current frame appearance feature vector, current frame motion feature vector, historical appearance feature vector, historical motion feature vector and historical motion trajectory information. The initial value of the target tracking query vector can be set to 0, and with the training and tracking process, it will be updated according to the detection result and motion prediction, and gradually accumulate the key features of the target.

[0151] In some embodiments, the expression of the above predicted displacement is:

[0152] ;

[0153] wherein, represents the predicted displacement, represents the target tracking query vector, represents the first learning weight, represents the second learning weight, represents the first bias term, represents the second bias term.

[0154] S1043, determining a predicted position coordinate value of the target zebrafish according to the predicted displacement.

[0155] Next, the predicted position coordinate value of the target zebrafish can be determined based on the predicted displacement determined in S1042 and the historical position coordinate value. In some embodiments, the expression of the predicted position coordinate value is as follows:

[0156] ;

[0157] wherein, represents the predicted position coordinate value of the target zebrafish in the k-th video frame image, represents the predicted position coordinate value of the target zebrafish in the k-1-th video frame image, , represents the Gaussian noise.

[0158] wherein, for when (the first frame), the bounding box coordinates output by the detection stage combined with the refining offset value determined, is the initial detection position of the target zebrafish.

[0159] when , is obtained through the tracking process of the k-1-th frame, that is, determined by adding the position of the k-2-th frame the displacement of the k-1 | k-2-th frame and the Gaussian noise

[0160] S1044, associating and matching the predicted position coordinate value with the historical motion trajectory information based on the center heat map information, the bounding box coordinates, the refining offset value and the target tracking query vector, to determine an association matching result.

[0161] Further, in the data association stage, the detection target can be matched with the existing trajectory, that is, the predicted position coordinate value is associated and matched with the historical motion trajectory information.

[0162] In some embodiments, to associate the predicted position coordinate value with the historical motion trajectory information, a cross-frame affinity matrix can be calculated to represent the matching probability of the detection and tracking query. Then the association matching result can be the cross-frame affinity matrix, which can be used to represent the matching probability of the predicted position coordinate value and the historical motion trajectory information. The expression of the cross-frame affinity matrix is as follows:

[0163] ;

[0164] wherein, represents a cross-frame affinity matrix, represents a matching score of the predicted position coordinate value of the i-th target zebrafish and the j-th historical motion trajectory information, is determined based on the center heat map information, the bounding box coordinate, the refining offset value, and the target tracking query vector.

[0165] The above The similarity between the new detection result (center heat map information, bounding box coordinate, refining offset value) and the target tracking query can be obtained by calculation. Specifically, it includes appearance similarity, motion similarity, and position similarity. The appearance similarity is the matching degree between the appearance features of the detected target (such as the target existence probability reflected by the center heat map information and the region features corresponding to the bounding box coordinate) and the current frame appearance feature vector associated with the trajectory. The motion similarity is calculated according to the consistency between the predicted motion trend of the detected target (such as the position change of the bounding box coordinate) and the historical motion features (such as displacement and speed) of the existing trajectory. The closer the spatial distance between the bounding box center of the detected target and the predicted position of the existing trajectory in the current frame, the higher the score.

[0166] S1045, determining target motion trajectory information corresponding to each of the plurality of target zebrafishes based on the association matching result.

[0167] In some embodiments, S1045 specifically includes:

[0168] First, the target pairing combination with the maximum matching probability is determined by the Hungarian algorithm or the KM algorithm, and the target pairing combination includes a set of predicted position coordinate values and corresponding historical motion trajectory information.

[0169] Then, in the case that the matching probability of the target pairing combination is greater than the matching threshold, the predicted position coordinate value and the corresponding historical motion trajectory information in the target pairing combination are associated and updated to determine the target motion trajectory information.

[0170] In one implementation, in the case that the matching probability of the target pairing combination is less than or equal to the matching threshold, the target zebrafish corresponding to the predicted position coordinate value can be determined as a new target, and a new trajectory is created. For the unmatched historical motion trajectory information, it can be temporarily retained, and if it is not matched for consecutive multiple frames, the historical motion trajectory information is deleted, and is regarded as the target zebrafish disappearing.

[0171] In some embodiments, the center heat map information comprises: original detection confidence. Then, S104, target detection and tracking are performed according to the multi-modal fusion feature, and target motion trajectory information corresponding to each of the plurality of target zebrafish is determined, and the process further comprises: refining the original detection confidence based on the center heat map information to obtain refined detection confidence, and the expression of the refined detection confidence is:

[0172] ;

[0173] ;

[0174] wherein, represents refined detection confidence corresponding to the i-th target zebrafish in the k-th video frame image, represents original detection confidence corresponding to the i-th target zebrafish in the k-th video frame image, is a fusion intensity coefficient, and the exemplary fusion intensity coefficient may be 2. represents harmonic mean of modal-specific confidence corresponding to the i-th target zebrafish in the k-th video frame image, represents confidence corresponding to the appearance modal in the k-th video frame image, represents confidence corresponding to the motion modal in the k-th video frame image, represents a minimum value.

[0175] Specifically, in order to further improve the target detection reliability, a cross-modal confidence fusion strategy is introduced, and the original detection confidence is refined by using the complementary advantages of appearance and motion clues to obtain refined detection confidence. Specifically, the original detection confidence is mainly based on the appearance feature (center heat map), and the refined process introduces the harmonic of modal-specific confidence (which can reflect the reliability of the motion modal), so that the confidence is more comprehensive. Moreover, it can effectively reduce the false detection rate: by adjusting the fusion intensity coefficient , the detection results with low reliability are suppressed, and the results with high reliability are enhanced, so that the final detection confidence is closer to the probability of the existence of the target.

[0176] In some embodiments, a multi-task loss function can be used to jointly optimize the processes of target detection, tracking and association in S104, and the expression of the specific multi-task loss function is:

[0177] ;

[0178] wherein, is a center focus loss for punishing heat map prediction error, To measure the bounding box regression accuracy using SmoothL1 loss, The target center refinement can be ensured, The predicted and real motion displacement are aligned by IoU, The data association matrix is optimized by binary cross-entropy.

[0179] In this way, the long-term target identity consistency can be maintained by joint optimization in the multi-task loss function, while the dynamic visual problems in the zebrafish multi-target tracking scene in the space science experiment are adapted to.

[0180] The zebrafish multi-target tracking method based on the space science experiment video provided in the embodiments of the present application comprises the following steps: first, acquiring a zebrafish space science experiment video. Then, extracting motion information based on the zebrafish space science experiment video. Next, performing multi-modal feature fusion based on the motion information and appearance information to determine multi-modal fusion features. Finally, performing target detection and tracking according to the multi-modal fusion features to determine target motion trajectory information corresponding to each of the multiple target zebrafishes.

[0181] The method extracts motion information, and models the depth interaction and fusion of the motion information and the appearance information (original RGB frame information) through a heterogeneous graph network, thereby enhancing the robustness of target detection, data association, and trajectory prediction. Specifically, the motion of zebrafish in a complex background interference in a microgravity environment is nonlinear and random, and the linear motion assumption of traditional Kalman filtering can cause trajectory prediction deviation problems. The method extracts reliable motion information through motion modeling, polar coordinate conversion, and adaptive threshold filtering, which provides complementary semantic enhancement for appearance features. The appearance of zebrafish is highly similar and there is occlusion, and a single modality (such as RGB) can easily cause identity confusion (IDSW) and cause modal noise to be superimposed by directly adding features. The method uses a cross-modal fusion method of a graph network structure, extracts motion-guided RGB features, enhances the robustness of motion blur area detection, and maintains the temporal consistency of identity during occlusion or rapid motion by introducing a cross-modal confidence fusion module. In this way, the method can effectively improve the stability and accuracy of multi-target tracking of zebrafish.

[0182] In some embodiments, the performance and effectiveness of the zebrafish multi-target tracking method based on the space science experiment video provided in the above embodiments of the present application are verified and analyzed.

[0183] Experimental dataset: The validation analysis is based on the SpaceAnimal public dataset, which is based on the space station life ecological experiment cabinet zebrafish video data. A representative time period is selected, and a zebrafish multi-target tracking dataset is finely annotated and prepared, containing 1,757 frames of images and 7,028 individual instances.

[0184] Evaluation method: The evaluation indexes for testing the performance of the zebrafish multi-target tracking method based on space science experiment video in the technical solution include MOTA (multi-target tracking accuracy), IDF1 (identity F1 score), and Frag (total number of track interruptions). The specific calculation methods of MOTA and IDF1 are as follows:

[0185] ;

[0186] ;

[0187] MOTA is normalized by aggregating frame-by-frame errors (including false positives (FP), false negatives (FN), and identity switching (IDSW)) and the total number of true targets (GT) to reflect the overall tracking accuracy. The higher the MOTA score, the better the matching degree between the predicted trajectory and the true trajectory. IDF1 measures the accuracy of identity retention based on identity true positives (IDTP), identity false positives (IDFP), and identity false negatives (IDFN). The closer the IDF1 value is to 1, the stronger the consistency of individual target zebrafish identification between different frames. The lower the Frag, the more continuous and stable the tracker can maintain the trajectory.

[0188] The following respectively adopts the zebrafish multi-target tracking method based on space science experiment video provided by the above-mentioned related technology one, related technology two and the embodiment of the present application (hereinafter referred to as: the present scheme) to conduct zebrafish multi-target tracking accuracy experiment comparison on zebrafish space science experiment video experimental dataset. Table 1 is the zebrafish multi-target tracking accuracy experiment result.

[0189] Table 1 Zebrafish multi-target tracking experiment results

[0190]

[0191] As shown in Table 1, the present scheme exceeds the related technical scheme in MOTA (82.21%), IDF1 (74.26%) and Frag (36) and the like. Among them, the excellent performance of MOTA shows that the present scheme has stronger robustness in dealing with background interference and complex motion mode of the target, and has significant advantages in dealing with high-density and fast-moving targets in microgravity environment. In terms of identity preservation ability, the present scheme is more than 10% higher than the other two schemes in IDF1 index, which fully proves that the cross-modal re-detection mechanism proposed effectively maintains the consistency of the target identity when dealing with occlusion and rapid motion. In terms of trajectory continuity, the present scheme realizes the lowest Frag value of 36, which is significantly better than the related technology one (Frag value: 85) and the related technology two (Frag value: 82), highlighting its ability to generate smooth and uninterrupted trajectories, which is crucial for capturing zebrafish clusters and social interactions and the like. It can be seen that the present scheme breaks through the limitations of other existing technical schemes relying on a single visual feature, and even if the appearance of zebrafish is highly similar, the uniqueness of the motion trajectory can still guarantee the accuracy and stability of tracking.

[0192] Further, in some embodiments, ablation experiment analysis is performed to verify the role of each step in the present scheme. By adding the extracted motion information, constructing the heterogeneous graph, and cross-modal confidence fusion step by step, the influence of each step on the accuracy of zebrafish multi-target tracking is analyzed, and the ablation experiment results are shown in Table 2. In Table 2, “√” indicates that the corresponding step method is adopted, and “-” indicates that the corresponding step method is not adopted.

[0193] Table 2 Ablation experiment results of zebrafish multi-target tracking

[0194]

[0195] As shown in Table 2, the MOTA of the baseline method is 74.12%, the IDF1 is 60.14%, and the Frag is 85. Although the detection effect is acceptable, the identity consistency and robustness are still insufficient in the complex motion scene under microgravity, and there are frequent trajectory interruption phenomena.

[0196] After adding the extracted motion information, the tracking accuracy and identity preservation ability are improved, the MOTA reaches 79.09% (+4.97%), the IDF1 is improved to 67.79% (+7.65%). The Frag index is also improved to 69, indicating that the trajectory loss is less and the tracking is more complete. These results show that extracting motion information helps to improve the motion estimation and tracking stability in dynamic environment.

[0197] Further, the heterogeneous graph is constructed, and the performance is further improved by fusing appearance and motion features. MOTA is 80.74% (+1.65%), IDF1 reaches 71.22% (+3.43%). Frag is optimized to 47, indicating that the target association ability is enhanced, and the trajectory loss is less. These results verify that the construction of the heterogeneous graph enhances the feature expression ability and robustness under the conditions of occlusion and visual blur.

[0198] Finally, cross-modal confidence fusion is introduced to maintain identity and recover lost trajectories during occlusion or rapid motion. Cross-modal confidence fusion has further performance improvement, MOTA improves to 82.21% (+1.47%), IDF1 reaches 74.26% (+3.04%). Frag index reaches 36, and it can be seen that cross-modal confidence fusion can effectively deal with the problems of identity drift and trajectory loss.

[0199] The embodiment of the application also provides a zebrafish multi-target tracking system based on a space science experiment video, specifically, Figure 4 The structure diagram of the zebrafish multi-target tracking system based on the space science experiment video provided by the embodiment of the application is shown in Figure 4 As shown in the figure, the zebrafish multi-target tracking system 400 based on the space science experiment video includes an acquisition module 401, a motion information extraction module 402, a multi-modal feature fusion module 403, and a target detection and tracking module 404.

[0200] The acquisition module 401 can be used to acquire a zebrafish space science experiment video, and the zebrafish space science experiment video is used to represent the motion process of multiple target zebrafish in the microgravity environment of the space station.

[0201] The motion information extraction module 402 can be used to extract motion information based on the zebrafish space science experiment video, and the motion information is used to represent the motion state of the pixel points in the zebrafish space science experiment video.

[0202] The multi-modal feature fusion module 403 can be used to perform multi-modal feature fusion based on the motion information and the appearance information, determine the multi-modal fusion features, and the appearance information is the video frame image of the zebrafish space science experiment video.

[0203] The target detection and tracking module 404 can be used to perform target detection and tracking according to the multi-modal fusion features, and determine the target motion trajectory information corresponding to the multiple target zebrafish respectively.

[0204] The zebrafish multi-target tracking system based on the spatial science experiment video provided in the embodiment of the application first acquires the zebrafish spatial science experiment video through an acquisition module. Then, motion information is extracted based on the zebrafish spatial science experiment video through a motion information extraction module. Next, multi-modal feature fusion is performed based on the motion information and appearance information to determine multi-modal fusion features. Finally, target detection and tracking are performed according to the multi-modal fusion features through a target detection and tracking module to determine target motion trajectory information corresponding to each of the multiple target zebrafish. The system extracts motion feature information representing dynamic changes based on adaptive threshold motion modeling, and then uses a heterogeneous graph network to model the interaction and fusion relationship between appearance and motion features at different levels, thereby enhancing the robustness of target detection, data association and trajectory prediction. In this way, the method can effectively improve the stability and accuracy of multi-target tracking of zebrafish.

[0205] The embodiment of the application also provides an electronic device, which can include a display screen, a memory and one or more processors. The display screen, the memory and the processor are coupled. The memory is used to store computer program code, which includes computer instructions. When the processor executes the computer instructions, the electronic device can execute each method or step performed in the above-mentioned zebrafish multi-target tracking method based on spatial science experiment video embodiment. Of course, the electronic device includes but is not limited to the above-mentioned display screen, memory and one or more processors.

[0206] The embodiment of the application also provides a computer readable storage medium for storing computer instructions for running the above-mentioned zebrafish multi-target tracking method based on spatial science experiment video.

[0207] Through the description of the above embodiments, those skilled in the art can clearly understand that, for the convenience and brevity of description, only the division of the above functional modules is exemplified, and in actual application, the above functions can be completed by different functional modules according to needs, that is, the internal structure of the device is divided into different functional modules to complete all or part of the functions described above.

[0208] In the description of the application, it should be understood that the terms "first", "second" are only for descriptive purposes, and cannot be understood as indicating or implying relative importance or implicitly indicating the number of indicated technical features. Therefore, the features defined as "first", "second" can explicitly or implicitly include at least one of the features. In the description of the application, the meaning of "multiple" is at least two, such as two, three, etc., unless otherwise specifically limited.

[0209] In the description of the specification, the description of the terms "one embodiment", "some embodiments", "an example", "a specific example", or "some examples" etc. means that the specific features, structures, materials or characteristics described in connection with the embodiment or example are included in at least one embodiment or example of the present application. In the specification, the illustrative description of the above terms does not necessarily refer to the same embodiment or example. Moreover, the specific features, structures, materials or characteristics described can be combined in any appropriate manner in any one or more embodiments or examples. In addition, the person skilled in the art can combine and combine the different embodiments or examples described in the specification and the features of the different embodiments or examples without contradiction.

[0210] The similar parts among the embodiments provided in the present application can be referred to each other, and the specific embodiments provided above are only a few examples under the general concept of the present application, and do not constitute a limitation on the protection scope of the present application. Any other embodiments extended according to the present application scheme without creative labor for the person skilled in the art are within the protection scope of the present application.

Claims

1. A zebrafish multi-target tracking method based on spatial science experiment video, characterized in that, The method comprises the following steps: acquire zebrafish space science experiment video, the zebrafish space science experiment video is used to characterize the movement process of a plurality of target zebrafish in the microgravity environment of space station; extract motion information based on the zebrafish space science experiment video, the motion information is used to characterize the motion state of the pixel point represented in the zebrafish space science experiment video; based on the motion information and appearance information, multi-modal feature fusion is carried out to determine the multi-modal fusion feature, and the appearance information is the video frame image of the zebrafish space science experiment video; target detection and tracking are carried out according to the multi-modal fusion feature, and target motion trajectory information corresponding to the plurality of target zebrafish is determined; the multi-modal feature fusion based on the motion information and the appearance information comprises: respectively extracting features from the motion information and the appearance information to obtain two-dimensional motion feature vectors and two-dimensional appearance feature vectors; constructing a heterogeneous graph based on the two-dimensional motion feature vectors and the two-dimensional appearance feature vectors, the heterogeneous graph comprising a node set and an edge set, and the expression of the heterogeneous graph being: ； ； ； wherein, denotes a heterogeneous graph corresponding to the k-th frame of video frame image, denotes a node set corresponding to the k-th frame of video frame image, denotes an appearance feature vector corresponding to the k-th frame of video frame image, denotes a motion feature vector corresponding to the k-th frame of video frame image, denotes motion trajectory information determined based on the appearance feature corresponding to the k-1-th frame of video frame image, denotes motion trajectory information determined based on the motion feature corresponding to the k-1-th frame of video frame image, denotes a temporal edge, for connecting and , and connecting and ; denotes a spatial edge, denotes a heterogeneous edge, for connecting and ; attention encoding is carried out based on the heterogeneous graph through a time attention mechanism, a space attention mechanism and a heterogeneous attention mechanism to obtain an aggregated encoding feature; the aggregated encoding feature is decoded through a deformable attention mechanism to obtain the multi-modal fusion feature.

2. The method of claim 1, wherein, the motion information extraction based on the zebrafish space science experiment video comprises: motion modeling is carried out according to the zebrafish space science experiment video to determine the first estimated displacement corresponding to each pixel point in the video frame image of the zebrafish space science experiment video, and the first estimated displacement is in the Cartesian coordinate system; the first estimated displacement corresponding to each pixel point is converted into polar coordinates to obtain the second estimated displacement corresponding to each pixel point, and the second estimated displacement is used to represent the motion amplitude and motion direction of the pixel point between two adjacent video frame images; adaptive threshold filtering is carried out based on the second estimated displacement corresponding to each pixel point to determine the motion information corresponding to the pixel point.

3. The method of claim 1, wherein, the motion modeling according to the zebrafish space science experiment video to determine the first estimated displacement corresponding to each pixel point in the video frame image of the zebrafish space science experiment video comprises: determining the image intensity of each pixel point in the first neighborhood according to the zebrafish space science experiment video; determining the first estimated displacement corresponding to each pixel point by a minimum error estimation algorithm according to the image intensity of each pixel point in the first neighborhood.

4. The method of claim 3, wherein, the adaptive threshold filtering based on the second estimated displacement corresponding to each pixel point to determine the motion information corresponding to the pixel point comprises: determining the mean and standard deviation of the motion amplitude of the pixel points in the first neighborhood according to the motion amplitude in the second estimated displacement corresponding to each pixel point. According to the mean and the standard deviation of the motion amplitude of the pixels in the first neighborhood, a first filtering threshold and a second filtering threshold of the pixels in the first neighborhood are determined, wherein the second filtering threshold is greater than the first filtering threshold, and the expression of the first filtering threshold is: ； wherein, represents a first filtering threshold, represents a mean value of the motion amplitude, represents a standard deviation of the motion amplitude, represents a first experience control value; and an expression of the second filtering threshold is: ； wherein, represents a second filter threshold, represents a second empirical control value; According to the motion amplitude corresponding to the pixel point and the first filtering threshold and the second filtering threshold of the first neighborhood in which the pixel point is located, the motion state of the pixel point is classified, and the motion information corresponding to the pixel point is determined, and the expression of the motion information is: ； wherein, represents a pixel point corresponding motion information, represents a pixel point corresponding motion amplitude; in case, the pixel point represents a static background; in case, the pixel point represents a target zebrafish with uncertain motion state; in case, the pixel point represents a target zebrafish determined to be in a motion state.

5. The method of claim 1, wherein, The target motion trajectory information corresponding to the multiple target zebrafish is determined by target detection and tracking based on the multi-modal fusion feature, and the target detection and tracking based on the multi-modal fusion feature comprises: The center heat map information, the bounding box coordinates and the fine offset value are obtained by processing the multi-modal fusion feature through a convolution layer, and the center heat map information is used to represent the confidence of the target zebrafish existing in the video frame image. The prediction displacement of the target zebrafish is predicted based on the target tracking query vector, and the target tracking query vector comprises: the identity recognition information of the target zebrafish, the current frame appearance feature vector, the current frame motion feature vector, the historical appearance feature vector, the historical motion feature vector and the historical motion trajectory information. The prediction position coordinate value of the target zebrafish is determined according to the prediction displacement. The prediction position coordinate value is associated and matched with the historical motion trajectory information based on the center heat map information, the bounding box coordinates, the fine offset value and the target tracking query vector, and the associated matching result is determined. The target motion trajectory information corresponding to the multiple target zebrafish is determined based on the associated matching result.

6. The method of claim 5, wherein, The expression of the center heat map information is: ； wherein, represents the center heat map information corresponding to the kth frame of video frame image, represents a multimodal fusion feature, represents a convolutional layer, represents a Sigmoid activation function, represents a ReLU activation function; The expression of the bounding box coordinates is: ； wherein, represents the boundary box coordinates corresponding to the kth frame of video frame image; The expression of the fine offset value is: ； wherein, denotes the refined offset value corresponding to the k-th frame of video frame image, denotes a Tanh activation function; The expression of the prediction displacement is: ； wherein, represents a predicted displacement, represents a target tracking query vector, represents a first learning weight, represents a second learning weight, represents a first bias term, represents a second bias term; The expression of the prediction position coordinate value is: ； wherein, represents the predicted position coordinate value of the target zebrafish in the kth video frame image, represents the predicted position coordinate value of the target zebrafish in the k-1th video frame image, , represents the Gaussian noise.

7. The method of claim 5, wherein, The associated matching result is a cross-frame affinity matrix, which is used to represent the matching probability of the prediction position coordinate value and the historical motion trajectory information, and the expression of the cross-frame affinity matrix is: ； wherein, represents a cross-frame affinity matrix, represents a matching score of a predicted position coordinate value of the i-th target zebrafish and the j-th historical motion trajectory information, is determined based on the center heat map information, the bounding box coordinate, the refining offset value, and the target tracking query vector. The target motion trajectory information corresponding to the multiple target zebrafish is determined based on the associated matching result, and the target motion trajectory information comprises: The target pairing combination with the maximum matching probability is determined by the Hungarian algorithm or the KM algorithm, and the target pairing combination comprises a group of prediction position coordinate values and corresponding historical motion trajectory information. In the case that the matching probability of the target pairing combination is greater than a matching threshold, the prediction position coordinate value and the corresponding historical motion trajectory information in the target pairing combination are associated and updated to determine the target motion trajectory information.

8. The method of claim 5, wherein, The center heat map information comprises: the original detection confidence, and the target motion trajectory information corresponding to the multiple target zebrafish is determined by target detection and tracking based on the multi-modal fusion feature, and the target detection and tracking based on the multi-modal fusion feature further comprises: The fine detection confidence is obtained by fine correction of the original detection confidence based on the center heat map information, and the expression of the fine detection confidence is: ； ； wherein, denotes a refined detection confidence corresponding to the i-th target zebrafish in the k-th video frame image, denotes a raw detection confidence corresponding to the i-th target zebrafish in the k-th video frame image, is a fusion strength coefficient, denotes a harmonic mean of modal confidences corresponding to the i-th target zebrafish in the k-th video frame image, denotes a confidence corresponding to the appearance modality in the k-th video frame image, denotes a confidence corresponding to the motion modality in the k-th video frame image, denotes a minimum value. 9.A zebrafish multi-target tracking system based on spatial science experiment video, characterized in that, It comprises: The acquisition module, the motion information extraction module, the multi-modal feature fusion module, and the target detection and tracking module are used. The acquisition module is configured to acquire a zebrafish space science experiment video, which is used to represent the motion process of a plurality of target zebrafish in a space station microgravity environment. The motion information extraction module is configured to extract motion information based on the zebrafish space science experiment video, wherein the motion information is used to represent the motion state of a pixel point in the zebrafish space science experiment video. The multi-modal feature fusion module is configured to perform multi-modal feature fusion based on the motion information and appearance information, and determine multi-modal fusion features, wherein the appearance information is a video frame image of the zebrafish space science experiment video. The target detection and tracking module is configured to perform target detection and tracking based on the multi-modal fusion features, and determine target motion trajectory information corresponding to the plurality of target zebrafish respectively. The multi-modal feature fusion module is specifically configured to: extract features from the motion information and the appearance information respectively to obtain a two-dimensional motion feature vector and a two-dimensional appearance feature vector; construct a heterogeneous graph based on the two-dimensional motion feature vector and the two-dimensional appearance feature vector, wherein the heterogeneous graph includes a node set and an edge set, and the expression of the heterogeneous graph is: ； ； ； wherein, denotes a heterogeneous graph corresponding to the k-th frame of video frame image, denotes a node set corresponding to the k-th frame of video frame image, denotes an appearance feature vector corresponding to the k-th frame of video frame image, denotes a motion feature vector corresponding to the k-th frame of video frame image, denotes motion trajectory information determined based on the appearance feature corresponding to the k-1-th frame of video frame image, denotes motion trajectory information determined based on the motion feature corresponding to the k-1-th frame of video frame image, denotes a temporal edge, for connecting and , and connecting and ; denotes a spatial edge, denotes a heterogeneous edge, for connecting and ; perform attention encoding based on the heterogeneous graph through a time attention mechanism, a space attention mechanism, and a heterogeneous attention mechanism to obtain an aggregated encoding feature; perform deformable decoding on the aggregated encoding feature through a deformable attention mechanism to obtain the multi-modal fusion features.

Images