Zebra fish multi-target tracking method and system based on space science experiment video

Abstract

The invention provides a zebra fish multi-target tracking method and system based on a space science experiment video. The method comprises the following steps: firstly, obtaining a zebra fish space science experiment video; then, motion information is extracted based on the zebra fish space science experiment video, and the motion information is used for representing a motion state represented by pixel points in the zebra fish space science experiment video; then, multi-modal feature fusion is carried out based on the motion information and the appearance information, and multi-modal fusion features are determined; and finally, target detection and tracking are carried out according to the multi-modal fusion features, and target movement track information corresponding to the target zebra fishes is determined. According to the method, motion feature information representing dynamic changes is extracted through motion modeling based on a self-adaptive threshold value, then the interaction and fusion relation between appearance and motion features is modeled step by step through a heterogeneous graph, and the robustness of target detection, data association and trajectory prediction is enhanced. Therefore, the stability and the accuracy of multi-target tracking of the zebra fish can be effectively improved.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of target tracking, and in particular to a zebrafish multi-target tracking method and system based on space science experiment video. BACKGROUND

[0002] With the rapid development of space station technology, as a space laboratory, the space station has unique advantages such as covering the whole field of space science, strong on-orbit support capability, supporting human participation and up-down transportation, and can carry out large-scale multidisciplinary space science research. At present, the life and ecological experiment cabinet carried by the space station can obtain zebrafish experiment observation video (i.e. space science experiment video), which provides an important data basis for studying the influence of space environment on biological behavior and physiological response. Under this background, it is urgent to use multi-target tracking technology to realize accurate positioning, identity maintenance and motion trajectory generation of multiple zebrafish targets under the condition of given space science experiment video input, so as to provide reliable technical support for in-depth analysis.

[0003] Generally, multi-target tracking technology can be divided into two types of schemes: tracking after detection and joint detection and tracking. Among them, the core of tracking after detection is to decouple the tracking task into two stages of target detection and data association, and to improve the overall performance by optimizing the detection accuracy and association strategy independently. For example: SORT first realizes the separation of detection and association, predicts the motion state of the target based on Kalman filter and matches the detection box and the trajectory by combining the Hungarian algorithm. DeepSORT introduces appearance feature matching based on SORT, which improves the robustness of complex scenes. MOTDT improves the association accuracy by multi-cue fusion (such as appearance feature, motion feature). IOUByteTrack only uses the intersection over union of the detection box for association, which is simple and efficient but sensitive to target motion speed and occlusion. Joint detection and tracking converts the detector into a tracker and integrates the two tasks in the same framework. For example: Tracktor uses Faster R-CNN regression head to directly predict the next frame position of the target. FairMOT uses anchor-free detection based on CenterNet and jointly learns detection and re-identification features. TransTrack is based on the Transformer architecture, which uses self-attention mechanism to realize the joint modeling of detection and tracking.

[0004] However, the above-mentioned related multi-target tracking method is mostly designed for general public data sets, and has significant adaptability bottleneck when applied to space station zebrafish video analysis, and it is difficult to meet the precise tracking demand. The specific technical problems include: first, the background interference suppression is insufficient. The strong reflection and diffuse reflection phenomenon caused by the limited light source in the space experiment environment causes multiple reflections of zebrafish, which easily leads to false matching and target splitting, and seriously affects the continuity and accuracy of tracking. Second, the ability to cope with violent movement and occlusion is limited. Zebrafish in microgravity environment shows stress response such as high-speed swimming, irregular turning and group gathering, which leads to frequent occlusion and overlap between targets, increases the difficulty of target segmentation and identity maintenance, and the joint detection and tracking method relies on the hypothesis of high overlap of inter-frame bounding boxes, which is not true in the case of fast target displacement. The pre-detection and then association method has the defects of discarding image appearance features or high feature extraction cost, and splitting the detection and tracking process. Third, the adaptability to environmental dynamic changes is poor. Bubbles generated in the aquatic system during the on-orbit experiment period will adhere to the observation window or float in the field of view, forming a dynamic occlusion area. The existing method cannot effectively cope with the influence of such dynamic environmental changes on tracking effect. It can be seen that the related multi-target tracking method cannot stably and accurately realize multi-target tracking of zebrafish, and thus it is difficult to meet the scientific demand of trajectory precision and long-term identity consistency for space station zebrafish behavior analysis. SUMMARY

[0005] The technical problem to be solved by the present application is the problem of unstable and accurate multi-target tracking of zebrafish.

[0006] To solve the above technical problems, the present application provides a zebrafish multi-target tracking method and system based on space science experiment video, which specifically adopts the following technical solutions: In a first aspect, the present application provides a zebrafish multi-target tracking method based on space science experiment video, which comprises: first, acquiring a zebrafish space science experiment video, the zebrafish space science experiment video being used to represent the movement process of multiple target zebrafish in the microgravity environment of the space station. Then, extracting movement information based on the zebrafish space science experiment video, the movement information being used to represent the motion state of the pixel points in the zebrafish space science experiment video. Next, performing multi-modal feature fusion based on the movement information and appearance information to determine multi-modal fusion features, the appearance information being the video frame image of the zebrafish space science experiment video. Finally, performing target detection and tracking according to the multi-modal fusion features to determine the target motion trajectory information corresponding to the multiple target zebrafish respectively.

[0007] The method extracts motion feature information representing dynamic changes through adaptive threshold-based motion modeling, and then uses a heterogeneous graph network to model the interaction and fusion relationship between appearance and motion features at each level, thereby enhancing the robustness of target detection, data association, and trajectory prediction. According to the multi-modal fusion features, detection and displacement prediction, trajectory matching, and confidence fusion are integrated to realize integrated target detection and tracking, and finally determine the target motion trajectory information corresponding to each of the multiple target zebrafish. In this way, the method can effectively improve the stability and accuracy of multi-target tracking of zebrafish.

[0008] In combination with the first aspect, in an alternative implementation, the above motion information extraction based on the zebrafish space science experiment video includes: first, motion modeling is performed according to the zebrafish space science experiment video to determine the first estimated displacement of each pixel point in the video frame image of the zebrafish space science experiment video, and the first estimated displacement is in the Cartesian coordinate system. Then, polar coordinate conversion is performed on the first estimated displacement of each pixel point to obtain the second estimated displacement of each pixel point, and the second estimated displacement is used to represent the motion amplitude and motion direction of the pixel point between two adjacent video frame images. Finally, adaptive threshold filtering is performed based on the second estimated displacement of each pixel point to determine the motion information corresponding to the pixel point.

[0009] In combination with the first aspect, in an alternative implementation, the above motion modeling according to the zebrafish space science experiment video to determine the first estimated displacement of each pixel point in the video frame image of the zebrafish space science experiment video includes: first, the image intensity of each pixel point in the first neighborhood is determined according to the zebrafish space science experiment video. Then, the first estimated displacement corresponding to each pixel point is determined by a minimum error estimation algorithm based on the image intensity of each pixel point in the first neighborhood.

[0010] In combination with the first aspect, in an alternative implementation, the above adaptive threshold filtering based on the second estimated displacement of each pixel point to determine the motion information corresponding to the pixel point includes: first, the motion amplitude mean and motion amplitude standard deviation of the pixel points in the first neighborhood are determined according to the motion amplitude in the second estimated displacement of each pixel point. Then, the first filtering threshold and the second filtering threshold of the pixel points in the first neighborhood are determined according to the motion amplitude mean and motion amplitude standard deviation of the pixel points in the first neighborhood. The second filtering threshold is greater than the first filtering threshold, and the expression of the first filtering threshold is: ; wherein, represents the first filtering threshold, represents the motion amplitude mean, represents the motion amplitude standard deviation, represents the first experience control value. The expression of the second filtering threshold is: ; wherein, represents the second filtering threshold, represents the second experience control value. Finally, according to the motion amplitude corresponding to the pixel point and the motion state represented by the pixel point according to the first filtering threshold and the second filtering threshold in the first neighborhood in which the pixel point is located, the motion information corresponding to the pixel point is determined, and the expression of the motion information is: ; wherein, represents the motion information corresponding to the pixel point, represents the motion amplitude corresponding to the pixel point, represents the motion information corresponding to the pixel point, represents the motion amplitude corresponding to the pixel point; in the case of , the pixel point represents a static background; in the case of , the pixel point represents a target zebrafish in an uncertain motion state; in the case of , the pixel point represents a target zebrafish in a motion state.

[0011] In combination with the first aspect, in an alternative implementation manner, the above multi-modal feature fusion based on the motion information and the appearance information to determine the multi-modal fusion feature includes: first, feature extraction is performed on the motion information and the appearance information respectively to obtain a two-dimensional motion feature vector and a two-dimensional appearance feature vector. Then, a heterogeneous graph is constructed based on the two-dimensional motion feature vector and the two-dimensional appearance feature vector, the heterogeneous graph includes a node set and an edge set, and the expression of the heterogeneous graph is: ; ; ; wherein, represents the heterogeneous graph corresponding to the kth video frame image, represents the node set corresponding to the kth video frame image, represents the edge set corresponding to the kth video frame image, represents the appearance feature vector corresponding to the kth video frame image, represents the motion feature vector corresponding to the kth video frame image, represents the motion trajectory information determined based on the appearance feature corresponding to the k-1th video frame image, represents the motion trajectory information determined based on the motion feature corresponding to the k-1th video frame image, represents a time sequence edge, For connection and and connections and ; Represents a spatial edge. Indicates heterogeneous edges. For connection and Secondly, attention encoding is performed on the heterogeneous graph using temporal attention, spatial attention, and heterogeneous attention mechanisms to obtain aggregated encoded features. Finally, deformable decoding is performed on the aggregated encoded features using a deformable attention mechanism to obtain multimodal fusion features.

[0012] In conjunction with the first aspect, in one alternative implementation, the above-mentioned target detection and tracking based on multimodal fusion features to determine the target motion trajectory information corresponding to multiple target zebrafish includes: First, processing the multimodal fusion features through convolutional layers to obtain center heatmap information, bounding box coordinates, and refined offset values. The center heatmap information is used to characterize the confidence that a target zebrafish exists in the video frame image. Then, predicting the predicted displacement of the target zebrafish based on the target tracking query vector, which includes: the target zebrafish's identification information, the current frame appearance feature vector, the current frame motion feature vector, historical appearance feature vector, historical motion feature vector, and historical motion trajectory information. Next, determining the predicted position coordinates of the target zebrafish based on the predicted displacement. Second, based on the center heatmap information, bounding box coordinates, refined offset values, and target tracking query vector, associating and matching the predicted position coordinates with the historical motion trajectory information to determine the association matching result. Finally, determining the target motion trajectory information corresponding to multiple target zebrafish based on the association matching result.

[0013] In conjunction with the first aspect, in one alternative implementation, the expression for the aforementioned central heatmap information is: ; in, This represents the center heatmap information corresponding to the k-th video frame image. Indicates multimodal fusion features, Indicates a convolutional layer. This represents the Sigmoid activation function. This represents the ReLU activation function. The expression for the bounding box coordinates is: ; in, This represents the bounding box coordinates corresponding to the k-th video frame. The expression for the refinement offset value is: ; wherein, denotes the refined offset value corresponding to the k-th frame of video frame image, denotes a Tanh activation function. The expression of the predicted displacement is: ; wherein, denotes the predicted displacement, denotes the target tracking query vector, denotes the first learning weight, denotes the second learning weight, denotes the first bias term, denotes the second bias term. The expression of the predicted position coordinate value is: ; wherein, denotes the predicted position coordinate value of the target zebrafish in the k-th frame of video frame image, denotes the predicted position coordinate value of the target zebrafish in the k-1-th frame of video frame image, , denotes the Gaussian noise.

[0014] In combination with the first aspect, in an alternative implementation manner, the association matching result is a cross-frame affinity matrix, the cross-frame affinity matrix is used to represent the matching probability of the predicted position coordinate value and the historical motion trajectory information, and the expression of the cross-frame affinity matrix is: ; wherein, denotes the cross-frame affinity matrix, denotes the matching score of the predicted position coordinate value of the i-th target zebrafish and the j-th historical motion trajectory information, is determined based on the center heat map information, the bounding box coordinate, the refined offset value and the target tracking query vector. Then, the target motion trajectory information corresponding to each of the plurality of target zebrafish is determined based on the association matching result, which includes: first, determining a target pairing combination with the largest matching probability by using the Hungarian algorithm or the KM algorithm, the target pairing combination including a group of predicted position coordinate values and corresponding historical motion trajectory information. Then, in the case that the matching probability of the target pairing combination is greater than a matching threshold, the predicted position coordinate value and the corresponding historical motion trajectory information in the target pairing combination are associated and updated to determine the target motion trajectory information.

[0015] In combination with the first aspect, in an optional implementation manner, the center heat map information includes: original detection confidence. The target detection and tracking according to the multi-modal fusion feature to determine the target motion trajectory information corresponding to the plurality of target zebrafish further includes: refining the original detection confidence based on the center heat map information to obtain refined detection confidence, and the expression of the refined detection confidence is: ; ; wherein, refined detection confidence corresponding to the i-th target zebrafish in the k-th video frame image, original detection confidence corresponding to the i-th target zebrafish in the k-th video frame image, is a fusion intensity coefficient, is a harmonic mean of the modal-specific confidence corresponding to the i-th target zebrafish in the k-th video frame image, confidence corresponding to the appearance modal in the k-th video frame image, confidence corresponding to the motion modal in the k-th video frame image, represents a minimum value.

[0016] In the second aspect, the present application provides a zebrafish multi-target tracking system based on a space science experiment video, which comprises: an acquisition module, a motion information extraction module, a multi-modal feature fusion module and a target detection and tracking module. The acquisition module can be used to acquire a zebrafish space science experiment video, which is used to represent the motion process of a plurality of target zebrafish in a space station microgravity environment. The motion information extraction module can be used to extract motion information based on the zebrafish space science experiment video, and the motion information is used to represent the motion state of a pixel point in the zebrafish space science experiment video. The multi-modal feature fusion module can be used to perform multi-modal feature fusion based on the motion information and appearance information to determine multi-modal fusion features, and the appearance information is a video frame image of the zebrafish space science experiment video. The target detection and tracking module can be used to perform target detection and tracking according to the multi-modal fusion features to determine target motion trajectory information corresponding to the plurality of target zebrafish respectively.

[0017] In the third aspect, the present application provides an electronic device, comprising: a memory, one or more processors; the memory is coupled with the processor; wherein the memory stores computer program code, the computer program code includes computer instructions, when the computer instructions are executed by the processor, the electronic device executes the method provided by the first aspect and any optional implementation manner thereof.

[0018] In a fourth aspect, the present application provides a computer readable storage medium, comprising computer instructions, when the computer instructions are executed on an electronic device, the electronic device is caused to perform the method provided by the first aspect and any possible implementation manner thereof.

[0019] It can be understood that the beneficial effects achieved by the zebrafish multi-target tracking system based on space science experiment video provided by the second aspect, the electronic device of the third aspect and the computer readable storage medium of the fourth aspect can refer to the beneficial effects in the first aspect and any possible design manner thereof, which will not be repeated here. BRIEF DESCRIPTION OF DRAWINGS

[0020] Figure 1 A principle schematic diagram of the zebrafish multi-target tracking method based on space science experiment video provided by the embodiments of the present application; Figure 2 A flow schematic diagram of the zebrafish multi-target tracking method based on space science experiment video provided by the embodiments of the present application; Figure 3 A structure relationship schematic diagram of the heterogeneous graph provided by the embodiments of the present application; Figure 4 A structure schematic diagram of the zebrafish multi-target tracking system based on space science experiment video provided by the embodiments of the present application. DETAILED DESCRIPTION

[0021] The embodiments will be described in detail below with reference to the drawings. The following description is with reference to the drawings, unless otherwise indicated, the same numbers in different drawings represent the same or similar elements. The embodiments described in the following embodiments do not represent all the embodiments consistent with the present application. They are only examples of systems and methods consistent with some aspects of the present application, which are detailed in the claims.

[0022] In general, multi-object tracking techniques can be divided into two categories: tracking-by-detection and joint detection and tracking. The core of tracking-by-detection is to decouple the tracking task into two stages: target detection and data association, and to improve the overall performance by optimizing the detection accuracy and association strategy independently. For example, SORT first separates detection and association, laying the foundation for the two-stage framework, and based on Kalman filter to predict the target motion state, combined with the Hungarian algorithm to match the detection box and the track, lightweight and efficient. DeepSORT introduces appearance feature matching based on SORT, solves the problem of target occlusion and similar appearance interference, and combines motion model and appearance feature to improve the robustness of complex scenes. MOTDT combines detection results and tracking prediction, and improves the association accuracy through multi-cue fusion (such as appearance features and motion features). IOUByteTrack only uses the IoU (Intersection over Union) of the detection box for association, which is simple and efficient, but sensitive to target motion speed and occlusion. This method has two shortcomings. First, data association discards image appearance features or needs to calculate expensive feature extractors. Second, it separates the integration process of detection and tracking.

[0023] Joint detection and tracking converts the detector into a tracker and integrates the two tasks, and combines the two tasks in the same framework. For example, Tracktor uses the regression head of the detector Faster R-CNN to directly predict the position of the target in the next frame, without the need for additional association steps. FairMOT is based on CenterNet, which uses an anchor-free detection method, and jointly learns detection and re-identification features, achieving a good balance between speed and accuracy. TransTrack is based on the Transformer architecture, which uses self-attention mechanisms to realize joint modeling of detection and tracking. This method uses the tracking box as the region proposal of the detector, which requires the assumption that the bounding box has a large overlap between frames, which is not true in scenarios where targets have fast displacement.

[0024] For example, a related technology one proposes a MOTDT method based on a pre-detection and post-association framework, which combines detection and tracking results as candidates and selects the optimal candidate based on a deep neural network to deal with unreliable detection interference in tracking. Further, a hierarchical data association strategy is designed to improve tracking performance. The overall idea and basic process of this method include: real-time target classification, trajectory confidence score, appearance feature representation, hierarchical data association. A related technology two proposes a FairMOT method based on a joint detection and tracking framework, which converts multi-target tracking into a pixel-level key point estimation and identity classification problem on a high-resolution feature map. Through a single-stage end-to-end architecture and feature alignment design, the speed bottleneck and identity switching problem of traditional two-stage methods are solved. The overall idea and basic process of this method include: high-resolution feature extraction, detection branch, re-identification branch, bounding box temporal association.

[0025] However, the above-mentioned related multi-target tracking methods are mostly designed for general public data sets and are suitable for multi-target tracking tasks in general scenarios. However, when applied to the analysis of zebrafish space science experiment videos in space station environment, these methods face significant adaptability bottlenecks. On the one hand, general methods are difficult to cope with a series of unique and complex challenges in this task; on the other hand, they also fail to effectively tap into and utilize the unique advantages of zebrafish space science experiment videos at the data level. When applied to the analysis of zebrafish space science experiment videos in space station, there are significant adaptability bottlenecks, which are difficult to meet the precise tracking requirements, and the specific technical problems include: (1) Serious interference from complex background: Due to the limited arrangement of light sources in the space experiment environment, strong reflection and diffuse reflection phenomena often occur, causing multiple reflections of zebrafish individuals on the container wall or water surface. These reflections are highly similar to the real targets, causing frequent mismatching and target splitting during target detection and tracking, which seriously affects the continuity and accuracy of tracking.

[0026] (2) Special behavior patterns in microgravity environment: Under the condition of space microgravity, zebrafish exhibit strong stress response in the early stage of the experiment, characterized by frequent high-speed swimming, irregular turning, and group gathering behavior. This highly dynamic and densely interactive movement pattern easily leads to frequent occlusion and overlap between targets, increasing the difficulty of target segmentation and identity maintenance.

[0027] (3) Significant dynamic changes in on-orbit experiment environment: As the experiment period progresses, bubbles are gradually generated and enlarged in the aquatic system, and some bubbles adhere to the observation window or float in the field of view, forming dynamic occlusion areas. This not only reduces the visibility of the target, but also may cause false detection or missed detection, further affecting the robustness of the tracking algorithm.

[0028] It can be seen that the related multi-target tracking method cannot stably and accurately realize multi-target tracking of zebrafish, and thus it is difficult to meet the scientific requirements of trajectory accuracy and long-term identity consistency for zebrafish behavior analysis in the space station.

[0029] To solve the above problems, the embodiment of the application provides a zebrafish multi-target tracking method and system based on space science experiment video. The method and system can be applied to zebrafish observation experiment video collected in a stable scene in a fixed field of view under a microgravity environment, for example, a space science experiment video collected in a zebrafish observation experiment in a space station. The method first extracts motion feature information representing dynamic changes based on adaptive threshold motion modeling, and then uses a heterogeneous graph network to model the interaction and fusion relationship of appearance and motion features at each level, enhancing the robustness of target detection, data association and trajectory prediction.

[0030] Specifically, Figure 1 The principle schematic diagram of the zebrafish multi-target tracking method based on space science experiment video provided by the embodiment of the application is shown in Figure 1 The method first extracts motion information based on zebrafish space science experiment video through motion modeling, polar coordinate conversion and adaptive threshold filtering. Then, based on the motion information and appearance information, it realizes multi-modal feature fusion by feature extraction, heterogeneous graph construction, attention encoding and deformable decoding in turn, and determines the multi-modal fusion features. Finally, according to the multi-modal fusion features, it realizes integrated target detection and tracking through detection and displacement prediction, trajectory matching and confidence fusion, and finally determines the target motion trajectory information corresponding to each target zebrafish. The method can effectively improve the stability and accuracy of multi-target tracking of zebrafish.

[0031] The scheme provided by the embodiment of the application will be described below with reference to the accompanying drawings.

[0032] Specifically, Figure 2 The flowchart of the zebrafish multi-target tracking method based on space science experiment video provided by the embodiment of the application is shown in Figure 2 The zebrafish multi-target tracking method based on space science experiment video provided by the embodiment of the application includes the following steps S101-S104: S101, acquiring zebrafish space science experiment video.

[0033] In the embodiment of the application, the zebrafish space science experiment video can be an experiment observation video collected when a zebrafish observation experiment is performed in a life ecological experiment cabinet carried on a space station. The zebrafish space science experiment video can be used to represent the motion process of multiple target zebrafish in the microgravity environment of the space station. Specifically, the zebrafish space science experiment video is composed of a plurality of continuous video frame images.

[0034] S102, extract motion information based on the zebrafish space science experiment video.

[0035] In the embodiments of the present application, the motion information is first extracted from the zebrafish space science experiment video obtained in S101, which can be used to represent the motion state of the pixel points in the zebrafish space science experiment video. In this way, the motion pattern of the zebrafish in the zebrafish space science experiment video and the noise artifacts are effectively separated based on the motion information, and complementary semantic enhancement is provided for the appearance features.

[0036] In some embodiments, motion modeling, polar coordinate conversion and adaptive threshold filtering can be used to extract reliable motion information for subsequent multi-modal fusion. Specifically, S102 can include the following steps S1021-S1023: S1021, motion modeling is performed according to the zebrafish space science experiment video to determine the first estimated displacement corresponding to each pixel point in the video frame image of the zebrafish space science experiment video. Wherein the first estimated displacement is in the Cartesian coordinate system.

[0037] In some embodiments, S1021 specifically includes: First, the image intensity of each pixel point in the first neighborhood is determined according to the zebrafish space science experiment video.

[0038] Then, the first estimated displacement corresponding to each pixel point is determined by minimizing the error estimation algorithm according to the image intensity of each pixel point in the first neighborhood.

[0039] Specifically, local motion modeling is performed on the zebrafish space science experiment video, which aims to model how the image intensity (brightness) changes with the spatial position in a certain pixel point neighborhood (i.e. a preset neighborhood) in the video frame image of the zebrafish space science experiment. Since the target zebrafish motion is relatively smooth in the small neighborhood of the pixel point in the video frame image, a quadratic function can be used to describe it. For this purpose, a second-order Taylor expansion can be used to locally approximate the image intensity function. The second-order Taylor can capture the curvature information (convex, concave, edge direction, etc.), which can accurately describe the local structure.

[0040] For example, for the pixel position in the video frame image, the expression of the image intensity is: ; Wherein, represents the local offset relative to the center pixel point in a preset neighborhood, i.e. the intensity distribution in the small neighborhood centered on the center pixel point. For example, the center point is set as the origin (0, 0) in the embodiments of the present application, and is the displacement relative to the center origin (0, 0). In the formula, c is the intensity value of the video frame image at the center point (0, 0) (i.e. constant term).

[0041] is the second derivative matrix (for and partial derivative). is the second partial derivative of the image intensity function with respect to , which represents the case of intensity change in the x direction, Similarly. is the mixed second partial derivative of the image intensity function with respect to x and y, which represents the case of intensity change in the x direction. This second derivative matrix can be used to capture the curvature information of the local region of the image, such as convex, concave, edge direction, etc., so as to accurately describe the local structure. Specifically, the elements of the second derivative matrix have a clear numerical meaning, which can be large or small, positive or negative. The larger the value, the more intense the curvature change in that direction. For example, The larger the value, the more significant the intensity curvature change in the x direction. Positive values usually correspond to "convex" (such as bright spots), and negative values may correspond to "concave" (such as dark spots). When the matrix as a whole is positive (combined with eigenvalue analysis), it can represent a convex feature in the local region; if the sign is complex, it may correspond to edge or corner structure.

[0042] is the gradient vector, which represents the rate of change of image intensity in the x and y directions. The gradient vector describes the trend of image intensity change in the x and y directions in the local region, and its length represents the degree of change (the larger the gradient, the more obvious the intensity change), and the direction points to the direction of the fastest increase in intensity.

[0043] The above second derivative matrix is a symmetric matrix, and there are two eigenvalues (λ1 and λ2). λ1 and λ2 can be obtained by solving the characteristic equation of the matrix, and λ1 and λ2 reflect the "scaling" ability of the matrix in different directions. The larger the eigenvalue value, the more significant the curvature change in that direction. When the eigenvalues have the same sign (both positive or both negative), it may correspond to convex or concave structure (such as bright spots, dark spots); when the signs are opposite, it indicates that there are two orthogonal directions of intense curvature change in the local region, i.e. it corresponds to "corner point". For example, when the two eigenvalues are large and the signs are opposite, it indicates that the local region presents strong convexity and concavity in two perpendicular directions, which conforms to the structural characteristics of the corner point. In this way, the above expression of image intensity can effectively approximate the image intensity change in the preset neighborhood.

[0044] Further, in combination with the change over time, it is assumed that the image intensity in the preset neighborhood at time t is described by the above quadratic model. In At this moment, there may be a translational motion in the preset neighborhood. Since the intensity change of the same physical point in the video frame images at different times is mainly affected by noise, illumination change, occlusion, etc., the error estimation algorithm can be minimized, i.e., the first estimated displacement is expressed in the form of minimizing error : ; wherein, represents a regularization coefficient, which prevents overfitting and improves stability.

[0045] S1022, polar coordinate conversion is performed on the first estimated displacement corresponding to each pixel point to obtain the second estimated displacement corresponding to each pixel point.

[0046] Specifically, the first estimated displacement determined in S1021 is in the Cartesian coordinate system, which is not conducive to directly analyzing the motion intensity and motion direction of the target zebrafish. The polar coordinate form clearly separates two key attributes: motion amplitude (indicating the size of the displacement of the pixel point, i.e., the size of the speed), which is used to determine whether significant motion has occurred. Motion direction, used to analyze the motion trend, such as up, left, etc. Therefore, the first estimated displacement corresponding to the pixel point can be polar coordinate converted to obtain the second estimated displacement corresponding to each pixel point. The second estimated displacement is used to represent the motion amplitude and motion direction of the pixel point in the adjacent two video frame images. Exemplarily, the expression of the second estimated displacement (v) can be:

[0047] . .

[0048] S1023, adaptive threshold filtering is performed based on the second estimated displacement corresponding to each pixel point to determine the motion information corresponding to the pixel point.

[0049] Next, the second estimated displacement corresponding to each pixel point is filtered based on the adaptive threshold, i.e., classified to determine the motion type represented by the pixel point. Exemplarily, the motion type can include: suppressing static background, target zebrafish in an uncertain motion state, and target zebrafish in a determined motion state. Wherein, the adaptive threshold filtering is that different thresholds are used in different preset neighborhoods, rather than using a fixed threshold globally (the whole video frame image).

[0050] In some embodiments, the specific implementation method of determining the motion information corresponding to the pixel point by adaptive threshold filtering in S1023 can include the following steps S10231-S10233: S10231, according to the motion amplitude in the second estimated displacement corresponding to each pixel point, the mean value and the standard deviation of the motion amplitude of the pixel point in the first neighborhood are determined.​

[0051] S10232. Determine the first filtering threshold and the second filtering threshold for the pixels in the first neighborhood based on the mean of the motion amplitude and the standard deviation of the motion amplitude.

[0052] The second filtering threshold is greater than the first filtering threshold. The first filtering threshold (i.e., the low threshold) can be used to initially distinguish between minute movements and potential movements, while the second filtering threshold (i.e., the high threshold) can be used to distinguish between certain movements, that is, to distinguish between uncertain and certain motion states.

[0053] Specifically, the expression for the first filtering threshold is: ; in, This represents the first filtering threshold. Indicates the mean amplitude of motion. Indicates the standard deviation of the amplitude of motion. This represents the first empirical control value.

[0054] The expression for the second filtering threshold is: ; in, This indicates the second filtering threshold. This represents the second empirical control value.

[0055] For example, the first empirical control value It can be 1.0, the second empirical control value. It can be 2.5. First empirical control value. Second empirical control value Presets can be made according to actual application needs; this application does not impose specific limitations on this.

[0056] S10233. Based on the motion amplitude corresponding to the pixel and the first and second filtering thresholds within the first neighborhood of the pixel, classify the motion state represented by the pixel to determine the motion information corresponding to the pixel. The expression for the motion information is: ; in, Represents pixels Corresponding motion information Represents pixels The corresponding amplitude of motion; in In the case of pixels Indicates suppression of static background; in In the case of pixels This represents a target zebrafish in an uncertain state of motion; in In the case of pixels representing determining a target zebrafish in a motion state.

[0057] S103, performing multi-modal feature fusion based on the motion information and the appearance information to determine multi-modal fusion features.

[0058] Next, to effectively integrate the appearance information and motion information of the target zebrafish in the spatial science experiment video, a multi-modal feature fusion method based on a multi-modal heterogeneous graph architecture can be used. This method determines multi-modal fusion features based on motion information and appearance information through feature extraction, heterogeneous graph modeling, attention encoding, and deformable decoding. The appearance information is the video frame image of the zebrafish spatial science experiment video.

[0059] In some embodiments, specifically, S103 can include the following steps S1031-S1034: S1031, respectively extracting features from the motion information and the appearance information to obtain a two-dimensional motion feature vector and a two-dimensional appearance feature vector.

[0060] In one implementation, the motion information is motion modal , and the appearance information is an RGB video frame of the appearance modal. Based on the above motion information and appearance information, feature extraction can be performed using a PVTv2 backbone network, which is a multi-scale output visual Transformer architecture that outputs feature maps at four down-sampling rates r∈{4,8,16,32} to form a feature pyramid , where k is the time step (when (appearance) front frame), a (appearance) represents the appearance modal (i.e., the modal corresponding to the appearance information), m (motion) represents the motion modal (i.e., the modal corresponding to the motion information), and r is the spatial scale (i.e., the resolution). The size of each feature decreases as r increases.

[0061] To convert the three-dimensional feature map at each scale into a set of two-dimensional query vector sets to construct the initial nodes that can match the graph neural network, the embodiments of the present application use stretching processing and linear projection to flatten the three-dimensional matrix into a two-dimensional feature by the following formula, i.e., to obtain a two-dimensional motion feature vector and a two-dimensional appearance feature vector: ; wherein, represents a two-dimensional motion feature vector or a two-dimensional appearance feature vector, represents a flattening function, is a learnable projection weight, represents multi-scale position encoding to explicitly add spatial position information.

[0062] S1032, construct a heterogeneous graph based on the two-dimensional motion feature vector and the two-dimensional appearance feature vector, the heterogeneous graph comprising a node set and an edge set.

[0063] Specifically, Figure 3 The structural relationship diagram of the heterogeneous graph provided by the embodiments of the present application is shown in Figure 3 The expression of the heterogeneous graph is as follows: ; ; ; Among them, represents the heterogeneous graph corresponding to the k-th frame of video frame image, represents the node set corresponding to the k-th frame of video frame image, represents the edge set corresponding to the k-th frame of video frame image, represents the appearance feature vector corresponding to the k-th frame of video frame image, represents the motion feature vector corresponding to the k-th frame of video frame image, represents the motion trajectory information determined based on the appearance feature corresponding to the k-1-th frame of video frame image, represents the motion trajectory information determined based on the motion feature corresponding to the k-1-th frame of video frame image, represents a temporal edge, is used to connect and , and connect and ; represents a spatial edge, represents a heterogeneous edge, is used to connect and .

[0064] S1033, based on the heterogeneous graph, attention encoding is performed through a temporal attention mechanism, a spatial attention mechanism and a heterogeneous attention mechanism, to obtain an aggregated encoding feature.

[0065] In the attention encoding stage, the context-aware query representation is generated based on the heterogeneous graph constructed in S1032 through the attention mechanism, specifically including: a temporal attention mechanism, a spatial attention mechanism and a heterogeneous attention mechanism.

[0066] Specifically, the temporal attention mechanism aims to model the temporal consistency within the modal, and enhances the current query by using the historical trajectory, taking the appearance modal encoding feature as an example, which can be represented as: In the formula, The current feature is combined with the historical feature to enhance the memory. , , is a learnable parameter matrix of the temporal attention mechanism, is a scaling factor, is a feature dimension, which is used to prevent the gradient from being unstable due to the result of the dot product being too large. represents a temporal dimension. Similarly, motion modality information can be processed to obtain motion modality encoding features .

[0067] The spatial attention mechanism aims to achieve cross-modal spatial alignment to solve the problem that the appearance region and the motion region do not completely coincide. Specifically, a relative position bias can be used to model the spatial structure: ; wherein, represents the spatial attention feature of the motion modality to the appearance modality, can be used as an appearance query, can provide a key-value for the motion modality to guide the appearance to focus on the motion active region. Similarly, the reverse operation is: . , , is a learnable parameter matrix of the spatial attention mechanism, and s represents a spatial dimension. , represents the relative coordinate offset between the encoding nodes i and j in the heterogeneous graph.

[0068] The heterogeneous attention mechanism is used to uniformly fuse information of all modalities and types to generate a unified representation. The bidirectional enhanced features are spliced as a query to aggregate information from the motion modality and the appearance modality to obtain aggregated encoding features : ; wherein, , , correspond to queries, keys, and values of different modalities (for example: appearance modality and motion modality), , , is a learnable parameter matrix of the heterogeneous attention mechanism, and h represents heterogeneity.

[0069] In this way, the attention encoding through the above-mentioned temporal attention mechanism, spatial attention mechanism, and heterogeneous attention mechanism can effectively learn rich cross-modal representations while retaining modality-specific features to accurately determine the aggregated encoding features.

[0070] S1034, performing deformable decoding on the aggregated encoded features through a deformable attention mechanism to obtain multi-modal fusion features.

[0071] Further, the aggregated encoded features determined in S1033 can be deformable decoded based on the deformable attention mechanism to obtain multi-modal fusion features.

[0072] Specifically, the deformable attention mechanism aims to efficiently sample key regions from the aggregated encoded features to generate multi-modal fusion features for tracking. Specifically, by dynamically sampling through the deformable attention mechanism, the multi-modal fusion features are determined : ; wherein Head is the number of attention heads, which can be set to 8, for example. is the number of sampling points for each attention head, which can be set to 4, for example. is a learnable normalized reference point coordinate, with a value range of 0-1, is a learnable offset, S represents bilinear interpolation sampling, is a learnable parameter matrix.

[0073] In this way, the robust integration of appearance features and motion features can be achieved through the above S1031-S1034, providing an important feature basis for accurate and context-aware tracking of target zebrafish in complex spatial environments.

[0074] S104, performing target detection and tracking based on the multi-modal fusion features to determine target motion trajectory information corresponding to each of the plurality of target zebrafish.

[0075] Finally, target detection and tracking can be performed based on the multi-modal fusion features determined in S103 to determine target motion trajectory information. The present application adopts an end-to-end trainable architecture to integrate target detection and tracking into one, achieving real-time, accurate and robust tracking of target zebrafish in spatial environments. Detection and tracking share multi-modal features, detection output provides initial target hypotheses, and tracking queries maintain inter-frame temporal continuity.

[0076] In some embodiments, specifically, S104 can include the following steps S1041-S1045: S1041, processing the multi-modal fusion features through a convolutional layer to obtain center heat map information, bounding box coordinates and refinement offset values, respectively.

[0077] In the detection stage, first, the multi-modal fusion features are reshaped and processed through a series of convolutional layers to generate center heat map information, bounding box coordinates and fine-tuning offset values. Among them, the center heat map information is used to represent the confidence of the presence of the target zebrafish in the video frame image.

[0078] In some embodiments, the expression of the above center heat map information is: ; Among them, represents the center heat map information corresponding to the k-th video frame image, represents the multi-modal fusion features, represents the convolutional layer, represents the Sigmoid activation function, represents the ReLU activation function.

[0079] The expression of the above bounding box coordinates is: ; Among them, represents the bounding box coordinates corresponding to the k-th video frame image.

[0080] The expression of the above fine-tuning offset value is: ; Among them, represents the fine-tuning offset value corresponding to the k-th video frame image, represents the Tanh activation function.

[0081] S1042, predict the predicted displacement of the target zebrafish based on the target tracking query vector.

[0082] Then, in the target tracking stage, the motion trajectory of the existing target is predicted, and the predicted displacement of the target zebrafish is predicted by learning the target tracking query vector. Among them, the target tracking query vector is a core variable for maintaining the inter-frame time sequence continuity of the target, and its essence is a learnable feature vector. Specifically, the target tracking query vector includes: identity information of the target zebrafish, current frame appearance feature vector, current frame motion feature vector, historical appearance feature vector, historical motion feature vector and historical motion trajectory information. The initial value of the target tracking query vector can be set to 0, and with the training and tracking process, it will be updated according to the detection result and motion prediction, and gradually accumulate the key features of the target.

[0083] In some embodiments, the expression of the above predicted displacement is: ; Among them, represents the predicted displacement, represents the target tracking query vector, denotes the first learning weight, denotes the second learning weight, denotes the first bias term, denotes the second bias term.

[0084] S1043, determining the predicted position coordinate value of the target zebrafish according to the predicted displacement.

[0085] Next, based on the predicted displacement and the historical position coordinate value determined in S1042, the predicted position coordinate value of the target zebrafish can be determined. In some embodiments, the expression of the predicted position coordinate value is as follows: ; wherein, denotes the predicted position coordinate value of the target zebrafish in the k-th video frame image, denotes the predicted position coordinate value of the target zebrafish in the k-1-th video frame image, , denotes the Gaussian noise.

[0086] wherein, for when (first frame), the bounding box coordinates output by the detection stage combined with the refining offset value determined, is the initial detection position of the target zebrafish.

[0087] when , is obtained through the tracking process of the k-1-th frame, that is, the position of the k-2-th frame is added to the displacement of the k-1 | k-2-th frame and the Gaussian noise determined.

[0088] S1044, based on the center heat map information, the bounding box coordinates, the refining offset value and the target tracking query vector, the predicted position coordinate value is associated and matched with the historical motion trajectory information to determine the associated matching result.

[0089] Further, in the data association stage, the detected target can be matched with the existing trajectory, that is, the predicted position coordinate value is associated and matched with the historical motion trajectory information.

[0090] In some embodiments, to associate the predicted position coordinate value with the historical motion trajectory information, the cross-frame affinity matrix , which represents the matching probability of the detection and tracking query. The association matching result can be a cross-frame affinity matrix, which can be used to represent the matching probability of the predicted position coordinate value and the historical motion trajectory information, and the expression of the cross-frame affinity matrix is: ; wherein, represents the cross-frame affinity matrix, represents the matching score of the predicted position coordinate value of the i th target zebrafish and the j th historical motion trajectory information, is determined based on the center heat map information, the bounding box coordinates, the refinement offset value, and the target tracking query vector.

[0091] The above can be obtained by calculating the similarity of the new detection result (center heat map information, bounding box coordinates, refinement offset value) and the target tracking query. Specifically, it includes appearance similarity, motion similarity, and position similarity. The appearance similarity is the matching degree of the appearance features of the detected target (such as the target existence probability reflected by the center heat map information and the region features corresponding to the bounding box coordinates) and the current frame appearance feature vector associated with the trajectory. The motion similarity is calculated according to the consistency of the predicted motion trend of the detected target (such as the position change of the bounding box coordinates) and the historical motion features (such as displacement and speed) of the existing trajectory. The closer the spatial distance between the bounding box center of the detected target and the predicted position of the existing trajectory in the current frame, the higher the score.

[0092] S1045, determining the target motion trajectory information corresponding to the plurality of target zebrafishes respectively based on the association matching result.

[0093] In some embodiments, S1045 specifically includes: First, determine the target pairing combination with the maximum matching probability by using the Hungarian algorithm or the KM algorithm, and the target pairing combination includes a group of predicted position coordinate values and corresponding historical motion trajectory information.

[0094] Then, in the case that the matching probability of the target pairing combination is greater than the matching threshold, the predicted position coordinate value and the corresponding historical motion trajectory information in the target pairing combination are associated and updated to determine the target motion trajectory information.

[0095] In one implementation, in the case that the matching probability of the target pairing combination is less than or equal to the matching threshold, the target zebrafish corresponding to the predicted position coordinate value can be determined as a new target, and a new trajectory is created. For the unmatched historical motion trajectory information, it can be temporarily retained, and if it is not matched for consecutive multiple frames, the historical motion trajectory information is deleted, and it is regarded as the disappearance of the target zebrafish.

[0096] In some embodiments, the center heat map information comprises: original detection confidence. Then, S104, target detection and tracking are performed according to the multi-modal fusion feature, and target motion trajectory information corresponding to each of the plurality of target zebrafish is determined, and the process further comprises: refining the original detection confidence based on the center heat map information to obtain refined detection confidence, and the expression of the refined detection confidence is: ; ; wherein, represents the refined detection confidence corresponding to the i th target zebrafish in the k th video frame image, represents the original detection confidence corresponding to the i th target zebrafish in the k th video frame image, is a fusion intensity coefficient, and the exemplary fusion intensity coefficient may be 2. represents the harmonic mean of the modal-specific confidence corresponding to the i th target zebrafish in the k th video frame image, represents the confidence corresponding to the appearance modal in the k th video frame image, represents the confidence corresponding to the motion modal in the k th video frame image, represents a minimum value.

[0097] Specifically, in order to further improve the target detection reliability, a cross-modal confidence fusion strategy is introduced, the original detection confidence is refined by using the complementary advantages of appearance and motion clues, and refined detection confidence is obtained. Specifically, the original detection confidence is mainly based on the appearance feature (center heat map), and the refined process introduces the harmonic of the modal-specific confidence (which can reflect the reliability of the motion modal), so that the confidence is more comprehensive. Moreover, it can effectively reduce the false detection rate: by adjusting the fusion intensity coefficient , the detection results with low reliability are suppressed, and the results with high reliability are enhanced, so that the final detection confidence is closer to the probability of the existence of the target.

[0098] In some embodiments, a multi-task loss function can be used to jointly optimize the processes of target detection, tracking and association in S104, and the expression of the specific multi-task loss function is as follows: ; wherein, is a center focus loss for punishing heat map prediction error, is a SmoothL1 loss for measuring the accuracy of the bounding box regression, can ensure target center refinement, The predicted and real motion displacement are aligned by IoU, The data association matrix is optimized by binary cross-entropy.

[0099] In this way, the long-term target identity consistency can be maintained by joint optimization in the multi-task loss function, while the dynamic visual problems in the zebrafish multi-target tracking scene in the space science experiment are adapted.

[0100] The method for tracking multiple zebrafish targets based on a space science experiment video provided in the embodiments of the application comprises the following steps: first, acquiring a zebrafish space science experiment video. Then, extracting motion information based on the zebrafish space science experiment video. Next, performing multi-modal feature fusion based on the motion information and appearance information to determine multi-modal fusion features. Finally, performing target detection and tracking based on the multi-modal fusion features to determine target motion trajectory information corresponding to each of the multiple target zebrafish.

[0101] The method extracts motion information, and models the depth interaction and fusion of the motion information and the appearance information (original RGB frame information) through a heterogeneous graph network, thereby enhancing the robustness of target detection, data association and trajectory prediction. Specifically, the motion of zebrafish in a complex background interference in a microgravity environment is nonlinear and random, and the linear motion assumption of traditional Kalman filtering can cause trajectory prediction deviation problems. The method is based on a motion modeling and adaptive threshold filtering motion information extraction scheme, and extracts reliable motion information through motion modeling, polar coordinate conversion and adaptive threshold filtering, which provides complementary semantic enhancement for appearance features. The appearance of zebrafish is highly similar and there is occlusion, and a single modality (such as RGB) can easily cause identity confusion (IDSW) and modal noise superposition caused by direct feature addition. The method uses a cross-modal fusion method based on a graph network structure, extracts motion-guided RGB features, enhances the robustness of motion blur area detection, and maintains the temporal consistency of identity during occlusion or rapid motion by introducing a cross-modal confidence fusion module. In this way, the method can effectively improve the stability and accuracy of multi-target tracking of zebrafish.

[0102] In some embodiments, the performance and effectiveness of the method for tracking multiple zebrafish targets based on a space science experiment video provided in the above embodiments of the application are verified and analyzed.

[0103] Experimental data set: verification analysis is performed based on the SpaceAnimal public data set. The data set is based on zebrafish video data of a space station life ecological experiment cabinet, selects a representative time period, and fine-tunes manual annotation to produce a zebrafish multi-target tracking data set, which contains 1,757 frames of images and 7,028 individual instances.

[0104] Evaluation method: the evaluation indexes for testing the performance of the zebrafish multi-target tracking method based on spatial science experiment video in the technical solution include: MOTA (multi-target tracking accuracy), IDF1 (identity F1 score), Frag (total number of track interruptions), and the specific calculation methods of MOTA and IDF1 are as follows: ; ; Wherein, MOTA is normalized by aggregating frame-by-frame errors (including false positives (FP), false negatives (FN) and identity switching (IDSW)) and the total number of true targets (GT) to reflect the overall tracking accuracy. The higher the MOTA score, the better the matching degree of the predicted trajectory and the true trajectory. IDF1 measures the accuracy of identity preservation based on identity true positives (IDTP), identity false positives (IDFP) and identity false negatives (IDFN). The closer the IDF1 value is to 1, the stronger the consistency of the identification of a single target zebrafish between different frames. The lower Frag indicates that the tracker can maintain more continuous and stable trajectories.

[0105] The following respectively adopts the zebrafish multi-target tracking method based on spatial science experiment video provided by the above-mentioned related technology one, related technology two and the embodiment of the present application (hereinafter referred to as: the present scheme) to conduct zebrafish multi-target tracking accuracy experiment comparison on zebrafish spatial science experiment video experimental data set. Table 1 is the zebrafish multi-target tracking accuracy experiment result.

[0106] Table 1 zebrafish multi-target tracking experiment result As shown in Table 1, the present scheme exceeds the related technical solutions in MOTA (82.21%), IDF1 (74.26) and Frag (36) and other indicators. The excellent performance of MOTA indicates that the present scheme has stronger robustness in dealing with background interference and complex target motion patterns, and has significant advantages in dealing with high-density and fast-moving targets in microgravity environment. In terms of identity preservation ability, the present scheme is more than 10% higher than the other two schemes in IDF1 index, which fully proves that the cross-modal re-detection mechanism effectively maintains the consistency of target identity when dealing with occlusion and fast motion. In terms of trajectory continuity, the present scheme realizes the lowest Frag value of 36, which is significantly better than related technology one (Frag value: 85) and related technology two (Frag value: 82), highlighting its ability to generate smooth and uninterrupted trajectories, which is crucial for capturing zebrafish clusters and social interactions and other fine-grained behavior patterns. It can be seen that the present scheme breaks through the limitations of other existing technical solutions relying on single visual features, and even if the appearance of zebrafish is highly similar, the uniqueness of the motion trajectory can still guarantee the accuracy and stability of tracking.

[0107] Further, in some embodiments, ablation experiment analysis is conducted to verify the effect of each step in the scheme. By adding the extracted motion information, constructing a heterogeneous graph, and cross-modal confidence fusion step by step, the influence of each step on the precision of zebrafish multi-target tracking is analyzed, and the ablation experiment results are shown in Table 2. In Table 2, “√” indicates that the corresponding step method is adopted, and “-” indicates that the corresponding step method is not adopted.

[0108] Table 2 Ablation experiment results of zebrafish multi-target tracking As shown in Table 2, the MOTA of the baseline method is 74.12%, the IDF1 is 60.14%, and the Frag is 85. Although the detection effect is acceptable, the identity consistency and robustness are still insufficient in the complex motion scene under microgravity, and there are frequent track interruption phenomena.

[0109] After adding the extracted motion information, the tracking precision and identity retention ability are improved, the MOTA reaches 79.09% (+4.97%), the IDF1 is improved to 67.79% (+7.65%). The Frag index is also improved to 69, indicating that the track loss is less and the tracking is more complete. These results show that extracting motion information helps to improve the motion estimation and tracking stability in dynamic environments.

[0110] Further constructing a heterogeneous graph, the performance is further improved by fusing appearance and motion features. The MOTA is 80.74% (+1.65%), the IDF1 reaches 71.22% (+3.43%). The Frag is optimized to 47, indicating that the target association ability is enhanced and the track loss is less. These results verify that constructing a heterogeneous graph enhances the feature expression ability and robustness under occlusion and visual blur conditions.

[0111] Finally, cross-modal confidence fusion is introduced to maintain identity and recover lost tracks during occlusion or rapid motion. Cross-modal confidence fusion has further performance improvement, MOTA is improved to 82.21% (+1.47%), IDF1 reaches 74.26% (+3.04%). The Frag index reaches 36, which shows that cross-modal confidence fusion can effectively deal with identity drift and track loss problems.

[0112] The embodiment of the application also provides a zebrafish multi-target tracking system based on a space science experiment video, in particular, Figure 4 The structure diagram of the zebrafish multi-target tracking system based on the space science experiment video provided by the embodiment of the application is shown in Figure 4As shown, the zebrafish multi-target tracking system 400 based on the space science experiment video includes an acquisition module 401, a motion information extraction module 402, a multi-modal feature fusion module 403, and a target detection and tracking module 404.

[0113] The acquisition module 401 can be configured to acquire a zebrafish space science experiment video, and the zebrafish space science experiment video is used to represent the motion process of a plurality of target zebrafish in a space station microgravity environment.

[0114] The motion information extraction module 402 can be configured to extract motion information based on the zebrafish space science experiment video, and the motion information is used to represent the motion state of a pixel point in the zebrafish space science experiment video.

[0115] The multi-modal feature fusion module 403 can be configured to perform multi-modal feature fusion based on the motion information and appearance information, determine multi-modal fusion features, and the appearance information is a video frame image of the zebrafish space science experiment video.

[0116] The target detection and tracking module 404 can be configured to perform target detection and tracking according to the multi-modal fusion features, and determine target motion trajectory information corresponding to the plurality of target zebrafish respectively.

[0117] The zebrafish multi-target tracking system based on the space science experiment video provided by the embodiment of the present application is used to acquire the zebrafish space science experiment video through the acquisition module. Then, the motion information extraction module extracts the motion information based on the zebrafish space science experiment video. Next, the multi-modal feature fusion module performs multi-modal feature fusion based on the motion information and appearance information, and determines the multi-modal fusion features. Finally, the target detection and tracking module performs target detection and tracking according to the multi-modal fusion features, and determines the target motion trajectory information corresponding to the plurality of target zebrafish respectively. The system extracts the motion feature information representing the dynamic change based on the adaptive threshold motion modeling, and then uses the heterogeneous graph network to model the interaction and fusion relationship of the appearance and motion features at each level, thereby enhancing the robustness of target detection, data association and trajectory prediction. In this way, the method can effectively improve the stability and accuracy of multi-target tracking of zebrafish.

[0118] The embodiment of the present application also provides an electronic device, which can include a display screen, a memory and one or more processors. The display screen, the memory and the processor are coupled. The memory is used to store computer program code, and the computer program code includes computer instructions. When the processor executes the computer instructions, the electronic device can execute each method or step executed in the above-mentioned zebrafish multi-target tracking method based on the space science experiment video. Of course, the electronic device includes but is not limited to the above-mentioned display screen, memory and one or more processors.

[0119] The embodiment of the present application also provides a computer readable storage medium for storing computer instructions of the above-mentioned zebrafish multi-target tracking method based on a space science experiment video.

[0120] Through the above description of the embodiments, those skilled in the art can clearly understand that, for the convenience and brevity of description, only the above-mentioned division of functional modules is taken as an example, and in actual application, the above-mentioned functions can be completed by different functional modules according to needs, that is, the internal structure of the device is divided into different functional modules to complete all or part of the functions described above.

[0121] In the description of the present application, it should be understood that the terms "first", "second" are only for the purpose of description, and cannot be understood as indicating or implying relative importance or implicitly indicating the number of indicated technical features. Therefore, the features defined as "first", "second" can be explicitly or implicitly included at least one of the features. In the description of the present application, the meaning of "a plurality of" is at least two, for example, two, three, etc., unless otherwise specifically limited.

[0122] In the description of the present application, the description of the terms "one embodiment", "some embodiments", "example", "specific example", or "some examples" means that the specific features, structures, materials or characteristics described in conjunction with the embodiment or example are included in at least one embodiment or example of the present application. In the present specification, the illustrative description of the above terms does not necessarily refer to the same embodiment or example. Moreover, the specific features, structures, materials or characteristics described can be combined in any appropriate manner in any one or more embodiments or examples. In addition, the skilled in the art can combine and combine the different embodiments or features of different embodiments or examples described in the present specification without contradiction.

[0123] The similar parts between the embodiments provided in the present application can be referred to each other, and the specific embodiments provided above are only a few examples under the general concept of the present application, and do not limit the protection scope of the present application. For those skilled in the art, any other embodiments extended according to the present application scheme without creative labor are within the protection scope of the present application.

Claims

1. A method for multi-target tracking of zebrafish based on space science experiment videos, characterized in that, include: Acquire zebrafish space science experiment videos, which are used to characterize the motion process of multiple target zebrafish in the microgravity environment of the space station. Motion information is extracted from the zebrafish space science experiment video, and the motion information is used to characterize the motion state represented by the pixels in the zebrafish space science experiment video. Multimodal feature fusion is performed based on the motion information and appearance information to determine the multimodal fusion features, wherein the appearance information is the video frame image of the zebrafish space science experiment video; Target detection and tracking are performed based on the multimodal fusion features to determine the target motion trajectory information corresponding to the multiple target zebrafish.

2. The method according to claim 1, characterized in that, The extraction of motion information based on the zebrafish space science experiment video includes: Motion modeling is performed based on the zebrafish space science experiment video to determine the first estimated displacement corresponding to each pixel in the video frame image of the zebrafish space science experiment video, and the first estimated displacement is in the Cartesian coordinate system. The first estimated displacement corresponding to each pixel is transformed by polar coordinates to obtain the second estimated displacement corresponding to each pixel. The second estimated displacement is used to characterize the motion amplitude and motion direction of the pixel in two adjacent video frames. Adaptive threshold filtering is performed based on the second estimated displacement corresponding to each pixel to determine the motion information corresponding to the pixel.

3. The method according to claim 1, characterized in that, The step of performing motion modeling based on the zebrafish space science experiment video to determine the first estimated displacement corresponding to each pixel in the video frame image of the zebrafish space science experiment video includes: The image intensity of each pixel in the first neighborhood is determined based on the zebrafish space science experiment video. The first estimated displacement corresponding to each pixel is determined by minimizing the error estimation method based on the image intensity of each pixel in the first neighborhood.

4. The method according to claim 3, characterized in that, The step of performing adaptive threshold filtering based on the second estimated displacement corresponding to each pixel to determine the motion information corresponding to the pixel includes: Based on the motion amplitude in the second estimated displacement corresponding to each pixel, determine the mean motion amplitude and standard deviation of the motion amplitude of the pixels in the first neighborhood. Based on the mean and standard deviation of the motion amplitude of pixels within the first neighborhood, a first filtering threshold and a second filtering threshold are determined for pixels within the first neighborhood; wherein the second filtering threshold is greater than the first filtering threshold, and the expression for the first filtering threshold is: ； in, This represents the first filtering threshold. Indicates the mean amplitude of motion. Indicates the standard deviation of the amplitude of motion. The first empirical control value is represented; the expression for the second filtering threshold is: ； in, This indicates the second filtering threshold. This represents the second empirical control value; Based on the motion amplitude corresponding to the pixel, and the first and second filtering thresholds within the first neighborhood of the pixel, the motion state represented by the pixel is classified to determine the motion information corresponding to the pixel. The expression for the motion information is: ； in, Represents pixels Corresponding motion information Represents pixels The corresponding amplitude of motion; in In the case of pixels Indicates suppression of static background; in In the case of pixels This represents a target zebrafish in an uncertain state of motion; in In the case of pixels This indicates that the target zebrafish is in motion.

5. The method according to any one of claims 1-4, characterized in that, The step of fusing multimodal features based on the motion information and appearance information to determine the multimodal fused features includes: Feature extraction is performed on the motion information and the appearance information respectively to obtain two-dimensional motion feature vectors and two-dimensional appearance feature vectors; A heterogeneous graph is constructed based on the two-dimensional motion feature vector and the two-dimensional appearance feature vector. The heterogeneous graph includes a set of nodes and a set of edges. The expression of the heterogeneous graph is: ； ； ； in, This represents the heterogeneous graph corresponding to the k-th video frame image. This represents the set of nodes corresponding to the k-th video frame. This represents the set of edges corresponding to the k-th video frame. This represents the appearance feature vector corresponding to the k-th video frame image. This represents the motion feature vector corresponding to the k-th video frame. This represents the motion trajectory information determined based on appearance features corresponding to the (k-1)th video frame. This represents the motion trajectory information determined based on motion features corresponding to the (k-1)th video frame. Representing temporal edges, For connection and and connections and ; Represents a spatial edge. Indicates heterogeneous edges. For connection and ; Based on the heterogeneous graph, attention encoding is performed using temporal attention mechanism, spatial attention mechanism and heterogeneous attention mechanism to obtain aggregated encoding features; The multimodal fusion features are obtained by deformably decoding the aggregated encoded features using a deformable attention mechanism.

6. The method according to claim 5, characterized in that, The step of detecting and tracking targets based on the multimodal fusion features, and determining the target motion trajectory information corresponding to the multiple target zebrafish, includes: Based on the multimodal fusion features, convolutional layers are used to process the data to obtain center heatmap information, bounding box coordinates, and fine-tuning offset values. The center heatmap information is used to characterize the confidence that a target zebrafish exists in the video frame image. The predicted displacement of the target zebrafish is predicted based on the target tracking query vector, which includes: the target zebrafish's identification information, the current frame appearance feature vector, the current frame motion feature vector, the historical appearance feature vector, the historical motion feature vector, and the historical motion trajectory information; The predicted position coordinates of the target zebrafish are determined based on the predicted displacement. Based on the central heat map information, the bounding box coordinates, the refined offset value, and the target tracking query vector, the predicted position coordinate value is associated and matched with the historical motion trajectory information to determine the association and matching result; Based on the association matching results, the target motion trajectory information corresponding to the multiple target zebrafish is determined.

7. The method according to claim 6, characterized in that, The expression for the central heat map information is: ； in, This represents the center heatmap information corresponding to the k-th video frame image. Indicates multimodal fusion features, Indicates a convolutional layer. This represents the Sigmoid activation function. Represents the ReLU activation function; The expression for the bounding box coordinates is: ； in, This represents the bounding box coordinates corresponding to the k-th video frame image; The expression for the fine-tuning offset value is: ； in, This represents the retouching offset value corresponding to the k-th video frame. This represents the Tanh activation function; The expression for the predicted displacement is: ； in, Indicates the predicted displacement. Represents the target tracking query vector. Indicates the first learning weight. This represents the second learning weight. Indicates the first bias term. Indicates the second bias term; The expression for the predicted location coordinates is: ； in, This represents the predicted location coordinates of the target zebrafish in the k-th video frame. This represents the predicted position coordinates of the target zebrafish in the (k-1)th frame of the video image. , This represents Gaussian noise.

8. The method according to claim 6, characterized in that, The association matching result is a cross-frame affinity matrix, which is used to characterize the matching probability between the predicted position coordinates and the historical motion trajectory information. The expression of the cross-frame affinity matrix is: ； in, Represents the cross-frame affinity matrix. This represents the matching score between the predicted position coordinates of the i-th target zebrafish and the j-th historical trajectory information. It is determined based on the central heatmap information, the bounding box coordinates, the refined offset value, and the target tracking query vector; The step of determining the target motion trajectory information corresponding to the multiple target zebrafish based on the association matching results includes: The target pairing combination with the highest matching probability is determined by the Hungarian algorithm or the KM algorithm. The target pairing combination includes a set of predicted position coordinates and corresponding historical motion trajectory information. If the matching probability of the target pairing is greater than the matching threshold, the predicted position coordinates and the corresponding historical motion trajectory information in the target pairing are associated and updated to determine the target motion trajectory information.

9. The method according to claim 6, characterized in that, The central heatmap information includes: the original detection confidence score; the step of performing target detection and tracking based on the multimodal fusion features to determine the target motion trajectory information corresponding to the multiple target zebrafish also includes: The original detection confidence score is refined based on the central heatmap information to obtain the refined detection confidence score, which is expressed as follows: ； ； in, This represents the refined detection confidence score for the i-th target zebrafish in the k-th video frame. This represents the original detection confidence score corresponding to the i-th target zebrafish in the k-th video frame image. The fusion strength coefficient, This represents the harmonic mean of the modality-specific confidence level corresponding to the i-th target zebrafish in the k-th video frame. This represents the confidence level corresponding to the appearance modality in the k-th video frame image. This represents the confidence level corresponding to the motion mode in the k-th video frame. This represents the minimum value.

10. A zebrafish multi-target tracking system based on space science experiment videos, characterized in that, include: The module includes an acquisition module, a motion information extraction module, a multimodal feature fusion module, and a target detection and tracking module; among which, The acquisition module is used to acquire zebrafish space science experiment videos, which are used to characterize the movement process of multiple target zebrafish in the microgravity environment of the space station. The motion information extraction module is used to extract motion information based on the zebrafish space science experiment video, and the motion information is used to characterize the motion state represented by the pixels in the zebrafish space science experiment video. The multimodal feature fusion module is used to perform multimodal feature fusion based on the motion information and appearance information to determine the multimodal fusion features, wherein the appearance information is a video frame image of the zebrafish space science experiment video; The target detection and tracking module is used to detect and track targets based on the multimodal fusion features, and to determine the target motion trajectory information corresponding to the multiple target zebrafish.

Images