Zebrafish three-dimensional trajectory tracking method and system based on cloud-edge collaboration

Abstract

The application discloses a cloud-edge collaborative zebra fish three-dimensional trajectory tracking method and system, belongs to the underwater visual perception computing technical field, and aims to solve the technical problem of how to stably and timely track and reconstruct the three-dimensional trajectories of multiple zebra fishes in multiple incubators. The method comprises the following steps: constructing a target recognition model based on a YOLOv5 algorithm, and performing the following on the edge side: obtaining the position information of the zebra fishes through the trained target recognition model, associating the position information and the ID of the zebra fishes through an OC-SORT tracking algorithm, and timely uploading the frame number of the current video image and the ID and position information of each zebra fish in the video image to the cloud side; three-dimensional trajectory construction: calculating the weight of the edges in the directed graph on the basis of realizing trajectory matching through the construction of the directed graph, and realizing the three-dimensional matching of the movement trajectories of the zebra fishes under two different visual angles through multi-view information data fusion.

Description

Technical Field

[0001] This invention relates to the field of underwater visual perception computing technology, specifically to a cloud-edge collaborative method and system for three-dimensional trajectory tracking of zebrafish. Background Technology

[0002] Animal behavior science primarily studies the occurrence, development, evolution, and function of animal behavior. It is a comprehensive discipline integrating multiple disciplines such as ethology, ecology, psychology, genetics, evolutionary biology, sociology, and economics. The research subjects in animal behavior science are diverse. Among them, zebrafish, due to their 87% genetic similarity to humans and their similar organ systems, represent a realistic model of human diseases. Furthermore, zebrafish are cost-effective and efficient in drug screening, making them a very important and popular model organism internationally. Compared to rodents, zebrafish have lower breeding and maintenance costs, shorter research cycles, and are easy to feed, have a high survival rate, and are relatively resistant to disease. Compared to other vertebrates, they have advantages such as high egg production, readily available embryos, short developmental cycles, and ease of observation. Therefore, in the past few decades, the frequency of zebrafish use as an animal model has increased significantly, making them another major experimental research subject after rats and mice.

[0003] The effects of neurological drugs can manifest in the behavior of model organisms. Zebrafish movement patterns reflect the state of the nervous system, making them a common drug screening model for research on human neurological diseases, addictive drugs, and social anxiety disorders. Analyzing the movement and behavior of zebrafish populations after drug administration is crucial for verifying drug efficacy and is a key part of neuroscience and biological research. However, most researchers at this stage still rely on traditional methods of manual labeling, which is time-consuming and labor-intensive. Furthermore, manual inspection is subjective and limited to small-scale experiments. Commercially available zebrafish behavior tracking systems can only track one zebrafish per culture chamber, failing to capture the gregarious behavior characteristics of zebrafish. Due to limitations in efficiency and objectivity, a system for automatically tracking and analyzing the behavior of zebrafish populations is needed to improve research speed and reduce experimental errors.

[0004] In recent years, target tracking has made significant breakthroughs, especially with the improvement in target detection accuracy, leading to an increasing number of detection-based target tracking models. However, most solutions are designed for tracking based on human appearance re-identification and linear motion prediction, and most research is based on 2D planar observations, making it difficult to directly extend to 3D scenes. Due to the nonlinearity of zebrafish movement, their similar appearance, and their gregarious social behavior, it is difficult to achieve accurate tracking through detection tracking on a single plane or existing tracking algorithms that rely on appearance re-identification.

[0005] The technical problem that needs to be solved is how to stably and in real time perform three-dimensional trajectory tracking and reconstruction of multiple zebrafish in multiple incubators. Summary of the Invention

[0006] The technical objective of this invention is to address the above-mentioned shortcomings by providing a cloud-edge collaborative method and system for three-dimensional trajectory tracking of zebrafish, thereby solving the technical problem of how to stably and in real-time perform three-dimensional trajectory tracking and reconstruction of multiple zebrafish in multiple incubators.

[0007] In a first aspect, the present invention provides a cloud-edge collaborative zebrafish three-dimensional trajectory tracking method, comprising the following steps:

[0008] Image acquisition: Simultaneously capture video of multiple zebrafish in the same environment from two different perspectives at the edge, and obtain video streams from the two perspectives;

[0009] Target recognition and tracking: A target recognition model is built based on the YOLOv5 algorithm. For each frame of video image in each video stream, the following is performed at the edge: The zebrafish in the video image is identified by the trained target recognition model to obtain the zebrafish's position information. Based on the zebrafish's position information, an ID is assigned to each zebrafish in the video image using the OC-SORT tracking algorithm. The zebrafish is tracked based on its motion and appearance features. The position information and ID of the zebrafish are associated to obtain the ID and position information of each zebrafish in the video image. The current frame number of the video image and the ID and position information of each zebrafish in the video image are uploaded to the cloud in real time.

[0010] 3D trajectory construction: Based on the position information and ID of zebrafish in each frame of video image from different perspectives, a directed graph is constructed to achieve trajectory matching. The weights of the edges in the directed graph are calculated based on the similarity of the zebrafish's appearance features and motion features from different perspectives. The 3D matching of the zebrafish's motion trajectory from two different perspectives is achieved through multi-view information data fusion.

[0011] As a preferred method, for the constructed target recognition model, multiple sample images are collected, each containing at least one zebrafish, and the positions of the zebrafish in the sample images are labeled. The sample images and labels are used as sample data. The target recognition model is trained and tested based on the sample data to obtain the weight file corresponding to the target recognition model. After pruning the weight file, the weight file is written to the edge end to load the trained target recognition model to the edge end.

[0012] As a preferred option, the edge device has a processor and two high-definition cameras. The processor is equipped with an RK3588S chip, and the high-definition cameras are electrically connected to the RK3588S chip via MIPI cables.

[0013] The server is equipped with a cloud server and a cloud database. The cloud server is an MQTT server, and the RK3588S chip interacts with the cloud server based on the MQTT protocol.

[0014] The weight file after pruning and the OC-SORT tracking algorithm are both written into the RK3588S chip;

[0015] During image acquisition, two cameras simultaneously capture video of multiple zebrafish in the same environment from their respective perspectives, resulting in video streams from two different perspectives.

[0016] During target recognition and tracking, for each frame of video image, the frame number of the video image, as well as the ID and location information of each zebrafish in the video image, are uploaded to the cloud in real time and stored in the cloud database;

[0017] During the 3D trajectory construction, the cloud server reads the frame number of each video image and the ID and location information of each zebrafish in the video image from the cloud database.

[0018] Preferably, the 3D trajectory construction includes the following steps:

[0019] Node generation: Taking individual zebrafish as the target, based on the two-dimensional motion trajectory of the zebrafish from two perspectives, the nodes are created and their weights are calculated by calculating the target that appears in both perspectives simultaneously. The nodes are then added to the directed acyclic graph. For the time period between the zebrafish's most recent detection time and its first detection time, a set of node pairs is constructed based on the two-dimensional trajectory of the zebrafish from both perspectives.

[0020] Connecting nodes: For each node pair in the set of node pairs, calculate the weight of the directed edge between each node pair.

[0021] Generating 3D trajectories: For a directed acyclic graph (DAG), the longest path in the DAG is calculated recursively and the nodes are output as 3D trajectories. When calculating the longest path, the weights of all nodes and edges in the path are summed and the maximum value is selected as the longest path. After each extraction of the longest path, the nodes related to the longest path and other nodes with the same 2D trajectory as the node are deleted from the DAG until the DAG is empty.

[0022] As a preferred method, node generation includes the following steps:

[0023] For two perspectives, a first-person view and a second-person view, using the zebrafish trajectory in the first-person view as a baseline, multiple candidate points can be found for individual zebrafish targets detected in the first-person view in the second-person view. For the same ID appearing simultaneously in both the first-person and second-person views, through... Calculate the frames of the video image where the ID is detected from both viewpoints, where, This represents the set of video images whose ID was detected in the first-person perspective. This refers to the set of frames of video images in which the ID was detected from a second-view perspective;

[0024] The formula for calculating the weight of the matched node in each frame of the video image is:

[0025]

[0026] Among them, through The calculated intersecting frames are as follows: , This refers to the zebrafish from a second-person perspective. Projection error in water, This represents the reciprocal of the average reprojection error of the training dataset. This indicates that zebrafish were detected. In the experimental environment, what is the total weight of the matched nodes within a predetermined time period? Weights of each frame Composition, total weight The calculation formula is as follows:

[0027]

[0028] when When a node is added to a directed acyclic graph;

[0029] When the first-view 2D trajectory and the second-view 2D trajectory are combined and added as nodes to a directed acyclic graph, a set of node pairs is constructed. During the time period Inside ,in Used for time evaluation Is it in Before, Used to ensure They will not overlap in time. ,node Comprising two-dimensional trajectories from a first-person perspective Two-dimensional trajectory from a second perspective 3D trajectory And the quadruple representation of the weights of the node pairs;

[0030] Correspondingly, the set of node pairs Each node pair in the time period Inside, from the node To the node weight of directed edges The calculation formula is as follows:

[0031]

[0032] The weights of the edges are calculated as the positive complement probabilities of the exponential cumulative distribution function (CDF). , For zebrafish from Nodes within the time period The two-dimensional trajectory motion characteristics, for The time difference between them Represents a node The weight, Represents a node The weight, The value was chosen based on experience. The values ​​given for the training data;

[0033] The training dataset is a dataset constructed from sample data used for model sequence and testing of the target recognition model. The sample data includes sample images and annotations. Each sample image contains at least one zebrafish, and the annotations are the labels for the zebrafish in the sample images.

[0034] In a second aspect, the present invention provides a cloud-edge collaborative zebrafish three-dimensional trajectory tracking system, comprising a cloud and multiple edge terminals, wherein the edge terminals and the cloud cooperate to execute a cloud-edge collaborative zebrafish three-dimensional trajectory tracking method as described in any of the first aspects.

[0035] The edge is used to perform the following:

[0036] Image acquisition: Simultaneously capture video of multiple zebrafish in the same environment from two different perspectives at the edge, and obtain video streams from the two perspectives;

[0037] Target recognition and tracking: A target recognition model is built based on the YOLOv5 algorithm. For each frame of video image in each video stream, the following is performed at the edge: The zebrafish in the video image is identified by the trained target recognition model to obtain the zebrafish's position information. Based on the zebrafish's position information, an ID is assigned to each zebrafish in the video image using the OC-SORT tracking algorithm. The zebrafish is tracked based on its motion and appearance features. The position information and ID of the zebrafish are associated to obtain the ID and position information of each zebrafish in the video image. The current frame number of the video image and the ID and position information of each zebrafish in the video image are uploaded to the cloud in real time.

[0038] Correspondingly, the cloud is used to execute the following:

[0039] 3D trajectory construction: Based on the position information and ID of zebrafish in each frame of video image from different perspectives, a directed graph is constructed to achieve trajectory matching. The weights of the edges in the directed graph are calculated based on the similarity of the zebrafish's appearance features and motion features from different perspectives. The 3D matching of the zebrafish's motion trajectory from two different perspectives is achieved through multi-view information data fusion.

[0040] As a preferred approach, before deploying the trained target recognition model on the development board at the edge, the following operations are performed on the constructed target recognition model: multiple sample images are collected, each containing at least one zebrafish, and the positions of the zebrafish in the sample images are labeled. The sample images and labels are used as sample data. The target recognition model is trained and tested based on the sample data to obtain the weight file corresponding to the target recognition model. After pruning the weight file, the weight file is written to the edge to load the trained target recognition model onto the edge.

[0041] As a preferred option, the edge device has a processor and two high-definition cameras. The processor is equipped with an RK3588S chip, and the high-definition cameras are electrically connected to the RK3588S chip via MIPI cables.

[0042] The server is equipped with a cloud server and a cloud database. The cloud server is an MQTT server, and the RK3588S chip interacts with the cloud server based on the MQTT protocol.

[0043] The weight file after pruning and the OC-SORT tracking algorithm are both written into the RK3588S chip;

[0044] During image acquisition, the edge device is used to simultaneously capture video of multiple zebrafish in the same environment from their respective perspectives using two cameras, resulting in video streams from two different perspectives.

[0045] During target recognition and tracking, for each frame of video image, the edge device uses the RK3588S chip to upload the frame number of the video image, as well as the ID and location information of each zebrafish in the video image, to the cloud and store them in the cloud database in real time.

[0046] During the 3D trajectory construction, the cloud server is used to read the frame number of each video image and the ID and location information of each zebrafish in the video image from the cloud database.

[0047] Preferably, the cloud is used to perform the following construction of the 3D trajectory:

[0048] Node generation: Taking individual zebrafish as the target, based on the two-dimensional motion trajectory of the zebrafish from two perspectives, the nodes are created and their weights are calculated by calculating the target that appears in both perspectives simultaneously. The nodes are then added to the directed acyclic graph. For the time period between the zebrafish's most recent detection time and its first detection time, a set of node pairs is constructed based on the two-dimensional trajectory of the zebrafish from both perspectives.

[0049] Connecting nodes: For each node pair in the set of node pairs, calculate the weight of the directed edge between each node pair.

[0050] Generating 3D trajectories: For a directed acyclic graph (DAG), the longest path in the DAG is calculated recursively and the nodes are output as 3D trajectories. When calculating the longest path, the weights of all nodes and edges in the path are summed and the maximum value is selected as the longest path. After each extraction of the longest path, the nodes related to the longest path and other nodes with the same 2D trajectory as the node are deleted from the DAG until the DAG is empty.

[0051] Preferably, the cloud is used to execute the following generation nodes:

[0052] For two perspectives, a first-person view and a second-person view, using the zebrafish trajectory in the first-person view as a baseline, multiple candidate points can be found for individual zebrafish targets detected in the first-person view in the second-person view. For the same ID appearing simultaneously in both the first-person and second-person views, through... Calculate the frames of the video image where the ID is detected from both viewpoints, where, This represents the set of video images whose ID was detected in the first-person perspective. This refers to the set of frames of video images in which the ID was detected from a second-view perspective;

[0053] The formula for calculating the weight of the matched node in each frame of the video image is:

[0054]

[0055] Among them, through The calculated intersecting frames are as follows: , This refers to the zebrafish from a second-person perspective. Projection error in water, This represents the reciprocal of the average reprojection error of the training dataset. This indicates that zebrafish were detected. In the experimental environment, what is the total weight of the matched nodes within a predetermined time period? Weights of each frame Composition, total weight The calculation formula is as follows:

[0056]

[0057] when When a node is added to a directed acyclic graph;

[0058] When the first-view 2D trajectory and the second-view 2D trajectory are combined and added as nodes to a directed acyclic graph, a set of node pairs is constructed. During the time period Inside ,in Used for time evaluation Is it in Before, Used to ensure They will not overlap in time. ,node Comprising two-dimensional trajectories from a first-person perspective Two-dimensional trajectory from a second perspective 3D trajectory And the quadruple representation of the weights of the node pairs;

[0059] Correspondingly, the set of node pairs Each node pair in the time period Inside, from the node To the node weight of directed edges The calculation formula is as follows:

[0060]

[0061] The weights of the edges are calculated as the positive complement probabilities of the exponential cumulative distribution function (CDF). , For zebrafish from Nodes within the time period The two-dimensional trajectory motion characteristics, for The time difference between them Represents a node The weight, Represents a node The weight, The value was chosen based on experience. The values ​​given for the training data;

[0062] The training dataset is a dataset constructed from sample data used for model sequence and testing of the target recognition model. The sample data includes sample images and annotations. Each sample image contains at least one zebrafish, and the annotations are the labels for the zebrafish in the sample images.

[0063] The cloud-edge collaborative zebrafish 3D trajectory tracking method and system of the present invention have the following advantages:

[0064] 1. Video streams are acquired from two perspectives. For each frame of video image in each perspective, a target recognition model based on the Yolov5 algorithm is used to identify zebrafish in the video image. A real-time online tracking (OC-SORT) algorithm is introduced to assign IDs to the tracked zebrafish and bind the IDs to their location information. Based on the zebrafish's location information and assigned IDs, nodes are created for fusion and reconstruction of the zebrafish's 3D motion trajectory. Compared to traditional 2D tracking algorithms, this reduces the ID exchange frequency. The accuracy of 2D tracking affects the association and fusion of multi-view trajectories in the cloud server, such as... Figure 3 As shown, the algorithm requires the same ID to exist in both views to create nodes for mapping. For low-precision 2D tracking algorithms, many nodes are discarded, resulting in a large difference between the reconstructed trajectory block and the real trajectory block. This invention proposes a Yolov5+OC-SORT algorithm scheme suitable for zebrafish tracking by comparing it with existing 2D trajectory tracking algorithms, which greatly improves the accuracy of zebrafish 3D trajectory reconstruction and tracking.

[0065] 2. By deploying the 2D trajectory tracking algorithm on the RK3588S edge device, the image acquisition and target recognition tracking tasks are offloaded to the edge. The 2D trajectory fusion is then used to reconstruct the 3D trajectory in the cloud. This reduces the computational power requirements of the cloud server for zebrafish trajectory reconstruction and alleviates the pressure on the cloud server-centric system architecture. The cloud server-centric system architecture requires uploading video to the cloud, where the cloud server performs video detection, tracking, and reconstruction. This places extremely high demands on the cloud server's computing power and bandwidth, which cannot meet real-time requirements. By adopting a cloud-edge collaborative system architecture, the detection and tracking tasks are offloaded to the edge, closer to the video source. The final results are uploaded to the cloud server only via MQTT, which greatly alleviates the pressure on the cloud server and ensures real-time performance. One server can simultaneously process real-time data from N different zebrafish culture chambers (the number of N depends on the performance of the cloud server), which meets actual needs. At the same time, the modular design ensures that the failure of equipment in one culture chamber does not affect the operation of other equipment in the system. Attached Figure Description

[0066] To more clearly illustrate the technical solutions in the embodiments of the present invention, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0067] The invention will be further described below with reference to the accompanying drawings.

[0068] Figure 1 This document describes the system workflow and architecture of a cloud-edge collaborative zebrafish 3D trajectory tracking method in Example 1.

[0069] Figure 2 The flowchart of the OC-SORT two-dimensional tracking algorithm in the cloud-edge collaborative zebrafish three-dimensional trajectory tracking method in Example 1 is shown.

[0070] Figure 3 This is a flowchart of the three-dimensional trajectory construction process in a cloud-edge collaborative zebrafish three-dimensional trajectory tracking method in Example 1.

[0071] Figure 4 This is a block diagram of the MQTT protocol communication architecture in a cloud-edge collaborative zebrafish 3D trajectory tracking method in Example 1;

[0072] Figure 5 This is an example diagram of data fusion during the 3D trajectory construction process in a cloud-edge collaborative zebrafish 3D trajectory tracking method in Example 1;

[0073] Figure 6 This is an example diagram of path selection during the 3D trajectory construction process in a cloud-edge collaborative zebrafish 3D trajectory tracking method in Example 1;

[0074] Figure 7 This is a visualization of the comparative experiment between two-dimensional trajectory tracking and traditional algorithms in a cloud-edge collaborative zebrafish three-dimensional trajectory tracking method in Example 1.

[0075] Figure 8 This relates to the real-time performance of edge devices performing two-dimensional trajectory tracking on the input video stream in a cloud-edge collaborative zebrafish 3D trajectory tracking method in Example 1 (influenced by the fact that DeepSORT's ReID network has a single inference time of approximately 3ms, which is greatly affected by the number of zebrafish because each detection box requires inference).

[0076] Figure 9 This is a schematic diagram showing the comparison between two-dimensional trajectory tracking and traditional algorithms in a cloud-edge collaborative zebrafish three-dimensional trajectory tracking method in Example 1 (02 in the table represents two fish, and top and front represent different 2D views respectively).

[0077] Figure 10 This is a schematic diagram showing the comparison between two-dimensional trajectory tracking and traditional algorithms in a cloud-edge collaborative zebrafish three-dimensional trajectory tracking method in Example 1 (02 in the table represents two fish, and top and front represent different 2D views respectively).

[0078] Figure 11This is a schematic diagram comparing the three-dimensional trajectory tracking method for zebrafish using cloud-edge collaboration in Example 1 with traditional algorithms. Detailed Implementation

[0079] The present invention will be further described below with reference to the accompanying drawings and specific embodiments, so that those skilled in the art can better understand and implement the present invention. However, the embodiments are not intended to limit the present invention. In the absence of conflict, the embodiments of the present invention and the technical features in the embodiments can be combined with each other.

[0080] Example 1:

[0081] This invention discloses a cloud-edge collaborative method and system for 3D trajectory tracking of zebrafish, comprising three steps: image acquisition, target recognition and tracking, and 3D trajectory reconstruction, as detailed below:

[0082] Step 100 Image Acquisition: Simultaneously acquire video from two different perspectives at the edge of the same environment for multiple zebrafish, and obtain video streams from the two perspectives;

[0083] Step 200 Target Recognition and Tracking: Based on the YOLOv5 algorithm, a target recognition model is constructed. For each frame of video image in each video stream, the following is performed at the edge: The zebrafish in the video image is identified by the trained target recognition model to obtain the zebrafish's position information. Based on the zebrafish's position information, an ID is assigned to each zebrafish in the video image using the OC-SORT tracking algorithm. The zebrafish is tracked based on its motion and appearance features. The zebrafish's position information and ID are associated to obtain the ID and position information of each zebrafish in the video image. The current frame number of the video image and the ID and position information of each zebrafish in the video image are uploaded to the cloud in real time.

[0084] Step S300: 3D trajectory construction: Based on the position information and ID of the zebrafish in each frame of video image from different perspectives, a directed graph is constructed to achieve trajectory matching. Based on the similarity of the zebrafish's appearance features and motion features from different perspectives, the weights of the edges in the directed graph are calculated. The 3D matching of the zebrafish's motion trajectory from two different perspectives is achieved through multi-view information data fusion.

[0085] In this embodiment, image acquisition and target recognition and tracking are performed at the edge, while 3D trajectory construction is performed in the cloud. The edge device consists of a processor and two high-definition cameras. The processor is equipped with an RK3588S chip, and the high-definition cameras are RGB cameras.

[0086] The cloud server is deployed in the cloud. In this embodiment, Alibaba Cloud ECS is selected. EMQX is used to build the server into an MQTT server. A database is installed in the cloud server as a cloud database and a Python environment is configured. The RK3588S chip interacts with the cloud server based on the MQTT protocol.

[0087] As a concrete implementation of edge computing and cloud computing, the camera uses the CAM-8MS1M sensor, which supports up to 800W pixels and a maximum resolution of 3840*2160. It is connected to the edge development board via a MIPI cable. The edge development board is equipped with the RK3588S chip, a new generation flagship high-end processor launched by Rockchip. It adopts an 8nm process design, is equipped with an eight-core CPU with four A76 cores and four A55 cores, and an Arm high-performance GPU. It also has a built-in NPU with 6T computing power, which has powerful visual processing capabilities and can be expanded for AI computing power, image data processing, etc. It can connect to the Internet via Ethernet cable and communicate with cloud servers and cloud databases via MQTT protocol.

[0088] Before performing step 100 (image acquisition), a zebrafish environment was constructed. Specifically, a standard glass aquarium was selected as the experimental tank, with dimensions of 30 × 30 × 30 cm and a water depth of 15 cm. To reduce reflection, three-quarters of the window glass was covered with translucent plastic. The plastic was 30 × 30 cm in size, with a luminous flux of 670 lumens and a color temperature of 4000K.

[0089] During image acquisition, the RGB camera captures video streams from both the frontal and top-down views of the fish tank.

[0090] In step S200, zebrafish recognition is performed on each frame of the video stream using the trained target model to obtain the zebrafish's position information in the video image. The trained target recognition model is obtained and deployed at the edge as follows: For the constructed target recognition model, multiple sample images are collected, each containing at least one zebrafish. The positions of the zebrafish in the sample images are labeled. The sample images and labels are used as sample data. Based on the sample data, the target recognition model is trained and tested to obtain a weight file corresponding to the target recognition model. After pruning the weight file, it is written to the edge, thus loading the trained target recognition model onto the edge.

[0091] In this embodiment, eight 2K offline video streams from different perspectives, namely frontal and top views of the fish tank, were acquired by an RGB camera. Each frame of the video stream was used as a sample image, and the position information of the zebrafish in the sample image was labeled. A sample set was constructed based on the sample images and the labels, and the sample set was divided into a training set and a test set to train and test the constructed target recognition model.

[0092] To ensure lightweight and real-time performance of target recognition and tracking after deployment on the RK3588S, a lightweight one-stage detection model, YOLOv5-Small, was selected for target recognition model construction through comparison. The weight file was obtained from the target recognition model constructed using the aforementioned training and testing sets. Since the RK3588S is an embedded platform, this embodiment optimized and pruned the weight file of the target recognition model, reducing its size by approximately 75% while decreasing the MAP by only 0.5%. This embodiment also rewrote the target recognition model using C++.

[0093] Target recognition is performed using a trained target recognition model. The bounding box detected in each frame of video image represents the zebrafish's position information in that frame. Initially, the OC-SORT algorithm assigns an ID to each zebrafish in the tank based on its position information. In the next frame, the algorithm detects zebrafish in real-time and associates trajectories with the same ID based on the zebrafish's position information tracked by motion and appearance. If no existing trajectory matches (indicating a new object or an object that has reappeared after being occluded), a new trajectory is created for that object and assigned a unique ID. In this embodiment, the OC-SORT algorithm was rewritten in C++ for deployment to adapt to the RK3588S. The algorithm is described below; see the flowchart for details. Figure 2 .

[0094] OC-SORT (Observation-Centric SORT) is an algorithm with lower noise and greater robustness to occlusion and nonlinear motion. It was proposed to address the limitations of the SORT algorithm, such as its sensitivity to state noise, the accumulation of errors over time, and its reliance on Kalman filter estimation as the primary observation in the event of occlusion. We introduce the momentum of zebrafish motion into the association phase of OC-SORT, employing an observation-centered online smoothing (OOS) strategy and adding an observation-centered momentum term (OCM) to the association cost to offset accumulated errors during periods of untracking. Furthermore, OC-SORT incorporates observation-centered recovery (OCR) to search for missing objects around its last observation.

[0095] Observation-centered online smoothing (OOS) strategy: When the trajectory of a zebrafish is in Lost at that time Re-association can be performed using different parameters to generate virtual trajectories. = Following this virtual trajectory, let it return to the lost moment, from The filter parameters are checked at each step of the KF prediction update process. With the supervision of the virtual trajectory, the error in the state estimation will not accumulate. The updated state estimate is... .

[0096] Center Momentum Term (OCM): Linear motion models assume consistent velocity direction, but this assumption is not valid in reality due to the nonlinearity and noise of zebrafish motion. While we can approximate zebrafish motion as linear over relatively short time intervals, noise still affects the consistency of velocity direction. OC-SORT proposes a method to reduce noise by adding a velocity consistency (momentum) term to the cost matrix. This uses observations associated with the trajectory for direction calculation, thus avoiding error accumulation in the estimated state.

[0097] Observation-Centered Reconstruction (OCR): Generally, trajectory loss is caused by zebrafish observation loss (occlusion or unreliable detection) and nonlinear motion. The SORT algorithm's reliance on KF estimation as the primary method and observation as a secondary method leads to a significant accumulation of trajectory errors. To reduce these errors, OC-SORT proposes Observation-Centered Reconstruction (OCR), which searches and correlates the location of the lost trajectory.

[0098] Considering that the RK3588S chip at the edge is an embedded platform, this embodiment uses the C / C++ version of the paho-mqtt-embended library to undertake the task of sending data to the server. The specific data includes the frame number of each video image detected by the RK3588S during runtime, the zebrafish's location information, and the ID of each zebrafish.

[0099] In step S200, the cloud database stores the position information and ID of each frame of video images of the zebrafish from different perspectives. Based on the zebrafish's position information from each perspective, the two-dimensional motion trajectory of the zebrafish from that perspective can be obtained. Figure 5As shown, assuming there are two zebrafish tracks in the top view and three tracks in the side view within the same time period, matching the zebrafish position information in the top view with the position information in the side view at the same moment will result in three possibilities. By calculating the length of the common occurrence in the top and side views at that moment, the probabilities of the three combinations can be obtained (data on the nodes in the right figure). Then, by calculating the appearance and motion characteristics of the zebrafish from different perspectives, the probability of matching the top and side views in the next frame can be obtained (data on the edge of the right figure), thus completing the reconstruction of different track segments. In this embodiment, the 3D trajectory construction includes three steps: generating nodes, connecting nodes, and generating 3D trajectories. The specific flowchart of the algorithm is shown below. Figure 3 As shown.

[0100] Node generation: Taking individual zebrafish as the target, based on the two-dimensional motion trajectory of the zebrafish from two perspectives, the nodes are created and their weights are calculated by calculating the target that appears in both perspectives simultaneously. The nodes are then added to the directed acyclic graph. For the time period between the zebrafish's most recent detection time and its first detection time, a set of node pairs is constructed based on the two-dimensional trajectory of the zebrafish from both perspectives.

[0101] The specific implementation of node generation includes the following steps:

[0102] For two perspectives, a first-person view and a second-person view, using the zebrafish trajectory in the first-person view as a baseline, multiple candidate points can be found for individual zebrafish targets detected in the first-person view in the second-person view. For the same ID appearing simultaneously in both the first-person and second-person views, through... Calculate the frames of the video image where the ID is detected from both viewpoints, where, This represents the set of video images whose ID was detected in the first-person perspective. This refers to the set of frames of video images in which the ID was detected from a second-view perspective;

[0103] The formula for calculating the weight of the matched node in each frame of the video image is:

[0104]

[0105] Among them, through The calculated intersecting frames are as follows: , This refers to the zebrafish from a second-person perspective. Projection error in water, This represents the reciprocal of the average reprojection error of the training dataset. This indicates that zebrafish were detected. In the experimental environment, what is the total weight of the matched nodes within a predetermined time period? Weights of each frame Composition, total weight The calculation formula is as follows:

[0106]

[0107] when When a node is added to a directed acyclic graph;

[0108] When the first-view 2D trajectory and the second-view 2D trajectory are combined and added as nodes to a directed acyclic graph, a set of node pairs is constructed. During the time period Inside ,in Used for time evaluation Is it in Before, Used to ensure They will not overlap in time. ,node Comprising two-dimensional trajectories from a first-person perspective Two-dimensional trajectory from a second perspective 3D trajectory And the weights of the node pairs are represented by quadruples.

[0109] Connecting nodes: For each node pair in the set of node pairs, calculate the weight of the directed edge between each node pair.

[0110] Node Pair Set Each node pair in the time period Inside, from the node To the node weight of directed edges The calculation formula is as follows:

[0111]

[0112] The weights of the edges are calculated as the positive complement probabilities of the exponential cumulative distribution function (CDF). , For zebrafish from Nodes within the time period The two-dimensional trajectory motion characteristics, for The time difference between them Represents a node The weight, Represents a node The weight, The value was chosen based on experience. The values ​​given for the training data;

[0113] The training dataset is a dataset constructed from sample data used for model sequence and testing of the target recognition model. The sample data includes sample images and annotations. Each sample image contains at least one zebrafish, and the annotations are the labels for the zebrafish in the sample images.

[0114] Generating 3D trajectories: For a directed acyclic graph (DAG), the longest path in the DAG is calculated recursively and the nodes are output as 3D trajectories. When calculating the longest path, the weights of all nodes and edges in the path are summed and the maximum value is selected as the longest path. After each extraction of the longest path, the nodes related to the longest path and other nodes with the same 2D trajectory as the node are deleted from the DAG until the DAG is empty.

[0115] The method in this embodiment provides a zebrafish 3D trajectory tracking based on RK3588S cloud-edge collaboration. This method combines computer vision, Internet of Things, and wireless communication technologies, and the resulting advantages include: not only reducing the bandwidth required for video transmission, but also reducing data latency due to the physical proximity of the edge server to the video source, but also reducing the computational power requirements of the cloud server for zebrafish trajectory reconstruction due to the offloading of detection and tracking tasks.

[0116] This embodiment also applies Yolov5 and the observation-centered real-time online tracking (OC-SORT) algorithm to the 2D scenario of zebrafish tracking to improve tracking accuracy, such as... Figure 7-10 As shown, compared with traditional 2D target tracking algorithms, the application of OC-SORT significantly reduces the frequency of ID exchange. Furthermore, by fusing and reconstructing the 3D trajectory of the zebrafish based on the positional information of the zebrafish tracked from different planar perspectives (top view, side view) and the assigned IDs, the accuracy of 3D trajectory tracking of the zebrafish is greatly improved.

[0117] Example 2:

[0118] The present invention discloses a cloud-edge collaborative zebrafish three-dimensional trajectory tracking system, comprising an edge terminal and a cloud terminal, wherein the edge terminal and the cloud terminal cooperate to execute the method disclosed in Example 1.

[0119] In this embodiment, the edge terminal is used to perform the following:

[0120] (1) Image acquisition: Simultaneously acquire video from two different perspectives at the edge of the same environment for multiple zebrafish, and obtain video streams from the two perspectives;

[0121] (2) Target recognition and tracking: A target recognition model is constructed based on the YOLOv5 algorithm. For each frame of video image in each video stream, the following is performed at the edge: The zebrafish in the video image is identified by the trained target recognition model to obtain the zebrafish's position information. Based on the zebrafish's position information, an ID is assigned to each zebrafish in the video image using the OC-SORT tracking algorithm. The zebrafish is tracked based on its motion and appearance features. The position information and ID of the zebrafish are associated to obtain the ID and position information of each zebrafish in the video image. The current frame number of the video image and the ID and position information of each zebrafish in the video image are uploaded to the cloud in real time.

[0122] Correspondingly, the cloud is used to perform 3D trajectory construction: based on the position information and ID of the zebrafish in each frame of video images from different perspectives, a directed graph is constructed to achieve trajectory matching. Based on the similarity of the zebrafish's appearance features and motion features from different perspectives, the weights of the edges in the directed graph are calculated. The 3D matching of the zebrafish's motion trajectory from two different perspectives is achieved through multi-view information data fusion.

[0123] In this embodiment, the edge device includes a processor and two high-definition cameras. The processor is equipped with an RK3588S chip, and the high-definition cameras are RGB cameras.

[0124] The cloud server is deployed in the cloud. In this embodiment, Alibaba Cloud ECS is selected. EMQX is used to build the server into an MQTT server. A database is installed in the cloud server as a cloud database and a Python environment is configured. The RK3588S chip interacts with the cloud server based on the MQTT protocol.

[0125] As a concrete implementation of edge computing and cloud computing, the camera uses the CAM-8MS1M sensor, which supports up to 800W pixels and a maximum resolution of 3840*2160. It is connected to the edge development board via a MIPI cable. The edge development board is equipped with the RK3588S chip, a new generation flagship high-end processor launched by Rockchip. It adopts an 8nm process design, is equipped with an eight-core CPU with four A76 cores and four A55 cores, and an Arm high-performance GPU. It also has a built-in NPU with 6T computing power, which has powerful visual processing capabilities and can be expanded for AI computing power, image data processing, etc. It can connect to the Internet via Ethernet cable and communicate with cloud servers and cloud databases via MQTT protocol.

[0126] Before image acquisition, a zebrafish environment was constructed. Specifically, a standard glass aquarium was selected as the experimental tank, measuring 30 × 30 × 30 cm with a water depth of 15 cm. To reduce reflection, three-quarters of the window glass was covered with translucent plastic. The plastic material measured 30 × 30 cm, had a luminous flux of 670 lumens, and a color temperature of 4000 K.

[0127] During image acquisition, the RGB camera captures video streams from both the frontal and top-down views of the fish tank.

[0128] The trained target model is used to identify zebrafish in each frame of the video stream to obtain the zebrafish's location information within the video image. This trained target recognition model is obtained and deployed at the edge as follows: For the constructed target recognition model, multiple sample images are acquired, each containing at least one zebrafish. The positions of the zebrafish in the sample images are labeled. The sample images and labels are used as sample data. Based on this sample data, the target recognition model is trained and tested to obtain a weight file corresponding to the target recognition model. After pruning the weight file, it is written to the edge, thus loading the trained target recognition model onto the edge.

[0129] In this embodiment, eight 2K offline video streams from different perspectives, namely frontal and top views of the fish tank, were acquired by an RGB camera. Each frame of the video stream was used as a sample image, and the position information of the zebrafish in the sample image was labeled. A sample set was constructed based on the sample images and the labels, and the sample set was divided into a training set and a test set to train and test the constructed target recognition model.

[0130] To ensure lightweight and real-time performance of target recognition and tracking after deployment on the RK3588S, a lightweight one-stage detection model, YOLOv5-Small, was selected for target recognition model construction through comparison. The weight file was obtained from the target recognition model constructed using the aforementioned training and testing sets. Since the RK3588S is an embedded platform, this embodiment optimized and pruned the weight file of the target recognition model, reducing its size by approximately 75% while decreasing the MAP by only 0.5%. This embodiment also rewrote the target recognition model using C++.

[0131] At the edge, a trained target recognition model performs target recognition. The bounding box detected in each frame of video image represents the zebrafish's position information in that frame. Initially, the OC-SORT algorithm assigns an ID to each zebrafish in the tank based on its position information. In the next frame, the algorithm detects zebrafish in real time and associates trajectories with the same ID based on the zebrafish's position information tracked by motion and appearance. If the detection does not match any existing trajectory (indicating a new object or an object that has reappeared after being occluded), a new trajectory is created for that object and a unique ID is assigned. In this embodiment, the OC-SORT algorithm was rewritten in C++ for deployment to adapt to RK3588S.

[0132] Considering that the RK3588S chip at the edge is an embedded platform, this embodiment uses the C / C++ version of the paho-mqtt-embended library to undertake the task of sending data to the server. The specific data includes the frame number of each video image detected by the RK3588S during runtime, the zebrafish's location information, and the ID of each zebrafish.

[0133] The cloud database stores the position information and ID of each frame of a zebrafish video image from different viewpoints. Based on the zebrafish's position information from each viewpoint, its two-dimensional motion trajectory can be obtained. In this embodiment, the cloud performs the following three operations to construct a three-dimensional trajectory: generating nodes, connecting nodes, and generating the three-dimensional trajectory.

[0134] Node generation: Taking individual zebrafish as the target, based on the two-dimensional motion trajectory of the zebrafish from two perspectives, the nodes are created and their weights are calculated by calculating the target that appears in both perspectives simultaneously. The nodes are then added to the directed acyclic graph. For the time period between the zebrafish's most recent detection time and its first detection time, a set of node pairs is constructed based on the two-dimensional trajectory of the zebrafish from both perspectives.

[0135] The specific implementation of node generation includes the following operations:

[0136] For two perspectives, a first-person view and a second-person view, using the zebrafish trajectory in the first-person view as a baseline, multiple candidate points can be found for individual zebrafish targets detected in the first-person view in the second-person view. For the same ID appearing simultaneously in both the first-person and second-person views, through... Calculate the frames of the video image where the ID is detected from both viewpoints, where, This represents the set of video images whose ID was detected in the first-person perspective. This refers to the set of frames of video images in which the ID was detected from a second-view perspective;

[0137] The formula for calculating the weight of the matched node in each frame of the video image is:

[0138]

[0139] Among them, through The calculated intersecting frames are as follows: , This refers to the zebrafish from a second-person perspective. Projection error in water, This represents the reciprocal of the average reprojection error of the training dataset. This indicates that zebrafish were detected. In the experimental environment, what is the total weight of the matched nodes within a predetermined time period? Weights of each frame Composition, total weight The calculation formula is as follows:

[0140]

[0141] when When a node is added to a directed acyclic graph;

[0142] When the first-view 2D trajectory and the second-view 2D trajectory are combined and added as nodes to a directed acyclic graph, a set of node pairs is constructed. During the time period Inside ,in Used for time evaluation Is it in Before, Used to ensure They will not overlap in time. ,node Comprising two-dimensional trajectories from a first-person perspective Two-dimensional trajectory from a second perspective 3D trajectory And the weights of the node pairs are represented by quadruples.

[0143] Connecting nodes: For each node pair in the set of node pairs, the cloud server calculates the weight of the directed edge between each node pair.

[0144] Node Pair Set Each node pair in the time period Inside, from the node To the node weight of directed edges The calculation formula is as follows:

[0145]

[0146] The weights of the edges are calculated as the positive complement probabilities of the exponential cumulative distribution function (CDF). , For zebrafish from Nodes within the time period The two-dimensional trajectory motion characteristics, for The time difference between them Represents a node The weight, Represents a node The weight, The value was chosen based on experience. The values ​​given for the training data;

[0147] The training dataset is a dataset constructed from sample data used for model sequence and testing of the target recognition model. The sample data includes sample images and annotations. Each sample image contains at least one zebrafish, and the annotations are the labels for the zebrafish in the sample images.

[0148] Generating 3D trajectories: For a directed acyclic graph (DAG), the longest path in the DAG is calculated recursively and the nodes are output as 3D trajectories. When calculating the longest path, the weights of all nodes and edges in the path are summed and the maximum value is selected as the longest path. After each extraction of the longest path, the nodes related to the longest path and other nodes with the same 2D trajectory as the node are deleted from the DAG until the DAG is empty.

[0149] In this embodiment, the three steps of generating nodes, connecting nodes, and generating three-dimensional trajectories are written into the cloud server as a multi-view information fusion algorithm, and the cloud server executes the algorithm to construct the three-dimensional trajectory of the zebrafish.

[0150] The present invention has been shown and described in detail above with reference to the accompanying drawings and preferred embodiments. However, the present invention is not limited to these disclosed embodiments. Based on the above embodiments, those skilled in the art will know that more embodiments of the present invention can be obtained by combining the code review methods in the different embodiments. These embodiments are also within the protection scope of the present invention.

Claims

1. A cloud-edge collaborative method for zebrafish 3D trajectory tracking, characterized in that, Includes the following steps: Image acquisition: Simultaneously capture video of multiple zebrafish in the same environment from two different perspectives at the edge, and obtain video streams from the two perspectives; Target recognition and tracking: A target recognition model is built based on the YOLOv5 algorithm. For each frame of video image in each video stream, the following is performed at the edge: The zebrafish in the video image is identified by the trained target recognition model to obtain the zebrafish's position information. Based on the zebrafish's position information, an ID is assigned to each zebrafish in the video image using the OC-SORT tracking algorithm. The zebrafish is tracked based on its motion and appearance features. The position information and ID of the zebrafish are associated to obtain the ID and position information of each zebrafish in the video image. The current frame number of the video image and the ID and position information of each zebrafish in the video image are uploaded to the cloud in real time. 3D trajectory construction: Based on the position information and ID of zebrafish in each frame of video image from different perspectives, a directed graph is constructed to achieve trajectory matching. The weights of the edges in the directed graph are calculated based on the similarity of the zebrafish's appearance features and motion features from different perspectives. The 3D matching of the zebrafish's motion trajectory from two different perspectives is achieved through multi-view information data fusion. For the constructed target recognition model, multiple sample images are collected, each containing at least one zebrafish. The positions of the zebrafish in the sample images are labeled. The sample images and labels are used as sample data. The target recognition model is trained and tested based on the sample data to obtain the weight file corresponding to the target recognition model. After pruning the weight file, the weight file is written to the edge to load the trained target recognition model to the edge. The edge device consists of a processor and two high-definition cameras. The processor is equipped with an RK3588S chip, and the high-definition cameras are electrically connected to the RK3588S chip via MIPI cables. The server-side deployment includes a cloud server and a cloud database. The cloud server is an MQTT server, and the RK3588S chip interacts with the cloud server based on the MQTT protocol. The weighted file after pruning and the OC-SORT tracking algorithm are written to the RK3588S chip. During image acquisition, the two cameras simultaneously capture video of multiple zebrafish in the same environment from their respective perspectives, obtaining video streams from both perspectives. During target recognition and tracking, for each frame of video image, the frame number of the video image and the ID and location information of each zebrafish in the video image are uploaded to the cloud in real time and stored in the cloud database. During 3D trajectory construction, the cloud server reads the frame number of each video image and the ID and location information of each zebrafish in the video image from the cloud database. The 3D trajectory construction includes the following steps: Node generation: Taking individual zebrafish as the target, based on the two-dimensional motion trajectory of the zebrafish from two perspectives, the nodes are created and their weights are calculated by calculating the target that appears in both perspectives simultaneously. The nodes are then added to the directed acyclic graph. For the time period between the zebrafish's most recent detection time and its first detection time, a set of node pairs is constructed based on the two-dimensional trajectory of the zebrafish from both perspectives. Connecting nodes: For each node pair in the set of node pairs, calculate the weight of the directed edge between each node pair; Generating 3D trajectories: For a directed acyclic graph (DAG), the longest path in the DAG is calculated recursively and the nodes are output as 3D trajectories. When calculating the longest path, the weights of all nodes and edges in the path are summed and the maximum value is selected as the longest path. After each extraction of the longest path, the nodes related to the longest path and other nodes with the same 2D trajectory as the node are deleted from the DAG until the DAG is empty.

2. The cloud-edge collaborative zebrafish three-dimensional trajectory tracking method according to claim 1, characterized in that, Generating nodes involves the following steps: For two perspectives, a first-person view and a second-person view, using the zebrafish trajectory in the first-person view as a baseline, multiple candidate points can be found for individual zebrafish targets detected in the first-person view in the second-person view. For the same ID appearing simultaneously in both the first-person and second-person views, through... Calculate the frames of the video image where the ID is detected from both viewpoints, where, This represents the set of video images whose ID was detected in the first-person perspective. This refers to the set of frames of video images in which the ID was detected from a second-view perspective; The formula for calculating the weight of the matched node in each frame of the video image is: ， Among them, through The calculated intersecting frames are as follows: , This refers to the zebrafish from a second-person perspective. Projection error in water, This represents the reciprocal of the average reprojection error of the training dataset. This indicates that zebrafish were detected. In the experimental environment, what is the total weight of the matched nodes within a predetermined time period? Weights of each frame Composition, total weight The calculation formula is as follows: ， when When a node is added to a directed acyclic graph; When the first-view 2D trajectory and the second-view 2D trajectory are combined and added as nodes to a directed acyclic graph, a set of node pairs is constructed. During the time period Inside ,in Used for time evaluation Is it in Before, Used to ensure They will not overlap in time. ,node Comprising two-dimensional trajectories from a first-person perspective Two-dimensional trajectory from a second perspective 3D trajectory And the quadruple representation of the weights of the node pairs; Correspondingly, the set of node pairs Each node pair in the time period Inside, from the node To the node weight of directed edges The calculation formula is as follows: ， The weights of the edges are calculated as the positive complement probabilities of the exponential cumulative distribution function (CDF). , For zebrafish in from Nodes within the time period The two-dimensional trajectory motion characteristics, for and The time difference between them Represents a node The weight, Represents a node The weight, The value was chosen based on experience. The values ​​given for the training data; The training dataset is a dataset constructed from sample data used for model sequence and testing of the target recognition model. The sample data includes sample images and annotations. Each sample image contains at least one zebrafish, and the annotations are the labels for the zebrafish in the sample images.

3. A cloud-edge collaborative zebrafish 3D trajectory tracking system, characterized in that, It includes cloud and multiple edge terminals, with the edge terminals and cloud working together to execute a cloud-edge collaborative zebrafish three-dimensional trajectory tracking method as described in any one of claims 1-2; The edge is used to perform the following: Image acquisition: Simultaneously capture video of multiple zebrafish in the same environment from two different perspectives at the edge, and obtain video streams from the two perspectives; Target recognition and tracking: A target recognition model is built based on the YOLOv5 algorithm. For each frame of video image in each video stream, the following is performed at the edge: The zebrafish in the video image is identified by the trained target recognition model to obtain the zebrafish's position information. Based on the zebrafish's position information, an ID is assigned to each zebrafish in the video image using the OC-SORT tracking algorithm. The zebrafish is tracked based on its motion and appearance features. The position information and ID of the zebrafish are associated to obtain the ID and position information of each zebrafish in the video image. The current frame number of the video image and the ID and position information of each zebrafish in the video image are uploaded to the cloud in real time. Correspondingly, the cloud is used to execute the following: 3D trajectory construction: Based on the position information and ID of zebrafish in each frame of video image from different perspectives, a directed graph is constructed to achieve trajectory matching. The weights of the edges in the directed graph are calculated based on the similarity of the zebrafish's appearance features and motion features from different perspectives. The 3D matching of the zebrafish's motion trajectory from two different perspectives is achieved through multi-view information data fusion. Before deploying the trained target recognition model on the development board at the edge, the following operations are performed on the constructed target recognition model: multiple sample images are collected, each containing at least one zebrafish, and the positions of the zebrafish in the sample images are labeled. The sample images and labels are used as sample data. The target recognition model is trained and tested based on the sample data to obtain the weight file corresponding to the target recognition model. After pruning the weight file, the weight file is written to the edge to load the trained target recognition model onto the edge. The edge device consists of a processor and two high-definition cameras. The processor is equipped with an RK3588S chip, and the high-definition cameras are electrically connected to the RK3588S chip via MIPI cables. The server-side deployment includes a cloud server and a cloud database. The cloud server is an MQTT server, and the RK3588S chip interacts with the cloud server based on the MQTT protocol. The weight file after pruning and the OC-SORT tracking algorithm are written to the RK3588S chip. During image acquisition, the edge device uses the two cameras to simultaneously capture video of multiple zebrafish in the same environment from their respective perspectives, obtaining video streams from two perspectives. During target recognition and tracking, for each frame of video image, the edge device uses the RK3588S chip to upload the frame number of the video image and the ID and location information of each zebrafish in the video image to the cloud in real time and store it in the cloud database. During 3D trajectory construction, the cloud server reads the frame number of each video image and the ID and location information of each zebrafish in the video image from the cloud database. The cloud is used to perform the following 3D trajectory construction: Node generation: Taking individual zebrafish as the target, based on the two-dimensional motion trajectory of the zebrafish from two perspectives, the nodes are created and their weights are calculated by calculating the target that appears in both perspectives simultaneously. The nodes are then added to the directed acyclic graph. For the time period between the zebrafish's most recent detection time and its first detection time, a set of node pairs is constructed based on the two-dimensional trajectory of the zebrafish from both perspectives. Connecting nodes: For each node pair in the set of node pairs, calculate the weight of the directed edge between each node pair; Generating 3D trajectories: For a directed acyclic graph (DAG), the longest path in the DAG is calculated recursively and the nodes are output as 3D trajectories. When calculating the longest path, the weights of all nodes and edges in the path are summed and the maximum value is selected as the longest path. After each extraction of the longest path, the nodes related to the longest path and other nodes with the same 2D trajectory as the node are deleted from the DAG until the DAG is empty.

4. The cloud-edge collaborative zebrafish three-dimensional trajectory tracking system according to claim 3, characterized in that, The cloud is used to execute the following generation nodes: For two perspectives, a first-person view and a second-person view, using the zebrafish trajectory in the first-person view as a baseline, multiple candidate points can be found for individual zebrafish targets detected in the first-person view in the second-person view. For the same ID appearing simultaneously in both the first-person and second-person views, through... Calculate the frames of the video image where the ID is detected from both viewpoints, where, This represents the set of video images whose ID was detected in the first-person perspective. This refers to the set of frames of video images in which the ID was detected from a second-view perspective; The formula for calculating the weight of the matched node in each frame of the video image is: ， Among them, through The calculated intersecting frames are as follows: , This refers to the zebrafish from a second-person perspective. Projection error in water, This represents the reciprocal of the average reprojection error of the training dataset. This indicates that zebrafish were detected. In the experimental environment, what is the total weight of the matched nodes within a predetermined time period? Weights of each frame Composition, total weight The calculation formula is as follows: ， when When a node is added to a directed acyclic graph; When the first-view 2D trajectory and the second-view 2D trajectory are combined and added as nodes to a directed acyclic graph, a set of node pairs is constructed. During the time period Inside ,in Used for time assessment Is it in Before, Used to ensure They will not overlap in time. ,node Comprising two-dimensional trajectories from a first-person perspective Two-dimensional trajectory from a second perspective 3D trajectory And the quadruple representation of the weights of the node pairs; Correspondingly, the set of node pairs Each node pair in the time period Inside, from the node To the node weight of directed edges The calculation formula is as follows: ， The weights of the edges are calculated as the positive complement probabilities of the exponential cumulative distribution function (CDF). , For zebrafish in from Nodes within the time period The two-dimensional trajectory motion characteristics, for The time difference between them Represents a node The weight, Represents a node The weight, The value was chosen based on experience. The values ​​given for the training data; The training dataset is a dataset constructed from sample data used for model sequence and testing of the target recognition model. The sample data includes sample images and annotations. Each sample image contains at least one zebrafish, and the annotations are the labels for the zebrafish in the sample images.

Images