Method, system and device for observing movement based on target biological crawling trajectory

Abstract

This invention discloses a method, system, and device for motion observation based on the crawling trajectory of target organisms. The method includes acquiring training videos of target organism crawling; assigning a tracker and lifecycle variables to each target for multi-target tracking across consecutive frames; performing cascaded matching between the filtered target detection boxes and the target tracking boxes of the current frame; calculating the pixel crawling speed for the matched tracking sequence; obtaining a scale based on the target pixel crawling distance and the target's physical distance in the real world, and then calculating the actual crawling distance and actual crawling speed of the target organism. This invention employs a multi-layer cascaded matching method to match detection boxes and target tracking boxes, enabling real-time observation of the crawling of various types of target organisms, including slow-moving target organisms, fast-moving target organisms, and moths that exhibit jumping crawling, and accurately recording motion information such as crawling trajectory, crawling speed, crawling time, crawling distance, and dwell time.

Description

Technical Field

[0001] This invention belongs to the field of big data processing technology, specifically relating to a method, system, and device for motion observation based on the crawling trajectory of a target organism. Background Technology

[0002] Currently, numerous scientific studies are being conducted on target organisms, such as observing the feeding preferences, phototaxis, and mating behavior of reptiles. Analyzing the behavior of target organisms can effectively record their movement patterns and habits, providing effective guidance for pests and beneficial insects, and having a crucial impact on the management and control of crop diseases and pests. Traditional methods for observing the movement of target organisms generally rely on visual observation. However, visual observation is time-consuming, and commercially available observation equipment has poor results, cannot achieve multi-target observation, and the target organism's trajectory lines are prone to breakage.

[0003] With the increasing application of artificial intelligence technology in agriculture, it has become possible to detect designated target organisms using deep learning techniques, enabling effective localization of these organisms and improving the generalization detection capability for various target organisms by expanding the dataset. However, in the field of target organism observation, there is still no effective method for real-time tracking and observation of target organisms, recording real-time trajectory, crawling speed, crawling distance, and other motion parameters. Furthermore, there is no unified and effective solution for the problems that arise during the crawling observation of various types of target organisms. Summary of the Invention

[0004] To address the shortcomings of existing technologies, this invention discloses a method, system, and device for motion observation based on the crawling trajectory of a target organism.

[0005] The motion observation method based on the crawling trajectory of the target organism includes the following steps:

[0006] Acquire a video of the target organism crawling, perform target detection on the video frames to obtain the target region, and filter out false targets in the target region to obtain a target detection box, wherein the target detection box includes at least one detection box;

[0007] Assign target markers to the target detection boxes and perform multi-target tracking on the target detection boxes to generate target tracking boxes;

[0008] The target detection box and the target tracking box are concatenated and matched to obtain the matched tracking sequence.

[0009] The displacement of the center point of the target tracking box is obtained through the matched tracking sequence, and then the pixel crawling distance is obtained.

[0010] The pixel crawling time is obtained by capturing the time difference between the center points of the tracking box. Based on the pixel crawling distance and the pixel crawling time, the target pixel crawling speed is obtained.

[0011] A scale is obtained by calculating the pixel crawling distance and the actual physical distance. The pixel crawling distance and pixel crawling speed are then calculated based on the scale to obtain the actual crawling distance and actual crawling speed of the target organism.

[0012] As one possible implementation, the step of cascading matching the target detection box and the target tracking box to obtain a matched tracking sequence, wherein the cascading matching includes at least three stages of cascading matching, includes the following steps:

[0013] The first stage of cascaded matching involves performing IOU matching between the target detection box and the target tracking box, calculating the IOU value between each target detection box and each target tracking box, and performing the first Hungarian matching on target detection boxes and target tracking boxes whose IOU value is greater than a first preset threshold.

[0014] Second-stage cascaded matching: For target detection boxes and target tracking boxes whose IOU values ​​are less than the first preset threshold in the first-stage cascaded matching, calculate the HOG feature cosine similarity between the target detection boxes and the target tracking boxes, and perform a second Hungarian matching by combining the Euclidean distance similarity.

[0015] The third stage is cascaded matching: matching the target detection box and the target tracking box based on ReID.

[0016] As one possible implementation, the first-stage cascaded matching includes the following steps:

[0017] Calculate the IOU value between each target detection box and each target tracking box, and perform IOU matching between the target detection boxes and the target tracking boxes;

[0018] For target detection boxes and target tracking boxes with an IOU value greater than the first preset threshold, the best match between target detection boxes and target tracking boxes in adjacent frames is obtained by using the Hungarian matching algorithm.

[0019] The formula for calculating the IOU value is:

[0020]

[0021] Where A represents the set of object detection boxes and B represents the set of object tracking boxes.

[0022] As one possible implementation, the second-stage cascaded matching includes the following steps:

[0023] In the first stage, the target detection boxes and target tracking boxes with an IOU value less than the first preset threshold are matched together. The HOG feature cosine similarity between the target detection boxes and the target tracking boxes is calculated.

[0024] Calculate the Euclidean distance between the center point of the target detection box and the center point of the target tracking box;

[0025] The second Hungarian matching result is obtained by weighted summation of Euclidean distance similarity and HOG feature cosine similarity.

[0026] The Euclidean distance between the center point of the target detection box and the center point of the target tracking box is calculated using the following formula:

[0027]

[0028] in, This represents the Euclidean distance between the center point of the target detection box and the center point of the target tracking box. This represents the coordinates of the center point of the target detection box. Indicates the coordinates of the center point of the target tracking box;

[0029] The formula for calculating the second Hungarian match result is as follows:

[0030]

[0031] in, Represents the similarity based on Euclidean distance. Represents the cosine similarity of HOG features. This represents the weighted sum of Euclidean distance similarity and HOG feature cosine similarity. This represents the normalized weighting factor for similarity based on Euclidean distance. This represents the normalized weighting factor based on the cosine similarity of HOG features.

[0032] As one possible implementation, the third-stage cascaded matching includes the following steps:

[0033] Acquire newly detected target organism video frames, retain several frames as a feature query library for the ROI region, and acquire video frames of the target organism corresponding to the target marker. The number of newly detected target organism video frames is the same as the number of target organism video frames corresponding to the target marker.

[0034] The pre-trained feature extraction network is trained using a metric learning loss function to obtain the trained feature extraction network.

[0035] The trained feature extraction network performs ROI feature extraction on newly detected target biological video frames and target biological video frames corresponding to target markers, and obtains the ROI feature extraction results of newly detected target biological video frames and target biological video frames corresponding to target markers.

[0036] The cosine similarity is calculated for the newly detected target biological video frame ROI feature extraction results and the target biological video frame ROI feature extraction results corresponding to the target marker, and the cosine similarity is added to obtain the total cosine similarity.

[0037] When the total cosine similarity is greater than the second preset threshold, the newly detected target creature is determined to be a target creature that has appeared before, and the target tag is assigned to the newly detected target creature; otherwise, a new target tag is assigned to the newly detected target creature, thereby completing the ReID matching of the target creature and obtaining the matched tracking sequence.

[0038] The formula for calculating cosine similarity is:

[0039]

[0040] in, This indicates the ROI feature extraction results of the newly detected target biological video frame. This indicates the ROI feature extraction results for the target biological video frame corresponding to the target marker;

[0041] The formula for calculating the total cosine similarity is:

[0042]

[0043] in, Represents the total cosine similarity. This represents the cosine similarity.

[0044] As one possible implementation, the step of acquiring a target organism crawling training video and performing target detection on consecutive video frames to obtain the target region includes the following steps:

[0045] Acquire videos of the target organism crawling, extract video frames to define the target bounding boxes, and obtain target bounding boxes to form a calibration dataset;

[0046] The calibration dataset is divided into a training set, a validation set, and a test set. The pre-trained object detection model is then trained and validated to obtain the object detection model.

[0047] The target region is obtained by detecting targets in consecutive video frames using a target detection model.

[0048] As one possible implementation, assigning target markers to the target detection boxes and generating target tracking boxes through multi-target tracking includes the following steps:

[0049] Assign target markers to the filtered target detection boxes and assign a tracker to each target;

[0050] The tracker tracks the target detection box, matches the detected target with the detected target appearing in the next frame, performs multi-target tracking on the target detection box, and incorporates life cycle rules into the tracking process to obtain the target tracking box.

[0051] A motion observation system based on the crawling trajectory of a target organism includes a data acquisition module, a target detection box allocation module, a cascaded matching module, a pixel crawling distance acquisition module, a target pixel crawling data acquisition module, and an actual crawling data acquisition module.

[0052] The data acquisition module is used to acquire video of the target organism crawling, perform target detection on the video frames to obtain the target region, and filter out false targets in the target region to obtain the target detection box, wherein the target detection box includes at least one detection box;

[0053] The target detection box allocation module is used to assign target markers to target detection boxes and generate target tracking boxes by performing multi-target tracking on the target detection boxes;

[0054] The cascaded matching module is used to cascade the target detection box and the target tracking box to obtain the matched tracking sequence;

[0055] The pixel crawling distance acquisition module is used to obtain the displacement of the center point of the target tracking box through the matched tracking sequence, and then obtain the pixel crawling distance;

[0056] The target pixel crawling data acquisition module is used to obtain the pixel crawling time by capturing the time difference between the center points of the tracking box, and to obtain the target pixel crawling speed based on the pixel crawling distance and pixel crawling time.

[0057] The actual crawling data acquisition module is used to calculate the scale based on the pixel crawling distance and the actual physical distance. The pixel crawling distance and pixel crawling speed are calculated based on the scale to obtain the actual crawling distance and actual crawling speed of the target organism.

[0058] As one possible implementation, the step of cascading matching the target detection box and the target tracking box to obtain a matched tracking sequence, wherein the cascading matching includes at least three stages of cascading matching, includes the following steps:

[0059] The first stage of cascaded matching involves performing IOU matching between the target detection box and the target tracking box, calculating the IOU value between each target detection box and each target tracking box, and performing the first Hungarian matching on target detection boxes and target tracking boxes whose IOU value is greater than a first preset threshold.

[0060] Second-stage cascaded matching: For target detection boxes and target tracking boxes whose IOU values ​​are less than the first preset threshold in the first-stage cascaded matching, calculate the HOG feature cosine similarity between the target detection boxes and the target tracking boxes, and perform a second Hungarian matching by combining the Euclidean distance similarity.

[0061] The third stage is cascaded matching: matching the target detection box and the target tracking box based on ReID.

[0062] As one possible implementation, assigning target markers to the target detection boxes and generating target tracking boxes through multi-target tracking includes the following steps:

[0063] Assign target markers to the filtered target detection boxes and assign a tracker to each target;

[0064] The tracker tracks the target detection box, matches the detected target with the detected target appearing in the next frame, performs multi-target tracking on the target detection box, and incorporates life cycle rules into the tracking process to obtain the target tracking box.

[0065] A computer-readable storage medium storing a computer program that, when executed by a processor, implements the following method:

[0066] Acquire a video of the target organism crawling, perform target detection on the video frames to obtain the target region, and filter out false targets in the target region to obtain a target detection box, wherein the target detection box includes at least one detection box;

[0067] Assign target markers to the target detection boxes and perform multi-target tracking on the target detection boxes to generate target tracking boxes;

[0068] The target detection box and the target tracking box are concatenated and matched to obtain the matched tracking sequence.

[0069] The displacement of the center point of the target tracking box is obtained through the matched tracking sequence, and then the pixel crawling distance is obtained.

[0070] The pixel crawling time is obtained by capturing the time difference between the center points of the tracking box. Based on the pixel crawling distance and the pixel crawling time, the target pixel crawling speed is obtained.

[0071] A scale is obtained by calculating the pixel crawling distance and the actual physical distance. The pixel crawling distance and pixel crawling speed are then calculated based on the scale to obtain the actual crawling distance and actual crawling speed of the target organism.

[0072] A motion observation device based on the crawling trajectory of a target organism includes a memory, a processor, and a computer program stored in the memory and running on the processor. When the processor executes the computer program, it implements the following method:

[0073] Acquire a video of the target organism crawling, perform target detection on the video frames to obtain the target region, and filter out false targets in the target region to obtain a target detection box, wherein the target detection box includes at least one detection box;

[0074] Assign target markers to the target detection boxes and perform multi-target tracking on the target detection boxes to generate target tracking boxes;

[0075] The target detection box and the target tracking box are concatenated and matched to obtain the matched tracking sequence.

[0076] The displacement of the center point of the target tracking box is obtained through the matched tracking sequence, and then the pixel crawling distance is obtained.

[0077] The pixel crawling time is obtained by capturing the time difference between the center points of the tracking box. Based on the pixel crawling distance and the pixel crawling time, the target pixel crawling speed is obtained.

[0078] A scale is obtained by calculating the pixel crawling distance and the actual physical distance. The pixel crawling distance and pixel crawling speed are then calculated based on the scale to obtain the actual crawling distance and actual crawling speed of the target organism.

[0079] This invention, by adopting the above technical solutions, has significant technical effects:

[0080] 1. This invention employs a multi-layer cascaded matching method to match detection boxes and target tracking boxes: IOU matching, HOG feature cosine similarity, and Euclidean distance between the target detection boxes and target tracking boxes enable real-time observation of crawling behavior of various types of target organisms, including slow-moving target organisms, fast-moving target organisms, moths with jumping and crawling behaviors, and target organisms with displacement mutations; the ReID algorithm is introduced into the observation of target organism crawling behavior, so that when the target organism leaves the video frame and reappears, effective identity continuation can still be performed, thereby more effectively recording the movement of the target organism;

[0081] 2. This invention also provides a method for measuring the crawling scale of the target organism and an effective scale conversion method, which effectively converts the pixel speed and pixel distance of the target organism crawling in the video into the actual physical speed and physical distance of crawling, thereby enabling more effective acquisition of the actual movement of the target organism;

[0082] 3. The motion observation method for the crawling trajectory of the target organism described in this invention is a target detection method based on deep learning, which has high target organism positioning accuracy and can accurately record motion information such as crawling trajectory, crawling speed, crawling time, crawling distance, and dwell time. Attached Figure Description

[0083] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0084] Figure 1 This is a schematic diagram of the motion observation method based on the crawling trajectory of the target organism as described in this invention;

[0085] Figure 2 This is an overall schematic diagram of the motion observation system based on the crawling trajectory of the target organism as described in this invention. Detailed Implementation

[0086] The present invention will be further described below with reference to the accompanying drawings and embodiments. The following embodiments are explanations of the present invention, but the present invention is not limited to the following embodiments.

[0087] Example 1:

[0088] A motion observation method based on the crawling trajectory of a target organism, such as Figure 1 As shown, it includes the following steps:

[0089] S100. Acquire a video of the target organism crawling, perform target detection on the video frames to obtain the target region, and filter out false targets in the target region to obtain a target detection box, wherein the target detection box includes at least one detection box.

[0090] S200: Assign target markers to the target detection boxes and perform multi-target tracking on the target detection boxes to generate target tracking boxes;

[0091] S300. Concatenate the target detection box and the target tracking box to obtain the matched tracking sequence.

[0092] S400. The displacement of the center point of the target tracking box is obtained through the matched tracking sequence, and then the pixel crawling distance is obtained.

[0093] S500: By capturing the time difference between the center points of the tracking box, the pixel crawling time is obtained. Based on the pixel crawling distance and the pixel crawling time, the target pixel crawling speed is obtained.

[0094] S600. A scale is obtained by calculating the pixel crawling distance and the actual physical distance. The pixel crawling distance and pixel crawling speed are calculated based on the scale to obtain the actual crawling distance and actual crawling speed of the target organism.

[0095] This invention employs a multi-layer cascaded matching method to match detection boxes and target tracking boxes. IOU matching, HOG feature cosine similarity, and Euclidean distance between the target detection boxes and target tracking boxes enable real-time observation of the crawling movements of various target organisms, including slow-moving, fast-moving, moth-like, and target organisms with sudden displacement changes. The ReID algorithm is introduced into the target organism crawling observation, allowing for effective identity continuation even when the target organism reappears after leaving the video frame, thus more effectively recording the target organism's movement. This invention also provides a method for measuring the target organism's crawling scale, offering an effective scale conversion method that effectively converts the pixel speed and pixel distance of the target organism's crawling movement in the video frame into the actual physical speed and distance of the crawling movement, thereby more effectively acquiring the target organism's actual movement. The motion observation method for the target organism's crawling trajectory described in this invention is based on a deep learning-based target detection method, possessing high target organism positioning accuracy and accurately recording motion information such as crawling trajectory, crawling speed, crawling time, crawling distance, and dwell time.

[0096] In this embodiment, step S100 involves acquiring a video of the target organism crawling, performing target detection on the video frames to obtain the target region, and filtering out falsely detected targets within the target region to obtain a target detection box. This specifically includes the following steps:

[0097] S110: Obtain the target organism crawling video as the training set, capture images every N frames and save them locally, use the labelImg calibration tool to define the target organism bounding boxes in the captured images, and obtain the target organism bounding box dataset of the video frames.

[0098] S120: Divide the target biological bounding box dataset into a training dataset, a validation dataset, and a test dataset, wherein the ratio of the number of the training dataset, the validation dataset, and the test dataset is 8:1:1;

[0099] S130: Train the object pre-detection model yolov8 using the training dataset to obtain the object detection model, and then perform object detection;

[0100] S140: Perform image processing on the dataset after object detection to remove false detection boxes with solid color backgrounds and obtain the filtered object detection boxes.

[0101] Step S200 assigns target markers to the target detection boxes and performs multi-target tracking on the target detection boxes to generate target tracking boxes, specifically including the following steps:

[0102] S210: Assign target markers to the target detection box and assign trackers and lifecycle variables to each target marker.

[0103] S220: Perform multi-target tracking on the target detection box for consecutive frames to generate a target tracking box, and construct a total tracking information module. The total tracking information module includes a target biological tracking information module and a target biological tracking information maintenance module. During this process, life cycle variable parameters are added.

[0104] Step S300 involves cascading matching of the target detection box and the target tracking box to obtain a matched tracking sequence. In this embodiment, the cascading matching includes a three-stage cascading matching process, specifically comprising the following steps:

[0105] S310: First-stage cascaded matching: Perform IOU matching between the target detection box and the target tracking box, calculate the IOU value between each target detection box and each target tracking box, and perform the first Hungarian matching on the target detection box and the target tracking box whose IOU value is greater than the first preset threshold.

[0106] The first stage of cascaded matching includes the following steps:

[0107] Calculate the IOU value between each target detection box and each target tracking box, and perform IOU matching between the target detection boxes and the target tracking boxes;

[0108] For target detection boxes and target tracking boxes with an IOU value greater than the first preset threshold, the best match between target detection boxes and target tracking boxes in adjacent frames is obtained by using the Hungarian matching algorithm.

[0109] The formula for calculating the IOU value is:

[0110]

[0111] Where A represents the set of object detection boxes and B represents the set of object tracking boxes.

[0112] S320: Second-stage cascaded matching: For target detection boxes and target tracking boxes whose IOU values ​​are less than the first preset threshold in the first-stage cascaded matching, calculate the HOG feature cosine similarity between the target detection boxes and the target tracking boxes, and perform a second Hungarian matching by combining the Euclidean distance similarity.

[0113] The second stage of cascaded matching includes the following steps:

[0114] In the first stage, the target detection boxes and target tracking boxes with an IOU value less than the first preset threshold are matched together. The HOG feature cosine similarity between the target detection boxes and the target tracking boxes is calculated.

[0115] Calculate the Euclidean distance between the center point of the target detection box and the center point of the target tracking box;

[0116] The second Hungarian matching result is obtained by weighted summation of Euclidean distance similarity and HOG feature cosine similarity.

[0117] The Euclidean distance between the center point of the target detection box and the center point of the target tracking box is calculated using the following formula:

[0118]

[0119] in, This represents the Euclidean distance between the center point of the target detection box and the center point of the target tracking box. This represents the coordinates of the center point of the target detection box. This indicates the coordinates of the center point of the target tracking box.

[0120] The formula for calculating the second Hungarian match result is as follows:

[0121]

[0122] in, Represents the similarity based on Euclidean distance. Represents the cosine similarity of HOG features. This represents the weighted sum of Euclidean distance similarity and HOG feature cosine similarity. This represents the normalized weighting factor for similarity based on Euclidean distance. This represents the normalized weighting factor based on the cosine similarity of HOG features.

[0123] S330: Third-stage cascaded matching: Matching the target detection box and the target tracking box based on ReID.

[0124] The third-stage cascaded matching includes the following steps:

[0125] Acquire newly detected target organism video frames, retain several frames as a feature query library for the ROI region, and acquire video frames of the target organism corresponding to the target marker. The number of newly detected target organism video frames is the same as the number of target organism video frames corresponding to the target marker.

[0126] The pre-trained feature extraction network is trained using a metric learning loss function to obtain the trained feature extraction network.

[0127] The trained feature extraction network performs ROI feature extraction on newly detected target biological video frames and target biological video frames corresponding to target markers, and obtains the ROI feature extraction results of newly detected target biological video frames and target biological video frames corresponding to target markers.

[0128] The cosine similarity is calculated for the newly detected target biological video frame ROI feature extraction results and the target biological video frame ROI feature extraction results corresponding to the target marker, and the cosine similarity is added to obtain the total cosine similarity.

[0129] When the total cosine similarity is greater than the second preset threshold, the newly detected target organism is determined to be a target organism that has appeared before, and the target marker is assigned to the newly detected target organism. Otherwise, a new target marker is assigned to the newly detected target organism, thereby completing the ReID matching of the target organism and obtaining the matched tracking sequence.

[0130] The formula for calculating cosine similarity is:

[0131]

[0132] in, This indicates the ROI feature extraction results of the newly detected target biological video frame. This indicates the ROI feature extraction results for the target biological video frame corresponding to the target marker;

[0133] The formula for calculating the total cosine similarity is:

[0134]

[0135] in, Represents the total cosine similarity. This represents the cosine similarity.

[0136] In step S400, the displacement of the center point of the target tracking box is obtained through the matched tracking sequence, and then the pixel crawling distance is obtained.

[0137] In the S500, the pixel crawling time is obtained by capturing the time difference between the center points of the tracking box, and the target pixel crawling speed is obtained based on the pixel crawling distance and pixel crawling time.

[0138] In S500, a scale is obtained by calculating the pixel crawling distance and the actual physical distance. The pixel crawling distance and pixel crawling speed are calculated based on the scale to obtain the actual crawling distance and actual crawling speed of the target organism.

[0139] The main improvements over traditional methods of observing target organisms with the naked eye are: effective recording of the actual crawling trajectory of the target organism with high accuracy. It provides good trajectory recording results for target organism crawling under various conditions (slow movement, fast movement, jumping movement, sudden displacement, and re-identification of the target organism). Furthermore, this embodiment also provides effective recording of indicators such as speed, distance, and crawling time. By converting the scale, the movement of the target organism in the image is effectively transformed into actual movement observation indicators, which is more helpful for researchers to study the movement, habits, and preferences of target organisms.

[0140] Example 2:

[0141] A motion observation system based on the crawling trajectory of a target organism, such as Figure 2 As shown, it includes: a data acquisition module 100, a target detection box allocation module 200, a cascading matching module 300, a pixel crawling distance acquisition module 400, a target pixel crawling data acquisition module 500, and an actual crawling data acquisition module 600.

[0142] The data acquisition module 100 is used to acquire video of the target organism crawling, perform target detection on the video frames to obtain the target region, and perform false target screening on the target region to obtain the target detection box, wherein the target detection box includes at least one detection box;

[0143] The target detection box allocation module 200 is used to allocate target markers to the target detection boxes and generate target tracking boxes by performing multi-target tracking on the target detection boxes;

[0144] The cascade matching module 300 is used to cascade and match the target detection box and the target tracking box to obtain the matched tracking sequence;

[0145] The pixel crawling distance acquisition module 400 is used to obtain the displacement of the center point of the target tracking box through the matched tracking sequence, and then obtain the pixel crawling distance;

[0146] The target pixel crawling data acquisition module 500 is used to obtain the pixel crawling time by capturing the time difference between the center points of the tracking box, and to obtain the target pixel crawling speed based on the pixel crawling distance and pixel crawling time.

[0147] The actual crawling data acquisition module 600 is used to calculate a scale based on the pixel crawling distance and the actual physical distance. The pixel crawling distance and pixel crawling speed are calculated based on the scale to obtain the actual crawling distance and actual crawling speed of the target organism.

[0148] This invention is divided into two parts: training and testing. The training part can be developed using Linux-Ubuntu 20.04, with an Intel Core i7-8700 CPU, 64GB of memory, and two NVIDIA GeForce RTX 4090 GPUs for model training. The GPU driver version is 515.57, CUDA version is 11.7, and TensorRT version is 8.5.1.7. The main development languages ​​are C++ and Python. OpenCV version 4.6 is used when configuring the convolutional network. Of course, other operating systems developed based on this method can also be used.

[0149] Various changes and modifications made without departing from the spirit and scope of this invention, and all equivalent technical solutions, also fall within the scope of this invention.

[0150] The various embodiments in this specification are described in a progressive manner, with each embodiment focusing on the differences from other embodiments. The same or similar parts between the various embodiments can be referred to each other.

[0151] Those skilled in the art will understand that embodiments of the present invention can be provided as methods, apparatus, or computer program products. Therefore, the present invention can take the form of a completely hardware embodiment, a completely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, the present invention can take the form of a computer program product embodied on one or more computer-usable storage media (including, but not limited to, disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0152] This invention is described with reference to flowchart illustrations and / or block diagrams of the method, terminal device (system), and computer program product according to the invention. It will be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, special-purpose computer, embedded processor, or other programmable data processing terminal device to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing terminal device, generate instructions for implementing the flowchart illustrations and / or block diagrams. Figure 1 One or more processes and / or boxes Figure 1A device that provides the functions specified in one or more boxes.

[0153] These computer program instructions may also be stored in a computer-readable storage medium that can direct a computer or other programmable data processing terminal device to operate in a particular manner, such that the instructions stored in the computer-readable storage medium produce an article of manufacture including instruction means, which are implemented in a process Figure 1 One or more processes and / or boxes Figure 1 The function specified in one or more boxes.

[0154] These computer program instructions can also be loaded onto a computer or other programmable data processing terminal equipment, causing a series of operational steps to be performed on the computer or other programmable terminal equipment to produce a computer-implemented process, thereby providing instructions that execute on the computer or other programmable terminal equipment for implementing the process. Figure 1 One or more processes and / or boxes Figure 1 The steps of the function specified in one or more boxes.

[0155] It should be noted that:

[0156] The phrase "an embodiment" or "an embodiment" used in this specification means that a particular feature, structure, or characteristic described in connection with the embodiment is included in at least one embodiment of the invention. Therefore, the phrase "an embodiment" or "an embodiment" appearing in various places throughout the specification does not necessarily refer to the same embodiment.

[0157] Furthermore, it should be noted that the shapes and names of the components in the specific embodiments described in this specification may differ. All equivalent or simple variations made to the structure, features, and principles described in this patent concept are included within the protection scope of this patent. Those skilled in the art to which this invention pertains may make various modifications or additions to the described specific embodiments or use similar methods to substitute them, as long as they do not depart from the structure of this invention or exceed the scope defined in these claims, they should all fall within the protection scope of this invention.

Claims

1. A motion observation method based on the crawling trajectory of a target organism, characterized in that, Includes the following steps: Acquire a video of the target organism crawling, perform target detection on the video frames to obtain the target region, and filter out false targets in the target region to obtain a target detection box, wherein the target detection box includes at least one detection box; Assign target markers to the target detection boxes and perform multi-target tracking on the target detection boxes to generate target tracking boxes; Assign target markers to the filtered target detection boxes and assign a tracker to each target; The tracker tracks the target detection box, matches the detected target with the detected target appearing in the next frame, performs multi-target tracking on the target detection box and incorporates life cycle rules during the tracking process to obtain the target tracking box; The target detection box and the target tracking box are concatenated and matched to obtain the matched tracking sequence. The cascading matching includes at least three stages of cascading matching, and therefore includes the following steps: The first stage of cascaded matching involves performing IOU matching between the target detection box and the target tracking box, calculating the IOU value between each target detection box and each target tracking box, and performing the first Hungarian matching on target detection boxes and target tracking boxes whose IOU value is greater than a first preset threshold. Second-stage cascaded matching: For target detection boxes and target tracking boxes whose IOU values ​​are less than the first preset threshold in the first-stage cascaded matching, calculate the HOG feature cosine similarity between the target detection boxes and the target tracking boxes, and perform a second Hungarian matching by combining the Euclidean distance similarity. The third stage of cascaded matching: matching the target detection box and the target tracking box based on ReID; The first stage of cascaded matching includes the following steps: Calculate the IOU value between each target detection box and each target tracking box, and perform IOU matching between the target detection boxes and the target tracking boxes; For target detection boxes and target tracking boxes with an IOU value greater than the first preset threshold, the best match between target detection boxes and target tracking boxes in adjacent frames is obtained by using the Hungarian matching algorithm. The formula for calculating the IOU value is: Where A represents the set of target detection boxes and B represents the set of target tracking boxes; The second stage of cascaded matching includes the following steps: In the first stage, the target detection boxes and target tracking boxes with an IOU value less than the first preset threshold are matched together. The HOG feature cosine similarity between the target detection boxes and the target tracking boxes is calculated. Calculate the Euclidean distance between the center point of the target detection box and the center point of the target tracking box; The second Hungarian matching result is obtained by weighted summation of Euclidean distance similarity and HOG feature cosine similarity. The Euclidean distance between the center point of the target detection box and the center point of the target tracking box is calculated using the following formula: in, This represents the Euclidean distance between the center point of the target detection box and the center point of the target tracking box. This represents the coordinates of the center point of the target detection box. Indicates the coordinates of the center point of the target tracking box; The formula for calculating the second Hungarian match result is as follows: in, Represents the similarity based on Euclidean distance. Represents the cosine similarity of HOG features. This represents the weighted sum of Euclidean distance similarity and HOG feature cosine similarity. This represents the normalized weighting factor for similarity based on Euclidean distance. This represents the normalized weighting factor based on HOG feature cosine similarity; The third-stage cascaded matching includes the following steps: Acquire newly detected target organism video frames, retain several frames as a feature query library for the ROI region, and acquire video frames of the target organism corresponding to the target marker. The number of newly detected target organism video frames is the same as the number of target organism video frames corresponding to the target marker. The pre-trained feature extraction network is trained using a metric learning loss function to obtain the trained feature extraction network. The trained feature extraction network performs ROI feature extraction on newly detected target biological video frames and target biological video frames corresponding to target markers, and obtains the ROI feature extraction results of newly detected target biological video frames and target biological video frames corresponding to target markers. The cosine similarity is calculated for the newly detected target biological video frame ROI feature extraction results and the target biological video frame ROI feature extraction results corresponding to the target marker, and the cosine similarity is added to obtain the total cosine similarity. When the total cosine similarity is greater than the second preset threshold, the newly detected target creature is determined to be a target creature that has appeared before, and the target tag is assigned to the newly detected target creature; otherwise, a new target tag is assigned to the newly detected target creature, thereby completing the ReID matching of the target creature and obtaining the matched tracking sequence. The formula for calculating cosine similarity is: in, This indicates the ROI feature extraction results of the newly detected target biological video frame. This indicates the ROI feature extraction results for the target biological video frame corresponding to the target marker; The formula for calculating the total cosine similarity is: in, Represents the total cosine similarity. Indicates cosine similarity; The displacement of the center point of the target tracking box is obtained through the matched tracking sequence, and then the pixel crawling distance is obtained. The pixel crawling time is obtained by capturing the time difference between the center points of the tracking box. Based on the pixel crawling distance and the pixel crawling time, the target pixel crawling speed is obtained. A scale is obtained by calculating the pixel crawling distance and the actual physical distance. The pixel crawling distance and pixel crawling speed are then calculated based on the scale to obtain the actual crawling distance and actual crawling speed of the target organism.

2. The motion observation method based on the crawling trajectory of a target organism according to claim 1, characterized in that, The process of acquiring a training video of the target organism crawling and performing target detection on consecutive video frames to obtain the target region includes the following steps: Acquire videos of the target organism crawling, extract video frames to define the target bounding boxes, and obtain target bounding boxes to form a calibration dataset; The calibration dataset is divided into a training set, a validation set, and a test set. The pre-trained object detection model is then trained and validated to obtain the object detection model. The target region is obtained by detecting targets in consecutive video frames using a target detection model.

3. A motion observation system based on the crawling trajectory of a target organism, characterized in that, It includes a data acquisition module, a target detection box allocation module, a cascading matching module, a pixel crawling distance acquisition module, a target pixel crawling data acquisition module, and an actual crawling data acquisition module. The data acquisition module is used to acquire video of the target organism crawling, perform target detection on the video frames to obtain the target region, and filter out false targets in the target region to obtain the target detection box, wherein the target detection box includes at least one detection box; The target detection box allocation module is used to assign target markers to target detection boxes and generate target tracking boxes by performing multi-target tracking on the target detection boxes; Assign target markers to the filtered target detection boxes and assign a tracker to each target; The tracker tracks the target detection box, matches the detected target with the detected target appearing in the next frame, performs multi-target tracking on the target detection box and incorporates life cycle rules during the tracking process to obtain the target tracking box; The cascaded matching module is used to cascade the target detection box and the target tracking box to obtain the matched tracking sequence; The cascading matching includes at least three stages of cascading matching, and therefore includes the following steps: The first stage of cascaded matching involves performing IOU matching between the target detection box and the target tracking box, calculating the IOU value between each target detection box and each target tracking box, and performing the first Hungarian matching on target detection boxes and target tracking boxes whose IOU value is greater than a first preset threshold. Second-stage cascaded matching: For target detection boxes and target tracking boxes whose IOU values ​​are less than the first preset threshold in the first-stage cascaded matching, calculate the HOG feature cosine similarity between the target detection boxes and the target tracking boxes, and perform a second Hungarian matching by combining the Euclidean distance similarity. The third stage of cascaded matching: matching the target detection box and the target tracking box based on ReID; The first stage of cascaded matching includes the following steps: Calculate the IOU value between each target detection box and each target tracking box, and perform IOU matching between the target detection boxes and the target tracking boxes; For target detection boxes and target tracking boxes with an IOU value greater than the first preset threshold, the best match between target detection boxes and target tracking boxes in adjacent frames is obtained by using the Hungarian matching algorithm. The formula for calculating the IOU value is: Where A represents the set of target detection boxes and B represents the set of target tracking boxes; The second stage of cascaded matching includes the following steps: In the first stage, the target detection boxes and target tracking boxes with an IOU value less than the first preset threshold are matched together. The HOG feature cosine similarity between the target detection boxes and the target tracking boxes is calculated. Calculate the Euclidean distance between the center point of the target detection box and the center point of the target tracking box; The second Hungarian matching result is obtained by weighted summation of Euclidean distance similarity and HOG feature cosine similarity. The Euclidean distance between the center point of the target detection box and the center point of the target tracking box is calculated using the following formula: in, This represents the Euclidean distance between the center point of the target detection box and the center point of the target tracking box. This represents the coordinates of the center point of the target detection box. Indicates the coordinates of the center point of the target tracking box; The formula for calculating the second Hungarian match result is as follows: in, Represents the similarity based on Euclidean distance. Represents the cosine similarity of HOG features. This represents the weighted sum of Euclidean distance similarity and HOG feature cosine similarity. This represents the normalized weighting factor for similarity based on Euclidean distance. This represents the normalized weighting factor based on HOG feature cosine similarity; The third-stage cascaded matching includes the following steps: Acquire newly detected target organism video frames, retain several frames as a feature query library for the ROI region, and acquire video frames of the target organism corresponding to the target marker. The number of newly detected target organism video frames is the same as the number of target organism video frames corresponding to the target marker. The pre-trained feature extraction network is trained using a metric learning loss function to obtain the trained feature extraction network. The trained feature extraction network performs ROI feature extraction on newly detected target biological video frames and target biological video frames corresponding to target markers, and obtains the ROI feature extraction results of newly detected target biological video frames and target biological video frames corresponding to target markers. The cosine similarity is calculated for the newly detected target biological video frame ROI feature extraction results and the target biological video frame ROI feature extraction results corresponding to the target marker, and the cosine similarity is added to obtain the total cosine similarity. When the total cosine similarity is greater than the second preset threshold, the newly detected target creature is determined to be a target creature that has appeared before, and the target tag is assigned to the newly detected target creature; otherwise, a new target tag is assigned to the newly detected target creature, thereby completing the ReID matching of the target creature and obtaining the matched tracking sequence. The formula for calculating cosine similarity is: in, This indicates the ROI feature extraction results of the newly detected target biological video frame. This indicates the ROI feature extraction results for the target biological video frame corresponding to the target marker; The formula for calculating the total cosine similarity is: in, Represents the total cosine similarity. Indicates cosine similarity; The pixel crawling distance acquisition module is used to obtain the displacement of the center point of the target tracking box through the matched tracking sequence, and then obtain the pixel crawling distance; The target pixel crawling data acquisition module is used to obtain the pixel crawling time by capturing the time difference between the center points of the tracking box, and to obtain the target pixel crawling speed based on the pixel crawling distance and pixel crawling time. The actual crawling data acquisition module is used to calculate the scale based on the pixel crawling distance and the actual physical distance. The pixel crawling distance and pixel crawling speed are calculated based on the scale to obtain the actual crawling distance and actual crawling speed of the target organism.

4. The motion observation system based on the crawling trajectory of a target organism according to claim 3, characterized in that, Assigning target markers to target detection boxes and generating target tracking boxes through multi-target tracking of target detection boxes includes the following steps: Assign target markers to the filtered target detection boxes and assign a tracker to each target; The tracker tracks the target detection box, matches the detected target with the detected target appearing in the next frame, performs multi-target tracking on the target detection box, and incorporates life cycle rules into the tracking process to obtain the target tracking box.

5. A computer-readable storage medium storing a computer program, characterized in that, When the computer program is executed by a processor, it implements the method according to any one of claims 1 to 2.

6. A motion observation device based on the crawling trajectory of a target organism, comprising a memory, a processor, and a computer program stored in the memory and running on the processor, characterized in that, When the processor executes the computer program, it implements the method according to any one of claims 1 to 2.

Images