A method, system, device, medium, and product for recognizing courtship behavior in small fish.

By employing background modeling, deep learning, and sparse optical flow tracking techniques, the problem of automatically identifying courtship behavior in small fish has been solved. This enables real-time differentiation between male and female individuals and accurate determination of courtship behavior, thereby improving recognition efficiency and accuracy.

CN123071631APending Publication Date: 2026-11-10INST OF AQUATIC LIFE ACAD SINICA
View PDF 0 Cites 0 Cited by

Patent Information

Application Number
CN202611410441.8
Authority / Receiving Office
CN · China
Patent Type
Applications(China)
Current Assignee / Owner
Filing Date
2026-09-11
Publication Date
2026-11-10

AI Technical Summary

Technical Problem

Existing technologies struggle to automatically and accurately identify courtship behavior in small fish in real time, especially in aquaculture container environments, where they cannot effectively distinguish between male and female individuals and identify their courtship behavior.

Method used

The method combines background modeling and inter-frame difference to extract moving targets, filters out interfering targets through geometric features, identifies gender categories using a multi-task deep learning classification network, extracts key feature points through a deep learning regression network, and determines the fish's posture and trajectory features by combining sparse optical flow tracking, ultimately achieving automatic determination of courtship behavior.

Benefits of technology

It enables real-time differentiation of male and female individuals of small fish in aquaculture containers and automatic and accurate identification of various courtship behaviors, improving the accuracy and automation level of identification.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure FT_1
    Figure FT_1
  • Figure FT_2
    Figure FT_2
  • Figure FT_3
    Figure FT_3
Patent Text Reader

Abstract

This application discloses a method, system, device, medium, and product for recognizing courtship behavior in small fish, relating to the field of fish behavior recognition. The method includes acquiring the current frame image of a target fish within a culture container and extracting the region of interest; extracting moving targets based on background modeling and inter-frame difference and filtering out interference; identifying the sex category of each target and eliminating non-fish using a multi-task deep learning classification network; inputting valid targets into a deep learning regression network to extract key feature points, and obtaining a posture descriptor based on the set of position coordinates of the key feature points; obtaining trajectory features using a sparse optical flow tracking method; and determining the courtship behavior judgment result based on the sex category, posture descriptor, and trajectory features. This application achieves real-time differentiation of male and female individuals in small fish based on the fusion of fish posture and movement trajectory, and automatically and accurately identifies their respective courtship behaviors, improving the accuracy and automation level of courtship behavior recognition.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] This application relates to the field of fish behavior recognition, and in particular to a method, system, device, medium and product for recognizing the courtship behavior of small fish. Background Technology

[0002] Small fish (such as zebrafish and killifish) play an important role as model organisms in genetics, developmental biology, neuroscience, and drug screening. Courtship behavior in fish is a crucial aspect of their reproductive process, directly reflecting their health, endocrine status, and reproductive capacity. Therefore, the automatic identification and real-time monitoring of courtship behavior in small fish has significant application value in biological research, drug screening, and aquaculture.

[0003] Currently, fish behavior analysis mainly relies on manual observation and recording, which suffers from high subjectivity, low efficiency, and the inability to conduct continuous monitoring for extended periods. Existing video analysis methods are primarily used for tracking fish swimming trajectories, but they lack precise quantitative descriptions of fish postures and struggle to distinguish between different courtship behavior categories, resulting in low accuracy and poor reliability in the automatic identification of courtship behaviors.

[0004] Therefore, there is an urgent need for a method to identify the courtship behavior of small fish, in order to solve the problem that it is impossible to distinguish between male and female small fish in aquaculture containers in real time and accurately identify their respective courtship behaviors based on the combined features of fish posture and movement trajectory, thereby improving the accuracy and automation level of courtship behavior identification of small fish. Summary of the Invention

[0005] The purpose of this application is to provide a method, system, device, medium, and product for recognizing courtship behavior in small fish. This method can achieve real-time differentiation between male and female individuals of small fish in aquaculture containers, and automatically and accurately identify various courtship behaviors of male and female fish based on the fusion analysis of fish posture descriptors and trajectory features, thereby improving the accuracy and automation level of fish courtship behavior recognition.

[0006] To achieve the above objectives, this application provides the following solution: Firstly, this application provides a method for recognizing courtship behavior in small fish, including: Acquire the current frame image of the target fish in the aquaculture container and extract the region of interest image from the current frame image; Based on the region of interest image, moving targets are extracted by combining background modeling and inter-frame difference to obtain a set of candidate moving targets for the current frame; Based on the candidate moving target set, interference targets are filtered out using a geometric feature filtering method to obtain the initial moving target set for the current frame; Each initial moving target in the initial moving target set of the current frame is input into a pre-trained multi-task deep learning classification network, which outputs the gender category of each initial moving target in the current frame; the gender category includes non-fish, male fish, and female fish. Remove the initial moving targets whose gender category is not fish from the initial moving target set to obtain the valid moving target set for the current frame; Each valid moving target in the set of valid moving targets in the current frame is input into a pre-trained deep learning regression network, which outputs a set of position coordinates of key feature points of each valid moving target in the current frame; the key feature points include the fish head, dorsal fin, caudal fin, anal fin, and pelvic fin; Based on the set of position coordinates of key feature points of each effective moving target in the current frame, the distance, direction, angle, normalized projection component and arc-to-direction ratio between key feature points are calculated to obtain the attitude descriptor of each effective moving target in the current frame. The trajectory features of each effective moving target in the current frame are determined by sparse optical flow tracking. Based on the sex category, posture descriptor, and trajectory features of each valid moving target in the current frame, the mating behavior determination results of each valid moving target in the current frame are determined; the mating behavior determination results include the mating behavior determination results of male fish and female fish; the mating behavior determination results of male fish include chasing, tail chasing, circling, figure-eight swimming, or trembling; the mating behavior determination results of female fish include approaching, escorting, displaying, leading, or spawning.

[0007] Secondly, this application provides a small fish courtship behavior recognition system, comprising: The image acquisition module is used to acquire the current frame image of the target fish in the aquaculture container and extract the region of interest image from the current frame image; The moving target extraction module is used to extract moving targets based on the region of interest image by combining background modeling and inter-frame difference to obtain a set of candidate moving targets for the current frame. The interference target filtering module is used to filter out interference targets based on the candidate moving target set using a geometric feature filtering method, so as to obtain the initial moving target set of the current frame; The multi-task classification module is used to input each initial moving target in the initial moving target set of the current frame into a pre-trained multi-task deep learning classification network, and output the gender category of each initial moving target in the current frame; the gender category includes non-fish, male fish and female fish; The effective target filtering module is used to delete initial moving targets whose gender category is not fish from the initial moving target set, and obtain the effective moving target set of the current frame; The key point regression module is used to input each effective moving target in the set of effective moving targets in the current frame into a pre-trained deep learning regression network, and output the set of position coordinates of key feature points of each effective moving target in the current frame; the key feature points include the fish head, dorsal fin, caudal fin, anal fin, and pelvic fin; The attitude quantization module is used to calculate the distance, direction, angle, normalized projection component and arc-to-direction ratio between key feature points based on the set of position coordinates of key feature points of each effective moving target in the current frame, so as to obtain the attitude descriptor of each effective moving target in the current frame. The trajectory tracking module is used to determine the trajectory features of each effective moving target in the current frame using a sparse optical flow tracking method. The behavior determination module is used to determine the courtship behavior determination results of each valid moving target in the current frame based on the sex category, posture descriptor, and trajectory features of each valid moving target in the current frame. The courtship behavior determination results include the courtship behavior determination results of male fish and female fish. The courtship behavior determination results of male fish include chasing, tail chasing, circling, figure-eight swimming, or trembling. The courtship behavior determination results of female fish include approaching, escorting, displaying, leading, or spawning.

[0008] Thirdly, this application provides a computer device, including: a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the computer program to implement the steps of the method for recognizing courtship behavior of small fish described above.

[0009] Fourthly, this application provides a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the steps of the method for recognizing the courtship behavior of small fish described above.

[0010] Fifthly, this application provides a computer program product, including a computer program that, when executed by a processor, implements the steps of the method for recognizing courtship behavior of small fish described above.

[0011] According to the specific embodiments provided in this application, this application has the following technical effects: This application provides a method, system, device, medium, and product for recognizing courtship behavior in small fish. By acquiring the current frame image of the target fish within a culture container and extracting the region of interest (ROI) image, it solves the problem of interference areas outside the culture container affecting processing efficiency and accuracy in video images. This enables rapid localization of the target fish's location, improving the computational efficiency and accuracy of subsequent processing. By extracting moving targets based on the ROI image and filtering candidate moving targets using geometric features, it addresses the problem of interference targets such as water ripples, light, and noise affecting detection accuracy in the culture environment. This effectively filters out interfering targets from candidate moving targets, improving the reliability of initial moving target detection. By inputting the initial set of moving targets into a pre-trained multi-task deep learning classification network for gender category recognition and deleting non-fish targets, it solves the problem of simultaneously distinguishing between fish and non-fish, as well as accurately distinguishing between male and female individuals. This achieves effective separation of male and female fish and automatic removal of non-fish targets, paving the way for subsequent pose recognition and behavior determination. It provides an accurate foundation for sex classification; by inputting effective moving targets into a pre-trained deep learning regression network to extract the location coordinates of key feature points and calculate posture descriptors, it solves the problem of lacking precise quantitative description of fish posture, realizing the quantification of fish posture based on key feature points of the head, dorsal fin, caudal fin, anal fin, and pelvic fin, providing fine posture information for courtship behavior recognition; by determining the trajectory features of effective moving targets through sparse optical flow tracking, it solves the problem of not being able to obtain continuous motion information by relying solely on single-frame images, realizing real-time tracking of the swimming trajectories of male and female fish and accurate extraction of trajectory features; by determining the courtship behavior judgment results based on sex classification, posture descriptors, and trajectory features, it solves the problem of not being able to distinguish male and female individuals of small fish in aquaculture containers in real time and accurately identify their respective courtship behaviors based on the combined features of fish posture and movement trajectory, realizing the automatic and accurate identification of 10 types of courtship behaviors, including male fish chasing, tail chasing, circling, figure-eight swimming, trembling, and female fish approaching, escorting, displaying, leading, and spawning. Attached Figure Description

[0012] To more clearly illustrate the technical solutions in the embodiments of this application or the prior art, the drawings used in the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0013] Figure 1 A flowchart illustrating a method for recognizing courtship behavior in small fish, provided as an embodiment of this application; Figure 2 A schematic diagram of the functional modules of a small fish courtship behavior recognition system provided in an embodiment of this application; Figure 3 This is a schematic diagram of the structure of a computer device provided in an embodiment of this application. Detailed Implementation

[0014] The technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, and not all embodiments. Based on the embodiments of this application, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application.

[0015] To make the above-mentioned objectives, features and advantages of this application more apparent and understandable, the application will be further described in detail below with reference to the accompanying drawings and specific embodiments.

[0016] In one exemplary embodiment, such as Figure 1 As shown, a method for recognizing courtship behavior in small fish is provided. This method is executed by a computer device, specifically a terminal or server, or both. In this embodiment, it includes steps 101 to 109. Wherein: Step 101: Obtain the current frame image of the target fish in the aquaculture container and extract the region of interest image from the current frame image.

[0017] Step 102: Based on the region of interest image, moving targets are extracted by combining background modeling and inter-frame difference to obtain a set of candidate moving targets for the current frame.

[0018] Step 103: Based on the candidate moving target set, interference targets are filtered out using a geometric feature filtering method to obtain the initial moving target set for the current frame.

[0019] Step 104: Input each initial moving target in the initial moving target set of the current frame into the pre-trained multi-task deep learning classification network, and output the gender category of each initial moving target in the current frame; the gender category includes non-fish, male fish and female fish.

[0020] Step 105: Delete the initial moving targets whose gender category is not fish from the initial moving target set to obtain the valid moving target set for the current frame.

[0021] Step 106: Input each effective moving target in the set of effective moving targets in the current frame into the pre-trained deep learning regression network, and output the set of position coordinates of key feature points of each effective moving target in the current frame; the key feature points include the fish head, dorsal fin, caudal fin, anal fin, and pelvic fin.

[0022] Step 107: Based on the set of position coordinates of key feature points of each effective moving target in the current frame, calculate the distance, direction, angle, normalized projection component and arc-to-direction ratio between key feature points to obtain the attitude descriptor of each effective moving target in the current frame.

[0023] Step 108: Determine the trajectory features of each effective moving target in the current frame using the sparse optical flow tracking method.

[0024] Step 109: Determine the courtship behavior judgment results of each valid moving target in the current frame based on the sex category, posture descriptor, and trajectory features of each valid moving target in the current frame; the courtship behavior judgment results include the courtship behavior judgment results of male fish and female fish; the courtship behavior judgment results of male fish include chasing, tail chasing, circling, figure-eight swimming, or trembling; the courtship behavior judgment results of female fish include approaching, escorting, displaying, leading, or spawning.

[0025] By implementing steps 101 to 109 above, this application can achieve precise separation of fish and non-fish, and male and female fish in complex aquaculture environments through a multi-task deep learning classification network. It can extract key feature points of fish and quantify posture descriptors through a deep learning regression network, and obtain trajectory features by combining sparse optical flow tracking. Then, based on the combined features of posture and trajectory, it can automatically and accurately identify 10 types of courtship behaviors of male fish, including chasing, tail chasing, circling, figure-eight swimming, trembling, and female fish approaching, escorting, displaying, leading, and spawning. This solves the problem of not being able to distinguish male and female individuals of small fish in aquaculture containers in real time and accurately identify their respective courtship behaviors based on the combined features of fish posture and movement trajectory, and improves the accuracy and automation level of courtship behavior identification of small fish.

[0026] In another exemplary embodiment of this application, step 102 specifically includes: The background image is obtained by averaging the pixels of the first preset number of frames preceding the current frame. Calculate the pixel difference between the current frame image and the background image at corresponding positions to obtain the first difference image; Calculate the pixel difference between the corresponding positions of the current frame image and the second preset number of frames before the current frame to obtain the second difference image; The pixel values ​​at corresponding positions in the first difference image and the second difference image are added together to obtain the fused difference image; Based on the fused difference image, a binarized output image is obtained through a preset binarization method. The preset binarization method includes taking any pixel in the fused difference image as the current pixel. If the absolute value of the pixel value of the current pixel is greater than a preset threshold, the current pixel is marked as a foreground pixel. If the absolute value of the pixel value of the current pixel is less than or equal to the preset threshold, the current pixel is marked as a background pixel. Contour extraction is performed on the binarized output image to obtain a contour set; Calculate the contour area of ​​each contour in the contour set, remove contours with a contour area less than a preset area threshold as noise contours, and retain contours with a contour area greater than or equal to the preset area threshold as valid contours. Calculate the minimum x-coordinate, maximum x-coordinate, minimum y-coordinate, and maximum y-coordinate of each of the effective contours; The bounding rectangles of each effective contour are determined based on the minimum x-coordinate, maximum x-coordinate, minimum y-coordinate, and maximum y-coordinate. The image regions defined by each bounding rectangle in the current frame image are then used as candidate moving targets, thus obtaining the candidate moving target set for the current frame.

[0027] In another exemplary embodiment of this application, the posture descriptor includes the directional angle between the line connecting the fish head and caudal fin and the horizontal direction, the normalized vertical component of the line connecting the dorsal fin and pelvic fin, the caudal fin bending direction, the normalized degree of fish body bending, the angle between the line connecting the anal fin and pelvic fin and the line connecting the pelvic fin and the fish head, the normalized horizontal component of the line connecting the dorsal fin and pelvic fin, the angle between the line connecting the anal fin and pelvic fin, and the ratio of the arc length of the fish body centerline to the straight-line distance from the fish head to the caudal fin (arc-to-straight ratio). Step 107 specifically includes: The pose descriptors for each valid moving target in the current frame are obtained using the following formula: ; ; ; ; ; ; ; ; In the pose descriptor calculation, the image's vertical coordinate is converted to an upward-positive coordinate, with the x-axis positive to the right and the positive angle direction being counterclockwise. The directional angle between the line connecting the fish's head and tail fin and the horizontal direction; and These represent the x-coordinate and y-coordinate of the fish head, respectively. and The x and y coordinates represent the caudal fin, respectively. Represents the normalized vertical component of the line connecting the dorsal and pelvic fins; and The x and y coordinates represent the dorsal fin, respectively. and The x and y coordinates of the pelvic fins are represented respectively. Indicates the direction of the tail fin's curvature; This indicates the degree of curvature of the fish body after normalization; the larger the value, the greater the degree of curvature. The directional angle between the line connecting the fish's head and tail fin and the vertical direction; Represents the normalized horizontal component of the line connecting the dorsal and pelvic fins; This represents the angle between the line connecting the anal and pelvic fins and the line connecting the pelvic fins and the fish's head. Indicates the arc length of the fish's centerline; It represents the ratio of the arc length of the fish's centerline to the straight-line distance from the fish's head to its tail fin; This represents the Euclidean distance from the fish's head to its dorsal fin. This represents the Euclidean distance from the dorsal fin to the caudal fin. This represents the Euclidean distance from the caudal fin to the anal fin. This represents the Euclidean distance from the anal fin to the pelvic fin. This represents the Euclidean distance from the pelvic fin to the fish's head. This represents the Euclidean distance from the fish's head to its tail fin. This represents the Euclidean distance from the dorsal fin to the pelvic fin. This represents the Euclidean distance from the dorsal fin to the anal fin. This indicates the Euclidean distance from the anal fin to the fish's head.

[0028] In another exemplary embodiment of this application, step 108 specifically includes: Within the regions of each effective moving target in the current frame, the corner response value of each pixel is calculated using a corner detection algorithm, and the third preset number of pixels with the largest corner response value are selected as tracking feature points to obtain the set of tracking feature points corresponding to each effective moving target in the current frame. For each set of tracking feature points corresponding to each effective moving target in the current frame, the sparse optical flow tracking method is used to obtain the displacement vector of each tracking feature point in the set of tracking feature points corresponding to each effective moving target from the previous frame to the current frame and the forward tracking position corresponding to the displacement vector. The reverse optical flow from the forward tracking position to the previous frame is calculated to obtain the reverse tracking position from the current frame to the previous frame. The displacement vector whose error between the forward tracking position and the reverse tracking position is less than a preset pixel threshold is determined as the effective displacement vector; The effective displacement vectors of each effective moving target are fused to obtain the overall displacement vector of each effective moving target. Based on the overall displacement vector of each effective moving target, update the trajectory position of each effective moving target in the current frame; Based on the trajectory position of each effective moving target in the current frame and the trajectory position of the fourth preset number of frames before the current frame, the trajectory characteristics of each effective moving target are determined; the trajectory characteristics include instantaneous velocity, instantaneous acceleration, and trajectory distance.

[0029] In another exemplary embodiment of this application, step 109 specifically includes: Take any one of the valid moving targets in the current frame as the current valid moving target; If the current valid moving target is male, calculate the spatial distance between the current valid moving target and all valid moving targets of the current frame that are female, and select the valid moving target of the current frame that is female with the closest spatial distance as the female fish that is paired with the current valid moving target. If the current valid moving target's gender category is female fish, then calculate the spatial distance between the current valid moving target and all valid moving targets with the gender category of male fish in the current frame, and select the valid moving target with the closest spatial distance as the male fish corresponding to the current valid moving target; The trajectory features also include the correlation between the trajectory of the current effective moving target and the trajectory of the paired female or paired male fish; If the current valid target is male, the following steps are used to determine the male fish's courtship behavior: If the angle between the line connecting the head and tail fin of the current effective moving target fish and the horizontal direction is within the range of 150 degrees to 180 degrees, and the instantaneous speed of the current effective moving target fish is greater than 1.5 times the instantaneous speed of the paired female fish, and the trajectory distance of the current effective moving target fish meets the preset rapid increase trend condition, then the mating behavior of the male fish of the current effective moving target fish is determined to be chasing; wherein, the preset rapid increase trend condition includes taking a sliding window with the current frame as the endpoint and a length of the fifth preset frame number N1 (N1=10 in this embodiment), and calculating the trajectory distance increment of the current effective moving target fish within the window Δd=d(t)-d(t-N1); if Δd is greater than the first preset distance threshold T_d (T_d is twice the body length of the current effective moving target fish in this embodiment), and Δd is greater than 1.2 times the trajectory distance increment of the paired female fish fish within the same window, then the trajectory distance of the current effective moving target fish is determined to show a rapid increase trend. In the above formula, d(t) represents the trajectory distance of the current effective moving target in frame t (i.e. the current frame), d(t-N1) represents the trajectory distance of the current effective moving target in frame t-N1, and Δd represents the increment of the trajectory distance within the sliding window (i.e. the difference between the trajectory distance in frame t and frame t-N1).

[0030] If the distance between the current active target and the paired female fish is less than twice the body length of the current active target, and the difference between the angle between the line connecting the head and tail fin of the current active target and the horizontal direction and the angle between the line connecting the head and tail fin of the paired female fish and the horizontal direction is 180 degrees, then the mating behavior of the male fish of the current active target is determined to be tail chasing. If the correlation between the trajectory of the current effective moving target and the paired female fish is less than 0.3, and the instantaneous acceleration of the current effective moving target exhibits periodic changes, and the trajectory position sequence of the current effective moving target is circularly distributed, then the mating behavior of the male fish of the current effective moving target is determined to be circling. The periodic changes in instantaneous acceleration refer to: taking the instantaneous acceleration sequence of the current effective moving target within the most recent sixth preset frame number N2 (N2=30 in this embodiment), performing autocorrelation or Fourier transform on the sequence, and if its dominant frequency f_a is within a preset range of 0.5Hz to 3Hz and the acceleration amplitude is greater than a second preset acceleration threshold, then the instantaneous acceleration is determined to be periodic. The circular distribution of the trajectory position sequence refers to: fitting a circle center and radius to the trajectory position sequence of the current effective moving target; if the root mean square error of the distance from each trajectory point to the fitted circle center is less than a third preset radius threshold (or the roundness index of the trajectory point is less than a preset roundness threshold), then the trajectory position sequence is determined to be circularly distributed.

[0031] If the curvature of the fish body of the current effective moving target is within the range of 0.5 to 0.8, and the correlation between the trajectory of the current effective moving target and the paired female fish is less than 0.2, and the trajectory position sequence of the current effective moving target satisfies the preset figure-eight distribution condition, then the mating behavior of the male fish of the current effective moving target is determined to be figure-eight swimming. The preset figure-eight distribution condition includes calculating the curvature sign of the trajectory position sequence of the current effective moving target. If the curvature sign alternates between positive and negative at least once in consecutive frames (i.e., the trajectory bends to one side first and then to the other side), and the trajectory as a whole forms two tangent loops after intersecting once, then the trajectory position sequence is determined to be figure-eight distributed.

[0032] If the frequency of change in the bending degree of the fish body of the current effective moving target is greater than 5 Hz, and the instantaneous speed of the current effective moving target is less than 0.5 times the preset normal swimming speed, and the position change of the current effective moving target from the previous frame to the current frame is less than the preset position change threshold, then the mating behavior of the male fish of the current effective moving target is determined as trembling. If the current valid target is female, the following steps are used to determine the mating behavior of the female fish: If the angle between the head direction of the current effective moving target and the head direction of the paired male fish is less than 30 degrees, and the absolute value of the vertical component of the line connecting the dorsal and pelvic fins of the current effective moving target is less than 0.1, and the direction of the tail fin bend is within the preset straight state range, and the rate of decrease in the positional distance between the current effective moving target and the paired male fish is greater than 5 mm per second, and the instantaneous speed of the current effective moving target is greater than 0.8 times the preset normal swimming speed, and the instantaneous acceleration of the current effective moving target is positive, then the mating behavior of the female fish of the current effective moving target is judged as close. If the absolute value of the difference between the angle between the line connecting the head and tail fin of the current effective moving target and the horizontal direction and the angle between the line connecting the head and tail fin of the paired male fish and the horizontal direction is less than 15 degrees, and the absolute value of the vertical component of the line connecting the dorsal and pelvic fins is less than 0.1, and the degree of body curvature is within the range of 0.3 to 0.6, and the absolute value of the difference in instantaneous speed between the current effective moving target and the paired male fish is less than 0.1 times the instantaneous speed of the paired male fish, and the spatial distance between the current effective moving target and the paired male fish is maintained at 1 to 2 times the current... If the length of the effective moving target fish is equal to the length of the fish body, and the trajectory of the current effective moving target is parallel to that of the paired male fish, then the mating behavior of the female fish of the current effective moving target is determined as escorting. Here, the parallelism of the trajectory of the current effective moving target and the paired male fish means that within the seventh preset frame number N3 (N3=15 in this embodiment) before the current frame, the overall displacement vectors of the two fish are calculated respectively. If the angle between the two vector directions is less than the fourth preset angle threshold θ_par (θ_par=15° in this embodiment), then the two trajectories are determined to be parallel.

[0033] If the vertical component of the line connecting the dorsal and pelvic fins of the current effective moving target is within the range of -0.3 to 0.3, the angle between the line connecting the head and tail fins and the vertical direction is greater than a preset vertical angle threshold or the horizontal component of the line connecting the dorsal and pelvic fins is greater than a preset horizontal component threshold, and the angle between the line connecting the anal and pelvic fins is greater than 0, and the instantaneous speed of the current effective moving target is less than 0.3 times the preset normal swimming speed, and the position change of the current effective moving target from the previous frame to the current frame is less than a preset position change threshold, and the trajectory position of the current effective moving target meets the preset small-range swing condition, then the result of the female fish courtship behavior of the current effective moving target is to be displayed; wherein, the preset small-range swing condition includes taking the trajectory position of the current effective moving target within the most recent eighth preset frame number N4, calculating its lateral offset perpendicular to the overall swimming direction, and if the amplitude of the lateral offset is less than the fifth preset swing amplitude threshold (0.3 times the fish length in this embodiment) and changes periodically back and forth, then the trajectory position is determined to be a small-range swing.

[0034] If the line connecting the head and tail fin of the current effective moving target points towards the bottom of the breeding container, and the vertical component of the line connecting the dorsal and pelvic fins is within the range of -0.2 to 0.2, and the direction of tail fin curvature is within the preset straight-line state range, and the instantaneous speed of the current effective moving target is greater than 0.8 times the instantaneous speed of the paired male fish, and the trajectory position of the current effective moving target is towards the bottom of the breeding container, and the spatial distance between the current effective moving target and the paired male fish meets the preset change conditions, then the mating behavior judgment result of the female fish of the current effective moving target is leading; wherein, the trajectory position is towards the bottom of the breeding container means that the vertical coordinate y of the trajectory position of the current effective moving target (downward is the positive direction of y in the image coordinate system) continuously increases and y is greater than the preset bottom vertical coordinate threshold Y_b (in this embodiment, Y_b is taken as 80% of the image height) continuously for no less than the ninth preset frame number N5 (in this embodiment, N5=10 frames); the preset change conditions include the rate of change of the spatial distance between the two fish being less than the sixth preset rate of change threshold (in this embodiment, it is taken as 5mm / s), that is, remaining stable or changing slowly.

[0035] If the vertical component of the line connecting the dorsal and pelvic fins of the current effective moving target is greater than 0.2, and the ratio of the arc length to the straight-line distance of the fish's centerline is greater than a preset curvature ratio threshold, the trajectory position of the current effective moving target is close to the bottom of the aquaculture container, and the instantaneous speed of the current effective moving target is less than 0.2 times the preset normal swimming speed, and the position change of the current effective moving target from the previous frame to the current frame is less than the second preset position change threshold, then the mating behavior of the female fish of the current effective moving target is determined to be spawning. Here, "trajectory position close to the bottom of the aquaculture container" means that the vertical coordinate y of the trajectory position of the current effective moving target is greater than the preset bottom vertical coordinate threshold Y_b (in this embodiment, Y_b is 80% of the image height) and this condition persists for at least the ninth preset frame number; in this case, the trajectory position is determined to be close to the bottom of the aquaculture container.

[0036] In another exemplary embodiment of this application, the multi-task deep learning classification network includes a feature extraction layer and a two-dimensional classification output layer connected in sequence; The feature extraction layer is used to extract fish morphological features based on the initial moving target; The two-dimensional classification output layer is used to determine whether the initial moving target is a fish or a non-fish based on the fish morphological features, and to determine whether the initial moving target is a male or female fish based on the fish morphological features of the initial moving target, thereby outputting the gender category of each initial moving target in the current frame. Deep learning regression networks consist of a feature extraction layer and a regression output layer connected in sequence; The feature extraction layer reuses the feature extraction layer in the multi-task deep learning classification network to extract fish morphological features based on the initial moving target. The regression output layer is used to output the set of position coordinates of key feature points of each effective moving target in the current frame based on fish morphological features.

[0037] The following example illustrates this application using the specific process of identifying courtship behavior in small fish.

[0038] This application uses video moving target extraction, deep learning target recognition and trajectory tracking methods to identify male and female fish in real time. By judging the trajectory, positional relationship and posture information of the target fish, it achieves the purpose of real-time dynamic analysis of the courtship behavior of small fish.

[0039] Before calculating distance or velocity in millimeters, scale calibration is performed. Specifically, a known length calibration object inside the aquaculture container or the actual size of the aquaculture container is used as a reference. Let Sp be the known length in the image corresponding to Sp pixels and Lm be the actual length in millimeters. Then, the pixel-to-millimeter conversion factor K = Sp / Lm, with the unit being pixels / millimeter. The actual displacement is obtained by removing the target's pixels using K, and then the velocity in millimeters per second is calculated by combining the time interval between adjacent frames.

[0040] To achieve the above objectives, this application provides the following solution: Step S1, automatic extraction of region of interest; Step S2, extraction of moving targets in the video; Step S3, filtering of non-interested targets; Step S4, classification and identification of male and female fish; Step S5, pose recognition; Step S6, target trajectory tracking; Step S7, recognition of courtship behavior. The specific steps are as follows: Step S1: Automatically extract the region of interest.

[0041] In addition to the aquaculture container (a fish tank is used as an example in this embodiment), the data collected by the camera also includes some interference areas. To improve the algorithm's recognition accuracy and computational efficiency, it is necessary to automatically extract the region of interest from the entire image. Based on the characteristics of the fish tank's edges, this application designs a method for locating the fish tank's position using line segment fitting. First, the current frame image is converted into a grayscale image. Second, gradient calculation is performed on the current frame grayscale image to obtain the edge points in the x and y directions. Then, line segment fitting is performed on the edge points, and finally, the fitted line segments are sorted according to their length, direction, and position.

[0042] The above line segments are processed as follows: First, the longest line segment is selected as one long side of the fish tank. Second, a line segment with the same direction and a distance of about half that of the first line segment is selected as the other long side of the fish tank. Third, a line segment perpendicular to the two line segments and with a length of about one segment of the long side is selected as the short side of the fish tank. Finally, the other short side of the fish tank is extracted based on the direction and position relationship.

[0043] The four line segments extracted through the above steps are used as the outline boundary of the fish tank. The region within the outline boundary is then extracted from the current frame image as the region of interest image for subsequent algorithm processing.

[0044] Step S2: Extract moving targets from the video.

[0045] To extract targets from video images, this application utilizes background modeling and information from preceding and following videos for moving target extraction. First, background modeling uses the average value of the most recent 20 frames of image data as the background u(x, y). Second, the difference d1(x, y) is calculated by subtracting the corresponding pixel position between the current frame image I(x, y) and the background image. Then, the pixel difference d2(x, y) is calculated between the current frame image and the images of the three frames preceding it. d1(x, y) and d2(x, y) are added together to obtain d(x, y). d(x, y) is compared with a threshold TH. Pixels with an absolute value greater than the preset threshold are marked as foreground pixels (pixel value 1), and pixels with an absolute value less than or equal to the preset threshold are marked as background pixels (pixel value 0), resulting in an output image output. Contour extraction and bounding rectangle calculation are performed on the output. The resulting rectangular region is used as the moving target. These moving targets include male and female fish, and may also include some interfering targets.

[0046] As an optional implementation, in step S2, the Suzuki-Abe boundary tracking algorithm is used to extract the contours of the binary image (output) obtained from background modeling. The specific implementation steps are as follows: Step S2.1: Contour extraction.

[0047] Starting from the top left corner of the binary image output, the system scans line by line. When a foreground pixel (with a value of 1) is encountered and its 8-neighborhood contains only one other foreground pixel, this pixel is identified as the boundary starting point. From this starting point, adjacent foreground pixels are traced counter-clockwise, and the coordinates of each boundary point are recorded. When the tracing returns to the starting point, a complete contour is formed. Extraction.

[0048] Step S2.2: Calculate the contour area.

[0049] For the extracted i-th contour Calculate its area using the Shoelace formula: ; in, Represents outline Area (unit: pixels); , () is the outline The x and y coordinates of the j-th pixel, j=1,...,n, where n is the total number of pixels in the outline; , () is the outline Set the x and y coordinates of the (j+1)th pixel; set an area threshold. (In this embodiment, it is set to 50 pixels), if < If it is determined to be a noise profile, it will be removed; if ≥ If the value is 1, then it is retained as a valid moving target. It should be noted that when j=n, j+1 in the formula takes the value 1, i.e. ( , ) = ( , This ensures that the beginning and end of the outline pixels are closed.

[0050] Step S2.3: Calculate the circumscribed rectangle.

[0051] For the preserved effective contour Find the minimum x-coordinate value respectively. Minimum ordinate value Maximum x-coordinate value Maximum ordinate value : ; ; ; ; The parameters of the circumscribed rectangle are: ; in, This represents the width of the bounding rectangle. ; Indicates the height of the bounding rectangle. The circumscribed rectangle defines the precise location of the moving target in the image, providing a region constraint for subsequent target filtering and recognition.

[0052] Step S3: Filtering of non-interested targets.

[0053] To effectively extract the target from step S2, non-interesting targets need to be filtered out. These typically include water ripples, light rays, and image noise. In this application, the aspect ratio of the target is considered... For filtering, the aspect ratio of the target refers to the aspect ratio of the target's bounding rectangle: ; Where T represents whether the target is a valid target, T=1 represents a valid target, and T=0 represents a distracting target; by using the above filtering conditions, distracting targets that are obviously not fish are filtered out, reducing the amount of subsequent calculations.

[0054] Step S4: Accurately classify and identify male and female fish.

[0055] To further accurately identify whether the target extracted in step S3 is a fish and distinguish between male and female fish, this application employs a deep learning-based multi-task classification algorithm. Its main steps include collecting sample data, labeling sample data, multi-task design, model training, and model inference. Specifically, sample data collection mainly refers to saving the image data acquired in step S3 and classifying it according to whether it is a fish, male, or female. Secondly, the sample data annotation assigns a label to each data image. For images that are not fish, the label is [0, 0]; for male fish images, the label is [1, 0]; and for female fish images, the label is [1, 1]. This two-dimensional label data corresponds to the multi-task label, where the first dimension indicates whether it is a fish, and the second dimension indicates whether it is male or female. Sexually mature fish exhibit significant differences in appearance between sexes. Sex can be determined by comprehensively considering the biological appearance characteristics of different fish species before labeling them as male or female. For example, zebrafish, as a model organism for small fish, have males that are slender and bright yellow, while females have rounder bodies and a more bluish, pale color. The sex determination is primarily based on a combination of body shape and color. There are slight differences between different small fish species, and the criteria for sex identification are adjusted according to the specific fish species. The training of the multi-task classification network is based on sample images labeled with the above features, enabling the network to automatically learn and extract these sex-related body color, body shape, and local morphological features, thereby achieving accurate differentiation between male and female fish.

[0056] Secondly, a multi-task design is employed, using a lightweight deep learning classification network. This embodiment employs the following network structure: four convolutional (conv) layers, five pooling (pooling) layers, four ReLU (Rectified Linear Unit) layers, and two fully connected layers. The main function of the conv layers is to extract features from the image that are translation-, rotation-, and scale-invariant, while simultaneously reducing the dimensionality of image features to prevent overfitting and improve the model's generalization ability. The convolutional layers are characterized by local perception, parameter sharing, and multi-kernel convolution. The first conv layer has 96 convolutional kernels, each with a size of 5. 5. With a stride of 2, the second conv layer has 64 convolutional kernels, each with a size of 3. 3. With a stride of 2, the third conv layer has 128 convolutional kernels, each with a size of 3. 3. With a stride of 2, the fourth conv layer has 256 convolutional kernels, each with a size of 3. 3, step size is 2.

[0057] The convolutional kernels in pooling layers only take the maximum or average value at corresponding positions (max pooling, average pooling), and are not modified through backpropagation. The main function of pooling layers is to ignore changes in the relative position of the target, such as tilt or rotation. This improves accuracy, reduces the dimensionality of the feature map, and can avoid overfitting to some extent.

[0058] ReLU layers are linear rectified function layers, commonly used activation functions in deep learning network models. Their main purpose is to increase the non-linearity of the neural network model. The advantages of the ReLU activation function are gradient non-saturation and fast computation speed.

[0059] Next comes model training. The main process of model training involves iteratively training the model on a large amount of sample data and corresponding label data, minimizing the loss function value between the model's predictions and the label data. During training, the loss value generated after one forward pass is used to update the values ​​of all parameters in the model. Through continuous iteration, a transferable learning model is finally generated. The model training uses a multi-task loss function: , , The total loss is .in, This indicates whether the target is a true label for the fish. This represents the predicted probability that the target is a fish; A true label indicating the target sex of fish. This represents the predicted probability that the target fish is a female; the sex classification loss only applies to... Calculation of fish samples with a value of 1; This indicates the total number of samples. This represents the coordinates of the predicted key feature points for the i-th sample. Represents the coordinates of the true key feature points of the i-th sample; , and These represent the weight coefficients for fish classification loss, sex classification loss, and key feature point regression loss, respectively, and their values ​​are determined based on the actual training settings. It should be noted that... , Used only for classification, and in contrast to the pose descriptor calculated based on key feature points in subsequent steps. to There is no direct relationship.

[0060] The training process mainly consists of four steps, which are divided into two phases: Phase 1, Forward Propagation Phase: a) Extract samples (X, Yp) from the sample set and input X into the network; b) Calculate the corresponding actual output Op.

[0061] In this process, the original image data is processed by the network model, undergoing four convolution operations and five pooling operations, followed by two fully connected operations, ultimately outputting the result. Essentially, it performs convolution operations, dimensionality reduction, and data stretching between the convolutional kernels of each layer of the model and the image.

[0062] The second stage, the feedback phase: a) Calculate the difference between the actual prediction result Op and the true label data Yp corresponding to the sample, and calculate the partial derivative of the difference with respect to each convolutional kernel. b) Adjust the model parameters using backpropagation based on the partial derivatives and learning rate.

[0063] Finally, there is model reasoning. Based on the model trained above, the data from the third step is input into the program to determine whether it is a fish. If it is a fish, it is further determined whether it is a male or female fish.

[0064] Step S5, pose recognition.

[0065] Step S4 identifies male and female fish. When recognizing the fish's posture, a vector calculation method based on key points is used. The main steps include extracting key points and calculating the vector relationships between key points. Key point extraction employs a deep learning-based key feature point regression algorithm, and the main steps include data annotation, regression task design, model training, and model inference.

[0066] First, data annotation is performed. In this application, the fish head, dorsal fin, caudal fin, anal fin, and pelvic fin are defined as key feature points. During the data annotation process, the positions of the five key feature points are marked on the target small image of the fish in step S4. The labeled data preprocessing involves normalizing the aforementioned location data, calculated using the original coordinates of the k-th key feature point. , The ratio of the image width to its height, with the data ranging from 0 to 1, in the following format: ;in, and The x and y coordinates (in pixels) of the k-th key feature point are normalized. k represents the index of the key feature point (k=1,2,3,4,5, corresponding to the fish head, dorsal fin, caudal fin, anal fin, and pelvic fin respectively). The normalized coordinate values ​​range from 0 to 1. , W represents the image width (in pixels); H represents the image height (in pixels).

[0067] Secondly, regarding the regression task design, a lightweight deep learning network structure is adopted. This application reuses the network structure from the fourth step of feature extraction, with the following modifications for the regression task: First, the network output is defined as 10-dimensional data according to the data labeling format. Second, the deviation between the network output and the labeled data is defined using the MSE (mean square error) metric. A smaller MSE indicates a closer approximation between the predicted and actual values; otherwise, a larger difference is indicated. The training process involves continuously reducing the MSE and adjusting the network model parameters. Through iterative training, the MSE loss function value decreases and tends to converge. When the MSE loss function value no longer decreases significantly or reaches the preset number of training epochs, model training is complete, and the model parameters at this point are the optimal model parameters obtained through training.

[0068] Model inference uses the trained model described above to extract key feature points from the image in step S4. Obtaining these key feature points provides fundamental information for calculating the fish's pose.

[0069] To obtain the fish's posture, a method based on the coordinate relationship of key feature points was used to calculate the Euclidean distance from the fish's head to the dorsal fin. Euclidean distance from dorsal fin to caudal fin Euclidean distance from caudal fin to anal fin Euclidean distance from anal fin to pelvic fin European distance from pelvic fin to fish head European distance from fish head to tail fin Euclidean distance from dorsal fin to pelvic fin Euclidean distance from dorsal fin to anal fin And the European distance from the anal fin to the fish head Based on the aforementioned distance, key feature point direction, included angle, normalized projection components, and fish body centerline arc length, the attitude descriptor is calculated.

[0070] Attitude calculation is performed based on the aforementioned position vector values. ; In step S5 of this application, eight pose descriptors are defined to quantify the pose information of the fish, as follows: This represents the directional angle between the line connecting the fish's head and tail fin and the horizontal direction, with a value ranging from -180° to 180°. Represents the normalized vertical component of the line connecting the dorsal and pelvic fins; This indicates the direction of the tail fin's bend: -1 indicates a bend to the left, 0 indicates a straight line, and 1 indicates a bend to the right. This indicates the degree of curvature of the fish body after normalization; the larger the value, the greater the degree of curvature. This represents the directional angle between the line connecting the fish's head and tail fin and the vertical direction, with a value ranging from -180° to 180°. This represents the normalized horizontal component of the line connecting the dorsal and pelvic fins. The angle between the line connecting the anal and pelvic fins and the line connecting the pelvic fins and the head of the fish is denoted as 0° to 180°. It represents the ratio of the arc length of the fish's centerline to the straight-line distance from the fish's head to its tail fin. and Not less than 1.

[0071] Based on the above pose descriptors, the fish poses are subdivided into 12 sub-poses, and their correspondence is as follows: Category 1: Head left, tail right state: Left and right straight line state: ∈[-10°, 10°], ∈[-0.1, 0.1] and ∈[0,0.1], the fish body is in a horizontal straight line, with the head on the left and the tail on the right; where, Indicates the direction of the tail fin's bend (range -1~1, -1 bends to the left, 0 is a straight line, 1 bends to the right). This indicates the degree of curvature of the fish body after normalization; a larger value indicates a higher degree of curvature.

[0072] Curve upwards and to the left: ∈[30°, 60°], ∈[-0.6, -0.4], ∈[0.3, 0.6], the fish body bends to the left with its back facing upwards; Bend downwards to the left and right: ∈[-60°, -30°], ∈[0.4, 0.6], ∈[0.3, 0.6], the fish body bends to the left with its belly facing upwards.

[0073] Category Two: Head Right, Tail Left State: Left and right straight line state: ∈[170°, 180°] or [-180°, -170°], ∈[-0.1, 0.1], ∈[0, 0.1], the fish body is in a horizontal straight line, with the head on the right and the tail on the left.

[0074] Curve upwards and to the left: ∈[120°,150°], ∈[-0.6, -0.4], ∈[0.3, 0.6], the fish body bends to the right with its back facing upwards; Bend downwards to the left and right: ∈[-150°, -120°], ∈[0.4, 0.6], ∈[0.3, 0.6], the fish body bends to the right with its belly facing upwards.

[0075] Category 3: Head up, tail down: Vertical straight line state: ∈[-10°, 10°], ∈[0, 0.1], the fish body is in a vertical straight line, with the head at the top and the tail at the bottom; Curve up and down to the left: ∈[30°,60°], ∈[0.4, 0.6], ∈[0.3, 0.6], the fish body bends downwards, with the left side facing outwards; Curve up and down to the right: ∈[-60°, -30°], ∈[-0.6, -0.4], ∈[0.3, 0.6], the fish body bends downwards, with the right side facing outwards.

[0076] Category 4: Head down, tail up: Vertical straight line state: ∈[170°, 180°] or [-180°, -170°], ∈[0, 0.1], the fish body is in a vertical straight line, with the head at the bottom and the tail at the top; Curve up and down to the left: ∈[120°, 150°], ∈[0.4, 0.6], ∈[0.3, 0.6], the fish body is bent upwards, with the left side facing outwards; Curve up and down to the right: ∈[-150°, -120°], ∈[-0.6, -0.4], ∈[0.3, 0.6], the fish body is bent upwards, with the right side facing outwards.

[0077] Posture information was calculated separately for male and female fish.

[0078] Step S6, target trajectory tracking.

[0079] After obtaining the pose information of a single fish, the trajectory information of the two fish, the distance between the two trajectories, and the correlation between them are further calculated. Therefore, real-time target tracking of the two fish is required. This application adopts an optical flow tracking algorithm based on sparse feature points. The main steps of the traditional optical flow tracking algorithm include: processing a continuous sequence of video image frames; detecting foreground targets for each frame; if a foreground target appears in a frame, finding its representative feature points (usually using corner points as feature points); for the next two adjacent video frames, finding the best position of the feature points that appeared in the previous frame in the current frame, thereby obtaining the position information of the foreground target in the current frame; and repeating this iteratively to achieve target tracking.

[0080] In step S6 of this application, an optical flow tracking algorithm based on sparse feature points is used to perform real-time target tracking of the two fish. The specific implementation steps are as follows: Step S6.1: Feature point selection.

[0081] The Shi-Tomasi corner detection algorithm is used to select representative feature points within the foreground target region. This algorithm determines whether a point is a corner by calculating the corner response function R of each pixel in the image, and selects the N points with the largest R values ​​as tracking feature points (N=100 in this embodiment) to ensure that the selected feature points have good tracking stability.

[0082] Step S6.2: Optical flow calculation.

[0083] The Lucas-Kanade sparse optical flow algorithm is used to calculate the displacement of feature points between adjacent frames. Assuming the brightness of feature points remains constant between adjacent frames and the optical flow is similar between adjacent pixels, a system of optical flow equations is established and the least-squares solution is obtained to yield the displacement vector of each feature point. The specific calculation formula is as follows: ; ; in, , These represent the displacement in the x-direction (the speed of the feature point in the horizontal direction (unit: pixels / frame)) and the displacement in the y-direction (the speed of the feature point in the vertical direction (unit: pixels / frame)). , The image gradients are defined as follows: x-axis (the rate of change of grayscale in the horizontal direction, reflecting the intensity of changes in image texture in the x-direction) and y-axis (the rate of change of grayscale in the vertical direction, reflecting the intensity of changes in image texture in the y-direction). The time gradient is the rate of change of grayscale in an image over time, reflecting the brightness change of the same pixel between adjacent frames.

[0084] Step S6.3: Forward and reverse optical flow verification.

[0085] To improve tracking robustness, a forward and reverse optical flow consistency check is adopted. The forward optical flow runs from frame t to frame t+1, and the reverse optical flow runs from frame t+1 to frame t. If the position error of the forward and reverse tracking is less than the threshold (set to 2 pixels in this embodiment), it is determined to be effective tracking; otherwise, it is determined to be tracking failure and the feature points are reinitialized.

[0086] Step S6.4: Track information acquisition.

[0087] The following motion features are obtained from the tracking results of consecutive frames: Instantaneous velocity: ; Instantaneous acceleration: ; Track distance: , =t-N+1,...,t; Trajectory Correlation: ; The swimming trajectory information of the two fish is obtained using the method described above. In this embodiment, the trajectory information of 50 historical frames is retained. The meanings of each letter in the above motion features are as follows: This represents the instantaneous velocity of the currently active moving target in frame t. and These represent the x-coordinate and y-coordinate of the current effective moving target in the image coordinate system at frame t, respectively. and These represent the x-coordinate and y-coordinate of the current effective moving target in the image coordinate system at frame t-1, respectively; Indicates the time interval between two adjacent frames; This represents the instantaneous acceleration of the currently effective moving target in frame t; This represents the instantaneous velocity of the currently effective moving target in frame t-1; This represents the cumulative trajectory distance of the currently valid moving target within the most recent N frames up to frame t; and These represent the current effective moving target in the image coordinate system. The x and y coordinates of the frame; and These represent the current effective moving target in the image coordinate system. -1 frame's x and y coordinates; This indicates the trajectory correlation between the currently effective moving target and its paired fish; This represents the function for calculating the correlation coefficient. This represents the velocity vector sequence of the currently active moving target and its paired male fish; This represents the velocity vector sequence of the currently active moving target and its paired female fish.

[0088] Step S7, courtship behavior recognition.

[0089] Courtship behaviors are identified based on the previously calculated posture and trajectory information. Male courtship behaviors include chasing, tail-nose, encircling, zig-zag swimming, and quivering; female courtship behaviors include approaching, escorting, presenting, leading, and laying eggs.

[0090] In step S7 of this application, the correspondence between the 10 identified courtship behaviors and the posture descriptors and trajectory information is as follows: (1) Male fish courtship behavior (5 categories): The meanings of the symbols in step S7 are consistent with those described above: ~ The meanings are the same as those defined in step S5; the ratio threshold (e.g., 1.5) represents the upper limit of the ratio between the instantaneous velocity of the current effective moving target and the instantaneous velocity of the paired female / male fish; the trajectory correlation threshold (e.g., 0.3, 0.2) represents the upper limit of the correlation coefficient for determining trajectory similarity; the fish body length represents the body length of the current effective moving target; the meanings of the other symbols are as described above.

[0091] Chase: The male fish swims rapidly behind the female. Characteristics: ∈[150°, 180°]; > ×1.5; Rapid increase; manifested by the male fish rapidly approaching the female fish.

[0092] Tail-nose: The male fish's head approaches the female's tail. Characteristics: <2 × fish body length; ∈[170°, 180°]; This is characterized by the male fish closely following the tail of the female fish.

[0093] Encircle: The male fish moves in a circular motion around the female fish. Characteristics: <0.3; It exhibits periodic changes; the trajectory is circular; and it is characterized by the male fish swimming around the female fish.

[0094] Zig-Zag (Figure-Eight Swimming): The male fish swims in a figure-eight pattern. Characteristics: ∈[0.5, 0.8]; <0.2; the trajectory is in the shape of an 8; it is manifested by the male fish making a figure 8 shape display.

[0095] Quiver: The male fish vibrates its body rapidly. Characteristics: Rapid changes (frequency > 5Hz); <0.5 times the normal swimming speed; little change in position; manifested as the male fish rapidly vibrating its body near the female fish.

[0096] (2) Courtship behavior of female fish (5 categories): Approach: The female actively swims towards the male to express her courtship intention. Characteristics: (1) Posture characteristics: The angle between the fish and the direction of the male fish is less than 30°. ∈[-0.1, 0.1] (body level); ∈[-0.1, 0.1] (body basically upright); (2) trajectory characteristics: Decrease rate > 5 mm / s; >0.8× ;A(t) is positive (accelerating the approach); (3) Behavioral performance: the female fish faces the male fish with its head facing the male fish, actively shortens the distance between the two fish, and gradually increases the speed.

[0097] Escort behavior: After accepting courtship, the female swims side by side with the male, who escorts her from the side. Characteristics: (1) Posture characteristics: The difference between D1_escort and D1_male is <15°; ∈[-0.1, 0.1]; ∈[0.3, 0.6] (body slightly bent); (2) trajectory characteristics: <0.1× ;The distance between the two fish is kept at 1-2 times the length of the fish body; the trajectory is parallel; (3) Behavioral performance: The female fish and the male fish swim side by side and synchronously, the two fish maintain a relatively stable distance, and the male fish escorts the female fish from the side or below.

[0098] Presentation behavior: The female fish displays her side or abdomen in front of the male fish, allowing the male fish to examine her. Characteristics: (1) Posture characteristics: ∈[-0.3, 0.3] (body tilted to show side profile); or Significant (fins spread); >0 (abdomen facing male fish); (2) Trajectory characteristics: <0.3× (Slow down display); small position change; trajectory swings in a small range; (3) Behavioral performance: the female fish swims sideways in front of the male fish, showing the side of her body or abdomen, the fins may be spread, and the swimming is slow and elegant.

[0099] Lead (leading behavior): The female actively leads the male to the spawning site (usually a location with suitable substrate). Characteristics: (1) Posture characteristics: Pointing in the direction of the spawning site; ∈[-0.2, 0.2] (body remains horizontal); ∈[-0.1, 0.1] (body basically upright, direction clear); (2) Trajectory characteristics: > ×0.8 (Female fish actively leads); the trajectory is towards the bottom or a specific location; (3) Behavioral behavior: The female fish swims in front and the male fish follows behind, with the two fish maintaining a certain distance. The female fish guides the direction of swimming towards the spawning area.

[0100] Egg lay (spawning behavior): The female lays eggs in a suitable location on the bottom, and the male simultaneously releases sperm for fertilization. Characteristics: (1) Postural characteristics: >0.2 (back up); (1) The bending ratio is greater than the preset bending ratio threshold (the body is in an egg-laying bending posture); the body is close to the bottom; (2) Trajectory characteristics: <0.2× (Extremely slow during spawning); trajectory close to the bottom (y coordinate > 80% of image height); position change < 0.3 times fish length / s; (3) Behavioral performance: the female fish swims slowly in the bottom area with her body in a bent posture, and may use her pectoral fin or caudal fin to dig spawning pits in the substrate before spawning.

[0101] Based on the same inventive concept, this application also provides a small fish courtship behavior recognition system for implementing the aforementioned method for recognizing the courtship behavior of small fish. The solution provided by this system is similar to the implementation described in the above method; therefore, the specific limitations of one or more embodiments of the small fish courtship behavior recognition system provided below can be found in the limitations of the small fish courtship behavior recognition method described above, and will not be repeated here.

[0102] In one exemplary embodiment, such as Figure 2 As shown, a system for recognizing courtship behavior in small fish is provided, comprising: The image acquisition module 201 is used to acquire the current frame image of the target fish in the aquaculture container and extract the region of interest image from the current frame image.

[0103] The moving target extraction module 202 is used to extract moving targets based on the region of interest image by combining background modeling and inter-frame difference to obtain a set of candidate moving targets for the current frame.

[0104] The interference target filtering module 203 is used to filter out interference targets based on the candidate moving target set by using a geometric feature filtering method to obtain the initial moving target set of the current frame.

[0105] The multi-task classification module 204 is used to input each initial moving target in the initial moving target set of the current frame into a pre-trained multi-task deep learning classification network and output the gender category of each initial moving target in the current frame; the gender category includes non-fish, male fish and female fish.

[0106] The effective target filtering module 205 is used to delete the initial moving targets whose gender category is not fish from the initial moving target set, so as to obtain the effective moving target set of the current frame.

[0107] The key point regression module 206 is used to input each effective moving target in the set of effective moving targets in the current frame into a pre-trained deep learning regression network, and output the set of position coordinates of key feature points of each effective moving target in the current frame; the key feature points include the fish head, dorsal fin, caudal fin, anal fin, and pelvic fin.

[0108] The attitude quantization module 207 is used to calculate the distance, direction, angle, normalized projection component and arc-to-direction ratio between key feature points based on the set of position coordinates of key feature points of each effective moving target in the current frame, so as to obtain the attitude descriptor of each effective moving target in the current frame.

[0109] The trajectory tracking module 208 is used to determine the trajectory features of each effective moving target in the current frame using a sparse optical flow tracking method.

[0110] The behavior determination module 209 is used to determine the mating behavior determination results of each valid moving target in the current frame based on the sex category, posture descriptor and trajectory features of each valid moving target in the current frame; the mating behavior determination results include the mating behavior determination results of male fish and female fish; the mating behavior determination results of male fish include chasing, tail chasing, circling, figure-eight swimming or trembling; the mating behavior determination results of female fish include approaching, escorting, displaying, leading or spawning.

[0111] In one exemplary embodiment, a computer device is provided, which may be a server or a terminal, and its internal structure diagram may be as follows. Figure 3As shown, this computer device includes a processor, memory, input / output (I / O) interfaces, and a communication interface. The processor, memory, and I / O interfaces are connected via a system bus, and the communication interface is also connected to the system bus via the I / O interfaces. The processor provides computational and control capabilities. The memory includes non-volatile storage media and internal memory. The non-volatile storage media stores the operating system, computer programs, and a database. The internal memory provides the environment for the operation of the operating system and computer programs stored in the non-volatile storage media. The database stores data related to the recognition of courtship behavior in small fish. The I / O interfaces are used for exchanging information between the processor and external devices. The communication interface is used for communication with external terminals via a network connection. When executed by the processor, the computer program implements a method for recognizing courtship behavior in small fish.

[0112] Those skilled in the art will understand that Figure 3 The structure shown is merely a block diagram of a portion of the structure related to the present application and does not constitute a limitation on the computer equipment to which the present application is applied. Specific computer equipment may include, for example, [the following is a list of possible additional structures]. Figure 3 The embodiments show more or fewer components, combinations of certain components, or different component arrangements. In one exemplary embodiment, a computer device is provided, including a memory and a processor, the memory storing a computer program, which the processor executes to implement the steps in the above-described method embodiments.

[0113] In one exemplary embodiment, a computer-readable storage medium is provided storing a computer program that, when executed by a processor, implements the steps in the above-described method embodiments.

[0114] In one exemplary embodiment, a computer program product is provided, including a computer program that, when executed by a processor, implements the steps in the above-described method embodiments.

[0115] It should be noted that the user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for analysis, data stored, data displayed, etc.) involved in this application are all information and data authorized by the user or fully authorized by all parties, and the collection, use and processing of the relevant data must comply with relevant regulations.

[0116] Those skilled in the art will understand that all or part of the processes in the above embodiments can be implemented by a computer program instructing related hardware. The computer program can be stored in a non-volatile computer-readable storage medium, and when executed, it can include the processes of the embodiments described above. Any references to memory, databases, or other media used in the embodiments provided in this application can include at least one of non-volatile and volatile memory. Non-volatile memory can include read-only memory (ROM), magnetic tape, floppy disk, flash memory, optical memory, high-density embedded non-volatile memory, resistive random access memory (ReRAM), magnetic random access memory (MRAM), ferroelectric random access memory (FRAM), phase change memory (PCM), graphene memory, etc. Volatile memory can include random access memory (RAM) or external cache memory, etc. By way of illustration and not limitation, RAM can take many forms, such as Static Random Access Memory (SRAM) or Dynamic Random Access Memory (DRAM).

[0117] The databases involved in the embodiments provided in this application may include at least one type of relational database and non-relational database. Non-relational databases may include, but are not limited to, blockchain-based distributed databases. The processors involved in the embodiments provided in this application may be general-purpose processors, central processing units, graphics processing units, digital signal processors, programmable logic devices, quantum computing-based data processing logic devices, etc., and are not limited to these.

[0118] The technical features of the above embodiments can be combined in any way. For the sake of brevity, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0119] This document uses specific examples to illustrate the principles and implementation methods of this application. The descriptions of the above embodiments are only for the purpose of helping to understand the methods and core ideas of this application. Furthermore, those skilled in the art will recognize that, based on the ideas of this application, there will be changes in the specific implementation methods and application scope. Therefore, the content of this specification should not be construed as a limitation of this application.

Claims

1. A method for recognizing courtship behavior in small fish, characterized in that, include: Acquire the current frame image of the target fish inside the aquaculture container, and extract the region of interest image from the current frame image; Based on the region of interest image, moving targets are extracted by combining background modeling and inter-frame difference to obtain a set of candidate moving targets for the current frame; Based on the candidate moving target set, interference targets are filtered out using a geometric feature filtering method to obtain the initial moving target set for the current frame; Each initial moving target in the initial moving target set of the current frame is input into a pre-trained multi-task deep learning classification network, which outputs the gender category of each initial moving target in the current frame; the gender category includes non-fish, male fish, and female fish. Remove the initial moving targets whose gender category is not fish from the initial moving target set to obtain the valid moving target set for the current frame; Each valid moving target in the set of valid moving targets in the current frame is input into a pre-trained deep learning regression network, which outputs a set of position coordinates of key feature points of each valid moving target in the current frame; the key feature points include the fish head, dorsal fin, caudal fin, anal fin, and pelvic fin; Based on the set of position coordinates of key feature points of each effective moving target in the current frame, the distance, direction, angle, normalized projection component and arc-to-direction ratio between key feature points are calculated to obtain the attitude descriptor of each effective moving target in the current frame. The trajectory features of each effective moving target in the current frame are determined by sparse optical flow tracking. Based on the sex category, posture descriptor, and trajectory features of each valid moving target in the current frame, the mating behavior determination results of each valid moving target in the current frame are determined; the mating behavior determination results include the mating behavior determination results of male fish and female fish; the mating behavior determination results of male fish include chasing, tail chasing, circling, figure-eight swimming, or trembling; the mating behavior determination results of female fish include approaching, escorting, displaying, leading, or spawning.

2. The method for recognizing courtship behavior in small fish according to claim 1, characterized in that, Based on the region of interest image, moving targets are extracted using a combination of background modeling and inter-frame differencing to obtain a candidate set of moving targets for the current frame, specifically including: The background image is obtained by averaging the pixels of the first preset number of frames preceding the current frame. Calculate the pixel difference between the current frame image and the background image at corresponding positions to obtain the first difference image; Calculate the pixel difference between the corresponding positions of the current frame image and the second preset number of frames before the current frame to obtain the second difference image; The pixel values ​​at corresponding positions in the first difference image and the second difference image are added together to obtain the fused difference image; Based on the fused difference image, a binarized output image is obtained through a preset binarization method. The preset binarization method includes taking any pixel in the fused difference image as the current pixel. If the absolute value of the pixel value of the current pixel is greater than a preset threshold, the current pixel is marked as a foreground pixel. If the absolute value of the pixel value of the current pixel is less than or equal to the preset threshold, the current pixel is marked as a background pixel. Contour extraction is performed on the binarized output image to obtain a contour set; Calculate the contour area of ​​each contour in the contour set, remove contours with a contour area less than a preset area threshold as noise contours, and retain contours with a contour area greater than or equal to the preset area threshold as valid contours. Calculate the minimum x-coordinate, maximum x-coordinate, minimum y-coordinate, and maximum y-coordinate of each of the effective contours; The bounding rectangles of each effective contour are determined based on the minimum x-coordinate, maximum x-coordinate, minimum y-coordinate, and maximum y-coordinate. The image regions defined by each bounding rectangle in the current frame image are then used as candidate moving targets, thus obtaining the candidate moving target set for the current frame.

3. The method for recognizing courtship behavior in small fish according to claim 1, characterized in that, The posture descriptor includes the directional angle between the line connecting the head and caudal fin and the horizontal direction, the normalized vertical component of the line connecting the dorsal and pelvic fins, the direction of caudal fin curvature, the normalized degree of body curvature, the angle between the line connecting the anal and pelvic fins and the line connecting the pelvic fins and the head, the normalized horizontal component of the line connecting the dorsal and pelvic fins, the angle between the line connecting the anal and pelvic fins, and the ratio of the arc length of the body centerline to the straight-line distance from the head to the caudal fin. Based on the set of position coordinates of key feature points of each effective moving target in the current frame, the distance, direction, angle, normalized projection component, and arc-to-direction ratio between the key feature points are calculated to obtain the attitude descriptor of each effective moving target in the current frame, specifically including: The pose descriptors for each valid moving target in the current frame are obtained using the following formula: ; ; ; ; ; ; ; ; in, The directional angle between the line connecting the fish's head and tail fin and the horizontal direction; and These represent the x-coordinate and y-coordinate of the fish head, respectively. and The x and y coordinates represent the caudal fin, respectively. Represents the normalized vertical component of the line connecting the dorsal and pelvic fins; and The x and y coordinates represent the dorsal fin, respectively. and The x and y coordinates represent the pelvic fins, respectively. This represents the Euclidean distance from the dorsal fin to the pelvic fin; Indicates the direction of the tail fin's curvature; This represents the Euclidean distance from the dorsal fin to the caudal fin. This represents the Euclidean distance from the fish's head to its tail fin. This indicates the degree of curvature of the fish body after normalization. This represents the Euclidean distance from the caudal fin to the anal fin. This represents the Euclidean distance from the anal fin to the pelvic fin; The directional angle between the line connecting the fish's head and tail fin and the vertical direction; Represents the normalized horizontal component of the line connecting the dorsal and pelvic fins; This represents the angle between the line connecting the anal and pelvic fins and the line connecting the pelvic fins and the fish's head. This represents the Euclidean distance from the pelvic fin to the fish's head. Indicates the arc length of the fish's centerline; It represents the ratio of the arc length of the fish's centerline to the straight-line distance from the fish's head to its tail fin; This indicates the Euclidean distance from the anal fin to the fish's head.

4. The method for recognizing courtship behavior in small fish according to claim 1, characterized in that, The trajectory features of each effective moving target in the current frame are determined using a sparse optical flow tracking method, specifically including: Within the regions of each effective moving target in the current frame, the corner response value of each pixel is calculated using a corner detection algorithm, and the third preset number of pixels with the largest corner response value are selected as tracking feature points to obtain the set of tracking feature points corresponding to each effective moving target in the current frame. For each set of tracking feature points corresponding to each effective moving target in the current frame, the sparse optical flow tracking method is used to obtain the displacement vector of each tracking feature point in the set of tracking feature points corresponding to each effective moving target from the previous frame to the current frame and the forward tracking position corresponding to the displacement vector. The reverse optical flow from the forward tracking position to the previous frame is calculated to obtain the reverse tracking position from the current frame to the previous frame. The displacement vector whose error between the forward tracking position and the reverse tracking position is less than a preset pixel threshold is determined as the effective displacement vector; The effective displacement vectors of each effective moving target are fused to obtain the overall displacement vector of each effective moving target. Based on the overall displacement vector of each effective moving target, update the trajectory position of each effective moving target in the current frame; Based on the trajectory position of each effective moving target in the current frame and the trajectory position of the fourth preset number of frames before the current frame, the trajectory characteristics of each effective moving target are determined; the trajectory characteristics include instantaneous velocity, instantaneous acceleration, and trajectory distance.

5. The method for recognizing courtship behavior in small fish according to claim 4, characterized in that, Based on the gender category, posture descriptor, and trajectory features of each valid moving target in the current frame, the mating behavior determination result of each valid moving target in the current frame is determined, specifically including: Take any one of the valid moving targets in the current frame as the current valid moving target; If the current valid moving target is male, calculate the spatial distance between the current valid moving target and all valid moving targets of the current frame that are female, and select the valid moving target of the current frame that is female with the closest spatial distance as the female fish that is paired with the current valid moving target. If the current valid moving target's gender category is female fish, then calculate the spatial distance between the current valid moving target and all valid moving targets with the gender category of male fish in the current frame, and select the valid moving target with the closest spatial distance as the male fish corresponding to the current valid moving target; The trajectory features also include the correlation between the trajectory of the current effective moving target and the trajectory of the paired female or paired male fish; If the current valid target is male, the following steps are used to determine the male fish's courtship behavior: If the angle between the line connecting the head and tail fin of the current effective moving target fish and the horizontal direction is within the range of 150 degrees to 180 degrees, and the instantaneous speed of the current effective moving target fish is greater than 1.5 times the instantaneous speed of the paired female fish, and the trajectory distance of the current effective moving target fish meets the preset rapid increase trend condition, then the mating behavior of the male fish of the current effective moving target fish is determined as chasing. If the distance between the current active target and the paired female fish is less than twice the body length of the current active target, and the difference between the angle between the line connecting the head and tail fin of the current active target and the horizontal direction and the angle between the line connecting the head and tail fin of the paired female fish and the horizontal direction is 180 degrees, then the mating behavior of the male fish of the current active target is determined to be tail chasing. If the correlation between the trajectory of the current effective moving target and the trajectory of the paired female fish is less than 0.3, and the instantaneous acceleration of the current effective moving target changes periodically, and the trajectory position sequence of the current effective moving target is circularly distributed, then the mating behavior of the male fish of the current effective moving target is determined as circling. If the curvature of the fish body of the current effective moving target is within the range of 0.5 to 0.8, and the trajectory correlation between the current effective moving target and the paired female fish is less than 0.2, and the trajectory position sequence of the current effective moving target satisfies the preset figure-eight distribution condition, then the mating behavior of the male fish of the current effective moving target is determined to be figure-eight swimming. If the frequency of change in the bending degree of the fish body of the current effective moving target is greater than 5 Hz, and the instantaneous speed of the current effective moving target is less than 0.5 times the preset normal swimming speed, and the position change of the current effective moving target from the previous frame to the current frame is less than the preset position change threshold, then the mating behavior of the male fish of the current effective moving target is determined as trembling. If the current valid target is female, the following steps are used to determine the mating behavior of the female fish: If the angle between the head direction of the current effective moving target and the head direction of the paired male fish is less than 30 degrees, and the absolute value of the vertical component of the line connecting the dorsal and pelvic fins of the current effective moving target is less than 0.1, and the direction of the tail fin bend is within the preset straight state range, and the rate of decrease in the positional distance between the current effective moving target and the paired male fish is greater than 5 mm per second, and the instantaneous speed of the current effective moving target is greater than 0.8 times the preset normal swimming speed, and the instantaneous acceleration of the current effective moving target is positive, then the mating behavior of the female fish of the current effective moving target is judged as close. If the absolute value of the difference between the angle between the line connecting the head and tail fin of the current effective moving target and the horizontal direction and the angle between the line connecting the head and tail fin of the paired male fish and the horizontal direction is less than 15 degrees, and the absolute value of the vertical component of the line connecting the dorsal and pelvic fins is less than 0.1, and the degree of curvature of the fish body is within the range of 0.3 to 0.6, and the absolute value of the difference in instantaneous speed between the current effective moving target and the paired male fish is less than 0.1 times the instantaneous speed of the paired male fish, and the spatial distance between the current effective moving target and the paired male fish is maintained at 1 to 2 times the body length of the current effective moving target, and the trajectories of the current effective moving target and the paired male fish are parallel, then the mating behavior of the female fish of the current effective moving target is determined as escort. If the vertical component of the line connecting the dorsal and pelvic fins of the current effective moving target is within the range of -0.3 to 0.3, the angle between the line connecting the head and tail fins and the vertical direction is greater than the preset vertical angle threshold or the horizontal component of the line connecting the dorsal and pelvic fins is greater than the preset horizontal component threshold, and the angle between the line connecting the anal and pelvic fins is greater than 0, the instantaneous speed of the current effective moving target is less than 0.3 times the preset normal swimming speed, the position change of the current effective moving target from the previous frame to the current frame is less than the preset position change threshold, and the trajectory position of the current effective moving target meets the preset small-range swaying condition, then the result of the female fish courtship behavior of the current effective moving target is to be displayed. If the line connecting the head and tail fin of the current effective moving target points towards the bottom of the breeding container, and the vertical component of the line connecting the dorsal and pelvic fins is within the range of -0.2 to 0.2, and the direction of tail fin curvature is within the preset straight state range, and the instantaneous speed of the current effective moving target is greater than 0.8 times the instantaneous speed of the paired male fish, and the trajectory position of the current effective moving target is towards the bottom of the breeding container, and the spatial distance between the current effective moving target and the paired male fish meets the preset change conditions, then the mating behavior of the female fish of the current effective moving target is determined as leading. If the vertical component of the line connecting the dorsal and pelvic fins of the current effective moving target is greater than 0.2, and the ratio of the arc length of the fish's centerline to the straight-line distance is greater than the preset curvature ratio threshold, the trajectory position of the current effective moving target is close to the bottom of the breeding container, and the instantaneous speed of the current effective moving target is less than 0.2 times the preset normal swimming speed, and the position change of the current effective moving target from the previous frame to the current frame is less than the second preset position change threshold, then the mating behavior of the female fish of the current effective moving target is determined to be spawning.

6. The method for recognizing courtship behavior in small fish according to claim 1, characterized in that, A multi-task deep learning classification network consists of a feature extraction layer and a two-dimensional classification output layer connected in sequence; The feature extraction layer is used to extract fish morphological features based on the initial moving target; The two-dimensional classification output layer is used to determine whether the initial moving target is a fish or a non-fish based on the fish morphological features, and to determine whether the initial moving target is a male or female fish based on the fish morphological features of the initial moving target, thereby outputting the gender category of each initial moving target in the current frame. Deep learning regression networks consist of a feature extraction layer and a regression output layer connected in sequence; The feature extraction layer reuses the feature extraction layer in the multi-task deep learning classification network to extract fish morphological features based on the initial moving target. The regression output layer is used to output the set of position coordinates of key feature points of each effective moving target in the current frame based on fish morphological features.

7. A system for recognizing courtship behavior in small fish, characterized in that, The small fish courtship behavior recognition system is used to implement the small fish courtship behavior recognition method according to any one of claims 1-6, and the small fish courtship behavior recognition system includes: The image acquisition module is used to acquire the current frame image of the target fish in the aquaculture container and extract the region of interest image from the current frame image; The moving target extraction module is used to extract moving targets based on the region of interest image by combining background modeling and inter-frame difference to obtain a set of candidate moving targets for the current frame. The interference target filtering module is used to filter out interference targets based on the candidate moving target set using a geometric feature filtering method, so as to obtain the initial moving target set of the current frame; The multi-task classification module is used to input each initial moving target in the initial moving target set of the current frame into a pre-trained multi-task deep learning classification network, and output the gender category of each initial moving target in the current frame; the gender category includes non-fish, male fish and female fish; The effective target filtering module is used to delete initial moving targets whose gender category is not fish from the initial moving target set, and obtain the effective moving target set of the current frame; The key point regression module is used to input each effective moving target in the set of effective moving targets in the current frame into a pre-trained deep learning regression network, and output the set of position coordinates of key feature points of each effective moving target in the current frame; the key feature points include the fish head, dorsal fin, caudal fin, anal fin, and pelvic fin; The attitude quantization module is used to calculate the distance, direction, angle, normalized projection component and arc-to-direction ratio between key feature points based on the set of position coordinates of key feature points of each effective moving target in the current frame, so as to obtain the attitude descriptor of each effective moving target in the current frame. The trajectory tracking module is used to determine the trajectory features of each effective moving target in the current frame using a sparse optical flow tracking method. The behavior determination module is used to determine the courtship behavior determination results of each valid moving target in the current frame based on the sex category, posture descriptor, and trajectory features of each valid moving target in the current frame. The courtship behavior determination results include the courtship behavior determination results of male fish and female fish. The courtship behavior determination results of male fish include chasing, tail chasing, circling, figure-eight swimming, or trembling. The courtship behavior determination results of female fish include approaching, escorting, displaying, leading, or spawning.

8. A computer device, comprising: A memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that the processor executes the computer program to implement the method for recognizing courtship behavior of small fish according to any one of claims 1-6.

9. A computer-readable storage medium having a computer program stored thereon, characterized in that, When executed by a processor, the computer program implements the method for recognizing the courtship behavior of small fish as described in any one of claims 1-6.

10. A computer program product, comprising a computer program, characterized in that, When executed by a processor, the computer program implements the method for recognizing the courtship behavior of small fish as described in any one of claims 1-6.