A computer vision-based fish fry dynamic technical method and system

Abstract

The application discloses a kind of fish fry dynamic technical method, system based on computer vision, related to image processing technical field, the visible light video data of fish fry group and multispectral image data are synchronously collected;The visible light video data and multispectral image data are fused and analyzed, and individual fry is identified and associated in time sequence;Kinematics features extracted from the visible light video data and spectral reflection features extracted from the multispectral image data are cooperatively encoded, and the encoding result is obtained.The application establishes multi-modal feature vector for each individual, which fuses motion representation and intrinsic physiological spectral information.Even when the individual motion trajectory mutates or is temporarily lost, its unique spectral features can serve as a stable identity auxiliary identifier.Combined with the appearance matching mechanism in multi-target tracking algorithms, the continuity of ID can be effectively maintained, significantly reducing the probability of losing or misidentifying individuals in complex and highly dynamic aquaculture environments.

Description

Technical Field

[0001] This invention relates to the field of image processing technology, specifically to a method and system for dynamic fish fry technology based on computer vision. Background Technology

[0002] In recent years, intelligent aquaculture technologies based on computer vision have developed rapidly. In the field of fish fry counting and sorting, mainstream solutions typically rely on building standardized operating environments to simplify the identification task. For example, by designing slides or pipes with specific tilt angles, fish fry are guided through the detection area in a relatively regular posture and direction. In such controlled scenarios, combining target detection (such as the YOLO series models) with multi-target tracking (such as SORT and DeepSORT algorithms) has enabled high-precision dynamic counting and tracking of fish fry with stable forward trajectories. These methods provide an effective automated approach to solving the problems of low efficiency and large errors associated with traditional manual counting.

[0003] However, in actual large-scale, high-density aquaculture scenarios, the behavior of fish fry exhibits high diversity and uncontrollability, resulting in insufficient generalization ability of the aforementioned methods that rely on environmental constraints. Specifically, this manifests as poor tolerance for identifying unconventional movement behaviors. When individual fish fry exhibit behaviors such as swimming backwards, stagnation, violent struggle in place, or jumping due to environmental stimuli, physical differences, or stress responses, their discontinuous, nonlinear, and highly dynamic movement trajectories easily exceed the prediction range of traditional tracking algorithms' motion models, leading to track interruptions, target loss, or frequent identity (ID) switching. This not only directly causes inaccurate counting results (such as double counting or undercounting) but also makes it impossible to effectively identify and quantify the stress state of the fish fry population and individual abnormal behaviors, limiting the reliable application of intelligent management systems in complex real-world scenarios. Summary of the Invention

[0004] The purpose of this invention is to provide a computer vision-based method and system for dynamic fish fry technology to address the shortcomings of the prior art.

[0005] To achieve the above objectives, the present invention provides the following technical solution: a computer vision-based method for dynamic fish fry monitoring, comprising:

[0006] Simultaneously collect visible light video data and multispectral image data of fish fry populations;

[0007] The visible light video data and multispectral image data are fused and analyzed to identify and temporally correlate individual fish fry;

[0008] The kinematic features extracted from the visible light video data and the spectral reflectance features extracted from the multispectral image data are co-encoded to obtain the encoding result;

[0009] Based on the encoding results, a multimodal feature vector is constructed for each associated individual fish fry;

[0010] Based on the associated individual fish fry and the multimodal feature vectors, a time-seriesd dynamic interaction map of the population is constructed.

[0011] The dynamic interaction graph of the group is input into a pre-trained graph neural network anomaly detection model to calculate the alienation index of group behavior.

[0012] Based on the aforementioned group behavior alienation indicators, abnormal group states are determined, and high-risk individual fish fry causing the abnormalities are located.

[0013] In a preferred embodiment, in the step of simultaneously acquiring visible light video data and multispectral image data of the fish fry population, the data is acquired by simultaneously deployed visible light cameras and multispectral cameras, wherein the operating band of the multispectral camera includes the near-infrared band.

[0014] In a preferred embodiment, the step of fusing and analyzing the visible light video data and multispectral image data to identify and temporally associate individual fish fry specifically includes: identifying and associating individuals based on the visible light video data using target detection and multi-target tracking algorithms, and aligning the multispectral image data with video frames using timestamps.

[0015] In a preferred embodiment, the step of co-encoding the kinematic features extracted from the visible light video data and the spectral reflectance features extracted from the multispectral image data to obtain the encoding result, wherein the co-encoding is specifically an adaptive weighted fusion, including: calculating a behavioral activity index based on the kinematic features, and dynamically adjusting the weight coefficient of the spectral reflectance features during fusion based on the index;

[0016] Specifically, when the behavioral activity index is lower than a preset threshold, the weighting coefficient of the spectral reflectance feature is increased.

[0017] In a preferred embodiment, in the step of constructing a multimodal feature vector for each associated individual fish fry based on the encoding result, the multimodal feature vector is formed by concatenating the kinematic features and the weighted spectral reflectance features.

[0018] In a preferred embodiment, the step of constructing a time-series dynamic interaction graph of the group based on the associated individual fish fry and the multimodal feature vector specifically involves: taking each associated individual fish fry as a node, taking its multimodal feature vector as a node attribute, and establishing edges based on the premise that the spatial Euclidean distance between individuals is less than a preset neighborhood radius, thereby constructing a dynamic graph structure.

[0019] The weight of the edge is calculated based on the Euclidean distance between the two nodes it connects and the cosine similarity between their multimodal feature vectors.

[0020] In a preferred embodiment, in the step of inputting the group dynamic interaction map into a pre-trained graph neural network anomaly detection model to calculate the group behavior alienation index, the graph neural network anomaly detection model is a spatiotemporal graph convolutional network, which is trained in an unsupervised manner using historical normal data, and the group behavior alienation index is the reconstruction error of the model on the current map.

[0021] In a preferred embodiment, the step of determining the abnormal state of the group based on the group behavior alienation index and locating the high-risk individual fish fry causing the abnormality specifically involves: calculating the contribution of each node to the alienation index through the attention weight of the graph neural network anomaly detection model, and locating the individual corresponding to the node with the highest contribution as a high-risk individual.

[0022] This invention also provides a computer vision-based fish fry dynamic technology system, comprising:

[0023] The data acquisition module is used to simultaneously acquire visible light video data and multispectral image data of fish fry populations;

[0024] The individual association module is used to fuse and analyze the visible light video data and multispectral image data, identify and associate individual fish fry in time sequence;

[0025] A collaborative coding module is used to collaboratively code the kinematic features extracted from the visible light video data and the spectral reflectance features extracted from the multispectral image data to obtain the coding result;

[0026] The feature construction module is used to construct a multimodal feature vector for each associated individual fish fry based on the encoding results;

[0027] The graph construction module is used to construct a time-series dynamic interaction graph of the group based on the associated individual fish fry and the multimodal feature vectors.

[0028] The alienation detection module is used to input the dynamic interaction graph of the group into a pre-trained graph neural network anomaly detection model to calculate the alienation index of group behavior.

[0029] The risk location module is used to determine the abnormal state of the group based on the group behavior alienation index and to locate the high-risk individual fish fry that caused the abnormality.

[0030] The technical effects and advantages provided by the present invention in the above technical solution are as follows:

[0031] This invention generates a multimodal feature vector for each fish fry by synchronously acquiring and strictly aligning visible light and multispectral data. This feature vector integrates internally stable spectral information with external motion information. When trajectory prediction fails, this feature vector can utilize the unique spectral reflectance characteristics of the fish fry's body surface as a strong identifier, working in conjunction with the appearance matching mechanism in multi-target tracking algorithms. This effectively maintains the continuity and uniqueness of individual identification in complex aquaculture environments, laying a reliable foundation for subsequent precise behavioral analysis.

[0032] This invention introduces an adaptive weighted fusion mechanism based on quantified behavioral activity indicators. This mechanism sets a specific behavioral activity threshold; when the system determines that a fish fry's activity level is below this threshold, it automatically increases the weight of its spectral features in the fusion encoding. The underlying principle is that low-activity fish fry exhibit convergent movement patterns and weak discriminative power, but changes in body color and surface moisture caused by physiological alterations can produce measurable reflectance differences in specific spectral bands (such as near-infrared). Therefore, this mechanism can intelligently shift the focus of analysis to a more sensitive spectral dimension when behavioral signals are weak, thereby achieving early warning of occult physiological abnormalities and breaking through the reliance of traditional methods on overt behavior.

[0033] This invention constructs a time-series dynamic interaction graph of a population and utilizes a graph neural network anomaly detection model for unsupervised learning and analysis. This method abstracts the population as a dynamic network and quantifies a "population behavior alienation index" by calculating the model's reconstruction error of the graph. This index is highly sensitive to subtle imbalances in the population's collaborative behavior. Furthermore, by analyzing the model's internal mechanisms, the contribution of each individual node to the overall anomaly can be calculated. This allows the invention not only to macroscopically determine the abnormal state of the population but also to trace back and precisely locate the few high-risk individuals causing the anomaly, achieving closed-loop management from global early warning to precise intervention, providing core technical support for refined aquaculture decision-making. Attached Figure Description

[0034] To more clearly illustrate the technical solutions in the embodiments of this application or the prior art, the drawings used in the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments recorded in this invention. For those skilled in the art, other drawings can be obtained based on these drawings.

[0035] Figure 1 This is a flowchart of the method of the present invention.

[0036] Figure 2 This is a system block diagram of the present invention. Detailed Implementation

[0037] To make the objectives, technical solutions, and advantages of the embodiments of the present invention clearer, the technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0038] Example 1, please refer to Figure 1 As shown in this embodiment, a computer vision-based fish fry dynamics technology method includes:

[0039] S1. Simultaneously collect visible light video data and multispectral image data of fish fry populations;

[0040] S2. Perform fusion analysis on the visible light video data and multispectral image data to identify and temporally correlate individual fish fry;

[0041] S3. Co-encode the kinematic features extracted from the visible light video data and the spectral reflectance features extracted from the multispectral image data to obtain the encoding result;

[0042] S4. Based on the encoding results, construct a multimodal feature vector for each associated individual fish fry;

[0043] S5. Based on the associated individual fish fry and the multimodal feature vector, construct a time-seriesd group dynamic interaction map;

[0044] S6. Input the dynamic interaction graph of the group into a pre-trained graph neural network anomaly detection model to calculate the alienation index of group behavior;

[0045] S7. Based on the aforementioned group behavior alienation index, determine the abnormal state of the group and locate the high-risk individual fish fry that caused the abnormality.

[0046] As described in S1-S7 above, this invention generates a multimodal feature vector for each fish fry by synchronously acquiring and strictly aligning visible light and multispectral data. This feature vector integrates internally stable spectral information and external motion information. When trajectory prediction fails, this feature vector can utilize the unique spectral reflectance characteristics of the fish fry's body surface as a strong identifier, working in conjunction with the appearance matching mechanism in multi-target tracking algorithms. This effectively maintains the continuity and uniqueness of individual identifiers in complex aquaculture environments, laying a reliable foundation for subsequent precise behavioral analysis.

[0047] This invention introduces an adaptive weighted fusion mechanism based on quantified behavioral activity indicators. This mechanism sets a specific behavioral activity threshold; when the system determines that a fish fry's activity level is below this threshold, it automatically increases the weight of its spectral features in the fusion encoding. The underlying principle is that low-activity fish fry exhibit convergent movement patterns and weak discriminative power, but changes in body color and surface moisture caused by physiological alterations can produce measurable reflectance differences in specific spectral bands (such as near-infrared). Therefore, this mechanism can intelligently shift the focus of analysis to a more sensitive spectral dimension when behavioral signals are weak, thereby achieving early warning of occult physiological abnormalities and breaking through the reliance of traditional methods on overt behavior.

[0048] This invention constructs a time-series dynamic interaction graph of a population and utilizes a graph neural network anomaly detection model for unsupervised learning and analysis. This method abstracts the population as a dynamic network and quantifies a "population behavior alienation index" by calculating the model's reconstruction error of the graph. This index is highly sensitive to subtle imbalances in the population's collaborative behavior. Furthermore, by analyzing the model's internal mechanisms, the contribution of each individual node to the overall anomaly can be calculated. This allows the invention not only to macroscopically determine the abnormal state of the population but also to trace back and precisely locate the few high-risk individuals causing the anomaly, achieving closed-loop management from global early warning to precise intervention, providing core technical support for refined aquaculture decision-making.

[0049] In one possible implementation, step S1 specifically includes:

[0050] S11, through a hardware synchronization triggering unit, controls a high frame rate visible light industrial camera and a multispectral camera to simultaneously expose and collect images of fish fry groups in the same aquaculture water body.

[0051] Specifically, a hardware synchronization trigger unit (such as a microcontroller or a dedicated synchronizer) generates precise timing pulses to ensure that the exposure start times of the two cameras are strictly aligned, with the error controlled at the microsecond level.

[0052] The multispectral camera operates in at least the 560nm, 680nm, and 760nm near-infrared bands. This band selection is based on the biological characteristics of fish fry: the 560nm band is sensitive to changes in fish fry surface pigments (such as carotenoids), reflecting their body color and vitality; the 680nm band is correlated with the chlorophyll a absorption peak and can be used to indirectly assess feed residue or algae attachment in the water; the 760nm near-infrared band is sensitive to changes in water content and tissue structure, helping to reflect changes in the health status of fish fry due to stress, disease, physiological tension, or tissue edema.

[0053] For example, a rigorously spatially calibrated array of cameras is deployed above a standardized recirculating aquaculture pond. Visible light cameras capture images of the rapid, subtle movements of fish fry at a high frame rate, while multispectral cameras, triggered by a synchronous signal, sequentially capture monochrome images of the three characteristic wavelengths mentioned above.

[0054] It should be noted that, to eliminate the interference of ambient light fluctuations and water surface reflections on image quality, a spectrum-stable and constant-intensity LED diffuse light source can be installed above the aquaculture water body. All cameras must undergo stereo joint calibration before data acquisition to obtain a unified world coordinate system and precise pixel-level mapping, and each frame of data should be appended with a synchronization timestamp accurate to the microsecond level, laying the foundation for subsequent pixel-level alignment of cross-modal data.

[0055] S12 preprocesses the collected raw data to form structured image data that can be used for analysis.

[0056] Specifically, the raw data contains noise and distortion, and preprocessing aims to improve data quality and consistency. For visible light video sequences, grayscale processing is first performed to simplify calculations, followed by histogram equalization to enhance the contrast between the foreground (fish fry) and the background (water body), and a Gaussian-filter-based spatial domain denoising algorithm is used to smooth image imperfections caused by micro-turbulence in the water or camera noise.

[0057] For multispectral image sequences, the preprocessing procedure is more complex. First, dark current correction and flat-field correction are performed to eliminate inherent dark current noise and pixel response inhomogeneity in the sensor. Then, using the joint calibration parameters (such as the homography matrix) obtained in step S11, the monochromatic images captured in all bands (560nm, 680nm, 760nm) are precisely registered onto the imaging plane of the visible light camera using bilinear interpolation. This step ensures that for the same physical point within the aquaculture pond, the pixel coordinates in the visible light image and the multispectral images of each band strictly correspond, achieving the prerequisite for spatial fusion.

[0058] In one possible implementation, step S2 specifically includes:

[0059] S21, perform individual identification and spatiotemporal correlation based on preprocessed visible light video data.

[0060] Specifically, the YOLOv8 object detection algorithm is used to process the video frame by frame, identifying all individual fish fry in each frame and outputting the bounding box coordinates, confidence score, and category for each individual. To eliminate detection noise, a confidence threshold of 0.8 is set to filter out suspected targets with low confidence scores. Subsequently, the DeepSORT multi-object tracking algorithm is used to associate detection boxes across frames. DeepSORT combines motion prediction based on Kalman filtering with metric learning based on deep appearance features (extracted through a lightweight ReID network). When fish fry briefly occlude with each other, the algorithm predicts their possible locations through motion trajectories, and when the target reappears, it recovers the correct ID association by comparing the cosine similarity of appearance features, thereby constructing a continuous and unique temporal motion trajectory for each fish fry.

[0061] S22. Utilize timestamps to achieve precise alignment between multispectral image data and visible light video frames. Specifically, based on the microsecond-level synchronization timestamps appended to each frame of data in step S1, a strict temporal correspondence is established between video frames and multispectral image frames. Since the frame rate of a multispectral camera (e.g., 5fps) is typically lower than the visible light video frame rate (e.g., 25fps) due to filter wheel switching, for video frames without a directly corresponding multispectral frame, the system employs temporal nearest neighbor interpolation or motion-compensated interpolation to assign or generate corresponding multispectral information, ensuring that each frame in the video stream has corresponding spectral data support, and controlling the overall alignment error within the time interval of a single video frame.

[0062] S23. Perform quality verification and screening on the individual data after association and alignment.

[0063] Specifically, the system checks the continuity of each motion trajectory and the completeness of its corresponding multispectral data. Abnormal individual data records with severely broken trajectories (e.g., missing more than 10 frames) or excessively high spectral data loss rates are discarded. Finally, a high-quality set of valid individual data containing both complete motion trajectories and corresponding temporal multispectral information is retained, providing reliable input for subsequent feature extraction and fusion encoding.

[0064] In step S3, specifically, collaborative coding adaptively integrates dynamic motion features and static spectral features based on an individual's real-time behavioral state to form a feature expression that better represents their overall health and behavioral state.

[0065] In one possible implementation, step S3 specifically includes:

[0066] S31, extracting temporal kinematic features from visible light video data.

[0067] Specifically, for individual i at time t, based on its motion trajectory, its displacement sequence within the current timeframe and the previous N frames (e.g., N=5, corresponding to 0.1 seconds) is calculated. Based on this sequence, the magnitude (i.e., velocity) and acceleration scalar of the instantaneous velocity vector are calculated. Furthermore, the motion pattern within this short time window is analyzed, and the rate of change of direction angle (angular velocity), trajectory curvature, and statistical features such as average velocity, velocity variance, and motion linearity are extracted.

[0068] S32 extracts spectral reflectance features from multispectral image data.

[0069] Specifically, based on the bounding box position of individual i at time t, corresponding image patches are cropped from the registered multispectral images of each band. The average intensity value of the pixels within each image patch (corresponding to the 560nm, 680nm, and 760nm bands) is calculated. , , This forms the original spectral reflectance vector. To further enhance the discriminative power of the features for physiological states, a normalized difference spectral index is constructed, for example: =( - ) / ( + +δ), this index is potentially related to the water content and metabolic activity of an organism, where δ is a correction factor to prevent the denominator from being zero, and takes a value of The index ranges from -1 to 1. A higher value indicates that the surface water content of the fish fry is more stable and the physiological state is healthier.

[0070] In one embodiment, the calculation of the behavioral activity index includes: calculating a weighted composite value of the individual's average speed and the degree of speed change based on the individual's movement trajectory at the current moment and within the previous short time window.

[0071] When the behavioral activity index is lower than the preset threshold, the weight coefficient of the spectral reflectance feature is increased to a preset enhancement weight value;

[0072] S33, perform adaptive weighted fusion coding based on behavioral activity assessment;

[0073] S331, First, calculate the behavioral activity index. (t). This index is used to quantify the intensity of an individual's movement at time t;

[0074] First, based on the position sequence of individual i at the current time t and within the previous N frames, (t-∆t), (t-∆t),…, (t), (t), calculate its instantaneous velocity sequence .

[0075] Next, the average velocity within this short time window is calculated. and speed standard deviation Behavioral activity index (t) is defined as the weighted sum of the two:

[0076] ;

[0077] in, and To use the pre-defined normalization constants based on the aquaculture species, This represents the maximum average speed of the fish fry under normal movement conditions. The maximum speed standard deviation under normal movement conditions of this species is obtained by statistically calibrating the movement data of more than 30 groups of healthy fry of this species, ensuring that the behavioral activity index value range is stable within [0, 1]. α and β are weighting coefficients (in this embodiment, α = 0.7, β = 0.3). The range of (t) is [0, 1], and the larger the value, the more active the motion.

[0078] S332. Dynamically adjust feature fusion weights based on activity index;

[0079] The preset activity threshold is =0.2. When the discriminative power of behavioral characteristics (motor) decreases, the contribution of physiological characteristics (spectral) increases;

[0080] when (t)< At this time, an individual is determined to be in a low-activity state, with a simple and indistinguishable movement pattern. However, their spectral characteristics, such as body color and surface moisture, are relatively stable and may contain important physiological information (such as dull body color or changes in moisture due to discomfort). In this case, the fusion weight of their spectral reflectance feature vector is used. Increased from the baseline value of 1.0 to =2.0;

[0081] Conversely (t)≧ At that time, individual movement is rich, and the kinematic characteristics themselves have strong discriminative power, maintaining... =1.0.

[0082] S333, the weighted spectral feature vector * (t) and kinematic eigenvectors (t) are input to a co-coding network with fully connected layers. This co-coding network contains two fully connected layers: the first layer has an output dimension of 16, and the second layer has an output dimension of 8. The activation function is ReLU. Training is performed by minimizing the feature reconstruction error to achieve nonlinear fusion of bimodal features. The network learns the nonlinear interaction between the two types of features and outputs a low-dimensional, dense co-coding vector. (t). This encoding result contains both the original information and the key modal information of the current moment emphasized by the adaptive weights.

[0083] In step S4, specifically, feature construction involves combining the collaborative coding results with key original feature dimensions to form a multimodal feature vector that ultimately represents the instantaneous state of an individual.

[0084] Specifically, based on the cooperative coding vector obtained in step S33 (t), the system retains it as the core representation after fusion. Meanwhile, to avoid losing key information with clear physical meaning, the dynamically weighted spectral principal component features are... * (t) and the most discriminative kinematic features (such as instantaneous velocity and acceleration) are concatenated. Finally, a multimodal feature vector is constructed for each body i at time t. (t)=[ (t); (t); * (t)]. This vector forms the basis of the node attributes in the subsequent graph structure construction.

[0085] In step S5, specifically, graph construction involves abstracting the population into a dynamically evolving complex network, where nodes represent individuals and edges represent potential interactions or influences between individuals, thereby modeling the population dynamics as a spatiotemporal graph sequence.

[0086] In one possible implementation, step S5 specifically includes:

[0087] S51, determine the nodes and node attributes. For each sampling time t, define all successfully tracked individual fish fry as a dynamic graph. The node set of (t) (t). The attributes of each node i are the multimodal feature vectors constructed at time t. (t);

[0088] S52, establish edges based on spatial proximity. Calculate the spatial Euclidean distance between any two nodes i and j in the image at the current time. (t). Set a preset neighborhood radius R. The specific value of R is determined by observing the natural interaction distance experiment of the fry of this aquaculture species. In the experiment, record the active approaching behaviors of more than 100 groups of healthy fry, take the median of the interaction distances as the benchmark, and then multiply by 1.2 times according to the aquaculture density for correction. Finally, determine the value range of 3 - 5 times the body length;

[0089] For example, it is set to 3 to 5 times the average body length of the fry. If (t) < R, it is considered that there is a possibility of interaction between the two individuals at this moment, and thus an undirected edge is established between nodes i and j (t), forming an edge set E(t);

[0090] S53, calculate the edge weight to quantify the connection strength. The weight of the edge is not simply 0 or 1, but a continuous value jointly determined by spatial distance attenuation and node state similarity. The specific calculation formula satisfies: ∝ ( ) * ( ), where is a monotonically decreasing function of the distance , is a monotonically increasing function of the cosine similarity of the node feature vectors and .

[0091] In one embodiment, the weight of the edge is calculated based on the Euclidean distance between the two nodes it connects and the cosine similarity between their multimodal feature vectors. Specifically:

[0092] ∝ exp ∙ , where a, b, c are constants.

[0093] In a preferred implementation of this embodiment, the edge weight is jointly determined by spatial proximity and feature similarity. The specific calculation formula is as follows:

[0094] = exp

[0095] where, is the Euclidean distance, is the scaling parameter; cosine_sim is the cosine similarity calculation function, and the second term linearly maps the range of cosine similarity values ​​from [−1, 1] to [0, 1]. This weighting formula ensures that only individuals that are spatially close and have similar behaviors or physiological states have strong connections, which is more consistent with biological observations of cooperative behavior in fish fry populations.

[0096] S54, by iteratively executing S51-S53, constructs a dynamic graph G(t) for each consecutive time sampling point, thus obtaining a time-seriesd population dynamic interaction graph sequence { (t1), (t2), ..., (tn)}, this sequence fully depicts the spatiotemporal evolution of group structure and individual state.

[0097] In step S6, specifically, alienation detection uses a deep learning model to learn patterns from normal group interaction patterns and quantifies the degree to which the current state deviates from the normal pattern.

[0098] In one embodiment, the group behavior alienation index is the error of the graph neural network anomaly detection model in reconstructing node features of the input group dynamic interaction graph, specifically the average normalized mean square error between the original feature vector and the reconstructed feature vector of all nodes.

[0099] In one possible implementation, step S6 specifically includes:

[0100] S61, Model Training. A spatiotemporal graph convolutional network, such as a graph autoencoder, is used as the architecture for the graph neural network anomaly detection model. The model is trained using unsupervised learning, with training data consisting of historical normal population time-series graph sequences collected and constructed under long-term stable and healthy aquaculture conditions. The training objective is to enable the model to efficiently encode features of these normal geographic patterns and accurately reconstruct them, thereby internalizing normal population dynamics into the model's parameters.

[0101] S62, Anomaly Calculation. During the online monitoring phase, the real-time dynamic interaction graph G(t) of the current group is input into the pre-trained model described above. The model (encoder-decoder structure) first encodes G(t) into a low-dimensional latent representation, and then the decoder attempts to reconstruct a graph Ĝ(t) from this representation.

[0102] The group behavior alienation index E(t) is defined as the error in this reconstruction. The model applies the input map... Reconstruction error of (t) (t) is defined as follows:

[0103] ;

[0104] in, The total number of nodes in the current graph. and These are the original and reconstructed multimodal feature vectors of node i, respectively. For a small constant to prevent division by zero (e.g.) This is used for numerical stability. (t) is the alienation index of group behavior, which quantifies the degree of deviation between the current group state and the normal pattern in memory.

[0105] In step S7, specifically, risk positioning involves tracing the source of the anomaly based on the determination of group anomalies, and accurately identifying high-risk individuals within the group.

[0106] In one possible implementation, step S7 specifically includes:

[0107] S71, Abnormal State Determination. Set a dynamic abnormal determination threshold. This threshold is a heterogeneity index calculated based on historical normal data. The statistical distribution of} is used to determine this. For example, the μ+3σ principle is adopted, where μ is the arithmetic mean of the reconstruction error of historical normal data, and σ is the standard deviation. This is calculated by collecting healthy fish fry population data for 30 consecutive days, 12 hours per day, ensuring that 99.7% of the normal state data falls within the range of [μ-3σ, μ+3σ], thus reducing false alarms. If the alienation index E(t) at the current moment > If this is detected, the fish fry population is determined to be in an abnormal state, and an early warning is generated.

[0108] S72, Calculate the contribution of node-level anomalies. To locate the source of anomalies, it is necessary to analyze the contribution of each node (individual) in the map to the overall reconstruction error E(t).

[0109] This embodiment achieves this by utilizing the attention mechanism within a pre-trained graph neural network model. The decoder reconstructs the features of each node. At that time, a set of attention weights will be generated. , This represents the node attention weight vector of the spatiotemporal graph convolutional network decoder. Its dimension is the same as the latent representation dimension, and it is normalized using the softmax function. Higher weight values ​​indicate a greater impact of the node's features on global graph reconstruction. This weight reflects which global information from the latent representation is prioritized for reconstructing this node. The anomaly contribution of node i... (t) is determined by its own reconstruction error and the norm of the attention weights:

[0110] .

[0111] S73 identifies high-risk individuals. All nodes are ranked according to their contribution. Sort the data in descending order. Identify the individual fish fry corresponding to the top-ranking nodes in terms of contribution or nodes whose contribution exceeds a preset contribution threshold as high-risk individuals for this population anomaly.

[0112] The S74 system outputs a comprehensive analysis report that includes abnormal time points, alienation index values, high-risk individual IDs, and their locations in the real-time video stream (usually marked with a highlighted box), providing farmers with intuitive and accurate intervention targets and decision-making basis.

[0113] In one embodiment, the process of locating high-risk individual fish fry that cause anomalies includes: calculating the contribution of each node in the graph neural network anomaly detection model to the reconstruction error, and locating the individuals corresponding to the nodes with the highest contribution or those exceeding a contribution threshold as high-risk individuals.

[0114] Example 2, please refer to Figure 2 As shown in this embodiment, a fish fry dynamic technology system based on computer vision includes: a data acquisition module, used to simultaneously acquire visible light video data and multispectral image data of fish fry populations;

[0115] The individual association module is used to fuse and analyze the visible light video data and multispectral image data, identify and associate individual fish fry in time sequence;

[0116] A collaborative coding module is used to collaboratively code the kinematic features extracted from the visible light video data and the spectral reflectance features extracted from the multispectral image data to obtain the coding result;

[0117] The feature construction module is used to construct a multimodal feature vector for each associated individual fish fry based on the encoding results;

[0118] The graph construction module is used to construct a time-series dynamic interaction graph of the group based on the associated individual fish fry and the multimodal feature vectors.

[0119] The alienation detection module is used to input the dynamic interaction graph of the group into a pre-trained graph neural network anomaly detection model to calculate the alienation index of group behavior.

[0120] The risk location module is used to determine the abnormal state of the group based on the group behavior alienation index and to locate the high-risk individual fish fry that caused the abnormality.

[0121] The output of the data acquisition module is connected to the input of the individual association module. The individual association module, collaborative coding module, feature construction module, map construction module, alienation detection module, and risk positioning module are connected in series. Each module transmits time-series data through a standardized data interface to ensure the continuity and synchronization of data flow.

[0122] The above description is merely a specific embodiment of this application, but the scope of protection of this application is not limited thereto. Any variations or substitutions that can be easily conceived by those skilled in the art within the scope of the technology disclosed in this application should be included within the scope of protection of this application. Therefore, the scope of protection of this application should be determined by the scope of the claims.

Claims

1. A computer vision-based method for dynamic fish fry monitoring, characterized in that, include: Simultaneously collect visible light video data and multispectral image data of fish fry populations; The visible light video data and multispectral image data are fused and analyzed to identify and temporally correlate individual fish fry; The kinematic features extracted from the visible light video data and the spectral reflectance features extracted from the multispectral image data are co-encoded to obtain the encoding result; The collaborative coding is specifically an adaptive weighted fusion, which includes: calculating a behavioral activity index based on the kinematic features, and dynamically adjusting the weight coefficient of the spectral reflectance features during fusion based on the index; Specifically, when the behavioral activity index is lower than a preset threshold, the weighting coefficient of the spectral reflectance feature is increased; Based on the encoding results, a multimodal feature vector is constructed for each associated individual fish fry; Based on the associated individual fish fry and the multimodal feature vectors, a time-seriesd dynamic interaction map of the population is constructed. The dynamic interaction graph of the group is input into a pre-trained graph neural network anomaly detection model to calculate the alienation index of group behavior; the graph neural network anomaly detection model is a spatiotemporal graph convolutional network, which is trained in an unsupervised manner using historical normal data, and the alienation index of group behavior is the reconstruction error of the model on the current graph; Based on the aforementioned group behavior alienation indicators, abnormal group states are determined, and high-risk individual fish fry causing the abnormalities are located.

2. The method for dynamic fish fry monitoring based on computer vision according to claim 1, characterized in that: In the step of simultaneously collecting visible light video data and multispectral image data of the fish fry population, the data is collected by simultaneously deployed visible light cameras and multispectral cameras, wherein the operating band of the multispectral camera includes the near-infrared band.

3. The method for dynamic fish fry monitoring based on computer vision according to claim 1, characterized in that: In the step of fusing and analyzing the visible light video data and multispectral image data to identify and temporally associate individual fish fry, the fusing and analysis specifically includes: identifying and associating individuals based on the visible light video data using target detection and multi-target tracking algorithms, and aligning the multispectral image data with video frames using timestamps.

4. The method for dynamic fish fry monitoring based on computer vision according to claim 1, characterized in that: In the step of constructing a multimodal feature vector for each associated individual fish fry based on the encoding result, the multimodal feature vector is formed by concatenating the kinematic features and the weighted spectral reflectance features.

5. The method for dynamic fish fry monitoring based on computer vision according to claim 1, characterized in that: In the step of constructing a time-series dynamic interaction graph of the group based on the associated individual fish fry and the multimodal feature vector, the construction of the time-series dynamic interaction graph of the group specifically involves: taking each associated individual fish fry as a node, taking its multimodal feature vector as a node attribute, and establishing edges based on the premise that the spatial Euclidean distance between individuals is less than a preset neighborhood radius, thereby constructing a dynamic graph structure. The weight of the edge is calculated based on the Euclidean distance between the two nodes it connects and the cosine similarity between their multimodal feature vectors.

6. The method for dynamic fish fry technology based on computer vision according to claim 1, characterized in that: In the step of determining the abnormal state of the group based on the group behavior alienation index and locating the high-risk individual fish fry that caused the abnormality, the specific step of locating the high-risk individual fish fry is as follows: by using the attention weight of the graph neural network anomaly detection model, the contribution of each node to the alienation index is calculated, and the individual corresponding to the node with the highest contribution is located as the high-risk individual.

7. A computer vision-based fish fry dynamics technology system, used to implement the computer vision-based fish fry dynamics technology method according to any one of claims 1-6, characterized in that, include: The data acquisition module is used to simultaneously acquire visible light video data and multispectral image data of fish fry populations; The individual association module is used to fuse and analyze the visible light video data and multispectral image data, identify and associate individual fish fry in time sequence; A collaborative coding module is used to collaboratively code the kinematic features extracted from the visible light video data and the spectral reflectance features extracted from the multispectral image data to obtain the coding result; The feature construction module is used to construct a multimodal feature vector for each associated individual fish fry based on the encoding results; The graph construction module is used to construct a time-series dynamic interaction graph of the group based on the associated individual fish fry and the multimodal feature vectors. The alienation detection module is used to input the dynamic interaction graph of the group into a pre-trained graph neural network anomaly detection model to calculate the alienation index of group behavior. The risk location module is used to determine the abnormal state of the group based on the group behavior alienation index and to locate the high-risk individual fish fry that caused the abnormality.

Images