Fish identification method and system combining behavior fingerprints and convolutional neural network
By combining behavioral fingerprinting with convolutional neural networks for fish identification, YOLO and DeepSORT are used to generate trajectory sequences and calculate behavioral fingerprint vectors, which are then fused with image features. This solves the problem of insufficient accuracy and robustness in fish identification in underwater monitoring environments, achieving higher identification accuracy and stability.
Patent Information
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2025-11-11
- Publication Date
- 2026-04-07
AI Technical Summary
Existing fish identification methods are easily affected by light interference, occlusion, and posture changes in underwater monitoring environments, leading to a decrease in identification accuracy, especially for similar species and rare fish species, and lacking robustness.
A fish identification method combining behavioral fingerprinting and convolutional neural networks is proposed. Trajectory sequences are generated using YOLO series target detection networks and DeepSORT tracking algorithm, 12-dimensional behavioral fingerprint vectors are calculated, and then fused with image features from convolutional neural networks. Multilayer perceptrons are used to generate weight vectors for channel weighting, and finally, classification is performed through fully connected layers.
It significantly improves the accuracy and stability of fish identification, increases the recall rate of rare fish species, enhances anti-interference capabilities, and makes data management more intelligent, supporting multi-condition retrieval.
Smart Images
Figure CN121811484A_ABST
Abstract
Description
Technical Field
[0001] This invention belongs to the field of computer vision and intelligent monitoring technology, specifically relating to a fish identification method and system that combines behavioral fingerprinting and convolutional neural networks. Background Technology
[0002] Existing fish identification methods generally rely on convolutional neural networks (CNNs) to extract features and classify single-frame images or short sequences. However, in underwater monitoring environments, fish exhibit significant changes in posture, severe light interference and occlusion, and marked differences in swimming speed, which can easily reduce the confidence level of single-frame images.
[0003] The existing methods have the following shortcomings: Difficulty in distinguishing similar species: CNN classification, which relies solely on appearance features, is prone to misclassifying fish species that look similar. Missed detection of rare species: In low-confidence scenarios, CNN outputs tend to favor common species, resulting in poor recognition of rare species. Insufficient robustness: Underwater noise, partial occlusion, and other factors cause instability in CNN classification outputs, leading to significant fluctuations.
[0004] Existing studies have utilized fish tracks for behavior recognition, but they have not fully leveraged track information to assist in fish species classification. Therefore, a new method combining appearance and movement characteristics is needed to improve the accuracy and robustness of fish identification. Summary of the Invention
[0005] The present invention aims to at least partially solve one of the technical problems in the related art.
[0006] Therefore, the first objective of this invention is to propose a fish identification method that combines behavioral fingerprinting with convolutional neural networks.
[0007] The main objective of this invention is to propose a fish identification method and system that combines behavioral fingerprinting and convolutional neural networks. By fusing image features and trajectory features, the accuracy and stability of fish species identification are significantly improved, especially in distinguishing similar species and identifying rare species.
[0008] The second objective of this invention is to propose a fish identification device that combines behavioral fingerprinting with a convolutional neural network.
[0009] To achieve the above objectives, a first aspect of the present invention proposes a fish identification method combining behavioral fingerprinting and convolutional neural networks, comprising: S1, the fish in the video frame is detected and the detection box is obtained by using the YOLO series target detection network. The detection box is then correlated across frames using the DeepSORT tracking algorithm to generate a trajectory sequence containing spatiotemporal coordinates. S2, Calculate a 12-dimensional behavioral fingerprint vector based on the trajectory sequence, which includes features such as average velocity, average acceleration, trajectory curvature, steering angle, dwell time, and population density. S3, input the detection box into the convolutional neural network to extract the image feature vector, map the behavior fingerprint vector through a multilayer perceptron to generate a weight vector, and use the weight vector to perform channel weighting on the image feature vector; S4. The weighted image feature vector is concatenated with the behavior fingerprint vector to form a fused feature vector, which is then input into the fully connected layer for classification and outputs the final category probability distribution.
[0010] In one embodiment of the present invention, S2 includes: S21, When calculating the trajectory curvature, the ratio of the cross product of the coordinate differences of three adjacent frames to the cube of the displacement is used for normalization. S22, Statistical Turning Angle At that time, through calculation The frequency percentage is used as a characteristic of extreme values of the shift.
[0011] In one embodiment of the present invention, S3 includes: S31, the multilayer perceptron contains at least two fully connected layers, each of which uses the ReLU activation function and Dropout regularization; S32, the channel weighting operation uses element-wise multiplication, where the dimension of the weight vector is the same as the number of channels of the image feature vector.
[0012] In one embodiment of the present invention, S4 further includes: S41, before the splicing operation, the behavioral fingerprint vector is subjected to L2 normalization; S42, the fully connected layer contains three hidden layers, with 512, 256 and 128 neurons in each layer, respectively.
[0013] In one embodiment of the present invention, it further includes: S5. When the determination result is that the target fish has passed through and the confidence level exceeds 0.80, a JSON index file containing fish category, behavioral fingerprint summary, uplink and downlink directions and video timestamp range is generated; the JSON index file is stored in the database to support multi-condition joint retrieval based on fish category, time range, direction and behavioral fingerprint features.
[0014] To achieve the above objectives, a second aspect of the present invention provides a fish identification device combining behavioral fingerprinting and a convolutional neural network, comprising: The target detection and trajectory generation module is used to detect fish in video frames and obtain detection boxes through the YOLO series target detection network, and to use the DeepSORT tracking algorithm to perform cross-frame association of the detection boxes to generate a trajectory sequence containing spatiotemporal coordinates. The behavioral fingerprint feature calculation module is used to calculate a 12-dimensional behavioral fingerprint vector based on the trajectory sequence, which includes features such as average velocity, average acceleration, trajectory curvature, turning angle, dwell time, and population density. The image feature extraction and channel weighting module is used to input the detection box into a convolutional neural network to extract image feature vectors, map the behavioral fingerprint vectors through a multilayer perceptron to generate weight vectors, and use the weight vectors to perform channel weighting on the image feature vectors. The feature fusion and classification module is used to concatenate the weighted image feature vector with the behavior fingerprint vector to form a fused feature vector, input it into the fully connected layer for classification, and output the final category probability distribution.
[0015] Beneficial effects of the embodiments of the present invention: Improved recognition accuracy: Compared to simple CNN classification, the CNN + behavioral fingerprint fusion improves recall by 10–15% in rare fish species identification. Enhanced robustness: Behavioral fingerprint features, based on temporal trajectories, can supplement discrimination information even under unstable lighting conditions or partial occlusion of the fish body. Improved robustness: The fusion mechanism suppresses fluctuations in single-frame CNN discrimination, resulting in more stable output. Smarter data management: The JSON index file provides retrieval conditions for subsequent scientific research and regulatory purposes, improving analysis efficiency.
[0016] Additional aspects and advantages of the invention will be set forth in part in the description which follows, and in part will be obvious from the description, or may be learned by practice of the invention. Attached Figure Description
[0017] The above and / or additional aspects and advantages of the present invention will become apparent and readily understood from the following description of the embodiments taken in conjunction with the accompanying drawings, wherein: Figure 1 This is a flowchart of a fish identification method combining behavioral fingerprinting and convolutional neural networks according to an embodiment of the present invention; Figure 2 This is an architecture diagram of a fish identification method combining behavioral fingerprinting and convolutional neural networks according to an embodiment of the present invention; Figure 3 This is a structural diagram of a fish identification device that combines behavioral fingerprinting and convolutional neural networks according to an embodiment of the present invention. Detailed Implementation
[0018] It should be noted that, unless otherwise specified, the embodiments and features described in the present invention can be combined with each other. The present invention will now be described in detail with reference to the accompanying drawings and embodiments.
[0019] To enable those skilled in the art to better understand the present invention, the technical solutions of the present invention will be clearly and completely described below with reference to the accompanying drawings of the embodiments of the present invention. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort should fall within the scope of protection of the present invention.
[0020] The following description, with reference to the accompanying drawings, describes a fish identification method and apparatus that combines behavioral fingerprinting and convolutional neural networks according to an embodiment of the present invention.
[0021] Example 1 Figure 1 This is a flowchart of a fish identification method combining behavioral fingerprinting and convolutional neural networks according to an embodiment of the present invention, as shown below. Figure 1 As shown, it includes: S1. The YOLO series target detection network is used to detect fish in the video frame and obtain detection boxes. The DeepSORT tracking algorithm is used to associate the detection boxes across frames to generate a trajectory sequence containing spatiotemporal coordinates.
[0022] In some implementations, this step uses the YOLO series of object detection networks to detect fish in video frames and combines it with the DeepSORT tracking algorithm to achieve cross-frame correlation, thereby generating a trajectory sequence containing spatiotemporal coordinates. This step is the foundational module of the entire fish identification system, providing crucial spatiotemporal data support for subsequent behavioral fingerprint generation and feature fusion.
[0023] The YOLO (You Only Look Once) network series employs a single-stage detection architecture, dividing the input image into grid cells, with each cell predicting multiple bounding boxes and their class probabilities. Versions such as YOLOv5 or YOLOv8 are preferred in this system due to their good balance between real-time performance and detection accuracy. The detection output is the bounding box coordinates of the fish in each frame. ,in Indicates the coordinates of the center point of the detection box. , The width and height are specified. The detection results are then fed into the DeepSORT tracking algorithm, which combines Kalman filtering and Hungarian matching strategies to achieve cross-frame association of the target. DeepSORT jointly models appearance features and motion states, assigning IDs to the detection boxes output by YOLO to generate continuous trajectory sequences.
[0024] Specifically, trajectory sequence Each trajectory point in The corresponding video number frame, The trajectory length is usually set to... To ensure the statistical validity of behavioral features, DeepSORT's matching threshold is typically set to Mahalanobis distance. The cost matrix of the Hungarian algorithm is composed of a weighted average of the Kalman prediction error and the cosine similarity of the appearance features, with weight coefficients... The typical value is To balance motion and appearance information.
[0025] S2, Calculate a 12-dimensional behavioral fingerprint vector based on the trajectory sequence, which includes features such as average velocity, average acceleration, trajectory curvature, steering angle, dwell time, and population density.
[0026] In some implementations, the behavioral fingerprint generation module of this invention transforms the trajectory sequence of fish in a video into a 12-dimensional quantitative behavioral feature vector. This provides crucial motion feature support for subsequent CNN image feature fusion. The technical implementation of this step is based on the spatiotemporal information of trajectory points, by calculating the average velocity. Average acceleration trajectory curvature Steering angle Duration of stay In addition to population density characteristics, a feature set comprehensively describing fish movement behavior was constructed.
[0027] Furthermore, trajectory sequence Output by the DeepSORT tracking algorithm, where This represents the total number of frames in the trajectory. This is the frame interval (usually 1 / 30 of a second). Speed Acceleration is calculated by dividing the Euclidean distance between the coordinate differences of adjacent frames by the time interval. This is the difference in velocity vectors divided by Trajectory curvature The cross product method is used to calculate the curvature of the trajectory across three consecutive frames. The denominator is the cube of the magnitude of the velocity vector, ensuring the comparability of curvature values across different velocity scales. Steering angle. It indicates the change in velocity direction and is used to assess the turning behavior of fish.
[0028] Specifically, the output of this step includes the average velocity. Speed standard deviation Average acceleration Acceleration standard deviation Mean curvature Standard deviation of curvature directional stability Mean steering angle Extreme value ratio of steering angle Duration of stay Average population density and density fluctuation amplitude These indicators together constitute a 12-dimensional behavioral fingerprint vector. Its dimensions are fixed, but the specific combination of features can be adjusted according to actual needs.
[0029] Furthermore, S2 includes: S21, when calculating the trajectory curvature, the ratio of the cross product of the coordinate differences of three adjacent frames to the cube of the displacement is used for normalization.
[0030] Specifically, in the behavioral fingerprint generation module, the trajectory curvature The calculation is one of the key steps, and its technical implementation is based on the geometric relationship between the difference vectors of coordinate points in three adjacent frames. Specifically, this step calculates the displacement vector between two adjacent frames. and Furthermore, by utilizing the ratio of its cross product to the modulus, a quantitative description of the curvature of the trajectory in a local region can be achieved.
[0031] In terms of technical implementation, curvature The calculation formula is:
[0032] This formula calculates the absolute value of the cross product of two displacement vectors and divides it by the cube of the magnitude of the displacement vectors, thus obtaining the trajectory at the [missing information - likely a specific point or time period]. Local curvature at the frame. The absolute value of the cross product reflects the degree of rotation between the two vectors, while the denominator normalizes the magnitude of the displacement, making the curvature value scale-invariant, suitable for fish behavior analysis at different speeds and trajectory lengths.
[0033] Specifically, trajectory length The number of consecutive frames output by the DeepSORT tracking algorithm is usually determined by the number of frames. Curvature can be effectively calculated. Furthermore, the time interval... It is usually the reciprocal of the video frame rate, for example, in a 30 FPS video. Seconds. Statistical indicators of curvature include mean curvature. and curvature variance It is used to describe the overall curvature and fluctuation of fish movement trajectories.
[0034] S22, Statistical Turning Angle At that time, through calculation Frequency percentage As a characteristic of turning to extreme values.
[0035] In some implementations, the statistical turning angle is... At that time, through calculation Frequency percentage As an extreme value feature in the behavioral fingerprint vector, the turning angle is used to characterize the intensity of turning behavior in fish during swimming. This feature is extracted based on the change in the angle between adjacent velocity vectors in the trajectory sequence, i.e., the turning angle. ,in Indicates the first The direction of fish movement in the frame and The included angle of the axis, measured in radians or degrees.
[0036] Specifically, firstly, the velocity vectors of the trajectory points output by the DeepSORT tracking module are calculated, and then the angle change between adjacent velocity vectors is determined. Subsequently, the steering angles in all frames are iterated. The absolute value is greater than The frequency of, denoted as And calculate its total number of steering angles. The proportion in, that is:
[0037] The range of values for this parameter is: A higher value indicates that the fish frequently exhibits dramatic turning behavior during swimming, which may be related to the movement patterns of a specific species. In practical applications, this feature is suitable for underwater video surveillance scenarios, such as fishways in hydroelectric power stations and fish lifts, to help distinguish fish that are similar in appearance but have significantly different movement behaviors.
[0038] S3, input the detection box into the convolutional neural network to extract the image feature vector, map the behavior fingerprint vector through a multilayer perceptron to generate a weight vector, and use the weight vector to perform channel weighting on the image feature vector.
[0039] In some implementations, the step involves using a behavioral fingerprint vector. Mapped to CNN image feature vectors Channel weight vector This enables dynamic weighted fusion of image features. The process is based on a combination of multilayer perceptron (MLP) and the Softmax function. Its core principle is to utilize the temporal motion features contained in behavioral fingerprints to adaptively adjust the image features extracted by CNN, thereby enhancing the expressive power of key channels and improving classification robustness.
[0040] Specifically, behavioral fingerprint vector ,in Including average speed velocity variance Average acceleration acceleration variance Mean trajectory curvature Curvature variance directional stability Mean steering angle Extreme value ratio of steering angle Duration of stay and population density and density fluctuation Equivalent quantization metrics. This vector undergoes non-linear mapping via an MLP, which typically consists of 2–3 fully connected layers, each using the ReLU activation function. The output dimension is the same as the CNN feature vector. (here) The MLP output is consistent with the standard output. The MLP output is normalized using the Softmax function to generate a weight vector. Its element value is in Within the interval, it represents the attention weight of each channel.
[0041] Furthermore, the image feature vectors extracted by CNN With weight vector Perform element-wise multiplication between them, that is... This process implements channel weighting. This operation effectively suppresses the influence of noisy channels and enhances the expression weights of image channels strongly correlated with behavioral features, thereby improving the classification model's sensitivity to fish behavioral characteristics. In practical applications, this step is commonly used in underwater monitoring scenarios such as fishways in hydropower stations and fish lifts, especially under conditions of unstable lighting, fish occlusion, or significant posture changes, significantly improving recognition stability and recall rate for rare fish species.
[0042] Furthermore, S3 includes: S31, the multilayer perceptron includes at least two fully connected layers, each employing the ReLU activation function and Dropout regularization.
[0043] In some implementations, the multilayer perceptron (MLP) comprises at least two fully connected layers, each employing a ReLU (Rectified Linear Unit) activation function and Dropout regularization to dynamically weight and fuse behavioral fingerprint features with CNN image features. The MLP module, acting as a feature fusion network, functions as a feature fusion network to integrate behavioral fingerprint vectors... Mapped to CNN image feature vectors Dimensionally consistent weight vector This enables channel-level weighting of CNN features, enhancing the expressive power of key features.
[0044] Specifically, the input to the MLP is a 12-dimensional behavioral fingerprint vector. Its structure typically includes two fully connected layers. The first layer optionally has 256 neurons, and the second layer outputs the same signal as the input signal. Same dimensions The output of each layer undergoes a nonlinear transformation using the ReLU activation function, mathematically expressed as follows: This enhances the model's expressive power and alleviates the gradient vanishing problem. Furthermore, to prevent overfitting and improve generalization ability, a Dropout mechanism is introduced after each fully connected layer, with a dropout probability typically set to 0.3–0.5, depending on the size and noise level of the training data.
[0045] S32, the channel weighting operation adopts element-wise multiplication, wherein the dimension of the weight vector is consistent with the number of channels of the image feature vector.
[0046] In some implementations, the channel weighting operation The core purpose of element-wise multiplication is to dynamically weight CNN image features using behavioral fingerprints, thereby enhancing the expressive power of key channels and suppressing the influence of noise or interference channels. Specifically, the weight vector used in this operation... It is composed of behavioral fingerprint vectors After mapping by a multilayer perceptron (MLP), the normalized weight vector generated by the softmax function has the same dimension as the image feature vector. The number of channels is the same, that is ,in , which represents the number of channels in the CNN output features.
[0047] Furthermore, this operation first converts the behavioral fingerprint vector The input is an MLP network, which typically consists of several fully connected layers. Each layer can use ReLU or LeakyReLU activation functions to enhance nonlinear expressiveness. The original weight vector output by the MLP is normalized using a softmax function to ensure that the sum of all channel weights is 1. Subsequently, this weight vector is compared with the CNN feature vector. Perform element-wise multiplication to obtain the weighted eigenvector. This operation is performed independently on each channel dimension, thereby enabling channel-level attention adjustment of CNN features.
[0048] Specifically, the structure of the MLP in this step can be configured according to actual needs, for example, the input layer size can be [missing information]. The output layer size is The intermediate layers can be set to 256 or 512 dimensions to ensure sufficient feature mapping capability. Furthermore, the temperature parameter of the softmax function can be set to the default value of 1, or dynamically adjusted according to the training process to optimize the smoothness of the weight distribution.
[0049] This channel weighting operation is primarily used in the feature fusion stage of fish recognition systems. When image features extracted by a CNN are affected by underwater environmental noise, occlusion, or changes in lighting, the weight vector generated by the behavioral fingerprint features through MLP mapping can adaptively adjust the CNN features, thereby improving the robustness and discriminative ability of the classification model. Especially when identifying fish with similar appearances or rare species, this operation helps to enhance discriminative feature channels and suppress the influence of interfering channels.
[0050] S4. The weighted image feature vector is concatenated with the behavior fingerprint vector to form a fused feature vector, which is then input into the fully connected layer for classification and outputs the final category probability distribution.
[0051] Specifically, this step involves weighting the image feature vector. With behavioral fingerprint vector Perform concatenation to construct a fused feature vector. It is then fed into a fully connected layer (FC) for the final classification output.
[0052] In some implementations, CNNs extract image feature vectors. (here) First, through behavioral fingerprint vectors (here) The generated weight vector Perform channel weighting to obtain ,in This represents an element-wise multiplication operation. This weighted process maps the behavioral fingerprint vector using a multilayer perceptron (MLP), and the output is... Dimensionally consistent weight vector And normalization is performed using the softmax function to ensure that the weights of each channel are within the normal range. Within this scope, enhance the expressive power of key features.
[0053] Specifically, behavioral fingerprint vector Includes 12 standardized temporal motion features, such as average velocity velocity variance Average acceleration acceleration variance Mean curvature Curvature variance directional stability Mean steering angle Extreme value ratio of steering angle Duration of stay and population density characteristics and These indicators are obtained through trajectory sequences. The calculations show that it has a clear physical meaning and is interpretable.
[0054] Furthermore, S4 includes: S41, the behavior fingerprint vector is processed before the splicing operation. Perform L2 normalization.
[0055] Specifically, the L2 normalization operation is performed on the behavioral fingerprint vector. Each sample undergoes a unit vector transformation to ensure it has a uniform magnitude in Euclidean space. Specifically, for any behavioral fingerprint vector... Its normalized form is:
[0056] in, This represents the L2 norm of the vector. This operation is typically performed after the behavioral fingerprint vector is generated and before it is input into the multilayer perceptron (MLP) to ensure that behavioral features are comparable in the subsequent weighted fusion process and to avoid the attention mechanism's over-reliance on certain dimensions due to differences in feature magnitudes.
[0057] Behavioral fingerprint vector Dimensions It is usually 12, which includes the average speed. velocity variance Average acceleration acceleration variance Mean curvature Curvature variance directional stability Mean steering angle Extreme value ratio of steering angle Duration of stay Average population density and density fluctuation amplitude Equivalent quantization index. After L2 normalization, the magnitude of the vector is 1, i.e. This ensures that the distance metric between different samples in the feature space has mathematical consistency.
[0058] L2 normalization helps improve the performance of the attention mechanism in the feature fusion module. This is because behavioral fingerprint vectors... With CNN feature vectors Significant differences exist in both dimension and magnitude; normalization allows behavioral features to generate weight vectors more effectively during the MLP mapping process. This allows for reasonable weighting of CNN features, improving the accuracy and stability of the final classification.
[0059] S42, the fully connected layer contains three hidden layers, with 256, 128 and 64 neurons in each layer, respectively.
[0060] This fully connected layer, as the core module for feature fusion and classification decision-making, is responsible for weighting the CNN image features. Behavioral fingerprint features After concatenation, a non-linear mapping and final classification output are performed. Specifically, the input vector... The dimension is ,in The original dimension of the CNN feature vector. Since the behavioral fingerprint vector has a fixed dimension, the dimension of the concatenated input vector is 524.
[0061] In some implementations, the fully connected layers employ a multilayer perceptron (MLP) structure, with each layer using the ReLU activation function to enhance the model's non-linear expressive power. The first layer contains 256 neurons, used for initial compression of input feature dimensions and extraction of high-level semantic information; the second layer contains 128 neurons, further abstracting and reducing the dimensionality of features; the third layer contains 64 neurons, serving as the final feature representation before classification. The weight matrix of each layer is optimized using the backpropagation algorithm, and the loss function uses cross-entropy loss to maximize the discriminative power of the classification.
[0062] Specifically, the fully connected layer has an input dimension of 524 and an output dimension of the number of categories. Typically, there are 10–20 target fish species. The number of neurons in each layer is set based on a balance between the complexity of feature fusion and the model's generalization ability; a decreasing structure of 256–64 neurons helps to progressively extract key features while avoiding overfitting. Furthermore, the output of the fully connected layers is normalized using a softmax function to obtain the final class probability distribution. .
[0063] This invention provides a fish identification method that combines behavioral fingerprinting and convolutional neural networks. By fusing image features extracted by convolutional neural networks with quantized trajectory behavioral fingerprint features, the accuracy and robustness of fish identification are improved, especially in distinguishing similar species and identifying rare fish species.
[0064] Example 2 The following describes in detail, with reference to the accompanying drawings, a fish identification method combining behavioral fingerprinting and convolutional neural networks according to an embodiment of the present invention.
[0065] This invention proposes a fish identification method that combines behavioral fingerprinting with convolutional neural networks, such as... Figure 2 As shown, the specific steps are as follows: S10, Detection and Tracking Module: The YOLO series of object detection networks were used to detect the fish body and obtain the detection box; The DeepSORT tracking algorithm is used to correlate detection boxes across frames to obtain the trajectory sequence of the fish in the video.
[0066] S20, CNN image feature extraction module: The image of the fish within the detection box is input into a CNN network to extract depth features; The CNN network outputs a preliminary probability distribution of species categories.
[0067] S30, Behavioral fingerprint generation module: Module function: Transforms the temporal trajectory of fish in videos into a quantitative set of behavioral features, which are then combined into a behavioral fingerprint vector to supplement the CNN image features.
[0068] S301, Input: Trajectory sequence from the DeepSORT tracking module:
[0069] Where N is the trajectory length (number of frames).
[0070] S302, Feature Calculation: Average velocity and acceleration: speed: ; Average speed: ; Acceleration: ; Average acceleration: ; Quantitative indicators: .
[0071] Trajectory curvature and directional stability: Curvature:
[0072] Mean curvature and curvature variance ; Directional stability is defined as the consistency of the angle between velocity vectors in adjacent frames:
[0073] (in (The angle between the velocity vector and the x-axis). Quantitative indicators: .
[0074] Steering angle distribution: Steering angle: Statistical distribution: mean steering angle, proportion of extreme values (taken here) Frequency percentage); Quantitative indicators: .
[0075] Length of stay: Defined as the duration for which a trajectory continuously appears within the frame: ; Quantitative indicator: Dwell time .
[0076] Population density changes: The radius of the statistical trajectory point in each frame Number of individuals within ; Calculate the average population density With density fluctuation amplitude Quantitative indicators: .
[0077] S303, Output: The above features are standardized and combined into a fixed-dimensional vector (12-dimensional in this case), denoted as the behavioral fingerprint vector. ; Depending on the species and characteristics of the fish, the fixed-dimensional vector can take different values. Under this approach, non-12-dimensional vectors are also within the scope of protection of this patent.
[0078] S40, Feature Fusion and Auxiliary Judgment Module: Module function: Improves fish classification accuracy by fusing CNN image features and behavioral fingerprint features through an attention mechanism.
[0079] S401, Input: CNN image feature vectors , here Behavioral fingerprint vector , here .
[0080] S402, Fusion Mechanism: Behavioral fingerprint weight generation: Behavioral fingerprint vector Input Multilayer Perceptron (MLP):
[0081] in , which serves as the weight vector for CNN features.
[0082] Feature weighting: Channel-weighted CNN features: ( (This indicates element-wise multiplication).
[0083] Integration and Classification: Weighted CNN features Concatenate the original fingerprint vector with the fingerprint vector to form a new vector containing CNN features and motion features: ; Input: Fully connected layer; Output: Class probability distribution: .
[0084] S403, Training and Output: Two-stage training using the new model: fixed Train the CNN classification head separately until convergence (20–30 epochs); load weights, add them to the fusion network, and train them end-to-end (30–50 epochs). After training, the model can directly output the predicted fish species category.
[0085] S50, Semantic Triggered Recording and Indexing Module: When the determination result is a fish target rather than debris (confidence level exceeding 0.80), the system automatically triggers video recording and saving; Simultaneously, a JSON index file is generated, containing: fish category and confidence level; behavioral fingerprint summary; uplink and downlink directions; and video recording timestamp range. The index file is stored in the database, supporting multi-condition retrieval.
[0086] Example 3 To achieve the above embodiments, such as Figure 3 As shown, this embodiment also provides a fish identification device 10 that combines behavioral fingerprinting and convolutional neural networks. The device 10 includes a target detection and trajectory generation module 100, a behavioral fingerprint feature calculation module 200, an image feature extraction and channel weighting module 300, and a feature fusion and classification module 400.
[0087] The target detection and trajectory generation module 100 is used to detect fish in video frames and obtain detection boxes through the YOLO series target detection network, and to use the DeepSORT tracking algorithm to perform cross-frame association of the detection boxes to generate a trajectory sequence containing spatiotemporal coordinates. The behavioral fingerprint feature calculation module 200 is used to calculate a 12-dimensional behavioral fingerprint vector based on the trajectory sequence, which includes features such as average velocity, average acceleration, trajectory curvature, turning angle, dwell time, and population density. The image feature extraction and channel weighting module 300 is used to input the detection box into a convolutional neural network to extract image feature vectors, map the behavioral fingerprint vectors through a multilayer perceptron to generate weight vectors, and use the weight vectors to perform channel weighting on the image feature vectors. The feature fusion and classification module 400 is used to concatenate the weighted image feature vector with the behavior fingerprint vector to form a fused feature vector, input it into a fully connected layer for classification, and output the final category probability distribution.
[0088] Furthermore, the aforementioned behavioral fingerprint feature calculation module 200 is also used for: When calculating the trajectory curvature, the ratio of the cross product of the coordinate differences of three adjacent frames to the cube of the displacement is used for normalization. Statistical turning angle At that time, through calculation The frequency percentage is used as a characteristic of extreme values of the shift.
[0089] Furthermore, the image feature extraction and channel weighting module 300 described above is also used for: The multilayer perceptron comprises at least two fully connected layers, each employing the ReLU activation function and Dropout regularization. The channel weighting operation employs element-wise multiplication, where the dimension of the weight vector is the same as the number of channels in the image feature vector.
[0090] Furthermore, the aforementioned feature fusion and classification module 400 is also used for: Before the splicing operation, the behavioral fingerprint vector is subjected to L2 normalization. The fully connected layer contains three hidden layers, with 512, 256, and 128 neurons in each layer, respectively.
[0091] Furthermore, the device 10 also includes: The index generation and database storage module is used to generate a JSON index file containing fish category, behavioral fingerprint summary, uplink and downlink directions, and video timestamp range when the determination result is a fish target and the confidence level exceeds 0.80; it is used to store the JSON index file into the database to support multi-condition joint retrieval based on fish category, time range, direction, and behavioral fingerprint features.
[0092] This invention discloses a fish identification device that combines behavioral fingerprinting and convolutional neural networks. By fusing image features extracted by convolutional neural networks with quantized trajectory behavioral fingerprint features, the device improves the accuracy and robustness of fish identification, especially in distinguishing similar species and identifying rare fish species.
[0093] In the description of this specification, the references to terms such as "one embodiment," "some embodiments," "example," "specific example," or "some examples," etc., refer to specific features, structures, materials, or characteristics described in connection with that embodiment or example, which are included in at least one embodiment or example of the present invention. In this specification, the illustrative expressions of the above terms do not necessarily refer to the same embodiment or example. Furthermore, the specific features, structures, materials, or characteristics described may be combined in any suitable manner in one or more embodiments or examples. Moreover, without contradiction, those skilled in the art can combine and integrate the different embodiments or examples described in this specification, as well as the features of different embodiments or examples.
[0094] Furthermore, the terms "first" and "second" are used for descriptive purposes only and should not be construed as indicating or implying relative importance or implicitly specifying the number of technical features indicated. Thus, a feature defined as "first" or "second" may explicitly or implicitly include at least one of that feature. In the description of this invention, "a plurality of" means at least two, such as two, three, etc., unless otherwise explicitly specified.
Claims
1. A fish identification method combining behavioral fingerprinting and convolutional neural networks, characterized in that, include: S1, the fish in the video frame is detected and the detection box is obtained by using the YOLO series target detection network. The detection box is then correlated across frames using the DeepSORT tracking algorithm to generate a trajectory sequence containing spatiotemporal coordinates. S2, Calculate a 12-dimensional behavioral fingerprint vector based on the trajectory sequence, which includes features such as average velocity, average acceleration, trajectory curvature, steering angle, dwell time, and population density. S3, input the detection box into the convolutional neural network to extract the image feature vector, map the behavior fingerprint vector through a multilayer perceptron to generate a weight vector, and use the weight vector to perform channel weighting on the image feature vector; S4. The weighted image feature vector is concatenated with the behavior fingerprint vector to form a fused feature vector, which is then input into the fully connected layer for classification and outputs the final category probability distribution.
2. The method as described in claim 1, characterized in that, The S2 includes: S21, When calculating the trajectory curvature, the ratio of the cross product of the coordinate differences of three adjacent frames to the cube of the displacement is used for normalization. S22, Statistical Turning Angle At that time, through calculation The frequency percentage is used as a characteristic of extreme values of the shift.
3. The method as described in claim 1, characterized in that, The S3 includes: S31, the multilayer perceptron contains at least two fully connected layers, each of which uses the ReLU activation function and Dropout regularization; S32, the channel weighting operation uses element-wise multiplication, where the dimension of the weight vector is the same as the number of channels of the image feature vector.
4. The method as described in claim 1, characterized in that, The S4 further includes: S41, before the splicing operation, the behavioral fingerprint vector is subjected to L2 normalization; S42, the fully connected layer contains three hidden layers, with 512, 256 and 128 neurons in each layer, respectively.
5. The method as described in claim 1, characterized in that, Also includes: S5. When the determination result is that the target fish has passed through and the confidence level exceeds 0.80, a JSON index file containing fish category, behavioral fingerprint summary, uplink and downlink directions and video timestamp range is generated; the JSON index file is stored in the database to support multi-condition joint retrieval based on fish category, time range, direction and behavioral fingerprint features.
6. A fish identification device combining behavioral fingerprinting and convolutional neural networks, characterized in that, include: The target detection and trajectory generation module is used to detect fish in video frames and obtain detection boxes through the YOLO series target detection network, and to use the DeepSORT tracking algorithm to perform cross-frame association of the detection boxes to generate a trajectory sequence containing spatiotemporal coordinates. The behavioral fingerprint feature calculation module is used to calculate a 12-dimensional behavioral fingerprint vector based on the trajectory sequence, which includes features such as average velocity, average acceleration, trajectory curvature, turning angle, dwell time, and population density. The image feature extraction and channel weighting module is used to input the detection box into a convolutional neural network to extract image feature vectors, map the behavioral fingerprint vectors through a multilayer perceptron to generate weight vectors, and use the weight vectors to perform channel weighting on the image feature vectors. The feature fusion and classification module is used to concatenate the weighted image feature vector with the behavior fingerprint vector to form a fused feature vector, input it into the fully connected layer for classification, and output the final category probability distribution.
7. The apparatus as claimed in claim 6, characterized in that, The behavioral fingerprint feature calculation module is also used for: When calculating the trajectory curvature, the ratio of the cross product of the coordinate differences of three adjacent frames to the cube of the displacement is used for normalization. Statistical turning angle At that time, through calculation The frequency percentage is used as a characteristic of extreme values of the shift.
8. The apparatus as claimed in claim 6, characterized in that, The image feature extraction and channel weighting module is also used for: The multilayer perceptron comprises at least two fully connected layers, each employing the ReLU activation function and Dropout regularization. The channel weighting operation employs element-wise multiplication, where the dimension of the weight vector is the same as the number of channels in the image feature vector.
9. The apparatus as claimed in claim 6, characterized in that, The feature fusion and classification module is also used for: Before the splicing operation, the behavioral fingerprint vector is subjected to L2 normalization. The fully connected layer contains three hidden layers, with 512, 256, and 128 neurons in each layer, respectively.
10. The apparatus as claimed in claim 6, characterized in that, Also includes: The index generation and database storage module is used to generate a JSON index file containing fish category, behavioral fingerprint summary, uplink and downlink directions, and video timestamp range when the judgment result is a fish-passing target and the confidence level exceeds 0.
80. This is used to store the JSON index file into the database, supporting multi-condition joint retrieval based on fish category, time range, direction, and behavioral fingerprint features.