A method, system, device and medium for processing images of fish passing through a fishway

Abstract

This invention discloses a method, system, device, and medium for processing fish passage images in a fishway, relating to the field of data processing. The method includes: acquiring initial monitoring images of a target fishway in real time; inputting the initial monitoring images into a target detection model and outputting detection information; cropping the target fishway image recording based on the detection information and set recording stop conditions to obtain a target monitoring image; extracting feature vectors from the target monitoring image using a deep convolutional neural network and determining the correlation between adjacent frames in the target monitoring image based on the feature vectors; determining uplink and downlink information based on the detection information and the correlation, and determining the number of fish passing through the target fishway based on the uplink and downlink information. This invention enables real-time processing of fish passage images, improving processing efficiency and operating speed.

Description

Technical Field

[0001] This invention relates to the field of data processing, and in particular to a method, system, device, and medium for processing images of fish passing through a fishway. Background Technology

[0002] Monitoring migratory fish during the operation of fishways is a crucial step in evaluating their effectiveness. The more indicators covered by the monitoring, the more comprehensive the data, and the better it is for assessing the fishway's passage efficiency.

[0003] Optical video monitoring refers to the method of acquiring video images by capturing them in the monitoring area using an optical camera, and then analyzing and processing the video images to obtain indicators such as the species and quantity of fish passing through the fishway. Depending on the monitoring needs, the camera can be deployed inside the fishway or outside the observation window. Considering the low light levels underwater, auxiliary equipment such as supplementary lights and fish guides are usually required to create a narrow, bright cross-section for the camera. Optical video monitoring allows for long-term, continuous monitoring of fishways, with indicators including the species and quantity of fish passing through. These indicators can be used to analyze the total annual fish passage and the annual distribution of fish passing through the fishway. Optical video monitoring requires manual review of fish passage images or the development of supporting monitoring software to count the fish that successfully pass through the fishway. Because the time periods for fish to pass through the video monitoring area are scattered and discontinuous, manual review of images is time-consuming and inefficient. Obtaining information such as the species, quantity, and swimming direction of fish passing through the fishway through manual monitoring is difficult to implement and cannot achieve all-weather monitoring. In addition, most of the software developed to support optical video monitoring methods uses background segmentation and motion monitoring, which means that fish are counted by capturing moving objects in the video. This method is prone to mistaking floating objects such as aquatic plants and garbage for fish, and it is difficult to identify the species of fish.

[0004] In recent years, the rapid development of deep learning technology has brought new opportunities for the development of fish passage monitoring technology. Researchers can acquire images of fish passages through photography and video recording and use image processing techniques such as image classification, target detection, and target tracking to analyze the fish passage situation within the fish passages. Compared with traditional methods, fish passage monitoring using computer vision technology has advantages such as high efficiency, non-invasiveness, and repeatable monitoring.

[0005] Currently, researchers mostly use offline processing methods when processing fish passage images in fishways using object detection and tracking algorithms. This involves acquiring the video images of fish passages and then processing them using the corresponding algorithms. This post-processing method struggles to obtain real-time fish passage information. Because the passage of migratory fish through fishways is discrete and non-uniform, sometimes no fish pass through for several hours. Analyzing fish passage images frame-by-frame using object detection and tracking algorithms throughout the entire timeframe would waste server computing resources. When a large number of fish pass through at the same time, it can easily overload the server, leading to crashes and accelerated equipment aging.

[0006] Therefore, it is essential to further optimize the processing of fish passage images by target detection and tracking algorithms based on deep learning technology, improve the processing efficiency and running speed of the algorithms, and achieve real-time acquisition of fish passage data. Summary of the Invention

[0007] The purpose of this invention is to provide a method, system, device, and medium for processing images of fish passing through fish passages, which can process images of fish passing through in real time, thereby improving processing efficiency and operating speed.

[0008] To achieve the above objectives, the present invention provides the following solution:

[0009] A method for processing images of fish passing through a fishway, the method comprising:

[0010] Acquire initial monitoring images of the target fishway in real time;

[0011] The initial monitoring image is input into the target detection model, which outputs detection information. The detection information includes fish information and time information. The fish information includes fish type and location coordinates. The time information includes the time when fish appear in the target fishway. The target detection model is constructed using machine learning methods based on a target detection algorithm.

[0012] Based on the detection information and the set recording stop condition, the target fishway is image recorded and edited to obtain the target monitoring image; the set recording stop condition is that no fish information is detected in the target fishway within a set time interval.

[0013] A deep convolutional neural network is used to extract feature vectors from the target monitoring image, and the correlation between adjacent frames in the target monitoring image is determined based on the feature vectors; the correlation includes similarity and motion information; the feature vectors are used to characterize the detection information of the target monitoring image;

[0014] Based on the detection information and the correlation, uplink and downlink information are determined based on a set baseline, and the number of fish that have passed through is determined based on the uplink and downlink information.

[0015] Optionally, the method for determining the target detection model specifically includes:

[0016] Acquire training data; the training data includes: initial training monitoring images and label data of multiple training fish passages; the label data includes: detection information corresponding to the initial training monitoring images;

[0017] The training data is divided into a training set and a validation set;

[0018] Construct a neural network based on object detection algorithms;

[0019] The training set and corresponding label data are input into the neural network, and the parameters of the neural network are trained with the goal of minimizing the loss function to obtain the trained neural network; the loss function is determined based on the error between the output of the neural network and the label data;

[0020] The parameters of the trained neural network are adjusted using the validation set and the corresponding label data to obtain the adjusted neural network.

[0021] The adjusted neural network is determined as the target detection model.

[0022] Optionally, determining the correlation between adjacent frames in the target monitoring image based on the feature vector specifically includes:

[0023] Based on the position information of the bounding boxes between adjacent frames in the target monitoring image, the Mahalanobis distance between adjacent frames in the target monitoring image is calculated;

[0024] The Mahalanobis distance is compared with a set threshold. If the Mahalanobis distance is less than the set threshold, the association is successful and the motion information is obtained.

[0025] Based on the feature vector, calculate the cosine feature distance between adjacent frames in the target monitoring image;

[0026] The similarity is determined based on the cosine feature distance.

[0027] Optionally, the functional expression for the Mahalanobis distance is:

[0028]

[0029] Where, d (1) (i,j) is the Mahalanobis distance; d j y represents the bounding box position information for the j-th frame; i S represents the bounding box position information for the i-th frame; i Let T be the covariance matrix; T is the transpose matrix.

[0030] Optionally, the formula for calculating the cosine characteristic distance is:

[0031]

[0032] in, r is the cosine feature distance; j Let j be the feature vector of the j-th frame; R is the feature vector of the i-th frame that was successfully associated k times; T is the transpose matrix; R i It is the set of feature vectors that were successfully associated k times.

[0033] Optionally, based on the detection information and the correlation, uplink and downlink information are determined based on a set baseline, and the number of fish that have passed is determined based on the uplink and downlink information, specifically including:

[0034] Using the Kalman filter method, state fusion is performed based on the detection information and the correlation relationship to obtain updated state information;

[0035] Based on the established baseline, uplink and downlink information are determined according to the detection information.

[0036] The number of fish that have crossed the line is determined by using the uplink and downlink information as a method of counting the number of fish that have crossed the line.

[0037] A fishway image processing system, the system comprising:

[0038] The image acquisition module is used to acquire initial monitoring images of the target fishway in real time;

[0039] The output module is used to input the initial monitoring image into the target detection model and output detection information; the detection information includes: fish information and time information; the fish information includes: category and location coordinates; the time information includes: the time when fish appeared in the target fishway; the target detection model is constructed based on the target detection algorithm and using machine learning methods;

[0040] The target monitoring image determination module is used to perform image recording and editing on the target fishway according to the detection information and the set recording stop condition to obtain the target monitoring image; the set recording stop condition is that no fish information is detected in the target fishway within a set time interval.

[0041] The association determination module is used to extract feature vectors from the target monitoring image using a deep convolutional neural network, and determine the association relationship between adjacent frames in the target monitoring image based on the feature vectors; the association relationship includes: similarity and motion information; the feature vectors are used to characterize the detection information of the target monitoring image;

[0042] The fish passage number determination module is used to determine the uplink and downlink information based on the detection information and the correlation relationship, and to determine the fish passage number based on the uplink and downlink information.

[0043] An electronic device includes a memory and a processor, the memory storing a computer program, and the processor running the computer program to enable the electronic device to perform the fish passage image processing method described above.

[0044] A computer-readable storage medium storing a computer program that, when executed by a processor, implements the fish passage image processing method described above.

[0045] According to specific embodiments provided by the present invention, the present invention discloses the following technical effects:

[0046] This invention provides a method, system, device, and medium for processing fish passage images in fishways. It acquires initial monitoring images of the target fishway in real time, inputs them into a target detection model, and outputs detection information. The target detection model is constructed using a target detection algorithm and machine learning methods. Based on the detection information and set recording stop conditions, the target fishway image is recorded and edited to obtain the target monitoring image. A deep convolutional neural network is used to extract feature vectors from the target monitoring image, and the correlation between adjacent frames in the target monitoring image is determined based on the feature vectors. Based on the detection information and the correlation, uplink and downlink information is determined based on a set baseline, and the number of fish passing through is determined based on the uplink and downlink information. By combining the target detection model and the deep convolutional neural network, the fish passage images can be processed in real time, improving processing efficiency and operating speed. Attached Figure Description

[0047] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the drawings used in the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0048] Figure 1 A flowchart of a fish passage image processing method provided in an embodiment of the present invention;

[0049] Figure 2 This is a schematic diagram of the process of using the YOLOv5 model for prediction provided in an embodiment of the present invention;

[0050] Figure 3 A flowchart for fish species identification and counting in fish passages provided in an embodiment of the present invention;

[0051] Figure 4This is a schematic diagram of the processing simulation of the AI ​​system provided in an embodiment of the present invention;

[0052] Figure 5 A flowchart of the AI ​​system provided in an embodiment of the present invention;

[0053] Figure 6 This is a structural diagram of the fish passage image processing system provided in an embodiment of the present invention.

[0054] Symbol explanation:

[0055] Image acquisition module-1, output module-2, target monitoring image determination module-3, correlation determination module-4, fish passage number determination module-5. Detailed Implementation

[0056] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0057] The purpose of this invention is to provide a method, system, device, and medium for processing images of fish passing through fish passages, which can process images of fish passing through in real time, thereby improving processing efficiency and operating speed.

[0058] To make the above-mentioned objects, features and advantages of the present invention more apparent and understandable, the present invention will be further described in detail below with reference to the accompanying drawings and specific embodiments.

[0059] Currently, in fish passage monitoring, researchers have used target detection and tracking algorithms to process fish passage images. However, most of these methods directly input the raw video images into these algorithms without initial screening to locate images showing fish passing through. This approach requires significant computational resources, demands high-end hardware, and primarily outputs textual information, making it difficult to output images showing fish passage. Furthermore, these methods do not effectively combine target detection and tracking algorithms, failing to fully leverage the advantages of optical image detection.

[0060] To address the problems of long monitoring video time series, random and discrete fish-spotting times, large computational costs, and untimely detection in current fishway monitoring, especially those based on optical video images, this invention proposes a deep learning-based discretized fishway image processing method and system. This method processes fishway images in real time, retaining only images of fish passing through and outputting information such as the type and quantity of fish.

[0061] Example 1

[0062] like Figure 1 As shown, this embodiment of the invention provides a method for processing images of fish passing through a fishway, the method comprising:

[0063] Step 100: Acquire initial monitoring images of the target fishway in real time.

[0064] Step 200: Input the initial monitoring image into the target detection model and output detection information; the detection information includes: fish information and time information; the fish information includes: category and location coordinates; the time information includes: the time when fish appeared in the target fishway; the target detection model is built based on the target detection algorithm and uses machine learning methods.

[0065] Step 300: Based on the detection information and the set recording stop condition, the target fishway is recorded and edited to obtain the target monitoring image; the recording stop condition is set to no fish information is detected within a set time interval in the target fishway.

[0066] Step 400: Use a deep convolutional neural network to extract feature vectors from the target monitoring image, and determine the correlation between adjacent frames in the target monitoring image based on the feature vectors; the correlation includes: similarity and motion information; the feature vectors are used to characterize the detection information of the target monitoring image.

[0067] Specifically, determining the correlation between adjacent frames in a target monitoring image based on feature vectors includes:

[0068] Based on the position information of the bounding boxes between adjacent frames in the target monitoring image, the Mahalanobis distance between adjacent frames in the target monitoring image is calculated.

[0069] The Mahalanobis distance is compared with a set threshold. If the Mahalanobis distance is less than the set threshold, the association is successful and motion information is obtained.

[0070] Based on the feature vector, calculate the cosine feature distance between adjacent frames in the target monitoring image.

[0071] Similarity is determined based on cosine feature distance.

[0072] Specifically, the functional expression for the Mahalanobis distance is:

[0073]

[0074] Where, d (1) (i,j) is the Mahalanobis distance; d j y represents the bounding box position information for the j-th frame; i S represents the bounding box position information for the i-th frame; i Let T be the covariance matrix; T is the transpose matrix.

[0075]

[0076] l is the indicator threshold; when the j-th detection box matches the ith tracking box, l = 1.

[0077] The formula for calculating the cosine characteristic distance is:

[0078]

[0079] in, r is the cosine feature distance; j Let j be the feature vector of the j-th frame; Let R be the feature vector of the i-th frame that was successfully associated k times; T is the transpose matrix; R i It is the set of feature vectors that were successfully associated k times.

[0080] Step 500: Based on the detection information and correlation, determine the uplink and downlink information based on the set baseline, and determine the number of fish that have passed through based on the uplink and downlink information.

[0081] This includes determining uplink and downlink information based on detection information and correlations, and then determining the number of fish that have passed through the watershed based on the uplink and downlink information. Specifically, this includes:

[0082] The Kalman filter method is used to perform state fusion based on detection information and correlation to obtain updated state information.

[0083] Based on the established baseline, uplink and downlink information are determined according to the detection information.

[0084] The number of fish that have crossed the line is determined by counting the number of fish that have crossed the line, based on the upstream and downstream information.

[0085] The methods for determining the target detection model specifically include:

[0086] Acquire training data; training data includes: initial monitoring images and label data of multiple training fish channels; label data includes: detection information corresponding to the initial monitoring images.

[0087] The training data is divided into a training set and a validation set.

[0088] A neural network is constructed based on object detection algorithms.

[0089] The training set and corresponding label data are input into the neural network, and the parameters of the neural network are trained with the goal of minimizing the loss function to obtain the trained neural network; the loss function is determined based on the error between the output of the neural network and the label data.

[0090] The parameters of the trained neural network are adjusted using the validation set and the corresponding label data to obtain the adjusted neural network.

[0091] The adjusted neural network was selected as the target detection model.

[0092] Example 2

[0093] The purpose of this invention is to provide a deep learning-based method for processing discretized fish passage images. This method is equivalent to setting up a hierarchical AI system to process the original fish passage images in real time, retaining only the images of fish passing through and outputting information such as the type and number of fish that passed through.

[0094] The AI ​​system mainly consists of three parts: a fish image initial screening module, a fish image tracking module, and a data storage and display module.

[0095] Fish Image Preliminary Screening Module: This module processes real-time fishway monitoring images. Based on the YOLOv5 target detection algorithm, it automatically extracts and saves images of fish passages that may be crossed by fish in the original video, using the appearance and disappearance of fish in the video stream as the start and stop conditions.

[0096] YOLOv5 is a deep learning-based object detection algorithm and the latest version of the YOLO series. Compared to previous versions, YOLOv5 offers significant improvements in both accuracy and speed. This algorithm employs a single-stage detection method, achieving object detection and localization with a single forward pass, resulting in fast detection speed. It utilizes the CSPDarknet feature extraction network, providing high computational efficiency and representational capabilities. Furthermore, YOLOv5 employs a multi-scale detection strategy, effectively handling targets of varying sizes and improving detection accuracy. Through rich data augmentation techniques and multi-class detection capabilities, YOLOv5 maintains high accuracy while achieving real-time performance, making it suitable for object recognition, real-time monitoring, and autonomous driving applications.

[0097] The main steps for extracting fish passage images using YOLOv5 are as follows:

[0098] Data preparation: Collect images of fish passing through fish passages, label them to create a dataset, and ensure that each target in the image has a corresponding category label and bounding box coordinates, i.e., the fish category and location coordinates.

[0099] Model training: The YOLOv5s model is trained using the training set in the dataset. The backpropagation algorithm and optimizer are used to adaptively adjust the model weight parameters so that it can accurately detect targets.

[0100] The training process of a YOLOv5 model involves inputting a labeled dataset into a network hosted on a computer for iterative processing. This mainly includes forward propagation, back propagation, loss function calculation, and parameter updates. The model will repeat these processes until predetermined stopping conditions are met, such as the maximum number of training epochs or loss function convergence.

[0101] Model evaluation: The trained model is evaluated using the validation set in the dataset. Metrics such as detection precision, recall, and average precision are calculated to assess the model's performance.

[0102] Object detection: The trained YOLOv5 model is used to detect fish passages in the video. When the model detects fish information in the image, it will output the detected object category and location information.

[0103] Fish image extraction: When the model outputs fish information, the original video is immediately recorded and edited until no fish are detected within 1 second, then recording stops and the video is saved.

[0104] See the flowchart for the prediction process using the YOLOv5 model. Figure 2 .

[0105] Fish Image Tracking Module: Based on the target tracking algorithm, the module detects fish passing through the fishway image after the fish influence screening module has processed the image, tracks each target fish in the image, and outputs the species, quantity, and up / down information of the fish.

[0106] The fish image tracking module is based on the DeepSORT algorithm, which is a multi-target tracking algorithm that combines deep learning and target association. The main steps are as follows:

[0107] Object detection: The YOLOv5 object detection algorithm is used to detect objects in the input image or video, and the location and category information of the objects in each frame are obtained.

[0108] Feature extraction: For each detected target, a deep convolutional neural network is used to extract its corresponding feature vector, which is then used in the subsequent target association process. The feature vector refers to the feature information of the target in the image.

[0109] A feature vector is a numerical vector extracted using a deep convolutional neural network to represent a target. This vector can be viewed as the target's representation in a feature space, where each dimension corresponds to a different feature. By using a deep convolutional neural network, the visual features of a target can be transformed into a numerical representation, making the target distinguishable and comparable in the feature space. In simpler terms, the information in a feature vector includes everything used to describe the target in an image, such as its shape, texture, and color.

[0110] Target association: Target association is performed between adjacent frames using the Hungarian algorithm to match the same target across different frames. The association relationship is determined by calculating the similarity of feature vectors and motion information.

[0111] Motion information is evaluated using Mahalanobis distance, which is then compared with a set threshold t. (1) If the Mahalanobis distance between the associations is less than the threshold, the association is successful.

[0112] Cosine similarity metric, based on appearance, serves as a second type of association measure.

[0113] Motion prediction: Based on the Kalman filter motion model, the position of undetected targets in future frames is predicted.

[0114] Position prediction is described by constructing an 8-dimensional state space to characterize the inter-frame displacement of each target:

[0115]

[0116] Where (u,v) represent the center coordinates of the target bounding box, γ represents the aspect ratio, and h is the height of the bounding box. This represents the velocity corresponding to the parameters (u,v,γ,h).

[0117] Status update: Based on the detection and correlation results of the current frame, the status of each tracked target is updated, including information such as position, velocity and appearance.

[0118] Target state update refers to updating the state information of detected and associated targets at each time step, including position, velocity, and appearance features. This update process is performed based on target trajectory prediction and measurements of associated targets.

[0119] There are two main situations:

[0120] a. For successfully associated targets, the target's state information is updated using the detection results of the current image frame. This includes updating the target's position, appearance features, and other relevant attributes. The method provided by this invention uses Kalman filtering to fuse and update the target state, taking into account both the predicted and measured target states.

[0121] b. For targets that fail to be successfully associated, it may be because the target was not detected in the current frame or could not be matched with any detection results. In this case, consider removing the target from the tracking list to avoid interfering with subsequent tracking.

[0122] Target Management: Based on the information obtained above, update the target's status and manage the tracked target, including target initialization, trajectory maintenance, and target deletion.

[0123] Tracking and counting: A virtual baseline is drawn in the middle of the video. Fish are counted by crossing the baseline. When the center point of the target fish detection box coincides with this baseline, the species information of the fish passing through is recorded and output. By comparing the coordinate values ​​of the center points before and after the fish passes through, the up / down information of the fish is determined, thus achieving fish species identification and counting. Figure 3 Flowchart for fish species identification and counting in fish passageways.

[0124] The data storage and display module fuses the fish passage images and the fish category, quantity, and upstream / downstream information output from the fish image tracking module according to a specific naming scheme, stores it in the database, and displays the fish passage images by calling and recognizing the encoded data. The fusion process involves naming the video image based on the information output after processing the fish passage images using the DeepSORT algorithm, facilitating data classification and processing.

[0125] The processed video should be named as follows:

[0126] D1F20211112_030000T20211112_031500U12D1.mp4;

[0127] D1 represents data from camera number 1.

[0128] F20211112_030000 represents the start time from 3:00:00 on November 12, 2021.

[0129] T20211112_031500 means the deadline is 3:15:00 on November 12, 2021.

[0130] U12 represents 12 entries upstream; D1 represents 1 entry downstream.

[0131] The processing flow of an AI system is as follows: Figure 4 and Figure 5 As shown, it can be described as follows:

[0132] 1) The fishway monitoring images are connected to the fish image screening module in real time, and the trained YOLOv5 algorithm is used to perform frame-by-frame detection on the input images.

[0133] 2) When the YOLOv5 algorithm detects fish, it immediately records the original video until no fish are detected for 1 second, then stops recording.

[0134] 3) Record the time when the fish appear and disappear, and save it to the database as the file name of the image of the fish passing through the channel.

[0135] 4) Input the fish passage images into the fish image tracking module for processing. Based on the DeepSORT target tracking algorithm, track each target fish in the image and finally output the fish category, number, and up / down information of the video.

[0136] 5) Merge the video saved in step 3) with the processing result output in step 4), that is, rename the video file according to the predetermined naming method, and then save it to the database.

[0137] 6) By analyzing the video file name encoding information, the fish passage image is displayed by calling the fish passage image.

[0138] Example 3

[0139] like Figure 6 As shown in the figure, this embodiment of the invention provides a fish passage image processing system, which includes: an image acquisition module 1, an output module 2, a target monitoring image determination module 3, a correlation determination module 4, and a fish passage number determination module 5.

[0140] Image acquisition module 1 is used to acquire initial monitoring images of the target fishway in real time.

[0141] Output module 2 is used to input the initial monitoring image into the target detection model and output detection information. The detection information includes: fish information and time information. The fish information includes: category and location coordinates. The time information includes: the time when fish appeared in the target fishway. The target detection model is built based on the target detection algorithm and uses machine learning methods.

[0142] The target monitoring image determination module 3 is used to record and edit the target fishway image based on the detection information and the set recording stop condition to obtain the target monitoring image; the set recording stop condition is that no fish information is detected in the target fishway within a set time interval.

[0143] The correlation determination module 4 is used to extract feature vectors of target monitoring images using a deep convolutional neural network, and determine the correlation between adjacent frames in the target monitoring images based on the feature vectors; the correlation includes: similarity and motion information; the feature vectors are used to characterize the detection information of the target monitoring images.

[0144] The fish passage number determination module 5 is used to determine the uplink and downlink information based on the detection information and correlation relationship, and to determine the fish passage number based on the uplink and downlink information.

[0145] Example 4

[0146] This invention provides an electronic device, including a memory and a processor. The memory stores a computer program, and the processor runs the computer program to enable the electronic device to perform the fish passage image processing method in Embodiment 1.

[0147] In one embodiment, the present invention also provides a computer-readable storage medium storing a computer program that, when executed by a processor, implements the fish passage image processing method of Embodiment 1.

[0148] The various embodiments in this specification are described in a progressive manner, with each embodiment focusing on its differences from other embodiments. Similar or identical parts between embodiments can be referred to interchangeably. For the systems disclosed in the embodiments, since they correspond to the methods disclosed in the embodiments, the descriptions are relatively simple; relevant parts can be referred to the method section.

[0149] This document uses specific examples to illustrate the principles and implementation methods of the present invention. The descriptions of the above embodiments are only for the purpose of helping to understand the method and core ideas of the present invention. Furthermore, those skilled in the art will recognize that, based on the ideas of the present invention, there will be changes in the specific implementation methods and application scope. Therefore, the content of this specification should not be construed as a limitation of the present invention.

Claims

1. A method for processing images of fish passing through a fishway, characterized in that, The method includes: Acquire initial monitoring images of the target fishway in real time; The initial monitoring image is input into the target detection model, which outputs detection information. The detection information includes fish information and time information. The fish information includes fish type and location coordinates. The time information includes the time when fish appear in the target fishway. The target detection model is constructed using machine learning methods based on a target detection algorithm. Based on the detection information and the set recording stop condition, the target fishway is image recorded and edited to obtain the target monitoring image; the set recording stop condition is that no fish information is detected in the target fishway within a set time interval. A deep convolutional neural network is used to extract feature vectors from the target monitoring image, and the correlation between adjacent frames in the target monitoring image is determined based on the feature vectors; the correlation includes similarity and motion information; the feature vectors are used to characterize the detection information of the target monitoring image; Based on the detection information and the correlation, uplink and downlink information are determined based on a set baseline, and the number of fish that have passed through is determined based on the uplink and downlink information; Based on the detection information and the correlation, uplink and downlink information are determined based on a set baseline, and the number of fish that have passed is determined based on the uplink and downlink information, specifically including: Using the Kalman filter method, state fusion is performed based on the detection information and the correlation relationship to obtain updated state information; Based on the established baseline, uplink and downlink information are determined according to the detection information. The number of fish that have crossed the line is determined by using the uplink and downlink information as a method of counting the number of fish that have crossed the line.

2. The fish passage image processing method according to claim 1, characterized in that, The method for determining the target detection model specifically includes: Acquire training data; the training data includes: initial training monitoring images and label data of multiple training fish passages; the label data includes: detection information corresponding to the initial training monitoring images; The training data is divided into a training set and a validation set; Construct a neural network based on object detection algorithms; The training set and corresponding label data are input into the neural network, and the parameters of the neural network are trained with the goal of minimizing the loss function to obtain the trained neural network; the loss function is determined based on the error between the output of the neural network and the label data; The parameters of the trained neural network are adjusted using the validation set and the corresponding label data to obtain the adjusted neural network. The adjusted neural network is determined as the target detection model.

3. The fish passage image processing method according to claim 1, characterized in that, Determining the correlation between adjacent frames in the target monitoring image based on the feature vector specifically includes: Based on the position information of the bounding boxes between adjacent frames in the target monitoring image, the Mahalanobis distance between adjacent frames in the target monitoring image is calculated; The Mahalanobis distance is compared with a set threshold. If the Mahalanobis distance is less than the set threshold, the association is successful and the motion information is obtained. Based on the feature vector, calculate the cosine feature distance between adjacent frames in the target monitoring image; The similarity is determined based on the cosine feature distance.

4. The fish passage image processing method according to claim 3, characterized in that, The functional expression for the Mahalanobis distance is: ； in, The Mahalanobis distance; For the first The bounding box position information of the frame; For the first The bounding box position information of the frame; It is the covariance matrix; This is the transpose of the matrix.

5. The fish passage image processing method according to claim 3, characterized in that, The formula for calculating the cosine characteristic distance is: ； in, The cosine feature distance; For the first Frame feature vector; For the first Frame in k Feature vectors that were successfully associated the first time; It is the transpose matrix; for k The set of feature vectors that were successfully associated.

6. A fish passage image processing system, characterized in that, The system includes: The image acquisition module is used to acquire initial monitoring images of the target fishway in real time; The output module is used to input the initial monitoring image into the target detection model and output detection information; the detection information includes: fish information and time information; the fish information includes: category and location coordinates; the time information includes: the time when fish appeared in the target fishway; the target detection model is constructed based on the target detection algorithm and using machine learning methods; The target monitoring image determination module is used to perform image recording and editing on the target fishway according to the detection information and the set recording stop condition to obtain the target monitoring image; the set recording stop condition is that no fish information is detected in the target fishway within a set time interval. The association determination module is used to extract feature vectors from the target monitoring image using a deep convolutional neural network, and determine the association relationship between adjacent frames in the target monitoring image based on the feature vectors; the association relationship includes: similarity and motion information; the feature vectors are used to characterize the detection information of the target monitoring image; The fish passage number determination module is used to determine uplink and downlink information based on a set baseline according to the detection information and the correlation relationship, and to determine the fish passage number according to the uplink and downlink information; Based on the detection information and the correlation, uplink and downlink information are determined based on a set baseline, and the number of fish that have passed is determined based on the uplink and downlink information, specifically including: Using the Kalman filter method, state fusion is performed based on the detection information and the correlation relationship to obtain updated state information; Based on the established baseline, uplink and downlink information are determined according to the detection information. The number of fish that have crossed the line is determined by using the uplink and downlink information as a method of counting the number of fish that have crossed the line.

7. An electronic device, characterized in that, The device includes a memory and a processor, the memory being used to store a computer program, and the processor running the computer program to cause the electronic device to perform the fish passage image processing method as described in any one of claims 1 to 5.

8. A computer-readable storage medium, characterized in that, It stores a computer program, which, when executed by a processor, implements the fish passage image processing method as described in any one of claims 1 to 5.

Images