Fish resource monitoring method and monitoring system based on sonar image and electronic equipment

Abstract

The invention discloses a fish resource monitoring method and system based on a sonar image and electronic equipment. The fish resource monitoring method comprises the following steps: acquiring the sonar image about a fish pond; detecting a fish target in more than two frames of sonar images, and tracking the fish target; and determining the number of the fish targets in the fishpond according to the tracking result. Through the above steps, the fish resource monitoring method can accurately monitor the number of fish targets in a fish pond, and does not cause adverse effects on the fish growth environment in the monitoring process.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of sonar image recognition, and particularly relates to a fish resource monitoring method based on a sonar image, a detection system and electronic equipment. BACKGROUND

[0002] In the field of modern aquaculture, monitoring the number of fish in a fish pond is the basis for scientific breeding, cost control and yield optimization. Through monitoring the number of fish, not only can the bait be accurately delivered and the growth cycle be scientifically managed, but also water quality deterioration, disease transmission caused by high fish density and water resource waste caused by low fish density can be avoided. At present, the monitoring methods for the number of fish in a fish pond include manual sampling method and marking recapture method. The manual sampling method is to capture samples in different areas of the fish pond, estimate the total number based on the sample quantity and capture rate. The defect of this method is that it is limited by sampling randomness and uneven fish distribution, resulting in high estimation error and inability to accurately monitor the number of fish. The marking recapture method is to release fish back into the fish pond after physical or chemical marking, and then estimate the total number based on the proportion of marked individuals when captured again. This method also has high error. Whether it is the manual sampling method or the marking recapture method, fish need to be captured multiple times, which not only takes time and effort, but also frequent capture can cause fish stress, thereby affecting fish growth. SUMMARY

[0003] An object of the present application is to provide a fish resource monitoring method based on a sonar image, a detection system and electronic equipment, wherein the fish resource monitoring method can accurately monitor the number of fish targets in a fish pond and will not cause adverse effects on the fish growth environment during the monitoring process.

[0004] An object of the present application is to provide a fish resource monitoring method based on a sonar image, a detection system and electronic equipment, wherein the fish resource monitoring method can accurately monitor the number of fish targets in a fish pond and will not cause adverse effects on the fish growth environment during the monitoring process.

[0005] An object of the present application is to provide a fish resource monitoring method based on a sonar image, a detection system and electronic equipment, wherein the fish resource monitoring method can accurately monitor the number of fish targets in a fish pond and will not cause adverse effects on the fish growth environment during the monitoring process.

[0006] An object of the present application is to provide a fish resource monitoring method based on sonar images, a detection system and an electronic device, wherein the fish resource monitoring method determines the number of fish targets by detecting the fish targets in sonar images of a fish pond, so that the fish growth environment is not adversely affected during the monitoring process.

[0007] According to a first aspect of the present application, the present application provides a fish resource monitoring method based on sonar images, wherein the fish resource monitoring method comprises the following steps: S1, obtaining sonar images of a fish pond; S2, detecting fish targets in two or more frames of the sonar images and tracking the fish targets; S3, determining the number of fish targets in the fish pond according to the tracking results.

[0008] According to an embodiment of the present application, the step S2 comprises: establishing a target list of the fish targets based on the detection results of the first frame of the sonar images, for tracking the fish targets; comparing the detection results of the subsequent frames of the sonar images with the fish targets in the target list, and if the fish targets in the detection results of the subsequent frames of the sonar images are not included in the target list, updating the target list.

[0009] According to an embodiment of the present application, the step S2 comprises: enhancing the sonar images; frame-by-frame detecting the fish targets in the enhanced sonar images to update the target list.

[0010] According to an embodiment of the present application, the step of enhancing the sonar images comprises: balancing the brightness of the sonar images at different distances; improving the dark details of the sonar images; unifying the brightness of the enhanced sonar images.

[0011] According to an embodiment of the present application, the step of balancing the brightness of the sonar images at different distances comprises: calculating the overall mean and the mean at different distances; calculating adaptive coefficients at different distances according to the proportion of the overall mean and the mean at different distances; adjusting the brightness at different distances based on the adaptive coefficients to balance the brightness of the sonar images at different distances.

[0012] According to one embodiment of the present invention, the step of improving the dark details of the sonar image includes: Take the log value from the sonar image to obtain a log image; Applying mean convolution to the log image yields the local mean of the log image. The difference between the local mean and the global mean of the log image is then calculated to obtain the highlight image. Calculate the difference between the log image and the highlight image to obtain the shadow image; The bright and dark images are added together with different weights and the result is taken as an exponent to improve the dark details of the sonar image.

[0013] According to an embodiment of the present invention, in step S2, the target list is updated based on an update strategy, wherein the update strategy includes: a matching relationship between the fish targets detected by the sonar image of the current frame and the retained targets in the target list corresponding to the fish targets detected by the sonar image of the previous frame.

[0014] According to an embodiment of the present invention, in step S2, the step of updating the target list includes: The fish targets detected in the sonar image of the previous frame are defined as retained targets in the target list, and a confidence level is set for the retained targets. At this time, the retained targets are set with the highest confidence level. The fish target detected in the sonar image of the current frame is defined as a new target. The new target is matched with the retained targets. If the retained target is not matched by the new target, the retained target is retained in the target list and its confidence level is reduced. If the retained target is matched by the new target, the new target is replaced by the retained target and its confidence level is set to the highest. If no retained target is matched by the new target, the new target is added as a new retained target to the target list and its confidence level is set to the highest.

[0015] According to one embodiment of the present invention, in step 2, the step of detecting the fish target in the enhanced sonar image includes: The enhanced sonar image is detected using a deep learning detection module to obtain deep learning detection results; The enhanced sonar image is detected based on the CFAR algorithm to obtain a CFAR feature map. The CFAR feature map is segmented using a threshold to obtain the CFAR detection result. The features of the fish target in the CFAR detection result are calculated. Using the deep learning detection results, the corresponding fish target is retrieved in the CFAR feature map, and the features of the retrieved fish target are taken as the baseline feature set. The feature fluctuation values ​​between features in the benchmark feature set are denoted as the benchmark feature fluctuation values. Calculate the average fluctuation of the features of the fish targets in the CFAR detection results and the features of the fish targets in the benchmark feature set, and calculate the ratio of the average fluctuation to the benchmark feature fluctuation value. CFAR detection results with a ratio within a certain range are retained as fusion results. All deep learning detection results are retained as the fusion result; Remove duplicate fusion results.

[0016] According to one embodiment of the present invention, the fish resource monitoring method includes a step of measuring the length of the fish target, the length measurement step of the fish target including: Based on the detection results of the fish target, the sonar image is cropped to obtain a sub-image containing a single fish; Calculate the mean of the sub-images, and use the mean as a threshold to perform threshold segmentation on the sub-images to obtain segmented images; Opening and closing operations are performed on the segmented image to smooth edges and fill image holes; Perform connectivity analysis on the segmented image and take the largest connected region as the set of segmentation points for the fish target; A polynomial curve is fitted to the set of fish target segmentation points to obtain a fitted curve; The pixel length of the fitted curve is calculated, and combined with the resolution of the sonar system, the actual length of the fish target is calculated.

[0017] According to a second aspect of the present invention, the present invention provides a fish resource monitoring system based on sonar images, comprising: Acquisition unit, used to acquire sonar images of the fishpond; A detection unit is used to detect fish targets in two or more frames of the sonar images; Tracking unit, used to track the fish target; The statistics unit is used to count the number of the fish targets based on the tracking results.

[0018] According to one embodiment of the present invention, the fish resource monitoring system further includes: an enhancement unit for enhancing the sonar image, thereby the detection unit detects the enhanced sonar image.

[0019] According to one embodiment of the present invention, the step of the enhancement unit enhancing the sonar image includes: balancing the brightness of the sonar image at near and far distances; improving the dark details of the sonar image; and unifying the brightness of the enhanced sonar image.

[0020] According to one embodiment of the present invention, the tracking unit tracks the fish target by: defining the fish target detected in the sonar image of the previous frame as a retained target in the target list and setting a confidence level for the retained target, at which time the retained target is set to the highest confidence level; defining the fish target detected in the sonar image of the current frame as a new target, matching the new target with the retained target; if the retained target is not matched by the new target, then the retained target is retained in the target list and the confidence level of the retained target is reduced; if the retained target is matched by the new target, then the new target is replaced by the retained target and the confidence level of the replaced retained target is set to the highest level; if no retained target is matched by the new target, then the new target is added as a new retained target to the target list and the highest confidence level is set.

[0021] According to a third aspect of the invention, the invention provides an electronic device comprising a processor and a memory; The memory stores program instructions; The processor is used to run the program instructions stored in the memory, causing the electronic device to execute a fish resource monitoring method, wherein the fish resource monitoring method includes the following steps: S1, acquiring sonar images of a fishpond; S2, detecting fish targets in two or more frames of the sonar images and tracking the fish targets; S3, determining the number of fish targets in the fishpond based on the tracking results. Attached Figure Description

[0022] Figure 1 This is a schematic diagram of the sonar system provided in an embodiment of the present invention.

[0023] Figure 2 This is a schematic diagram of an application scenario of the sonar system provided in an embodiment of the present invention.

[0024] Figure 3 This is a schematic flowchart of the fish resource monitoring method provided in the embodiments of the present invention.

[0025] Figure 4 These are sonar images provided in embodiments of the present invention.

[0026] Figure 5 This is a before-and-after comparison of the brightness of the sonar image provided in the embodiment of the present invention after the brightness at near and far distances was balanced.

[0027] Figure 6 It is the log image of the sonar image provided in the embodiment of the present invention.

[0028] Figure 7 This is a highlight image of a sonar image provided in an embodiment of the present invention.

[0029] Figure 8 This is a dark area image of a sonar image provided in an embodiment of the present invention.

[0030] Figure 9 This is an enhanced sonar image provided in an embodiment of the present invention.

[0031] Figure 10 This is a schematic diagram of the structure of the deep learning detection model provided in the embodiment of the present invention.

[0032] Figure 11 This is a schematic diagram of the structure of the multi-head self-attention module of the deep learning detection model provided in this embodiment of the invention.

[0033] Figure 12 This is the detection result of the sonar image of the first frame provided in the embodiment of the present invention.

[0034] Figure 13 This is the state of the target list after detecting the sonar image of the first frame, as provided in this embodiment of the invention.

[0035] Figure 14 This is the detection result of the second frame of sonar image detection provided in the embodiment of the present invention.

[0036] Figure 15 This is the state of the target list after sonar image detection in the second frame, as provided in this embodiment of the invention.

[0037] Figure 16 This is the detection result of the sonar image of the third frame provided in the embodiment of the present invention.

[0038] Figure 17 This is the state of the target list after sonar image detection in the third frame, as provided in this embodiment of the invention.

[0039] Figure 18 This is the detection result of the sonar image of the fourth frame provided in the embodiment of the present invention.

[0040] Figure 19 This is the state of the target list after sonar image detection in the fourth frame, as provided in this embodiment of the invention.

[0041] Figure 20 This is a sub-image obtained by cropping the enhanced sonar image, as provided in the embodiments of the present invention.

[0042] Figure 21 This is a segmented image obtained after thresholding a sub-image, as provided in an embodiment of the present invention.

[0043] Figure 22 This is the image obtained by performing opening and closing operations on the segmented image, as provided in the embodiments of the present invention.

[0044] Figure 23 This is an image obtained by curve fitting of a set of points, as provided in an embodiment of the present invention.

[0045] Figure 24 This is a schematic block diagram of the fish resource monitoring system provided in an embodiment of the present invention.

[0046] Figure 25 This is a schematic block diagram of an electronic device provided in an embodiment of the present invention. Detailed Implementation

[0047] The technical solutions in the embodiments of this application will now be described with reference to the accompanying drawings.

[0048] The terminology used in the following embodiments is for the purpose of describing particular embodiments only and is not intended to be used as a basis for interpretation. Limitations of this application. As used in the specification and appended claims of this application, the singular expressions “a,” “an,” “the,” “the,” “the,” and “this” are intended to also include expressions such as “one or more,” unless the context clearly indicates otherwise. It should also be understood that in the following embodiments of this application, “at least one” and “one or more” refer to one, two, or more than two. The term “and / or” is used to describe the relationship between related objects, indicating that three relationships may exist; for example, A and / or B can represent: A alone, A and B simultaneously, or B alone, where A and B can be singular or plural. The character “ / ” generally indicates that the preceding and following related objects are in an “or” relationship.

[0049] References to "one embodiment" or "some embodiments" as described in this specification mean that one or more embodiments of this application include a specific feature, structure, or characteristic described in connection with that embodiment. Therefore, the phrases "in one embodiment," "in some embodiments," "in other embodiments," "in still other embodiments," etc., appearing in different parts of this specification do not necessarily refer to the same embodiment, but rather mean "one or more, but not all, embodiments," unless otherwise specifically emphasized. The terms "comprising," "including," "having," and variations thereof mean "including but not limited to," unless otherwise specifically emphasized.

[0050] Figure 1A preferred example of a sonar system 100 is shown, the sonar system 100 including an FPGA processor 110, a D / A conversion circuit 120, an amplification drive circuit 130, a transducer 140, an A / D conversion circuit 150, a filter amplification circuit 160, a wireless communication module 170, and a wired communication module 180. The FPGA processor 110, the D / A conversion circuit 120, the amplification drive circuit 130, and the transducer 140 are connected in sequence. The transducer 140, the A / D conversion circuit 150, and the filter amplification circuit 160 are connected in sequence. The wireless communication module 170 and the wired communication module 180 are connected to the FPGA processor 110 to realize communication between the sonar system 100 and external devices. The D / A conversion circuit 120 receives the digital signal from the FPGA processor 110 and converts it into an analog signal. After processing by the amplification and driving circuit 130, the signal is output to the transducer 140, which then transmits an acoustic signal into the fishpond. The echo signal received by the transducer 140 is received by the A / D conversion circuit 150, filtered and amplified by the filtering and amplification circuit 160, and then output to the FPGA processor 110, thus obtaining an image of the fishpond.

[0051] It is understood that the structure illustrated in this invention does not constitute a specific limitation on the sonar system 100. In some embodiments, the sonar system 100 may include more or fewer components than illustrated, or combine and / or split certain components. The illustrated components may be implemented in hardware, software, or a combination of hardware and software.

[0052] It should be noted that after the sonar system 100 is deployed in the fishpond, even if the sonar system 100 is kept in the activated state, it does not mean that the sonar system 100 must transmit underwater acoustic signals and receive echo signals into the fishpond. The sonar system 100 can only transmit underwater acoustic signals and receive echo signals into the fishpond when the FPGA processor 110 sends digital signals to the D / A conversion circuit 120.

[0053] Figure 2An application scenario of the sonar system 100 is illustrated, wherein the sonar system 100 is deployed in a fishpond. In this specific example of the invention, the field of view of the sonar system 100 is 120°. To obtain a 360° sonar image of the fishpond, three sonar systems 100 are arranged adjacently. It should be noted that the number of sonar systems 100 is related to their field of view, depending on the requirement to obtain a 360° sonar image of the fishpond. Therefore, if the field of view of the sonar system 100 is 180° or more, two sonar systems 100 arranged adjacently are sufficient to obtain a 360° sonar image of the fishpond. Conversely, if the field of view of the sonar system 100 is less than 120°, four or more sonar systems 100 need to be arranged adjacently to obtain a 360° sonar image of the fishpond.

[0054] For ease of understanding, the following description uses the example of three sonar systems 100 arranged adjacent to each other to disclose the content and features of the present invention, wherein the three sonar systems 100 are defined as a first sonar system 100a, a second sonar system 100b and a third sonar system 100c, and the sum of the field of view of the first sonar system 100a, the second sonar system 100b and the third sonar system 100c is greater than or equal to 360°.

[0055] During operation, the first sonar system 100a, the second sonar system 100b, and the third sonar system 100c are all kept in the activated state, and the first sonar system 100a, the second sonar system 100b, and the third sonar system 100c are allowed to sequentially transmit underwater acoustic signals to the fishpond and receive echoes.

[0056] Specifically, when the first sonar system 100a acquires a first raw image of a first area of ​​the fishpond, both the second sonar system 100b and the third sonar system 100c are prevented from transmitting underwater acoustic signals and receiving echo signals into the fishpond, in order to avoid interference from the second sonar system 100b and the third sonar system 100c with the operation of the first sonar system 100a; when the second sonar system 100b acquires a second raw image of a second area of ​​the fishpond, the first sonar system 100a and the third sonar system 100c are prevented from transmitting underwater acoustic signals and receiving echo signals into the fishpond. The first sonar system 100a and the second sonar system 100c are prevented from transmitting underwater acoustic signals and receiving echo signals into the fishpond to avoid interference between the first sonar system 100a and the third sonar system 100c and the second sonar system 100b. When the third sonar system 100c acquires a third original image of a third area of ​​the fishpond, the first sonar system 100a and the second sonar system 100b are prevented from transmitting underwater acoustic signals and receiving echo signals into the fishpond to avoid interference between the first sonar system 100a and the second sonar system 100b and the third sonar system 100c.

[0057] After the first, second, and third original images are acquired, they are stitched together to obtain a sonar image of the fishpond, wherein the sonar image is 360° to avoid monitoring blind spots.

[0058] It should be noted that the specific method of stitching together the first original image, the second original image, and the third original image to obtain the 360° sonar image of the fishpond is not limited in this invention. For example, in some embodiments of this invention, the first original image acquired by the first sonar system 100a, the second original image acquired by the second sonar system 100b, and the third original image acquired by the third sonar system 100c are of the same size. Therefore, the first original image, the second original image, and the third original image can be arranged into a circular image, thereby achieving the stitching of the first original image, the second original image, and the third original image.

[0059] Appendix Figure 3 An exemplary flowchart of a fish resource monitoring method based on sonar images provided by the present invention is shown, wherein the fish resource monitoring method includes the following steps: S310, acquiring the sonar image of a fishpond; S320, detecting fish targets in two or more frames of the sonar image and tracking the fish targets; S330, determining the number of fish targets in the fishpond based on the tracking results.

[0060] Specifically, in step S310, two or more sonar systems 100 are deployed in the fishpond, wherein the sum of the field of view of these sonar systems 100 is 360° or more. For example, in one embodiment of the present invention, the number of sonar systems 100 deployed in the fishpond is three, wherein the first sonar system 100a, the second sonar system 100b, and the third sonar system 100c are located adjacent to each other, and the sum of the field of view of the first sonar system 100a, the second sonar system 100b, and the third sonar system 100c is 360° or more.

[0061] After the first sonar system 100a, the second sonar system 100b, and the third sonar system 100c are activated and maintained in the activated state, the first sonar system 100a, the second sonar system 100b, and the third sonar system 100c are sequentially allowed to transmit underwater acoustic signals and receive echoes toward the fishpond, so as to acquire the first original image of the first area of ​​the fishpond, the second original image of the second area of ​​the fishpond, and the third original image of the third area of ​​the fishpond, respectively. In this process, to avoid mutual interference between the first sonar system 100a, the second sonar system 100b, and the third sonar system 100c, when the first sonar system 100a acquires the first raw image of the first area of ​​the fishpond, the second sonar system 100b and the third sonar system 100c do not acquire images; when the second sonar system 100b acquires the second raw image of the second area of ​​the fishpond, the first sonar system 100a and the third sonar system 100c do not acquire images; and when the third sonar system 100c acquires the third raw image of the third area of ​​the fishpond, the first sonar system 100a and the second sonar system 100b do not acquire images.

[0062] Then, the first original image, the second original image, and the third original image are stitched together to obtain the sonar image of the fishpond in 360°. Figures 4 to 9 This shows a portion of the stitched-together sonar image.

[0063] In some embodiments of the present invention, step S320 further includes the step: S321, enhancing the sonar image to improve the contrast between the fish target and the background, facilitating subsequent detection and improving detection accuracy. The enhanced sonar image is as follows: Figure 9 As shown.

[0064] Specifically, the step of enhancing the sonar image includes: S3211, balancing the brightness of the sonar image at near and far distances. In a fishpond, the fish targets usually appear in groups and swim, so the brightness differences in the first original image, the second original image, and the third original image acquired by the first sonar system 100a, the second sonar system 100b, and the third sonar system 100c are large, resulting in large brightness differences in the stitched sonar image. This manifests as a weak contrast between distant fish targets and the background, and an inconsistency between the intensity of distant fish targets and the intensity of nearby fish targets, making detection difficult and affecting the accuracy of the detection results. By balancing the brightness of the sonar image at near and far distances, not only can the contrast between distant fish targets and the background be improved, but the intensity of distant fish targets can also be made consistent with the intensity of nearby fish targets, thus facilitating detection and improving the accuracy of the detection results.

[0065] The specific steps for balancing the brightness of the sonar image at near and far distances can be as follows: First, calculate the average value of the entire image and the average value at different distances; second, calculate the adaptive coefficient at different distances based on the ratio of the average value of the entire image to the average value at different distances; finally, adjust the brightness at different distances based on the adaptive coefficient to balance the brightness of the sonar image at near and far distances. (Refer to Appendix) Figure 5 . Figure 5 The area within the red box clearly shows the result of balancing the brightness of the sonar image at near and far distances. In other words, in this invention, the fish resource monitoring method can adaptively balance the brightness of the sonar image at near and far distances to better restore real scene details, avoid target misses caused by brightness imbalance, and enable the fish resource monitoring method of this invention to meet different usage environments.

[0066] In some embodiments of the present invention The mean of the entire graph. For pixel values, is the x-coordinate of the pixel. y is the ordinate of the pixel. The number of pixels, the average value of the entire image. The calculation formula is: ; The distance from the pixel to the center of the sonar system 100. for Mean of distance for Pixel values ​​in distance for Number of pixels in distance mean of distance The calculation formula is: Overall mean and the mean at different distances proportion Satisfying the relation ,in This is the adaptive coefficient. for The pixel values ​​after adjusting brightness based on distance satisfy the following relationship: ,in this way Brightness can be balanced across distances.

[0067] The fishpond environment is complex, resulting in three types of sonar images: very bright areas, which may be the fish targets; moderately bright areas; and moderately dark areas, which may be the background. Typically, the dark background corresponding to the darker areas may be due to the uneven terrain of the fishpond or the shadows cast by the fish targets. Therefore, step S320 further includes step S322, enhancing the dark details of the sonar image to unify the bright background corresponding to the brighter areas and the dark background corresponding to the darker areas, so that the sonar image contains only the bright targets corresponding to the very bright areas and a relatively uniform background. Finally, step S320 further includes step S323, unifying the brightness of the enhanced sonar image to facilitate acceptance by the deep learning detection model 1000.

[0068] Specifically, the steps to improve the dark details of the sonar image include: first, taking the log value of the sonar image to obtain a log image, such as... Figure 6 As shown; secondly, apply mean convolution to the log image to obtain the local mean of the log image. Calculate the difference between the local mean and the global mean of the log image, which is then used as the highlight image. The highlight image is shown below. Figure 7 As shown; next, calculate the difference between the log image and the highlight image, which is taken as the shadow image, as shown. Figure 8 As shown; finally, the bright and dark images are added together with different weights and the result is taken as an exponent to improve the dark details of the sonar image, such as... Figure 9 As shown. For example, in some embodiments of the present invention, the weight ratio of the brightness image and the dark image may be 7:3.

[0069] Step S320 further includes step S322, performing frame-by-frame detection on the enhanced sonar image to detect the fish target in each frame of the sonar image. In some embodiments of the present invention, the deep learning detection model 600 is used to detect the enhanced sonar image, and the specific structure of the deep learning detection model 600 is as follows: Figure 10 andFigure 11 As shown, it includes an input module 1001, a multi-head self-attention module 1002, a 5-convolutional group module 1003, a 3-convolutional group module 1004, a 3-convolutional group module 1005, and an output module 1006. It is understood that the data obtained after the sonar image is enhanced is enhanced underwater acoustic echo intensity data, not standard image data. Therefore, the target features input to the deep learning detection model 1000 via the input module 1001 are largely represented in terms of data rather than shape. Thus, the data needs to be processed by the multi-head self-attention module 1002 to make the deep learning detection model 1000 more sensitive to the input data features. Then, it undergoes convolution sequentially through the 5-convolutional group module 1003, the 3-convolutional group module 1004, and the 3-convolutional group module 1005, and finally output through the output module 1006 to achieve the detection of the fish target.

[0070] Step S320 further includes step S323, updating the target list. After detecting the fish targets in the enhanced sonar image, the step of updating the target list includes: first, defining the fish targets detected in the previous frame of the sonar image as retained targets in the target list, and setting a confidence level for the retained targets, at which point the retained targets are set to the highest confidence level; second, defining the fish targets detected in the current frame of the sonar image as new targets, matching the new targets with the retained targets, if the retained targets are not matched by the new targets, then the retained targets are retained in the target list and their confidence levels are reduced; if the retained targets are matched by the new targets, then the new targets are replaced by the retained targets, and the confidence levels of the replaced retained targets are set to the highest; if no retained targets are matched by the new targets, then the new targets are added as new retained targets to the target list and their confidence levels are set to the highest. In other words, during the monitoring of the fish targets, the number of fish targets included in the target list increases. In some embodiments, if the target list is not updated within a preset time (e.g., 30 minutes or 60 minutes), the number of fish targets included in the target list is taken as the total number of fish targets currently in the fishpond. In other embodiments, when the sonar system 100 no longer acquires new sonar images, the number of fish targets included in the target list is taken as the total number of fish targets currently in the fishpond.

[0071] In some embodiments of the present invention, after a new target is replaced by a retained target in the target list, the retained target is adjusted back to the highest confidence level. Correspondingly, if no new target matches a retained target in the target list, its confidence level is lowered. If no new target matches a retained target in the target list, and the new target is added to the target list as another retained target, the newly added retained target has its confidence level set to the highest confidence level. For a retained target in the target list, a lower confidence level corresponds to a lower search threshold, meaning the retained target is more likely to be found in the next match. This also means that in the next match, the same new target may be matched with more than two retained targets. In this case, based on the position and / or size of the fish target, the most similar retained target is selected to be replaced by the new target.

[0072] In a typical example of the present invention, refer to the appendix. Figure 12 and Figure 13 When detecting the sonar image of the first frame, fish target 101, fish target 102 and fish target 103 can be detected. Correspondingly, the target list is established and the reserved target 101a, reserved target 102a and reserved target 103a are added. The confidence level of the reserved target 101a, reserved target 102a and reserved target 103a is set to 1.0. At this time, the current count is 3 and the total count is 3.

[0073] Reference Appendix Figure 14 and Figure 15When detecting the sonar image in the second frame, fish targets 104, 105, and 106 can be detected. All three fish targets are new targets. Fish targets 104, 105, and 106 are matched with retained targets 101a, 102a, and 103a, respectively. Fish targets 104 and 101a can be matched, meaning they are the same fish target. At this point, the position and size of fish target 104 are replaced with the position and size of retained target 101a, and the confidence level is set to 1.0 again to track this fish target. Fish targets 105 and 102a can also be matched. That is, fish target 102 and fish target 105 are the same fish target. In this case, the position and size of fish target 105 are replaced with the position and size of the retained target 102a, and the confidence level is set to 1.0 again to track this fish target. Fish target 106 and the retained target 103a can be matched, meaning fish target 103 and fish target 106 are the same fish target. In this case, the position and size of fish target 106 are replaced with the position and size of the retained target 103a, and the confidence level is set to 1.0 again to track this fish target. Through these steps, the fish resource monitoring method updates the target list. At this point, the current quantity count is 3, and the total quantity count is 3.

[0074] Reference Appendix Figure 16 and Figure 17When detecting the sonar image in the third frame, fish targets 107, 108, and 109 are detected, all of which are new targets. Fish targets 107, 108, and 109 are matched with reserved targets 101a, 102a, and 103a, respectively. Since reserved target 101a is not matched by any new target, it remains in the target list, and its confidence level is reduced to 0.9. Since fish targets 107 and 102a can be matched (i.e., fish targets 107 and 105 are the same fish target), the position and size of fish target 107 are replaced with the position and size of reserved target 102a. The fish target 108 is not matched with any of the reserved targets. Therefore, fish target 108 is added to the target list as a new reserved target 104a, with a confidence level of 1.0. Fish target 109 and reserved target 103a are matched, meaning they are the same fish target. The position and size of fish target 109 are then replaced with the position and size of reserved target 103a, and the confidence level is again set to 1.0, thus tracking this fish target. Through these steps, the fish resource monitoring method updates the target list. At this point, the current count is 3, and the total count is 4.

[0075] Reference Appendix Figure 18 and Figure 19When detecting the sonar image of the fourth needle, fish target 110, fish target 111 and fish target 112 can be detected, and fish target 110, fish target 111 and fish target 112 are all new targets. Fish targets 110, 111, and 112 are matched with reserved targets 101a, 102a, 103a, and 104a, respectively. Although the positions of fish targets 110 and 101a are far apart, the confidence level of reserved target 101a is low (0.9). Therefore, fish targets 110 and 101a can still be matched, meaning they are the same fish target. In this case, the position and size of fish target 110 are replaced with the position and size of reserved target 101a, and the confidence level is again set to 1.0. Fish targets 111 and 102a can then be matched. Fish target 111 and fish target 107 are the same fish target. In this case, the position and size of fish target 104 are replaced with the position and size of retained target 102a, and the confidence level is set to 1.0 again to track this fish target. Fish target 112 and retained target 104a can be matched, meaning fish target 112 and fish target 108 are the same fish target. In this case, the position and size of fish target 112 are replaced with the position and size of retained target 104a, and the confidence level is set to 1.0 again to track this fish target. Retained target 103a has not been matched by any new target, so retained target 103a is kept in the target list, and the confidence level of retained target 103a is reduced to 0.9. Through these steps, the fish resource monitoring method updates the target list. At this point, the current quantity count is 3, and the total quantity count is 4.

[0076] In some embodiments of the present invention, the step of detecting the fish target in the enhanced sonar image includes: First, detecting the enhanced sonar image based on the deep learning detection module 1000 to obtain a deep learning detection result; Second, detecting the enhanced sonar image based on the CFAR algorithm to obtain a CFAR feature map, segmenting the CFAR feature map using a threshold to obtain a CFAR detection result, calculating the features of the fish target in the CFAR detection result, retaining the portion above the threshold and discarding the portion below the threshold; Third, using the deep learning detection result, retrieving the corresponding fish target in the CFAR feature map, and taking the features of the retrieved fish target (including feature intensity and regional contrast) as a reference feature set; Fourth, recording the feature fluctuation values ​​between features in the reference feature set as reference feature fluctuation values; Fifth, calculating the average fluctuation of the features of the fish target in the CFAR detection result and the features of the fish target in the reference feature set, and calculating the ratio of the average fluctuation to the reference feature fluctuation value, retaining the CFAR detection results with a ratio within a certain range as fusion results; Sixth, retaining all deep learning detection results as fusion results; Finally, removing duplicate fusion results. In this way, the fish resource monitoring method obtains more comprehensive and accurate detection results.

[0077] In some embodiments of the present invention, step S320 further includes the following steps: First, based on the detection results of the first frame of the sonar image, establish the target list; second, compare the detection results of the subsequent frames of the sonar image with the target list, and if the fish target in the detection results of the subsequent frames of the sonar image is not included in the target list, then add the fish target to the target list to update the target list.

[0078] Continue to refer to the appendix Figure 3 The fish resource monitoring method of the present invention further includes the step of: S340, measuring the length of the fish target. It is understood that after the length of the fish target is measured, the weight of the fish target can be estimated, thereby obtaining the size and weight distribution of various fish targets in the fishpond. This is beneficial for monitoring the growth of the fish targets and for facilitating precise feed delivery and scientific feeding in the fishpond.

[0079] Specifically, the length measurement step of the fish target includes: S341, based on the detection result of the fish target, cropping the sonar image to obtain a sub-image containing a single fish, that is, the sub-image of a single fish is extracted from the sonar image, see attached figure. Figure 20 S342, calculate the mean of the sub-image, use the mean as a threshold to perform threshold segmentation on the sub-image, and obtain the segmented image. (Refer to Appendix)Figure 21 S343, Perform opening and closing operations on the segmented image to smooth edges and fill image holes, refer to Appendix Figure 22 S344, Perform connected component analysis on the segmented image and take the largest connected component as the fish target segmentation point set; S345, Perform polynomial curve fitting on the fish target segmentation point set to obtain the fitted curve, refer to Appendix Figure 23 S346, calculate the pixel length of the fitted curve, and combine it with the resolution of the sonar system 100 to calculate the actual length of the fish target. Through the above steps, the fish resource monitoring method can measure the actual length of the fish target in a swimming state.

[0080] Reference Appendix Figure 24 The present invention further provides a fish resource monitoring system 2400 based on sonar images, which includes an acquisition unit 2410, a detection unit 2420, a tracking unit 2430 and a statistics unit 2440.

[0081] The acquisition unit 2410 is used to acquire the sonar image of the fishpond. In some embodiments of the present invention, after acquiring the first original image of a first area of ​​the fishpond from the first sonar system 100a, the second original image of a second area of ​​the fishpond from the second sonar system 100b, and the third original image of a third area of ​​the fishpond from the third sonar system 100c, the acquisition unit 2410 stitches together the first original image, the second original image, and the third original image to obtain a 360° sonar image of the fishpond.

[0082] The detection unit 2420 detects fish targets in two or more frames of the sonar images. To facilitate detection by the detection unit 2420, the fish resource monitoring system 2400 further includes an enhancement unit 2450, which enhances the sonar images so that the detection unit 2420 can detect the enhanced sonar images, thereby facilitating detection and improving the accuracy of the detection results.

[0083] In some embodiments of the present invention, the enhancement unit 2450 enhances the sonar image by balancing the brightness of the sonar image at near and far distances, improving the dark details of the sonar image, and making the brightness of the enhanced sonar image uniform. In this way, the brightness of the sonar image at near and far distances can be adaptively balanced, which is beneficial to restoring the real scene details, avoiding target omissions caused by brightness imbalance, and the enhanced sonar image should have only bright targets corresponding to the bright areas and relatively uniform backgrounds as much as possible.

[0084] The detection unit 2420 uses the deep learning detection model 1000 to detect the enhanced sonar image. The specific process is as follows: First, the enhanced underwater acoustic echo intensity data is input into the deep learning detection model 1000 via the input module 1001. Second, it is processed by the multi-head self-attention module 1002 to make the deep learning detection model 1000 more sensitive to the features of the input data. Then, it is convolved sequentially by the 5-convolution group module 1003, the 3-convolution group module 1004, and the 3-convolution group module 1005. Finally, it is output by the output module 1006 to achieve the detection of the fish target.

[0085] In one embodiment of the present invention, the tracking unit 2430 updates the target list based on an update strategy to track the fish target, wherein the update strategy includes: a matching relationship between the fish target detected by the sonar image in the current frame and the retained targets in the target list corresponding to the fish target detected by the sonar image in the previous frame. Specifically, firstly, the tracking unit 2430 defines the fish targets detected in the sonar image of the previous frame as the retained targets in the target list and sets a confidence level for the retained targets, at which point the retained targets are set to the highest confidence level. Secondly, the tracking unit 2430 defines the fish targets detected in the sonar image of the current frame as the new targets and matches the new targets with the retained targets. If the retained targets are not matched by the new targets, the retained targets are retained in the target list and the confidence level of the retained targets is reduced. If the retained targets are matched by the new targets, the new targets are replaced with the retained targets, and the confidence level of the replaced retained targets is set to the highest level. If no retained targets are matched by the new targets, the new targets are added to the target list as new retained targets and set to the highest confidence level.

[0086] The statistics unit 2440 determines the number of fish targets in the fishpond based on the tracking results of the tracking unit 2430 (i.e., the number of retained targets in the target list).

[0087] Continue to refer to the appendix Figure 24The fish resource monitoring system 2400 further includes a measurement unit 2460, which is used to measure the length of the fish target. Specifically, the process of the measurement unit 2460 measuring the length of the fish target is as follows: First, based on the detection results of the fish target, the sonar image is cropped to obtain a sub-image containing a single fish; second, the mean of the sub-image is calculated, and the mean is used as a threshold to perform threshold segmentation on the sub-image to obtain a segmented image; third, opening and closing operations are performed on the segmented image to smooth edges and fill image holes; fourth, connected component analysis is performed on the segmented image, and the largest connected component is taken as the segmentation point set of the fish target; fifth, polynomial curve fitting is performed on the segmentation point set of the fish target to obtain a fitted curve; sixth, the pixel length of the fitted curve is calculated, and the actual length of the fish target is calculated in combination with the resolution of the sonar system.

[0088] According to another aspect of the invention, reference is made to the appendix. Figure 25 The present invention further provides an electronic device 2500, which includes a processor 2510 and a memory 2520. The memory 2520 stores program instructions, and the processor 2510 is used to run the program instructions stored in the memory 2520, so that the electronic device executes the fish resource monitoring method.

[0089] The memory 2520 may include a program storage area and a data storage area. The program storage area may store the operating system, and the data storage area may store data created during the use of the electronic device 2500. For example, data related to the target list may be stored in the data storage area. The memory 2520 may include high-speed random access memory and non-volatile memory.

[0090] Those skilled in the art should understand that the embodiments of the present invention described above and shown in the accompanying drawings are merely examples and do not limit the present invention. The objectives of the present invention have been fully and effectively achieved. The functions and structural principles of the present invention have been demonstrated and explained in the embodiments, and any variations or modifications may be made to the implementation of the present invention without departing from the stated principles.

Claims

1. A method for monitoring fish resources based on sonar images, characterized in that, The fish resource monitoring method includes the following steps: S1, acquire sonar images of the fishpond; S2, detect fish targets in two or more frames of the sonar images and track the fish targets; S3, Based on the tracking results, determine the number of the fish targets in the fishpond.

2. The fish resource monitoring method according to claim 1, wherein step S2 includes: Based on the detection results of the first frame of the sonar image, a target list of the fish targets is established for tracking the fish targets; The detection results of the sonar images in subsequent frames are compared with the fish targets in the target list. If the fish targets in the detection results of the sonar images in subsequent frames are not included in the target list, the target list is updated.

3. The fish resource monitoring method according to claim 1, wherein step S2 includes: Enhance the sonar image; The fish targets in the enhanced sonar image are detected frame by frame to update the target list.

4. The fish resource monitoring method according to claim 3, wherein the step of enhancing the sonar image includes: Balance the brightness of the sonar image at near and far distances; Improve the dark details of the sonar image; To unify the brightness of the enhanced sonar image.

5. The fish resource monitoring method according to claim 4, wherein the step of balancing the brightness of the sonar image at near and far distances includes: Calculate the mean of the entire map and the mean at different distances; Calculate the adaptive coefficients at different distances based on the proportions of the overall map mean and the mean at different distances; The brightness of the sonar image is adjusted at different distances based on an adaptive coefficient to balance the brightness at both near and far distances.

6. The fish resource monitoring method according to claim 4, wherein the step of improving the dark details of the sonar image includes: Take the log value from the sonar image to obtain a log image; Applying mean convolution to the log image yields the local mean of the log image. The difference between the local mean and the global mean of the log image is then calculated to obtain the highlight image. Calculate the difference between the log image and the highlight image to obtain the shadow image; The bright and dark images are added together with different weights and the result is taken as an exponent to improve the dark details of the sonar image.

7. The fish resource monitoring method according to claim 3, wherein in step S2, the target list is updated based on an update strategy, wherein the update strategy includes: The matching relationship between the fish targets detected by the sonar image in the current frame and the reserved targets in the target list that correspond to the fish targets detected by the sonar image in the previous frame.

8. The fish resource monitoring method according to claim 3, wherein the step of updating the target list in step S2 includes: The fish targets detected in the sonar image of the previous frame are defined as retained targets in the target list, and a confidence level is set for the retained targets. At this time, the retained targets are set with the highest confidence level. The fish target detected in the sonar image of the current frame is defined as a new target. The new target is matched with the retained targets. If the retained target is not matched by the new target, the retained target is retained in the target list and its confidence level is reduced. If the retained target is matched by the new target, the new target is replaced by the retained target and its confidence level is set to the highest. If no retained target is matched by the new target, the new target is added as a new retained target to the target list and its confidence level is set to the highest.

9. The fish resource monitoring method according to claim 3, wherein the step of detecting the fish target in the enhanced sonar image in step 2 includes: The enhanced sonar image is detected using a deep learning detection module to obtain deep learning detection results; The enhanced sonar image is detected based on the CFAR algorithm to obtain a CFAR feature map. The CFAR feature map is segmented using a threshold to obtain the CFAR detection result. The features of the fish target in the CFAR detection result are calculated. Using the deep learning detection results, the corresponding fish target is retrieved in the CFAR feature map, and the features of the retrieved fish target are taken as the baseline feature set. The feature fluctuation values ​​between features in the benchmark feature set are denoted as the benchmark feature fluctuation values. Calculate the average fluctuation of the features of the fish targets in the CFAR detection results and the features of the fish targets in the benchmark feature set, and calculate the ratio of the average fluctuation to the benchmark feature fluctuation value. CFAR detection results with a ratio within a certain range are retained as fusion results. All deep learning detection results are retained as the fusion result; Remove duplicate fusion results.

10. The fish resource monitoring method according to any one of claims 2 to 9, wherein the fish resource monitoring method includes a step of measuring the length of the fish target, the length measurement step of the fish target comprising: Based on the detection results of the fish target, the sonar image is cropped to obtain a sub-image containing a single fish; Calculate the mean of the sub-images, and use the mean as a threshold to perform threshold segmentation on the sub-images to obtain segmented images; Opening and closing operations are performed on the segmented image to smooth edges and fill image holes; Perform connectivity analysis on the segmented image and take the largest connected region as the set of segmentation points for the fish target; A polynomial curve is fitted to the set of fish target segmentation points to obtain a fitted curve; The pixel length of the fitted curve is calculated, and combined with the resolution of the sonar system, the actual length of the fish target is calculated.

11. A fish resource monitoring system based on sonar imagery, characterized in that, include: Acquisition unit, used to acquire sonar images of the fishpond; A detection unit is used to detect fish targets in two or more frames of the sonar images; Tracking unit, used to track the fish target; The statistics unit is used to count the number of the fish targets based on the tracking results.

12. The fish resource monitoring system according to claim 11, wherein the fish resource monitoring system further comprises: An enhancement unit is used to enhance the sonar image, so that the detection unit detects the enhanced sonar image.

13. The fish resource monitoring system according to claim 12, wherein the step of the enhancement unit enhancing the sonar image includes: Balance the brightness of the sonar image at near and far distances; Improve the dark details of the sonar image; To unify the brightness of the enhanced sonar image.

14. The fish resource monitoring system according to any one of claims 11 to 13, wherein the tracking unit tracks the fish target by: defining the fish target detected in the sonar image of the previous frame as a retained target in the target list, and setting a confidence level for the retained target, wherein the retained target is set to the highest confidence level; defining the fish target detected in the sonar image of the current frame as a new target, matching the new target with the retained target; if the retained target is not matched by the new target, then the retained target is retained in the target list and the confidence level of the retained target is reduced; if the retained target is matched by the new target, then the new target is replaced by the retained target, and the confidence level of the replaced retained target is set to the highest level; if no retained target is matched by the new target, then the new target is added as a new retained target to the target list and the highest confidence level is set.

15. An electronic device, characterized in that it includes a processor and a memory; The memory stores program instructions; The processor is used to run the program instructions stored in the memory, causing the electronic device to perform the fish resource monitoring method as described in any one of claims 1 to 10.

Images