Fish species detection method and system based on video surveillance of ocean fishing vessels

Abstract

The present invention relates to the field of image enhancement technology, and more specifically to a fish species detection method and system based on video surveillance of distant-water fishing vessels. The present invention determines the analysis duration by analyzing changes in the degree of sea surface ripples. The present invention then analyzes the correlation between changes in bright lines on the fish's surface and, in combination with the degree of sea surface ripples, obtains a bright line attenuation coefficient for the bright lines on the fish's surface. Image enhancement is then performed to detect fish species. By analyzing the correspondence between changes in bright lines between different frames of a fish's swimming and changes in sea surface ripples, the present invention distinguishes bright lines on the fish from its own stripes, and unevenly enhances the image, making the fish's own stripes more prominent in the image. This improves the accuracy of fish species identification through video surveillance.

Description

Technical Field

[0001] The present invention relates to the technical field of image enhancement, and in particular to a fish species detection method and system based on video monitoring of ocean fishing vessels. Background Art

[0002] During distant-water fishing operations, distant-water fishing vessels identify fish species by their distinctive surface stripes. Marine fish species detection technology plays a vital role in fishery resource assessment, aquaculture, and marine ecosystem protection. However, traditional identification methods rely primarily on manual evaluation, which is inefficient and prone to errors. Therefore, neural network-based detection models are now being used to analyze the characteristics of fish surface stripes to identify species.

[0003] During deep-sea fishing, underwater video images captured by a device installed on the bottom of the boat are used to identify the unique stripes on the surface of fish bodies to detect and identify fish species. However, because fishing is carried out in the sea, the sea surface will produce waves due to sea breezes and the rotation of the earth. The fluctuations of the waves will refract light differently, which will be reflected on the fish to form bright stripes. This can easily interfere with the recognition of the fish's own stripes, resulting in errors in the recognition of fish surface stripes by the neural network-based detection model. Summary of the Invention

[0004] In order to solve the above technical problems, the purpose of the present invention is to provide a fish species detection method and system based on video monitoring of ocean fishing vessels.

[0005] According to a first aspect of an embodiment of the present invention, a method for detecting fish species based on video surveillance of a distant-water fishing vessel is provided, wherein the technical solution adopted is as follows:

[0006] Acquire multiple frames of continuous sea surface ripple images and underwater fish surface images;

[0007] Analyzing the fluctuation characteristics of the sea surface ripples based on the sea surface ripple image to obtain the fluctuation degree of the sea surface ripples at each moment;

[0008] Analyzing the changes in the fluctuation degree of the sea surface ripples at different times to obtain the fluctuation stability of the sea surface ripples at each time, and then determining the analysis time;

[0009] Within the analysis time range, based on the underwater fish body surface image, analyzing the temporal variation characteristics of the bright lines on the surface of each fish body, and obtaining the bright line variation correlation of the bright lines on the surface of each fish body;

[0010] Obtaining a bright line attenuation coefficient of each bright line on the surface of the fish body according to the fluctuation degree and the correlation between the bright line changes;

[0011] performing image enhancement on the underwater fish body surface image according to the bright fringe attenuation coefficient;

[0012] Based on the enhanced image, the neural network detection model is used to detect fish species.

[0013] In some embodiments of the present invention, analyzing the fluctuation characteristics of the sea surface ripples based on the sea surface ripple image to obtain the fluctuation degree of the sea surface ripples at each moment includes:

[0014] determining a plurality of ripples on the sea surface ripple image at each moment;

[0015] Analyzing the grayscale value differences of the pixels on the ripples to obtain the fluctuation amplitude of the ripples;

[0016] For the sea surface ripple image at each moment, all ripples are traversed to calculate the fluctuation amplitude to obtain the fluctuation degree of the sea surface ripples at each moment.

[0017] In some embodiments of the present invention, analyzing the change in the fluctuation degree of the sea surface ripples at different moments to obtain the fluctuation stability of the sea surface ripples at each moment includes:

[0018] Preset analysis duration;

[0019] Within the preset analysis time range, the difference between the fluctuation degree of the sea surface ripples at one moment and at other moments is analyzed, and the discrete degree of the fluctuation degree of the sea surface ripples at all moments is analyzed to obtain the fluctuation stability of the sea surface ripples at each moment.

[0020] In some embodiments of the present invention, determining the analysis duration includes:

[0021] According to the fluctuation stability, a bright pattern persistence coefficient of the bright pattern on the fish body surface is obtained;

[0022] The analysis duration is determined according to the bright streak persistence coefficient and the preset analysis duration.

[0023] In some embodiments of the present invention, within the analysis time range, based on the underwater fish body surface image, analyzing the temporal variation characteristics of the bright lines on the surface of each fish body to obtain the bright line variation correlation of the bright lines on the surface of each fish body includes:

[0024] Within the analysis time range, based on the underwater fish body surface image, the similarity of the brightness change of each fish body surface bright line and the similarity of the corner point position change in time sequence are analyzed to obtain the bright line change correlation of each fish body surface bright line.

[0025] In some embodiments of the present invention, within the analysis time range, analyzing the similarity of brightness changes of each bright pattern on the surface of the fish body in time sequence based on the underwater fish body surface image includes:

[0026] Obtaining a bright pattern connected domain of bright patterns on the surface of the fish body on the underwater fish body surface image;

[0027] Calculating the mean grayscale value of pixels in the connected domain of the bright lines to obtain the bright line grayscale value;

[0028] Within the analysis time range, the differences between the grayscale values ​​of the bright lines on the surface of each fish body at all adjacent moments are analyzed to obtain the similarity of the brightness changes of the bright lines on the surface of each fish body in time sequence.

[0029] In some embodiments of the present invention, within the analysis time range, analyzing the similarity of the corner position changes of each bright pattern on the surface of the fish body in time sequence based on the underwater fish body surface image includes:

[0030] Obtaining a coordinate sequence of corner points of bright lines on the surface of the fish body on the underwater fish body surface image;

[0031] Within the analysis time range, analyzing the degree of discreteness between the differences in the coordinate sequences of the corner points of each bright pattern on the surface of the fish body at all adjacent moments, and obtaining the similarity of the initial corner point position changes of each bright pattern on the surface of the fish body;

[0032] Within the analysis time range, on each bright pattern on the fish body surface, the difference between the mean distances between the coordinates of each corner point and the coordinates of other corner points at different times is analyzed to obtain the relative stability of the corner points of each bright pattern on the fish body surface;

[0033] The similarity of the initial corner point position changes is corrected by using the relative stability of the corner points to obtain the similarity of the corner point position changes of each bright pattern on the fish body surface in time sequence.

[0034] In some embodiments of the present invention, acquiring multiple frames of continuous sea surface ripple images and underwater fish surface images includes:

[0035] High-speed cameras are installed on and under the ocean-going fishing vessels to capture video surveillance of sea surface ripples and underwater fish surfaces.

[0036] The sea surface ripple video monitoring and the underwater fish body surface video monitoring are split separately to obtain multiple frames of continuous sea surface ripple images and underwater fish body surface images.

[0037] According to a second aspect of an embodiment of the present invention, a fish species detection system based on video surveillance of a distant-water fishing vessel is provided, comprising: a memory and a processor, wherein:

[0038] The memory is used to store program code;

[0039] The processor is configured to read the program code stored in the memory and execute the method described in the first aspect of the embodiment of the present invention.

[0040] In some embodiments of the present invention, the processor includes:

[0041] An image acquisition module is used to obtain multiple frames of continuous sea surface ripple images and underwater fish surface images;

[0042] a sea surface ripple analysis module configured to analyze the fluctuation characteristics of the sea surface ripples based on the sea surface ripple image to obtain the degree of fluctuation of the sea surface ripples at each moment; and to analyze the changes in the degree of fluctuation of the sea surface ripples at different moments to obtain the degree of fluctuation stability of the sea surface ripples at each moment, thereby determining the analysis duration;

[0043] a fish body surface bright pattern analysis module, configured to analyze, within the analysis time range, based on the underwater fish body surface image, the temporal variation characteristics of the bright pattern on the surface of each fish body, and obtain the bright pattern variation correlation of each fish body surface bright pattern; and to obtain the bright pattern attenuation coefficient of each fish body surface bright pattern based on the fluctuation degree and the bright pattern variation correlation;

[0044] An image enhancement module, configured to enhance the underwater fish body surface image according to the bright fringe attenuation coefficient;

[0045] The fish species detection module is used to detect fish species based on enhanced images using a neural network detection model.

[0046] Compared with the existing technology, the fish species detection method and system based on ocean fishing vessel video monitoring provided by the present invention have the following beneficial effects:

[0047] The present invention obtains the degree of fluctuation of the sea surface ripples at each moment by analyzing the fluctuation characteristics of the sea surface ripples; and obtains the degree of stability of the fluctuation of the sea surface ripples at each moment by analyzing the changes in the degree of fluctuation of the sea surface ripples at different moments, thereby determining the analysis time. The determination of the analysis time can effectively avoid the loss of information caused by selecting a short time range for research, and can also avoid the presence of redundant information in the image due to a long time range for selection, which interferes with detection. Within the analysis time range, based on the underwater fish body surface image, the temporal variation characteristics of the bright lines on the surface of each fish body are analyzed to obtain the correlation of the bright lines on the surface of each fish body; based on the degree of fluctuation and the correlation of the bright lines, the bright line attenuation coefficient of the bright lines on the surface of each fish body is obtained; based on the bright line attenuation coefficient, the underwater fish body surface image is enhanced; based on the enhanced image, the fish species are detected using a neural network detection model. The present invention analyzes the correspondence between the changes in the bright patterns between different frames of the fish's swimming process and the changes in the sea surface ripples, distinguishes the bright patterns on the fish from the fish's own stripes, and unevenly enhances the image, making the fish's own stripes more obvious in the image, thereby improving the accuracy of fish species identification through video monitoring. BRIEF DESCRIPTION OF THE DRAWINGS

[0048] In order to more clearly illustrate the technical solutions and advantages of the embodiments of the present invention or the prior art, the following briefly introduces the drawings required for use in the embodiments or the description of the prior art. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative work.

[0049] Figure 1 A schematic diagram of the basic flow of a method for detecting fish species based on video surveillance of a distant-water fishing vessel provided by one embodiment of the present invention;

[0050] Figure 2 A schematic diagram of edge detection results of a sea surface ripple image provided by one embodiment of the present invention;

[0051] Figure 3 A schematic diagram of edge detection results of an underwater fish surface image provided by one embodiment of the present invention;

[0052] Figure 4 A schematic diagram of bright lines on a fish's surface mapped after light is refracted by sea surface ripples, provided by one embodiment of the present invention;

[0053] Figure 5 A schematic diagram of the basic composition of a fish species detection system based on video surveillance of ocean-going fishing vessels provided by one embodiment of the present invention. DETAILED DESCRIPTION

[0054] To further illustrate the technical means and effectiveness of the present invention to achieve its intended purpose, the following, in conjunction with the accompanying drawings and preferred embodiments, describes in detail the specific implementation, structure, features, and effectiveness of the fish species detection method and system based on video surveillance of distant-water fishing vessels proposed by the present invention. In the following description, different references to "one embodiment" or "another embodiment" do not necessarily refer to the same embodiment. Furthermore, specific features, structures, or characteristics of one or more embodiments may be combined in any suitable manner.

[0055] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as commonly understood by those skilled in the art to which this invention pertains. Terms such as "comprises," "comprising," or any other variations thereof are intended to encompass non-exclusive inclusion, such that a circuit structure, article, or device comprising a series of elements includes not only those elements but also other elements not explicitly listed, or elements inherent to such article or device. In the absence of further limitations, the phrase "comprising a ..." to define an element does not preclude the presence of other identical elements in the article or device comprising the element.

[0056] The present invention specifically targets a scenario where deep-sea fishing vessels capture images of fish in the water using underwater cameras. The fish's surface stripes are then used to identify the species. However, the refraction of light by the waves can create bright streaks on the fish's surface, interfering with the identification of these stripes. Therefore, the present invention aims to reduce these bright streaks, enhance the surface stripes, and improve the accuracy of fish species identification.

[0057] The following describes in detail a method for detecting fish species based on video surveillance of ocean-going fishing vessels provided by the present invention with reference to the accompanying drawings.

[0058] See also Figure 1 , which shows the basic process of a fish species detection method based on video monitoring of ocean fishing vessels provided by an embodiment of the present invention.

[0059] like Figure 1 As shown, an embodiment of the present invention provides a method for detecting fish species based on video surveillance of ocean-going fishing vessels, which specifically includes:

[0060] S100: Acquire multiple frames of continuous sea surface ripple images and underwater fish surface images.

[0061] Acquire multiple continuous frames of sea surface ripple images and underwater fish surface images. A specific implementation method is as follows: high-speed cameras are deployed on and under ocean-going fishing vessels to capture high-precision video surveillance of sea surface ripples and underwater fish surface images during ocean-going fishing (i.e., simultaneously obtain video surveillance images from two perspectives); the captured video surveillance is transmitted to a fish species detection system in real time, and the sea surface ripple video surveillance and underwater fish surface video surveillance are split at a splitting density of 30 frames per second to obtain multiple continuous frames of sea surface ripple images and underwater fish surface images; the sea surface ripple images and underwater fish surface images are grayscaled to obtain sea surface ripple grayscale images and underwater fish surface grayscale images. It should be noted that in the subsequent description, unless otherwise specified, sea surface ripple images refer to sea surface ripple grayscale images, and underwater fish surface images refer to underwater fish surface grayscale images.

[0062] At this point, multiple frames of continuous sea surface ripple images and underwater fish surface images are obtained.

[0063] During deep-sea fishing, underwater video images captured by a high-speed camera mounted on the bottom of a boat are used to detect and identify fish species by identifying the distinctive stripes on the fish's body surface. However, due to winds and the Earth's motion, the sea surface creates waves. These wave fluctuations refract light differently, projecting onto the underwater fish's surface as bright streaks, which can interfere with the identification of the fish's stripes. The bright streaks created by light on the fish's surface fluctuate accordingly with the waves, and the fluctuations in these bright streaks are significantly correlated with the waves. Therefore, by analyzing the correlation between the bright streaks and the ripples caused by wave fluctuations, the better the correlation, the more likely they are light streaks caused by light refraction. Conversely, a negative correlation indicates that the bright streaks are likely bright streaks on the fish's body. The image is then adaptively enhanced to identify the fish species. Specifically, the process includes steps S200 to S700.

[0064] S200: Analyze the fluctuation characteristics of the sea surface ripples based on the sea surface ripple image to obtain the fluctuation degree of the sea surface ripples at each moment.

[0065] The generation of sea surface ripples is caused by sea breeze and the rotation of the earth. Therefore, the texture characteristics of sea surface ripples are relatively regular, that is, sea surface ripples have obvious periodic fluctuation patterns, and the periodic changes of sea surface ripples do not vary with the propagation position. The fluctuation amplitude of sea surface ripples at different positions is basically the same; in the same sea area, the fluctuation degree of sea surface ripples is only related to the size of the sea surface wind at that moment, that is to say, the fluctuation regularity of sea surface ripples is relatively strong.

[0066] Therefore, in an embodiment of the present invention, based on the sea surface ripple image, the fluctuation characteristics of the sea surface ripples are analyzed to obtain the fluctuation degree of the sea surface ripples at each moment.

[0067] First, multiple ripples are determined on the sea surface ripple image at each moment. The specific implementation method is: perform Sobel edge detection on the sea surface ripple image to obtain the edge curve of the sea surface ripple, such as Figure 2 As shown, one of the edge curves represents a ripple.

[0068] Then, the grayscale value differences of the pixels on the ripples are analyzed to obtain the fluctuation amplitude of the ripples. The specific implementation method is: morphologically dilate the edge curve of the sea surface ripples to obtain the highlighted area after expansion, traverse and calculate the pixel with the largest grayscale value in the highlighted area, and the difference between the grayscale value of the pixel and the grayscale value of the pixel in the highlighted area is a positive value, which is the bright spot (that is, the reflection at the location of the pixel is the largest, and the grayscale value of the pixel is the largest). The fluctuation pattern of the sea surface ripples is like a sine wave. The bright spot of the sea surface ripples is the maximum point of the sine wave. As the sea surface ripples fluctuate, the convexity of the ripples turns into concavity, and the darkest point of the ripples is the minimum point of the sine wave. Because the fluctuations of the sea surface ripples are uniform and similar, 1 / 2 of the maximum difference in the grayscale values ​​of the pixels on the sea surface ripples is used as the fluctuation amplitude of the sea surface ripples.

[0069] Finally, for the sea surface ripple image at each moment, all ripples are traversed to calculate the fluctuation amplitude and obtain the degree of fluctuation of the sea surface ripple at each moment. The calculation formula for the fluctuation degree of sea surface ripples at a certain moment is:

[0070]

[0071] Where, Indicates the The degree of fluctuation of the sea surface ripples in the sea surface ripple image at each moment; Indicates the The number of ripples on the sea surface ripple graph at any moment; Indicates the The first moment on the sea surface ripple image The maximum grayscale value of a ripple; Indicates the The first moment on the sea surface ripple image The minimum gray value of a ripple; represents the linear normalization function.

[0072] Indicates the amplitude of a ripple. Indicates the The average fluctuation amplitude of the ripples on the sea surface ripple graph at that moment. The larger the value, the greater the fluctuation degree of the sea surface ripples at that moment.

[0073] S300: Analyze the changes in the fluctuation degree of the sea surface ripples at different times, obtain the fluctuation stability of the sea surface ripples at each time, and then determine the analysis time.

[0074] Different degrees of sea surface ripples have varying durations, resulting in different durations of bright streaks reflected from these waves onto the fish surface. Therefore, when analyzing underwater fish surface images, the time range for analyzing the degree of sea ripples needs to be adjusted based on the ripples' fluctuations to avoid information loss from a shorter timeframe and redundant information from a longer timeframe, which could interfere with species detection.

[0075] Based on the above analysis, in an embodiment of the present invention, by analyzing the changes in the fluctuation degree of the sea surface ripples at different times, the fluctuation stability of the sea surface ripples at each time is obtained, and then the analysis time is determined.

[0076] First, preset the analysis time. Specifically, according to the characteristics of fish swimming and the changes in the degree of fluctuation of sea surface ripples, the preset analysis time is , 1s corresponds to 30 frames of images, and the number of frame moments corresponding to the preset analysis time is recorded as X.

[0077] For example, within a given time length (such as 1 second), the total number of frames is calculated based on the frame rate (FPS), and the timestamp (moment) corresponding to each frame is determined.

[0078] Then, according to the regularity of the sea surface ripple fluctuation, its energy attenuation is less and the ripple fluctuation degree is stable, so the fluctuation stability of the sea surface ripple is calculated. Since the smaller the change in the fluctuation degree of the sea surface ripple between different frames, the greater the fluctuation stability of the ripple. Therefore, within the preset analysis time range, the difference between the fluctuation degree of the sea surface ripple at one moment and other moments is analyzed, as well as the discrete degree of the fluctuation degree of the sea surface ripple at all moments is analyzed to obtain the fluctuation stability of the sea surface ripple at each moment. Construct the first The calculation formula for the stability of sea surface ripples at a given moment is:

[0079]

[0080] Where, Indicates the The stability of sea surface ripples at all times; Indicates the The variance of the fluctuation degree of all ripples on the sea surface ripple image at that moment; X represents the number of frame moments corresponding to the preset analysis time; Indicates the The degree of fluctuation of the sea surface ripples in the sea surface ripple image at each moment; Indicates the The degree of fluctuation of the sea surface ripples in the sea surface ripple image at the moment The time is Always moving forward moments within a duration range; Represents natural base An exponential function with base .

[0081] The variance of the ripple degree on the sea surface ripple image is used as the weight to calculate the The difference in the degree of fluctuation of the sea surface ripples at this moment and the sea surface ripples at other moments, the smaller the difference in the degree of fluctuation, the smaller the degree of attenuation of the fluctuation amplitude during the propagation of the ripples, that is, the greater the degree of fluctuation stability of the sea surface ripples.

[0082] The analysis duration is determined based on the stability of the sea surface ripples. Further analysis includes:

[0083] First, the persistence coefficient of the bright lines on the fish's body is obtained based on the degree of wave stability. Specifically, because there are ripples of different sizes on the sea surface, some are larger, their wave stability is better, and they exist for a longer time; some are smaller, their wave stability is poor, and they exist for a shorter time. Therefore, the duration of the bright lines produced by the refraction of light on the fish's body by different ripples is also different. Therefore, based on the degree of wave stability of the sea surface ripples, the persistence coefficient of the bright lines on the fish's body is obtained as:

[0084]

[0085] Where, Indicates the The bright pattern persistence coefficient of the bright pattern on the fish body at this moment; Indicates the The stability of sea surface ripples at all times; represents the normalization function; it should be noted that the addition of 0.4 is to make The value range of is distributed around 1, among which The value range of is (0, 1], then The value range of is (0.5, 0.731], then The value range is (0.9, 1.131].

[0086] No. The greater the degree of stability of the sea surface ripples at a certain moment, the longer the sea surface ripples last at that moment. The longer the duration of the bright pattern on the fish body at a certain moment, the greater the bright pattern persistence coefficient of the bright pattern on the fish body at that moment.

[0087] Then, based on the bright streak persistence coefficient and the preset analysis duration, the analysis duration is determined as:

[0088]

[0089] Where, Indicates the The analysis time corresponding to the sea surface ripple image at that moment; Indicates the preset analysis duration; Indicates the The bright pattern persistence coefficient of the bright pattern on the fish body at all times.

[0090] when When The research time of the underwater fish body surface image at the moment is enhanced, that is, extended, otherwise it means that the first The time spent studying the surface image of the underwater fish body is weakened, that is, shortened.

[0091] At this point, the analysis time for the underwater fish body surface image is determined.

[0092] S400: Within the analysis time range, based on the underwater fish body surface image, analyzing the temporal variation characteristics of the bright lines on the surface of each fish body, and obtaining the bright line variation correlation of the bright lines on the surface of each fish body.

[0093] During the analysis timeframe (the duration of the bright streak), light changes minimally, and the degree of variation in the bright streaks reflected on the fish by the refraction of sea surface ripples is relatively small. The temporal similarity of these bright streak variations is therefore high. In other words, within the duration, the similarity in the changes in the bright streaks on the fish's body is high. When the bright streaks on the fish's body exhibit similar changes within the duration, i.e., the greater the temporal correlation of the bright streaks, the more likely they are caused by the refraction of light through sea surface ripples, and the greater the need to mitigate these changes.

[0094] Based on the above analysis, in an embodiment of the present invention, within the analysis duration, based on the underwater fish surface image, the temporal variation characteristics of the bright lines on each fish's body surface are analyzed to obtain the correlation of the bright line variation of each fish's body surface bright line. Furthermore, within the analysis duration, based on the underwater fish surface image, the temporal similarity of the brightness variation and the similarity of the corner position variation of each fish's body surface bright line are analyzed to obtain the correlation of the bright line variation of each fish's body surface bright line.

[0095] Within the analysis time range, based on the underwater fish surface image, the similarity of the brightness change of each fish surface bright pattern in time sequence is analyzed. The specific implementation method is as follows:

[0096] First, on the underwater fish surface image, the bright pattern connected domain of the bright patterns on the fish body surface is obtained. More specifically, the bright patterns formed on the fish body by refraction in water usually appear as wavy stripes or locally concentrated bright spots. The bright patterns on the fish body are brighter than the stripes on the fish body surface. Therefore, the canny edge detection is used to obtain the bright pattern connected domain of the bright patterns on the underwater fish body surface image, that is, to obtain the bright patterns on the fish body surface, such as Figure 3 shown.

[0097] Then, the mean grayscale value of all pixels in the bright fringe connected domain is calculated to obtain the bright fringe grayscale value.

[0098] Finally, within the duration of the bright ripples, the fluctuation stability of the sea surface ripples is relatively good, and their refraction of light is relatively stable, that is, the brightness changes of the bright ripples produced by their light have a certain similarity between adjacent frames. Therefore, within the analysis time range, the differences between the bright ripple grayscale values ​​of each fish body surface bright ripple at all adjacent moments are analyzed to obtain the similarity of the brightness changes of each fish body surface bright ripple in the time series. The calculation formula for the similarity of the brightness changes of the bright lines on the surface of the fish body in time series is:

[0099]

[0100] Where, Represents the bright lines on the surface of the fish in the underwater fish surface image Similarity of brightness changes in time series; Indicates the Bright lines on the fish surface in the underwater fish surface image at this moment Gray value of bright lines; Indicates the The first image on the surface of the underwater fish body at this moment The grayscale value of the bright lines on the surface of the fish body; Indicates the The analysis time of the sea surface ripple image at the moment Moment and The time is Time is the starting point forward A moment within a duration range; Represents natural base An exponential function with base .

[0101] Indicates the Time and adjacent Bright lines on the fish's surface at the moment The smaller the difference in the grayscale mean of the bright lines at adjacent moments, the smaller the difference in the brightness change of the bright lines. When the difference in the brightness change of the bright lines is smaller within the entire analysis time (bright line duration), the greater the similarity of the brightness change of the bright lines on the fish body surface.

[0102] Because the bright lines caused by light refraction usually appear as stripes or bright spots, there are sharp ripple corners in the shape of the bright lines. Within the duration of the bright lines, the ripple corners of the bright lines on the fish body are similar, that is, the correlation between the changes in the ripple corners in time series is good.

[0103] Therefore, within the analysis time range, based on the underwater fish body surface image, the similarity of the corner position changes of each bright line on the fish body surface in time sequence is analyzed. The specific implementation method is as follows:

[0104] First, on the underwater fish surface image, obtain the corner point coordinate sequence of the fish surface bright lines. More specifically, for the fish surface bright lines in the underwater fish surface image at all moments within the analysis time range, Use Harris corner detection algorithm to get the bright lines on the fish body surface The coordinate sequence of the corner points.

[0105] Then, within the analysis time range, the degree of dispersion of the corner point coordinate sequence differences of each fish body surface bright pattern at all adjacent moments is analyzed to obtain the similarity of the initial corner point position changes of each fish body surface bright pattern. More specifically, for the fish body surface bright patterns in the underwater fish body surface image within the analysis time range, , calculate the difference of the corner point coordinate sequence corresponding to all adjacent moments, and then calculate the variance between all differences to obtain the bright pattern on the fish body surface The degree of discreteness between the differences in the coordinate sequences of the corner points at all adjacent moments is taken as the inverse of the discreteness between the differences to obtain the similarity of the initial corner point position changes of the bright lines on the surface of each fish body, which is recorded as . The larger it is, the more stable the changes in the corner points of the bright fringe are, that is, the greater the correlation between the changes in the bright fringe and the changes in illumination.

[0106] Because the fish is swimming during the duration of the bright pattern, the bending of the fish's trunk, that is, the change of the fish's spine, will interfere with the corner point matching, causing the position of the corner points of the bright pattern on the fish's surface to change, thereby affecting the variance of the corner point coordinate changes. Therefore, it is necessary to correct the similarity of the initial corner point position changes.

[0107] The formation of bright lines on the surface of the fish body is only related to the refraction of light by the fluctuation of waves. The bright lines on the surface of the fish body are caused by the ups and downs of the sea surface ripples, which cause angles on the sea surface and produce uneven refraction, which is mapped to the fish body to form bright lines. The relative positions of the angles between the bright lines are only related to the angle of refraction of light, that is, the relative positions of the corner points of the bright lines are only related to the fluctuation of the ripples. Therefore, within the duration of the bright lines, that is, the time range when the waves are stable, the relative positions of the corner points of the bright lines on the fish body change little.

[0108] Therefore, within the analysis time range, on each bright pattern on the fish body surface, the difference between the mean distances between each corner coordinate and other corner coordinates at different times is analyzed to obtain the relative stability of each corner of the bright pattern on the fish body surface. Bright lines on the fish body surface No. As an example, the first corner point is calculated. From the corner point to the bright line on the fish body The mean distance of the remaining corner points is recorded as , traversal analysis time For all moments in the range, calculate the The average distance of the corner points is obtained during the analysis time. Within the scope of The similarity of the mean distance of the corner points. Within the scope of Similarity of the mean distances of the corner points , expressed as:

[0109]

[0110] Where, Indicates the analysis time Bright lines on the fish surface within the range No. The similarity of the mean distance of the corner points; Indicates the The analysis time corresponding to the sea surface ripple image at that moment; Indicates the Bright lines on the fish body surface No. From the corner point to the bright line on the fish body The mean distance of the remaining corner points; Indicates the Bright lines on the fish body surface No. From the corner point to the bright line on the fish body The mean distance of the remaining corner points; Indicates the natural base The prime function with base .

[0111] Indicates the The difference in the relative positions of the corner points at different times. The smaller the value, the more stable the relative position of the corner point within the duration of the bright pattern. Indicates that within the duration of the bright pattern, the bright pattern on the fish body surface No. The relative stability of the position of each corner point. The larger the value, the better the relative stability of the position of the corner point.

[0112] Similarly, calculate the analysis time Bright lines on the fish surface within the range The similarity of the distance mean of the remaining corner points is normalized after accumulation and expressed as the bright lines on the fish body surface The relative stability of the corner points is expressed as:

[0113]

[0114] Where, Indicates bright lines on the fish's surface The relative stability of the corner points; Indicates bright lines on the fish's surface No. Corner points in the analysis time Similarity of distance means within the range; Indicates bright lines on the fish's surface Total number of upper corner points; represents the linear normalization function.

[0115] Finally, the similarity of the initial corner position changes is corrected by the relative stability of the corner points, and the similarity of the corner position changes of each bright pattern on the fish body surface in the time series is obtained, that is:

[0116]

[0117] Where, Indicates bright lines on the fish's surface The similarity of the corner position changes in time series; Indicates bright lines on the fish's surface Similarity of the changes in the initial corner positions in time series; Indicates bright lines on the fish's surface The relative stability of the corner points; represents the linear normalization function.

[0118] Similarity of corner position changes It reflects the degree of illumination correlation of corner position changes. The larger the value, the greater the correlation between the corner position change and illumination.

[0119] Finally, the fish body surface bright lines Similarity of corner position changes in time series Bright lines on the fish's surface Similarity of brightness changes in time series The weight of the fish body is used to obtain the bright lines on the surface of the fish body. The correlation of bright fringe changes is:

[0120]

[0121] Where, Indicates bright lines on the fish's surface Correlation of bright streak changes; Indicates bright lines on the fish's surface The similarity of the corner position changes in time series; Represents the bright lines on the surface of the fish in the underwater fish surface image The similarity of brightness changes in time series.

[0122] During analysis Within the range (the duration range of the bright lines), the greater the correlation of the bright line changes, the greater the correlation between the bright lines on the fish body surface and the refraction of light, that is, the greater the possibility that the bright lines on the fish body surface are caused by sea surface ripples, and the greater the degree of weakening that needs to be done.

[0123] S500: Obtaining a bright pattern attenuation coefficient of each bright pattern on the surface of the fish body according to the fluctuation degree and the correlation between the bright pattern changes.

[0124] The fluctuation of sea surface ripples will interfere with the propagation of light. The fluctuation of sea surface ripples will change the reflection of light, causing the light refracted to the fish in the sea water to be unevenly distributed, thus forming bright lines of various shapes. The brightness of these bright lines is significantly correlated with the fluctuation of sea surface ripples. The greater the fluctuation of sea surface ripples, the greater the reflection intensity of ripples on light, and the brighter the brightness of the bright lines mapped to the surface of fish body. Figure 4 As shown, the greater the degree of weakening of the bright lines on the fish body surface is, the greater the degree of weakening is.

[0125] Based on the above analysis, according to the correlation between the fluctuation degree and the change of bright lines, the bright line weakening coefficient of each fish body surface bright line is obtained. For example, first calculate the analysis time The average fluctuation degree of the sea surface ripples on the sea surface ripple image at all times within the range (the duration of the bright ripples) is recorded as , and then analyze the duration The corresponding fluctuation degree within the range is weighted to weight the correlation of the bright pattern change, and the bright pattern on the fish body surface is obtained. The bright fringe attenuation coefficient is:

[0126]

[0127] Where, Indicates bright lines on the fish's surface Bright streak attenuation coefficient; Indicates analysis duration The average fluctuation degree of the sea surface ripples on the sea surface ripple image at all times within the range; Indicates bright lines on the fish's surface Correlation of bright streak changes; represents the linear normalization function.

[0128] The degree of fluctuation of sea surface ripples The larger the value, the greater the reflection intensity of the ripples on the light, the brighter the brightness of the bright lines mapped to the surface of the fish body, and the greater the degree of weakening of the bright lines on the surface of the fish body; Correlation of bright line changes The larger the value is, the greater the correlation between the bright lines on the fish body surface and the refraction of light, that is, the greater the possibility that the bright lines on the fish body surface are caused by sea surface ripples, and the greater the degree of weakening is needed; therefore The larger it is, the more likely it is a bright streak caused by fluctuations on the sea surface, and the more it needs to be weakened.

[0129] S600: Perform image enhancement on the underwater fish surface image based on the bright ripple attenuation coefficient.

[0130] The bright pattern attenuation coefficients of bright patterns on the surfaces of different fish bodies are used as the enhancement ratios of bright patterns in the image. The existing algorithm is used to adaptively enhance the bright patterns in the image to obtain the enhanced image.

[0131] S700: Fish species detection using a neural network detection model based on enhanced images.

[0132] The enhanced image is fed into a neural network detection model to identify regions of interest, enabling fish species detection. Because the fish stripes are enhanced while the remaining bright areas are weakened through adaptive image enhancement, the neural network detection model can better identify the fish's active areas, thereby improving the neural network detection model's ability to detect fish species.

[0133] The specific operational steps for detecting fish species in images based on the neural network detection model are as follows: obtain a priori data sample set to ensure that fish images of different angles, lighting conditions and growth stages are covered; annotate the sample set, use rectangular boxes to annotate different fish species, and use the VIA annotation platform to generate XML or JSON format annotation files; label partitioning, label the images according to fish species, and select 70% of the data set for training and 30% for data verification in the training set-validation set ratio, and pre-train fish species detection based on the EfficientNet model; neural network selection, use the Faster R-CNN neural network target detection model; loss function, use the cross-entropy loss function; train the output result digital code, use different digital codes to represent different fish species, and the returned result is the digital code representing the fish species; input the current enhanced image, perform target detection based on the Faster R-CNN neural network, and obtain the output detection result; match the detection result to the species to complete the fish species detection.

[0134] Based on the same inventive concept as the above method, this embodiment also provides a fish species detection system based on video monitoring of ocean fishing vessels.

[0135] See also Figure 5 , which shows the basic composition of a fish species detection system based on video surveillance of ocean fishing vessels provided by an embodiment of the present invention.

[0136] like Figure 5 As shown, a fish species detection system based on video surveillance of ocean fishing vessels includes: a memory 10 and a processor 20, wherein:

[0137] Memory 10, for storing program code;

[0138] The processor 20 is used to read the program code stored in the memory 10 and execute it to obtain multiple frames of continuous sea surface ripple images and underwater fish body surface images; based on the sea surface ripple images, analyze the fluctuation characteristics of the sea surface ripples to obtain the fluctuation degree of the sea surface ripples at each moment; analyze the changes in the fluctuation degree of the sea surface ripples at different moments to obtain the fluctuation stability of the sea surface ripples at each moment, and then determine the analysis time; within the analysis time, based on the underwater fish body surface image, analyze the time-series change characteristics of the bright lines on the surface of each fish body to obtain the bright line change correlation of the bright lines on the surface of each fish body; according to the fluctuation degree and the bright line change correlation, obtain the bright line attenuation coefficient of the bright lines on the surface of each fish body; according to the bright line attenuation coefficient, perform image enhancement on the underwater fish body surface image; based on the enhanced image, use the neural network detection model to detect fish species.

[0139] Furthermore, the processor 20 includes an image acquisition module 21, a sea surface ripple analysis module 22, a fish surface bright pattern analysis module 23, an image enhancement module 24, and a fish species detection module 25. Among them:

[0140] An image acquisition module 21 is used to acquire multiple frames of continuous sea surface ripple images and underwater fish surface images;

[0141] The sea surface ripple analysis module 22 is used to analyze the fluctuation characteristics of the sea surface ripples based on the sea surface ripple image to obtain the fluctuation degree of the sea surface ripples at each moment; and to analyze the changes in the fluctuation degree of the sea surface ripples at different moments to obtain the fluctuation stability of the sea surface ripples at each moment, thereby determining the analysis time;

[0142] The fish surface bright pattern analysis module 23 is configured to analyze the temporal variation characteristics of the bright patterns on the surface of each fish body based on the underwater fish body surface image within the analysis time range, and obtain the bright pattern variation correlation of each fish body surface bright pattern; and to obtain the bright pattern attenuation coefficient of each fish body surface bright pattern based on the fluctuation degree and the bright pattern variation correlation;

[0143] An image enhancement module 24 is used to enhance the underwater fish surface image according to the bright fringe attenuation coefficient;

[0144] The fish species detection module 25 is used to detect fish species based on the enhanced image using a neural network detection model.

[0145] It should be noted that the order in which the embodiments of the present invention are described above is for illustrative purposes only and does not necessarily represent the superiority or inferiority of the embodiments. The processes depicted in the accompanying drawings do not necessarily require the specific order or sequential order shown to achieve the desired results. In certain embodiments, multitasking and parallel processing are also possible or may be advantageous.

[0146] The various embodiments in this specification are described in a progressive manner. The same or similar parts between the various embodiments can be referred to each other. Each embodiment focuses on the differences from other embodiments.

Claims

1. A method for detecting fish species based on video surveillance of ocean-going fishing vessels, characterized in that: The method comprises: Acquire multiple frames of continuous sea surface ripple images and underwater fish surface images; Analyzing the fluctuation characteristics of the sea surface ripples based on the sea surface ripple image to obtain the fluctuation degree of the sea surface ripples at each moment; Analyzing the changes in the fluctuation degree of the sea surface ripples at different times to obtain the fluctuation stability of the sea surface ripples at each time, and then determining the analysis time; Within the analysis time range, based on the underwater fish body surface image, analyzing the temporal variation characteristics of the bright lines on the surface of each fish body, and obtaining the bright line variation correlation of the bright lines on the surface of each fish body; Obtaining a bright line attenuation coefficient of each bright line on the surface of the fish body according to the fluctuation degree and the correlation between the bright line changes; performing image enhancement on the underwater fish body surface image according to the bright fringe attenuation coefficient; Based on the enhanced image, the neural network detection model is used to detect fish species.

2. The method for detecting fish species based on video surveillance of ocean-going fishing vessels according to claim 1, characterized in that: Based on the sea surface ripple image, the fluctuation characteristics of the sea surface ripples are analyzed to obtain the fluctuation degree of the sea surface ripples at each moment, including: determining a plurality of ripples on the sea surface ripple image at each moment; Analyzing the grayscale value differences of the pixels on the ripples to obtain the fluctuation amplitude of the ripples; For the sea surface ripple image at each moment, all ripples are traversed to calculate the fluctuation amplitude to obtain the fluctuation degree of the sea surface ripples at each moment.

3. The method for detecting fish species based on video surveillance of ocean-going fishing vessels according to claim 2, characterized in that: Analyze the changes in the fluctuation degree of the sea surface ripples at different times to obtain the fluctuation stability of the sea surface ripples at each time, including: Preset analysis duration; Within the preset analysis time range, the difference between the fluctuation degree of the sea surface ripples at one moment and at other moments is analyzed, and the discrete degree of the fluctuation degree of the sea surface ripples at all moments is analyzed to obtain the fluctuation stability of the sea surface ripples at each moment.

4. The method for detecting fish species based on video surveillance of ocean-going fishing vessels according to claim 3, characterized in that: Determine the length of the analysis, including: According to the fluctuation stability, a bright pattern persistence coefficient of the bright pattern on the fish body surface is obtained; The analysis duration is determined according to the bright streak persistence coefficient and the preset analysis duration.

5. The method for detecting fish species based on video surveillance of ocean-going fishing vessels according to claim 1, characterized in that: Within the analysis time range, based on the underwater fish body surface image, analyzing the temporal variation characteristics of the bright lines on the surface of each fish body, and obtaining the bright line variation correlation of the bright lines on the surface of each fish body, including: Within the analysis time range, based on the underwater fish body surface image, the similarity of the brightness change of each fish body surface bright line and the similarity of the corner point position change in time sequence are analyzed to obtain the bright line change correlation of each fish body surface bright line.

6. The method for detecting fish species based on video surveillance of ocean-going fishing vessels according to claim 5, characterized in that: Within the analysis time range, based on the underwater fish body surface image, analyzing the similarity of brightness changes of each bright pattern on the fish body surface in time sequence, including: Obtaining a bright pattern connected domain of bright patterns on the surface of the fish body on the underwater fish body surface image; Calculating the mean grayscale value of pixels in the connected domain of the bright lines to obtain the bright line grayscale value; Within the analysis time range, the differences between the grayscale values ​​of the bright lines on the surface of each fish body at all adjacent moments are analyzed to obtain the similarity of the brightness changes of the bright lines on the surface of each fish body in time sequence.

7. The method for detecting fish species based on video surveillance of ocean-going fishing vessels according to claim 5, characterized in that: Within the analysis time range, based on the underwater fish body surface image, analyzing the similarity of the corner point position changes of each bright pattern on the fish body surface in time sequence, including: Obtaining a coordinate sequence of corner points of bright lines on the surface of the fish body on the underwater fish body surface image; Within the analysis time range, analyzing the degree of discreteness between the differences in the coordinate sequences of the corner points of each bright pattern on the surface of the fish body at all adjacent moments, and obtaining the similarity of the initial corner point position changes of each bright pattern on the surface of the fish body; Within the analysis time range, on each bright pattern on the fish body surface, the difference between the mean distances between the coordinates of each corner point and the coordinates of other corner points at different times is analyzed to obtain the relative stability of the corner points of each bright pattern on the fish body surface; The similarity of the initial corner point position changes is corrected by using the relative stability of the corner points to obtain the similarity of the corner point position changes of each bright pattern on the fish body surface in time sequence.

8. The method for detecting fish species based on video surveillance of ocean-going fishing vessels according to claim 1, characterized in that: Acquire multiple frames of continuous sea surface ripple images and underwater fish surface images, including: High-speed cameras are installed on and under the ocean-going fishing vessels to capture video surveillance of sea surface ripples and underwater fish surfaces. The sea surface ripple video monitoring and the underwater fish body surface video monitoring are split separately to obtain multiple frames of continuous sea surface ripple images and underwater fish body surface images.

9. A fish species detection system based on video surveillance of ocean-going fishing vessels, characterized in that: The system comprises: a memory and a processor, wherein: The memory is used to store program code; The processor is configured to read the program code stored in the memory and execute the method according to any one of claims 1 to 8.

10. The fish species detection system based on ocean fishing vessel video monitoring according to claim 9, characterized in that: The processor includes: An image acquisition module is used to obtain multiple frames of continuous sea surface ripple images and underwater fish surface images; a sea surface ripple analysis module configured to analyze the fluctuation characteristics of the sea surface ripples based on the sea surface ripple image to obtain the degree of fluctuation of the sea surface ripples at each moment; and to analyze the changes in the degree of fluctuation of the sea surface ripples at different moments to obtain the degree of fluctuation stability of the sea surface ripples at each moment, thereby determining the analysis duration; a fish body surface bright pattern analysis module, configured to analyze, within the analysis time range, based on the underwater fish body surface image, the temporal variation characteristics of the bright pattern on the surface of each fish body, and obtain the bright pattern variation correlation of each fish body surface bright pattern; and to obtain the bright pattern attenuation coefficient of each fish body surface bright pattern based on the fluctuation degree and the bright pattern variation correlation; An image enhancement module, configured to enhance the underwater fish body surface image according to the bright fringe attenuation coefficient; The fish species detection module is used to detect fish species based on enhanced images using a neural network detection model.

Images