River fishing-prohibited-ship monitoring method and system based on visible light and infrared fusion

Abstract

The application provides a river fishing-ban-ship monitoring method and system based on visible light and infrared fusion, and relates to the technical field of computer vision. First, visible light image sequences and infrared image sequences of a river monitoring area are acquired. Then, visible light image sequences and infrared image sequences are subjected to ship visual feature extraction and thermal feature extraction respectively to obtain visible light ship feature sets and infrared ship feature sets. Cross-source feature fusion processing is performed on the two feature sets to generate a cross-source fusion feature set. A pre-trained fishing-ban-ship identification model is called to analyze the cross-source fusion feature set to generate a ship monitoring result containing a ship type, a position change track and a suspected fishing-ban behavior identifier. Finally, fishing-ban warning information containing a real-time position of a suspected fishing-ban ship and a behavior feature description of the suspected fishing-ban ship is generated based on the ship monitoring result, which helps to quickly take measures to stop fishing-ban behavior.

Description

Technical Field

[0001] This invention relates to the field of computer vision technology, and more specifically, to a method and system for monitoring fishing vessels in river channels based on the fusion of visible light and infrared light. Background Technology

[0002] In river ecological protection, fishing ban management is an important measure to ensure the ecological balance of aquatic areas and maintain the sustainable development of fishery resources. Accurate and timely monitoring of vessel activities in the river and identifying whether fishing activities are prohibited are key aspects of fishing ban management.

[0003] Currently, common methods for monitoring river vessels mainly rely on data from a single type of sensor. For example, monitoring methods based on visible light cameras can clearly capture the texture details of the river surface and its surrounding environment, such as the shape and color of the vessels, thus enabling preliminary visual identification. However, the quality of visible light images deteriorates significantly at night or in adverse weather conditions (such as dense fog or heavy rain), making vessel identification difficult and prone to missed or false detections.

[0004] Another method is based on infrared sensors. Infrared image sequences can reflect the thermal radiation distribution characteristics within the river area, and can detect vessels with heat sources even in low-light conditions. However, infrared images lack rich texture information and have limited ability to identify detailed features and types of vessels, making it difficult to accurately distinguish different types of vessels and determine their specific behaviors. Summary of the Invention

[0005] In view of the aforementioned problems, and in conjunction with the first aspect of the present invention, the present invention provides a method for monitoring fishing vessels in river channels based on visible light and infrared fusion, the method comprising:

[0006] Acquire visible light image sequences and infrared image sequences of the river monitoring area. The visible light image sequences contain texture details of the river surface and surrounding environment, and the infrared image sequences contain thermal radiation distribution characteristics within the river area.

[0007] The visible light image sequence is processed to extract ship visual features to obtain a visible light ship feature set, and the infrared image sequence is processed to extract thermal features to obtain an infrared ship feature set.

[0008] Perform cross-source feature fusion processing on the visible light ship feature set and the infrared ship feature set to generate a cross-source fused feature set;

[0009] The pre-trained fishing ban vessel identification model is invoked to perform vessel attribute and behavior analysis on the cross-source fusion feature set, generating vessel monitoring results that include vessel type, location change trajectory, and suspected fishing ban behavior identifiers;

[0010] Based on the vessel monitoring results, a fishing ban early warning information is generated, which includes the real-time location and behavioral characteristics description of suspected vessels subject to the fishing ban.

[0011] In another aspect, the present invention also provides a river channel fishing ban vessel monitoring system based on visible light and infrared fusion, including a processor and a machine-readable storage medium connected to the processor. The machine-readable storage medium is used to store programs, instructions or code, and the processor is used to execute the programs, instructions or code in the machine-readable storage medium to implement the above-mentioned method.

[0012] Based on the above, this invention acquires visible light image sequences and infrared image sequences of the river monitoring area, fully leveraging the advantages of visible light images containing detailed texture features of the river surface and surrounding environment, and infrared images containing thermal radiation distribution features. Visual features of vessels are extracted from the visible light image sequences to obtain a visible light vessel feature set, and thermal features are extracted from the infrared image sequences to obtain an infrared vessel feature set. This allows for the accurate capture of key vessel features from different dimensions. Cross-source feature fusion processing is performed on the two feature sets to generate a cross-source fused feature set. A pre-trained fishing ban vessel identification model is then used to analyze vessel attributes and behaviors on the cross-source fused feature set. This accurately generates vessel monitoring results containing vessel type, location change trajectory, and suspected fishing ban behavior identifiers, significantly improving the accuracy and comprehensiveness of vessel monitoring. Based on the vessel monitoring results, fishing ban early warning information containing real-time location and behavioral characteristics descriptions of suspected fishing ban vessels is generated, facilitating rapid measures to stop fishing ban behaviors and effectively protecting the river's ecological environment and fishery resources. Attached Figure Description

[0013] Figure 1 This is a schematic diagram of the execution flow of the river fishing ban vessel monitoring method based on visible light and infrared fusion provided in the embodiments of the present invention.

[0014] Figure 2 This is a schematic diagram of exemplary hardware and software components of a river channel fishing ban vessel monitoring system based on visible light and infrared fusion provided in an embodiment of the present invention. Detailed Implementation

[0015] The present invention will now be described in detail with reference to the accompanying drawings. Figure 1This is a flowchart illustrating a method for monitoring fishing vessels in river channels based on visible light and infrared fusion, according to an embodiment of the present invention. The following is a detailed description of this method.

[0016] Step S110: Obtain visible light image sequences and infrared image sequences of the river monitoring area. The visible light image sequences contain texture details of the river surface and surrounding environment, and the infrared image sequences contain thermal radiation distribution features within the river area.

[0017] In this embodiment, a specific natural river is selected as the monitoring target. This river is subject to a fishing ban during its closed season, necessitating monitoring of vessels within the area to prevent illegal fishing. To acquire the required visible light and infrared image sequences, multiple sets of image acquisition devices can be installed at suitable locations along the riverbank. Each set includes one visible light camera and one infrared thermal imager. These devices are fixed with brackets, and their lenses are pointed towards the river surface and surrounding area to ensure coverage of the preset monitoring range.

[0018] The visible light camera uses an industrial-grade lens with parameters set to suit long-distance water surface photography, capable of capturing details such as water ripples, color variations of boats, and patterns on the hull. The infrared thermal imager is a model with high thermal sensitivity, capable of detecting minute temperature changes on object surfaces, thus reflecting the thermal radiation of different objects, such as areas of high thermal radiation generated by a ship's engine or the thermal radiation trails created by people moving on the deck.

[0019] Furthermore, all image acquisition devices are controlled via a unified time synchronization mechanism to ensure that visible light cameras and infrared thermal imagers acquire images simultaneously. The acquisition frequency is set to a fixed value, ensuring a one-to-one temporal correspondence between the generated visible light image sequences and infrared image sequences. During acquisition, to protect potential personal privacy in the images, real-time preprocessing is performed. A deep learning-based image segmentation algorithm automatically identifies regions containing potentially sensitive information such as faces and vehicle license plates, and performs pixel-level blurring on these regions to ensure privacy is not compromised. The processed visible light and infrared image sequences are stored in a designated directory on a local server, awaiting further processing.

[0020] Step S120: Perform ship visual feature extraction processing on the visible light image sequence to obtain a visible light ship feature set; perform thermal feature extraction processing on the infrared image sequence to obtain an infrared ship feature set.

[0021] In this embodiment, after acquiring the visible light image sequence and the infrared image sequence, feature extraction is performed on these two different types of image sequences respectively. For the visible light image sequence, the focus is on extracting visual features that can reflect the appearance and motion of the ship; for the infrared image sequence, the focus is on extracting features related to the ship's thermal radiation for subsequent fusion processing.

[0022] Step S121: Perform ship region localization processing on consecutive frames in the visible light image sequence. By traversing the pixel region of each frame image through a sliding window, and combining prior knowledge of ship contours to filter out candidate regions that may contain ships, the boundary information of the candidate regions is enhanced by an edge detection algorithm to determine the range of the pixel region where the ship is located in each frame image. The range of the pixel region is represented by a set of bounding box coordinates.

[0023] In this embodiment, a single frame from a visible light image sequence is used as an example for ship region localization. First, the size of the sliding window is determined, set based on the approximate proportions of common ships in the image to ensure coverage of the pixel range that ships of different sizes might occupy. The sliding window starts from the top left corner of the image and moves sequentially in both horizontal and vertical directions. The step size for each movement is set according to the image resolution and processing efficiency, allowing the window to traverse every possible region in the image.

[0024] During the traversal, at each position, the image features within that window are extracted and compared with pre-defined prior knowledge of ship contours. This prior knowledge of ship contours is derived from analyzing a large number of ship sample images and includes common ship shape features such as approximate length-to-width ratios and overall streamlined or rectangular outlines. When the matching degree between the image features within a window and this prior knowledge reaches a set threshold, that window is marked as a candidate region.

[0025] After screening the candidate regions, an edge detection algorithm is applied to each candidate region. The Canny edge detection algorithm is selected. First, the image of the candidate region is converted to grayscale. Then, the gradient magnitude and direction of the image are calculated. Non-maximum suppression is used to eliminate stray responses in the edge detection results. Then, a double thresholding algorithm is used to determine the real and potential edges. Finally, an edge connection algorithm is used to connect these edges into a complete contour, thereby strengthening the boundary information of the candidate region.

[0026] Based on the enhanced boundary information, the pixel region where the ship is located is determined. For each determined ship region, a bounding box is used to mark it. The bounding box coordinate set consists of the pixel coordinates of the upper left and lower right corners of the bounding box, for example (a1, b1, a2, b2), where (a1, b1) is the upper left corner coordinate and (a2, b2) is the lower right corner coordinate. The above coordinate set can accurately define the position of the ship in the image.

[0027] Step S122: Extract visible light texture features within the pixel area, calculate pixel gray-level correlation features under different directions and distances using the gray-level co-occurrence matrix algorithm, combine image details after Gaussian filtering to generate pattern distribution features on the ship surface, extract region edge points using the Canny operator and fit contour curves to obtain edge contour features, and calculate color gradient features based on the color difference between adjacent pixels using the HSV color space. The visible light texture features include pattern distribution features, edge contour features, and color gradient features.

[0028] In this embodiment, after determining the pixel region of the ship, visible light texture features are extracted from the image within that region. First, a gray-level co-occurrence matrix (GLCM) algorithm is used to calculate pixel gray-level correlation features. The image of the ship region is converted into a grayscale image. Then, multiple different directions, such as 0 degrees, 45 degrees, 90 degrees, and 135 degrees, and multiple different pixel spacings are set in the grayscale image. For each direction and spacing, the probability of two pixels in the grayscale image having a specific combination of gray values ​​is statistically analyzed, thereby generating a gray-level co-occurrence matrix. Feature parameters such as energy, entropy, contrast, and correlation are extracted from the gray-level co-occurrence matrix. These parameters together constitute the pixel gray-level correlation features under different directions and distances.

[0029] Next, Gaussian filtering is applied to the image of the ship area. Gaussian filtering uses a preset Gaussian kernel, the size of which is selected based on the richness of detail in the image. Convolution operations are used to smooth the image, removing noise and preserving important details. The Gaussian-filtered image is then combined with the previously obtained pixel grayscale correlation features to analyze the pattern distribution on the ship's surface, such as the position, shape, and density of patterns like stripes, spots, and markings, generating pattern distribution features for the ship's surface.

[0030] Then, the edge points of the region are extracted using the Canny operator. The Canny edge detection algorithm is applied again to the grayscale image of the ship region to obtain a set of edge points. A curve fitting algorithm is used to process these edge points, and based on the coordinate distribution of the edge points, curves that can reflect the shape of the ship's outline are fitted, such as the side outline curve of the hull, the curves of the bow and stern, etc. These curves together constitute the edge outline features.

[0031] Finally, color gradient features are calculated based on the HSV color space. The image of the ship area is converted from the RGB color space to the HSV color space to obtain the hue, saturation, and lightness values ​​of each pixel. The differences between adjacent pixels in the three channels of hue, saturation, and lightness are calculated; these differences reflect the color variations. These differences are arranged according to pixel position to form a color gradient feature, which can reflect the color transitions in different areas of the ship's surface.

[0032] Step S123: Analyze the changing trend of the ship pixel region range in the continuous frames, calculate the center coordinate offset and bounding box size change rate of the ship in adjacent frames by inter-frame difference method and integrate them into displacement parameters. At the same time, combine optical flow method to track the motion trajectory of key ship pixels, supplement the subtle motion information not covered in the displacement parameters, and generate ship motion dynamic features.

[0033] In this embodiment, the changing trend of the pixel region range of the ship is analyzed for consecutive frames of a visible light image sequence to obtain motion dynamic features. First, the inter-frame difference method is used to select two consecutive image frames. Assuming the previous frame is image F(t) and the current frame is image F(t+1), the bounding box coordinates of the ship in these two images are obtained respectively.

[0034] Calculate the center coordinates of the bounding box. For the bounding box (a1, b1, a2, b2) in image F(t), its center coordinates are ((a1+a2) / 2, (b1+b2) / 2), denoted as (c1, d1); the center coordinates of the bounding box in image F(t+1) are (c2, d2). The center coordinate offset is (c2-c1, d2-d1), which reflects the overall distance the ship has moved in the horizontal and vertical directions.

[0035] Simultaneously, the rate of change of the bounding box size is calculated. In image F(t), the width of the bounding box is (a2-a1), and the height is (b2-b1); in image F(t+1), the width is (a2'-a1'), and the height is (b2'-b1'). The rate of change of width is ((a2'-a1')-(a2-a1)) / (a2-a1), and the rate of change of height is ((b2'-b1')-(b2-b1)) / (b2-b1). The center coordinate offset and the rate of change of the bounding box size are integrated to form the displacement parameter.

[0036] Next, optical flow is used to track the motion trajectories of key pixels on the ship. Multiple key pixels are selected within the ship's region; these are typically representative pixels such as turning points and corners of the ship's edges. Using the Lucas-Kanade optical flow algorithm, assuming that the motion of these key pixels between adjacent frames satisfies the assumptions of constant brightness and smooth motion, the motion vector of each key pixel is obtained by calculating the positional changes of these points in images F(t) and F(t+1).

[0037] These motion vectors are combined with the previously obtained displacement parameters, which reflect the overall motion of the ship. The motion vectors of key pixels supplement the subtle motion information not covered by the displacement parameters, such as slight rotation and local swaying of the ship. Integrating this information generates dynamic features of the ship's motion, which comprehensively reflects the ship's motion state in consecutive frames.

[0038] Step S124: Perform feature dimension mapping processing on the visible light texture features and the ship motion dynamic features. After performing dimensionality reduction or dimensionality increase operations through principal component analysis, the feature values ​​are mapped to the same numerical range through feature scaling, and the visible light texture features and the ship motion dynamic features are converted into feature vectors of the same dimension.

[0039] In this embodiment, since visible light texture features and ship motion dynamic features may have different dimensions, feature dimension mapping processing is required to enable them to be fused in subsequent operations. First, principal component analysis algorithm is applied to the visible light texture features and ship motion dynamic features respectively.

[0040] For visible light texture features, they are represented as a feature matrix, where each row represents a sample and each column represents a feature dimension. By calculating the covariance matrix of this matrix, and then solving for the eigenvalues ​​and eigenvectors, the top k eigenvectors that can explain most of the data variance are selected. The visible light texture features are then projected into a new space formed by these eigenvectors, achieving dimensionality reduction. If the dimensionality of the visible light texture features is low, an extension method from principal component analysis is used to increase the dimensionality, generating a higher-dimensional feature representation.

[0041] The same processing is applied to the ship motion dynamics features. Based on the relationship between the original dimension and the target dimension, corresponding dimensionality reduction or dimensionality increase operations are performed so that the processed visible light texture features and the ship motion dynamics features have the same dimension.

[0042] After dimensionality adjustment, feature scaling is performed. A min-max scaling method is used, where each feature value in the processed visible light texture features is calculated according to the formula (x-min_x) / (max_x-min_x), where x is the feature value, min_x is the minimum value in that feature dimension, and max_x is the maximum value in that feature dimension, mapping the feature value to the interval [0, 1]. The same scaling process is applied to the ship motion dynamics features, ensuring that the feature values ​​of both are within the same numerical range. After these processes, the visible light texture features and ship motion dynamics features are converted into feature vectors of the same dimension.

[0043] Step S125: The feature vectors of the same dimension are concatenated according to the time sequence, the feature vectors corresponding to each frame are arranged in the order of timestamps, the cosine similarity of adjacent timestamp feature vectors is calculated to verify the temporal continuity, and the feature vectors that are temporally discontinuous in the temporal continuity state are subjected to smooth transition processing to generate a visible light ship feature set containing temporal correlation.

[0044] In this embodiment, feature vectors of the same dimension obtained after dimension mapping are concatenated according to a time sequence. Each feature vector corresponds to the processing result of one frame of image. These feature vectors are arranged sequentially according to the timestamp order of image acquisition to form a sequence.

[0045] To verify temporal continuity, the cosine similarity of adjacent timestamp feature vectors is calculated. For feature vector V(t) corresponding to timestamp t and feature vector V(t+1) corresponding to timestamp t+1, the cosine similarity is calculated by dividing the dot product of the two vectors by the product of their magnitudes. The closer the result is to 1, the higher the similarity between the two feature vectors and the better the temporal continuity; conversely, the closer the result is to 1, the worse the temporal continuity.

[0046] When the calculated cosine similarity is lower than a preset threshold, it is determined to be temporally discontinuous. In this case, a smooth transition process is applied to these discontinuous feature vectors. A linear interpolation method is used to generate several transition feature vectors between V(t) and V(t+1), making the change in feature vectors more stable. For example, if there is temporal discontinuity between V(t) and V(t+1), transition vectors such as V(t,1) and V(t,2) are generated, where V(t,1) = V(t) + 1 / n * (V(t+1) - V(t)), V(t,2) = V(t) + 2 / n * (V(t+1) - V(t)), and n is the number of transition vectors.

[0047] After the above processing, a visible light vessel feature set containing temporal correlation is formed. The feature vectors in this visible light vessel feature set are arranged in chronological order, and there is good coherence between adjacent feature vectors, which can reflect the feature changes of the vessel at different points in time.

[0048] Step S130: Perform thermal feature extraction processing on the infrared image sequence to obtain an infrared ship feature set.

[0049] Step S131: Perform thermal radiation region segmentation processing on each frame of the infrared image sequence. Use an adaptive threshold segmentation algorithm to determine the segmentation threshold based on the thermal radiation intensity distribution, divide the image into multiple connected regions, and filter out region units with continuous thermal radiation characteristics and conforming to the size characteristics of ships by calculating the morphological parameters of each connected region. The region units correspond to the regions where ships may exist.

[0050] In this embodiment, thermal radiation region segmentation is performed on each frame of the infrared image sequence. First, the grayscale value of each pixel in the infrared image is converted into a corresponding thermal radiation intensity value; a higher grayscale value indicates a greater thermal radiation intensity. An adaptive threshold segmentation algorithm is used, which automatically determines the segmentation threshold based on the distribution of thermal radiation intensity in the image. Specifically, the image is divided into multiple sub-regions, a local threshold is calculated for each sub-region, and then the entire image is segmented based on these local thresholds. Regions with thermal radiation intensity higher than the threshold are marked as foreground, and regions with intensity lower than the threshold are marked as background.

[0051] After segmentation, multiple connected regions are obtained, each composed of interconnected foreground pixels. Morphological parameters of each connected region are calculated, including its area, perimeter, and the aspect ratio of its bounding rectangle. Based on the general size characteristics of a ship, filtering conditions such as area and aspect ratio ranges are set to filter the connected regions. For example, regions with excessively small areas may be noise or small floating objects and are excluded; regions with aspect ratios that do not conform to the characteristics of a ship are also excluded.

[0052] After screening, the selected regional units exhibit continuous thermal radiation characteristics and their morphological parameters conform to the size characteristics of ships. These regional units are considered to be areas where ships may exist.

[0053] Step S132: Extract the thermal radiation intensity distribution features of the region unit. By scanning each pixel in the region unit, record the location of the maximum thermal radiation intensity as the thermal radiation peak position. Calculate the thermal radiation intensity attenuation rate from the thermal radiation peak position to the edge of the region unit to obtain the thermal radiation gradient change features. Also, compare the area expansion of the region unit in consecutive frames and combine the thermal radiation intensity changes of the edge pixels to generate thermal radiation range expansion features. The thermal radiation intensity distribution features include the thermal radiation peak position, thermal radiation gradient change features, and thermal radiation range expansion features.

[0054] In this embodiment, after determining the region unit, its thermal radiation intensity distribution characteristics are extracted. First, each pixel within the region unit is scanned, and the thermal radiation intensity value of each pixel is recorded. The maximum value is then identified, and the pixel coordinates of the maximum value are the thermal radiation peak position. This thermal radiation peak position typically corresponds to parts that generate a lot of heat, such as ship engines.

[0055] Next, the decay rate of thermal radiation intensity from the peak position to the edge of the region cell is calculated. Multiple rays are drawn from the peak position to the edge of the region cell. Along each ray, the ratio of the change in thermal radiation intensity from the peak position to the distance is calculated; this ratio is the decay rate. The decay rates of multiple rays are combined to form a thermal radiation gradient characteristic, which reflects the change in thermal radiation intensity from the peak to the edge.

[0056] Then, a thermal radiation range expansion feature is generated. By comparing the area changes of the same region unit in consecutive frames, the expansion or contraction rate of the area is calculated. Simultaneously, the thermal radiation intensity changes of the edge pixels of the region unit are monitored. If the thermal radiation intensity of the edge pixels increases, it may indicate that the ship is expanding in that direction or that the thermal radiation range is increasing. Combining the area expansion and the change in edge pixel thermal radiation intensity generates a thermal radiation range expansion feature that reflects the dynamic changes in the ship's thermal radiation area.

[0057] Step S133: Track the position change of the region unit in continuous infrared image frames, match the region unit of the current frame with the previous frame using template matching method, determine the movement path of the center coordinate, calculate the coordinate change per unit time to obtain the movement rate feature, analyze the fluctuation of the movement rate, and extract the frequency and amplitude features of the movement rate change to supplement the movement rate feature.

[0058] In this embodiment, a template matching method is used to track the positional changes of region units in consecutive infrared image frames. The region units in the previous frame are used as templates, and the template is slid across a preset search range in the current frame. The similarity between the template and each region within the search range is calculated, and the region with the highest similarity is the region in the current frame that matches the region unit from the previous frame.

[0059] Based on the matching results, the center coordinates of the region units in each frame of the image are determined. These center coordinates are then connected in chronological order to form the movement path of the center coordinates. According to the movement path, the change in the center coordinates per unit time is calculated, i.e., the movement rate. The magnitude and direction of the movement rate together constitute the basic part of the movement rate feature.

[0060] Further analysis of the movement rate fluctuations is conducted, calculating the frequency and amplitude of changes in the movement rate over different time periods. Frequency refers to the number of rate changes per unit time, and amplitude refers to the difference between the maximum and minimum values ​​of the rate change. Incorporating frequency and amplitude characteristics into the movement rate features allows the movement rate features to more comprehensively reflect the motion state of the regional unit.

[0061] Step S134: Perform correlation mapping processing on the thermal radiation intensity distribution feature and the movement rate feature, calculate the correlation coefficient, construct a feature mapping matrix based on the correlation coefficient, and project the thermal radiation intensity distribution feature and the movement rate feature into the correlation space of the feature mapping matrix respectively to generate a thermal radiation motion correlation feature vector.

[0062] In this embodiment, after obtaining the thermal radiation intensity distribution characteristics and movement speed characteristics, it is necessary to perform correlation mapping between the two to establish a connection between thermal radiation characteristics and motion state. First, the correlation coefficient between the thermal radiation intensity distribution characteristics and the movement speed characteristics is calculated. The thermal radiation intensity distribution characteristics include parameters in multiple dimensions, such as the coordinates of the thermal radiation peak position, the change value of thermal radiation gradient in different directions, and the thermal radiation range expansion rate; the movement speed characteristics also include parameters in multiple dimensions, such as the magnitude, direction, frequency of change, and amplitude of change of the movement speed.

[0063] For each dimension parameter in the thermal radiation intensity distribution feature and each dimension parameter in the movement rate feature, the Pearson correlation coefficient method is used to calculate the degree of linear correlation between them, resulting in multiple correlation coefficients. These correlation coefficients form a two-dimensional matrix, where the rows of the matrix correspond to the dimensions of the thermal radiation intensity distribution feature, the columns correspond to the dimensions of the movement rate feature, and each element represents the degree of correlation between the corresponding two dimension parameters.

[0064] A feature mapping matrix is ​​constructed based on the obtained correlation coefficients. The element values ​​of this feature mapping matrix are determined by the correlation coefficients, reflecting the correlation weights between the thermal radiation intensity distribution features and the movement rate features in each dimension. Then, the thermal radiation intensity distribution feature vector and the movement rate feature vector are multiplied by the feature mapping matrix respectively, and projected onto the correlation space defined by the feature mapping matrix.

[0065] The projected thermal radiation intensity distribution and movement rate features exhibit stronger correlation in the correlation space. Concatenating these two projected feature vectors by channel generates a thermal radiation motion correlation feature vector. This feature vector simultaneously contains information on both thermal radiation intensity distribution and movement rate, demonstrating the inherent correlation between the two.

[0066] Step S135: Arrange the thermal radiation motion-related feature vectors according to the time sequence, organize the feature vectors according to the time stamp order, calculate the feature distance between adjacent feature vectors to evaluate the temporal consistency, and perform interpolation adjustment on feature vectors whose feature distance exceeds the preset distance range to generate an infrared ship feature set containing the temporal change pattern. Each feature vector in the infrared ship feature set contains thermal radiation and motion-related information of the corresponding time stamp.

[0067] In this embodiment, the thermal radiation motion-related feature vectors are arranged in chronological order according to the timestamps of image acquisition, forming an ordered sequence of feature vectors. Each feature vector corresponds to a timestamp and contains the thermal radiation and motion-related information of the region unit at that time point.

[0068] To assess temporal consistency, the characteristic distance between the thermal radiation motion associated feature vectors corresponding to adjacent timestamps is calculated. The characteristic distance is calculated using the Euclidean distance method: for feature vector U(t) at timet t and feature vector U(t+1) at timet t+1, the square root of the sum of the squares of the differences in the corresponding dimensions of the two vectors is calculated; the result is the Euclidean distance between them. A smaller distance value indicates more stable changes in adjacent feature vectors and better temporal consistency; conversely, a larger distance value indicates poorer temporal consistency.

[0069] A preset feature distance range is defined. When the calculated Euclidean distance between adjacent feature vectors exceeds this range, the feature vectors are interpolated and adjusted. A cubic spline interpolation method is used to generate several transition feature vectors between U(t) and U(t+1) based on their values, making the change trend of the feature vectors smoother. For example, if the distance between U(t) and U(t+1) is too large, transition vectors such as U(t, 1) and U(t, 2) are generated. These transition vectors are calculated using a cubic spline function and can better fit the change process from U(t) to U(t+1).

[0070] After the above processing, an infrared vessel feature set containing temporal variation patterns is formed. The feature vectors in this infrared vessel feature set are arranged in chronological order, and adjacent feature vectors have good temporal consistency, which can reflect the dynamic changes of thermal radiation and motion correlation information of regional units at different time points.

[0071] Step S140: Perform cross-source feature fusion processing on the visible light ship feature set and the infrared ship feature set to generate a cross-source fused feature set.

[0072] In this embodiment, the visible light ship feature set and the infrared ship feature set describe the characteristics of the ship from different perspectives. In order to make comprehensive use of this information, cross-source feature fusion processing is required. Through fusion processing, features from two different sources are organically combined to generate a more comprehensive and discriminative cross-source fused feature set.

[0073] Step S141: Traverse the timestamp information in the visible light ship feature set and the infrared ship feature set, establish a timestamp correspondence table, and extract the visible light feature vector and infrared feature vector under the same timestamp according to the timestamp correspondence table.

[0074] In this embodiment, the timestamp information corresponding to all feature vectors in the visible light ship feature set and the infrared ship feature set is first traversed to collect the timestamp of each feature vector. Since the visible light image sequence and the infrared image sequence are acquired synchronously, theoretically their timestamps should correspond one-to-one. However, there may be cases where some frames are lost or the timestamp recording is inaccurate. Therefore, it is necessary to establish a timestamp correspondence table.

[0075] For each timestamp in the visible light vessel feature set, the infrared vessel feature set is searched for timestamps that are the same or closest, and the two are matched and recorded in a timestamp correspondence table. For feature vectors that cannot be matched with a corresponding timestamp, interpolation is performed based on the matching situation before and after the timestamp to generate a corresponding virtual feature vector to ensure the integrity of the time series.

[0076] Based on the established timestamp correspondence table, the visible light feature vector and infrared feature vector corresponding to each timestamp are extracted to ensure that the two feature vectors at the same time point are used in subsequent processing.

[0077] Step S142: Call the cross-source attention mechanism module to calculate the association weights of the visible light feature vector and infrared feature vector under the same timestamp, and generate a feature association weight matrix.

[0078] In this embodiment, the cross-source attention mechanism module is invoked to calculate the correlation weight between the visible light feature vector and the infrared feature vector at the same timestamp. The cross-source attention mechanism module adopts a multilayer perceptron structure, and first concatenates the visible light feature vector and the infrared feature vector to form a concatenated feature vector.

[0079] The concatenated feature vector is input into the first fully connected layer of the multilayer perceptron, and after processing by the activation function, an intermediate feature vector is obtained. The intermediate feature vector is then input into the second fully connected layer, whose output dimension is the product of the visible light feature vector dimension and the infrared feature vector dimension. The output result is used to reshape the feature association weight matrix.

[0080] Each element in the feature association weight matrix represents the correlation strength between the corresponding dimension in the visible light feature vector and the corresponding dimension in the infrared feature vector. The larger the weight value, the stronger the correlation between the two dimensions, and the more attention should be paid to them during fusion.

[0081] Step S143: Perform weighted fusion processing on the visible light feature vector and the infrared feature vector according to the feature association weight matrix to generate a preliminary fused feature vector.

[0082] In this embodiment, the visible light feature vector and the infrared feature vector are weighted and fused according to the feature association weight matrix. First, the feature association weight matrix is ​​multiplied with the visible light feature vector to obtain the weighted visible light feature vector. Each dimension value in the visible light feature vector is the result of weighting the corresponding dimension of the original visible light feature vector with the weight value of the corresponding row in the association weight matrix.

[0083] Similarly, the transpose of the feature association weight matrix is ​​multiplied with the infrared feature vector to obtain a weighted infrared feature vector. Then, the weighted visible light feature vector and the weighted infrared feature vector are concatenated by channel to generate a preliminary fused feature vector.

[0084] The initial fusion of feature vectors combines important information from visible light and infrared features, and highlights the strong correlation between the two through the correlation weight matrix, while weakening the weak correlation, so that the fused features can better reflect the essential characteristics of the ship.

[0085] Step S144: Perform multi-scale feature mapping processing on the preliminary fused feature vector to obtain local detail features at different scales, and at the same time extract global statistical information of the features as global distribution features through global average pooling operation.

[0086] In this embodiment, multi-scale feature mapping is performed on the preliminary fused feature vector. Multiple convolutional kernels of different sizes are used to convolve the preliminary fused feature vector, with each kernel corresponding to a scale. For example, small-sized convolutional kernels are used to extract fine-grained local detail features, such as the fine texture of a ship's surface or local abrupt changes in thermal radiation; large-sized convolutional kernels are used to extract coarse-grained local detail features, such as the overall outline of a ship or the wide-area distribution of thermal radiation.

[0087] Each convolution operation is followed by an activation function to obtain local detail features at different scales. These local detail features describe the local information in the preliminary fused feature vector from different spatial scales.

[0088] Simultaneously, a global average pooling operation is performed on the preliminary fused feature vectors to calculate the average value of each feature channel. These average values ​​are then combined to form a global distribution feature. The global distribution feature reflects the statistical distribution of the preliminary fused feature vectors throughout the feature space, supplementing the global information missing from local detailed features.

[0089] Step S145: Perform channel concatenation processing on the local detail features and the global distribution features, concatenate the feature channels of the local detail features and the global distribution features in sequence, adjust the numerical distribution between channels through batch normalization operation, and generate a cross-source fusion feature set containing multi-scale information. Each feature vector in the cross-source fusion feature set contains multi-scale fusion feature information corresponding to the timestamp.

[0090] In this embodiment, local detail features and global distribution features at different scales are concatenated using channels. First, all local detail features are concatenated using channels in ascending order of scale to form a local feature set. Then, the global distribution features are concatenated with the local feature set using channels, so that the concatenated feature vector contains both local detail information at multiple scales and global statistical information.

[0091] After concatenation, batch normalization is performed on the resulting feature vectors. The mean and variance of each feature channel are calculated, and the feature value of each channel is subtracted from the mean of that channel and then divided by the standard deviation of that channel, so that the feature values ​​of each channel have a distribution characteristic of zero mean and unit variance.

[0092] Batch normalization adjusts the numerical distribution differences between different channels, preventing excessively large or small feature values ​​in some channels from affecting subsequent processing. After the above processing, a cross-source fusion feature set containing multi-scale information is generated. Each feature vector in this cross-source fusion feature set corresponds to a timestamp, containing the multi-scale fusion feature information of the ship at that time point.

[0093] Step S150: Call the pre-trained fishing ban vessel identification model to perform vessel attribute and behavior analysis on the cross-source fusion feature set, and generate vessel monitoring results including vessel type, position change trajectory and suspected fishing ban behavior identifiers.

[0094] In this embodiment, the cross-source fusion feature set contains rich vessel feature information. A pre-trained fishing ban vessel identification model is invoked to process this information, enabling analysis of vessel attributes and behavior. The fishing ban vessel identification model, trained on a large amount of labeled data, can extract key information from the cross-source fusion features, identify vessel types, track position change trajectories, and determine whether there are suspected fishing ban behaviors, ultimately generating comprehensive vessel monitoring results.

[0095] Step S151: Input the cross-source fusion feature set into the feature encoding layer of the fishing ban vessel identification model, and generate an encoded feature vector by performing layer-by-layer abstraction through a multi-layer convolutional neural network.

[0096] In this embodiment, each feature vector in the cross-source fusion feature set is sequentially input into the feature encoding layer of the fishing ban vessel identification model. The feature encoding layer consists of multiple convolutional neural networks. The first convolutional layer uses multiple convolutional kernels to perform convolution operations on the input cross-source fusion feature vectors to extract low-level features, such as edges and local textures.

[0097] The result of the convolution operation, after being processed by the activation function, is input into the pooling layer for downsampling, reducing the feature dimensionality while retaining important features. The downsampled features are then input into a second convolutional layer, which uses more convolutional kernels to extract higher-level features, such as the combination of ship components and the correlation pattern between thermal radiation and morphology.

[0098] Subsequent convolutional and pooling layers repeat the above process, with each layer further abstracting and extracting features based on the previous layer. After processing by multiple layers of convolutional neural networks, a high-dimensional feature map is obtained. This feature map is flattened into a one-dimensional vector, which is the encoded feature vector. The encoded feature vector is a highly abstract representation of the cross-source fusion features and contains the key feature information of the ship.

[0099] Step S152: Input the encoded feature vector into the vessel type identification branch, output the probability distribution of various types of vessels, and determine the vessel type information based on the probability distribution. The vessel type information includes category identifiers for fishing boats, transport boats, and sightseeing boats.

[0100] In this embodiment, the encoded feature vector is input into the vessel type recognition branch of the fishing ban vessel identification model. This vessel type recognition branch consists of multiple fully connected layers. The first fully connected layer receives the encoded feature vector and processes it through matrix operations and activation functions, mapping the features to an intermediate feature space.

[0101] The feature vectors from the intermediate feature space are input to the second fully connected layer. The output dimension of this second fully connected layer is consistent with the preset number of ship types, and each output node corresponds to a score for one ship type. The softmax function is applied to these scores to convert them into a probability distribution for each type of ship. The sum of the probability values ​​is 1, and each probability value represents the likelihood that the input feature vector belongs to the corresponding ship type.

[0102] Based on the probability distribution, the vessel type with the highest probability value is selected as the identification result, the vessel type information is determined, and corresponding category labels are added to it, such as "Y" for fishing boats, "T" for transport boats, and "G" for sightseeing boats.

[0103] Step S153: Input the encoded feature vector into the trajectory prediction branch, construct a temporal convolutional network by combining it with time series information, capture the dependencies of different time scales through multiple dilated convolutional layers, output the predicted position coordinates for multiple future time steps, and combine the predicted position coordinates with historical coordinates to generate a position change trajectory.

[0104] In this embodiment, the encoded feature vector is input into the trajectory prediction branch of the fishing ban vessel identification model. This trajectory prediction branch first combines the encoded feature vector with the corresponding timestamp information to construct a feature sequence containing time-series information.

[0105] The feature sequence is input into a temporal convolutional network, which contains multiple dilated convolutional layers, each with a different dilation rate. The dilation rate determines the receptive field size of the convolutional kernel on the input feature sequence; a smaller dilation rate is used to capture feature dependencies over short time scales, while a larger dilation rate is used to capture feature dependencies over longer time scales.

[0106] Through layer-by-layer processing using multiple dilated convolutional layers, temporal convolutional networks can effectively capture the dependencies of feature sequences at different time scales and extract the temporal patterns of ship motion. The processed features are input into a fully connected layer, which outputs predicted position coordinates for multiple future time steps. The position coordinates at each time step include predicted values ​​for both horizontal and vertical dimensions.

[0107] By connecting the predicted position coordinates of multiple future time steps with the position coordinates corresponding to historical timestamps (extracted from cross-source fusion features or parsed from the original image) in chronological order, a complete position change trajectory is formed. The position change trajectory can intuitively reflect the movement path and trend of the ship.

[0108] Step S154: Input the encoded feature vector into the behavior analysis branch and extract the pattern features related to the fishing ban behavior. The pattern features include features of vessel dwell time, abnormal trajectory turning, and wandering in specific areas.

[0109] Step S1541: Perform time-division processing on the encoded feature vector, dividing the continuous feature vector into multiple sub-sequences according to a fixed time interval, with each sub-sequence corresponding to a time period, to obtain sub-feature vector sequences corresponding to different time periods.

[0110] In this embodiment, the encoded feature vector is first segmented along the time dimension before extracting pattern features. Based on the time granularity requirements of ship behavior analysis, a fixed time interval is set; for example, continuous encoded feature vectors are divided into time periods of N timestamps.

[0111] For each time period, the corresponding encoded feature vectors are combined to form a sub-feature vector sequence. For example, the encoded feature vectors from timestamps t1 to tN form the first sub-sequence, the encoded feature vectors from timestamps tN+1 to t2N form the second sub-sequence, and so on. For the last portion with fewer than N timestamps, if its duration exceeds half of the fixed time interval, it is treated as a separate sub-sequence; otherwise, it is merged into the previous sub-sequence.

[0112] By segmenting the data along the time dimension, we obtain sub-feature vector sequences corresponding to different time periods. Each sub-feature vector sequence can reflect the characteristic changes of the ship within the corresponding time period.

[0113] Step S1542: Calculate the ship position distribution density in each sub-feature vector sequence, calculate the density of ship position coordinates using the kernel density estimation method, and select dense region features based on the density calculation results.

[0114] In this embodiment, for each sub-feature vector sequence, the ship's position coordinate information is extracted from the sub-feature vectors, and these position coordinates constitute a coordinate set. The position distribution density of this coordinate set is calculated using a kernel density estimation method. A Gaussian kernel function is selected as the kernel function, and the bandwidth is adaptively determined according to the distribution of the coordinate set.

[0115] The kernel density estimation process involves, for each location coordinate point, calculating its density contribution to the surrounding area using a Gaussian kernel function, centered on that point. The density contributions of all points are then superimposed to obtain a location density map of the entire region. Based on this density map, a density threshold is set, and areas with density values ​​exceeding the threshold are marked as dense regions.

[0116] Feature parameters of densely populated areas are extracted, such as the center coordinates, area, outline shape, and number of location coordinates within the area. These parameters are then combined to form densely populated area features. These features reflect the aggregation of vessels within a given time period and are of great significance in determining whether there is coordinated multi-vehicle fishing ban activities.

[0117] Step S1543: Analyze the trajectory direction change in the sub-feature vector sequence, calculate the trajectory direction angle at adjacent time points, determine whether the direction has changed significantly by the angle difference between the trajectory direction angles at adjacent time points, mark the angle difference as a mutation point when the angle difference exceeds a preset angle threshold, and record the angle change value of the mutation point to generate trajectory abnormal turning features.

[0118] In this embodiment, for each sub-feature vector sequence, the position coordinates of the ship at different time points are extracted from the sub-feature vectors. These position coordinates are arranged in chronological order to form the ship's trajectory. To analyze the changes in trajectory direction, the trajectory direction angle between adjacent time points is first calculated. For time points t1 and t2 (t2 is later than t1), the corresponding position coordinates are (x1, y1) and (x2, y2), respectively. The vector (x2-x1, y2-y1) is calculated using these two coordinates. The angle between this calculated vector and the positive horizontal direction is the trajectory direction angle from time point t1 to t2, denoted as θ1. Similarly, the trajectory direction angle θ2 from time point t2 to t3 is calculated, and so on, to obtain the sequence of trajectory direction angles between adjacent time points.

[0119] Next, the angular difference between adjacent trajectory directional angles is calculated, i.e., Δθ = θ_after - θ_before (where θ_after is the directional angle corresponding to the later time point, and θ_before is the directional angle corresponding to the earlier time point). This angular difference is compared with a preset angular threshold, which is set based on the range of directional changes during normal navigation of vessels in the river, for example, determined based on the range of normal turning angles of legal vessels in historical monitoring data. When the absolute value of the angular difference Δθ exceeds the preset angular threshold, this location is marked as a point of abrupt change, indicating that the vessel has undergone a significant directional change at that time point.

[0120] For each abrupt change point, the corresponding angle change value Δθ is recorded, along with the time point of the abrupt change and the vessel's position coordinates at that time. This information is then integrated to generate an abnormal trajectory turning feature. This feature reflects whether the vessel exhibits any unusual or sudden turns during its journey, which may be related to fishing ban activities, such as rapidly approaching areas where fish congregate or evading law enforcement vessels.

[0121] Step S1544: Statistically count the frequency and duration of the appearance of vessels in the prohibited fishing area, and calculate the wandering coefficient in the area in combination with the position change trajectory. The wandering coefficient is positively correlated with the frequency and duration. Specifically, by comparing the position of the vessel with the boundary of the prohibited fishing area, the time points when the vessel enters and leaves the prohibited fishing area are determined, the number of times it appears is calculated as the frequency, and the time point difference is calculated as the duration.

[0122] In this embodiment, the boundary of the fishing ban area is first defined. This boundary is pre-stored in the form of geographic coordinates, for example, using a set of vertex coordinates of a polygon to define the fishing ban area. For each time point in the sub-feature vector sequence, the vessel's position coordinates are extracted and compared with the boundary coordinates of the fishing ban area to determine whether the vessel is within the fishing ban area.

[0123] The determination method is as follows: using a ray-based point-in-polygon algorithm, the position coordinates of the vessel are determined to be inside the polygon corresponding to the fishing ban area. When the vessel enters the fishing ban area from outside, the time point at this moment is recorded as the entry time point; when the vessel leaves the fishing ban area from inside, the time point at this moment is recorded as the departure time point.

[0124] Based on the entry and exit times, the frequency and duration of vessel appearances within the prohibited fishing area are calculated. Frequency refers to the number of times a vessel enters the prohibited fishing area, i.e., the total number of entry times. Duration is the difference between each entry time and its corresponding exit time. If a vessel is still within the prohibited fishing area at the end of the sub-feature vector sequence, the exit time is used as the exit time to calculate the duration. All durations are then summed to obtain the total duration.

[0125] The wandering coefficient within the prohibited fishing area is calculated by combining the vessel's position change trajectory. The calculation method is as follows: First, multiple sampling points are selected along the position change trajectory, and the sum of the distances between these sampling points is calculated. This sum of distances reflects the length of the vessel's travel path within the prohibited fishing area. Then, the diagonal length of the smallest bounding rectangle of the vessel's activity range within the prohibited fishing area is calculated as a reference distance. Finally, the wandering coefficient is equal to (travel path length / reference distance) × (frequency of occurrence × weight 1 + duration × weight 2), where weight 1 and weight 2 are coefficients set according to actual conditions to adjust the influence of frequency of occurrence and duration on the wandering coefficient. Both weight 1 and weight 2 are positive numbers to ensure that the wandering coefficient is positively correlated with frequency of occurrence and duration. The larger the wandering coefficient, the higher the degree of wandering of the vessel within the prohibited fishing area, and the more likely it is to be engaging in prohibited fishing activities.

[0126] Step S1545: Perform feature aggregation processing on the dense area features, trajectory abnormal turning features and wandering coefficient to generate pattern features. Each dimension of the pattern features corresponds to a feature manifestation related to the fishing ban behavior.

[0127] In this embodiment, dense region features, trajectory anomaly turning features, and wandering coefficients are subjected to feature aggregation processing. First, these three features are converted into vectors with the same number of dimensions. For dense region features, the center coordinates, area, contour shape parameters, etc., can be converted into multi-dimensional values; the abrupt change angle values, time points, and position coordinates in trajectory anomaly turning features are also converted into corresponding dimensional values; the wandering coefficient, as a numerical value, can be incorporated into the vector by expanding it into multiple dimensions with the same value or by associating it with other features.

[0128] Next, the transformed vectors are standardized using the z-score standardization method, which converts the values ​​of each feature dimension into a distribution with a mean of 0 and a standard deviation of 1, thus eliminating dimensional differences between different features. The standardized vectors are then aggregated through feature concatenation, where the three vectors are joined end-to-end in sequence to form pattern features.

[0129] Each dimension of the pattern features corresponds to a specific feature among dense area features, abnormal trajectory turning features, or wandering coefficients. For example, a certain dimension corresponds to the size of the dense area, a certain dimension corresponds to the angle change value of abnormal trajectory turning, and a certain dimension corresponds to the value of the wandering coefficient, thus comprehensively integrating various feature information related to fishing ban behavior.

[0130] Step S155: Based on the matching results between the pattern features and the preset fishing ban behavior pattern library, calculate the similarity between the pattern features and each preset fishing ban behavior pattern in the preset fishing ban behavior pattern library, select the behavior tag corresponding to the preset fishing ban behavior pattern with the highest similarity as the suspected fishing ban behavior identifier, and combine the vessel type information, position change trajectory and suspected fishing ban behavior identifier into the vessel monitoring result.

[0131] In this embodiment, the preset fishing ban behavior pattern library stores a variety of preset fishing ban behavior patterns. Each pattern is represented in the form of a feature vector. These feature vectors are obtained by analyzing and extracting features from historical fishing ban cases. They cover common fishing ban behavior characteristics, such as lingering in the fishing ban area for a long time, multiple boats densely gathering in a certain area, and suddenly changing the direction of travel to approach the fish school.

[0132] The similarity between the pattern features and each preset fishing ban behavior pattern in the preset fishing ban behavior pattern library is calculated using the cosine similarity method. For a pattern feature vector V and a preset fishing ban behavior pattern feature vector U, the similarity S is calculated as the dot product of vector V and vector U divided by the product of the magnitudes of vector V and vector U, i.e., S = (V・U) / (||V||×||U||). The similarity between the pattern features and each preset pattern in the library is calculated using this method.

[0133] From all the calculated similarities, the preset fishing ban behavior pattern corresponding to the maximum value is selected. The behavior label of this preset fishing ban behavior pattern is the suspected fishing ban behavior identifier. The behavior label contains a description of the type of fishing ban behavior, such as "lingering in the fishing ban area" or "multiple boats densely gathering". The previously obtained vessel type information, position change trajectory, and the suspected fishing ban behavior identifier are integrated to form the vessel monitoring result. This vessel monitoring result comprehensively reflects the attributes, movement trajectory, and possible fishing ban behaviors of the monitored vessels.

[0134] Step S1551: Extract a set of typical fishing ban behavior patterns from a preset fishing ban behavior pattern library. This preset fishing ban behavior pattern library contains various fishing ban behavior feature templates extracted from historical fishing ban enforcement records, covering fishing ban behavior performance in different seasons and at different times. The set of typical fishing ban behavior pattern features contains behavioral feature templates of vessels in historical fishing ban cases.

[0135] In this embodiment, the construction of the pre-set fishing ban behavior pattern library is based on a large number of historical fishing ban enforcement records. When extracting the feature set of typical fishing ban behavior patterns, the historical fishing ban enforcement records are first screened to select representative cases. These cases cover fishing ban behaviors in different seasons (such as fish breeding season and non-breeding season) and different time periods (such as daytime, nighttime, early morning, etc.).

[0136] For each historical case, the behavioral characteristics of the vessels are extracted, including the frequency of their appearance in the prohibited fishing area, the duration of their presence, changes in their trajectory direction, and their aggregation with other vessels. These characteristics are then converted into feature vectors to form behavioral feature templates. All behavioral feature templates are integrated to form a typical prohibited fishing behavior pattern feature set. Each feature template in this set corresponds to a specific prohibited fishing behavior, such as "lingering in spawning areas at night in spring" or "multiple vessels gathering at fish migration channels in the early morning."

[0137] Step S1552: Calculate the feature distance between the pattern feature and each fishing ban behavior feature template in the typical fishing ban behavior pattern feature set, and arrange all fishing ban behavior feature templates in ascending order of feature distance, and select the fishing ban behavior feature template corresponding to the smallest feature distance as the matching feature template.

[0138] In this embodiment, the feature distance is calculated using the Euclidean distance method. For a pattern feature vector V and a certain fishing ban behavior feature template vector T in the typical fishing ban behavior pattern feature set, the feature distance D is calculated as the square root of the sum of the squares of the differences between the corresponding dimensions of vector V and vector T, i.e., D=[(v1-t1)²+(v2-t2)²+…+(vn-tn)²]^1 / 2, where v1 to vn are the values ​​of each dimension of vector V, and t1 to tn are the values ​​of each dimension of vector T.

[0139] Calculate the feature distance between the pattern features and each fishing ban behavior feature template in the typical fishing ban behavior pattern feature set, obtaining a set of distance values. Arrange these distance values ​​in ascending order, and the corresponding fishing ban behavior feature templates are also sorted accordingly. Select the fishing ban behavior feature template with the smallest feature distance at the top of the sorted list, and determine it as the matching feature template. This matching feature template has the smallest difference from the current pattern features and best reflects the current vessel behavior.

[0140] Step S1553: Calculate the feature overlap between the pattern feature and the matching feature template. When the feature overlap exceeds a preset overlap threshold, generate an identifier indicating the existence of suspected fishing ban behavior, including the matched fishing ban behavior type and overlap value. When the feature overlap does not exceed the preset overlap threshold, generate an identifier indicating the absence of suspected fishing ban behavior, and record the behavior type with the highest overlap as a reference. The suspected fishing ban behavior identifier includes behavior type and feature overlap information.

[0141] In this embodiment, the feature overlap degree is calculated as follows: the number of dimensions in which the numerical differences between the statistical pattern features and the matching feature templates are within a preset allowable range, and the ratio of this number of dimensions to the total number of feature dimensions is the feature overlap degree. For example, if the total feature dimensions are m, and k of the dimensions have numerical differences within the allowable range, then the feature overlap degree is k / m.

[0142] The preset overlap threshold is set based on the accuracy requirements for judging fishing ban behavior in historical data, for example, it is set to 0.6. When the calculated feature overlap exceeds 0.6, an identifier indicating suspected fishing ban behavior is generated. This identifier includes the matched fishing ban behavior type (such as "illegal stay in the fishing ban area") and the specific overlap value (such as 0.75).

[0143] When the feature overlap is less than 0.6, an identifier indicating no suspected fishing ban behavior is generated. However, the behavior type with the highest overlap with the pattern feature among all fishing ban behavior feature templates, along with its overlap value, is recorded as a reference for subsequent monitoring and analysis. Regardless of whether suspected fishing ban behavior exists, the generated suspected fishing ban behavior identifier includes behavior type and feature overlap information to comprehensively reflect the matching situation.

[0144] Step S160: Generate a fishing ban warning based on the vessel monitoring results. The fishing ban warning includes the real-time location and behavioral characteristics description of suspected fishing ban vessels.

[0145] In this embodiment, the vessel monitoring results include key information such as vessel type, location change trajectory, and signs of suspected fishing ban activities. Based on this information, a fishing ban early warning message is generated so that relevant law enforcement personnel can promptly understand the situation and take appropriate measures. The generation process requires analyzing the monitoring results, extracting key elements, and integrating the information to ultimately form a complete and clearly worded early warning message.

[0146] Step S161: Analyze the suspected fishing ban behavior identifiers in the vessel monitoring results, extract the behavior type codes and feature overlap values ​​contained therein, query the preset behavior severity level table according to the behavior type codes, and determine the type and confidence level of the suspected fishing ban behavior by combining the overlap values.

[0147] In this embodiment, the suspected fishing ban behavior identifier is stored in a structured data format, including behavior type code and feature overlap value. First, the identifier is parsed to extract the behavior type code (e.g., "001" represents "lingering in the fishing ban area", "002" represents "multiple boats gathering") and feature overlap value (e.g., 0.82).

[0148] In the pre-defined severity level table, each behavior type code corresponds to a type of prohibited fishing behavior and a severity level classification for that type of behavior. For example, "001" corresponds to "loitering in a prohibited fishing area," which is divided into three levels: mild, moderate, and severe. The corresponding behavior type is determined by querying this level table based on the extracted behavior type code.

[0149] The confidence level is determined by combining the feature overlap value. The higher the overlap value, the higher the credibility of the suspected fishing ban. The confidence level is divided into three levels: high, medium, and low. For example, an overlap value greater than 0.7 indicates high confidence, between 0.5 and 0.7 indicates medium confidence, and less than 0.5 indicates low confidence. Through the above process, the type and confidence level of the suspected fishing ban are identified.

[0150] Step S162: Extract the position change trajectory from the vessel monitoring results, obtain the vessel motion curve through a trajectory fitting algorithm, take the position coordinates at the end of the vessel motion curve as the current position coordinates of the suspected fishing ban vessel, and calculate the tangent direction at the end of the vessel motion curve as the direction of movement.

[0151] In this embodiment, the position change trajectory is extracted from the ship monitoring results. This trajectory consists of a series of position coordinates arranged in chronological order. A trajectory fitting algorithm is used to process these coordinates. A polynomial fitting method is selected, and the order of the polynomial is determined based on the curvature of the trajectory and the number of sampling points. The polynomial coefficients are solved using the least squares method to obtain a curve that can smoothly represent the ship's motion trajectory, i.e., the ship motion curve.

[0152] Take the position coordinates of the end of the motion curve, i.e., the latest position coordinates in time, as the current position coordinates of the suspected fishing ban vessel. Calculate the tangent direction at the end of the curve, and obtain the slope of the tangent by taking the derivative of the curve at the end point. Determine the direction of movement based on the slope. For example, a positive slope indicates movement in a certain direction, and a negative slope indicates movement in the opposite direction. The direction of movement is expressed in the form of azimuth (e.g., 30 degrees means 30 degrees east of north).

[0153] Step S163: Spatial comparison of the current location coordinates with the boundary coordinates of the fishing ban area, calculation of the shortest distance and azimuth from the current location coordinates to the boundary coordinates of the fishing ban area, analysis of the trend of suspected fishing ban vessels moving towards the fishing ban area, and generation of location association features.

[0154] In this embodiment, the coordinate information of the boundary of the fishing ban area is first obtained and stored in the form of polygon vertex coordinates. The current position coordinates of the suspected fishing ban vessel are then converted to the same coordinate system as the boundary coordinates of the fishing ban area for spatial comparison.

[0155] The shortest distance from the current location coordinates to the boundary coordinates of the fishing ban area is calculated as follows: traverse each edge of the fishing ban area boundary, calculate the perpendicular distance from the current location coordinates to each edge, and select the minimum value as the shortest distance. Simultaneously, calculate the azimuth angle of the current location coordinates relative to the boundary of the fishing ban area, which is the angle between the direction from the current location towards the center of the fishing ban area and true north.

[0156] The analysis examines the movement trends of suspected fishing vessels moving towards the prohibited fishing area. This is achieved by comparing the changes in distance from the current location to the boundary of the prohibited area with the locations at previous time points. A decreasing distance indicates the vessel is moving towards the prohibited area, while an increasing distance indicates it is moving away. By integrating information such as the shortest distance, azimuth, and movement trend, a location correlation feature is generated. This feature reflects the spatial relationship and dynamic changes between the vessel's current location and the prohibited fishing area.

[0157] Step S1631: Obtain the geographical boundary information of the fishing ban area. The geographical boundary information includes the coordinates of the polygonal boundary vertices of the fishing ban area. The geographical boundary information is represented by a set of polygonal vertex coordinates.

[0158] In this embodiment, the geographical boundary information of the fishing ban area is determined through prior geographic surveying and stored in the system's database. This geographical boundary information is specifically represented as a set of polygon boundary vertex coordinates, such as the latitude and longitude coordinates of multiple vertices (lat1, lon1), (lat2, lon2), ..., (latn, lonn). These vertices are connected sequentially to form a closed polygon, and the area enclosed by this polygon is the fishing ban area. When spatial comparison is required, this set of polygon vertex coordinates is read from the database as the basis for subsequent calculations.

[0159] Step S1632: Convert the current position coordinates of the suspected fishing ban vessel into coordinate points in the same coordinate system as the geographical boundary information, calculate the straight-line distance from the coordinate points to each vertex in the polygon vertex coordinate set, and arrange the straight-line distances in vertex order to generate a distance set.

[0160] In this embodiment, the current location coordinates of the suspected fishing vessel may be in image pixel coordinates or other local coordinate systems. These need to be converted to a coordinate system identical to the geographic boundary information, typically a latitude and longitude coordinate system. The conversion process is implemented using a coordinate transformation algorithm. This algorithm uses preset transformation parameters (such as the geographic location of the image acquisition device, lens parameters, etc.) to establish a mapping relationship between pixel coordinates and latitude and longitude coordinates, thereby converting the current location's pixel coordinates into latitude and longitude coordinate points (lat0, lon0).

[0161] Calculate the straight-line distance from the given coordinate point to each vertex in the set of polygon vertex coordinates. According to the distance formula between two points, for a vertex (lati, loni), the distance di = [(lat0 - lati)² + (lon0 - loni)²]^1 / 2, where i ranges from 1 to n. Arrange all the calculated distances d1, d2, ..., dn in the order of the vertex in the set to form a distance set.

[0162] Step S1633: Analyze the spatial positional relationship between the coordinate point and the polygon boundary, and calculate the mean and variance of the distance set as distance features based on the distance set, spatial positional relationship and movement direction of the suspected fishing ban vessel. Convert the spatial positional relationship into numerical code as positional state features, and use the angle between the movement direction and the direction pointing to the center of the fishing ban area as direction features. Construct a positional association feature that includes distance features, positional state features and direction features. The positional association feature is used to describe the dynamic positional relationship between the suspected fishing ban vessel and the fishing ban area.

[0163] In this embodiment, when analyzing the spatial relationship between the coordinate point (lat0, lon0) and the polygon boundary, the ray casting method is used to determine whether the coordinate point is located inside the no-fishing zone. Specifically, a ray is emitted from the coordinate point (lat0, lon0) in any direction, and the number of intersections between the ray and the polygon boundary is counted. If the number of intersections is odd, the coordinate point is determined to be inside the no-fishing zone; if the number of intersections is even, the coordinate point is determined to be outside the no-fishing zone; if the ray overlaps with the polygon boundary, the coordinate point is determined to be on the boundary of the no-fishing zone.

[0164] After obtaining the distance set, the mean and variance of the set are calculated. The mean is calculated by summing all the straight-line distances in the distance set and then dividing by the number of elements in the distance set, thus reflecting the average distance from the coordinate point to each vertex of the polygon. The variance is calculated by squared the difference between each straight-line distance and the mean, summing the squares, and then dividing by the number of elements in the distance set, which reflects the degree of deviation of each straight-line distance from the mean. These two parameters together constitute the distance characteristics.

[0165] For the numerical encoding of spatial location relationships, three values ​​are set to correspond to different location states: when the coordinate point is inside the fishing ban area, the code is 0; when the coordinate point is outside the fishing ban area, the code is 1; when the coordinate point is on the boundary of the fishing ban area, the code is 2. This numerical code is the location state feature.

[0166] When calculating the directional feature, first determine the center coordinates (lat_center, lon_center) of the fishing ban area. Then, calculate the direction vector pointing towards the center of the fishing ban area using the coordinate points (lat0, lon0) and the center coordinates (lat_center, lon_center). The direction of this vector is from the coordinate point to the center coordinates. Next, combine this with the movement direction vector of the suspected fishing vessel to calculate the angle between these two vectors. In the calculation process, first calculate the magnitudes of the two vectors and their dot product. Then, obtain the cosine value of the angle by the ratio of the dot product to the product of the magnitudes. Finally, determine the size of the angle based on the cosine value; this angle is the directional feature.

[0167] Distance features, position status features, and direction features are combined in sequence to form positional correlation features. These features can comprehensively describe the dynamic positional relationship between suspected fishing vessels and fishing-prohibited areas from three aspects: distance, position status, and direction of movement.

[0168] Step S164: Integrate and process the information of the suspected fishing ban behavior, including type, confidence level, current location coordinates, direction of movement, and location association features, to generate a structured set of early warning elements.

[0169] In this embodiment, after obtaining the type, confidence level, current location coordinates, movement direction, and location association features of the suspected fishing ban behavior, this information is integrated and processed. First, a corresponding field is set for each information item. For example, the "behavior type" field is used to store the type of suspected fishing ban behavior, the "confidence level" field is used to store the corresponding confidence level value, the "current location" field is used to store the current location coordinates, the "movement direction" field is used to store the movement direction information, and the "location association" field is used to store the location association features.

[0170] Then, each information item is filled into its corresponding field to ensure the accuracy of the information in each field. For numerical information, such as confidence level and numerical values ​​in location coordinates, their original numerical form is maintained; for textual descriptive information, such as behavior type, standardized textual descriptions are used; for composite information, such as location association features, the distance features, location status features, and direction features contained therein are combined according to a preset format and then stored in the corresponding field.

[0171] After the above processing, a structured set of early warning elements is formed. The information in this set of early warning elements is organized in an orderly manner and in a uniform format, which facilitates the generation of fishing ban early warning information based on this information.

[0172] Step S165: Based on the contents of the warning element set, generate a fishing ban warning information containing the real-time location and behavioral characteristics description of suspected fishing ban vessels.

[0173] In this embodiment, fishing ban warning information is generated based on a set of warning elements. First, the real-time location information of suspected fishing ban vessels is extracted from the set of warning elements, i.e., their current location coordinates, and converted into actual geographic coordinates, such as latitude and longitude, so that relevant personnel can accurately know the location of the vessels.

[0174] Next, information such as behavior type, confidence level, direction of movement, and location association features are extracted and described in text. For example, when describing the behavior type, the type of suspected fishing ban behavior is clearly indicated; when describing the confidence level, the credibility of the behavior being judged as a suspected fishing ban behavior is explained; when describing the direction of movement, the current movement trend of the vessel is described; and combined with location association features, the positional relationship between the vessel and the fishing ban area and the impact of possible movement trends on the fishing ban area are described.

[0175] The real-time location information and behavioral characteristics description are organized according to a preset template. The template structure includes a title, real-time location, and behavioral characteristics description. The title clearly indicates that this is a fishing ban warning; the real-time location clearly lists the latitude and longitude coordinates of the vessel; and the behavioral characteristics description details the vessel's behavior type, confidence level, direction of movement, and positional relationship with the prohibited fishing area.

[0176] After the above organization is completed, a fishing ban early warning information is generated, which allows relevant law enforcement personnel to quickly understand the specific situation of suspected fishing vessels.

[0177] Figure 2 The illustration shows exemplary hardware and software components of a river fishing vessel monitoring system 100 based on visible light and infrared fusion, which can implement the ideas of this application, according to some embodiments of this application. For example, a processor 120 can be used in the river fishing vessel monitoring system 100 based on visible light and infrared fusion and to perform the functions in this application.

[0178] For example, a river channel fishing vessel monitoring system 100 based on visible light and infrared fusion may include a network port 110 connected to a network, one or more processors 120 for executing program instructions, a communication bus 130, and various forms of storage media 140, such as a disk, ROM, or RAM, or any combination thereof. Exemplarily, the river channel fishing vessel monitoring system 100 based on visible light and infrared fusion may also include program instructions stored in ROM, RAM, or other types of non-transitory storage media, or any combination thereof. The methods of this application can be implemented according to these program instructions. The river channel fishing vessel monitoring system 100 based on visible light and infrared fusion also includes an I / O interface 150 between the computer and other input / output devices.

[0179] Furthermore, this embodiment of the invention also provides a readable storage medium, wherein computer-executable instructions are preset in the readable storage medium, and when the processor executes the computer-executable instructions, the above-mentioned method for monitoring river fishing vessels based on visible light and infrared fusion is implemented.

[0180] It should be noted that, in order to simplify the description of the present invention and thus help to understand one or more embodiments of the invention, multiple features may sometimes be grouped into one embodiment, drawing or description thereof in the foregoing description of the embodiments of the present invention.

Claims

1. A method for monitoring fishing vessels in river channels under fishing ban conditions based on visible light and infrared fusion, characterized in that, The method includes: Acquire visible light image sequences and infrared image sequences of the river monitoring area. The visible light image sequences contain texture details of the river surface and surrounding environment, and the infrared image sequences contain thermal radiation distribution characteristics within the river area. The visible light image sequence is processed to extract ship visual features to obtain a visible light ship feature set, and the infrared image sequence is processed to extract thermal features to obtain an infrared ship feature set. Perform cross-source feature fusion processing on the visible light ship feature set and the infrared ship feature set to generate a cross-source fused feature set; The pre-trained fishing ban vessel identification model is invoked to perform vessel attribute and behavior analysis on the cross-source fusion feature set, generating vessel monitoring results that include vessel type, location change trajectory, and suspected fishing ban behavior identifiers; Based on the vessel monitoring results, a fishing ban early warning information is generated, which includes the real-time location and behavioral characteristics description of suspected vessels subject to the fishing ban. The process of extracting ship visual features from the visible light image sequence to obtain a set of visible light ship features includes: Ship region localization processing is performed on consecutive frames in the visible light image sequence. The pixel region of each frame image is traversed by sliding window. After filtering out candidate regions that may contain ships by combining prior knowledge of ship outlines, the boundary information of the candidate regions is enhanced by edge detection algorithm to determine the range of the pixel region where the ship is located in each frame image. The range of the pixel region is represented by a set of bounding box coordinates. The visible light texture features within the pixel region are extracted, and the gray-level co-occurrence matrix algorithm is used to calculate the pixel gray-level correlation features under different directions and distances. Combined with the image details after Gaussian filtering, the pattern distribution features of the ship surface are generated. The edge points of the region are extracted by the Canny operator and the contour curve is fitted to obtain the edge contour features. The color gradient features are obtained by calculating the color difference between adjacent pixels based on the HSV color space. The visible light texture features include pattern distribution features, edge contour features and color gradient features. The variation trend of the ship pixel region range in the continuous frames is analyzed. The center coordinate offset and the rate of change of the ship bounding box in adjacent frames are calculated by the inter-frame difference method and integrated into the displacement parameters. At the same time, the motion trajectory of the ship's key pixels is tracked by the optical flow method to supplement the subtle motion information not covered in the displacement parameters and generate the ship motion dynamic features. The visible light texture features and the ship motion dynamic features are processed by feature dimension mapping. After performing dimensionality reduction or dimensionality increase operations through principal component analysis, the feature values ​​are mapped to the same numerical range through feature scaling, thereby converting the visible light texture features and the ship motion dynamic features into feature vectors of the same dimension. The feature vectors of the same dimension are concatenated according to the time sequence, and the feature vectors corresponding to each frame are arranged in the order of timestamps. The cosine similarity of adjacent timestamp feature vectors is calculated to verify the temporal continuity. The feature vectors that are temporally discontinuous in the temporal continuity state are smoothed to generate a set of visible light ship features containing temporal correlation. The process of extracting thermal features from the infrared image sequence to obtain an infrared ship feature set includes: Each frame of the infrared image sequence is segmented into thermal radiation regions. An adaptive threshold segmentation algorithm is used to determine the segmentation threshold based on the thermal radiation intensity distribution. The image is divided into multiple connected regions. By calculating the morphological parameters of each connected region, region units with continuous thermal radiation characteristics and conforming to the size characteristics of ships are selected. The region units correspond to the regions where ships may exist. The thermal radiation intensity distribution features of the region unit are extracted. By scanning each pixel in the region unit, the location of the maximum thermal radiation intensity is recorded as the thermal radiation peak position. The thermal radiation intensity decay rate from the thermal radiation peak position to the edge of the region unit is calculated to obtain the thermal radiation gradient change features. In addition, the area expansion of the region unit in consecutive frames is compared, and the thermal radiation range expansion features are generated by combining the thermal radiation intensity changes of the edge pixels. The thermal radiation intensity distribution features include the thermal radiation peak position, thermal radiation gradient change features, and thermal radiation range expansion features. The positional changes of the region unit in continuous infrared image frames are tracked. The region unit of the current frame is matched with the previous frame by template matching to determine the movement path of the center coordinate. The coordinate change per unit time is calculated to obtain the movement rate feature. The fluctuation of the movement rate is analyzed, and the frequency and amplitude features of the movement rate change are extracted and added to the movement rate feature. The thermal radiation intensity distribution feature and the movement rate feature are correlated and mapped, the correlation coefficient is calculated, a feature mapping matrix is ​​constructed based on the correlation coefficient, and the thermal radiation intensity distribution feature and the movement rate feature are projected into the correlation space of the feature mapping matrix respectively to generate a thermal radiation motion correlation feature vector. The thermal radiation motion-related feature vectors are arranged in a time sequence, organized according to the order of timestamps, and the feature distance between adjacent feature vectors is calculated to evaluate temporal consistency. Feature vectors whose feature distance exceeds a preset distance range are interpolated and adjusted to generate an infrared ship feature set containing temporal change patterns. Each feature vector in the infrared ship feature set contains thermal radiation and motion-related information corresponding to the timestamp.

2. The method for monitoring fishing vessels in river channels based on visible light and infrared fusion according to claim 1, characterized in that, The step of performing cross-source feature fusion processing on the visible light vessel feature set and the infrared vessel feature set to generate a cross-source fused feature set includes: Traverse the timestamp information in the visible light ship feature set and the infrared ship feature set, establish a timestamp correspondence table, and extract the visible light feature vector and infrared feature vector under the same timestamp according to the timestamp correspondence table; The cross-source attention mechanism module is invoked to calculate the association weights of the visible light feature vectors and infrared feature vectors under the same timestamp, generating a feature association weight matrix. The visible light feature vector and the infrared feature vector are weighted and fused according to the feature association weight matrix to generate a preliminary fused feature vector. The preliminary fused feature vector is subjected to multi-scale feature mapping to obtain local detail features at different scales. At the same time, global statistical information of the features is extracted as global distribution features through global average pooling operation. The local detail features and the global distribution features are channel concatenated. The feature channels of the local detail features and the global distribution features are concatenated in sequence. The numerical distribution between the channels is adjusted by batch normalization to generate a cross-source fusion feature set containing multi-scale information. Each feature vector in the cross-source fusion feature set contains multi-scale fusion feature information corresponding to the timestamp.

3. The method for monitoring fishing vessels in river channels based on visible light and infrared fusion according to claim 1, characterized in that, The pre-trained fishing ban vessel identification model is invoked to perform vessel attribute and behavior analysis on the cross-source fusion feature set, generating vessel monitoring results including vessel type, location change trajectory, and suspected fishing ban behavior identifiers, including: The cross-source fusion feature set is input into the feature encoding layer of the fishing ban vessel identification model, and the feature vector is generated by layer-by-layer abstraction through a multi-layer convolutional neural network. The encoded feature vector is input into the vessel type identification branch, and the probability distribution of various types of vessels is output. The vessel type information is determined based on the probability distribution, and the vessel type information includes category identifiers for fishing boats, transport boats, and sightseeing boats. The encoded feature vector is input into the trajectory prediction branch, and a temporal convolutional network is constructed by combining it with time series information. Multiple dilated convolutional layers are used to capture the dependencies at different time scales, and the predicted position coordinates for multiple future time steps are output. The predicted position coordinates are then combined with historical coordinates to generate a position change trajectory. The encoded feature vector is input into the behavior analysis branch to extract pattern features related to the fishing ban behavior. The pattern features include features of vessel dwell time, abnormal trajectory turning, and loitering in specific areas. Based on the matching results between the pattern features and the preset fishing ban behavior pattern library, the similarity between the pattern features and each preset fishing ban behavior pattern in the preset fishing ban behavior pattern library is calculated. The behavior tag corresponding to the preset fishing ban behavior pattern with the highest similarity is selected as the suspected fishing ban behavior identifier. The vessel type information, position change trajectory and suspected fishing ban behavior identifier are combined into the vessel monitoring result.

4. The method for monitoring fishing vessels in river channels based on visible light and infrared fusion according to claim 3, characterized in that, The step of inputting the encoded feature vector into the behavior analysis branch and extracting pattern features related to the fishing ban behavior includes: The encoded feature vector is segmented in the time dimension, and the continuous feature vector is divided into multiple subsequences according to a fixed time interval. Each subsequence corresponds to a time period, resulting in a sub-feature vector sequence corresponding to different time periods. Calculate the ship position distribution density in each sub-feature vector sequence, calculate the density of ship position coordinates using the kernel density estimation method, and filter out dense region features based on the density calculation results; Analyze the trajectory direction changes in the sub-feature vector sequence, calculate the trajectory direction angles at adjacent time points, determine whether the direction has changed significantly by the angle difference between the trajectory direction angles at adjacent time points, mark the abrupt change point when the angle difference exceeds the preset angle threshold, and record the angle change value of the abrupt change point to generate trajectory abnormal turning features. The frequency and duration of vessel appearances within the prohibited fishing area are statistically analyzed, and the wandering coefficient within the area is calculated by combining the position change trajectory. The wandering coefficient is positively correlated with the frequency and duration of appearance. Specifically, by comparing the vessel's position with the boundary of the prohibited fishing area, the time points when the vessel enters and leaves the prohibited fishing area are determined, the number of appearances is calculated as the frequency of appearance, and the time point difference is calculated as the duration. The dense area features, trajectory anomaly turning features, and wandering coefficient are subjected to feature aggregation processing to generate pattern features. Each dimension of the pattern features corresponds to a feature manifestation related to the fishing ban behavior.

5. The method for monitoring fishing vessels in river channels based on visible light and infrared fusion according to claim 3, characterized in that, The step of generating a suspected fishing ban behavior identifier based on the matching results of the pattern features and a preset fishing ban behavior pattern library includes: A set of typical fishing ban behavior patterns is extracted from a pre-set fishing ban behavior pattern library. This pre-set fishing ban behavior pattern library contains various fishing ban behavior pattern templates extracted from historical fishing ban enforcement records, covering fishing ban behavior performance in different seasons and at different times. The set of typical fishing ban behavior pattern features includes behavioral feature templates of vessels in historical fishing ban cases. Calculate the feature distance between the pattern feature and each fishing ban behavior feature template in the typical fishing ban behavior pattern feature set, and arrange all fishing ban behavior feature templates in ascending order of feature distance, and select the fishing ban behavior feature template corresponding to the smallest feature distance as the matching feature template; The feature overlap between the pattern feature and the matching feature template is calculated. When the feature overlap exceeds a preset overlap threshold, an identifier indicating the existence of suspected fishing ban behavior is generated, which includes the matched fishing ban behavior type and the overlap value. When the feature overlap does not exceed the preset overlap threshold, an identifier indicating the absence of suspected fishing ban behavior is generated, and the behavior type with the highest overlap is recorded as a reference. The suspected fishing ban behavior identifier includes behavior type and feature overlap information.

6. The method for monitoring fishing vessels in river channels based on visible light and infrared fusion according to claim 1, characterized in that, The generation of fishing ban early warning information based on the vessel monitoring results includes: The suspected fishing ban behavior identifiers in the vessel monitoring results are analyzed, and the behavior type codes and feature overlap values ​​contained therein are extracted. The behavior type codes are queried from the preset behavior severity level table, and the type and confidence level of the suspected fishing ban behavior are determined by combining the overlap values. The position change trajectory in the vessel monitoring results is extracted, and the vessel motion curve is obtained through a trajectory fitting algorithm. The position coordinates at the end of the vessel motion curve are taken as the current position coordinates of the suspected fishing ban vessel, and the tangent direction at the end of the vessel motion curve is calculated as the direction of movement. The current location coordinates are spatially compared with the boundary coordinates of the fishing ban area. The shortest distance and azimuth from the current location coordinates to the boundary coordinates of the fishing ban area are calculated. The trend of suspected fishing ban vessels moving towards the fishing ban area is analyzed, and location association features are generated. The types, confidence levels, current location coordinates, direction of movement, and location-related features of the suspected fishing violations are integrated and processed to generate a structured set of early warning elements. Based on the content of the aforementioned set of early warning elements, a fishing ban early warning message is generated, which includes the real-time location and behavioral characteristics of suspected fishing vessels.

7. The method for monitoring fishing vessels in river channels based on visible light and infrared fusion according to claim 6, characterized in that, The process involves spatially comparing the current location coordinates with the boundary coordinates of the fishing ban area, calculating the shortest distance and azimuth from the current location coordinates to the boundary coordinates of the fishing ban area, analyzing the trend of suspected fishing ban vessels moving towards the fishing ban area, and generating location association features, including: Obtain the geographic boundary information of the fishing ban area, which includes the coordinates of the vertices of the polygon boundary of the fishing ban area, and the geographic boundary information is represented by a set of polygon vertex coordinates. The current location coordinates of the suspected fishing ban vessel are converted into coordinate points in the same coordinate system as the geographical boundary information, and the straight-line distances from the coordinate points to each vertex in the set of polygon vertex coordinates are calculated. The straight-line distances are then arranged in vertex order to generate a distance set. The spatial relationship between the coordinate points and the polygon boundary is analyzed. Based on the distance set, the spatial relationship, and the movement direction of the suspected fishing ban vessel, the mean and variance of the distance set are calculated as distance features. The spatial relationship is converted into numerical codes as positional status features. The angle between the movement direction and the direction pointing to the center of the fishing ban area is used as a direction feature. A positional association feature including distance features, positional status features, and direction features is constructed. The positional association feature is used to describe the dynamic positional relationship between the suspected fishing ban vessel and the fishing ban area.

8. A river channel fishing vessel monitoring system based on visible light and infrared fusion, characterized in that, The device includes a processor and a memory, the memory and the processor being connected. The memory is used to store programs, instructions or code, and the processor is used to execute the programs, instructions or code in the memory to implement the river channel fishing ban vessel monitoring method based on visible light and infrared fusion as described in any one of claims 1-7.

Images