Visible light and infrared fusion-based river no-fishing ship monitoring method and system

Abstract

The invention provides a visible light and infrared fusion-based river channel fish banning ship monitoring method and system, and relates to the technical field of computer vision, and the method comprises the steps: firstly obtaining a visible light image sequence and an infrared image sequence of a river channel monitoring region; ship visual feature extraction and thermal feature extraction are carried out on the visible light image sequence and the infrared image sequence respectively to obtain a visible light ship feature set and an infrared ship feature set, and cross-source feature fusion processing is carried out on the two feature sets to generate a cross-source fusion feature set; and calling a pre-trained fish forbidding ship identification model to analyze the cross-source fusion feature set, generating a ship monitoring result including the ship type, the position change track and the suspected fish forbidding behavior identifier, and finally generating fish forbidding early warning information including the real-time position and behavior feature description of the suspected fish forbidding ship based on the ship monitoring result. And measures can be quickly taken to stop the fishing forbidding behavior.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of computer vision, in particular to a river fishing-ban ship monitoring method and system based on visible light and infrared fusion. BACKGROUND

[0002] In the river ecological protection work, fishing-ban management is an important measure to protect the ecological balance of water area and maintain the sustainable development of fishery resources. Accurate and timely monitoring of ship activities in the river and identifying whether there is fishing-ban behavior are key links of fishing-ban management.

[0003] At present, the common river ship monitoring method mainly relies on single type of sensor data. For example, the monitoring method based on visible light camera can clearly capture the texture detail features of river surface and surrounding environment, such as the appearance shape, color and the like of the ship, so as to preliminarily identify the ship visually. However, the quality of visible light image will be greatly reduced in the night or in bad weather (such as thick fog, heavy rain and the like), which leads to difficulty in ship identification and easy occurrence of missed detection and false detection.

[0004] Another is the monitoring method based on infrared sensor. The infrared image sequence can reflect the heat radiation distribution features in the river area, and the ship with heat source can be detected even in the environment with insufficient light. However, the infrared image lacks rich texture information, and the ability to identify the detail features and type of the ship is limited, which is difficult to accurately distinguish different types of ships and judge the specific behavior of the ship. SUMMARY

[0005] In view of the above-mentioned problems, in combination with the first aspect of the present application, the present application provides a river fishing-ban ship monitoring method based on visible light and infrared fusion, which comprises: obtaining a visible light image sequence and an infrared image sequence of a river monitoring area, the visible light image sequence containing texture detail features of river surface and surrounding environment, and the infrared image sequence containing heat radiation distribution features in the river area; performing ship visual feature extraction processing on the visible light image sequence to obtain a visible light ship feature set, and performing heat feature extraction processing on the infrared image sequence to obtain an infrared ship feature set; performing cross-source feature fusion processing on the visible light ship feature set and the infrared ship feature set to generate a cross-source fusion feature set; calling a pre-trained fishing-ban ship identification model to analyze the cross-source fusion feature set for ship attribute and behavior, and generating a ship monitoring result containing ship type, position change trajectory and suspected fishing-ban behavior identification; Generate a fishing ban warning information based on the ship monitoring result, and the fishing ban warning information contains the real-time position and behavior characteristic description of the suspected fishing ban ship.

[0006] In another aspect, the application also provides a river fishing ban ship monitoring system based on visible light and infrared fusion, comprising a processor and a machine readable storage medium, the machine readable storage medium is connected with the processor, the machine readable storage medium is used to store programs, instructions or codes, and the processor is used to execute the programs, instructions or codes in the machine readable storage medium to realize the above method.

[0007] Based on the above aspects, the application fully utilizes the advantages of the texture detail features of the river surface and the surrounding environment contained in the visible light image and the thermal radiation distribution features contained in the infrared image by acquiring the visible light image sequence and the infrared image sequence of the river monitoring area, performs visual feature extraction processing on the visible light image sequence to obtain a visible light ship feature set, performs thermal feature extraction processing on the infrared image sequence to obtain an infrared ship feature set, can accurately capture the key features of the ship from different dimensions respectively, performs cross-source feature fusion processing on the two feature sets to generate a cross-source fusion feature set, calls a pre-trained fishing ban ship identification model to analyze the ship attributes and behaviors of the cross-source fusion feature set, can accurately generate a ship monitoring result containing the ship type, position change trajectory and suspected fishing ban behavior identification, greatly improves the accuracy and comprehensiveness of ship monitoring, generates a fishing ban warning information containing the real-time position and behavior characteristic description of the suspected fishing ban ship based on the ship monitoring result, which helps to quickly take measures to stop the fishing ban behavior and effectively protect the river ecological environment and fishery resources. BRIEF DESCRIPTION OF DRAWINGS

[0008] Figure 1 is the execution flow diagram of the river fishing ban ship monitoring method based on visible light and infrared fusion provided by the embodiment of the application.

[0009] Figure 2 is the schematic diagram of exemplary hardware and software components of the river fishing ban ship monitoring system based on visible light and infrared fusion provided by the embodiment of the application. DETAILED DESCRIPTION

[0010] The application will be specifically described below in conjunction with the drawings of the specification, Figure 1 is the flow diagram of the river fishing ban ship monitoring method based on visible light and infrared fusion provided by an embodiment of the application, and the river fishing ban ship monitoring method based on visible light and infrared fusion will be described in detail below.

[0011] Step S110: Obtain a visible light image sequence and an infrared image sequence of the river monitoring area, the visible light image sequence containing texture detail features of the river surface and the surrounding environment, and the infrared image sequence containing thermal radiation distribution features in the river area.

[0012] In this embodiment, a natural river is selected as the monitoring object. The river has a fishing prohibition period, and the ships in the area need to be monitored to prevent illegal fishing behavior. To obtain the required visible light image sequence and infrared image sequence, a plurality of image acquisition devices can be installed at appropriate positions along the river bank. Each image acquisition device includes a visible light camera and an infrared thermal imager. These image acquisition devices are fixed by a support and the lenses are directed towards the river surface and the surrounding area to ensure that the preset monitoring range can be covered.

[0013] Among them, the visible light camera adopts an industrial grade camera, and the lens parameter is set to a specification suitable for long distance shooting of the water surface scene, which can capture the wave pattern of the water surface, the color difference of the ship, the pattern on the surface of the ship body, etc. The infrared thermal imager is selected to have a higher thermal sensitivity, which can detect small temperature changes on the surface of the object, thereby reflecting the thermal radiation of different objects, such as the high thermal radiation area generated by the ship engine when working, the thermal radiation track formed by the personnel on the deck, etc.

[0014] In addition, all image acquisition devices are controlled by a unified time synchronization device to ensure that the visible light camera and the infrared thermal imager collect images at the same time, and the collection frequency is set to a fixed value, so that the generated visible light image sequence and infrared image sequence correspond to each other in time. During the collection process, in order to protect the privacy of individuals that may appear in the image, real-time preprocessing is performed on the image. An image segmentation algorithm based on deep learning is used to automatically identify the privacy information area such as face and vehicle license plate that may be contained in the image, and pixel-level blur processing is performed on these areas to ensure that the privacy information is not leaked. The processed visible light image sequence and infrared image sequence are stored in the specified directory of the local server, waiting for subsequent processing.

[0015] Step S120: Perform ship visual feature extraction processing on the visible light image sequence to obtain a visible light ship feature set, and perform thermal feature extraction processing on the infrared image sequence to obtain an infrared ship feature set.

[0016] In this embodiment, after obtaining the visible light image sequence and the infrared image sequence, feature extraction is performed on the two different types of image sequences respectively. For the visible light image sequence, the visual features that can reflect the appearance and motion state of the ship are extracted; for the infrared image sequence, the features related to the thermal radiation of the ship are extracted for subsequent fusion processing.

[0017] Step S121: Ship region positioning processing is performed on the continuous frames in the visible light image sequence. By sliding window traversal of the pixel region of each frame of image, after screening out the candidate region possibly containing the ship in combination with the prior knowledge of ship contour, the boundary information of the candidate region is strengthened by using the edge detection algorithm, and the pixel region range where the ship is located in each frame of image is determined. The pixel region range is expressed by a boundary box coordinate set.

[0018] In this embodiment, taking one frame of image in the visible light image sequence as an example, the ship region positioning is performed. First, the size of the sliding window is determined. The size is set according to the approximate proportion of common ships in the image, which ensures that the pixel range possibly occupied by ships of different sizes can be covered. The sliding window starts from the top left corner of the image and moves in the horizontal and vertical directions in turn. The step length of each movement is set according to the resolution of the image and the processing efficiency, so that the window can traverse each possible region in the image.

[0019] During the traversal process, the image features in the window are extracted every time the window moves to a position, and are compared with the preset prior knowledge of ship contour. The prior knowledge of ship contour is summarized by analyzing a large number of sample ship images, including common shape features of ships, such as approximate length-width ratio, overall streamline or rectangular contour, etc. When the matching degree of the image features in the window with the prior knowledge reaches a set threshold, the window is marked as a candidate region.

[0020] After the screening of the candidate region is completed, the edge detection algorithm is applied to each candidate region. The Canny edge detection algorithm is selected. First, the image of the candidate region is processed by grayscale, then the gradient amplitude and direction of the image are calculated, the stray response in the edge detection result is eliminated by non-maximum suppression, and finally the real and potential edges are determined by using the double-threshold algorithm, and the edges are connected into a complete contour by the edge connection algorithm, so as to strengthen the boundary information of the candidate region.

[0021] According to the strengthened boundary information, the pixel region range where the ship is located is determined. For each determined ship region, a boundary box is used for marking. The boundary box coordinate set is composed of the pixel coordinates of the upper left corner and the lower right corner of the boundary box, for example (a1, b1, a2, b2), wherein (a1, b1) is the upper left corner coordinate, and (a2, b2) is the lower right corner coordinate. The above coordinate set can accurately frame the position of the ship in the image.

[0022] Step S122: Extract visible light texture features in the pixel region range, calculate pixel gray correlation features in different directions and distances using a gray level co-occurrence matrix algorithm, combine image details after Gaussian filtering to generate pattern distribution features on the surface of the ship, extract regional edge points by a Canny operator and fit a contour curve to obtain edge contour features, calculate color difference values of adjacent pixels based on an HSV color space to obtain color gradient features. The visible light texture features include the pattern distribution features, the edge contour features, and the color gradient features.

[0023] In this embodiment, after determining the pixel region range of the ship, visible light texture features are extracted from the image in the region. First, pixel gray correlation features are calculated using a gray level co-occurrence matrix algorithm. The image of the ship region is converted into a gray image, and then in the gray image, multiple different directions such as 0 degrees, 45 degrees, 90 degrees, and 135 degrees, and multiple different pixel spacings are set. For each direction and spacing, the occurrence probability of two pixel points having a specific gray value combination in the gray image is calculated, thereby generating a gray level co-occurrence matrix. From the gray level co-occurrence matrix, feature parameters such as energy, entropy, contrast, and correlation are extracted, which together constitute pixel gray correlation features in different directions and distances.

[0024] Next, the image of the ship region is subjected to Gaussian filtering. A preset Gaussian kernel is used for the Gaussian filtering, and the size of the Gaussian kernel is selected according to the richness of the details of the image. The image is smoothed by convolution operation to remove noise in the image and retain important detail information. The image after Gaussian filtering is combined with the previously obtained pixel gray correlation features to analyze the pattern distribution on the surface of the ship, such as the position, shape, and distribution density of patterns such as stripes, spots, and markings, to generate pattern distribution features on the surface of the ship.

[0025] Then, regional edge points are extracted by a Canny operator. The Canny edge detection algorithm is applied again to the gray image of the ship region to obtain a set of edge points. Curve fitting algorithms are used to process these edge points, and according to the coordinate distribution of the edge points, curves that reflect the shape of the ship's contour are fitted, such as the side contour curve of the ship body, the curves of the bow and stern, and the like. These curves together constitute edge contour features.

[0026] Finally, color gradient features are calculated based on an HSV color space. The image of the ship region is converted from an RGB color space to an HSV color space to obtain hue, saturation, and lightness values for each pixel. The difference values of adjacent pixels in the hue, saturation, and lightness channels are calculated, which reflect the color changes. These difference values are arranged according to the position of the pixels to form color gradient features, which can reflect the color transition of different regions on the surface of the ship.

[0027] Step S123: Analyzing the trend of the change of the ship pixel region range in the continuous frames, calculating the center coordinate offset of the ship bounding box in the adjacent frames and the boundary box size change rate by the inter-frame difference method and integrating them into the displacement parameter, and combining the optical flow method to track the motion trajectory of the key pixel points of the ship to supplement the subtle motion information not covered in the displacement parameter, and generating the ship motion dynamic characteristics.

[0028] In this embodiment, for the continuous frames of the visible light image sequence, the trend of the change of the ship pixel region range is analyzed to obtain the motion dynamic characteristics. First, the inter-frame difference method is used to select two continuous frames of images, assuming that the previous frame is image F(t) and the current frame is image F(t+1), and the bounding box coordinates of the ship in the two frames of images are obtained.

[0029] The center coordinates of the bounding box are calculated. For the bounding box (a1, b1, a2, b2) in the image F(t), the center coordinates are ((a1+a2) / 2, (b1+b2) / 2), denoted as (c1, d1); the center coordinates of the bounding box in the image F(t+1) are (c2, d2). The center coordinate offset is (c2-c1, d2-d1), which reflects the overall movement distance of the ship in the horizontal and vertical directions.

[0030] At the same time, the size change rate of the bounding box is calculated. The width of the bounding box in the image F(t) is (a2-a1), and the height is (b2-b1); the width of the bounding box in the image F(t+1) is (a2'-a1'), and the height is (b2'-b1'). The width change rate is ((a2'-a1')-(a2-a1)) / (a2-a1), and the height change rate is ((b2'-b1')-(b2-b1)) / (b2-b1). The center coordinate offset and the size change rate of the bounding box are integrated together to form the displacement parameter.

[0031] Next, the motion trajectory of the key pixel points of the ship is tracked by combining the optical flow method. A plurality of key pixel points are selected in the ship region, which are usually turning points, corner points and other representative pixels on the edge of the ship. Using the Lucas-Kanade optical flow algorithm, assuming that the motion of these key pixel points between adjacent frames satisfies the brightness constancy and motion smoothness assumptions, the position change of each key pixel point in the images F(t) and F(t+1) is calculated to obtain the motion vector of each key pixel point.

[0032] The motion vectors are combined with the displacement parameters obtained previously, which reflect the overall motion of the ship, and the motion vectors of the key pixels, which can supplement the fine motion information not covered by the displacement parameters, such as slight rotation of the ship, local shaking, etc. The information is integrated to generate a ship motion dynamic feature, which can comprehensively reflect the motion state of the ship in consecutive frames.

[0033] Step S124: Perform feature dimension mapping processing on the visible light texture feature and the ship motion dynamic feature. After dimension reduction or dimension increase operation by principal component analysis algorithm, map the feature values to the same numerical interval by feature scaling, and convert the visible light texture feature and the ship motion dynamic feature into feature vectors of the same dimension.

[0034] In this embodiment, since the visible light texture feature and the ship motion dynamic feature may have different dimensions, feature dimension mapping processing is needed to enable them to perform subsequent fusion operation. First, apply principal component analysis algorithm to the visible light texture feature and the ship motion dynamic feature respectively.

[0035] For the visible light texture feature, it is represented as a feature matrix, each row of which represents a sample and each column of which represents a feature dimension. By calculating the covariance matrix of the matrix, then solving the eigenvalues and eigenvectors of the covariance matrix, the first k eigenvectors that can explain most of the data variance are selected, the visible light texture feature is projected into the new space composed of these eigenvectors to realize dimension reduction operation; if the dimension of the visible light texture feature is low, the extension method in the principal component analysis algorithm is used for dimension increase to generate a higher dimensional feature representation.

[0036] The same processing is performed on the ship motion dynamic feature. According to the relationship between the original dimension and the target dimension, corresponding dimension reduction or dimension increase operation is performed to make the processed visible light texture feature and the ship motion dynamic feature have the same dimension.

[0037] After completing the dimension adjustment, feature scaling processing is performed. The min-max scaling method is adopted. For each feature value in the processed visible light texture feature, it is calculated according to the formula (x-min_x) / (max_x-min_x), where x is the feature value, min_x is the minimum value of the feature dimension, and max_x is the maximum value of the feature dimension, and the feature value is mapped to the interval [0, 1]. The same scaling processing is also performed on the ship motion dynamic feature to make the feature values of the two in the same numerical interval. After the above processing, the visible light texture feature and the ship motion dynamic feature are converted into feature vectors of the same dimension.

[0038] Step S125: The feature vectors of the same dimension are spliced in time sequence, the feature vectors corresponding to each frame are arranged in time stamp order, the cosine similarity of adjacent time stamp feature vectors is calculated to verify the time sequence continuity, the feature vectors with time sequence discontinuity are processed by smooth transition, and a visible light ship feature set containing time sequence correlation is generated.

[0039] In this embodiment, the feature vectors of the same dimension obtained after the dimension mapping processing are spliced in time sequence. Each feature vector corresponds to the processing result of a frame of image, and these feature vectors are arranged in sequence according to the time stamp order of image acquisition, forming a sequence.

[0040] To verify the time sequence continuity, the cosine similarity of adjacent time stamp feature vectors is calculated. For the feature vector V(t) corresponding to the time stamp t and the feature vector V(t+1) corresponding to the time stamp t+1, the cosine similarity is calculated by dividing the dot product of the two vectors by the product of their moduli. The closer the result is to 1, the higher the similarity of the two feature vectors and the better the time sequence continuity; otherwise, the time sequence continuity is poor.

[0041] When the calculated cosine similarity is lower than the preset threshold, it is determined that the time sequence is not continuous. At this time, the feature vectors that are not continuous are processed by smooth transition, and linear interpolation is used to generate a plurality of transition feature vectors between V(t) and V(t+1), so that the change of the feature vector is more stable. For example, if there is time sequence discontinuity between V(t) and V(t+1), transition vectors V(t,1), V(t,2), etc. are generated, where V(t,1)=V(t)+1 / n*(V(t+1)-V(t)), V(t,2)=V(t)+2 / n*(V(t+1)-V(t)), and n is the number of transition vectors.

[0042] After the above processing, a visible light ship feature set containing time sequence correlation is formed. The feature vectors in the visible light ship feature set are arranged in time sequence, and the adjacent feature vectors have good continuity, which can reflect the feature change of the ship at different time points.

[0043] Step S130: Perform heat feature extraction processing on the infrared image sequence to obtain an infrared ship feature set.

[0044] Step S131: Perform heat radiation region segmentation processing on each frame of image in the infrared image sequence, determine a segmentation threshold according to the heat radiation intensity distribution by using an adaptive threshold segmentation algorithm, divide the image into a plurality of connected regions, and select a region unit with continuous heat radiation characteristics and in line with the size characteristics of the ship by calculating the morphological parameters of each connected region. The region unit corresponds to a region where a ship may exist.

[0045] In this embodiment, for each frame of image in the infrared image sequence, thermal radiation region segmentation is performed. First, the gray value of each pixel in the infrared image is converted into a corresponding thermal radiation intensity value, and the higher the gray value, the greater the thermal radiation intensity. An adaptive threshold segmentation algorithm is used, which automatically determines the segmentation threshold according to the distribution of thermal radiation intensity in the image. The specific process is to divide the image into multiple sub-regions, calculate a local threshold for each sub-region, and then segment the entire image according to these local thresholds, mark the region with thermal radiation intensity higher than the threshold as foreground, and mark the region with thermal radiation intensity lower than the threshold as background.

[0046] After segmentation, multiple connected regions are obtained, each connected region being composed of mutually connected foreground pixels. The morphological parameters of each connected region are calculated, including the area, perimeter, aspect ratio of the circumscribed rectangle, etc. of the region. According to the general size characteristics of the ship, the range of area, the range of aspect ratio, etc. are set as screening conditions, and the connected regions are screened. For example, regions with too small area may be noise or small floating objects, which are excluded; regions with aspect ratio not conforming to the characteristics of the ship are also excluded.

[0047] After screening, the obtained region unit has continuous thermal radiation characteristics, and its morphological parameters conform to the size characteristics of the ship. These region units are considered as regions where ships may exist.

[0048] Step S132: Extract the thermal radiation intensity distribution characteristics of the region unit, record the position of the maximum thermal radiation intensity value as the thermal radiation peak position by scanning each pixel point in the region unit, calculate the thermal radiation intensity decay rate from the thermal radiation peak position to the edge of the region unit to obtain the thermal radiation gradient change characteristics, and generate the thermal radiation range expansion characteristics by comparing the area expansion of the region unit in consecutive frames and combining the thermal radiation intensity change of the edge pixel. The thermal radiation intensity distribution characteristics include the thermal radiation peak position, the thermal radiation gradient change characteristics, and the thermal radiation range expansion characteristics.

[0049] In this embodiment, after determining the region unit, the thermal radiation intensity distribution characteristics thereof are extracted. First, each pixel point in the region unit is scanned, and the thermal radiation intensity value of each pixel point is recorded. The maximum value is found out, and the pixel coordinates of the maximum value are the thermal radiation peak position. The thermal radiation peak position usually corresponds to the part of the ship with more heat generation, such as the engine.

[0050] Then, the thermal radiation intensity decay rate from the thermal radiation peak position to the edge of the region unit is calculated. A plurality of rays are drawn from the thermal radiation peak position to the edge of the region unit, and the ratio of the change in thermal radiation intensity to the distance from the peak position to the edge along each ray is calculated, which is the decay rate. The decay rates of the plurality of rays are combined to form a thermal radiation gradient change feature, which reflects the change in thermal radiation intensity from the peak to the edge.

[0051] Then, a thermal radiation range expansion feature is generated. By comparing the area changes of the same region unit in consecutive frames, the expansion rate or contraction rate of the area is calculated. At the same time, the change in thermal radiation intensity of the edge pixel of the region unit is monitored. If the thermal radiation intensity of the edge pixel increases, it may mean that the ship is expanding in that direction or the thermal radiation range is expanding. The area expansion and the change in thermal radiation intensity of the edge pixel are combined to generate the thermal radiation range expansion feature, which can reflect the dynamic changes of the thermal radiation region of the ship.

[0052] Step S133: Track the position change of the region unit in consecutive infrared image frames, match the region unit of the current frame with the previous frame by template matching method, determine the moving path of the center coordinates, calculate the coordinate change in unit time to obtain the moving speed feature, and analyze the fluctuation of the moving speed to extract the frequency and amplitude features of the moving speed change and supplement them to the moving speed feature.

[0053] In this embodiment, the template matching method is used to track the position change of the region unit in consecutive infrared image frames. The region unit in the previous frame image is taken as a template, and the template is slid in the current frame image according to a preset search range. The similarity of the template and each region in the search range is calculated, and the region with the highest similarity is the region in the current frame that matches the region unit in the previous frame.

[0054] Through the matching result, the center coordinates of the region unit in each frame image are determined, and these center coordinates are connected in time sequence to form a moving path of the center coordinates. According to the moving path, the change amount of the center coordinates in unit time, i.e. the moving speed, is calculated. The size and direction of the moving speed together constitute the basic part of the moving speed feature.

[0055] The fluctuation of the moving speed is further analyzed, and the change frequency and amplitude of the moving speed in different time periods are calculated. The frequency refers to the number of rate changes in unit time, and the amplitude refers to the difference between the maximum and minimum values of the rate change. The frequency and amplitude features are supplemented to the moving speed feature, so that the moving speed feature can more comprehensively reflect the motion state of the region unit.

[0056] Step S134: The thermal radiation intensity distribution feature and the moving speed feature are associated and mapped, a correlation coefficient is calculated, a feature mapping matrix is constructed according to the correlation coefficient, the thermal radiation intensity distribution feature and the moving speed feature are projected into the correlation space of the feature mapping matrix respectively, and a thermal radiation motion correlation feature vector is generated.

[0057] In this embodiment, after obtaining the thermal radiation intensity distribution feature and the moving speed feature, the two are associated and mapped to establish the correlation between the thermal radiation characteristics and the motion state. First, the correlation coefficient between the thermal radiation intensity distribution feature and the moving speed feature is calculated. The thermal radiation intensity distribution feature contains multiple dimensional parameters, such as the coordinates of the thermal radiation peak position, the thermal radiation gradient change value in different directions, the thermal radiation range expansion rate, etc.; the moving speed feature also contains multiple dimensional parameters, such as the size, direction, change frequency and change amplitude of the moving speed, etc.

[0058] For each dimensional parameter in the thermal radiation intensity distribution feature and each dimensional parameter in the moving speed feature, the linear correlation degree between them is calculated by using the Pearson correlation coefficient calculation method, and a plurality of correlation coefficients are obtained. The above correlation coefficients constitute a two-dimensional matrix, the rows of the matrix correspond to the dimensions of the thermal radiation intensity distribution feature, the columns correspond to the dimensions of the moving speed feature, and each element represents the correlation degree between the corresponding two dimensional parameters.

[0059] According to the obtained correlation coefficients, a feature mapping matrix is constructed, the element values of the feature mapping matrix are determined by the correlation coefficients, and the correlation weights between the dimensions of the thermal radiation intensity distribution feature and the moving speed feature are reflected. Then, the thermal radiation intensity distribution feature vector and the moving speed feature vector are respectively subjected to matrix multiplication operation with the feature mapping matrix, and are projected into the correlation space defined by the feature mapping matrix.

[0060] The projected thermal radiation intensity distribution feature and moving speed feature have stronger correlation in the correlation space. The two projected feature vectors are spliced together according to the channel to generate a thermal radiation motion correlation feature vector. The thermal radiation motion correlation feature vector contains thermal radiation intensity distribution information and motion speed information, and the inherent correlation between the two is reflected.

[0061] Step S135: The thermal radiation motion correlation feature vectors are arranged in time sequence, the feature vectors are organized in the order of time stamp, the feature distance of adjacent feature vectors is calculated to evaluate the time sequence consistency, the feature vectors with feature distance exceeding the preset distance range are interpolated and adjusted, an infrared ship feature set containing time sequence change rule is generated, and each feature vector in the infrared ship feature set contains thermal radiation and motion correlation information corresponding to the time stamp.

[0062] In this embodiment, the thermal radiation and motion correlation feature vectors are arranged in the order of the timestamps of image acquisition to form an ordered feature vector sequence. Each feature vector corresponds to a timestamp and contains the thermal radiation and motion correlation information of the region unit at the timestamp.

[0063] To evaluate the time sequence consistency, the feature distance between the thermal radiation and motion correlation feature vectors corresponding to adjacent timestamps is calculated. The feature distance is calculated using the Euclidean distance method, that is, for the feature vector U(t) of timestamp t and the feature vector U(t+1) of timestamp t+1, the square root of the sum of squares of the dimension differences between the two vectors is calculated, and the result is the Euclidean distance between the two. The smaller the distance value, the more stable the change of adjacent feature vectors, and the better the time sequence consistency; otherwise, the time sequence consistency is poor.

[0064] A feature distance range is preset, and when the Euclidean distance of adjacent feature vectors calculated exceeds the range, the feature vectors are interpolated and adjusted. A cubic spline interpolation method is used to generate several transition feature vectors between U(t) and U(t+1) according to their numerical values, so that the change trend of the feature vectors is smoother. For example, if the distance between U(t) and U(t+1) is too large, transition vectors U(t,1), U(t,2), etc. are generated. These transition vectors are calculated by a cubic spline function and can well fit the change process from U(t) to U(t+1).

[0065] After the above processing, an infrared ship feature set containing time sequence change rules is formed. The feature vectors in the infrared ship feature set are arranged in time sequence, and the adjacent feature vectors have good time sequence consistency, which can reflect the dynamic changes of the thermal radiation and motion correlation information of the region unit at different time points.

[0066] Step S140: performing cross-source feature fusion processing on the visible light ship feature set and the infrared ship feature set to generate a cross-source fusion feature set.

[0067] In this embodiment, the visible light ship feature set and the infrared ship feature set respectively describe the features of the ship from different angles. In order to comprehensively utilize this information, cross-source feature fusion processing is needed. Through fusion processing, the two different source features are organically combined to generate a more comprehensive and more discriminative cross-source fusion feature set.

[0068] Step S141: traversing the timestamp information in the visible light ship feature set and the infrared ship feature set to establish a timestamp correspondence table, and extracting the visible light feature vector and the infrared feature vector under the same timestamp according to the timestamp correspondence table.

[0069] In this embodiment, the timestamp information corresponding to all feature vectors in the visible light ship feature set and the infrared ship feature set is first traversed to collect the timestamp of each feature vector. Since the visible light image sequence and the infrared image sequence are acquired synchronously, theoretically their timestamps should correspond one-to-one. However, there may be cases where some frames are lost or the timestamp recording is inaccurate. Therefore, it is necessary to establish a timestamp correspondence table.

[0070] For each timestamp in the visible light vessel feature set, the infrared vessel feature set is searched for timestamps that are the same or closest, and the two are matched and recorded in a timestamp correspondence table. For feature vectors that cannot be matched with a corresponding timestamp, interpolation is performed based on the matching situation before and after the timestamp to generate a corresponding virtual feature vector to ensure the integrity of the time series.

[0071] Based on the established timestamp correspondence table, the visible light feature vector and infrared feature vector corresponding to each timestamp are extracted to ensure that the two feature vectors at the same time point are used in subsequent processing.

[0072] Step S142: Call the cross-source attention mechanism module to calculate the association weights of the visible light feature vector and infrared feature vector under the same timestamp, and generate a feature association weight matrix.

[0073] In this embodiment, the cross-source attention mechanism module is invoked to calculate the correlation weight between the visible light feature vector and the infrared feature vector at the same timestamp. The cross-source attention mechanism module adopts a multilayer perceptron structure, and first concatenates the visible light feature vector and the infrared feature vector to form a concatenated feature vector.

[0074] The concatenated feature vector is input into the first fully connected layer of the multilayer perceptron, and after processing by the activation function, an intermediate feature vector is obtained. The intermediate feature vector is then input into the second fully connected layer, whose output dimension is the product of the visible light feature vector dimension and the infrared feature vector dimension. The output result is used to reshape the feature association weight matrix.

[0075] Each element in the feature association weight matrix represents the correlation strength between the corresponding dimension in the visible light feature vector and the corresponding dimension in the infrared feature vector. The larger the weight value, the stronger the correlation between the two dimensions, and the more attention should be paid to them during fusion.

[0076] Step S143: Perform weighted fusion processing on the visible light feature vector and the infrared feature vector according to the feature association weight matrix to generate a preliminary fused feature vector.

[0077] In this embodiment, the visible light feature vector and the infrared feature vector are weighted and fused according to the feature correlation weight matrix. First, the feature correlation weight matrix is multiplied by the visible light feature vector to obtain a weighted visible light feature vector. Each dimension value in the visible light feature vector is the weighted result of the corresponding dimension of the original visible light feature vector and the weight value in the corresponding row of the correlation weight matrix.

[0078] Similarly, the transpose of the feature correlation weight matrix is multiplied by the infrared feature vector to obtain a weighted infrared feature vector. Then, the weighted visible light feature vector and the weighted infrared feature vector are spliced by channel to generate a preliminary fusion feature vector.

[0079] The preliminary fusion feature vector combines important information of the visible light feature and the infrared feature, and highlights the strong correlation dimensions between the two and weakens the weak correlation dimensions through the correlation weight matrix, so that the fused feature can better reflect the essential features of the ship.

[0080] Step S144: performing multi-scale feature mapping processing on the preliminary fusion feature vector to obtain local detail features at different scales, and extracting global statistical information of the features as global distribution features through a global average pooling operation.

[0081] In this embodiment, the preliminary fusion feature vector is subjected to multi-scale feature mapping processing. A plurality of convolution kernels of different sizes are used to perform convolution operations on the preliminary fusion feature vector, and each convolution kernel corresponds to a scale. For example, a small-size convolution kernel is used to extract fine-grained local detail features such as small textures on the surface of the ship and local mutations of thermal radiation; and a large-size convolution kernel is used to extract coarse-grained local detail features such as the overall contour region of the ship and the large-scale distribution of thermal radiation.

[0082] Each convolution operation is subjected to an activation function processing to obtain local detail features at different scales. The above local detail features describe the local information in the preliminary fusion feature vector from different spatial scales.

[0083] Meanwhile, the preliminary fusion feature vector is subjected to a global average pooling operation, and the average values of each feature channel are calculated. These average values are combined to form global distribution features. The global distribution features reflect the statistical distribution of the preliminary fusion feature vector in the entire feature space, and can supplement the global information missing in the local detail features.

[0084] Step S145: Perform channel concatenation processing on the local detail features and the global distribution features, sequentially splice the feature channels of the local detail features and the global distribution features, adjust the numerical value distribution between the channels through batch normalization operation, and generate a cross-source fusion feature set containing multi-scale information. Each feature vector in the cross-source fusion feature set contains multi-scale fusion feature information of the corresponding timestamp.

[0085] In this embodiment, the local detail features and the global distribution features at different scales are processed through channel concatenation. First, all the local detail features are spliced in the order of increasing scale to form a local feature set. Then, the global distribution features are spliced with the local feature set to make the spliced feature vector contain both local detail information at multiple scales and global statistical information.

[0086] After splicing, batch normalization operation is performed on the obtained feature vector. The mean and variance of each feature channel are calculated, and the feature value of each channel is subtracted from the mean of the channel and then divided by the standard deviation of the channel, so that the feature values of each channel have a distribution characteristic of zero mean and unit variance.

[0087] Through batch normalization operation, the numerical value distribution difference between different channels is adjusted, avoiding the influence of excessively large or small feature values of some channels on subsequent processing. After the above processing, a cross-source fusion feature set containing multi-scale information is generated. Each feature vector in the cross-source fusion feature set corresponds to a timestamp and contains multi-scale fusion feature information of the ship at that time point.

[0088] Step S150: Call the pre-trained illegal fishing ship identification model to analyze the attributes and behaviors of the cross-source fusion feature set, and generate a ship monitoring result containing the ship type, location change trajectory, and suspected illegal fishing behavior identifier.

[0089] In this embodiment, the cross-source fusion feature set contains rich ship feature information, and the pre-trained illegal fishing ship identification model is called to process it to analyze the attributes and behaviors of the ship. The illegal fishing ship identification model is trained by a large amount of labeled data, can extract key information from the cross-source fusion feature, identify the type of the ship, track the location change trajectory, and judge whether there is suspected illegal fishing behavior, and finally generate a comprehensive ship monitoring result.

[0090] Step S151: Input the cross-source fusion feature set into the feature encoding layer of the illegal fishing ship identification model, and perform layer-by-layer abstraction through a multi-layer convolutional neural network to generate an encoded feature vector.

[0091] In this embodiment, each feature vector in the cross-source fusion feature set is sequentially input into the feature encoding layer of the illegal fishing vessel identification model. The feature encoding layer is composed of multiple layers of convolutional neural networks. The first convolutional layer uses multiple convolutional kernels to perform convolution operations on the input cross-source fusion feature vector, extracting low-level features such as edges and local textures.

[0092] The results of the convolution operation are processed by an activation function and input into a pooling layer for downsampling, which reduces the feature dimension while preserving important features. The downsampled features are input into the second convolutional layer, which uses more convolutional kernels to extract higher-level features such as component combinations of vessels, correlation patterns of thermal radiation and morphology, etc.

[0093] The subsequent convolutional layers and pooling layers repeat the above process, with each layer performing more in-depth abstraction and extraction of features based on the previous layer. After processing by multiple layers of convolutional neural networks, a high-dimensional feature map is obtained, which is flattened into a one-dimensional vector, i.e., the encoding feature vector. The encoding feature vector is a highly abstracted representation of the cross-source fusion features, containing key feature information of the vessel.

[0094] Step S152: input the encoding feature vector into the vessel type identification branch, output the probability distribution of each type of vessel, and determine the vessel type information based on the probability distribution, which includes the categories of fishing vessels, transport vessels, and sightseeing vessels.

[0095] In this embodiment, the encoding feature vector is input into the vessel type identification branch of the illegal fishing vessel identification model. The vessel type identification branch is composed of multiple fully connected layers. The first fully connected layer receives the encoding feature vector and maps the features to an intermediate feature space through matrix operations and activation function processing.

[0096] The feature vector of the intermediate feature space is input into the second fully connected layer, and the output dimension of the second fully connected layer is consistent with the preset number of vessel types. Each output node corresponds to the score of a vessel type. The softmax function is applied to these scores to convert them into a probability distribution of each type of vessel, with the sum of the probability values being 1. Each probability value represents the likelihood of the input feature vector belonging to the corresponding vessel type.

[0097] According to the probability distribution, the vessel type with the largest probability value is selected as the identification result, and the vessel type information is determined and added with the corresponding category identifier, such as the identifier "Y" for fishing vessels, the identifier "T" for transport vessels, and the identifier "G" for sightseeing vessels.

[0098] Step S153: input the encoded feature vector into the trajectory prediction branch, combine time series information to construct a time series convolution network, capture the dependence relationship at different time scales through multiple dilated convolution layers, output the position coordinate prediction value at multiple future time steps, and combine the position coordinate prediction value with the historical coordinates to generate the position change trajectory.

[0099] In this embodiment, the encoded feature vector is input into the trajectory prediction branch of the illegal fishing vessel identification model. The trajectory prediction branch first combines the encoded feature vector with the corresponding timestamp information to construct a feature sequence containing time series information.

[0100] The feature sequence is input into the time series convolution network, which contains multiple dilated convolution layers, each with a different dilation rate. The dilation rate determines the receptive field size of the convolution kernel on the input feature sequence. A smaller dilation rate is used to capture feature dependencies within a short time scale, and a larger dilation rate is used to capture feature dependencies within a long time scale.

[0101] Through layer-by-layer processing of multiple dilated convolution layers, the time series convolution network can effectively capture the dependence relationship of the feature sequence at different time scales and extract the time series pattern of the vessel movement. The processed features are input into a fully connected layer, which outputs the position coordinate prediction value at multiple future time steps. The position coordinates at each time step include horizontal and vertical prediction values.

[0102] The position coordinate prediction values at multiple future time steps are connected in time sequence with the position coordinates corresponding to the historical timestamps (extracted from the cross-source fusion features or obtained from the original image) to form a complete position change trajectory. The position change trajectory can intuitively reflect the movement path and trend of the vessel.

[0103] Step S154: input the encoded feature vector into the behavior analysis branch, extract pattern features related to illegal fishing behavior from the features, and the pattern features include vessel stay time features, trajectory abnormal turning features, and specific area lingering features.

[0104] Step S1541: perform time dimension segmentation processing on the encoded feature vector, divide the continuous feature vector into multiple subsequences according to a fixed time interval, each subsequence corresponds to a time period, and obtain a sub-feature vector sequence corresponding to different time periods.

[0105] In this embodiment, before extracting the pattern features, the encoded feature vector is first subjected to time dimension segmentation processing. According to the time granularity requirement of vessel behavior analysis, a fixed time interval is set, for example, the continuous encoded feature vector is divided into a time period every N timestamp.

[0106] For each time period, the corresponding encoding feature vectors are combined together to form a sub-feature vector sequence. For example, the encoding feature vectors with time stamps from t1 to tN form the first sub-sequence, the encoding feature vectors with time stamps from tN+1 to t2N form the second sub-sequence, and so on. For the last part of less than N time stamps, if its time length exceeds half of the fixed time interval, it is separately regarded as a sub-sequence; otherwise, it is merged into the previous sub-sequence.

[0107] By time dimension segmentation, sub-feature vector sequences corresponding to different time periods are obtained, and each sub-feature vector sequence can reflect the feature change of the ship in the corresponding time period.

[0108] Step S1542: Calculate the ship position distribution density in each sub-feature vector sequence, calculate the density of the ship position coordinates by the kernel density estimation method, and select the dense area feature according to the density calculation result.

[0109] In this embodiment, for each sub-feature vector sequence, the position coordinate information of the ship is extracted from the sub-feature vector, and these position coordinates form a coordinate set. The kernel density estimation method is used to calculate the position distribution density of the coordinate set, the kernel function is selected as the Gaussian kernel function, and the bandwidth is adaptively determined according to the distribution of the coordinate set.

[0110] The process of kernel density estimation is that for each position coordinate point, the density contribution of the surrounding area is calculated by the Gaussian kernel function with the point as the center, the density contributions of all points are superimposed, and the position distribution density map of the whole area is obtained. According to the density map, a density threshold is set, and the area with a density value higher than the threshold is marked as a dense area.

[0111] The feature parameters of the dense area are extracted, such as the center coordinates, area, contour shape of the dense area, and the number of position coordinates in the area, etc. These parameters are combined together to form the dense area feature. The dense area feature can reflect the aggregation of the ship in the corresponding time period, and is of great significance for judging whether there is multi-ship cooperative fishing ban behavior or not.

[0112] Step S1543: Analyze the trajectory direction change in the sub-feature vector sequence, calculate the trajectory direction angle of adjacent time points, judge whether the direction changes significantly by the angle difference between the trajectory direction angles of adjacent time points, mark as a mutation point when the angle difference exceeds a preset angle threshold, and record the angle change value of the mutation point to generate the trajectory abnormal turning feature.

[0113] In this embodiment, for each sub-feature vector sequence, the position coordinates of the ship at different time points are extracted from the sub-feature vectors. These position coordinates are arranged in chronological order to form the ship's trajectory. To analyze the changes in trajectory direction, the trajectory direction angle between adjacent time points is first calculated. For time points t1 and t2 (t2 is later than t1), the corresponding position coordinates are (x1, y1) and (x2, y2), respectively. The vector (x2-x1, y2-y1) is calculated using these two coordinates. The angle between this calculated vector and the positive horizontal direction is the trajectory direction angle from time point t1 to t2, denoted as θ1. Similarly, the trajectory direction angle θ2 from time point t2 to t3 is calculated, and so on, to obtain the sequence of trajectory direction angles between adjacent time points.

[0114] Next, the angular difference between adjacent trajectory directional angles is calculated, i.e., Δθ = θ_after - θ_before (where θ_after is the directional angle corresponding to the later time point, and θ_before is the directional angle corresponding to the earlier time point). This angular difference is compared with a preset angular threshold, which is set based on the range of directional changes during normal navigation of vessels in the river, for example, determined based on the range of normal turning angles of legal vessels in historical monitoring data. When the absolute value of the angular difference Δθ exceeds the preset angular threshold, this location is marked as a point of abrupt change, indicating that the vessel has undergone a significant directional change at that time point.

[0115] For each abrupt change point, the corresponding angle change value Δθ is recorded, along with the time point of the abrupt change and the vessel's position coordinates at that time. This information is then integrated to generate an abnormal trajectory turning feature. This feature reflects whether the vessel exhibits any unusual or sudden turns during its journey, which may be related to fishing ban activities, such as rapidly approaching areas where fish congregate or evading law enforcement vessels.

[0116] Step S1544: Statistically count the frequency and duration of the appearance of vessels in the prohibited fishing area, and calculate the wandering coefficient in the area in combination with the position change trajectory. The wandering coefficient is positively correlated with the frequency and duration. Specifically, by comparing the position of the vessel with the boundary of the prohibited fishing area, the time points when the vessel enters and leaves the prohibited fishing area are determined, the number of times it appears is calculated as the frequency, and the time point difference is calculated as the duration.

[0117] In this embodiment, the boundary of the fishing ban area is first defined. This boundary is pre-stored in the form of geographic coordinates, for example, using a set of vertex coordinates of a polygon to define the fishing ban area. For each time point in the sub-feature vector sequence, the vessel's position coordinates are extracted and compared with the boundary coordinates of the fishing ban area to determine whether the vessel is within the fishing ban area.

[0118] The judgment method is: through a point-in-polygon judgment algorithm such as a ray method, it is determined whether the ship position coordinates are located inside the polygon corresponding to the fishing prohibited area. When the ship enters the interior from the exterior of the fishing prohibited area, the time point at this time is recorded as the entering time point; when the ship leaves the interior to the exterior of the fishing prohibited area, the time point at this time is recorded as the leaving time point.

[0119] According to the entering time point and the leaving time point, the occurrence frequency and the duration of the ship in the fishing prohibited area are counted. The occurrence frequency is the number of times the ship enters the fishing prohibited area, that is, the total number of entering time points. The duration is the difference between each entering time point and the corresponding leaving time point. If the ship is still in the fishing prohibited area at the end of the sub-feature vector sequence, the sequence end time point is taken as the leaving time point to calculate the duration. Then, all the durations are added to obtain the total duration.

[0120] The lingering coefficient in the area is calculated in combination with the position change trajectory. The calculation method is: first, a plurality of sampling points are selected on the position change trajectory, and the distance sum between these sampling points is calculated. The distance sum reflects the length of the driving path of the ship in the fishing prohibited area. Then, the diagonal length of the minimum circumscribed rectangle of the ship activity range in the fishing prohibited area is calculated as the reference distance. Finally, the lingering coefficient is equal to (driving path length / reference distance) x (occurrence frequency x weight 1 + duration x weight 2), wherein weight 1 and weight 2 are coefficients set according to actual conditions, used to adjust the influence degree of occurrence frequency and duration in the lingering coefficient, and weight 1 and weight 2 are both positive numbers to ensure that the lingering coefficient is positively correlated with the occurrence frequency and the duration. The larger the lingering coefficient is, the higher the degree of lingering of the ship in the fishing prohibited area is, and the more likely the fishing prohibited behavior exists.

[0121] Step S1545: The dense area feature, the trajectory abnormal turning feature, and the lingering coefficient are processed by feature aggregation to generate a pattern feature. Each dimension of the pattern feature corresponds to a feature performance related to the fishing prohibited behavior.

[0122] In this embodiment, the dense area feature, the trajectory abnormal turning feature, and the lingering coefficient are processed by feature aggregation. First, the three features are converted into vector forms with the same number of dimensions. The center coordinates, area, contour shape parameters, etc. contained in the dense area feature can be converted into a plurality of dimension values; the mutation point angle change value, time point, position coordinates, etc. in the trajectory abnormal turning feature are also converted into corresponding dimension values; the lingering coefficient as a numerical value can be integrated into the vector by extending it into a plurality of same numerical dimension or associating it with other features.

[0123] Then, the converted vector is normalized, and a z-score normalization method is used to convert the numerical value of each feature dimension to a distribution with a mean of 0 and a standard deviation of 1, so as to eliminate the dimensional difference between different features. The normalized vector is aggregated by feature splicing, that is, the three vectors are connected in order and form a pattern feature.

[0124] Each dimension of the pattern feature corresponds to a specific feature performance in the dense area feature, the trajectory abnormal turning feature or the wandering coefficient, for example, a certain dimension corresponds to the area size of the dense area, a certain dimension corresponds to the angle change value of the trajectory abnormal turning, and a certain dimension corresponds to the numerical value of the wandering coefficient, thereby comprehensively integrating various feature information related to the illegal fishing behavior.

[0125] Step S155: According to the matching result of the pattern feature and the preset illegal fishing behavior pattern library, the similarity of the pattern feature and each preset illegal fishing behavior pattern in the preset illegal fishing behavior pattern library is calculated, and the behavior label corresponding to the preset illegal fishing behavior pattern with the highest similarity is selected as the suspected illegal fishing behavior label. The ship type information, position change trajectory and suspected illegal fishing behavior label are combined to form a ship monitoring result.

[0126] In the embodiment, the preset illegal fishing behavior pattern library stores a plurality of preset illegal fishing behavior patterns, each of which is represented in the form of a feature vector. These feature vectors are obtained by analyzing and extracting features from historical illegal fishing cases, and cover common illegal fishing behavior features, such as long-time wandering in an illegal fishing area, dense gathering of multiple ships in an area, sudden change of driving direction to approach a fish school, etc.

[0127] The similarity of the pattern feature and each preset illegal fishing behavior pattern in the preset illegal fishing behavior pattern library is calculated using the cosine similarity calculation method. For the pattern feature vector V and a preset illegal fishing behavior pattern feature vector U, the similarity S is calculated as the dot product of the vector V and the vector U divided by the product of the norm of the vector V and the norm of the vector U, that is, S=(V・U) / (||V||×||U||). In this way, the similarity of the pattern feature and each preset pattern in the library is calculated.

[0128] From all the calculated similarities, the preset illegal fishing behavior pattern corresponding to the maximum value is selected. The behavior label of this preset illegal fishing behavior pattern is the suspected illegal fishing behavior label, which contains the type description of the illegal fishing behavior, such as "wandering in an illegal fishing area", "dense gathering of multiple ships", etc. The ship type information, position change trajectory and suspected illegal fishing behavior label obtained before are integrated together to form a ship monitoring result, which comprehensively reflects the attributes, motion trajectory and possible illegal fishing behavior of the monitored ship.

[0129] Step S1551: Extract a typical illegal fishing behavior pattern feature set from a preset illegal fishing behavior pattern library, which contains a plurality of illegal fishing behavior feature templates extracted from historical illegal fishing law enforcement records, covering illegal fishing behavior performances in different seasons and different time periods. The typical illegal fishing behavior pattern feature set contains behavior feature templates of ships in historical illegal fishing cases.

[0130] In this embodiment, the construction of the preset illegal fishing behavior pattern library is based on a large number of historical illegal fishing law enforcement records. When extracting the typical illegal fishing behavior pattern feature set, first, the historical illegal fishing law enforcement records are screened, and representative cases are selected, which cover illegal fishing behaviors in different seasons (such as fish spawning period, non-spawning period), different time periods (such as daytime, night, early morning, etc.).

[0131] For each historical case, the behavior features of the ship are extracted, including the appearance frequency, duration, trajectory direction change, and aggregation with other ships in the illegal fishing area, etc. These features are converted into feature vector form to form a behavior feature template. All behavior feature templates are integrated together to form a typical illegal fishing behavior pattern feature set. Each feature template in the typical illegal fishing behavior pattern feature set corresponds to a specific illegal fishing behavior performance, such as "wandering in the spawning area at night in spring" and "aggregating in the fish migration channel in the early morning".

[0132] Step S1552: Calculate the feature distance between the pattern feature and each illegal fishing behavior feature template in the typical illegal fishing behavior pattern feature set, and arrange all illegal fishing behavior feature templates in order from small to large according to the feature distance, and select the illegal fishing behavior feature template corresponding to the smallest feature distance as the matching feature template.

[0133] In this embodiment, the feature distance uses the Euclidean distance calculation method. For the pattern feature vector V and a certain illegal fishing behavior feature template vector T in the typical illegal fishing behavior pattern feature set, the calculation method of the feature distance D is the square root of the sum of the squares of the differences between the corresponding dimensions of the vector V and the vector T, that is, D=[(v1-t1)²+(v2-t2)²+…+(vn-tn)²]^1 / 2, where v1 to vn are the dimension values of vector V, and t1 to tn are the dimension values of vector T.

[0134] The feature distance between the pattern feature and each illegal fishing behavior feature template in the typical illegal fishing behavior pattern feature set is calculated respectively to obtain a set of distance values. These distance values are arranged in order from small to large, and the corresponding illegal fishing behavior feature templates are also sorted. The illegal fishing behavior feature template corresponding to the smallest feature distance is selected, which is determined as the matching feature template. The matching feature template has the smallest difference with the current pattern feature and best reflects the characteristics of the current ship behavior.

[0135] Step S1553: Calculate the feature coincidence degree of the pattern feature and the matching feature template. When the feature coincidence degree exceeds a preset coincidence degree threshold, generate an identification indicating the existence of suspected illegal fishing behavior, including the matched illegal fishing behavior type and the coincidence degree value; and when the feature coincidence degree does not exceed the preset coincidence degree threshold, generate an identification indicating the non-existence of suspected illegal fishing behavior, while recording the behavior type with the highest coincidence degree as a reference. The suspected illegal fishing behavior identification includes the behavior type and the feature coincidence degree information.

[0136] In this embodiment, the calculation method of the feature coincidence degree is as follows: count the number of dimensions whose value difference is within the preset allowed range between the pattern feature and the matching feature template, and the ratio of the number of dimensions to the total number of features is the feature coincidence degree. For example, if the total number of features is m, and the value difference of k dimensions is within the allowed range, then the feature coincidence degree is k / m.

[0137] The preset coincidence degree threshold is set according to the accuracy requirement of illegal fishing behavior judgment in historical data, for example, set to 0.6. When the calculated feature coincidence degree exceeds 0.6, an identification of the existence of suspected illegal fishing behavior is generated, which includes the matched illegal fishing behavior type (such as "illegal stay in illegal fishing area") and the specific coincidence degree value (such as 0.75).

[0138] When the feature coincidence degree does not exceed 0.6, an identification of the non-existence of suspected illegal fishing behavior is generated, but at the same time, the behavior type with the highest coincidence degree and its coincidence degree value in all illegal fishing behavior feature templates are recorded as a reference for subsequent monitoring and analysis. Whether there is suspected illegal fishing behavior or not, the generated suspected illegal fishing behavior identification includes the behavior type and the feature coincidence degree information, so as to fully reflect the matching situation.

[0139] Step S160: Generate illegal fishing warning information based on the ship monitoring result, which includes the real-time position of the suspected illegal fishing ship and the description of the behavior characteristics.

[0140] In this embodiment, the ship monitoring result includes the type, position change trajectory, suspected illegal fishing behavior identification and other key information of the ship. Based on these information, illegal fishing warning information is generated so that relevant law enforcement personnel can timely understand the situation and take appropriate measures. During the generation process, the monitoring result needs to be analyzed, key elements are extracted, and information is integrated to finally form complete and clear warning information.

[0141] Step S161: Analyze the suspected illegal fishing behavior identification in the ship monitoring result, extract the behavior type code and the feature coincidence degree value contained therein, query the preset behavior severity level table according to the behavior type code, and determine the type and confidence level of the suspected illegal fishing behavior in combination with the coincidence degree value.

[0142] In this embodiment, the suspected illegal fishing behavior identifier is stored in the form of structured data, containing behavior type code and feature coincidence value information. First, the identifier is parsed to extract the behavior type code (such as "001" representing "loitering in a closed fishing area" and "002" representing "multiple ship aggregation") and the feature coincidence value (such as 0.82).

[0143] In the preset behavior severity level table, each behavior type code corresponds to an illegal fishing behavior type and the severity level division of the type of behavior, for example, "001" corresponding to "loitering in a closed fishing area" is divided into three levels of slight, general and serious. According to the extracted behavior type code, the corresponding behavior type is determined by querying the level table.

[0144] The confidence level is determined in combination with the feature coincidence value. The higher the coincidence value, the higher the credibility of the suspected illegal fishing behavior. The confidence level is divided into three levels of high, medium and low, for example, the coincidence value is greater than 0.7 for high confidence, 0.5 to 0.7 for medium confidence, and less than 0.5 for low confidence. Through the above process, the type and confidence level of the suspected illegal fishing behavior are determined.

[0145] Step S162: Extracting the position change trajectory in the ship monitoring result, obtaining the ship motion curve through the trajectory fitting algorithm, taking the position coordinates at the end of the ship motion curve as the current position coordinates of the suspected illegal fishing ship, and calculating the tangent direction at the end of the ship motion curve as the moving direction.

[0146] In this embodiment, the position change trajectory is extracted from the ship monitoring result, which is composed of a series of position coordinates arranged in time sequence. Trajectory fitting algorithm is used to process these coordinates, polynomial fitting method is selected, the degree of polynomial is determined according to the bending degree of trajectory and the number of sampling points, and the polynomial coefficients are solved by least square method to obtain a curve that can smoothly represent the ship motion trajectory, i.e. the ship motion curve.

[0147] The position coordinates at the end of the motion curve, i.e. the latest position coordinates in time, are taken as the current position coordinates of the suspected illegal fishing ship. The tangent direction at the end of the curve is calculated, the slope of the tangent is obtained by taking the derivative of the curve at the end point, and the moving direction is determined according to the slope, for example, the slope is positive, indicating moving in a certain direction, and the slope is negative, indicating moving in the opposite direction. The moving direction is represented in the form of azimuth angle (such as 30 degrees representing north 30 degrees east).

[0148] Step S163: Spatially compare the current position coordinates with the forbidden fishing area boundary coordinates, calculate the shortest distance and azimuth angle from the current position coordinates to the forbidden fishing area boundary coordinates, analyze the trend of the suspected forbidden fishing vessel moving towards the forbidden fishing area, and generate the position correlation feature.

[0149] In this embodiment, the coordinate information of the forbidden fishing area boundary is first obtained, which is stored in the form of polygon vertex coordinates. The current position coordinates of the suspected forbidden fishing vessel are converted into the same coordinate system as the forbidden fishing area boundary coordinates for spatial comparison.

[0150] The shortest distance from the current position coordinates to the forbidden fishing area boundary coordinates is calculated. The specific method is to traverse each edge of the forbidden fishing area boundary, calculate the perpendicular distance from the current position coordinates to each edge, and select the minimum value as the shortest distance. At the same time, the azimuth angle of the current position coordinates relative to the forbidden fishing area boundary is calculated, i.e. the angle between the direction from the current position to the center of the forbidden fishing area and the north direction.

[0151] The trend of the suspected forbidden fishing vessel moving towards the forbidden fishing area is analyzed by comparing the distance changes of the current position and the positions at previous time points to the forbidden fishing area boundary. If the distance gradually decreases, it indicates that the vessel is moving towards the forbidden fishing area; if the distance gradually increases, it indicates that the vessel is moving away from the forbidden fishing area. The shortest distance, azimuth angle and movement trend are integrated together to generate the position correlation feature, which reflects the spatial relationship and dynamic changes of the vessel's current position and the forbidden fishing area.

[0152] Step S1631: Obtain the geographical boundary information of the forbidden fishing area, which includes the polygon boundary vertex coordinates of the forbidden fishing area. The geographical boundary information is represented by a set of polygon vertex coordinates.

[0153] In this embodiment, the geographical boundary information of the forbidden fishing area is determined through preliminary geographic mapping and stored in the database of the system. The geographical boundary information is specifically represented by a set of polygon boundary vertex coordinates, such as a set of latitude and longitude coordinates (lat1, lon1), (lat2, lon2), …, (latn, lonn) containing multiple vertices. These vertices are connected in order to form a closed polygon, and the area enclosed by the polygon is the forbidden fishing area. When spatial comparison is needed, the set of polygon vertex coordinates is read from the database as the basis for subsequent calculations.

[0154] Step S1632: Convert the current position coordinates of the suspected forbidden fishing vessel into coordinate points in the same coordinate system as the geographical boundary information, and calculate the straight-line distances from the coordinate points to each vertex in the set of polygon vertex coordinates. Arrange the straight-line distances in order of the vertices to generate a distance set.

[0155] In this embodiment, the current position coordinates of the suspected fishing-ban vessel can be in image pixel coordinates or other local coordinate systems, which need to be converted into the same coordinate system as the geographical boundary information, usually the latitude and longitude coordinate system. The conversion process is realized by a coordinate conversion algorithm, which establishes a mapping relationship between the pixel coordinates and the latitude and longitude coordinates by using preset conversion parameters (such as the geographical position of the image acquisition device, lens parameters, etc.), so as to convert the pixel coordinates of the current position into a latitude and longitude coordinate point (lat0, lon0).

[0156] The straight-line distances of the coordinate point to each vertex in the vertex coordinate set of the polygon are calculated, and according to the distance formula between two points, for a vertex (lati, loni), the distance di=[(lat0-lati)²+(lon0-loni)²]^1 / 2, where i is from 1 to n. All the calculated distances d1, d2, …, dn are arranged in order according to the order of the vertices in the set, forming a distance set.

[0157] Step S1633: analyzing the spatial position relationship between the coordinate point and the polygon boundary, and according to the distance set, the spatial position relationship and the moving direction of the suspected fishing-ban vessel, calculating the mean and variance of the distance set as distance features, converting the spatial position relationship into a numerical code as a position state feature, and calculating the included angle between the moving direction and the direction pointing to the center of the fishing-ban area as a direction feature, constructing a position association feature containing the distance features, the position state feature and the direction feature, which is used to describe the dynamic position relationship between the suspected fishing-ban vessel and the fishing-ban area.

[0158] In this embodiment, when analyzing the spatial position relationship between the coordinate point (lat0, lon0) and the polygon boundary, the ray method is used to determine whether the coordinate point is located inside the fishing-ban area. Specifically, a ray is emitted from the coordinate point (lat0, lon0) in any direction, and the number of intersection points of the ray with the polygon boundary is counted. If the number of intersection points is odd, it is determined that the coordinate point is located inside the fishing-ban area; if the number of intersection points is even, it is determined that the coordinate point is located outside the fishing-ban area; if the ray coincides with the polygon boundary, it is determined that the coordinate point is located on the boundary of the fishing-ban area.

[0159] After obtaining the distance set, the mean and variance of the set are calculated. The mean is calculated by adding all the straight-line distances in the distance set and dividing by the number of elements in the distance set, which reflects the average distance of the coordinate point to each vertex of the polygon. The variance is calculated by squaring the difference between each straight-line distance and the mean and then adding them up, and dividing by the number of elements in the distance set, which reflects the degree of deviation of each straight-line distance from the mean. These two parameters together constitute the distance feature.

[0160] For numerical coding of spatial position relationship, three numerical values are set to correspond to different position states: when the coordinate point is located inside the closed fishing area, the code is 0; when the coordinate point is located outside the closed fishing area, the code is 1; when the coordinate point is located on the boundary of the closed fishing area, the code is 2, and the numerical code is the position state feature.

[0161] When calculating the direction feature, first determine the center coordinates (lat_center, lon_center) of the closed fishing area, and calculate the direction vector pointing to the center of the closed fishing area through the coordinate point (lat0, lon0) and the center coordinates (lat_center, lon_center). The direction of the direction vector is from the coordinate point to the center coordinates. Then, combined with the moving direction vector of the suspected closed fishing vessel, the included angle between the two vectors is calculated. In the calculation process, the magnitudes of the two vectors and their dot products are first calculated, and then the cosine value of the included angle is obtained by the ratio of the dot product to the product of the magnitudes, and finally the size of the included angle is determined according to the cosine value, which is the direction feature.

[0162] The distance feature, position state feature and direction feature are combined in order to form the position correlation feature. The position correlation feature can comprehensively describe the dynamic position relationship between the suspected closed fishing vessel and the closed fishing area from the aspects of distance, position state and moving direction.

[0163] Step S164: Information integration processing is performed on the type of suspected closed fishing behavior, confidence degree, current position coordinates, moving direction and position correlation feature to generate a structured early warning element set.

[0164] In this embodiment, after obtaining the type of suspected closed fishing behavior, confidence degree, current position coordinates, moving direction and position correlation feature, these information are integrated and processed. First, set a corresponding field for each information item, for example, the "behavior type" field is used to store the type of suspected closed fishing behavior, the "confidence degree" field is used to store the corresponding confidence degree value, the "current position" field is used to store the current position coordinates, the "moving direction" field is used to store the moving direction information, and the "position correlation" field is used to store the position correlation feature.

[0165] Then, fill in each information item according to the corresponding field to ensure that the information in each field is accurate. For numerical information, such as confidence degree, numerical value in position coordinates, etc., keep its original numerical form; for text description type information, such as behavior type, use standardized text expression; for composite information, such as position correlation feature, combine the distance feature, position state feature and direction feature contained therein according to the preset format and store them in the corresponding field.

[0166] After the above processing, a structured early warning element set is formed, and the information in the early warning element set is organized in order and unified in format, facilitating subsequent generation of the fishing ban early warning information based on the information.

[0167] Step S165: Generate the fishing ban early warning information containing the real-time position of the suspected fishing ban vessel and the description of the behavior characteristics based on the content of the early warning element set.

[0168] In this embodiment, the fishing ban early warning information is generated based on the early warning element set. First, the real-time position information, i.e., the current position coordinates, of the suspected fishing ban vessel is extracted from the early warning element set, and is converted into an actual geographic coordinate representation, such as a latitude and longitude representation, so that the relevant personnel can accurately know the position of the vessel.

[0169] Then, the information such as the behavior type, the confidence degree, the moving direction, and the position correlation characteristics is extracted and described in words. For example, when describing the behavior type, it is explicitly pointed out that the suspected fishing ban behavior is of which type; when describing the confidence degree, it is explained that the behavior is determined to be the suspected fishing ban behavior with which confidence degree; when describing the moving direction, it is explained that the current moving trend of the vessel; and in combination with the position correlation characteristics, the position relationship between the vessel and the fishing ban area and the influence of the possible moving trend of the vessel on the fishing ban area are described.

[0170] The real-time position information and the behavior characteristic description are organized according to a preset template. The structure of the template includes a title, a real-time position, and a behavior characteristic description. The title part explicitly indicates that this is a fishing ban early warning information; the real-time position part clearly lists the latitude and longitude coordinates of the vessel; and the behavior characteristic description part elaborates the behavior type, the confidence degree, the moving direction, and the position relationship with the fishing ban area of the vessel.

[0171] After the above organization is completed, the fishing ban early warning information is generated, which can enable the relevant law enforcement personnel to quickly understand the specific situation of the suspected fishing ban vessel.

[0172] Figure 2 Fig. 1 shows a schematic diagram of exemplary hardware and software components of a river fishing ban vessel monitoring system 100 based on visible light and infrared fusion that can implement the idea of the present application. For example, the processor 120 can be used in the river fishing ban vessel monitoring system 100 based on visible light and infrared fusion, and used to perform the functions in the present application.

[0173] For example, the visible and infrared fusion based river fishing prohibition vessel monitoring system 100 can include a network port 110 connected to a network, one or more processors 120 for executing program instructions, a communication bus 130, and different forms of storage media 140, such as a disk, a ROM, or a RAM, or any combination thereof. Exemplarily, the visible and infrared fusion based river fishing prohibition vessel monitoring system 100 can also include program instructions stored in a ROM, a RAM, or other types of non-transitory storage media, or any combination thereof. The method of the present application can be implemented according to these program instructions. The visible and infrared fusion based river fishing prohibition vessel monitoring system 100 further includes an I / O interface 150 between the computer and other input and output devices.

[0174] In addition, the embodiment of the present application further provides a readable storage medium, wherein computer executable instructions are preset in the readable storage medium, and when a processor executes the computer executable instructions, the visible and infrared fusion based river fishing prohibition vessel monitoring method is realized.

[0175] It should be noted that, in order to simplify the description of the present application and to help the understanding of one or more embodiments of the present application, in the foregoing description of the embodiments of the present application, various features are sometimes combined into one embodiment, drawing or description thereof.

Claims

1. A method for monitoring fishing vessels in river channels under fishing ban conditions based on visible light and infrared fusion, characterized in that, The method includes: Acquire visible light image sequences and infrared image sequences of the river monitoring area. The visible light image sequences contain texture details of the river surface and surrounding environment, and the infrared image sequences contain thermal radiation distribution characteristics within the river area. The visible light image sequence is processed to extract ship visual features to obtain a visible light ship feature set, and the infrared image sequence is processed to extract thermal features to obtain an infrared ship feature set. Perform cross-source feature fusion processing on the visible light ship feature set and the infrared ship feature set to generate a cross-source fused feature set; The pre-trained fishing ban vessel identification model is invoked to perform vessel attribute and behavior analysis on the cross-source fusion feature set, generating vessel monitoring results that include vessel type, location change trajectory, and suspected fishing ban behavior identifiers; Based on the vessel monitoring results, a fishing ban early warning information is generated, which includes the real-time location and behavioral characteristics description of suspected vessels subject to the fishing ban.

2. The method for monitoring fishing vessels in river channels based on visible light and infrared fusion according to claim 1, characterized in that, The process of extracting ship visual features from the visible light image sequence to obtain a set of visible light ship features includes: Ship region localization processing is performed on consecutive frames in the visible light image sequence. The pixel region of each frame image is traversed by sliding window. After filtering out candidate regions that may contain ships by combining prior knowledge of ship outlines, the boundary information of the candidate regions is enhanced by edge detection algorithm to determine the range of the pixel region where the ship is located in each frame image. The range of the pixel region is represented by a set of bounding box coordinates. The visible light texture features within the pixel region are extracted, and the gray-level co-occurrence matrix algorithm is used to calculate the pixel gray-level correlation features under different directions and distances. Combined with the image details after Gaussian filtering, the pattern distribution features of the ship surface are generated. The edge points of the region are extracted by the Canny operator and the contour curve is fitted to obtain the edge contour features. The color gradient features are obtained by calculating the color difference between adjacent pixels based on the HSV color space. The visible light texture features include pattern distribution features, edge contour features and color gradient features. The variation trend of the ship pixel region range in the continuous frames is analyzed. The center coordinate offset and the rate of change of the ship bounding box in adjacent frames are calculated by the inter-frame difference method and integrated into the displacement parameters. At the same time, the motion trajectory of the ship's key pixels is tracked by the optical flow method to supplement the subtle motion information not covered in the displacement parameters and generate the ship motion dynamic features. The visible light texture features and the ship motion dynamic features are processed by feature dimension mapping. After performing dimensionality reduction or dimensionality increase operations through principal component analysis, the feature values ​​are mapped to the same numerical range through feature scaling, thereby converting the visible light texture features and the ship motion dynamic features into feature vectors of the same dimension. The feature vectors of the same dimension are concatenated according to the time sequence, and the feature vectors corresponding to each frame are arranged in the order of timestamps. The cosine similarity of adjacent timestamp feature vectors is calculated to verify the temporal continuity. The feature vectors that are temporally discontinuous are processed to smooth the transition, and a set of visible light ship features containing temporal correlation is generated.

3. The method for monitoring fishing vessels in river channels based on visible light and infrared fusion according to claim 1, characterized in that, The process of extracting thermal features from the infrared image sequence to obtain an infrared ship feature set includes: Each frame of the infrared image sequence is segmented into thermal radiation regions. An adaptive threshold segmentation algorithm is used to determine the segmentation threshold based on the thermal radiation intensity distribution. The image is divided into multiple connected regions. By calculating the morphological parameters of each connected region, region units with continuous thermal radiation characteristics and conforming to the size characteristics of ships are selected. The region units correspond to the regions where ships may exist. The thermal radiation intensity distribution features of the region unit are extracted. By scanning each pixel in the region unit, the location of the maximum thermal radiation intensity is recorded as the thermal radiation peak position. The thermal radiation intensity decay rate from the thermal radiation peak position to the edge of the region unit is calculated to obtain the thermal radiation gradient change features. In addition, the area expansion of the region unit in consecutive frames is compared, and the thermal radiation range expansion features are generated by combining the thermal radiation intensity changes of the edge pixels. The thermal radiation intensity distribution features include the thermal radiation peak position, thermal radiation gradient change features, and thermal radiation range expansion features. The positional changes of the region unit in continuous infrared image frames are tracked. The region unit of the current frame is matched with the previous frame by template matching to determine the movement path of the center coordinate. The coordinate change per unit time is calculated to obtain the movement rate feature. The fluctuation of the movement rate is analyzed, and the frequency and amplitude features of the movement rate change are extracted and added to the movement rate feature. The thermal radiation intensity distribution feature and the movement rate feature are correlated and mapped, the correlation coefficient is calculated, a feature mapping matrix is ​​constructed based on the correlation coefficient, and the thermal radiation intensity distribution feature and the movement rate feature are projected into the correlation space of the feature mapping matrix respectively to generate a thermal radiation motion correlation feature vector. The thermal radiation motion-related feature vectors are arranged in a time sequence, organized according to the order of timestamps, and the feature distance between adjacent feature vectors is calculated to evaluate temporal consistency. Feature vectors whose feature distance exceeds a preset distance range are interpolated and adjusted to generate an infrared ship feature set containing temporal change patterns. Each feature vector in the infrared ship feature set contains thermal radiation and motion-related information corresponding to the timestamp.

4. The method for monitoring fishing vessels in river channels based on visible light and infrared fusion according to claim 1, characterized in that, The step of performing cross-source feature fusion processing on the visible light vessel feature set and the infrared vessel feature set to generate a cross-source fused feature set includes: Traverse the timestamp information in the visible light ship feature set and the infrared ship feature set, establish a timestamp correspondence table, and extract the visible light feature vector and infrared feature vector under the same timestamp according to the timestamp correspondence table. The cross-source attention mechanism module is invoked to calculate the association weights of the visible light feature vectors and infrared feature vectors under the same timestamp, generating a feature association weight matrix. The visible light feature vector and the infrared feature vector are weighted and fused according to the feature association weight matrix to generate a preliminary fused feature vector. The preliminary fused feature vector is subjected to multi-scale feature mapping to obtain local detail features at different scales. At the same time, global statistical information of the features is extracted as global distribution features through global average pooling operation. The local detail features and the global distribution features are channel concatenated. The feature channels of the local detail features and the global distribution features are concatenated in sequence. The numerical distribution between the channels is adjusted by batch normalization to generate a cross-source fusion feature set containing multi-scale information. Each feature vector in the cross-source fusion feature set contains multi-scale fusion feature information corresponding to the timestamp.

5. The method for monitoring fishing vessels in river channels based on visible light and infrared fusion according to claim 1, characterized in that, The pre-trained fishing ban vessel identification model is invoked to perform vessel attribute and behavior analysis on the cross-source fusion feature set, generating vessel monitoring results including vessel type, location change trajectory, and suspected fishing ban behavior identifiers, including: The cross-source fusion feature set is input into the feature encoding layer of the fishing ban vessel identification model, and the feature vector is generated by layer-by-layer abstraction through a multi-layer convolutional neural network. The encoded feature vector is input into the vessel type identification branch, and the probability distribution of various types of vessels is output. The vessel type information is determined based on the probability distribution, and the vessel type information includes category identifiers for fishing boats, transport boats, and sightseeing boats. The encoded feature vector is input into the trajectory prediction branch, and a temporal convolutional network is constructed by combining it with time series information. Multiple dilated convolutional layers are used to capture the dependencies at different time scales, and the predicted position coordinates for multiple future time steps are output. The predicted position coordinates are then combined with historical coordinates to generate a position change trajectory. The encoded feature vector is input into the behavior analysis branch to extract pattern features related to the fishing ban behavior. The pattern features include features of vessel dwell time, abnormal trajectory turning, and loitering in specific areas. Based on the matching results between the pattern features and the preset fishing ban behavior pattern library, the similarity between the pattern features and each preset fishing ban behavior pattern in the preset fishing ban behavior pattern library is calculated. The behavior tag corresponding to the preset fishing ban behavior pattern with the highest similarity is selected as the suspected fishing ban behavior identifier. The vessel type information, position change trajectory and suspected fishing ban behavior identifier are combined into the vessel monitoring result.

6. The method for monitoring fishing vessels in river channels based on visible light and infrared fusion according to claim 5, characterized in that, The step of inputting the encoded feature vector into the behavior analysis branch and extracting pattern features related to the fishing ban behavior includes: The encoded feature vector is segmented in the time dimension, and the continuous feature vector is divided into multiple subsequences according to a fixed time interval. Each subsequence corresponds to a time period, resulting in a sub-feature vector sequence corresponding to different time periods. Calculate the ship position distribution density in each sub-feature vector sequence, calculate the density of ship position coordinates using the kernel density estimation method, and filter out dense region features based on the density calculation results; Analyze the trajectory direction changes in the sub-feature vector sequence, calculate the trajectory direction angles at adjacent time points, determine whether the direction has changed significantly by the angle difference between the trajectory direction angles at adjacent time points, mark the abrupt change point when the angle difference exceeds the preset angle threshold, and record the angle change value of the abrupt change point to generate trajectory abnormal turning features. The frequency and duration of vessel appearances within the prohibited fishing area are statistically analyzed, and the wandering coefficient within the area is calculated by combining the position change trajectory. The wandering coefficient is positively correlated with the frequency and duration of appearance. Specifically, by comparing the vessel's position with the boundary of the prohibited fishing area, the time points when the vessel enters and leaves the prohibited fishing area are determined, the number of appearances is calculated as the frequency of appearance, and the time point difference is calculated as the duration. The dense area features, trajectory anomaly turning features, and wandering coefficient are subjected to feature aggregation processing to generate pattern features. Each dimension of the pattern features corresponds to a feature manifestation related to the fishing ban behavior.

7. The method for monitoring fishing vessels in river channels based on visible light and infrared fusion according to claim 5, characterized in that, The step of generating a suspected fishing ban behavior identifier based on the matching results of the pattern features and a preset fishing ban behavior pattern library includes: A set of typical fishing ban behavior patterns is extracted from a pre-set fishing ban behavior pattern library. This pre-set fishing ban behavior pattern library contains various fishing ban behavior pattern templates extracted from historical fishing ban enforcement records, covering fishing ban behavior performance in different seasons and at different times. The set of typical fishing ban behavior pattern features includes behavioral feature templates of vessels in historical fishing ban cases. Calculate the feature distance between the pattern feature and each fishing ban behavior feature template in the typical fishing ban behavior pattern feature set, and arrange all fishing ban behavior feature templates in ascending order of feature distance, and select the fishing ban behavior feature template corresponding to the smallest feature distance as the matching feature template; The feature overlap between the pattern feature and the matching feature template is calculated. When the feature overlap exceeds a preset overlap threshold, an identifier indicating the existence of suspected fishing ban behavior is generated, which includes the matched fishing ban behavior type and the overlap value. When the feature overlap does not exceed the preset overlap threshold, an identifier indicating the absence of suspected fishing ban behavior is generated, and the behavior type with the highest overlap is recorded as a reference. The suspected fishing ban behavior identifier includes behavior type and feature overlap information.

8. The method for monitoring fishing vessels in river channels based on visible light and infrared fusion according to claim 1, characterized in that, The generation of fishing ban early warning information based on the vessel monitoring results includes: The suspected fishing ban behavior identifiers in the vessel monitoring results are analyzed, and the behavior type codes and feature overlap values ​​contained therein are extracted. The behavior type codes are queried from the preset behavior severity level table, and the type and confidence level of the suspected fishing ban behavior are determined by combining the overlap values. The position change trajectory in the vessel monitoring results is extracted, and the vessel motion curve is obtained through a trajectory fitting algorithm. The position coordinates at the end of the vessel motion curve are taken as the current position coordinates of the suspected fishing ban vessel, and the tangent direction at the end of the vessel motion curve is calculated as the direction of movement. The current location coordinates are spatially compared with the boundary coordinates of the fishing ban area. The shortest distance and azimuth from the current location coordinates to the boundary coordinates of the fishing ban area are calculated. The trend of suspected fishing ban vessels moving towards the fishing ban area is analyzed, and location association features are generated. The types, confidence levels, current location coordinates, direction of movement, and location-related features of the suspected fishing violations are integrated and processed to generate a structured set of early warning elements. Based on the content of the aforementioned set of early warning elements, a fishing ban early warning message is generated, which includes the real-time location and behavioral characteristics of suspected fishing vessels.

9. The method for monitoring fishing vessels in river channels based on visible light and infrared fusion according to claim 8, characterized in that, The process involves spatially comparing the current location coordinates with the boundary coordinates of the fishing ban area, calculating the shortest distance and azimuth from the current location coordinates to the boundary coordinates of the fishing ban area, analyzing the trend of suspected fishing ban vessels moving towards the fishing ban area, and generating location association features, including: Obtain the geographic boundary information of the fishing ban area, which includes the coordinates of the vertices of the polygon boundary of the fishing ban area, and the geographic boundary information is represented by a set of polygon vertex coordinates. The current location coordinates of the suspected fishing ban vessel are converted into coordinate points in the same coordinate system as the geographical boundary information, and the straight-line distances from the coordinate points to each vertex in the set of polygon vertex coordinates are calculated. The straight-line distances are then arranged in vertex order to generate a distance set. The spatial relationship between the coordinate points and the polygon boundary is analyzed. Based on the distance set, the spatial relationship, and the movement direction of the suspected fishing ban vessel, the mean and variance of the distance set are calculated as distance features. The spatial relationship is converted into numerical codes as positional status features. The angle between the movement direction and the direction pointing to the center of the fishing ban area is used as a direction feature. A positional association feature including distance features, positional status features, and direction features is constructed. The positional association feature is used to describe the dynamic positional relationship between the suspected fishing ban vessel and the fishing ban area.

10. A river channel fishing vessel monitoring system based on visible light and infrared fusion, characterized in that, The device includes a processor and a memory, the memory and the processor being connected. The memory is used to store programs, instructions or code, and the processor is used to execute the programs, instructions or code in the memory to implement the river channel fishing ban vessel monitoring method based on visible light and infrared fusion as described in any one of claims 1-9.

Images