Bimodal-based breeding operation fishing boat target detection device and method in long-distance complex scene

Abstract

The invention discloses a bimodal-based breeding operation fishing boat target detection device and method in a long-distance complex scene, belongs to the field of image target detection, and is used for solving the problems that in the long-distance complex scene, a fishing boat in long-distance fishing operation is not obvious in feature and is easily confused with seeing a sea house target and a reef target. The first backbone network is used for inputting a visible light modal image and comprises a first feature extraction layer, and the output of the first feature extraction layer comprises visible light features; the second backbone network is used for inputting an optical flow modal image, the second backbone network comprises a first feature extraction layer, and the output of the first feature extraction layer comprises optical flow features; and the fusion module is arranged between the first feature extraction layer of the first backbone network and the first feature extraction layer of the second backbone network, inputs visible light features and optical flow features, and outputs fusion features including channel attention features and space attention features.

Description

TECHNICAL FIELD

[0001] The application belongs to the field of image target detection, and relates to a dual-mode-based target detection device and method for fish farming work fishing boats in a long-distance complex scene. BACKGROUND

[0002] Fishing boats are important tools for fish farming work, and are of great significance to the development of the fishing industry and the life of fishermen. Fishing boat management is an important means to protect the safety of fishermen in their daily work and to improve the efficiency of fishing work. Traditional fishing boat management requires manual observation and manual recording, which is prone to errors or intentional concealment, resulting in loopholes in illegal fishing supervision. Moreover, it is impossible to achieve real-time supervision all day long. Under this background, intelligent detection technology based on YOLO series algorithms has become the core of the intelligent transformation of the fishing industry. However, the camera is far away from the fishing boat in the real fishing boat operation environment, and the imaging quality is poor, which reduces the recognition accuracy. The complexity of the coastline environment (such as sea-view houses, reefs, and densely parked scenes) also poses a serious challenge to detection technology. Therefore, there is an urgent need to develop a fishing boat detection method in a complex environment.

[0003] The YOLO algorithm series is a high-efficiency architecture paradigm in the field of target detection, and its core feature is to integrate target positioning and classification tasks into a single network framework, which can complete the detection task through a single forward propagation, and has always been committed to achieving a balance between detection accuracy and real-time performance. Due to its advantages in real-time processing, this series of algorithms has been more widely used in actual object detection tasks, especially in scenarios that require real-time performance, such as real-time detection of fishing boats. Huang et al. improved YOLOv4 to detect ships near the port berth. By using the k-means algorithm to strengthen the clustering of input data, the detection accuracy was improved, but the detection of small targets of fishing boats at a distance was not very accurate. Han et al. proposed a ship detection algorithm for unmanned aerial vehicle aerial photography, which incorporated SGE attention mechanism and MCA attention mechanism into YOLOv8n to enhance feature extraction capability, and improved the fusion method to strengthen information understanding, thereby improving detection performance, but the target detection capability in complex backgrounds needs to be improved. In another study, Zhang et al. proposed a detection algorithm for maritime personnel search and rescue. They improved YOLOv7 by introducing a SA attention mechanism for efficient target search area positioning and adding a small object detection head to enhance the detection capability of small targets. Although the YOLO series has been continuously improved, the detection performance is commendable, but there is still room for improvement in detection accuracy, especially in long-distance complex scenes, where the features of fishing boats in distant fishing operations are not obvious, and are easily confused with sea-view house targets and reef targets. The above models have not solved this problem.

[0004] In summary, the YOLO target detection technology still faces the following challenges in the detection task of fishing vessels in fishing operations: (1) The camera is placed far away from the shore, and the focal length of the camera is limited, resulting in poor image quality. The target features of the fishing vessels in operation are confused with the sea-view room; (2) The fishing vessel target in the real scene is small, and the detection layer and detection head of the original YOLO model cannot better detect small targets; (3) The sea surface environment is complex, and the gradient fluctuation is large due to the change of sea wave and water surface light, and the original YOLO model cannot better locate the small fishing vessel target; (4) The fishing vessels in the fishing operation may be blocked by each other. SUMMARY

[0005] In order to solve the problem that the features of fishing vessels in the fishing operation at a long distance in a complex scene are not obvious and are easily confused with the sea-view room target and the reef target, in a first aspect, an image target detection device according to some embodiments of the application comprises a first backbone network configured to input a visible light modality image, the first backbone network comprising a first feature extraction layer, an output of the first feature extraction layer comprising visible light features; a second backbone network configured to input an optical flow modality image, the second backbone network comprising a first feature extraction layer, an output of the first feature extraction layer comprising optical flow features; a fusion module disposed between the first feature extraction layer of the first backbone network and the first feature extraction layer of the second backbone network, configured to input the visible light features and the optical flow features, and output fusion features comprising channel attention features and spatial attention features; a neck network, an input of the neck network comprising the fusion features; a detection head, an input of the detection head comprising an output of the neck network, an output of the detection head comprising image targets; The fusion module comprises a concatenation layer configured to concatenate the visible light features and the optical flow features to obtain concatenated features; a channel attention mechanism layer configured to obtain channel prior features of the concatenated features; a spatial attention mechanism layer configured to obtain spatial attention features of the channel prior features; a fusion layer configured to fuse the channel prior features and the spatial attention features to obtain the fusion features.

[0006] The image target detection device according to some embodiments of the application, the channel attention mechanism layer comprises a max-pooling layer configured to perform a max-pooling operation on the concatenated features to output a max-pooling feature vector; an average-pooling layer configured to perform an average-pooling operation on the concatenated features to output an average-pooling feature vector; The multi-layer perceptron takes the maximum-pooled feature vector as input and outputs a first output vector; and the multi-layer perceptron takes the average-pooled feature vector as input and outputs a second output vector; The first addition fusion layer performs an addition operation on the first output vector and the second output vector to generate a fusion feature vector; The activation function takes the fusion feature vector as input and generates a channel attention mechanism map; The second multiplication fusion layer multiplies the spliced feature and the channel attention mechanism map to obtain a channel prior feature.

[0007] According to some embodiments of the present application, the spatial attention mechanism layer includes a depth separable convolution and a second addition fusion layer, wherein the depth separable convolution includes The first depth convolution takes the channel prior feature as input, and has at least two outputs; The second depth convolution includes at least two, and the input of each second depth convolution is connected to the output of one first depth convolution; The second addition fusion layer performs an addition operation on the features output by each second depth convolution to obtain a spatial attention feature.

[0008] According to some embodiments of the present application, the image target detection device, the neck network includes a first feature fusion layer and a second feature fusion layer; wherein the neck network further includes an efficient multi-scale attention mechanism module, the efficient multi-scale attention mechanism module is arranged between the first feature fusion layer and the detection head of the neck network and / or is arranged between the first feature fusion layer and the second feature fusion layer of the neck network, and the fusion feature output by the fusion module is input into the first feature fusion layer.

[0009] According to some embodiments of the present application, the image target detection device, wherein the fusion module includes four modules arranged at different layers; The first feature extraction layer of the first layer of the first backbone network includes four modules arranged at different layers; The first feature extraction layer of the first layer of the second backbone network includes four modules arranged at different layers; The first feature extraction layer of the first layer of the first backbone network includes a first output and a second output; the visible light feature of the first output is input into the first feature extraction layer of the second layer of the first backbone network; and the visible light feature of the second output is input into the first input of the splicing layer of the fusion module of the first layer; The first feature extraction layer of the first layer of the second backbone network includes a first output and a second output; the optical flow feature of the first output is input into the first feature extraction layer of the second layer of the second backbone network; and the optical flow feature of the second output is input into the second input of the splicing layer of the fusion module of the first layer; The first feature extraction layer of the second layer of the first backbone network comprises a first output and a second output; the visible light feature of the first output is input into the first feature extraction layer of the third layer of the first backbone network; and the visible light feature of the second output is input into the first input of the concatenation layer of the fusion module of the second layer. The first feature extraction layer of the second layer of the second backbone network comprises a first output and a second output; the optical flow feature of the first output is input into the first feature extraction layer of the third layer of the second backbone network; and the optical flow feature of the second output is input into the second input of the concatenation layer of the fusion module of the second layer. The first feature extraction layer of the third layer of the first backbone network comprises a first output and a second output; the visible light feature of the first output is input into the first feature extraction layer of the fourth layer of the first backbone network; and the visible light feature of the second output is input into the first input of the concatenation layer of the fusion module of the third layer. The first feature extraction layer of the third layer of the second backbone network comprises a first output and a second output; the optical flow feature of the first output is input into the first feature extraction layer of the fourth layer of the second backbone network; and the optical flow feature of the second output is input into the second input of the concatenation layer of the fusion module of the third layer. The first feature extraction layer of the fourth layer of the first backbone network comprises a second output; and the visible light feature of the second output is input into the first input of the concatenation layer of the fusion module of the fourth layer. The first feature extraction layer of the fourth layer of the second backbone network comprises a second output; and the optical flow feature of the second output is input into the second input of the concatenation layer of the fusion module of the fourth layer.

[0010] The image target detection device according to some embodiments of the present application further comprises an SPPF module and a C2PSA module. The neck network comprises a first feature fusion layer, a second feature fusion layer and a high-efficiency multi-scale attention mechanism module. The first feature fusion layer comprises three layers arranged at different layers; the second feature fusion layer comprises three layers arranged at different layers; and the high-efficiency multi-scale attention mechanism module comprises two layers arranged at different layers. The input of the SPPF module is the fusion feature output by the fusion module of the fourth layer, the input of the C2PSA module is the output of the SPPF module, and the output of the C2PSA module comprises a first output and a second output. The first feature fusion layer comprises a first input and a second input, and comprises a first output. The first input of the first feature fusion layer of the third layer is the first output of the C2PSA module, and the second input is the fusion feature output by the fusion module of the third layer. The first input of the first feature fusion layer of the second layer is the first output of the first feature fusion layer of the third layer, and the second input is the fusion feature output by the fusion module of the second layer; the first feature fusion layer of the second layer further comprises a second output; The first input of the first feature fusion layer of the first layer is the first output of the first feature fusion layer of the second layer, and the second input is the fusion feature output by the fusion module of the first layer; The input of the high-efficiency multi-scale attention mechanism module of the second layer is the second output of the first feature fusion layer of the second layer. The input of the high-efficiency multi-scale attention mechanism module of the first layer is the first output of the first feature fusion layer of the first layer, and the output of the high-efficiency multi-scale attention mechanism module of the first layer comprises the first output and the second output. The second feature fusion layer comprises a first input and a first output. The second feature fusion layer of the second layer further comprises a second input and a second output, the first input is the second output of the high-efficiency multi-scale attention mechanism module of the first layer, and the second input is the output of the high-efficiency multi-scale attention mechanism module of the second layer. The second feature fusion layer of the third layer further comprises a second output, and the first input is the first output of the second feature fusion layer of the second layer. The first input of the second feature fusion layer of the fourth layer is the first output of the second feature fusion layer of the third layer, and the second input is the second output of the C2PSA module.

[0011] The image target detection device according to some embodiments of the present application comprises four detection heads arranged at different layers; wherein the input of the detection head of the first layer is the first output of the high-efficiency multi-scale attention mechanism module of the first layer. The input of the detection head of the second layer is the second output of the second feature fusion layer of the second layer. The input of the detection head of the second layer is the second output of the second feature fusion layer of the third layer. The input of the detection head of the fourth layer is the first output of the second feature fusion layer of the third layer.

[0012] The image target detection device according to some embodiments of the present application comprises a first multiplication fusion layer, which performs a multiplication operation on the channel prior feature and the spatial attention feature to obtain a fusion feature. The image target detection device further comprises a first convolution layer and a first activation function, which are arranged between the concatenation layer and the channel attention mechanism layer. The image target detection device further comprises a second convolution layer, which is arranged between the second addition fusion layer and the first multiplication fusion layer. The image target detection device further comprises a third convolution layer, which is arranged downstream of the first addition fusion layer. The image target detection device further comprises a fourth convolutional layer arranged at an input end of the first backbone network. The image target detection device further comprises a fifth convolutional layer arranged at an input end of the second backbone network.

[0013] The image target detection device according to some embodiments of the present application, the visible light modality image under a long-distance complex scene containing a fish farming vessel, the optical flow modality image is an optical flow modality image under a long-distance complex scene containing a fish farming vessel, and the image target includes the fish farming vessel.

[0014] In a second aspect, the image target detection method according to some embodiments of the present application is based on any one of the devices, comprising inputting the visible light modality image into the first backbone network; inputting the optical flow modality image into the second backbone network; the first backbone network extracts visible light features, and the second backbone network extracts optical flow features; the fusion module splices the visible light features and the optical flow features to obtain spliced features, acquires channel prior features of the spliced features, and acquires spatial attention features according to the channel prior features; the fusion module fuses the channel prior features and the spatial attention features to obtain fusion features, inputs the fusion features into the neck network, and outputs the image target by the detection head.

[0015] Advantages: The present application takes YOLOv11 as a basic model, proposes a fish vessel target detection device working in a low-quality image environment, and the network used in the device is YOLOv11-OPNet.

[0016] (1) The fish vessel small target is confused with the reef features, resulting in a decrease in recognition accuracy. The present application introduces an optical flow modality, uses dynamic information in the optical flow to distinguish from static reefs, and at the same time, proposes a new strategy of YOLO optical flow modality and visible light modality fusion CPF module, and improves the recognition effect.

[0017] (2) In order to compare the accurate detection of some distant dense small target scenes, the present application adds a P2 layer with larger resolution and a detection head for detecting small targets, and enhances the semantic expression ability of small targets.

[0018] (3) The present application adds an EMA attention mechanism in the neck network (neck), improves the capture ability of the network to key features, significantly improves the model performance, and shows stronger robustness in complex scenes.

[0019] (4) The application introduces Focaler-IoU as a new loss function, focuses on difficult samples such as small occluded targets, optimizes the geometric parameters of the bounding box, reduces missed detection and mispositioning, and improves the detection recall rate.

[0020] (5) The application combines the channel attention and spatial attention of the visible light mode and the optical flow mode fusion module (CPF) to realize dynamic weight distribution and enhance key information. The channel attention globally evaluates the channel features of the visible light (RGB) and the optical flow (OpticalFlow), suppresses the redundant channels, and enhances the channels sensitive to fishing boat detection. The obtained channel prior graph is input into the depth separable convolution module to obtain the spatial attention graph. The spatial attention is captured through multi-scale depth separable convolution to capture the complementary spatial information between modes (such as the motion trajectory in the optical flow and the texture details in the visible light), avoid noise interference caused by the simple superposition of Concat, and the dynamic weight mechanism enables the model to focus on key features rather than treating all features equally like Concat, thereby improving the detection accuracy, especially for the breeding operation fishing boat target in the long-distance complex scene. The long shooting distance is realized, and the accuracy is greatly improved.

[0021] Additional aspects and advantages of the application will be in part apparent and in part pointed out hereinafter. BRIEF DESCRIPTION OF DRAWINGS

[0022] Figure 1 is a flow chart of the method of the application.

[0023] Figure 2 Visible light scene diagram, wherein (a) is the Golden Toad Port of Dachangshan Island, (b) is the Sea View House of Liu Fu Wharf, (c) is the Sea View House of Li Changqun, and (d) is the Huolong Breeding Farm.

[0024] Figure 3 Optical flow visualization diagram, wherein (a) is the Linyang International of Dachangshan Island, (b) is the Changshan Bridge, (c) is the Sea View House of Liu Fu Wharf, and (d) is the Sea View House of Li Changqun.

[0025] Figure 4 YOLOv11-OPNet network structure diagram.

[0026] Figure 5 YOLOv11-OPNet detection flow chart.

[0027] Figure 6 Visible light mode and optical flow mode fusion module (CPF), wherein (a) is a CPF module structure diagram, and (b) is a ChannelAttention module structure diagram.

[0028] Figure 7EMA attention mechanism structure diagram.

[0029] Figure 8 YOLOv11 and YOLOv11-OPNet detection effect comparison, wherein (a) YOLOv11 effect, (b) model detection effect of the application.

[0030] Figure 9 YOLOv11-OPNet accuracy line comparison chart.

[0031] Figure 10 Each model heat map comparison effect diagram, wherein (a) RT-DETR, (b) YOLOv8, (c) YOLOv11, (d) YOLOv12, (e) YOLOv11-OPNet. DETAILED DESCRIPTION

[0032] The embodiments of the present application will be described in detail below with reference to the accompanying drawings. The present application collects shore camera shooting data in different scenes in the shore environment, conducts experiments on each scene respectively, and constructs a fishing boat data set after sorting the data. The data set is from more than ten fixed cameras of fishing boat ports (Dalian Changhe Dachangshan Island Jinchan Port, Dalian Changhe Guangludao, etc.), and the source data format is.MP4 video file. After experimental analysis, the video of each scene is extracted as an image at an interval of 20 frames, 150-200 pictures are selected for each scene, 2414 visible light image data are obtained in.PNG format, and the obtained visible light image data are as shown in Figure 2 Optical flow is a technique for estimating the motion speed and direction of pixel points in an image sequence. The core idea is to analyze the gray level change of adjacent frames to infer the instantaneous motion vector field of the object in the two-dimensional imaging plane. Traditional optical flow methods (such as Horn-Schunck and Lucas-Kanade) are based on hand-designed energy functions, and solve the optical flow by variational method or difference method, relying on the brightness constancy assumption (the brightness of the same target does not change in adjacent frames) and the small motion assumption (the pixel displacement of adjacent frames is small). For example, the HS algorithm introduces global smoothing constraints and iteratively solves the optical flow field through the Euler-Lagrange equation. Such methods are prone to failure in complex scenes, such as Figure 3 , and have high computational complexity. Therefore, the RAFT optical flow algorithm is used to extract optical flow images in the experiment. RAFT uses an end-to-end deep learning architecture to construct pixel-level global similarity using a 4D correlation volume, and adopts a high-resolution fixed flow field processing to avoid information loss. The adjacent two frames of the extracted visible light pictures are input into the RAFT model, and the visible light data are converted into optical flow modal data.

[0033] After acquiring the fishing boat image, the fishing boat target in the image is drawn a bounding box in a manual annotation manner to ensure the accuracy of the box selection. Since the optical flow data is obtained by comparing two frames of visible light images, it corresponds to the visible light image one by one, so only the visible light data is labeled. The fishing boat label is unified as "ship".

[0034] In order to achieve better training effect of the network, the labeled fishing boat data set is randomly divided according to the ratio of 8:1:1, and the number of training set, validation set and test set is 1928, 241 and 241 respectively. In order to make the evaluation result of the test set more reliable, in the data set division process, the test set and the training set come from different video segments, effectively avoiding the problem of data leakage, and improving the generalization ability of the model.

[0035] Embodiment: YOLOv11-OPNet network of the present application. Analyzing the fishing boat data in the culture operation, it is found that most of the scenes are small targets in the distance, so YOLOv11 is selected as the basic model. The latest YOLOv12 focuses on improving the attention mechanism, and reduces the number of network layers. These changes lead to the detection effect of small targets in the distance to decline, and there is no good adaptability to the optical flow-visible light dual mode, so it is not applicable. In addition, considering the real-time and high efficiency of aquaculture, the model size used by the present application is YOLOv11L.

[0036] The core of YOLOv11 is the modular architecture reconstruction and dynamic training strategy, which realizes the breakthrough balance of precision and efficiency through C3k2 module, C2PSA attention mechanism and NMS-Free detection head. The model adopts adaptive image enhancement technology, supports multi-modal input such as visible light, infrared and depth map, uniformly scales the input image to 640x640 resolution, and integrates dynamic Mosaic enhancement (four image splicing) and Mixup mixed enhancement, which improves the generalization ability in complex scenes.

[0037] The backbone network is based on an improved CSP Darknet framework, introduces a C3k2 module (allowing custom convolution kernel size, enhancing feature extraction flexibility) and a C2PSA component (integrating a position-sensitive attention mechanism, dynamically focusing on key areas through spatial weights), and compared with YOLOv8, the parameter quantity is reduced by 22%, and the small target detection accuracy (COCO dataset) is improved by 5.2%. The neck adopts an improved PAN architecture, fuses multi-scale information through a bidirectional feature pyramid, adds a SPFF spatial pyramid fast pooling module to expand the receptive field, and reduces the false detection rate in dense scenes by 18%. The head adopts a double-head design: in the training stage, a one-to-many prediction head is activated (generating multiple candidate boxes to enhance the supervision signal), and in the inference stage, the one-to-one prediction head is switched to directly output the optimal result, eliminating the delay of traditional non-maximum suppression (NMS), and the end-to-end inference speed is improved by 30%.

[0038] Although the performance of YOLOv11 is very good, it has good performance in most normal scenes. However, when the imaging quality of the camera is low, the features of distant targets and small targets are not obvious, which reduces the recognition accuracy of YOLOv11 in real fishing environments. At the same time, in the complex scene with many reefs, and under the condition of low-resolution images, the features of reefs and fishing boats are confused, and accurate recognition is not possible.

[0039] In view of the above problems, the present application provides a small target fishing boat recognition detection network in a low-quality image environment, as shown in Figures 4-6 The network structure includes a first backbone network for inputting a visible light modal image, the first backbone network including a first feature extraction layer, the output of the first feature extraction layer including visible light features; a second backbone network for inputting a light flow modal image, the second backbone network including a first feature extraction layer, the output of the first feature extraction layer including light flow features; a fusion module arranged between the first feature extraction layer of the first backbone network and the first feature extraction layer of the second backbone network, inputting visible light features and light flow features, and outputting fusion features including channel attention features and spatial attention features; a neck network, the input of the neck network including the fusion features; a detection head, the input of the detection head including the output of the neck network, and the output of the detection head including image targets.

[0040] The fusion module includes a splicing layer for splicing the visible light features and the light flow features to obtain spliced features; a channel attention mechanism layer for obtaining channel prior features of the spliced features; a spatial attention mechanism layer for obtaining spatial attention features of the channel prior features; and a fusion layer for fusing the channel prior features and the spatial attention features to obtain the fusion features.

[0041] The channel attention mechanism layer includes a max pooling layer, which performs a max pooling operation on the spliced features to output a max-pooled feature vector; an average pooling layer, which performs an average pooling operation on the spliced features to output an average-pooled feature vector; a multilayer perceptron, which inputs the max-pooled feature vector to output a first output vector; and a multilayer perceptron, which inputs the average-pooled feature vector to output a second output vector; a first addition fusion layer, which performs an addition operation on the first output vector and the second output vector to generate a fusion feature vector; and an activation function, which inputs the fusion feature vector to generate a channel attention mechanism map; and a second multiplication fusion layer, which multiplies the spliced features and the channel attention mechanism map to obtain channel prior features.

[0042] The spatial attention mechanism layer includes a depth separable convolution and a second addition fusion layer, wherein the depth separable convolution includes a first depth convolution, an input of the first depth convolution includes the channel prior features, the first depth convolution has at least two outputs; and a second depth convolution, the second depth convolution includes at least two, an input of each second depth convolution is connected to an output of one first depth convolution; wherein the second addition fusion layer performs an addition operation on the features output by each second depth convolution to obtain spatial attention features.

[0043] The neck network includes a first feature fusion layer and a second feature fusion layer; wherein the neck network further includes an efficient multi-scale attention mechanism module, the efficient multi-scale attention mechanism module is arranged between the first feature fusion layer and the detection head of the neck network and / or arranged between the first feature fusion layer and the second feature fusion layer of the neck network, and the fusion features output by the fusion module are input into the first feature fusion layer.

[0044] The fusion module includes four modules arranged at different layers; the first feature extraction layer of the first backbone network includes four modules arranged at different layers; the first feature extraction layer of the second backbone network includes four modules arranged at different layers; wherein the first feature extraction layer of the first layer of the first backbone network includes a first output and a second output; the visible light feature of the first output is input into the first feature extraction layer of the second layer of the first backbone network; the visible light feature of the second output is input into the first input of the splicing layer of the fusion module of the first layer; the first feature extraction layer of the first layer of the second backbone network includes a first output and a second output; the optical flow feature of the first output is input into the first feature extraction layer of the second layer of the second backbone network; and the optical flow feature of the second output is input into the second input of the splicing layer of the fusion module of the first layer.

[0045] The first feature extraction layer of the second layer of the first backbone network comprises a first output and a second output; the visible light features of the first output are input into the first feature extraction layer of the third layer of the first backbone network; the visible light features of the second output are input into the first input of the concatenation layer of the fusion module of the second layer; the first feature extraction layer of the second layer of the second backbone network comprises a first output and a second output; the optical flow features of the first output are input into the first feature extraction layer of the third layer of the second backbone network; and the optical flow features of the second output are input into the second input of the concatenation layer of the fusion module of the second layer.

[0046] The first feature extraction layer of the third layer of the first backbone network comprises a first output and a second output; the visible light features of the first output are input into the first feature extraction layer of the fourth layer of the first backbone network; the visible light features of the second output are input into the first input of the concatenation layer of the fusion module of the third layer; the first feature extraction layer of the third layer of the second backbone network comprises a first output and a second output; the optical flow features of the first output are input into the first feature extraction layer of the fourth layer of the second backbone network; and the optical flow features of the second output are input into the second input of the concatenation layer of the fusion module of the third layer. The first feature extraction layer of the fourth layer of the first backbone network comprises a second output; the visible light features of the second output are input into the first input of the concatenation layer of the fusion module of the fourth layer; the first feature extraction layer of the fourth layer of the second backbone network comprises a second output; and the optical flow features of the second output are input into the second input of the concatenation layer of the fusion module of the fourth layer.

[0047] The network further comprises an SPPF module and a C2PSA module; the neck network comprises a first feature fusion layer, a second feature fusion layer and a high-efficiency multi-scale attention mechanism module; the first feature fusion layer comprises three layers arranged at different levels; the second feature fusion layer comprises three layers arranged at different levels; and the high-efficiency multi-scale attention mechanism module comprises two layers arranged at different levels.

[0048] The input of the SPPF module is the fusion features output by the fusion module of the fourth layer, the input of the C2PSA module is the output of the SPPF module, and the output of the C2PSA module comprises a first output and a second output.

[0049] The first feature fusion layer includes a first input and a second input, and includes a first output; the first input of the first feature fusion layer of the third layer is the first output of the C2PSA module, and the second input is the fusion feature output by the fusion module of the third layer; the first input of the first feature fusion layer of the second layer is the first output of the first feature fusion layer of the third layer, and the second input is the fusion feature output by the fusion module of the second layer; the first feature fusion layer of the second layer further includes a second output; the first input of the first feature fusion layer of the first layer is the first output of the first feature fusion layer of the second layer, and the second input is the fusion feature output by the fusion module of the first layer.

[0050] The input of the efficient multi-scale attention mechanism module of the second layer is the second output of the first feature fusion layer of the second layer; the input of the efficient multi-scale attention mechanism module of the first layer is the first output of the first feature fusion layer of the first layer, and the efficient multi-scale attention mechanism module of the first layer outputs includes a first output and a second output; wherein the second feature fusion layer includes a first input and a first output; the second feature fusion layer of the second layer further includes a second input and a second output, the first input is the second output of the efficient multi-scale attention mechanism module of the first layer, and the second input is the output of the efficient multi-scale attention mechanism module of the second layer; the second feature fusion layer of the third layer further includes a second output, and the first input is the first output of the second feature fusion layer of the second layer; the first input of the second feature fusion layer of the fourth layer is the first output of the second feature fusion layer of the third layer, and the second input is the second output of the C2PSA module.

[0051] The detection head includes four, which are arranged at different layers; wherein the input of the detection head of the first layer is the first output of the efficient multi-scale attention mechanism module of the first layer; the input of the detection head of the second layer is the second output of the second feature fusion layer of the second layer; the input of the detection head of the second layer is the second output of the second feature fusion layer of the third layer; the input of the detection head of the fourth layer is the first output of the second feature fusion layer of the third layer.

[0052] The fusion layer includes a first multiplication fusion layer, which performs multiplication operation on the channel prior feature and the spatial attention feature to obtain the fusion feature; the image target detection device further includes a first convolution layer and a first activation function, which are arranged between the splicing layer and the channel attention mechanism layer; the image target detection device further includes a second convolution layer, which is arranged between the second addition fusion layer and the first multiplication fusion layer; the image target detection device further includes a third convolution layer, which is arranged downstream of the first addition fusion layer; the image target detection device further includes a fourth convolution layer, which is arranged at the input end of the first backbone network; the image target detection device further includes a fifth convolution layer, which is arranged at the input end of the second backbone network.

[0053] The first feature extraction layer includes Conv, C3K2. The first feature fusion layer includes Up, Concat and C3K2. The second feature fusion layer includes Conv, C3K2 and Concat.

[0054] Specifically, the application first introduces an optical flow branch on the basis of YOLOv11, extracts the features of visible light and optical flow, then performs feature fusion of the visible light mode and the optical flow mode through the proposed CPF module at the P2-P5 layer, combines the dynamic information of the optical flow mode, solves the confusion problem of fishing boats and reefs in complex environments, at the same time, the newly added P2 neck network (Neck) feature extraction layer and the P2 level detection head make the detection of small targets more accurate. The position of the feature map with higher resolution increases the EMA spatial attention mechanism, so that the network can dynamically pay attention to small targets that are difficult to detect in the distance. Finally, the Focaler-IoU loss function is introduced to solve the problem that the traditional cross-entropy loss and the basic IoU loss are difficult to distinguish between difficult and easy samples, leading to the model being dominated by simple samples, and the overall performance of fishing boat detection is improved.

[0055] The dual-mode structure of the application is improved. The traditional YOLO fishing boat detection algorithm is a visible light single-mode input. The visible light image has rich color and texture details, but when the camera imaging quality is poor and the target distance is far, the visible light feature is very poor. At this time, the optical flow branch is introduced to supplement the motion features of the fishing boat, and the data of the two modes are used for information complementation to improve the accuracy of fishing boat detection. Figure 4 YOLOv11-OPNet network structure diagram.

[0056] Figure 5 YOLOv11-OPNet detection flow chart, as Figure 5 shown, first, the input visible light video is divided into two branches, one directly enters the first backbone network of YOLOv11-OPNet for visible light feature extraction, and the other branch enters the RAFT data preprocessing model to convert the visible light video into optical flow mode information, and then enters the first backbone network of YOLOv11-OPNet for optical flow feature extraction.

[0057] Subsequently, the features extracted from the two modes are sent to the visible light mode and optical flow mode fusion module (CPF) of the application for feature fusion. The visible light mode and optical flow mode fusion module (CPF) is referred to as "fusion module" hereinafter.

[0058] Finally, the fused information enters the neck network (Neck) for finer feature extraction, and the detection is completed through the detection head (Detect).

[0059] In the fishery farming scene, the fishing boat image obtained by the fishing boat target due to the distance between the shore camera and the sea fishing boat is far away, or the influence of extreme weather, the pixel of the fishing boat target is missing, and the information of the reef target is similar, which directly leads to the reduction of the accuracy of the fishing boat target detection. Therefore, it is very important to fully utilize the motion information in the optical flow mode for fishing boat small target detection. Selecting fusion in the backbone network can extract more information and achieve better fusion effect.

[0060] The traditional Concat method only increases the channel number by splicing, and cannot effectively utilize the spatial correlation of cross-modal. Meanwhile, in the real fishing boat detection task, the visible light mode may cause the target to be blurred due to light changes, and the optical flow mode may produce noise due to sea wave interference.

[0061] In summary, the application designs a visible light mode and optical flow mode fusion module (CPF) combining channel attention and spatial attention, realizes dynamic weight distribution and enhances key information. The channel attention will globally evaluate the channel features of the visible light (RGB) and the optical flow (OpticalFlow), suppress the redundant channels and enhance the channels sensitive to fishing boat detection; the spatial attention captures the complementary spatial information between modes (such as the motion trajectory in the optical flow and the texture details in the visible light) through multi-scale depth separable convolution, avoids the noise interference caused by the simple superposition of Concat, and this dynamic weight mechanism enables the model to focus more on key features, rather than treating all features equally like Concat, thereby improving the detection accuracy.

[0062] Figure 6 For the visible light mode and optical flow mode fusion module (CPF), as shown in Figure 6 (a), first, the visible light features and the optical flow features are spliced, and after channel matching through 1x1 Conv operation and activation function, they are input into the channel attention mechanism. The channel attention mechanism structure is as shown in Figure 6 (b), the spliced visible light features and the optical flow features are subjected to average pooling and maximum pooling operations to summarize the spatial information in the feature map, and then the processed information is input into a shared multi-layer perceptron (MLP). The output of the shared multi-layer perceptron is summed by element, passed through an activation function, and multiplied by the spliced features to obtain the channel prior graph (channel prior feature), as shown in formula 1: (1) In the formula, represents the channel prior feature, F represents the input spliced feature, represents an activation function.

[0063] as shown in Figure 6(a) shown, the obtained channel prior map is input into a deep separable convolution module to obtain a spatial attention map (spatial attention feature), as shown in formula 2: (2) wherein, represents the spatial attention feature, represents the first deep convolution, represents the second deep convolution, ∈{1,2,3} represents the first branch. represents a 1x1 Conv operation for matching.

[0064] As Figure 6 (a) shown, finally, the channel prior map is multiplied by the spatial attention map to obtain a fusion feature, as shown in formula 3: (3) As Figure 4 shown, the visible light modal and the optical flow modal fusion module (CPF) is located in the YOLOv11-OPNet network structure as shown, which is a Cross Feture Enhence module.

[0065] In a real fishing boat operation scene, a small fishing boat target in the distance often occupies a very small pixel area in the image, and traditional single-scale feature extraction is easy to ignore details. The efficient multi-scale attention mechanism module (EMA) is a new technology used in the field of computer vision, which can improve the feature extraction and representation ability of the model in a multi-scale scene. The EMA attention mechanism can capture different scale features through a parallel multi-branch structure, and the 1x1 convolution branch combines adaptive average pooling to extract global semantic information, which assists in the positioning of small targets in complex backgrounds. The 3x3 convolution branch expands the receptive field through grouped convolution to enhance the sensitivity to local features such as fishing boat edges and textures. This multi-scale fusion strategy enables the model to focus on both the overall position and local details of small targets, improving detection accuracy in fishing boat monitoring data sets, significantly enhancing the semantic expression ability of small targets, and improving the robustness of the model. In summary, the EMA attention mechanism is introduced. The structure of the EMA attention mechanism is shown in Figure 7 , and the EMA is located in the YOLOv11-OPNet network structure as Figure 4 shown, which is an Attention module.

[0066] The Focaler-IoU loss function of the application. In the fishery farming scene, most of the fishing boat targets in the image are relatively small compared to the background, and some scenes are too dense. The cross-entropy loss used by YOLOv11 itself is difficult to focus on difficult samples. Therefore, in view of the above problems, the application introduces Focaler-IoU as a loss function to enhance the network's learning ability for difficult samples and improve the accuracy and robustness of detection.

[0067] Focaler-IoU is a loss function that improves the regression of the bounding box of target detection by dynamically adjusting the attention of samples and fusing a geometric optimization term. Its core formula is shown in formulas 3-4. Wherein And are threshold parameters for dynamically adjusting the attention of the model to different difficulty samples. is the lower threshold value, which controls the attention to low-quality samples (for example =0.00 ignores extremely low IoU samples). is the upper threshold value, which filters the boundary of high-quality samples (for example =0.95 focuses on difficult samples with IoU≥95%).

[0068] (3) (4) In the formula, indicates the improved calculation method of the intersection over union; indicates the intersection over union, which is an index for measuring the overlap of two bounding boxes; indicates the loss function using GIoU, indicates the final loss function calculation method.

[0069] Model training and evaluation of the application: In order to ensure the accuracy and reliability of the experimental data, the training and testing of the small target fishing boat detection network are run in the same environment. The experimental environment is shown in Table 1. The batch size is 16, the training period is 300 rounds, and the initial learning rate is 0.0001.

[0070] Table 1 Experimental environment Evaluation metrics: The detection accuracy and high inference speed play an important role in real-time fishing vessel target detection. To comprehensively evaluate the performance of the proposed method, the confusion matrix is used as an evaluation tool, including the following key indicators: true positive (TP), true negative (TN), false positive (FP), and false negative (FN).

[0071] In terms of network detection accuracy, precision rate (P), recall rate (R), and mean average precision (mAP@0.5) are selected as evaluation metrics. In addition, fishing vessel detection is a task with specific requirements, and the accuracy rate of fishing vessels should be the primary indicator for evaluating the detection accuracy of the model compared to the recall rate. The calculation formulas are shown in Equations 5-8: (5) (6) (7) (8) Experimental results and analysis: (1) Ablation experiment To verify the effectiveness of the model improvement part, an ablation experiment was designed. The experimental scheme is as follows: the optical flow modality, P2 feature map detection layer, EMA attention mechanism module, and Focaler-IoU loss function are added to the baseline model in turn to construct multiple models with different configurations, named YOLOv11-OP, YOLOv11-OE, YOLOv11-OF, YOLOv11-OPE, and YOLOv11-OPF. The suffix represents the combination of modules contained in the model. The experimental results are shown in Table 2, which shows that the above improvements have certain performance improvement for the baseline model. The final network model achieves an accuracy rate of 87.1% and a fishing vessel recall rate of 79.6%, which is a significant improvement of 7.5% and 14.3% compared to the baseline model.

[0072] In the real fishing scene, most fishing boats are small targets far away and too dense, which seriously affects the accuracy of detection. By adding a P2 feature extraction layer with higher resolution, the detection performance of small targets is enhanced, and the accuracy and recall rates are improved by 3.7% and 9.3%, respectively. Due to the limited focal length of the camera device and the complex environment by the sea, the quality of the visible light image is low, the effective information carried by the target is missing, and the static target such as the reef cannot be well distinguished from the fishing boat target. The optical flow mode is introduced to extract the motion information in the optical flow mode to assist the model in distinguishing other interference objects in the scene, effectively improving the accuracy by 2.4%. Subsequently, in order to solve the problem that small fishing boat targets far away often occupy a very small pixel area in the image, and traditional single-scale feature extraction easily ignores the details of the fishing boat target, the EMA attention mechanism module is introduced to better assist the positioning of small targets in complex backgrounds, significantly improving the recall rate. Finally, Focaler-IoU is used as the loss function to dynamically adjust the attention of the model to different samples, enhancing the network's learning of difficult samples, and finally improving the accuracy and recall rates by 1.5% and 1.6%, respectively. Compared with the baseline model, the final network has a significant advantage in the accuracy and speed of fishing boat detection. Figure 8 and Figure 9 The intuitive comparison of the detection effect and performance of the method of the application and the baseline model is shown in Table 1 and Table 2, respectively, which further verifies the effectiveness of the improvement strategies and the good synergy between them.

[0073] Table 2 Ablation experiment results (2) Model loss function comparison test To further verify the effectiveness of the selected loss function on the detection accuracy of the model, a comparison experiment of different loss functions is designed. YOLOv11 is added with optical flow mode, P2 detection layer, and EMA attention mechanism, which is recorded as Base. Base is added with SlideLoss, GIoU, DIoU, CIoU, and Focaler-IoU as loss functions, respectively. The experimental results are shown in Table 3, which clearly shows the influence of different loss functions on the accuracy (P), recall rate (R), and average precision (mAP@0.5) of the model.

[0074] SlideLoss solves the problem of sample imbalance by dividing samples, improves model accuracy, and performs well in classification tasks. However, in the fishing vessel detection task of fishing operations, the samples are relatively single and there is no imbalance problem, so the improvement is limited. GIoU (Generalized Intersection over Union) introduces the minimum closed rectangle formula based on IoU, solves the problem that the gradient is 0 when there is no overlap, and enhances the detection accuracy of non-overlapping regions, but it is not suitable for fishing vessel scenes with overlapping conditions, so the improvement is limited. DIoU (Distance-IoU) uses a center point distance penalty method to solve the problem of reduced accuracy when detecting horizontally or vertically arranged boxes with IoU, but it is very unstable in small target or dense scenes, which directly leads to a 0.6% reduction in model accuracy. CIoU (Complete IoU) adds a length-width ratio penalty term based on DIoU, which improves accuracy, but it is ineffective when the length-width ratio is the same, such as the scenario of a large box inside a small box when fishing vessels overlap. Focaler-IoU combines the advantages of SlideLoss, GIoU and CIoU, enabling the model to focus on difficult samples (such as occlusion scenarios) while improving the detection ability of small targets. Ultimately, compared with other loss functions, the model's accuracy is improved by 1.8%.

[0075] Table 3 Comparison of detection performance of different models (3) Comparative experiment To further verify the effectiveness and advancement of the network model of the present application for fishing vessel detection, a comparative experiment is conducted by selecting the currently higher detection effect model in this field, including RT-DETR, YOLOv8, YOLOv12 and baseline model YOLOv11. The experimental results are shown in Table 4, compared with other algorithms, the method proposed in the present application achieves the best effect in comprehensive performance.

[0076] Table 4 Comparison of detection performance of different models Compared with RT-DETR, the method of the application significantly improves the detection accuracy and recall rate by 12.8% and 8.7%. This is because in the small target detection task, although RT-DETR adopts multi-scale feature fusion, the global attention mechanism of the Transformer weakens the local details, resulting in a detection accuracy of small targets that is much lower than that of some CNN models (such as the P AN-FPN structure of YOLOv8). In addition, in comparison with the other three YOLO series algorithms, the method of the application also shows advancement, achieving a significant improvement of 7.7%, 7.5% and 7.7% in fishing boat accuracy, which is mainly due to the fact that the low image quality causes the feature information carried by small targets in the distance to be not obvious, and at the same time, the original model of YOLO has only three detection heads, which weakens the detection ability of small target individuals and affects the overall detection performance of the model. Figure 10 The heat map directly reflects the attention degree of the region to the model. Through the color contrast of the heat map, it is shown that YOLOv11-OPnet has high accuracy in the detection of small target fishing boats.

[0077] In the task of detecting fishing boats in complex scenes at a distance, the application proposes a detection method YOLOv11-OPNet (YOLOv11-OpticalFlowNet) for fishing boats in the case where the features of targets at a distance are not obvious, aiming to accurately identify fishing boats with poor image quality in the fishing environment. By designing the introduction of an optical flow mode in the backbone network, combined with the dynamic information in the optical flow, the problem of confusion between fishing boats and reefs in complex environments is solved, significantly enhancing the adaptability of the model in complex environments. Secondly, in order to better combine the channel priori and spatial attention mechanism, a CPF cross-modal fusion module is proposed to solve the combination of information of the visible light mode and the optical flow mode, so that more information can be extracted to achieve better fusion effect. In the neck network, an EMA attention mechanism is proposed to mainly improve the network's ability to capture key features and significantly improve the model's performance, showing stronger robustness in complex scenes. Finally, Focaler-IoU is used as a new loss function to solve the problem that the traditional cross-entropy loss and the basic IoU loss cannot distinguish between difficult and easy samples, leading to the model being dominated by simple samples, and the overall performance of fishing boat detection is improved. Experimental results show that the improved model improves the accuracy, recall rate, and average precision by 7.5%, 14.3%, and 11.6%, respectively.

[0078] Based on the above embodiments, the application also provides a computer program which, when running on a computer, causes the computer to execute the method provided by the above embodiments.

[0079] Based on the above embodiments, the embodiments of the present application further provide a computer storage medium, which stores a computer program. The computer program is executed by a computer, so that the computer executes the method provided in the above embodiments.

[0080] The storage medium can be any available medium that can be accessed by a computer. By way of example, and not limitation, such computer-readable media can comprise RAM, ROM, EEPROM, CD-ROM or other optical disk storage, magnetic disk storage or other magnetic storage devices, or any other medium that can be used to carry or store desired program code in the form of instructions or data structures and that can be accessed by a computer. Computer-readable media can further include, but are not limited to, bus- based memory cores, program

[0081] Based on the above embodiments, the embodiments of the present application further provide a chip for reading a computer program stored in a memory, and implementing the method provided in the above embodiments.

[0082] Based on the above embodiments, the embodiments of the present application provide a computer program product. When the computer program product runs on an electronic device, the method provided in the above embodiments is implemented.

[0083] Those skilled in the art should understand that the embodiments of the present application can be provided as a method, a system, or a computer program product. Therefore, the present application can take the form of an entirely hardware embodiment, an entirely software embodiment, or an embodiment combining software and hardware aspects. Moreover, the present application can take the form of a computer program product implemented on one or more computer-usable storage media (including, but not limited to, disk storage, CD-ROMs, optical storage devices, etc.) containing computer-usable program code.

[0084] The present application is described with reference to the flowcharts and / or block diagrams of the methods, devices (systems), and computer program products according to this application. It should be understood that each flow and / or block in the flowcharts and / or block diagrams, as well as a combination of flows and / or blocks in the flowcharts and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing apparatus to produce a machine, so that the instructions executed by the processor of the computer or other programmable data processing apparatus produce the functions specified in the flowcharts and / or block diagrams. Figure 1 The functions specified in a flow or multiple flows and / or blocks Figure 1 The functions specified in a flow or multiple flows and / or blocks

[0085] These computer program instructions can also be stored in a computer-readable memory that can direct the computer or other programmable data processing apparatus to work in a specific manner, so that the instructions stored in the computer-readable memory produce a manufactured product including instruction apparatus, which implements the functions specified in the flowcharts and / or block diagrams. Figure 1one or more processes and / or blocks Figure 1 the function(s) specified in the block or blocks.

[0086] These computer program instructions can also be loaded into computer or other programmable data processing devices, so that a series of operation steps are performed on the computer or other programmable data processing devices to generate computer-implemented processes, thus the instructions executed on the computer or other programmable data processing devices provide processes for implementing the flow Figure 1 one or more processes and / or blocks Figure 1 the function(s) specified in the block or blocks.

[0087] Obviously, those skilled in the art can make various modifications and variations to the present application without departing from the spirit and scope of the present application. Thus, if these modifications and variations of the present application belong to the scope of the claims of the present application and their equivalents, the present application is also intended to include these modifications and variations.

Claims

1. An image target detection device, characterized in that, include A first backbone network is used to input visible light modal images. The first backbone network includes a first feature extraction layer, and the output of the first feature extraction layer includes visible light features. The second backbone network is used to input the optical flow modality image. The second backbone network includes a first feature extraction layer, and the output of the first feature extraction layer includes optical flow features. The fusion module is located between the first feature extraction layer of the first backbone network and the first feature extraction layer of the second backbone network. The input includes visible light features and optical flow features, and the output includes fused features of channel attention features and spatial attention features. The neck network, whose input includes fused features; The detection head's input includes the output of the neck network, and its output includes the image target. The fusion module includes The splicing layer combines visible light features with optical flow features to obtain spliced ​​features; The channel attention mechanism layer acquires the channel prior features for splicing features; The spatial attention mechanism layer acquires spatial attention features of channel prior features; The fusion layer combines channel prior features with spatial attention features to obtain fused features.

2. The image target detection device according to claim 1, characterized in that, Channel attention mechanism layer, including The max pooling layer performs max pooling on the concatenated features and outputs a max pooled feature vector. The average pooling layer performs average pooling on the concatenated features and outputs an average pooled feature vector. For a multilayer perceptron, the max-pooled feature vector is input into the multilayer perceptron and outputs a first output vector; the average-pooled feature vector is input into the multilayer perceptron and outputs a second output vector. The first additive fusion layer adds the first output vector and the second output vector to generate a fused feature vector; Activation function, fused feature vector input activation function, generates channel attention mechanism diagram; The second multiplication and fusion layer multiplies the spliced ​​features with the channel attention mechanism graph to obtain the channel prior features.

3. The image target detection device according to claim 2, characterized in that, The spatial attention mechanism layer includes depthwise separable convolutions and a second additive fusion layer. The depthwise separable convolutions include... The first depthwise convolution has channel prior features as input and at least two outputs. The second depthwise convolution includes at least two, and the input of each second depthwise convolution is connected to the output of a first depthwise convolution. The second addition fusion layer performs an addition operation on the features output by each second depth convolution to obtain spatial attention features.

4. The image target detection device according to claim 1, characterized in that, The neck network includes a first feature fusion layer and a second feature fusion layer; the neck network also includes an efficient multi-scale attention mechanism module, which is set between the first feature fusion layer of the neck network and the detection head and / or between the first feature fusion layer and the second feature fusion layer of the neck network, and the fused features output by the fusion module are input to the first feature fusion layer.

5. The image target detection apparatus according to any one of claims 1-4, characterized in that, in, The fusion module includes four modules set in different layers; The first feature extraction layer of the first backbone network includes four layers set in different layers; The first feature extraction layer of the second backbone network includes four layers set in different layers; The first feature extraction layer of the first layer of the first backbone network includes a first output and a second output; the visible light features of the first output are input to the first feature extraction layer of the second layer of the first backbone network; the visible light features of the second output are input to the first input of the splicing layer of the fusion module of the first layer. The first feature extraction layer of the first layer of the second backbone network includes a first output and a second output; the optical flow features of the first output are input to the first feature extraction layer of the second layer of the second backbone network; the optical flow features of the second output are input to the second input of the splicing layer of the fusion module of the first layer. The first feature extraction layer of the second layer of the first backbone network includes a first output and a second output; the visible light features of the first output are input to the first feature extraction layer of the third layer of the first backbone network; and the visible light features of the second output are input to the first input of the splicing layer of the fusion module of the second layer. The second feature extraction layer of the second layer of the second backbone network includes a first output and a second output; the optical flow feature of the first output is input to the first feature extraction layer of the third layer of the second backbone network; the optical flow feature of the second output is input to the second input of the splicing layer of the fusion module of the second layer. The first feature extraction layer of the third layer of the first backbone network includes a first output and a second output; the visible light features of the first output are input to the first feature extraction layer of the fourth layer of the first backbone network; the visible light features of the second output are input to the first input of the splicing layer of the fusion module of the third layer. The first feature extraction layer of the third layer of the second backbone network includes a first output and a second output; the optical flow feature of the first output is input to the first feature extraction layer of the fourth layer of the second backbone network; the optical flow feature of the second output is input to the second input of the splicing layer of the fusion module of the third layer. The first feature extraction layer of the fourth layer of the first backbone network includes a second output; the visible light features of the second output are input to the first input of the splicing layer of the fusion module of the fourth layer. The first feature extraction layer of the fourth layer of the second backbone network includes a second output; the optical flow features of the second output are input to the second input of the splicing layer of the fusion module of the fourth layer.

6. The image target detection device according to claim 5, characterized in that, It also includes the SPPF module and the C2PSA module; The neck network includes a first feature fusion layer, a second feature fusion layer, and an efficient multi-scale attention mechanism module. The first feature fusion layer comprises three layers set in different layers; the second feature fusion layer comprises three layers set in different layers; and the efficient multi-scale attention mechanism module comprises two layers set in different layers. The SPPF module takes as input the fusion feature output by the fourth-layer fusion module, and the C2PSA module takes as input the output of the SPPF module. The C2PSA module output includes a first output and a second output. The first feature fusion layer includes a first input and a second input, and includes a first output; The first input of the first feature fusion layer of the third layer is the first output of the C2PSA module, and the second input is the fused feature output by the fusion module of the third layer; The first input of the first feature fusion layer of the second layer is the first output of the first feature fusion layer of the third layer, and the second input is the fused feature output by the fusion module of the second layer; the first feature fusion layer of the second layer also includes a second output; The first input of the first feature fusion layer of the first layer is the first output of the first feature fusion layer of the second layer, and the second input is the fused feature output by the fusion module of the first layer; The input of the efficient multi-scale attention mechanism module in the second layer is the second output of the first feature fusion layer in the second layer; The input of the efficient multi-scale attention mechanism module of the first layer is the first output of the first feature fusion layer of the first layer, and the output of the efficient multi-scale attention mechanism module of the first layer includes the first output and the second output. The second feature fusion layer includes a first input and a first output; The second feature fusion layer of the second layer also includes a second input and a second output. The first input is the second output of the efficient multi-scale attention mechanism module of the first layer, and the second input is the output of the efficient multi-scale attention mechanism module of the second layer. The second feature fusion layer of the third layer also includes a second output, and the first input is the first output of the second feature fusion layer of the second layer. The first input of the second feature fusion layer of the fourth layer is the first output of the second feature fusion layer of the third layer, and the second input is the second output of the C2PSA module.

7. The image target detection device according to claim 6, characterized in that, The detection head consists of four units, which are set in different layers; the input of the detection head in the first layer is the first output of the efficient multi-scale attention mechanism module in the first layer. The input to the detection head of the second layer is the second output of the second feature fusion layer of the second layer; The input to the detection head of the second layer is the second output of the second feature fusion layer of the third layer; The input to the fourth layer's detection head is the first output of the second feature fusion layer of the third layer.

8. The image target detection device according to claim 4, characterized in that, The fusion layer includes a first multiplication fusion layer, which performs a multiplication operation on the channel prior features and the spatial attention features to obtain the fused features; The image target detection device also includes a first convolutional layer and a first activation function, which are disposed between the stitching layer and the channel attention mechanism layer; The image target detection device also includes a second convolutional layer, which is disposed between the second addition fusion layer and the first multiplication fusion layer; The image target detection device also includes a third convolutional layer, which is located downstream of the first additive fusion layer; The image target detection device also includes a fourth convolutional layer, which is located at the input end of the first backbone network; The image target detection device also includes a fifth convolutional layer, which is located at the input of the second backbone network.

9. The image target detection device according to claim 1, characterized in that, Visible light modal images are visible light modal images of distant complex scenes containing aquaculture fishing vessels, while optical flow modal images are optical flow modal images of distant complex scenes containing aquaculture fishing vessels, with the target of the image including the aquaculture fishing vessels.

10. An image target detection method, characterized in that, Based on any one of claims 1-9, including Input visible light modal images into the first backbone network; Input optical flow modal images into the second backbone network; The first backbone network extracts visible light features, and the second backbone network extracts optical flow features; The fusion module stitches visible light features and optical flow features to obtain stitched features, and obtains the channel prior features of the stitched features; based on the channel prior features, spatial attention features are obtained. The fusion module fuses channel prior features with spatial attention features to obtain fused features, which are then input into the neck network, and the detection head outputs the image target.

Images