Dual-mode based device and method for detecting target of aquaculture work fishing boat in long-distance complex scene

Abstract

The application discloses a dual-mode-based target detection device and method for fishing boats in long-distance complex scenes, belongs to the field of image target detection, and aims to solve the problem that the features of fishing boats in long-distance fishing operations are not obvious and are easily confused with sea-view house targets and reef targets in long-distance complex scenes, and the device and method are characterized by comprising a first backbone network for inputting a visible light mode image, the first backbone network comprising a first feature extraction layer, the output of the first feature extraction layer comprising visible light features; a second backbone network for inputting an optical flow mode image, the second backbone network comprising a first feature extraction layer, the output of the first feature extraction layer comprising optical flow features; and a fusion module arranged between the first feature extraction layer of the first backbone network and the first feature extraction layer of the second backbone network, the fusion module inputting the visible light features and the optical flow features and outputting fusion features comprising channel attention features and spatial attention features.

Description

TECHNICAL FIELD

[0001] The application belongs to the field of image target detection, and relates to a dual-mode-based target detection device and method for fish farming work fishing boats in a long-distance complex scene. BACKGROUND

[0002] Fishing boats are important tools for fish farming work, and are of great significance to the development of the fishing industry and the life of fishermen. Fishing boat management is an important means to protect the safety of fishermen in their daily work and to improve the efficiency of fishing work. Traditional fishing boat management requires manual observation and manual recording, which is prone to errors or intentional concealment, resulting in loopholes in illegal fishing supervision. Moreover, it is impossible to achieve real-time supervision all day long. Under this background, intelligent detection technology based on YOLO series algorithms has become the core of the intelligent transformation of the fishing industry. However, the camera is far away from the fishing boat in the real fishing boat operation environment, and the imaging quality is poor, which reduces the recognition accuracy. The complexity of the coastline environment (such as sea-view houses, reefs, and densely parked scenes) also poses a serious challenge to detection technology. Therefore, there is an urgent need to develop a fishing boat detection method in a complex environment.

[0003] The YOLO algorithm series is a high-efficiency architecture paradigm in the field of target detection, and its core feature is to integrate target positioning and classification tasks into a single network framework, which can complete the detection task through a single forward propagation, and has always been committed to achieving a balance between detection accuracy and real-time performance. Due to its advantages in real-time processing, this series of algorithms has been more widely used in actual object detection tasks, especially in scenarios that require real-time performance, such as real-time detection of fishing boats. Huang et al. improved YOLOv4 to detect ships near the port berth. By using the k-means algorithm to strengthen the clustering of input data, the detection accuracy was improved, but the detection of small targets of fishing boats at a distance was not very accurate. Han et al. proposed a ship detection algorithm for unmanned aerial vehicle aerial photography, which incorporated SGE attention mechanism and MCA attention mechanism into YOLOv8n to enhance feature extraction capability, and improved the fusion method to strengthen information understanding, thereby improving detection performance, but the target detection capability in complex backgrounds needs to be improved. In another study, Zhang et al. proposed a detection algorithm for maritime personnel search and rescue. They improved YOLOv7 by introducing a SA attention mechanism for efficient target search area positioning and adding a small object detection head to enhance the detection capability of small targets. Although the YOLO series has been continuously improved, the detection performance is commendable, but there is still room for improvement in detection accuracy, especially in long-distance complex scenes, where the features of fishing boats in distant fishing operations are not obvious, and are easily confused with sea-view house targets and reef targets. The above models have not solved this problem.

[0004] In summary, the YOLO target detection technology still faces the following challenges in the detection task of fishing vessels in fishing operations: (1) The camera is placed far away from the shore, and the focal length of the camera is limited, resulting in poor image quality. The target features of the fishing vessels in operation are confused with the sea-view room; (2) The fishing vessel target in the real scene is small, and the detection layer and detection head of the original YOLO model cannot better detect small targets; (3) The sea surface environment is complex, and the gradient fluctuation is large due to the change of sea wave and water surface light. The original YOLO model cannot better locate the small fishing vessel target; (4) The fishing vessels in the fishing operation may be blocked by each other. SUMMARY

[0005] In order to solve the problem that the features of fishing vessels in the fishing operation at a long distance in a complex scene are not obvious and are easily confused with the sea-view room target and the reef target, in a first aspect, an image target detection device according to some embodiments of the application comprises

[0006] a first backbone network configured to input a visible light modality image, the first backbone network comprising a first feature extraction layer, an output of the first feature extraction layer comprising visible light features;

[0007] a second backbone network configured to input an optical flow modality image, the second backbone network comprising a first feature extraction layer, an output of the first feature extraction layer comprising optical flow features;

[0008] a fusion module disposed between the first feature extraction layer of the first backbone network and the first feature extraction layer of the second backbone network, configured to input the visible light features and the optical flow features, and output fusion features comprising channel attention features and spatial attention features;

[0009] a neck network, an input of the neck network comprising the fusion features;

[0010] a detection head, an input of the detection head comprising an output of the neck network, an output of the detection head comprising image targets;

[0011] wherein the fusion module comprises

[0012] a concatenation layer configured to concatenate the visible light features and the optical flow features to obtain concatenated features;

[0013] a channel attention mechanism layer configured to obtain channel prior features of the concatenated features;

[0014] a spatial attention mechanism layer configured to obtain spatial attention features of the channel prior features;

[0015] a fusion layer configured to fuse the channel prior features and the spatial attention features to obtain the fusion features.

[0016] The image target detection device according to some embodiments of the application, the channel attention mechanism layer comprises

[0017] a max-pooling layer, which performs a max-pooling operation on the concatenated features and outputs a max-pooling feature vector;

[0018] an average-pooling layer, which performs an average-pooling operation on the concatenated features and outputs an average-pooling feature vector;

[0019] a multi-layer perceptron, which inputs the max-pooling feature vector and outputs a first output vector, and inputs the average-pooling feature vector and outputs a second output vector;

[0020] a first addition fusion layer, which performs an addition operation on the first output vector and the second output vector to generate a fusion feature vector;

[0021] an activation function, which inputs the fusion feature vector and generates a channel attention map;

[0022] a second multiplication fusion layer, which multiplies the concatenated features and the channel attention map to obtain channel prior features.

[0023] According to the image target detection device of some embodiments of the present application, the spatial attention mechanism layer includes a depth separable convolution and a second addition fusion layer, wherein the depth separable convolution includes

[0024] a first depth convolution, an input of the first depth convolution includes the channel prior features, and the first depth convolution has at least two outputs;

[0025] a second depth convolution, the second depth convolution includes at least two, and an input of each second depth convolution is connected to an output of one first depth convolution;

[0026] The second addition fusion layer performs an addition operation on the features output by each second depth convolution to obtain spatial attention features.

[0027] According to the image target detection device of some embodiments of the present application, the neck network includes a first feature fusion layer and a second feature fusion layer; wherein the neck network further includes an efficient multi-scale attention mechanism module, the efficient multi-scale attention mechanism module is arranged between the first feature fusion layer and the detection head of the neck network and / or arranged between the first feature fusion layer and the second feature fusion layer of the neck network, and the fusion features output by the fusion module are input into the first feature fusion layer.

[0028] According to the image target detection device of some embodiments of the present application, the fusion module includes four modules arranged at different layers.

[0029] The first feature extraction layer of the first backbone network includes four modules arranged at different layers.

[0030] The first feature extraction layer of the second backbone network includes four modules arranged at different layers.

[0031] The first feature extraction layer of the first layer of the first backbone network comprises a first output and a second output; the visible light feature of the first output is input into the first feature extraction layer of the second layer of the first backbone network; and the visible light feature of the second output is input into the first input of the concatenation layer of the fusion module of the first layer.

[0032] The first feature extraction layer of the first layer of the second backbone network comprises a first output and a second output; the optical flow feature of the first output is input into the first feature extraction layer of the second layer of the second backbone network; and the optical flow feature of the second output is input into the second input of the concatenation layer of the fusion module of the first layer.

[0033] The first feature extraction layer of the second layer of the first backbone network comprises a first output and a second output; the visible light feature of the first output is input into the first feature extraction layer of the third layer of the first backbone network; and the visible light feature of the second output is input into the first input of the concatenation layer of the fusion module of the second layer.

[0034] The first feature extraction layer of the second layer of the second backbone network comprises a first output and a second output; the optical flow feature of the first output is input into the first feature extraction layer of the third layer of the second backbone network; and the optical flow feature of the second output is input into the second input of the concatenation layer of the fusion module of the second layer.

[0035] The first feature extraction layer of the third layer of the first backbone network comprises a first output and a second output; the visible light feature of the first output is input into the first feature extraction layer of the fourth layer of the first backbone network; and the visible light feature of the second output is input into the first input of the concatenation layer of the fusion module of the third layer.

[0036] The first feature extraction layer of the third layer of the second backbone network comprises a first output and a second output; the optical flow feature of the first output is input into the first feature extraction layer of the fourth layer of the second backbone network; and the optical flow feature of the second output is input into the second input of the concatenation layer of the fusion module of the third layer.

[0037] The first feature extraction layer of the fourth layer of the first backbone network comprises a second output; and the visible light feature of the second output is input into the first input of the concatenation layer of the fusion module of the fourth layer.

[0038] The first feature extraction layer of the fourth layer of the second backbone network comprises a second output; and the optical flow feature of the second output is input into the second input of the concatenation layer of the fusion module of the fourth layer.

[0039] The image target detection device according to some embodiments of the present application further comprises an SPPF module and a C2PSA module.

[0040] The neck network comprises a first feature fusion layer, a second feature fusion layer and a high-efficiency multi-scale attention mechanism module.

[0041] wherein the first feature fusion layer includes three layers arranged at different levels; the second feature fusion layer includes three layers arranged at different levels; and the high-efficiency multi-scale attention mechanism module includes two layers arranged at different levels.

[0042] wherein the input of the SPPF module is the fusion feature output by the fusion module of the fourth layer, the input of the C2PSA module is the output of the SPPF module, and the output of the C2PSA module includes a first output and a second output.

[0043] wherein the first feature fusion layer includes a first input and a second input, and includes a first output.

[0044] The first input of the first feature fusion layer of the third layer is the first output of the C2PSA module, and the second input is the fusion feature output by the fusion module of the third layer.

[0045] The first input of the first feature fusion layer of the second layer is the first output of the first feature fusion layer of the third layer, and the second input is the fusion feature output by the fusion module of the second layer. The first feature fusion layer of the second layer further includes a second output.

[0046] The first input of the first feature fusion layer of the first layer is the first output of the first feature fusion layer of the second layer, and the second input is the fusion feature output by the fusion module of the first layer.

[0047] wherein the input of the high-efficiency multi-scale attention mechanism module of the second layer is the second output of the first feature fusion layer of the second layer.

[0048] The input of the high-efficiency multi-scale attention mechanism module of the first layer is the first output of the first feature fusion layer of the first layer, and the output of the high-efficiency multi-scale attention mechanism module of the first layer includes a first output and a second output.

[0049] wherein the second feature fusion layer includes a first input and a first output.

[0050] The second feature fusion layer of the second layer further includes a second input and a second output, the first input is the second output of the high-efficiency multi-scale attention mechanism module of the first layer, and the second input is the output of the high-efficiency multi-scale attention mechanism module of the second layer.

[0051] The second feature fusion layer of the third layer further includes a second output, and the first input is the first output of the second feature fusion layer of the second layer.

[0052] The first input of the second feature fusion layer of the fourth layer is the first output of the second feature fusion layer of the third layer, and the second input is the second output of the C2PSA module.

[0053] According to the image target detection device of some embodiments of the present application, the detection head includes four, which are arranged at different layers; wherein the input of the detection head of the first layer is the first output of the high-efficiency multi-scale attention mechanism module of the first layer;

[0054] The input of the detection head of the second layer is the second output of the second feature fusion layer of the second layer;

[0055] The input of the detection head of the second layer is the second output of the second feature fusion layer of the third layer;

[0056] The input of the detection head of the fourth layer is the first output of the second feature fusion layer of the third layer.

[0057] According to the image target detection device of some embodiments of the present application, the fusion layer includes a first multiplication fusion layer, which performs multiplication operation on the channel prior feature and the spatial attention feature to obtain the fusion feature;

[0058] The image target detection device further includes a first convolutional layer and a first activation function, which are arranged between the splicing layer and the channel attention mechanism layer;

[0059] The image target detection device further includes a second convolutional layer, which is arranged between the second addition fusion layer and the first multiplication fusion layer;

[0060] The image target detection device further includes a third convolutional layer, which is arranged downstream of the first addition fusion layer;

[0061] The image target detection device further includes a fourth convolutional layer, which is arranged at the input end of the first backbone network;

[0062] The image target detection device further includes a fifth convolutional layer, which is arranged at the input end of the second backbone network.

[0063] According to the image target detection device of some embodiments of the present application, the visible light modality image under the far distance complex scene containing the fishery ship for aquaculture operation, the optical flow modality image is the optical flow modality image under the far distance complex scene containing the fishery ship for aquaculture operation, and the image target includes the fishery ship for aquaculture operation.

[0064] In the second aspect, according to the image target detection method of some embodiments of the present application, based on any one device, including

[0065] Input the visible light modality image into the first backbone network;

[0066] Input the optical flow modality image into the second backbone network;

[0067] The first backbone network extracts the visible light feature, and the second backbone network extracts the optical flow feature;

[0068] The fusion module splices the visible light feature and the optical flow feature to obtain a spliced feature, and acquires a channel prior feature of the spliced feature;

[0069] The fusion module fuses the channel prior feature and the spatial attention feature to obtain a fusion feature, and inputs the fusion feature into a neck network, and outputs an image target by a detection head.

[0070] Beneficial effects: the present application takes YOLOv11 as a basic model, and proposes a fishing boat target detection device working in a low-quality image environment.

[0071] (1) The fishing boat small target is confused with the reef feature, resulting in a decrease in recognition accuracy. The present application introduces an optical flow mode, uses dynamic information in the optical flow to distinguish from static reefs, and at the same time, proposes a new strategy of YOLO optical flow mode and visible light mode fusion CPF module, which improves the recognition effect.

[0072] (2) In order to compare the accurate detection of some distant dense small target scenes, the present application adds a P2 layer with larger resolution, and a detection head is used to detect small targets, which enhances the semantic expression ability of small targets.

[0073] (3) The present application adds an EMA attention mechanism in the neck network (Neck), improves the capture ability of the network to key features, and significantly improves the model performance, and shows stronger robustness in complex scenes.

[0074] (4) The present application introduces Focaler-IoU as a new loss function, focuses on difficult samples such as occluded small targets, optimizes the geometric parameters of the bounding box, reduces missed detection and mispositioning, and improves the detection recall rate.

[0075] (5) The present application combines the channel attention and the spatial attention of the visible light mode and the optical flow mode fusion module (CPF) to realize dynamic weight distribution and enhance key information. The channel attention will globally evaluate the channel features of the visible light (RGB) and the optical flow (OpticalFlow), suppress redundant channels and enhance channels sensitive to fishing boat detection; and the obtained channel prior map is input into a depth separable convolution module to obtain a spatial attention map, so that the spatial attention captures the complementary spatial information between modes (such as motion trajectories in the optical flow and texture details in the visible light) through multi-scale depth separable convolution, avoids noise interference caused by simple stacking of Concat, and this dynamic weight mechanism enables the model to focus more on key features, rather than treating all features equally like Concat, thereby improving the detection accuracy, especially for the breeding operation fishing boat target in a long-distance complex scene, which realizes great improvement in accuracy.

[0076] Additional aspects and advantages of the invention will be set forth in part in the description which follows, and in part will be obvious from the description, or may be learned by practice of the invention. Attached Figure Description

[0077] Figure 1 This is a flowchart of the method of the present invention.

[0078] Figure 2 Visible light scene images, including (a) Jinchan Port on Dachangshan Island, (b) Liufu Wharf Seaview House, (c) Li Changqun Seaview House, and (d) Hualong Seedling Farm.

[0079] Figure 3 Optical flow visualization, in which (a) Linyang International on Dachangshan Island, (b) Changshan Bridge, (c) Liufu Wharf sea-view room, and (d) Li Changqun sea-view room.

[0080] Figure 4 YOLOv11-OPNet network architecture diagram.

[0081] Figure 5 YOLOv11-OPNet detection flowchart.

[0082] Figure 6 Visible light mode and optical flow mode fusion module (CPF), where (a) is the structure diagram of the CPF module and (b) is the structure diagram of the ChannelAttention module.

[0083] Figure 7 EMA attention mechanism structure diagram.

[0084] Figure 8 A comparison of detection performance between YOLOv11 and YOLOv11-OPNet, where (a) shows the performance of YOLOv11 and (b) shows the detection performance of the model of the present invention.

[0085] Figure 9 YOLOv11-OPNet accuracy curve comparison chart.

[0086] Figure 10 Comparison of heatmap results for each model, including (a) RT-DETR, (b) YOLOv8, (c) YOLOv11, (d) YOLOv12, and (e) YOLOv11-OPNet. Detailed Implementation

[0087] Embodiments of the present application are described in detail below with reference to the accompanying drawings. The present application collects shore camera shooting data in different scenarios in the shore environment, conducts experiments for each scenario, and constructs a fishing boat dataset after sorting the data. The dataset is sourced from fixed-point camera real shots of more than ten fishing boat ports (Dalian Changhe Dachangshan Island Jinchan Port, Dalian Changhe Guangludao, etc.), and the source data format is.MP4 video files. After experimental analysis, 150-200 pictures are selected for each scene, and 2414 visible light image data in.PNG format are obtained. The obtained visible light image data is shown in FIG. 8. Figure 2 Optical flow is a technique for estimating the motion speed and direction of pixel points in an image sequence. The core idea is to infer the instantaneous motion vector field of the object in the two-dimensional imaging plane by analyzing the gray level changes of adjacent frames. Traditional optical flow methods (such as Horn-Schunck and Lucas-Kanade) are based on manually designed energy functions, and solve the optical flow by variational or difference methods, relying on the brightness constancy assumption (the brightness of the same target does not change in adjacent frames) and the small motion assumption (the pixel displacement of adjacent frames is small). For example, the HS algorithm introduces global smoothing constraints and iteratively solves the optical flow field through the Euler-Lagrange equation. Such methods are prone to failure in complex scenes, such as Figure 3 , and have high computational complexity. Therefore, the RAFT optical flow algorithm is used to extract optical flow images in the experiment. RAFT uses an end-to-end deep learning architecture to construct pixel-level global similarity using a 4D correlation volume, and adopts a high-resolution fixed flow field processing to avoid information loss. The adjacent two frames of the extracted visible light pictures are input into the RAFT model, and the visible light data is converted into optical flow modal data.

[0088] After obtaining the fishing boat images, the present application uses manual annotation to draw bounding boxes for the fishing boat targets in the images, ensuring accurate box selection. Since optical flow data is obtained by comparing two frames of visible light images, it corresponds one-to-one with visible light images, so only visible light data is labeled. The fishing boat label is unified as "ship".

[0089] In order to achieve better training effect of the network, the labeled fishing boat dataset is randomly divided according to the ratio of 8:1:1, and the number of training set, validation set and test set is 1928, 241 and 241 respectively. In order to make the evaluation results of the test set more reliable, in the data set division process, the test set and the training set come from different video segments, effectively avoiding the problem of data leakage, and improving the generalization ability of the model.

[0090] Embodiment: YOLOv11-OPNet network of the present application. Analysis of fishing boat data in aquaculture operations found that most scenarios are small targets in the distance, so YOLOv11 is selected as the basic model. The latest YOLOv12 focuses on improving the attention mechanism, and reduces the number of network layers. These changes result in a decrease in the detection effect of small targets in the distance, and the light flow-visible light dual mode is not well adapted, so it is not suitable. In addition, considering the real-time and efficiency of aquaculture, the model size used by the present application is YOLOv11L.

[0091] The core of YOLOv11 is the modular architecture reconstruction and dynamic training strategy, which realizes the breakthrough balance of precision and efficiency through C3k2 module, C2PSA attention mechanism and NMS-Free detection head. The model uses adaptive image enhancement technology, supports multi-modal input such as visible light, infrared and depth map, uniformly scales the input image to 640x640 resolution, and integrates dynamic Mosaic enhancement (four image splicing) and Mixup mixed enhancement, which improves the generalization ability in complex scenes.

[0092] The backbone network is based on the improved CSP Darknet framework, which introduces C3k2 module (allows custom convolution kernel size, enhances feature extraction flexibility) and C2PSA component (integrates position-sensitive attention mechanism, dynamically focuses on key areas through spatial weight), which reduces the parameter quantity by 22% compared with YOLOv8, and improves the small target detection accuracy (COCO dataset) by 5.2%. The neck uses an improved PAN architecture, which fuses multi-scale information through a bidirectional feature pyramid, adds a SPFF spatial pyramid fast pooling module to expand the receptive field, and reduces the false detection rate in dense scenes by 18%. The head uses a double-head design: in the training stage, activate a one-to-many prediction head (generate multiple candidate boxes to enhance the supervision signal), and in the inference stage, switch to a one-to-one prediction head to directly output the optimal result, eliminating the delay of traditional non-maximum suppression (NMS), and the end-to-end inference speed is improved by 30%.

[0093] Although the performance of YOLOv11 is already very good, it has good performance in most normal scenarios. However, when the imaging quality of the camera is low, the features of distant targets and small targets are not obvious, which reduces the recognition accuracy of YOLOv11 in real fishing environments. At the same time, in complex scenes with many reefs, and under low-resolution image conditions, the features of reefs and fishing boats are confused, and accurate identification is not possible.

[0094] To solve the above problems, the present application proposes a small target fishing boat recognition and detection network in a low-quality image environment, and the network structure is as follows Figures 4-6As shown, the first backbone network is configured to input the visible light modality image, and includes a first feature extraction layer, and an output of the first feature extraction layer includes visible light features; the second backbone network is configured to input the optical flow modality image, and includes a first feature extraction layer, and an output of the first feature extraction layer includes optical flow features; the fusion module is arranged between the first feature extraction layer of the first backbone network and the first feature extraction layer of the second backbone network, and is configured to input the visible light features and the optical flow features, and output fused features including channel attention features and spatial attention features; the neck network is configured to input the fused features; and the detection head is configured to input an output of the neck network, and output image targets.

[0095] The fusion module includes a concatenation layer configured to concatenate the visible light features and the optical flow features to obtain concatenated features; a channel attention mechanism layer configured to obtain channel prior features of the concatenated features; a spatial attention mechanism layer configured to obtain spatial attention features of the channel prior features; and a fusion layer configured to fuse the channel prior features and the spatial attention features to obtain the fused features.

[0096] The channel attention mechanism layer includes a max-pooling layer configured to perform a max-pooling operation on the concatenated features to output a max-pooling feature vector; an average-pooling layer configured to perform an average-pooling operation on the concatenated features to output an average-pooling feature vector; a multi-layer perceptron configured to input the max-pooling feature vector into the multi-layer perceptron to output a first output vector, and input the average-pooling feature vector into the multi-layer perceptron to output a second output vector; a first addition fusion layer configured to perform an addition operation on the first output vector and the second output vector to generate a fused feature vector; and an activation function configured to input the fused feature vector into the activation function to generate a channel attention mechanism map; and a second multiplication fusion layer configured to multiply the concatenated features and the channel attention mechanism map to obtain the channel prior features.

[0097] The spatial attention mechanism layer includes a depthwise separable convolution and a second addition fusion layer, wherein the depthwise separable convolution includes a first depthwise convolution, an input of the first depthwise convolution includes the channel prior features, and the first depthwise convolution has at least two outputs; and a second depthwise convolution, the second depthwise convolution includes at least two, and an input of each second depthwise convolution is connected to an output of one first depthwise convolution; wherein the second addition fusion layer performs an addition operation on features output by each second depthwise convolution to obtain the spatial attention features.

[0098] The neck network includes a first feature fusion layer and a second feature fusion layer; the neck network further includes an efficient multi-scale attention mechanism module, the efficient multi-scale attention mechanism module is arranged between the first feature fusion layer and the detection head of the neck network and / or arranged between the first feature fusion layer and the second feature fusion layer of the neck network, and the fused features output by the fusion module are input into the first feature fusion layer.

[0099] wherein the fusion module includes four being arranged at different layers; the first feature extraction layer of the first backbone network includes four being arranged at different layers; the first feature extraction layer of the first layer of the first backbone network includes a first output and a second output; the visible light feature of the first output is input into the first feature extraction layer of the second layer of the first backbone network; the visible light feature of the second output is input into the first input of the splicing layer of the fusion module of the first layer; the first feature extraction layer of the first layer of the second backbone network includes a first output and a second output; the optical flow feature of the first output is input into the first feature extraction layer of the second layer of the second backbone network; the optical flow feature of the second output is input into the second input of the splicing layer of the fusion module of the first layer.

[0100] wherein the first feature extraction layer of the second layer of the first backbone network includes a first output and a second output; the visible light feature of the first output is input into the first feature extraction layer of the third layer of the first backbone network; the visible light feature of the second output is input into the first input of the splicing layer of the fusion module of the second layer; the first feature extraction layer of the second layer of the second backbone network includes a first output and a second output; the optical flow feature of the first output is input into the first feature extraction layer of the third layer of the second backbone network; the optical flow feature of the second output is input into the second input of the splicing layer of the fusion module of the second layer.

[0101] wherein the first feature extraction layer of the third layer of the first backbone network includes a first output and a second output; the visible light feature of the first output is input into the first feature extraction layer of the fourth layer of the first backbone network; the visible light feature of the second output is input into the first input of the splicing layer of the fusion module of the third layer; the first feature extraction layer of the third layer of the second backbone network includes a first output and a second output; the optical flow feature of the first output is input into the first feature extraction layer of the fourth layer of the second backbone network; the optical flow feature of the second output is input into the second input of the splicing layer of the fusion module of the third layer.

[0102] wherein the first feature extraction layer of the fourth layer of the first backbone network includes a second output; the visible light feature of the second output is input into the first input of the splicing layer of the fusion module of the fourth layer; the first feature extraction layer of the fourth layer of the second backbone network includes a second output; the optical flow feature of the second output is input into the second input of the splicing layer of the fusion module of the fourth layer.

[0103] wherein the network further includes an SPPF module and a C2PSA module; wherein the neck network includes a first feature fusion layer, a second feature fusion layer and a high-efficiency multi-scale attention mechanism module; wherein the first feature fusion layer includes three being arranged at different layers; the second feature fusion layer includes three being arranged at different layers; the high-efficiency multi-scale attention mechanism module includes two being arranged at different layers.

[0104] The input of the SPPF module is the fusion feature output by the fusion module of the fourth layer, the input of the C2PSA module is the output of the SPPF module, and the output of the C2PSA module includes a first output and a second output.

[0105] The first feature fusion layer includes a first input and a second input, and includes a first output; the first input of the first feature fusion layer of the third layer is the first output of the C2PSA module, and the second input is the fusion feature output by the fusion module of the third layer; the first input of the first feature fusion layer of the second layer is the first output of the first feature fusion layer of the third layer, and the second input is the fusion feature output by the fusion module of the second layer; the first feature fusion layer of the second layer further includes a second output; the first input of the first feature fusion layer of the first layer is the first output of the first feature fusion layer of the second layer, and the second input is the fusion feature output by the fusion module of the first layer.

[0106] The input of the efficient multi-scale attention mechanism module of the second layer is the second output of the first feature fusion layer of the second layer; the input of the efficient multi-scale attention mechanism module of the first layer is the first output of the first feature fusion layer of the first layer, and the output of the efficient multi-scale attention mechanism module of the first layer includes a first output and a second output; the second feature fusion layer includes a first input and a first output; the second feature fusion layer of the second layer further includes a second input and a second output, the first input is the second output of the efficient multi-scale attention mechanism module of the first layer, and the second input is the output of the efficient multi-scale attention mechanism module of the second layer; the second feature fusion layer of the third layer further includes a second output, and the first input is the first output of the second feature fusion layer of the second layer; the first input of the second feature fusion layer of the fourth layer is the first output of the second feature fusion layer of the third layer, and the second input is the second output of the C2PSA module.

[0107] The detection head includes four, which are arranged at different layers; the input of the detection head of the first layer is the first output of the efficient multi-scale attention mechanism module of the first layer; the input of the detection head of the second layer is the second output of the second feature fusion layer of the second layer; the input of the detection head of the second layer is the second output of the second feature fusion layer of the third layer; the input of the detection head of the fourth layer is the first output of the second feature fusion layer of the third layer.

[0108] The fusion layer includes a first multiplication fusion layer, and the channel prior feature and the spatial attention feature are subjected to a multiplication operation to obtain a fusion feature.

[0109] The first feature extraction layer includes Conv and C3K2. The first feature fusion layer includes Up, Concat and C3K2.

[0110] Specifically, the application first introduces an optical flow branch on the basis of YOLOv11, extracts the features of visible light and optical flow, then performs feature fusion of the visible light mode and the optical flow mode through the proposed CPF module at the P2-P5 layer, combines the dynamic information of the optical flow mode, solves the problem of confusion between fishing boats and reefs in a complex environment, at the same time, the newly added P2 neck network (Neck) feature extraction layer and the P2 level detection head make the detection of small targets more accurate. The position of the feature map with higher resolution increases the EMA spatial attention mechanism, so that the network can dynamically pay attention to small targets that are difficult to detect in the distance. Finally, the Focaler-IoU loss function is introduced to solve the problem that the traditional cross-entropy loss and the basic IoU loss are difficult to distinguish between difficult and easy samples, so that the model is dominated by simple samples, and the overall performance of fishing boat detection is improved.

[0111] The dual-modal structure of the application is improved. The traditional YOLO fishing boat detection algorithm is a visible light single-modal input. The visible light image has rich color and texture details, but when the camera imaging quality is poor and the target distance is far, the visible light feature is very poor. At this time, the optical flow branch is introduced to supplement the motion features of the fishing boat, and the data of the two modes are used for information complementation to improve the accuracy of fishing boat detection. Figure 4 YOLOv11-OPNet network structure diagram.

[0112] Figure 5 YOLOv11-OPNet detection flow chart, as Figure 5As shown, first, the input visible light video is divided into two branches, one directly enters the first backbone network of YOLOv11-OPNet for visible light feature extraction, and the other branch enters the RAFT data preprocessing model to convert the visible light video into optical flow modal information, and then enters the first backbone network of YOLOv11-OPNet for optical flow feature extraction.

[0113] Subsequently, the features extracted by the two modalities are sent to the visible light modal and optical flow modal fusion module (CPF) of the application for feature fusion. The visible light modal and optical flow modal fusion module (CPF) is referred to as the "fusion module" hereinafter.

[0114] Finally, the fused information enters the neck network (Neck) for finer feature extraction, and the detection is completed through the detection head (Detect).

[0115] In the fishery farming scene, the fishboat image obtained by the fishboat target is far away from the shore camera and the sea surface fishboat, or is affected by extreme weather, resulting in missing pixels of the fishboat target, and the information of the fishboat target is similar to that of the reef target, which directly leads to the reduction of the accuracy of fishboat target detection. Therefore, it is crucial to fully utilize the motion information in the optical flow modal for fishboat small target detection. By selecting fusion in the backbone network, more information can be extracted to achieve better fusion effect.

[0116] The traditional Concat method only increases the number of channels by splicing, and cannot effectively utilize the spatial correlation of cross-modalities. Meanwhile, in the real fishboat detection task, the visible light modal may cause the target to be blurred due to changes in light, and the optical flow modal may produce noise due to sea wave interference.

[0117] In summary, the visible light modal and optical flow modal fusion module (CPF) combining channel attention and spatial attention is designed in the application, which realizes dynamic weight distribution and enhances key information. The channel attention globally evaluates the channel features of the visible light (RGB) and the optical flow (OpticalFlow), suppresses redundant channels, and enhances channels sensitive to fishboat detection; the spatial attention captures the complementary spatial information between modalities (such as the motion trajectory in the optical flow and the texture details in the visible light) through multi-scale depth separable convolution, avoids noise interference caused by simple superposition of Concat, and this dynamic weight mechanism enables the model to focus more on key features, rather than treating all features equally like Concat, thereby improving the detection accuracy.

[0118] Figure 6 For the visible light modal and optical flow modal fusion module (CPF), as Figure 6(a) shows that, first, the visible light features and the optical flow features are concatenated, and then input into the channel attention mechanism after 1x1 Conv operation and activation function for channel matching. The structure of the channel attention mechanism is as shown in Figure 6 (b) shows that the concatenated visible light features and optical flow features are subjected to average pooling and maximum pooling operations to summarize the spatial information in the feature map, and then the processed information is input into a shared multi-layer perceptron (MLP). The output of the shared multi-layer perceptron is summed, passed through an activation function, and multiplied by the concatenated features to obtain the channel prior map (channel prior feature), as shown in formula 1:

[0119] (1)

[0120] In the formula, represents the channel prior feature, F represents the input concatenated feature, represents an activation function.

[0121] As shown in Figure 6 (a), the obtained channel prior map is input into a depth separable convolution module to obtain a spatial attention map (spatial attention feature), as shown in formula 2:

[0122] (2)

[0123] wherein, represents the spatial attention feature, represents a first depth convolution, represents a second depth convolution, represents the i-th branch, and i ∈ {1, 2, 3}. represents a 1x1 Conv operation for matching. As shown in

[0124] (a), finally, the channel prior map and the spatial attention map are multiplied to obtain the fused feature, as shown in formula 3: Figure 6

[0125] (3)

[0126] The position of the visible light modality and optical flow modality fusion module (CPF) in the YOLOv11-OPNet network structure is as shown in Figure 4 , which is a Cross Feture Enhence module.

[0127] ​In real-world fishing vessel operations, small fishing boats in the distance often occupy a tiny pixel area in the image, and traditional single-scale feature extraction easily overlooks details. The Efficient Multi-Scale Attention (EMA) mechanism is a novel technique in computer vision that enhances the model's feature extraction and representation capabilities in multi-scale scenes. The EMA attention mechanism captures features at different scales through a parallel multi-branch structure. A 1×1 convolutional branch combined with adaptive average pooling extracts global semantic information, assisting in the localization of small targets in complex backgrounds. A 3×3 convolutional branch expands the receptive field through grouped convolutions, enhancing sensitivity to local features such as fishing boat edges and textures. This multi-scale fusion strategy allows the model to simultaneously focus on the overall position and local details of small targets, improving detection accuracy in fishing vessel monitoring datasets, significantly enhancing the semantic representation of small targets, and improving the model's robustness. Therefore, the EMA attention mechanism is introduced. The EMA attention mechanism structure is as follows: Figure 7 As shown, the position of EMA in the YOLOv11-OPNet network structure is as follows: Figure 4 As shown, this is the Attention module.

[0128] The present invention introduces the Focaler-IoU loss function. In aquaculture scenarios, most fishing boat targets in images are relatively small compared to the background, and in some scenarios, the fishing boat targets are too dense, making it difficult for YOLOv11's cross-entropy loss to focus on difficult samples. Therefore, to address the above problems, the present invention introduces Focaler-IoU as a loss function to enhance the network's learning ability for difficult samples, thereby improving detection accuracy and robustness.

[0129] Focaler-IoU is a loss function that improves object detection bounding box regression by dynamically adjusting sample attention and incorporating geometric optimization terms. Its core formula is shown in Equation 3-4. and It is a threshold parameter used to dynamically adjust the model's attention to samples of different difficulty. The lower threshold controls the attention given to low-quality samples (e.g.) (Ignore extremely low IoU samples when IoU = 0.00). The upper limit threshold is used to filter the boundary of high-quality samples (e.g.) =0.95 focuses on difficult samples with IoU ≥ 95%.

[0130] (3)

[0131] (4)

[0132] In the formula, This indicates the improved method for calculating the intersection-union ratio (IU). denotes the intersection over union, which is an indicator of the overlap of two bounding boxes; denotes the loss function using GIoU, denotes the final loss function calculation method.

[0133] Model training and evaluation of the application:

[0134] In order to ensure the accuracy and reliability of experimental data, the training and testing of the small target fishing boat detection network are run in the same environment. The experimental environment is shown in Table 1. The batch size is 16, the training period is 300 rounds, and the initial learning rate is 0.0001.

[0135] Table 1 Experimental environment

[0136]

[0137] Evaluation index: detection accuracy and high reasoning speed play an important role in real-time fishing boat target detection. In order to comprehensively evaluate the performance of the proposed method, the confusion matrix is used as an evaluation tool, including the following key indicators: true positive (TP), true negative (TN), false positive (FP) and false negative (FN).

[0138] In terms of network detection accuracy, precision rate (P), recall rate (R) and mean average precision (mAP@0.5) are selected as evaluation indicators. In addition, fishing boat detection is a task with specific requirements, and compared with recall rate, the accuracy of fishing boat should be the primary indicator for evaluating the detection accuracy of the model. The calculation formula is shown in formula 5 to formula 8:

[0139] (5)

[0140] (6)

[0141] (7)

[0142] (8)

[0143] Experimental results and analysis:

[0144] (1) Ablation experiment

[0145] To verify the effectiveness of the improved part of the model, an ablation experiment is designed. The experimental scheme is as follows: the optical flow mode, the P2 feature map detection layer, the EMA attention mechanism module and the Focaler-IoU loss function are added to the baseline model in turn, and a plurality of different configuration model variants are constructed, which are named YOLOv11-OP, YOLOv11-OE, YOLOv11-OF, YOLOv11-OPE, YOLOv11-OPF respectively. The suffix represents the module combination contained in the model, and the experimental results are shown in Table 2, and the results show that the above improvements have a certain performance improvement on the baseline model. The final network model reaches 87.1% and 79.6% in accuracy and fishing boat recall rate, which is improved by 7.5% and 14.3% compared with the baseline model.

[0146] In the real fishing scene, most fishing boats are small targets in the distance and too dense, which seriously affects the accuracy of detection. By adding the P2 extraction layer with larger feature map resolution, the detection performance of small targets is enhanced, and the accuracy and recall rate are improved by 3.7% and 9.3% respectively. Due to the limited focal length of the camera device and the complex environment by the sea, the visible light image quality is low, the effective information carried by the target is missing, and the static target such as the reef cannot be well distinguished from the fishing boat target. The optical flow mode is introduced, and the motion information in the optical flow mode is extracted to assist the model to distinguish other interference objects in the scene, which effectively improves the accuracy by 2.4%. Then, in order to solve the problem that the small fishing boat target in the distance often occupies a very small pixel area in the image, and the traditional single scale feature extraction easily ignores the details of the fishing boat target, the EMA attention mechanism module is introduced, which better assists the positioning of small targets in complex background and significantly improves the recall rate. Finally, Focaler-IoU is used as the loss function, which dynamically adjusts the attention degree of the model to different samples, enhances the learning of the network to difficult samples, and finally improves the accuracy and recall rate by 1.5% and 1.6% respectively. Compared with the baseline model, the final network has significant advantages in the accuracy and speed of fishing boat detection. Figure 8 and Figure 9 The intuitive comparison of the detection effect and performance of the method and the baseline model is shown, which further verifies the effectiveness of the improved strategy and the good synergistic effect between them.

[0147] Table 2 Ablation experiment results

[0148]

[0149] (2) Model loss function comparison test

[0150] To further verify the effectiveness of the selected loss function on the detection accuracy of the model, a comparative experiment of different loss functions is designed, and YOLOv11 is added to the light flow mode, P2 detection layer and EMA attention mechanism, which is recorded as Base, and SlideLoss, GIoU, DIoU, CIoU and Focaler-IoU are added to Base as loss functions, respectively. The experimental results are shown in Table 3, which clearly shows the influence of different loss functions on the accuracy (P), recall rate (R) and average precision (mAP@0.5) of the model.

[0151] SlideLoss solves the sample imbalance problem by dividing samples, improves the accuracy of the model, and performs well in classification tasks. However, in the fishing vessel detection task of fishing operations, the sample is relatively single and there is no imbalance problem, so the improvement is limited. GIoU (Generalized Intersection over Union) introduces the minimum closure rectangle formula based on IoU, solves the problem that the gradient is 0 when there is no overlap in IoU, and enhances the detection accuracy of non-overlapping regions, but it is not suitable for fishing vessel scenes with overlapping conditions, so the improvement is limited. DIoU (Distance-IoU) uses a center point distance penalty method to solve the problem of reduced accuracy when detecting horizontally or vertically arranged frames with IoU, but it is very unstable in small target or dense scenes, which directly leads to a 0.6% reduction in model accuracy. CIoU (Complete IoU) adds a length-width ratio penalty term based on DIoU, which improves accuracy, but is ineffective when the length-width ratio is the same, such as the scene of a large frame fitting a small frame when fishing vessels overlap. Focaler-IoU combines the advantages of SlideLoss, GIoU and CIoU, enabling the model to focus on difficult samples (such as occlusion scenes) while improving the detection ability of small targets. Finally, compared with other loss functions, the accuracy of the model is improved by 1.8%.

[0152] Table 3 Comparison of detection performance of different models

[0153]

[0154] (3) Comparative experiment

[0155] To further verify the effectiveness and advancement of the network model of the application in fishing vessel detection, a comparative experiment is conducted by selecting the currently higher detection effect model in this field, including RT-DETR, YOLOv8, YOLOv12 and the baseline model YOLOv11. The experimental results are shown in Table 4, and compared with other algorithms, the method proposed in the application achieves the best effect in comprehensive performance.

[0156] Table 4 Comparison of detection performance of different models

[0157]

[0158] Compared with RT-DETR, the method of the application significantly improves the detection accuracy and recall rate by 12.8% and 8.7%. This is because in the small target detection task, although RT-DETR adopts multi-scale feature fusion, the global attention mechanism of the Transformer weakens the local details, resulting in a much lower small target detection accuracy than some CNN models (such as the PAN-FPN structure of YOLOv8). In addition, in comparison with the other three YOLO series algorithms, the method of the application also shows advancement, achieving a significant improvement of 7.7%, 7.5%, and 7.7% in fishing boat accuracy, which is mainly due to the low image quality, which causes the feature information carried by the small target in the distance to be not obvious, and the original model of YOLO has only three detection heads, which weakens the detection ability of small target individuals and affects the overall detection performance of the model. Figure 10 The heat map directly reflects the attention degree of the region to the model. Through the color contrast of the heat map, it is shown that YOLOv11-OPnet has high accuracy in the detection of small target fishing boats.

[0159] In the task of detecting farming fishing boats in a complex scene at a distance, the application proposes a detection method YOLOv11-OPNet (YOLOv11-OpticalFlowNet) for fishing boats in a situation where the features of the target at a distance are not obvious, aiming to accurately identify fishing boats with poor image quality in a fishing environment. By designing to introduce an optical flow mode in the backbone network and combining the dynamic information in the optical flow, the problem of confusion between fishing boats and reefs in a complex environment is solved, significantly enhancing the adaptability of the model in a complex environment. Secondly, in order to better combine the channel prior and spatial attention mechanism, a CPF cross-modal fusion module is proposed to solve the combination of information of the visible light mode and the optical flow mode, so that more information can be extracted to achieve better fusion effect. In the neck network, an EMA attention mechanism is proposed to improve the network's ability to capture key features and significantly improve the model's performance, showing stronger robustness in complex scenes. Finally, Focaler-IoU is used as a new loss function to solve the problem that the traditional cross-entropy loss and the basic IoU loss cannot distinguish between difficult and easy samples, leading to the model being dominated by simple samples, and the overall performance of fishing boat detection is improved. Experimental results show that the improved model improves the accuracy, recall rate, and average precision by 7.5%, 14.3%, and 11.6%, respectively.

[0160] Based on the above embodiments, the embodiments of the present application further provide a computer program, which, when running on a computer, causes the computer to execute the method provided in the above embodiments.

[0161] Based on the above embodiments, the embodiments of the present application further provide a computer storage medium, which stores a computer program, and the computer program, when executed by a computer, causes the computer to execute the method provided in the above embodiments.

[0162] The storage medium can be any available medium that can be accessed by a computer. By way of example, and not limitation, such computer-readable media can comprise RAM, ROM, EEPROM, CD-ROM or other optical disk storage, magnetic disk storage or other magnetic storage devices, or any other medium that can be used to carry or store desired program code in the form of instructions or data structures and that can be accessed by a computer.

[0163] Based on the above embodiments, the embodiments of the present application further provide a chip for reading a computer program stored in a memory, and implementing the method provided in the above embodiments.

[0164] Based on the above embodiments, the embodiments of the present application provide a computer program product, which, when running on an electronic device, implements the method provided in the above embodiments.

[0165] Those skilled in the art should understand that the embodiments of the present application can be provided as a method, a system, or a computer program product. Therefore, the present application can take the form of an entirely hardware embodiment, an entirely software embodiment, or an embodiment combining software and hardware aspects. Moreover, the present application can take the form of a computer program product implemented on one or more computer-usable storage media (including, but not limited to, disk storage, CD-ROM, optical storage, etc.) containing computer usable program code.

[0166] The present application is described with reference to the flowcharts and / or block diagrams of the methods, devices (systems), and computer program products according to the present application. It should be understood that each flow and / or block in the flowcharts and / or block diagrams, and combinations of flows and / or blocks in the flowcharts and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing apparatus to produce a machine, so that the instructions executed by the processor of the computer or other programmable data processing apparatus produce a device that implements the functions specified in the flowcharts and / or block diagrams. Figure 1 The functions specified in a flow or multiple flows and / or blocks Figure 1 The functions specified in a flow or multiple flows and / or blocks

[0167] These computer program instructions can also be stored in a computer readable memory that can direct a computer or other programmable data processing apparatus to function in a particular manner, such that the instructions stored in the computer readable memory produce an article of manufacture including instructions which implement the flow Figure 1 The flow or flows and / or blocks Figure 1 The flow or flows and / or blocks

[0168] These computer program instructions can also be loaded onto a computer or other programmable data processing apparatus to cause a series of operational steps to be performed on the computer or other programmable apparatus to produce a computer implemented process such that the instructions that execute on the computer or other programmable apparatus provide steps for implementing the flow Figure 1 The flow or flows and / or blocks Figure 1 The flow or flows and / or blocks

[0169] Obviously, numerous modifications and variations of the present application are possible in light of the above teachings. It is therefore to be understood that within the scope of the appended claims and their equivalents, the application can be practiced otherwise than as specifically described.

Claims

1. An image target detection apparatus characterized by comprising: Comprising a first backbone network for inputting a visible light modality image, the first backbone network comprising a first feature extraction layer, an output of the first feature extraction layer comprising visible light features; a second backbone network for inputting an optical flow modality image, the second backbone network comprising a first feature extraction layer, an output of the first feature extraction layer comprising optical flow features; a fusion module arranged between the first feature extraction layer of the first backbone network and the first feature extraction layer of the second backbone network, inputting the visible light features and the optical flow features, outputting fusion features comprising channel attention features and spatial attention features; a neck network, an input of the neck network comprising the fusion features; wherein the neck network comprises a first feature fusion layer and a second feature fusion layer; wherein the neck network further comprises a high-efficiency multi-scale attention mechanism module, the high-efficiency multi-scale attention mechanism module being arranged between the first feature fusion layer and a detection head of the neck network and / or being arranged between the first feature fusion layer and the second feature fusion layer of the neck network, the fusion features output by the fusion module being input to the first feature fusion layer; a detection head, an input of the detection head comprising an output of the neck network, an output of the detection head comprising image targets. wherein the fusion module comprises a concatenation layer for concatenating the visible light features and the optical flow features to obtain concatenated features; a channel attention mechanism layer for obtaining channel prior features of the concatenated features; a spatial attention mechanism layer for obtaining spatial attention features of the channel prior features; a fusion layer for fusing the channel prior features and the spatial attention features to obtain the fusion features.

2. The image target detection apparatus according to claim 1, characterized by The channel attention mechanism layer comprises a max-pooling layer for performing a max-pooling operation on the concatenated features to output a max-pooling feature vector; an average-pooling layer for performing an average-pooling operation on the concatenated features to output an average-pooling feature vector; a multi-layer perceptron, the max-pooling feature vector being input to the multi-layer perceptron to output a first output vector, the average-pooling feature vector being input to the multi-layer perceptron to output a second output vector; a first addition fusion layer for performing an addition operation on the first output vector and the second output vector to generate a fusion feature vector; an activation function, the fusion feature vector being input to the activation function to generate a channel attention mechanism map; a second multiplication fusion layer for multiplying the concatenated features and the channel attention mechanism map to obtain the channel prior features.

3. The image target detection apparatus according to claim 2, characterized by The spatial attention mechanism layer comprises a depthwise separable convolution and a second addition fusion layer, wherein the depthwise separable convolution comprises a first depthwise convolution, an input of the first depthwise convolution comprising the channel prior features, the first depthwise convolution having at least two outputs; a plurality of second depthwise convolutions, an input of each second depthwise convolution being connected to an output of one first depthwise convolution; wherein the second addition fusion layer performs an addition operation on the features output by each second depthwise convolution to obtain the spatial attention features.

4. The image target detection apparatus according to any one of claims 1 to 3, characterized by wherein the fusion module comprises four modules arranged at different layers; the first feature extraction layer of the first backbone network comprises four modules arranged at different layers; the first feature extraction layer of the second backbone network comprises four modules arranged at different layers; The first feature extraction layer of the first layer of the first backbone network comprises a first output and a second output; the visible light feature of the first output is input into the first feature extraction layer of the second layer of the first backbone network; and the visible light feature of the second output is input into the first input of the concatenation layer of the fusion module of the first layer. The first feature extraction layer of the first layer of the second backbone network comprises a first output and a second output; the optical flow feature of the first output is input into the first feature extraction layer of the second layer of the second backbone network; and the optical flow feature of the second output is input into the second input of the concatenation layer of the fusion module of the first layer. The first feature extraction layer of the second layer of the first backbone network comprises a first output and a second output; the visible light feature of the first output is input into the first feature extraction layer of the third layer of the first backbone network; and the visible light feature of the second output is input into the first input of the concatenation layer of the fusion module of the second layer. The first feature extraction layer of the second layer of the second backbone network comprises a first output and a second output; the optical flow feature of the first output is input into the first feature extraction layer of the third layer of the second backbone network; and the optical flow feature of the second output is input into the second input of the concatenation layer of the fusion module of the second layer. The first feature extraction layer of the third layer of the first backbone network comprises a first output and a second output; the visible light feature of the first output is input into the first feature extraction layer of the fourth layer of the first backbone network; and the visible light feature of the second output is input into the first input of the concatenation layer of the fusion module of the third layer. The first feature extraction layer of the third layer of the second backbone network comprises a first output and a second output; the optical flow feature of the first output is input into the first feature extraction layer of the fourth layer of the second backbone network; and the optical flow feature of the second output is input into the second input of the concatenation layer of the fusion module of the third layer. The first feature extraction layer of the fourth layer of the first backbone network comprises a second output; and the visible light feature of the second output is input into the first input of the concatenation layer of the fusion module of the fourth layer. The first feature extraction layer of the fourth layer of the second backbone network comprises a second output; and the optical flow feature of the second output is input into the second input of the concatenation layer of the fusion module of the fourth layer.

5. The image target detection apparatus according to claim 4, characterized by The SPPF module and the C2PSA module are further included. The neck network comprises a first feature fusion layer, a second feature fusion layer and a high-efficiency multi-scale attention mechanism module. The first feature fusion layer comprises three layers arranged at different levels; the second feature fusion layer comprises three layers arranged at different levels; and the high-efficiency multi-scale attention mechanism module comprises two layers arranged at different levels. The input of the SPPF module is the fusion feature output by the fusion module of the fourth layer, the input of the C2PSA module is the output of the SPPF module, and the output of the C2PSA module comprises a first output and a second output. The first feature fusion layer comprises a first input and a second input, and comprises a first output. The first input of the first feature fusion layer of the third layer is the first output of the C2PSA module, and the second input is the fusion feature output by the fusion module of the third layer. The first input of the first feature fusion layer of the second layer is the first output of the first feature fusion layer of the third layer, and the second input is the fusion feature output by the fusion module of the second layer; the first feature fusion layer of the second layer further comprises a second output; The first input of the first feature fusion layer of the first layer is the first output of the first feature fusion layer of the second layer, and the second input is the fusion feature output by the fusion module of the first layer; The input of the efficient multi-scale attention mechanism module of the second layer is the second output of the first feature fusion layer of the second layer; The input of the efficient multi-scale attention mechanism module of the first layer is the first output of the first feature fusion layer of the first layer, and the output of the efficient multi-scale attention mechanism module of the first layer comprises a first output and a second output; The second feature fusion layer comprises a first input and a first output; The second feature fusion layer of the second layer further comprises a second input and a second output, the first input is the second output of the efficient multi-scale attention mechanism module of the first layer, and the second input is the output of the efficient multi-scale attention mechanism module of the second layer; The second feature fusion layer of the third layer further comprises a second output, and the first input is the first output of the second feature fusion layer of the second layer; The first input of the second feature fusion layer of the fourth layer is the first output of the second feature fusion layer of the third layer, and the second input is the second output of the C2PSA module.

6. The image target detection apparatus according to claim 5, characterized by The detection head comprises four, which are arranged at different layers; wherein the input of the detection head of the first layer is the first output of the efficient multi-scale attention mechanism module of the first layer; The input of the detection head of the second layer is the second output of the second feature fusion layer of the second layer; The input of the detection head of the second layer is the second output of the second feature fusion layer of the third layer; The input of the detection head of the fourth layer is the first output of the second feature fusion layer of the third layer.

7. The image target detection apparatus according to claim 1, characterized by The fusion layer comprises a first multiplication fusion layer, which performs multiplication operation on the channel prior feature and the spatial attention feature to obtain the fusion feature; The image target detection device further comprises a first convolution layer and a first activation function, which are arranged between the concatenation layer and the channel attention mechanism layer; The image target detection device further comprises a second convolution layer, which is arranged between the second addition fusion layer and the first multiplication fusion layer; The image target detection device further comprises a third convolution layer, which is arranged downstream of the first addition fusion layer; The image target detection device further comprises a fourth convolution layer, which is arranged at the input end of the first backbone network; The image target detection device further comprises a fifth convolution layer, which is arranged at the input end of the second backbone network.

8. The image target detection apparatus according to claim 1, characterized by The visible light modality image is a visible light modality image containing a fish farming boat in a long-distance complex scene, the optical flow modality image is an optical flow modality image containing a fish farming boat in a long-distance complex scene, and the image target comprises a fish farming boat.

9. An image target detection method, characterized by, Based on the device of any one of claims 1-8, comprising inputting the visible light modality image into the first backbone network; inputting the optical flow modality image into the second backbone network; the first backbone network extracts visible light features, and the second backbone network extracts optical flow features; the fusion module concatenates the visible light features and the optical flow features to obtain concatenated features, acquires channel prior features of the concatenated features, and acquires spatial attention features according to the channel prior features; The fusion module fuses the channel prior feature and the spatial attention feature to obtain a fusion feature, and inputs the fusion feature into a neck network, and outputs an image target by a detection head.

Images