Fishing net detection model training method, device, computer equipment and storage medium

Abstract

The present application relates to a training method, device, computer equipment and storage medium for a fishing net detection model. The method comprises: obtaining a sample image; marking the fishing net in the sample image to obtain a first image; preprocessing the first image to obtain a second image; performing a wavelet transform on the second image to obtain a plurality of spectral feature graphs; inputting the second image and the plurality of spectral feature graphs into an SSD detection framework for training to obtain a fishing net detection model. Among them, since the spectral feature graph obtained after the wavelet transform has a large number of fishing net features, the spectral feature graph and the second image are added into the SSD detection framework for fusion training, which increases the complexity of information encoding during training, thereby improving the recognition rate of the fishing net detection model.

Description

Technical Field

[0001] The present application relates to the field of image processing technology, and in particular to a training method, device, computer equipment and storage medium for a fishing net detection model. Background Art

[0002] With the diversified development of fisheries, there are a large number of fishing nets in the offshore waters. The fishing nets are scattered in the sea and can easily get entangled in propellers, posing a great potential threat to the safe navigation of ships.

[0003] The traditional fishing net detection method is to perform edge detection on the fishing nets in the aerial images through the Canny edge detection operator after acquiring the aerial images. However, since the fishing nets in aerial images are usually small, it is difficult to accurately detect the fishing nets by edge detection. Summary of the invention

[0004] Based on this, it is necessary to provide a training method, device, computer equipment and storage medium for a fishing net detection model that can improve the accuracy of fishing net detection in response to the above technical problems.

[0005] In a first aspect, the present application provides a method for training a fishing net detection model. The method for training a fishing net detection model comprises:

[0006] Get a sample image;

[0007] Annotate the fishing net in the sample image to obtain a first image;

[0008] Preprocessing the first image to obtain a second image;

[0009] Performing wavelet transformation on the second image to obtain a plurality of spectral characteristic graphs;

[0010] The second image and multiple spectral feature maps are input into the SSD detection framework for training to obtain a fishing net detection model.

[0011] In one embodiment, the second image is subjected to wavelet transformation to obtain a plurality of spectral characteristic graphs, including:

[0012] The second image is decomposed into three layers using Haar wavelet to obtain a first spectral feature map, a second spectral feature map and a third spectral feature map in sequence.

[0013] In one embodiment, the backbone network of the SSD detection framework is a VGG network, and the second image and the plurality of spectral feature maps are input into the SSD detection framework for training, including:

[0014] Input the second image into the VGG network for convolution feature extraction and pooling to obtain a first feature map;

[0015] Merging the first characteristic graph and the first spectral characteristic graph to obtain a second characteristic graph;

[0016] Perform convolution feature extraction and pooling on the second feature map to obtain a third feature map;

[0017] Merging the third characteristic map and the second spectral characteristic map to obtain a fourth characteristic map;

[0018] Perform convolution feature extraction and pooling on the fourth feature map to obtain a fifth feature map;

[0019] The fifth characteristic map and the third spectral characteristic map are combined to obtain a sixth characteristic map;

[0020] The sixth feature map is used for training.

[0021] In one embodiment, after marking the fishing net in the sample image to obtain the first image, the method further includes:

[0022] determining color distribution information of the first image;

[0023] The first image is normalized according to the color distribution information.

[0024] In one embodiment, preprocessing the first image to obtain the second image includes:

[0025] Performing a convolution operation on the first image using a mean template;

[0026] The data obtained after the convolution operation is logarithmically transformed to obtain a second image.

[0027] In one embodiment, the training method of the fishing net detection model further includes:

[0028] Acquire image data containing a fishing net;

[0029] The image data is input into the fishing net detection model to obtain the fishing net information output by the fishing net detection model.

[0030] In a second aspect, the present application also provides a training device for a fishing net detection model. The training device for a fishing net detection model comprises:

[0031] A sample acquisition module, used for acquiring a sample image;

[0032] An image annotation module, used for annotating the fishing net in the sample image to obtain a first image;

[0033] A preprocessing module, used for preprocessing the first image to obtain a second image;

[0034] A wavelet transform module, used for performing wavelet transform on the second image to obtain a spectral characteristic graph;

[0035] The model training module is used to input the second image and the spectral feature map into the SSD detection framework for training to obtain a fishing net detection model.

[0036] In a third aspect, the present application further provides a computer device. The computer device includes a memory and a processor, the memory stores a computer program, and the processor implements the following steps when executing the computer program:

[0037] Get a sample image;

[0038] Annotate the fishing net in the sample image to obtain a first image;

[0039] Preprocessing the first image to obtain a second image;

[0040] Performing wavelet transformation on the second image to obtain a plurality of spectral characteristic graphs;

[0041] The second image and multiple spectral feature maps are input into the SSD detection framework for training to obtain a fishing net detection model.

[0042] In a fourth aspect, the present application further provides a computer-readable storage medium. The computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the following steps are implemented:

[0043] Get a sample image;

[0044] Annotate the fishing net in the sample image to obtain a first image;

[0045] Preprocessing the first image to obtain a second image;

[0046] Performing wavelet transformation on the second image to obtain a plurality of spectral characteristic graphs;

[0047] The second image and multiple spectral feature maps are input into the SSD detection framework for training to obtain a fishing net detection model.

[0048] In a fifth aspect, the present application further provides a computer program product. The computer program product includes a computer program, and when the computer program is executed by a processor, the following steps are implemented:

[0049] Get a sample image;

[0050] Annotate the fishing net in the sample image to obtain a first image;

[0051] Preprocessing the first image to obtain a second image;

[0052] Performing wavelet transformation on the second image to obtain a plurality of spectral characteristic graphs;

[0053] The second image and multiple spectral feature maps are input into the SSD detection framework for training to obtain a fishing net detection model.

[0054] The training method, device, computer equipment, storage medium and computer program product of the above-mentioned fishing net detection model obtains a sample image; annotates the fishing net in the sample image to obtain a first image; preprocesses the first image to obtain a second image; performs a wavelet transform on the second image to obtain multiple spectral feature maps; inputs the second image and the multiple spectral feature maps into the SSD detection framework for training to obtain a fishing net detection model. Among them, since the spectral feature map obtained after the wavelet transform has more fishing net features, the spectral feature map and the second image are added to the SSD detection framework for fusion training, which increases the complexity of information encoding during training, thereby improving the recognition rate of the fishing net detection model. BRIEF DESCRIPTION OF THE DRAWINGS

[0055] Figure 1 A schematic diagram of a flow chart of a method for training a fishing net detection model in one embodiment;

[0056] Figure 2 is a schematic diagram of performing the first layer of Haar wavelet decomposition on an image in one embodiment;

[0057] Figure 3 A schematic diagram of VGG network training in one embodiment;

[0058] Figure 4 is a flow chart of a method for training a fishing net detection model in another embodiment;

[0059] Figure 5 is a structural block diagram of a training device for a fishing net detection model in one embodiment;

[0060] Figure 6 FIG. 4 is a diagram showing the internal structure of a computer device in one embodiment. DETAILED DESCRIPTION

[0061] In order to make the purpose, technical solution and advantages of the present application more clearly understood, the present application is further described in detail below in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present application and are not used to limit the present application.

[0062] Figure 1 FIG. 1 is a flow chart of a method for training a fishing net detection model in an embodiment. Figure 1As shown, a training method for a fishing net detection model is provided. This embodiment takes the method applied to a terminal as an example. It can be understood that the method can also be applied to a server, and can also be applied to a system including a terminal and a server, and is implemented through the interaction between the terminal and the server. In this embodiment, the method can include the following steps:

[0063] Step 102: Acquire a sample image.

[0064] The sample image refers to the original image required for training the model. In this embodiment, the sample image can be a patrol image collected by a drone during a historical ship shipping patrol mission.

[0065] Specifically, an image acquisition device can be set on the drone, and the image acquisition device can be used to collect patrol images. Among the patrol images collected by the image acquisition device, some images are shot at a long distance, and the effective information of the fishing net is less, which is not suitable for the later identification of the fishing net area. Some images are shot at a short distance, and the effective information of the fishing net is more detailed, which can be used for the high-precision identification and application of the fishing net area in the later stage. Therefore, the sample image can also be an image obtained by screening the patrol images according to the amount of effective information of the fishing net.

[0066] The terminal can obtain sample images in a variety of ways. For example, when the image acquisition device on the drone captures the sample image on the flight route of the drone, the terminal can directly obtain the sample image from the image acquisition device, or obtain the sample image from the server after the drone sends the sample image to the server.

[0067] It should be noted that the terminal may be, but is not limited to, devices such as drones, aircraft, personal computers, laptops, smart phones, tablet computers, and IoT devices.

[0068] Step 104: annotate the fishing net in the sample image to obtain a first image.

[0069] In this embodiment, the model needs to be trained through a supervised learning strategy, so the fishing nets in the sample images need to be labeled to provide labeled information during the training process.

[0070] It should be understood that image object detection usually includes two aspects: first, locating the position where the object appears in the image, and second, determining the type of the object. In order for the trained fishing net detection model to detect fishing nets, it is necessary to annotate the fishing nets in the sample image, for example, annotate the type information and position information of the fishing nets. Among them, the position information of the fishing net can be, but is not limited to, the image coordinates of the upper left point of the fishing net area, the width of the fishing net area, the height of the fishing net area, and other information. The type information of the fishing net is the type of the fishing net area. Specifically, the type of the fishing net area can be determined according to the shape characteristics of the fishing net area, such as a circular fishing net, a linear fishing net, and a square fishing net.

[0071] Of course, after marking the types of the fishing net areas, the number of fishing net areas can also be counted to obtain the corresponding relationship between the type information and the number of different fishing net areas, as shown in Table 1 below:

[0072] Table 1

[0073] type Round fishing net Linear fishing net Square fishing net total quantity 1168 1128 1471 3767

[0074] Step 106: preprocess the first image to obtain a second image.

[0075] Among them, the main purpose of preprocessing is to eliminate irrelevant information in the first image, restore useful real information, enhance the detectability of relevant information and simplify the data to the maximum extent.

[0076] Specifically, the first image may be preprocessed in a variety of ways to obtain the second image. For example, the first image may be scaled, convolved, statistically analyzed, enhanced, and the like.

[0077] For the scaling and convolution processing, a convolution template may be used to implement the convolution operation of the first image, and the first image may be scaled to a multiple of 32 resolution, such as 416*416 resolution, during the convolution operation. In one example, the mathematical expression of the convolution operation is as follows:

[0078] I(x,y)=S(x,y)*T(x,y);

[0079] Among them, S is the input original image, I is the output result of the convolution operation, T is the convolution template, and x and y are the horizontal and vertical coordinates of the pixel point in the image coordinate system, respectively.

[0080] For statistical analysis processing, the size of the annotation box and the color distribution information of the first image can be statistically analyzed, and the statistical analysis results can be used as the prior values ​​of the hyperparameters of the fishing net detection model. Better model parameter values ​​can help improve the convergence speed of the model during training and accelerate the search for the optimal solution of the model.

[0081] Regarding enhancement processing, it should be understood that image enhancement is to add some information or transform data to the original image through certain means, selectively highlight the features of interest in the image or cover up some unnecessary features in the image, so that the image matches the visual response characteristics. Specifically, a linear function is used to transform the grayscale value of the image to achieve image contrast enhancement; a nonlinear function can be used to transform the grayscale value of the image to achieve widening of the image area of ​​interest and compression of the background area of ​​no interest, etc.

[0082] Step 108: Perform wavelet transform on the second image to obtain a plurality of spectral feature graphs.

[0083] Among them, wavelet transform is a time scale analysis method for signals. It is also a time-frequency localization analysis method with a fixed window size, variable shape, and changeable time window and frequency window. It has the characteristics of multi-resolution analysis and the ability to characterize the local characteristics of signals in both time and frequency domains.

[0084] The sampling step size of the wavelet transform in the time domain for different frequencies is adjustable. At low frequencies, the time resolution of the wavelet transform is low and the frequency resolution is high; at high frequencies, the time resolution of the wavelet transform is high and the frequency resolution is low. This feature is consistent with the characteristics of the second image that the low-frequency signal changes slowly and the high-frequency signal changes quickly. When using the wavelet transform to process the second image, an appropriate wavelet function can be selected to decompose the second image to obtain multiple spectral feature maps.

[0085] Specifically, the second image may be decomposed using Cmor wavelet, Fbsp wavelet, Shan wavelet or Haar wavelet, etc., to extract spectral features from the second image.

[0086] Among them, Cmor wavelet is the abbreviation of complex wavelet Morlet, which has good time-frequency analysis accuracy. The time-frequency and frequency domain analysis capabilities of Cmor wavelet are different. Generally speaking, the frequency domain analysis capability of this wavelet is good, but the time domain analysis capability is not outstanding. Haar wavelet is a wavelet transform based on the Haar wavelet function, which has good frequency domain analysis capability and time domain analysis capability.

[0087] It should be understood that when different wavelet functions are used for wavelet transform, the number of spectral characteristic graphs obtained is not the same, and the amount of fishing net information contained in the spectral characteristic graphs is also different.

[0088] Step 110: input the second image and the multiple spectral feature maps into the SSD detection framework for training to obtain a fishing net detection model.

[0089] Among them, the SSD (full name Single Shot Multibox Detector) detection framework is a single-stage and multi-layer target detection model. It generates PriorBox, classifies, and regresses on multiple feature maps of different depths, and finally integrates them to obtain the final prediction result. The SSD detection framework has a significant speed advantage over the Faster R-CNN model, and a significant accuracy advantage over the YOLO model.

[0090] It should be understood that the image-based object detection method can be divided into two major types: 1. Object detection method based on feature engineering; 2. Regression method based on deep neural network. Among them, the object detection method based on feature engineering includes three steps: region selection, feature extraction, and classification. The algorithm has a large amount of computation, the strategy effect of region selection is poor, and the degree of adaptability of manually extracted features is low, resulting in high time and space complexity of the algorithm, and the maximum object detection accuracy can only reach 35.1%. However, since functions of any form can be fitted by increasing the number of nodes in the neural network, the object detection algorithm based on the neural network can automatically learn the features of the target to be detected in the image by utilizing the powerful fitting ability of the neural network, and input the features extracted by the neural network into the fully connected network, and perform position regression and classification on it, and the object detection accuracy can reach 92%. Under the mutual trade-off between detection accuracy and processing speed, the SSD detection framework is used for model training in this embodiment.

[0091] Specifically, the backbone network in the SSD detection framework may be VGG (full name Visual Geometry Group), ResNet or MobileNets, etc., which is not limited in this embodiment.

[0092] Among them, the VGG network is a deep convolutional neural network, which mainly explores the relationship between the depth of the convolutional neural network and its performance. By repeatedly stacking 3*3 small convolution kernels and 2*2 maximum pooling layers, a 16-19-layer deep convolutional neural network was successfully built, which has the characteristics of good error rate and generalization ability.

[0093] ResNet is a residual network. This series of networks is widely used in fields such as target classification. As part of the classic neural network backbone for computer vision tasks, it is a typical convolutional network. ResNet can be divided into ResNet18, ResNet50, etc. according to the number of layers. The ResNet50 network first performs a convolution operation on the input, then passes through 4 residual modules, and finally performs a full connection operation for classification tasks. ResNet50 contains 50 convolution operations.

[0094] MobileNets is a set of mobile-first computer vision models that are based on a streamlined structural design and use deep separable convolutions to build lightweight deep neural networks. MobileNets introduces two hyperparameters that can freely select the scale of the model according to the constraints of different tasks. Through these two simple global hyperparameters, MobileNets achieves a good balance between speed and accuracy. As a lightweight deep network, MobileNets is universal and effective in computer vision tasks such as recognition, positioning, and detection.

[0095] In a specific implementation, the second image can be used as an input image, and multiple spectral feature maps can be input into different stage layers of the backbone network of the SSD detection framework for training to obtain a fishing net detection model.

[0096] In the training method of the above-mentioned fishing net detection model, a sample image is obtained; the fishing net in the sample image is marked to obtain a first image; the first image is preprocessed to obtain a second image; the second image is subjected to wavelet transformation to obtain multiple spectral feature maps; the second image and the multiple spectral feature maps are input into the SSD detection framework for training to obtain a fishing net detection model. Among them, since the spectral feature map obtained after the wavelet transformation has more fishing net features, the spectral feature map and the second image are added into the SSD detection framework for fusion training, which increases the complexity of information encoding during training, thereby improving the recognition rate of the fishing net detection model.

[0097] In one embodiment, the step of performing wavelet transform on the second image to obtain multiple spectral feature maps includes: performing three-layer decomposition on the second image using Haar wavelet to sequentially obtain a first spectral feature map, a second spectral feature map, and a third spectral feature map.

[0098] Among them, the haar wavelet is a wavelet transform based on the haar wavelet function. For the haar transform of a two-dimensional image, low-pass and high-pass filtering are performed in the horizontal and vertical directions, thereby generating low-frequency features and high-frequency features. The high-frequency features are similar to the edges in the image, which are areas with drastic changes in the image, such as the fishing net area.

[0099] Figure 2 FIG. 1 is a schematic diagram of performing the first-layer Haar wavelet decomposition on an image in one embodiment. Figure 2As shown, when the first layer of haar wavelet decomposition is performed on the second image, since the second image is usually a visible light image, that is, a three-channel image composed of three primary colors (Red, Green, Blue, RGB), the haar wavelet is used to decompose the three channels of R, G, and B respectively, and each channel will generate 4 sub-images. The resolution of the sub-image becomes 0.5 times that of the original, and a total of 12 sub-images (3*4=12) are generated for the three channels. The 12 sub-images are stacked to form a first spectral feature map of the 12 channels.

[0100] Specifically, the image can be decomposed by Haar wavelet based on the following formula:

[0101]

[0102] Among them, t is the independent variable, j is a set of non-negative integers, and K represents the translation of the Haar wavelet scaling function.

[0103] After the first spectral feature map, the second spectral feature map and the third spectral feature map are extracted using the haar wavelet, these feature maps can be input into different stage layers of the backbone network of the SSD detection framework.

[0104] This embodiment uses Haar wavelet to decompose the second image into three layers to obtain the first spectral characteristic map, the second spectral characteristic map and the third spectral characteristic map in sequence, so as to obtain more fishing net area information and improve the diversity and comprehensiveness of the input data of the model.

[0105] In one embodiment, the backbone network of the SSD detection framework is a VGG network, and the step of inputting the second image and multiple spectral feature maps into the SSD detection framework for training includes: inputting the second image into the VGG network for convolutional feature extraction and pooling to obtain a first feature map; merging the first feature map and the first spectral feature map to obtain a second feature map; performing convolutional feature extraction and pooling on the second feature map to obtain a third feature map; merging the third feature map and the second spectral feature map to obtain a fourth feature map; performing convolutional feature extraction and pooling on the fourth feature map to obtain a fifth feature map; merging the fifth feature map and the third spectral feature map to obtain a sixth feature map; and using the sixth feature map for training.

[0106] Figure 3 Schematic diagram of VGG network training in one embodiment. Figure 3As shown in the figure, assuming that the resolution of the second image input to the VGG network is 224*224*3, after the first layer of convolution feature extraction and pooling, the first feature map with a resolution of 112*112*128 is obtained. At the same time, the second image is input to the haar wavelet transform for the first layer of feature extraction. Here, each channel of the RGB image is processed separately, and each channel is transformed by the wavelet transform to obtain a feature map with a resolution of 112*112*4. After the three channels are processed, a first spectral feature map with a resolution of 112*112*12 is obtained. Finally, the first feature map of the VGG network is merged with the first spectral feature map obtained by the haar wavelet transform to obtain a second feature map with a resolution of 112*112*140.

[0107] The second feature map is subjected to the second layer of convolution feature extraction and pooling to obtain the third feature map with a resolution of 56*56*256. At the same time, the second image is input into the haar wavelet transform for the second layer of feature extraction to obtain the second spectral feature map with a resolution of 56*56*12. Finally, the third feature map of the VGG network is merged with the second spectral feature map obtained by the haar wavelet transform to obtain the fourth feature map with a resolution of 56*56*268.

[0108] The fourth feature map is subjected to the third layer of convolution feature extraction and pooling to obtain the fifth feature map with a resolution of 28*28*512. At the same time, the second image is input into the haar wavelet transform for the third layer of feature extraction to obtain the third spectral feature map with a resolution of 28*28*12. Finally, the fifth feature map of the VGG network is merged with the third spectral feature map obtained by the haar wavelet transform to obtain the sixth feature map with a resolution of 28*28*524.

[0109] It should be understood that the CNN network is a global transformation that is completely transformed in the time domain or completely transformed in the frequency domain. It cannot characterize the time-frequency properties of the signal, but the time-frequency regional properties are precisely the most essential and critical characteristics of non-stationary signals. The wavelet transform has the characteristics of multi-resolution analysis and has the ability to characterize the local characteristics of the signal in both the time and frequency domains. This embodiment uses the haar wavelet to extract the spectral features of the second image, obtains three spectral feature maps, and then inputs these spectral feature maps into different stage layers of the VGG network, so that more fishing net area information is input into the VGG network for training, thereby improving the accuracy of the model.

[0110] In one embodiment, after the fishing nets in the sample image are marked to obtain the first image, the training method of the fishing net detection model may further include the following steps: determining the color distribution information of the first image; and normalizing the first image according to the color distribution information.

[0111] Specifically, the color distribution information of the first image can be determined by color histogram, color moment, color set, color aggregation vector or color correlation diagram. After statistical analysis of the color distribution information, the statistical mean and standard deviation can be calculated, and then the first image is standardized according to a preset formula to convert the first image into a standard normal distribution image. The preset formula can be referred to as follows:

[0112]

[0113] Wherein, x is the color distribution information of the first image, y is the color distribution information of the standardized image, which obeys the standard normal distribution, μ is the statistical mean, and σ is the standard deviation.

[0114] After the first image is standardized, the processed image can be used as an input image and input into the SSD detection framework for training.

[0115] This embodiment determines the color distribution information of the first image and standardizes the first image according to the color distribution information so that the data input to the SSD detection framework conforms to the standard normal distribution law, which can accelerate the convergence of the model and improve the speed of model training.

[0116] In one embodiment, the step of preprocessing the first image to obtain the second image may include the following steps: performing a convolution operation on the first image using a mean template; and performing a logarithmic transformation on the data obtained after the convolution operation to obtain the second image.

[0117] Among them, the mean template is a type of convolution template, which refers to averaging the pixel values ​​in the area covered by the template, that is, average pooling. By using the mean template for convolution operation, image downsampling can be achieved and the image can be made smoother.

[0118] By using logarithmic transformation, the low gray value area in the image can be expanded and its high gray value part can be compressed to emphasize the low gray value area of ​​the image, thereby achieving the purpose of enhancing the image. In an example, the formula of logarithmic transformation can be:

[0119]

[0120] Among them, r is the control coefficient, which controls the intensity of the change of the enhancement curve. The larger r is, the steeper the curve is, and the more obvious the stretching of the low gray value area is. c is the scaling coefficient, and v is the pixel value of the input image.

[0121] Of course, in a specific implementation, after logarithmic transformation of the data obtained after the convolution operation, the data with too high or too low grayscale can be corrected by gamma transformation to enhance the contrast of the data. In addition, piecewise linear transformation can be used to enhance the contrast of each part of the data.

[0122] By inputting the preprocessed second image into the SSD object detection framework, the backbone network of the SSD object detection framework extracts features from the second image and uses a fixed frame to generate regression regions, utilizing multi-layer feature information, which greatly improves the model training speed and detection accuracy.

[0123] This embodiment uses a mean template to perform a convolution operation on the first image and performs a logarithmic transformation on the data obtained after the convolution operation, so that the second image obtained after preprocessing has lower noise and is smoother, and reduces the impact of light changes on fishing net detection, thereby improving the training speed and detection accuracy.

[0124] In one embodiment, after the fishing net detection model is trained, image data containing the fishing net may be obtained; the image data is input into the fishing net detection model to obtain the fishing net information output by the fishing net detection model.

[0125] Image data refers to the collection of grayscale values ​​of each pixel expressed in numerical values. It has the characteristics of non-contact, long-distance, rich information, and low hardware cost, and is widely used in various industries. Fishing net information can include the type information and location information of the fishing net area.

[0126] Specifically, the image data containing the fishing net can be obtained in a variety of ways. In one example, an image acquisition device (e.g., a visible light imaging device) can be installed on the drone, and the image acquisition device takes image data containing the fishing net on the flight route of the drone, and stores the image data in the terminal. In another example, an image acquisition device and an image transmission device can be installed on the drone, and the image acquisition device takes image data containing the fishing net on the flight route of the drone, and the image transmission device transmits the image data to the server, and the terminal obtains the image data from the server.

[0127] In a specific implementation, the terminal may be an embedded device installed on a drone. After the fishing net detection model is generated, the fishing net detection model may be burned into the embedded device. The image acquisition device captures image data containing fishing nets on the flight route of the drone and stores the image data in the embedded device. The embedded device receives the image data, obtains the image data containing fishing nets, and uses the fishing net detection model to predict the image data to obtain fishing net information. In this way, the drone can analyze the category of the fishing net area in the current scene and the location of the fishing net in the scene in real time during the shipping inspection process.

[0128] Of course, after detecting the fishing net information, the distance from the fishing net to the ship can also be determined based on the fishing net information. When the distance between the fishing net and the ship reaches a preset value, the ship is judged to be in danger and an early warning message is output to the dangerous ship.

[0129] It should be understood that the application of computer vision technology in ship and shipping production inspection management is mainly concentrated in the following aspects: 1. By using drones for investigation, evidence collection and emergency response, the rapid response and timeliness of investigation can be guaranteed to identify and track hit-and-run ships to prevent them from escaping; 2. Use drones equipped with cameras to check ship overloads; 3. Use drones equipped with thermal imaging to check the number of crew members on ships; 4. Use airborne video and photography equipment to record and preserve evidence for investigation and processing. 5. Use drones to organize and guide ship traffic order. In other words, at present, drones usually only need to obtain data and use the data as evidence, and no further detection and analysis of the data is carried out, let alone using it in dangerous warnings for ship navigation.

[0130] This embodiment obtains image data containing fishing nets and inputs the image data into a fishing net detection model to obtain fishing net information output by the fishing net detection model, thereby realizing the application of the fishing net detection model in ship navigation and expanding the application scenarios of the fishing net detection model.

[0131] Figure 4 FIG. 1 is a flow chart of a method for training a fishing net detection model in another embodiment. Figure 4 As shown, the training method of the fishing net detection model includes the following steps:

[0132] Step 401, obtaining a sample image;

[0133] Step 402, marking the fishing net in the sample image to obtain a first image;

[0134] Step 403, preprocessing the first image to obtain a second image;

[0135] Step 404, performing wavelet transform on the second image to obtain a plurality of spectral characteristic graphs;

[0136] Step 405, inputting the second image and the plurality of spectral feature maps into the SSD detection framework for training to obtain a fishing net detection model;

[0137] Step 406, obtaining image data including the fishing net;

[0138] Step 407, input the image data into the fishing net detection model to obtain the fishing net information output by the fishing net detection model.

[0139] It should be understood that, although the steps in the flowcharts involved in the above embodiments are displayed in sequence according to the indication of the arrows, these steps are not necessarily executed in sequence according to the order indicated by the arrows. Unless there is a clear explanation in this article, the execution of these steps is not strictly limited in order, and these steps can be executed in other orders. Moreover, at least a part of the steps in the flowcharts involved in the above embodiments may include multiple steps or multiple stages, and these steps or stages are not necessarily executed at the same time, but can be executed at different times, and the execution order of these steps or stages is not necessarily carried out in sequence, but can be executed in turn or alternately with other steps or at least a part of the steps or stages in other steps.

[0140] Based on the same inventive concept, the embodiment of the present application also provides a training device for a fishing net detection model for implementing the training method for the fishing net detection model involved above. The implementation scheme for solving the problem provided by the device is similar to the implementation scheme recorded in the above method, so the specific limitations in the embodiments of the training device for one or more fishing net detection models provided below can refer to the limitations of the training method for the fishing net detection model above, and will not be repeated here.

[0141] In one embodiment, Figure 5 As shown, a training device for a fishing net detection model is provided, comprising: a sample acquisition module 502, an image annotation module 504, a preprocessing module 506, a wavelet transformation module 508 and a model training module 510, wherein:

[0142] The sample acquisition module 502 is used to acquire a sample image;

[0143] An image annotation module 504 is used to annotate the fishing net in the sample image to obtain a first image;

[0144] A preprocessing module 506, configured to preprocess the first image to obtain a second image;

[0145] A wavelet transform module 508, used to perform wavelet transform on the second image to obtain a spectral characteristic graph;

[0146] The model training module 510 is used to input the second image and the spectral feature map into the SSD detection framework for training to obtain a fishing net detection model.

[0147] In one embodiment, the wavelet transform module 508 is further used to decompose the second image into three layers using Haar wavelet to obtain a first spectral feature map, a second spectral feature map and a third spectral feature map in sequence.

[0148] In one embodiment, the model training module 510 is also used to input the second image into the VGG network for convolutional feature extraction and pooling to obtain a first feature map; merge the first feature map and the first spectral feature map to obtain a second feature map; perform convolutional feature extraction and pooling on the second feature map to obtain a third feature map; merge the third feature map and the second spectral feature map to obtain a fourth feature map; perform convolutional feature extraction and pooling on the fourth feature map to obtain a fifth feature map; merge the fifth feature map and the third spectral feature map to obtain a sixth feature map; and use the sixth feature map for training.

[0149] In one embodiment, the preprocessing module 506 is further configured to determine color distribution information of the first image; and perform standardization processing on the first image according to the color distribution information.

[0150] In one embodiment, the preprocessing module 506 is further used to perform a convolution operation on the first image using a mean template; and perform a logarithmic transformation on the data obtained after the convolution operation to obtain a second image.

[0151] In one embodiment, the training device of the fishing net detection model also includes a model prediction module (not shown) for acquiring image data containing a fishing net; inputting the image data into the fishing net detection model to obtain the fishing net information output by the fishing net detection model.

[0152] Each module in the training device of the above-mentioned fishing net detection model can be implemented in whole or in part by software, hardware and their combination. Each of the above-mentioned modules can be embedded in or independent of the processor in the computer device in the form of hardware, or can be stored in the memory of the computer device in the form of software, so that the processor can call and execute the operations corresponding to each of the above modules.

[0153] In one embodiment, a computer device is provided. The computer device may be a terminal, and its internal structure diagram may be as follows: Figure 6As shown. The computer device includes a processor, a memory, a communication interface, a display screen and an input device connected through a system bus. Among them, the processor of the computer device is used to provide computing and control capabilities. The memory of the computer device includes a non-volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system and a computer program. The internal memory provides an environment for the operation of the operating system and the computer program in the non-volatile storage medium. The communication interface of the computer device is used to communicate with an external terminal in a wired or wireless manner, and the wireless manner can be achieved through WIFI, a mobile cellular network, NFC (near field communication) or other technologies. When the computer program is executed by the processor, a training method for a fishing net detection model is implemented. The display screen of the computer device can be a liquid crystal display screen or an electronic ink display screen, and the input device of the computer device can be a touch layer covering the display screen, or a key, trackball or touchpad set on the computer device housing, or an external keyboard, touchpad or mouse, etc.

[0154] Those skilled in the art will understand that Figure 6 The structure shown in the figure is only a block diagram of a part of the structure related to the solution of the present application, and does not constitute a limitation on the computer device to which the solution of the present application is applied. The specific computer device may include more or fewer components than those shown in the figure, or combine certain components, or have a different arrangement of components.

[0155] In one embodiment, a computer device is provided, including a memory and a processor, wherein a computer program is stored in the memory, and when the processor executes the computer program, the following steps are implemented:

[0156] Get a sample image;

[0157] Annotate the fishing net in the sample image to obtain a first image;

[0158] Preprocessing the first image to obtain a second image;

[0159] Performing wavelet transformation on the second image to obtain a plurality of spectral characteristic graphs;

[0160] The second image and multiple spectral feature maps are input into the SSD detection framework for training to obtain a fishing net detection model.

[0161] In one embodiment, when the processor executes the computer program, the processor further implements the following steps:

[0162] The second image is decomposed into three layers using Haar wavelet to obtain a first spectral feature map, a second spectral feature map and a third spectral feature map in sequence.

[0163] In one embodiment, when the processor executes the computer program, the processor further implements the following steps:

[0164] The second image is input into the VGG network for convolution feature extraction and pooling to obtain a first feature map; the first feature map and the first spectral feature map are merged to obtain a second feature map; the second feature map is subjected to convolution feature extraction and pooling to obtain a third feature map; the third feature map and the second spectral feature map are merged to obtain a fourth feature map; the fourth feature map is subjected to convolution feature extraction and pooling to obtain a fifth feature map; the fifth feature map and the third spectral feature map are merged to obtain a sixth feature map; and the sixth feature map is used for training.

[0165] In one embodiment, when the processor executes the computer program, the processor further implements the following steps:

[0166] Determine color distribution information of the first image; and perform standardization processing on the first image according to the color distribution information.

[0167] In one embodiment, when the processor executes the computer program, the processor further implements the following steps:

[0168] A convolution operation is performed on the first image using a mean template; and a logarithmic transformation is performed on the data obtained after the convolution operation to obtain a second image.

[0169] In one embodiment, when the processor executes the computer program, the processor further implements the following steps:

[0170] Obtain image data containing a fishing net; input the image data into a fishing net detection model to obtain fishing net information output by the fishing net detection model.

[0171] In one embodiment, a computer readable storage medium is provided, on which a computer program is stored, and when the computer program is executed by a processor, the following steps are implemented:

[0172] Get a sample image;

[0173] Annotate the fishing net in the sample image to obtain a first image;

[0174] Preprocessing the first image to obtain a second image;

[0175] Performing wavelet transformation on the second image to obtain a plurality of spectral characteristic graphs;

[0176] The second image and multiple spectral feature maps are input into the SSD detection framework for training to obtain a fishing net detection model.

[0177] In one embodiment, when the computer program is executed by a processor, the following steps are also implemented:

[0178] The second image is decomposed into three layers using Haar wavelet to obtain a first spectral feature map, a second spectral feature map and a third spectral feature map in sequence.

[0179] In one embodiment, when the computer program is executed by a processor, the following steps are also implemented:

[0180] The second image is input into the VGG network for convolution feature extraction and pooling to obtain a first feature map; the first feature map and the first spectral feature map are merged to obtain a second feature map; the second feature map is subjected to convolution feature extraction and pooling to obtain a third feature map; the third feature map and the second spectral feature map are merged to obtain a fourth feature map; the fourth feature map is subjected to convolution feature extraction and pooling to obtain a fifth feature map; the fifth feature map and the third spectral feature map are merged to obtain a sixth feature map; and the sixth feature map is used for training.

[0181] In one embodiment, when the computer program is executed by a processor, the following steps are also implemented:

[0182] Determine color distribution information of the first image; and perform standardization processing on the first image according to the color distribution information.

[0183] In one embodiment, when the computer program is executed by a processor, the following steps are also implemented:

[0184] A convolution operation is performed on the first image using a mean template; and a logarithmic transformation is performed on the data obtained after the convolution operation to obtain a second image.

[0185] In one embodiment, when the computer program is executed by a processor, the following steps are also implemented:

[0186] Obtain image data containing a fishing net; input the image data into a fishing net detection model to obtain fishing net information output by the fishing net detection model.

[0187] In one embodiment, a computer program product is provided, comprising a computer program, which, when executed by a processor, implements the following steps:

[0188] Get a sample image;

[0189] Annotate the fishing net in the sample image to obtain a first image;

[0190] Preprocessing the first image to obtain a second image;

[0191] Performing wavelet transformation on the second image to obtain a plurality of spectral characteristic graphs;

[0192] The second image and multiple spectral feature maps are input into the SSD detection framework for training to obtain a fishing net detection model.

[0193] In one embodiment, when the computer program is executed by a processor, the following steps are also implemented:

[0194] The second image is decomposed into three layers using Haar wavelet to obtain a first spectral feature map, a second spectral feature map and a third spectral feature map in sequence.

[0195] In one embodiment, when the computer program is executed by a processor, the following steps are also implemented:

[0196] The second image is input into the VGG network for convolution feature extraction and pooling to obtain a first feature map; the first feature map and the first spectral feature map are merged to obtain a second feature map; the second feature map is subjected to convolution feature extraction and pooling to obtain a third feature map; the third feature map and the second spectral feature map are merged to obtain a fourth feature map; the fourth feature map is subjected to convolution feature extraction and pooling to obtain a fifth feature map; the fifth feature map and the third spectral feature map are merged to obtain a sixth feature map; and the sixth feature map is used for training.

[0197] In one embodiment, when the computer program is executed by a processor, the following steps are also implemented:

[0198] Determine color distribution information of the first image; and perform standardization processing on the first image according to the color distribution information.

[0199] In one embodiment, when the computer program is executed by a processor, the following steps are also implemented:

[0200] A convolution operation is performed on the first image using a mean template; and a logarithmic transformation is performed on the data obtained after the convolution operation to obtain a second image.

[0201] In one embodiment, when the computer program is executed by a processor, the following steps are also implemented:

[0202] Obtain image data containing a fishing net; input the image data into a fishing net detection model to obtain fishing net information output by the fishing net detection model.

[0203] It should be noted that the user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for analysis, stored data, displayed data, etc.) involved in this application are all information and data authorized by the user or fully authorized by all parties.

[0204] Those of ordinary skill in the art can understand that all or part of the processes in the above-mentioned embodiment methods can be completed by instructing the relevant hardware through a computer program, and the computer program can be stored in a non-volatile computer-readable storage medium. When the computer program is executed, it can include the processes of the embodiments of the above-mentioned methods. Among them, any reference to the memory, database or other medium used in the embodiments provided in the present application can include at least one of non-volatile and volatile memory. Non-volatile memory can include read-only memory (ROM), magnetic tape, floppy disk, flash memory, optical memory, high-density embedded non-volatile memory, resistive random access memory (ReRAM), magnetic random access memory (MRAM), ferroelectric random access memory (FRAM), phase change memory (PCM), graphene memory, etc. Volatile memory can include random access memory (RAM) or external cache memory, etc. As an illustration and not limitation, RAM can be in various forms, such as static random access memory (SRAM) or dynamic random access memory (DRAM). The database involved in each embodiment provided in this application may include at least one of a relational database and a non-relational database. Non-relational databases may include distributed databases based on blockchains, etc., but are not limited thereto. The processors involved in each embodiment provided in this application may be general-purpose processors, central processing units, graphics processors, digital signal processors, programmable logic devices, data processing logic devices based on quantum computing, etc., but are not limited thereto.

[0205] The technical features of the above embodiments may be combined arbitrarily. To make the description concise, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0206] The above-described embodiments only express several implementation methods of the present application, and the descriptions thereof are relatively specific and detailed, but they cannot be understood as limiting the scope of the present application. It should be pointed out that, for a person of ordinary skill in the art, several variations and improvements can be made without departing from the concept of the present application, and these all belong to the protection scope of the present application. Therefore, the protection scope of the present application shall be subject to the attached claims.

Claims

1. A method for training a fishing net detection model, characterized in that: The method comprises: Get a sample image; Annotating the fishing net in the sample image to obtain a first image; Preprocessing the first image to obtain a second image; Decomposing the second image into three layers by using Haar wavelet to obtain a first spectral characteristic graph, a second spectral characteristic graph and a third spectral characteristic graph in sequence; Inputting the second image and the multiple spectral feature maps into an SSD detection framework for training to obtain a fishing net detection model; wherein the backbone network of the SSD detection framework is a VGG network, and inputting the second image and the multiple spectral feature maps into the SSD detection framework for training comprises: Inputting the second image into the VGG network for convolutional feature extraction and pooling to obtain a first feature map; Merging the first characteristic graph and the first spectral characteristic graph to obtain a second characteristic graph; Performing convolution feature extraction and pooling on the second feature map to obtain a third feature map; Combining the third characteristic graph and the second spectral characteristic graph to obtain a fourth characteristic graph; Performing convolution feature extraction and pooling on the fourth feature map to obtain a fifth feature map; Combining the fifth characteristic graph and the third spectral characteristic graph to obtain a sixth characteristic graph; Using the sixth feature map for training; The resolution of the third feature map is smaller than the resolution of the first feature map, and larger than the resolution of the fifth feature map.

2. The method according to claim 1, characterized in that: After the fishing net in the sample image is marked to obtain the first image, the method further includes: determining color distribution information of the first image; The first image is normalized according to the color distribution information.

3. The method according to claim 1, characterized in that The preprocessing of the first image to obtain the second image includes: Performing a convolution operation on the first image using a mean template; Perform logarithmic transformation on the data obtained after the convolution operation to obtain the second image.

4. The method according to claim 3, characterized in that After performing logarithmic transformation on the data obtained after the convolution operation, the method further includes: Gamma transformation is used to correct data with too high or too low gray levels in order to enhance the contrast of the data.

5. The method according to claim 1, characterized in that The method further comprises: Acquire image data containing a fishing net; The image data is input into the fishing net detection model to obtain the fishing net information output by the fishing net detection model.

6. The method according to claim 1, characterized in that The marking of the fishing net in the sample image includes marking the type information and location information of the fishing net.

7. A training device for a fishing net detection model, characterized in that: The device comprises: A sample acquisition module, used for acquiring a sample image; An image annotation module, used for annotating the fishing net in the sample image to obtain a first image; A preprocessing module, used for preprocessing the first image to obtain a second image; A wavelet transform module, used for decomposing the second image into three layers by using Haar wavelet, so as to obtain a first spectral characteristic graph, a second spectral characteristic graph and a third spectral characteristic graph in sequence; A model training module, used for inputting the second image and the spectral feature map into an SSD detection framework for training to obtain a fishing net detection model; Among them, the backbone network of the SSD detection framework is the VGG network; The model training module is also used to input the second image into the VGG network for convolution feature extraction and pooling to obtain a first feature map; Merging the first characteristic graph and the first spectral characteristic graph to obtain a second characteristic graph; Performing convolution feature extraction and pooling on the second feature map to obtain a third feature map; Combining the third characteristic graph and the second spectral characteristic graph to obtain a fourth characteristic graph; Performing convolution feature extraction and pooling on the fourth feature map to obtain a fifth feature map; Combining the fifth characteristic graph and the third spectral characteristic graph to obtain a sixth characteristic graph; Using the sixth feature map for training; The resolution of the third feature map is smaller than the resolution of the first feature map, and larger than the resolution of the fifth feature map.

8. A computer device comprising a memory and a processor, wherein the memory stores a computer program, wherein: When the processor executes the computer program, the steps of the method according to any one of claims 1 to 6 are implemented.

9. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the steps of the method according to any one of claims 1 to 6 are implemented.

10. A computer program product, comprising a computer program, characterized in that When the computer program is executed by a processor, the steps of the method according to any one of claims 1 to 6 are implemented.

Images