An image splicing and target recognition method and system based on sea cucumber catching

Abstract

The application discloses an image splicing and target recognition method based on sea cucumber catching, comprising the following steps: acquiring an underwater sea cucumber data set and marking and dividing the data set; building a deep learning model and setting model training parameters, running the model to start training whole-process monitoring; storing the trained underwater sea cucumber recognition model; pre-processing the imported image, calculating the Ratio index, and selecting a suitable image sorting algorithm to sort the image sequence; splicing the image, repairing the spliced image; and recognizing the sea cucumber in the acquired image through the trained model deployed on an embedded platform. Through image sorting and splicing and other processing on the acquired underwater optical or acoustic image, and based on the YOLOv5 deep learning model and its deployment in an embedded system, the application realizes high-precision and high-efficiency recognition of the sea cucumber, and provides a feasible design scheme for the visual system design of an underwater sea cucumber intelligent catching robot for replacing manual fishing.

Description

Technical Field

[0001] This invention relates to the field of underwater image processing, and in particular to a method and system for image stitching and target recognition based on sea cucumber harvesting. Background Technology

[0002] In traditional sea cucumber harvesting methods, the selection of densely populated areas and the identification of sea cucumbers rely on the experience of professionals. Replacing costly and risky manual harvesting with an intelligent sea cucumber harvesting system is an inevitable trend. Visual recognition is a crucial component of sea cucumber harvesting systems. However, underwater images often contain noise and distortion, and the visibility of the underwater environment is limited. Furthermore, the small size of target marine organisms like sea cucumbers makes underwater image processing and analysis, as well as the selection of densely populated areas and the identification of sea cucumbers, challenging. This invention aims to obtain smooth, seamless underwater wide-field-of-view images by sorting and stitching underwater optical or acoustic images. This provides more complete and accurate information for the initial selection of densely populated areas for sea cucumber farming. Moreover, based on a deep learning model deployed on an embedded platform, it achieves high-precision and high-efficiency identification of sea cucumbers, providing a feasible design solution for the vision system of an intelligent underwater sea cucumber harvesting robot.

[0003] Due to the unknown underwater environment and limited visibility, the influence of underwater microorganisms and impurities, and the absorption and scattering of light by water, underwater images often suffer from various noises, distortions, and degradations, resulting in a decline in optical image quality. Sonar images are also subject to significant interference, with underwater noise and reverberation greatly affecting the quality of sonar images. Furthermore, the small size of marine organisms such as sea cucumbers makes underwater identification extremely difficult. Manually sorting the large number of underwater images acquired is inefficient. The design of a visual recognition system is a crucial aspect of the design of an intelligent underwater sea cucumber harvesting system, significantly impacting the subsequent precise and efficient harvesting operations and motion control of the underwater harvesting robot. Summary of the Invention

[0004] The purpose of this section is to outline some aspects of embodiments of the present invention and to briefly describe some preferred embodiments. Simplifications or omissions may be made in this section, as well as in the abstract and title of this application, to avoid obscuring the purpose of these documents; however, such simplifications or omissions should not be construed as limiting the scope of the invention.

[0005] In view of the high risk and low efficiency of the above and / or existing traditional manual harvesting methods for sea cucumbers, as well as the problems that make it difficult for underwater harvesting systems to automatically detect and identify sea cucumbers due to factors such as noise, distortion, and degradation of underwater images, this invention is proposed.

[0006] Therefore, the problem to be solved by this invention is how to enable the underwater intelligent sea cucumber harvesting robot system to accurately and efficiently detect and identify underwater sea cucumbers.

[0007] To solve the above-mentioned technical problems, the present invention provides the following technical solution:

[0008] In a first aspect, embodiments of the present invention provide an image stitching and target recognition method based on sea cucumber harvesting, comprising: acquiring an underwater sea cucumber dataset and labeling and dividing the dataset; building a deep learning model and setting model training parameters; storing model training weights to obtain a trained underwater sea cucumber recognition model; acquiring underwater optical or acoustic images and storing imported auxiliary information; preprocessing the imported images; calculating the correlation between each image in the imported image sequence and other images, and calculating the Ratio index based on the correlation, and sorting the image sequence using an image sorting algorithm; stitching the images, repairing the stitched images to obtain an underwater wide-field-of-view image, and determining dense sea cucumber areas; acquiring optical images in the dense sea cucumber areas, and recognizing sea cucumbers in the acquired images using the trained model.

[0009] As a preferred embodiment of the image stitching and target recognition method based on sea cucumber harvesting described in this invention, the specific formula for calculating the correlation between each image in the imported image sequence and other images is as follows:

[0010]

[0011] Where F1(u,v) is the Fourier transform of the image f1(x,y). Let f2(x,y) be the complex conjugate function of the Fourier transform of the image f2(x,y), and x0 and y0 be the translations of the image f2(x,y) relative to the image f1(x,y) in the horizontal and vertical directions, respectively.

[0012]

[0013] Among them, F -1 This represents the inverse transformation from the frequency domain to the spatial domain, where x0 and y0 are the horizontal and vertical translations of image f2(x,y) relative to image f1(x,y), respectively.

[0014] As a preferred embodiment of the image stitching and target recognition method based on sea cucumber harvesting described in this invention, the calculation of the Ratio index based on correlation includes the following steps:

[0015] The formula for calculating the Ratio indicator is as follows:

[0016]

[0017] Where corr(:) refers to the column vector composed of the correlation values ​​calculated between image i and all other images. The Ratio value calculated for each image is a function of the correlation values ​​between that image and other images. The minimum value among all Ratios is the selection index for the image sequence sorting algorithm, which is the basis for the selection of the image sorting algorithm. Max(corr(:)) is the maximum value among the correlation values ​​calculated between image i and all other images. Mean(corr(:)) is the average value of all the correlation values ​​calculated between image i and all other images.

[0018] The obtained Ratio is compared with the threshold given in the algorithm, and a sorting algorithm is selected. If the Ratio is greater than the threshold given in the algorithm, the phase correlation method is selected for automatic image sorting; if the Ratio is less than the threshold given in the algorithm, the SSIM method is selected for automatic image sorting.

[0019] As a preferred embodiment of the image stitching and target recognition method based on sea cucumber harvesting described in this invention, the phase correlation method includes the following steps: calculating the normalized power spectrum of two images; transforming the power spectrum to the spatial domain to obtain the peak size and translation parameters; and sorting the sequence images according to the calculated peak size.

[0020] As a preferred embodiment of the image stitching and target recognition method based on sea cucumber harvesting described in this invention, the SSIM method includes the following steps: calculating the SSIM value between every two images in a nested loop, when image i traverses from i=1 to i=num_imgs, image j traverses from j=i+1 to j=num_imgs, and calculating the SSIM value ssim_val between image i and image j; assigning ssim_val to ssim_mat(i,j) and ssim_mat(j,i) of the two-dimensional matrix ssim_mat; sorting the rows of ssim_mat in descending order, storing the indices in the array sorted_idx, and rearranging imgs using sorted_idx to obtain sorted_imgs.

[0021] As a preferred embodiment of the image stitching and target recognition method based on sea cucumber harvesting described in this invention, the specific formula for repairing the stitched image is as follows:

[0022]

[0023] Among them, (x1,y1), (x1,y2), (x2,y1), and (x2,y2) are the four known pixels closest to (x,y).

[0024] As a preferred embodiment of the image stitching and target recognition method based on sea cucumber harvesting described in this invention, the image stitching includes underwater optical image stitching and underwater acoustic image stitching. The underwater optical image stitching includes feature extraction, feature matching, and optical image stitching. The underwater acoustic image stitching includes acoustic image registration and acoustic image stitching.

[0025] Secondly, embodiments of the present invention provide an image stitching and target recognition system based on sea cucumber harvesting, comprising: a data import and preprocessing module for importing optical or acoustic images, storing auxiliary information, and performing necessary preprocessing operations; an image stitching module for sorting and stitching underwater optical or acoustic image sequences, and repairing the stitched images; a data annotation and partitioning module for annotating and pre-partitioning the collected sea cucumber dataset for training a deep learning model; a deep learning model training module for building and configuring a deep learning network, setting model training parameters, and running the model to enable full-process monitoring of training; and an embedded platform image processing module for deploying the trained deep learning model on an embedded platform for image processing, thereby realizing real-time recognition and acquisition by the sea cucumber harvesting vision system.

[0026] Thirdly, embodiments of the present invention provide a computer device, including a memory and a processor, wherein the memory stores a computer program, wherein: when the computer program instructions are executed by the processor, they implement the steps of the image stitching and target recognition method based on sea cucumber harvesting as described in the first aspect of the present invention.

[0027] Fourthly, embodiments of the present invention provide a computer-readable storage medium having a computer program stored thereon, wherein: when the computer program instructions are executed by a processor, they implement the steps of the image stitching and target recognition method based on sea cucumber harvesting as described in the first aspect of the present invention.

[0028] The beneficial effects of this invention are as follows: By processing the acquired underwater optical or acoustic images through image sorting and stitching, a smooth and seamless underwater large field-of-view image is obtained, which facilitates the selection of sea cucumber farming and dense areas. Furthermore, based on the YOLOv5 deep learning model and its deployment in embedded systems, high-precision and high-efficiency recognition of sea cucumbers is achieved, providing a feasible design scheme for the vision system design of intelligent underwater sea cucumber harvesting robots that can replace manual harvesting. Attached Figure Description

[0029] The purpose of this section is to outline some aspects of embodiments of the present invention and to briefly describe some preferred embodiments. Simplifications or omissions may be made in this section, as well as in the abstract and title of this application, to avoid obscuring the purpose of these documents; however, such simplifications or omissions should not be construed as limiting the scope of the invention.

[0030] Figure 1 This is a flowchart illustrating the image stitching and target recognition method based on sea cucumber harvesting in Example 1.

[0031] Figure 2 This is the main network structure of YOLO-V5, an image stitching and target recognition method based on sea cucumber harvesting, in Example 1.

[0032] Figure 3 This is the sea cucumber dataset for Example 2, based on image stitching and target recognition methods for sea cucumber harvesting.

[0033] Figure 4 This is an image showing the evaluation results of the deep learning model training effect based on the image stitching and target recognition method for sea cucumber harvesting in Example 2.

[0034] Figure 5 The images shown are the results of image sorting before (top) and automatic sorting based on the SSIM algorithm (bottom) in Example 2, based on the image stitching and target recognition method for sea cucumber harvesting.

[0035] Figure 6 The images shown are the results of image sorting before (top) and automatic sorting based on phase correlation method (bottom) in Example 2, which is based on image stitching and target recognition method for sea cucumber harvesting.

[0036] Figure 7 This is an example of the left and right image feature extraction and matching results using SIFT and KNN algorithms based on the image stitching and target recognition method for sea cucumber harvesting in Example 2.

[0037] Figure 8 The image shows the sorting results of the image stitching and target recognition method based on sea cucumber harvesting in Example 2, and the image stitching effect of the SIFT algorithm.

[0038] Figure 9 The image shows the sorting results of the image stitching and target recognition method based on sea cucumber harvesting in Example 2, and the image stitching effect of the phase correlation method.

[0039] Figure 10 This is a diagram showing the result of processing some sea cucumber images by the model trained based on the image stitching and target recognition method of sea cucumber harvesting in Example 2.

[0040] Figure 11 This is a diagram showing the final result of the model of the image stitching and target recognition method based on sea cucumber harvesting in Example 2 running on the ROS system of the embedded device TX2. Detailed Implementation

[0041] To make the above-mentioned objects, features and advantages of the present invention more apparent and understandable, the specific embodiments of the present invention will be described in detail below with reference to the accompanying drawings.

[0042] Many specific details are set forth in the following description in order to provide a full understanding of the invention. However, the invention may also be practiced in other ways different from those described herein, and those skilled in the art can make similar extensions without departing from the spirit of the invention. Therefore, the invention is not limited to the specific embodiments disclosed below.

[0043] Secondly, the term "one embodiment" or "embodiment" as used herein refers to a specific feature, structure, or characteristic that may be included in at least one implementation of the present invention. The phrase "in one embodiment" appearing in different places in this specification does not necessarily refer to the same embodiment, nor is it a single or selective embodiment that is mutually exclusive with other embodiments.

[0044] Example 1

[0045] Reference Figures 1-2 This is the first embodiment of the present invention, which provides an image stitching and target recognition method based on sea cucumber harvesting, including:

[0046] S1: Obtain the underwater sea cucumber dataset and label and divide the dataset.

[0047] Specifically, in this embodiment, the LabelImg tool is mainly used to annotate the valid images in the dataset, and the annotation information is obtained and saved. Then, the dataset is divided into training set and validation set according to a certain ratio. Depending on the size of the dataset, common division ratios are 3:1, 4:1, 7:1, etc. In this embodiment, the training set and validation set are divided into 4:1.

[0048] S2: Build a deep learning model and set the model training parameters.

[0049] Specifically, the YOLOv5 network model is adopted. The main structure is to first divide the network into grids, with each grid responsible for predicting the target in its corresponding region. Finally, the output is processed after non-maximum suppression. In this embodiment, for sea cucumber detection, the number of detection categories is 1. The network model needs to configure the size of the prior boxes, the backbone, and the head. The relevant parameters include which layer the input comes from, the number of layers, the module type, and other parameters (args). When building the network model, the depth coefficient depth_multiple is multiplied by the number of layers to adjust the depth, and the width coefficient width_multiple is multiplied by the number of convolutional kernels to obtain the actual number of convolutional kernels. The width of the network model is adjusted in this way.

[0050] Furthermore, set the model training parameters and run the model to enable full training monitoring. The main settings to be modified are the training epochs, batch size, and learning rate. Monitor the changes in training set bounding box loss, target loss, and classification loss during the training process.

[0051] S3: Store the model training weights to obtain the trained underwater sea cucumber recognition model.

[0052] Preferably, the training time for terminating the model is determined based on the previously set training rounds. After training is completed, the weights of the convolutional neural network are stored to obtain the trained underwater sea cucumber recognition model.

[0053] S4: Acquire underwater optical or acoustic images and store imported auxiliary information.

[0054] Specifically, the image format is JPG. The underwater optical or acoustic images are acquired from the imaging and detection equipment such as cameras or side-scan sonars carried by the underwater acquisition system, and image auxiliary information (such as image storage directory, GPS information and depth information of sonar images) is stored.

[0055] It is important to note that storing auxiliary information supplements the spatial information of the scanned two-dimensional image, making it easier to identify the dense sea cucumber areas in the stitched image and obtain their spatial location information.

[0056] S5: Preprocess the imported image.

[0057] Furthermore, when storing images, damaged or unrecognizable images are removed. The preprocessing of images mainly involves filtering and noise reduction, which includes Gaussian filtering, median filtering, wavelet transform filtering, and mean filtering. For a fixed working scenario, appropriate noise reduction processing can be selected based on filtering effect evaluation indicators such as VIF and GMSD. In this embodiment, Gaussian filtering is used.

[0058] It is important to note that Gaussian filtering was chosen because, after performing various filtering operations on the acquired images, Gaussian filtering showed better VIF and GMSD performance compared to other filters. Gaussian filtering often yields the best results for processing sonar side-scan images, so it was selected for image denoising.

[0059] Furthermore, the Gaussian filtering preprocessing of the image involves sliding a filter kernel along the image and performing operations on it. The filter kernel is a matrix generated by a Gaussian distribution, and the expression for the two-dimensional Gaussian function is shown below:

[0060]

[0061] Where σ is the standard deviation of the Gaussian distribution, and x and y are the pixel coordinates of the image.

[0062] S6: Calculate the correlation between each image in the imported image sequence and all other images, calculate the Ratio index based on the correlation, and sort the image sequence using an image sorting algorithm.

[0063] Furthermore, for different types of image data, such as underwater optical images or acoustic images, a novel method for automatically sorting images based on self-defined rules and thresholds is proposed, specifically including the following steps:

[0064] S6.1: The specific formula for calculating the correlation between each image in the imported image sequence and all other images is as follows:

[0065]

[0066] Where F1(u,v) is the Fourier transform of the image f1(x,y). Let f2(x,y) be the complex conjugate function of the Fourier transform of the image f2(x,y), and x0 and y0 be the translations of the image f2(x,y) relative to the image f1(x,y) in the horizontal and vertical directions, respectively.

[0067]

[0068] Among them, F -1 This represents the inverse transformation from the frequency domain to the spatial domain, where x0 and y0 are the horizontal and vertical translations of image f2(x,y) relative to image f1(x,y), respectively.

[0069] S6.2: Calculate the Ratio index based on correlation.

[0070] Specifically, it includes the following steps:

[0071] S6.2.1: The formula for calculating the Ratio index is as follows:

[0072]

[0073] Where corr(:) refers to the column vector composed of the correlation values ​​calculated between image i and all other images. The Ratio value calculated for each image is a function of the correlation values ​​between that image and other images. The minimum value among all Ratios is the selection index for the image sequence sorting algorithm, which is the basis for the selection of the image sorting algorithm. Max(corr(:)) is the maximum value among the correlation values ​​calculated between image i and all other images. Mean(corr(:)) is the average value of all the correlation values ​​calculated between image i and all other images.

[0074] Furthermore, the correlation value between the two images is calculated by using Fourier transform to obtain the phase information of the cross power spectrum of the two images, transforming it to the spatial domain to obtain the impulse function and determine its peak value. The magnitude of the peak value reflects the correlation between the two images. A larger peak value indicates a greater correlation because images with greater correlation have a larger overlapping area in the spatial domain, and the cross power spectrum obtained by transforming it to the frequency domain is larger. Therefore, the peak value of the impulse function obtained by transforming the cross power spectrum back to the spatial domain is larger.

[0075] S6.2.2: Compare the obtained Ratio with the threshold given in the algorithm and select the sorting algorithm.

[0076] Specifically, based on the ratio, the sorting algorithm can be selected from two options: SSIM, which utilizes spatial information of the image, and phase correlation, which utilizes frequency domain information of the image. The ratio is used to select the algorithm that combines the two methods.

[0077] S6.2.2.1: If Ratio is greater than the threshold given in the algorithm, then the phase correlation method is selected for automatic image sorting.

[0078] Furthermore, if the calculated value of the image sequence is greater than the threshold given in the algorithm, it indicates that the maximum value of the peak calculated by the phase correlation method differs significantly from other values. In this case, the phase correlation method is more effective, so the correlation method is selected for automatic image sorting. The specific steps are as follows: calculate the normalized power spectrum of the two images; transform the power spectrum to the spatial domain to obtain the peak size and translation parameters; sort the sequence images according to the calculated peak size.

[0079] S6.2.2.2: If the Ratio is less than the threshold given in the algorithm, the SSIM method is selected for automatic image sorting.

[0080] Specifically, the SSIM method includes the following steps: Calculate the SSIM value between every two images in a nested loop. When image i is traversed from i=1 to i=num_imgs, image j is traversed from j=i+1 to j=num_imgs. Calculate the SSIM value ssim_val between image i and image j. Assign ssim_val to ssim_mat(i,j) and ssim_mat(j,i) of the two-dimensional matrix ssim_mat. Sort the rows of ssim_mat in descending order, store the indices in the array sorted_idx, and rearrange the images using sorted_idx to obtain sorted_imgs.

[0081] S7: Stitch the images together, repair the stitched images, obtain a wide field-of-view underwater image, and determine the dense sea cucumber area.

[0082] Furthermore, image stitching includes underwater optical image stitching and underwater acoustic image stitching. Underwater optical image stitching includes feature extraction, feature matching, and optical image stitching, while underwater acoustic image stitching includes acoustic image registration and acoustic image stitching.

[0083] Specifically, underwater optical image stitching includes feature extraction, feature matching, and optical image stitching. Feature extraction involves detecting key points and extracting feature descriptions using methods such as scale-invariant feature transformation. In this embodiment, the SIFT algorithm is used. Feature matching uses KNN detection. Optical image stitching is based on perspective distortion of the image using RANSAC, and the images are stitched together sequentially according to the order arranged in the previous step.

[0084] Specifically, underwater acoustic image stitching includes acoustic image registration and acoustic image stitching. Acoustic image registration calculates the translation parameters of the two images based on the phase correlation method and aligns the sorted images to be stitched. Acoustic image stitching translates and stitches the images sequentially according to their order.

[0085] Furthermore, the specific formula for repairing the stitched image is as follows:

[0086]

[0087] Among them, (x1,y1), (x1,y2), (x2,y1), and (x2,y2) are the four known pixels closest to (x,y).

[0088] S8: Acquire optical images in areas with high sea cucumber density, and use a trained model to identify sea cucumbers from the acquired images.

[0089] Specifically, after identifying areas with high sea cucumber density, color underwater optical images are acquired in real time from a depth camera mounted on an underwater harvesting robot in that area. The acquired images are then used to identify sea cucumbers by a trained model deployed on an embedded platform.

[0090] Furthermore, the trained model is deployed to the ROS system on the TX2 embedded platform to process the images acquired by the underwater RBGD camera, identify sea cucumbers in the images, and determine the three-dimensional position of the sea cucumbers by combining the depth information obtained from the depth camera. This information is then published through the ROS system.

[0091] Furthermore, this embodiment also provides an image stitching and target recognition system based on sea cucumber harvesting, including a data import and preprocessing module for importing optical or acoustic images, storing auxiliary information, and performing necessary preprocessing operations; an image stitching module for sorting and stitching underwater optical or acoustic image sequences, and repairing the stitched images; a data annotation and partitioning module for annotating and pre-partitioning the collected sea cucumber dataset for training a deep learning model; a deep learning model training module for building and configuring a deep learning network, setting model training parameters, and running the model to enable full-process monitoring of training; and an embedded platform image processing module for deploying the trained deep learning model on an embedded platform for image processing, realizing real-time recognition and acquisition by the sea cucumber harvesting vision system.

[0092] This embodiment also provides a computer device applicable to the image stitching and target recognition method based on sea cucumber harvesting, including a memory and a processor; the memory is used to store computer-executable instructions, and the processor is used to execute the computer-executable instructions to realize the image stitching and target recognition method based on sea cucumber harvesting as proposed in the above embodiment.

[0093] The computer device can be a terminal, comprising a processor, memory, communication interface, display screen, and input devices connected via a system bus. The processor provides computing and control capabilities. The memory includes non-volatile storage media and internal memory. The non-volatile storage media stores the operating system and computer programs. The internal memory provides an environment for the operation of the operating system and computer programs stored in the non-volatile storage media. The communication interface is used for wired or wireless communication with external terminals; wireless communication can be achieved through Wi-Fi, carrier networks, NFC (Near Field Communication), or other technologies. The display screen can be an LCD screen or an e-ink screen. The input devices can be a touch layer covering the display screen, buttons, a trackball, or a touchpad on the computer device's casing, or an external keyboard, touchpad, or mouse.

[0094] This embodiment also provides a storage medium storing a computer program that, when executed by a processor, performs the following steps: acquiring an underwater sea cucumber dataset and labeling and dividing the dataset; building a deep learning model and setting model training parameters; storing model training weights to obtain a trained underwater sea cucumber recognition model; acquiring underwater optical or acoustic images and storing imported auxiliary information; preprocessing the imported images; calculating the correlation between each image in the imported image sequence and other images, and calculating the Ratio index based on the correlation, and sorting the image sequence using an image sorting algorithm; stitching the images together, repairing the stitched images to obtain an underwater wide-field-of-view image, and determining dense sea cucumber areas; acquiring optical images in the dense sea cucumber areas, and using the trained model to identify sea cucumbers from the acquired images.

[0095] In summary, this invention obtains smooth, seamless underwater wide-field-of-view images by processing the acquired underwater optical or acoustic images through image sorting and stitching, which facilitates the selection of sea cucumber farming and densely populated areas. Furthermore, based on the YOLOv5 deep learning model and its deployment in embedded systems, it achieves high-precision and high-efficiency recognition of sea cucumbers, providing a feasible design scheme for the vision system design of intelligent underwater sea cucumber harvesting robots that can replace manual harvesting.

[0096] Example 2

[0097] Reference Figures 3 to 11 This is the second embodiment of the present invention, which provides a design method for a sea cucumber harvesting vision system based on underwater image stitching and target recognition. In order to verify the beneficial effects of the present invention, scientific demonstration is carried out through economic benefit calculation and simulation experiments.

[0098] First, a sea cucumber dataset for training deep learning networks is acquired from an underwater camera, such as... Figure 3 As shown, the dataset includes sea cucumber images under different conditions such as clear underwater images, blurred underwater images, images above water, images with a small number of sea cucumbers, and images with a large number of sea cucumbers. The dataset is labeled and divided into training and validation sets in a 4:1 ratio.

[0099] Furthermore, a YOLOv5 network model was built, and training parameters were set. The initial learning rate was set to 0.01, the number of training epochs to 98, the batch size to 32, and the weight decay to 0.0005. Training began. After 98 epochs, training was completed, and the training effect of the model was as follows. Figure 4 As shown, the PR curve obtained after training is displayed, and the value of mAP@0.5 can reach 98.8%, which proves that the model has a very good recognition effect.

[0100] Furthermore, acoustic image sequences are acquired from the side-scan sonar mounted on the vehicle of the intelligent sea cucumber harvesting system, or optical image sequences are acquired from the camera mounted on the underwater harvesting robot. The image sequence contains six images. For each image, its correlation value with the other five images is calculated, and the algorithm selects the Ratio index.

[0101] Furthermore, based on the calculated Ratio index and the threshold of 1.4 determined in this embodiment, an image sorting algorithm is selected. Figure 5 The image sequence shown has a Ratio of 1.32, and the final sorting result was obtained using the SSIM method. Figure 6 The image sequence shown is a Ratio index of 1.52. The final sorting results using the phase correlation method are shown in Table 1.

[0102] Table 1 shows the image sorting algorithm selected based on the Ratio index and threshold.

[0103] Ratio metric (threshold 1.4) Domain Information algorithm 1.32(<1.4) Utilizing spatial domain information SSIM method 1.52(>1.4) Using frequency domain information Phase correlation method

[0104] Furthermore, by Figure 5 and Figure 6 The two processing results show that their corresponding algorithms process image spatial and frequency information respectively. By selecting indicators in the algorithm to link the two methods, the spatial and frequency domain information of the image can be fully utilized.

[0105] Furthermore, for the acquired underwater optical images, feature extraction, feature matching, and image stitching are performed based on SIFT, KNN, and RANSAC algorithms. Figure 7 The image above shows the feature points detected by the SIFT algorithm, as well as the matching effect based on the KNN algorithm. Based on this matching result and the RANSAC algorithm, the image is deformed and stitched together. The sorted result and the final image stitching effect of the underwater optical image are shown in the figure below. Figure 8 As shown.

[0106] Furthermore, for the acquired underwater acoustic images, image registration and stitching are performed based on the phase correlation method. According to the translation parameters obtained from the previously performed correlation calculations, the sorted images to be stitched are aligned and stitched together sequentially according to their order. Figure 9 The image shows the stitched result of the sorting and the final underwater acoustic image.

[0107] Furthermore, images of densely populated sea cucumber areas were acquired in real time from underwater cameras. A pre-trained model was deployed on the ROS system of the TX2 embedded platform to process the real-time images and identify the sea cucumbers in the images, verifying the practical feasibility of the underwater sea cucumber target detection system. Figure 10The results of processing some of the acquired sea cucumber images are shown, and the final effect of running them on the ROS system of TX2 is as follows. Figure 11 As shown.

[0108] It should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention and are not intended to limit it. Although the present invention has been described in detail with reference to preferred embodiments, those skilled in the art should understand that modifications or equivalent substitutions can be made to the technical solutions of the present invention without departing from the spirit and scope of the technical solutions of the present invention, and all such modifications or substitutions should be covered within the scope of the claims of the present invention.

Claims

1. A sea cucumber image capturing and target recognition method based on image stitching and target recognition, characterized in that: include, Obtain an underwater sea cucumber dataset and label and divide the dataset; Build a deep learning model and set the model training parameters; Store the model training weights to obtain a trained underwater sea cucumber recognition model; Acquire underwater optical or acoustic images and store and import auxiliary information; Preprocess the imported image; Calculate the correlation between each image in the imported image sequence and all other images, calculate the Ratio index based on the correlation, and sort the image sequence using an image sorting algorithm; The images are stitched together, and the stitched images are repaired to obtain a wide field-of-view underwater image and identify areas with high sea cucumber density. Optical images were acquired in areas with high sea cucumber density, and a trained model was used to identify sea cucumbers from the acquired images. Calculating the Ratio index based on correlation involves the following steps: The formula for calculating the Ratio indicator is as follows: ； wherein, a column vector composed of the correlation values calculated for image i with all other images, a value that is a function of the correlation values of image i with other images, the minimum value among all Ratios is the selection indicator for the image sequence algorithm, and is the basis for the selection of the image sequence algorithm, a maximum value among the correlation values calculated for image i with other images, an average value of all the correlation values calculated for image i with other images; The resulting compare with the threshold given in the algorithm and select the sorting algorithm; like If the image size exceeds the threshold given in the algorithm, the phase correlation method is selected for automatic image sorting. like If the image is smaller than the threshold given in the algorithm, the SSIM method is selected for automatic image sorting. The specific formula for repairing the stitched image is as follows: ； in, , , , It is distance The four most recent known pixels.

2. The image stitching and target recognition method based on sea cucumber harvesting as described in claim 1, characterized in that: The specific formula for calculating the correlation between each image in the imported image sequence and the other images is as follows: ； in, For image , For image The complex conjugate function of the Fourier transform, and Images Relative to image Translation amounts in the horizontal and vertical directions; ； in, This represents the inverse transform from the frequency domain to the spatial domain. and Images Relative to image Translation amount in the horizontal and vertical directions.

3. The image stitching and target recognition method based on sea cucumber harvesting as described in claim 2, characterized in that: The phase correlation method includes the following steps: Calculate the normalized power spectrum of the two images; Transform the power spectrum to the spatial domain to obtain the peak size and translation parameters; The sequence images are sorted according to the calculated peak size.

4. The image stitching and target recognition method based on sea cucumber harvesting as described in claim 3, characterized in that: The SSIM method includes the following steps: Calculate the SSIM value between every two images in a nested loop, when the image... from Traversing to At that time, the image from Traversal to Calculate the SSIM value between image i and image j. ; Will Assigned to a two-dimensional matrix of and ; sorted in descending order The rows will have their indices stored in an array. In, and use Rearrange To obtain .

5. The image stitching and target recognition method based on sea cucumber harvesting as described in claim 1, characterized in that: The image stitching includes underwater optical image stitching and underwater acoustic image stitching. Underwater optical image stitching includes feature extraction, feature matching, and optical image stitching, while underwater acoustic image stitching includes acoustic image registration and acoustic image stitching.

6. An image stitching and target recognition system based on sea cucumber harvesting, based on the image stitching and target recognition method based on sea cucumber harvesting as described in any one of claims 1 to 5, characterized in that: It also includes, The data import and preprocessing module is used to import optical or acoustic images, store auxiliary information, and perform necessary preprocessing operations. The image stitching module sorts and stitches underwater optical or acoustic image sequences, and repairs the stitched images. The data labeling and partitioning module is used to label and pre-partition the collected sea cucumber dataset in order to train the deep learning model. The deep learning model training module is used to build and configure deep learning networks, set model training parameters, and run the model to enable full monitoring of the training process. The embedded platform image processing module is used to deploy the trained deep learning model on the embedded platform for image processing, enabling real-time recognition and acquisition by the sea cucumber harvesting vision system.

7. A computer device comprising a memory and a processor, wherein the memory stores a computer program, characterized in that: When the processor executes the computer program, it implements the steps of the method according to any one of claims 1 to 5.

8. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, it implements the steps of the method according to any one of claims 1 to 5.

Images