Shrimp body grading method and device based on deep learning

Abstract

The application provides a shrimp body grading method and device based on deep learning, and belongs to the field of agricultural information technology. The method comprises the following steps: inputting a target image into a trained target detection network model, outputting trunk information of a shrimp body, and determining a trunk area of the shrimp body according to the trunk information; extracting a local hyperspectral image of a trunk part according to a hyperspectral image containing the shrimp body and the trunk information, inputting the local hyperspectral image into a trained HybridSN network model, and obtaining a place of origin type; extracting local near-infrared spectral data of the trunk part according to near-infrared spectral data containing the shrimp body and the trunk information, inputting the local near-infrared spectral data into a trained SAE-FNN network model, and outputting a texture index of the shrimp body; and determining a quality grade of the shrimp body according to the trunk area, the place of origin type and the texture index. The method can avoid the problems that it is difficult to grade by weighing at present, and it is difficult to quickly grade due to differences in places of origin and differences in tastes, can realize efficient and accurate grading of shrimp bodies, and is beneficial to industrial popularization and application.

Description

TECHNICAL FIELD

[0001] The present application relates to the field of agricultural information technology, and in particular to a shrimp body grading method and device based on deep learning. BACKGROUND

[0002] With the increasing demand for crayfish, crayfish products have become a major focus of the development of the aquaculture industry. In order to avoid some fraudulent behavior in the crayfish industry chain, such as false reporting of origin, reducing the meat quality standard, etc., resulting in uneven quality of crayfish sold, it is necessary to complete the intelligent operation of crayfish grading.

[0003] At present, the crayfish grading mainly adopts a mechanical processing method, but this method has low overall efficiency. Although there is a crayfish grading method based on machine learning at present, the accuracy is often unsatisfactory, and it is difficult to be applied on a large scale. SUMMARY

[0004] In view of the grading difficulty existing in the prior art, the present application provides a shrimp body grading method and device based on deep learning.

[0005] The present application provides a shrimp body grading method based on deep learning, comprising: acquiring a target image containing a to-be-detected shrimp body, inputting the target image into a trained target detection network model, outputting the trunk information of the shrimp body, determining the trunk area of the shrimp body according to the trunk information, wherein the trunk is the part after the head and tail of the shrimp body; according to the hyperspectral image containing the shrimp body and the trunk information, extracting a local hyperspectral image of the trunk part, and inputting the local hyperspectral image into a trained HybridSN (Hybrid Spectral Net, a spectral network mixed with 2D and 3D convolution) network model to obtain the origin type of the to-be-detected shrimp body; according to the near-infrared spectral data containing the shrimp body and the trunk information, extracting local near-infrared spectral data of the trunk part, and inputting the near-infrared spectral data into a trained SAE-FNN (a neural network constructed by combining a stacked auto-encoder (SAE) and a fully connected neural network (FNN)) network model to output the texture index of the shrimp body; determining the quality grade of the shrimp body according to the trunk area, the origin type and the texture index; wherein the target detection network model is trained according to the shrimp body sample target image with determined trunk information; the HybridSN network model is trained according to the shrimp body sample local hyperspectral image with determined origin type; and the SAE-FNN network model is trained according to the shrimp body sample local near-infrared spectral data with determined texture index.

[0006] The application provides a shrimp body grading method based on deep learning.

[0007] The application provides a shrimp body grading method based on deep learning.

[0008] The application provides a shrimp body grading method based on deep learning.

[0009] The application provides a shrimp body grading method based on deep learning.

[0010] The application provides a shrimp body grading method based on deep learning.

[0011] According to the shelly crab body grading method based on deep learning provided by the application, the trunk information is the position of the detection box containing the trunk in the image, and accordingly, the determination of the trunk area of the shelly crab body according to the trunk information comprises: inputting the detection box containing the trunk into a trained DeepLabv3+ detection model to obtain the detection result of each pixel of the detection box as a trunk or background; determining the trunk area of the shelly crab body according to the pixel area of the detection result as a trunk; wherein the DeepLabv3+ detection model is obtained according to sample images with trunk and background distinguishing labels.

[0012] The application further provides a shelly crab body grading device based on deep learning, comprising: a body positioning module configured to obtain a target image containing a shelly crab body to be detected, input the target image into a trained target detection network model, output trunk information of the shelly crab body, and determine a trunk area of the shelly crab body according to the trunk information, wherein the trunk is a part of the shelly crab body after decapitation and tail removal; an origin detection module configured to extract a local hyperspectral image of a trunk part according to a hyperspectral image containing the shelly crab body and the trunk information, and input the local hyperspectral image into a trained HybridSN network model to obtain an origin type of the shelly crab body to be detected; a texture detection module configured to extract a local near-infrared spectrum data of a trunk part according to near-infrared spectrum data containing the shelly crab body and the trunk information, and input the near-infrared spectrum data into a trained SAE-FNN network model to output a texture index of the shelly crab body; and a quality detection module configured to determine a quality level of the shelly crab body according to the trunk area, the origin type and the texture index; wherein the target detection network model is trained according to shelly crab body sample target images with determined trunk information; the HybridSN network model is trained according to shelly crab body sample local hyperspectral images with determined origin types; and the SAE-FNN network model is trained according to shelly crab body sample local near-infrared spectrum data with determined texture indexes.

[0013] The application further provides a shrimp body grading system based on deep learning, which comprises a screening device, a camera and the shrimp body grading device based on deep learning.

[0014] The shrimp body grading system based on deep learning provided by the application further comprises a second panel, which is located below the first panel and is provided with N drop-off holes corresponding to the drop-off holes; the support and the shell are rotationally connected through a gear transmission mechanism, the gear transmission mechanism comprises meshed gears and a gear ring, the gear transmission mechanism is installed on the second panel, and the gear ring is installed on the inner wall of the shell.

[0015] The application further provides an electronic device, which comprises a memory, a processor and a computer program stored in the memory and capable of being run on the processor, and the processor implements the shrimp body grading method based on deep learning according to any one of the above-mentioned methods when the program is executed.

[0016] The application further provides a non-transitory computer readable storage medium, which stores a computer program, and the computer program implements the shrimp body grading method based on deep learning according to any one of the above-mentioned methods when the program is executed by a processor.

[0017] The application further provides a computer program product, which comprises a computer program, and the computer program implements the shrimp body grading method based on deep learning according to any one of the above-mentioned methods when the program is executed by a processor.

[0018] The shrimp body grading method and device based on deep learning provided by the application can quickly obtain the trunk information of the shrimp body through a target detection network model, determine the trunk area based on the trunk information, and extract the hyperspectral image of the trunk, so that accurate spatial and spectral information is obtained, and the accuracy of the origin information is ensured; meanwhile, the infrared spectrum information reflecting the internal structure of the sample is further extracted based on the trunk information, so that the detection accuracy of the texture index is improved. Further, the three types of neural networks are associated with each other to realize rapid and accurate acquisition of the trunk area, the origin type, and the texture index parameter, the spatial structure of the shrimp meat, the geographical characteristics, and the fusion characteristics of the trunk area are comprehensively considered, the problems that the trunk and limbs are not proportional and are difficult to weigh and grade, the origin difference leads to differences in the meat content of shrimp bodies of the same size, and the same meat content leads to differences in taste are avoided, and then the shrimp bodies can be efficiently and accurately graded, which is beneficial to industrial popularization and application. BRIEF DESCRIPTION OF DRAWINGS

[0019] In order to more clearly illustrate the technical solutions in the application or the prior art, the following will briefly introduce the drawings needed to be used in the embodiments or the prior art description. Obviously, the drawings in the following description are some embodiments of the application, and other drawings can be obtained by those skilled in the art without creative effort on the basis of these drawings.

[0020] Figure 1 is one of the process schematic diagrams of the shrimp body grading method based on deep learning provided by the application;

[0021] Figure 2 is the second process schematic diagram of the shrimp body grading method based on deep learning provided by the application;

[0022] Figure 3 is the structural schematic diagram of the shrimp body grading device based on deep learning provided by the application;

[0023] Figure 4 is one of the structural schematic diagrams of the screening device provided by the application;

[0024] Figure 5 is the second structural schematic diagram of the screening device provided by the application;

[0025] Figure 6 is the third structural schematic diagram of the screening device provided by the application; Figure 7 is the structural schematic diagram of the electronic device provided by the application. DETAILED DESCRIPTION

[0026] In order to make the objects, technical solutions and advantages of the present application clearer, the technical solutions in the present application will be described clearly and completely below in conjunction with the drawings in the present application. Obviously, the described embodiments are part of the embodiments of the present application, rather than all the embodiments. Based on the embodiments in the present application, all other embodiments obtained by those of ordinary skill in the art without creative labor fall within the scope of protection of the present application.

[0027] The technical solutions in the present application will be described clearly and completely below in conjunction with the drawings in the present application. Figures 1-7 The deep learning-based shrimp body grading method and device of the present application are described. Figure 1 The deep learning-based shrimp body grading method provided by the present application is shown in the flowchart. Figure 1 The deep learning-based shrimp body grading method provided by the present application comprises:

[0028] 101, obtaining a target image containing a shrimp body to be detected, inputting the target image into a trained target detection network model, outputting trunk information of the shrimp body, and determining a trunk area of the shrimp body according to the trunk information, wherein the trunk is the part of the shrimp body after the head and tail are removed.

[0029] The shrimp body to be detected is a crayfish to be detected and graded. First, the crayfish image is collected and preprocessed. An image acquisition system suitable for crayfish can be built, because during the acquisition process, the image acquisition system directly affects the recognition accuracy of the crayfish. If the angle of the photo collected by the image acquisition system is not correct, or the light is too dark and the clarity is too low, it will affect the subsequent recognition effect. By establishing the acquisition system, the crayfish image acquisition is completed, and the crayfish image is processed, denoised, etc., the purpose is to produce the most suitable image for neural network learning from the processing result.

[0030] The target detection network model can be an SSD network model. SSD (Single Shot MultiBox Detector) network is a more novel target detection neural network compared with YOLO, and has developed into one of the preferred frameworks for current target detection. The SSD network inherits the idea of regression instead of detection from the YOLO series, so it is naturally faster than Fast R-CNN, but its Map (average precision) is higher than YOLO, so the actual effect is good at present.

[0031] In the present application, the SSD network is applied to realize the crayfish target recognition, the crayfish image is learned, and the network model is trained to produce a model capable of locating and recognizing the head and tail of the shrimp body, and output the recognition image and the loss curve.

[0032] For SSD, because there are multiple feature maps with different convolution kernel sizes in the whole network, it is in the form of a pyramid, and each pyramid classifies and regresses the image during recognition, so the result is more accurate. And in SSD, there is a Prior box designed by referring to the Anchor in RCNN network. Prior box represents some pre-selected boxes for specific targets before resizing. After that, the specific position of these boxes is adjusted through classification and regression to obtain the accurate position of the final target. In SSD, the size is calculated as a scaling ratio relative to the original image, called "scale". The calculation formula of scale is as follows:

[0033]

[0034] Where, S k represents the size of the prediction box of the kth layer feature map, k represents the current feature map layer number, and M represents the total number of feature map layers; S min is the size of the prediction box of the shallowest layer feature map, S max is the size of the prediction box of the deepest layer feature map.

[0035] Each prior box has its own maximum and minimum size, and the size of the prediction box of each layer feature map is between Smin and Smax.

[0036] After separating the crayfish from the image background by the SSD model, the area of the crayfish is calculated as an external evaluation index.

[0037] 102. According to the hyperspectral image containing the shrimp body and the torso information, a local hyperspectral image of the torso part is extracted, and the local hyperspectral image is input into the trained HybridSN network model to obtain the type of origin of the detected shrimp body.

[0038] Using hyperspectral technology can obtain spatial and spectral information at the same time, and then obtain the chemical and physical information of the crayfish. The collected data is mathematically modeled using chemometrics. Thus, the origin of the crayfish is output as an internal index for nondestructive testing.

[0039] Hyperspectral image (HSI) includes images of different wavebands. In hyperspectral data processing work, convolutional neural network (CNN) is one of the most commonly used deep learning-based hyperspectral data classification methods. Most of these methods are based on two-dimensional CNN, but two-dimensional CNN alone cannot extract high-discrimination feature maps from the spectral dimension. 3D convolution is more complex in calculation, and has poorer performance on many spectral bands for data with similar textures. On the other hand, the classification performance of HSI is highly dependent on spatial and spectral information. Therefore, the mixed use of 3D and 2D convolution can solve the above problems and reduce the complexity of the model.

[0040] In order to process hyperspectral data of shrimp bodies and classify the origins of the shrimp bodies, the application solves the hyperspectral image classification problem through a hybrid network HybridSN. First, 3D convolution is used to help the network represent spatial-spectral features, and then 2D convolution is used to further learn more abstract spatial representations. The hybrid use of 3D and 2D convolution reduces the complexity of the model, fully utilizes the spectral and spatial feature maps, and achieves the maximum possible accuracy.

[0041] 103. According to the near-infrared spectral data of the shrimp body and the trunk information, local near-infrared spectral data of the trunk part is extracted, and the near-infrared spectral data is input into the trained SAE-FNN network model to output the texture index of the shrimp body.

[0042] Hyperspectral data is generally collected by transmission and reflection, while near-infrared spectral data can be collected by diffuse reflection. Diffuse reflection light is light that is returned to the surface after multiple reflections, refractions, diffractions and absorptions after the analysis light enters the inside of the sample. Diffuse reflection light is light that has interacted with the molecules in the sample, so it contains information about the structure and composition of the sample. After correction of the diffuse reflection spectrum by the Kubelka-Munk equation, quantitative analysis can be performed.

[0043] Near-infrared spectral data collected by diffuse reflection contains information about the structure and composition of the sample, and can better express the internal information of the material than hyperspectral data. In the present application, after the trunk information is determined in 102, the local near-infrared spectral data of the trunk is input into the trained SAE-FNN network model, so that an accurate texture index can be obtained. Further, the near-infrared spectral data of the shrimp body can be obtained by a diffuse reflection near-infrared spectrometer.

[0044] The local near-infrared spectral data obtained can be in the form of a wavelength reflectivity vector, and the wavelength range can be 1100-2526 nm.

[0045] The texture index includes hardness, brittleness, cohesion, elasticity, gumminess, chewiness and resilience.

[0046] 104. According to the trunk area, origin type and texture index, the quality level of the shrimp body is determined.

[0047] The shrimp body grading method based on deep learning of the present application quickly obtains the trunk information of the shrimp body through a target detection network model, determines the trunk area based on this, and extracts the hyperspectral image of the trunk, so as to obtain accurate spatial and spectral information, and ensure the accuracy of the origin information; meanwhile, the infrared spectrum information reflecting the internal structure of the sample is further extracted based on the trunk information, so as to improve the detection precision of the texture index. Further, the present application realizes the rapid and accurate acquisition of the trunk area, the origin type and the texture index parameter through three types of neural networks that are interrelated, integrates the fusion characteristics of the spatial structure of the shrimp body, the geographical characteristics and the trunk area, avoids the current problems that the trunk and the limbs are not proportional and are difficult to weigh and grade, that the origin difference leads to the difference in the meat content of the shrimp bodies of the same size, and that the taste is different under the same meat content, and further realizes the efficient and accurate grading of the shrimp body, which is conducive to the industrial popularization and application.

[0048] In some embodiments, the target detection network model is a Mobilenet network replacing a backbone network, an SSD network model with an RFB network structure added after a prediction layer and an FC7 layer, and obtained after training based on a Soft-NMS algorithm.

[0049] In the embodiment of the present application, the backbone network of the SSD network is replaced by Mobilenet, and the precision of the retrained and modeled test network is tested. Because MobileNet is a lightweight deep neural network designed for some mobile embedded devices, the core of its innovative idea is to decompose the convolution kernel, thereby reducing the number of network parameters and the time required for training. Compared with the unimproved SSD network before, it can determine a relatively better model. The target detection algorithm is used to detect the whole crayfish, the head, the tail and the pincers, in the model training, the present application trains the SSD network and the SSD-MobileNet under the same experimental conditions, compares the results of the two network training, and verifies that the SSD-MobileNet has stronger recognition ability.

[0050] The structure of the RFB module is composed of multiple branches with different kernels and dilated convolution layers. This convolution part uses 1x1, 1x3, 3x1 and 3x3 convolution kernels. Because this structure can effectively reduce the number of convolution parameters. Therefore, the dilated convolution part is to generate higher resolution feature maps and increase the receptive field without increasing the number of convolution parameters. For example, the size of the input image is 300x300, and the 38x38 size feature map is used to predict small objects, but the semantic information of the 38x38 feature map is less enough to describe the object information. The present application adds the RFB network structure after the 38x38 prediction layer to extract more rich semantic information. At the same time, the RFB network structure is also added after the FC7 layer to further improve the model performance.

[0051] Soft-NMS uses the method of confidence, first from high to low score, and then this continuous function will weaken the detection score of the non-maximum detection box, rather than completely deleting them. This process is continuously recursively applied to the remaining t prior boundary boxes, so that the score of the prior boundary box can be reduced, but not directly and roughly deleted, so that a part of the required prior boundary box can be retained.

[0052] In some embodiments, before the obtaining includes the target image of the shrimp body to be detected, further comprising: obtaining a first shrimp body sample hyperspectral image of different known origin types; inputting the local hyperspectral image of the trunk part in the first shrimp body sample hyperspectral image into the constructed HybridSN network model, and training in combination with the corresponding known origin type as the label.

[0053] Optionally, the HybridSN network model is trained based on a ncf cycle function setting learning rate. The learning rate is the most important hyperparameter of the deep learning model, and a smaller learning rate will increase the training time of the deep learning model, and a larger learning rate will cause the model to deviate from the global minimum value. Therefore, the adaptive learning rate is generally introduced to dynamically change the value of the learning rate, and the cycle learning rate method changes the learning rate constantly each time. The introduced cycle function maintains a high learning rate value in the continuous iteration period, which may cause the model using the cycle learning method to deviate from the global minimum value. Therefore, in the training of the HybridSN, a new cycle function (ncf) is used to overcome this problem. The ncf cycle strategy is to set the maximum and minimum boundaries of the learning rate by itself. With the increase of the number of training iterations, the learning rate also increases, and once the learning rate reaches the highest learning rate value, the learning rate of the next stage is adjusted to the base value.

[0054] Specifically, four categories can be taken as labels: Anhui system, Hubei system, Jiangsu system, and Jiangxi system. Then, the collected hyperspectral data is reduced in dimension by PCA, and the data is divided into training set and validation set by spxy sample division. The prepared training set is used to train the network model HybridSN, and the accuracy of the model is tested by the validation set.

[0055] Based on the trained HybridSN network model, the local hyperspectral data of the shrimp to be detected is input in 102, and the origin of the shrimp body is output. Combined with the results of 101 and 103, the shrimp body is graded.

[0056] In one embodiment, before the target image containing the shrimp to be detected is acquired, the near-infrared spectrum data of a plurality of second shrimp samples is acquired, and the texture index value of each second shrimp sample measured according to the experiment is combined as a training sample to train the SAE-FNN network model.

[0057] To distinguish the samples trained by the HybridSN model, the SAE-FNN network model is trained as a second shrimp sample in the embodiment of the application.

[0058] For example, every 1 day, 15 lobster samples are randomly selected, full texture experiments are performed, and texture indexes: hardness, brittleness, cohesiveness, elasticity, gumminess, chewiness, and resilience are obtained, and spectral images are collected. In the experiment, 135 samples are randomly selected as a calibration set to train the network model, and the remaining 45 samples are used as a prediction set to estimate the performance of the calibration model.

[0059] At the same time, in order to predict the hardness, brittleness, cohesiveness, elasticity, etc. of the shrimp by using the deep learning method, 1000 pixel spectra are randomly selected in the ROI of each shrimp sample to create a large data set. In the SAE-FNN model, first, the randomly selected pixel spectra are used to train the SAE, then the trained SAE is applied to the average spectrum to extract useful spectral features, and finally, it is used as the input of the FNN in the SAE-FNN to predict the texture indexes of the shrimp such as hardness, brittleness, cohesiveness, and elasticity.

[0060] The improved network model SAE-FNN is trained using the prepared training set, and the accuracy of the model is tested by the validation set. Finally, the near-infrared spectrum data of the shrimp to be detected is input in 103, and the texture indexes of the shrimp are output.

[0061] In one embodiment, the HybridSN network model is obtained by adding a 3D multi-head self-attention mechanism after a 3D-convolution layer, and the 3D multi-head self-attention mechanism is used to: input the local hyperspectral image into three convolution kernels to obtain three feature maps with a size of HxWxCxD, wherein H, W, C and D are the height, width, channel number and depth of the feature map, respectively; reshape the three feature maps to obtain three matrices Q, K and V with a size of NxD / NxHWC, wherein N is the number of feature maps; initialize three position matrices with a size of NxD / NxHx1x1, NxD / Nx1xWx1 and NxD / Nx1x1xC respectively, add the three position matrices to obtain a matrix with a size of NxD / NxHxWxC, reshape the added matrix and multiply it by the matrix Q to obtain a final position encoding matrix P; multiply the matrices Q and K to obtain an attention matrix with a size of NxHWCxHWC, add the final position encoding matrix P to the attention matrix, multiply the matrix V after a Softmax activation function, output a matrix with a size of NxD / NxHWC, and perform a reshaping operation to output a feature map with a size of HxWxCxD.

[0062] In the embodiment of the present application, the HybridSN network model is further improved by adding a 3D multi-head self-attention mechanism. The improved method can include the following steps:

[0063] Given a feature map x with a shape of {H, W, C, D}, first pass it through three convolution kernels, such as a convolution kernel with a size of 1x1x1 combined with a three-dimensional convolution with a step of 1x1x1, to obtain three feature maps with a shape of {H, W, C, D}. After reshape operation, three matrices Q, K and V are obtained with a size of {N, D / N, HWC}, which retains the context information and local feature details. Then, multiply the matrices Q and K to obtain an attention matrix with a size of {N, HWC, HWC}. In order to confirm the position information between images, the position encoding information is introduced in the embodiment of the present application. Initialize three matrices with a size of {N, D / N, H, 1, 1}, {N, D / N, 1, W, 1} and {N, D / N, 1, 1, C}. It should be noted that H, W and C here are the H, W and C of the matrix Q. First, add the three position matrices to obtain a matrix with a size of {N, D / N, H, W, C}, perform a reshaping operation, and multiply it by the matrix Q to obtain a final position encoding matrix P. Multiply the matrices Q and K to obtain an attention matrix with a size of NxHWCxHWC, add the position encoding matrix P to the attention matrix, multiply the matrix V after a Softmax activation function, and output a matrix with a shape of {N, D / N, HWC}. After performing a reshaping operation, a feature map with a size of {H, W, C, D} is output.

[0064] The method of the embodiment of the present application firstly extracts multi-scale features by using convolution kernels of different sizes through the multi-head attention mechanism, samples different granularities of feature maps, and effectively fuses the spatial and spectral features of the feature maps.

[0065] In some embodiments, the torso information is the position of the detection box containing the torso in the image, and accordingly, the determining the torso area of the shrimp body according to the torso information comprises: inputting the detection box containing the torso into a trained DeepLabv3+ detection model to obtain the detection result of each pixel of the detection box being a torso or a background; and determining the torso area of the shrimp body according to the area of the pixels detected as the torso; wherein the DeepLabv3+ detection model is trained according to sample images with labels distinguishing the torso and the background.

[0066] In the embodiment of the present application, the Deeplab v3+ algorithm is used to train the model to obtain a reasonable model capable of performing semantic segmentation on shrimp bodies. Based on the shrimp body segmentation image recognized by Deeplab v3+, the area of the image is calculated, corresponding to different levels, so as to realize shrimp body grading.

[0067] The deep learning-based shrimp body grading device provided by the present application will be described below. The deep learning-based shrimp body grading device described below can be referred to in conjunction with the deep learning-based shrimp body grading method described above.

[0068] Figure 3 is a structural schematic diagram of the deep learning-based shrimp body grading device provided by the present application, like Figure 3As shown, the shrimp body grading device based on deep learning includes a body positioning module 301, an origin detection module 302, a texture detection module 303, and a quality detection module 304. The body positioning module 301 is configured to obtain a target image containing a shrimp body to be detected, input the target image into a trained target detection network model, output trunk information of the shrimp body, and determine a trunk area of the shrimp body according to the trunk information, wherein the trunk is a part of the shrimp body after decapitation. The origin detection module 302 is configured to extract a local hyperspectral image of the trunk part according to a hyperspectral image containing the shrimp body and the trunk information, and input the local hyperspectral image into a trained HybridSN network model to obtain an origin type of the shrimp body to be detected. The texture detection module 303 is configured to extract local near-infrared spectral data of the trunk part according to near-infrared spectral data containing the shrimp body and the trunk information, and input the near-infrared spectral data into a trained SAE-FNN network model to output a texture index of the shrimp body. The quality detection module 304 is configured to determine a quality grade of the shrimp body according to the trunk area, the origin type, and the texture index. The target detection network model is trained according to shrimp sample target images with determined trunk information. The HybridSN network model is trained according to shrimp sample local hyperspectral images with determined origin types. The SAE-FNN network model is trained according to shrimp sample local near-infrared spectral data with determined texture indexes.

[0069] The device embodiment provided by the embodiment of the present application is to realize the above-mentioned method embodiments, and the specific process and detailed content are referred to the above-mentioned method embodiments, which will not be repeated here.

[0070] The shrimp body grading device based on deep learning provided by the embodiment of the present application has the same implementation principle, technical effects and the above-mentioned shrimp body grading method embodiments based on deep learning. For brief description, the parts not mentioned in the shrimp body grading device embodiment based on deep learning can refer to the corresponding contents in the above-mentioned shrimp body grading method embodiment based on deep learning.

[0071] The present application also provides a shrimp body grading system based on deep learning, comprising a screening device, a camera and the above-mentioned shrimp body grading device based on deep learning, Figure 4 is one of the structural schematic diagrams of the screening device provided by the present application, like Figure 4The screening device shown. The screening device includes a support 1 that can rotate relative to a housing 2; N+2 baffles 21 are installed at intervals on the inner wall of the housing; a first panel 11 is installed vertically on the support 1 below the baffles 21; the N+2 baffles divide the first panel 11 into N+2 areas, wherein N areas correspond one-to-one to quality levels, and each area is provided with a drop hole 111, and each drop hole is provided with an automatically opening and closing door plate 112; the other two areas are a feeding port and an image acquisition area; a camera is provided above the image acquisition area to capture target images and spectral images containing shrimp bodies and send them to the shrimp body grading device based on deep learning; a door plate control mechanism 113 is provided below the first panel 11 to control the opening and closing of the corresponding door plates in the N areas according to the quality level obtained by the shrimp body grading device based on deep learning.

[0072] Figure 5 is a structural schematic diagram of the screening device provided by the present application, Figure 6 is a structural schematic diagram of the screening device provided by the present application, which can be referred to in combination. Specifically, taking the quality level as 4 as an example, six partitions are installed at intervals on the inner wall of the housing, which can divide the space on the first panel into six areas. The support 1 and the cylindrical housing 2 produce relative rotation. The first area is above the feeding port, and each time one lobster is transported, i.e., only one small lobster exists in one area. The lobster falls on the first panel 11, and the rotation of the circumferential housing 2 pushes the shrimp body forward. The second area is above the camera (which can be a multifunctional camera including an RGB camera module, a hyperspectral camera module, and a diffuse reflectance near-infrared spectrum camera module; or three cameras with RGB shooting, hyperspectral shooting, and diffuse reflectance near-infrared spectrum shooting, which are collectively referred to as the aforementioned camera), which takes a picture. After that, the grading device of the above embodiment classifies the small lobster, which can be classified into four levels. The device platform corresponding to the last four areas has a mechanically opening and closing door plate. When the corresponding classified small lobster reaches the corresponding area, the door plate opens, and the small lobster falls to complete the grading operation. If the small lobster reaches the area that does not correspond, the door plate will not open, and the small lobster will be pushed forward by the partition until it falls after reaching the corresponding area, and the cycle continues.

[0073] In some embodiments, a second panel 12 is further included, which is located below the first panel 11, and the second panel 12 is provided with N drop holes 121 corresponding to the drop holes. The support 1 and the housing 2 are connected by a gear transmission mechanism 13, which includes a gear 131 and a ring gear 22 that mesh with each other. The gear transmission mechanism 13 is installed on the second panel 12, and the ring gear 22 is installed on the inner wall of the housing 2.

[0074] Figure 7 is a structural schematic diagram of the electronic device provided by the present application, asFigure 7 As shown, the electronic device can include a processor 701, a communications interface 702, a memory 703, and a communications bus 704, wherein the processor 701, the communications interface 702, and the memory 703 complete mutual communication through the communications bus 704. The processor 701 can invoke a logical instruction in the memory 703 to execute a shrimp body grading method based on deep learning, which includes: acquiring a target image containing a shrimp body to be detected, inputting the target image into a trained target detection network model, outputting trunk information of the shrimp body, determining a trunk area of the shrimp body according to the trunk information, wherein the trunk is a part of the shrimp body after decapitation; according to a hyperspectral image containing the shrimp body and the trunk information, a local hyperspectral image of the trunk part is extracted, and the local hyperspectral image is input into a trained HybridSN network model to obtain a place of origin type of the shrimp body to be detected; according to near-infrared spectral data containing the shrimp body and the trunk information, local near-infrared spectral data of the trunk part is extracted, and the near-infrared spectral data is input into a trained SAE-FNN network model to output a texture index of the shrimp body; according to the trunk area, the place of origin type, and the texture index, a quality grade of the shrimp body is determined; wherein the target detection network model is trained according to shrimp body sample target images with determined trunk information; the HybridSN network model is trained according to shrimp body sample local hyperspectral images with determined place of origin type; and the SAE-FNN network model is trained according to shrimp body sample local near-infrared spectral data with determined texture index.

[0075] In addition, the logical instruction in the memory 703 described above can be implemented in the form of a software function unit and sold or used as an independent product, which can be stored in a computer readable storage medium. Based on such understanding, the technical solutions of the present application essentially or parts of the prior art that contribute to the present application or parts of the technical solutions can be embodied in the form of a software product, and the computer software product is stored in a storage medium, including a plurality of instructions to make a computer device (which can be a personal computer, a server, or a network device, etc.) execute all or part of the steps of the methods described in various embodiments of the present application. The aforementioned storage medium includes: a U disk, a mobile hard disk, a read-only memory (ROM, Read-Only Memory), a random access memory (RAM, Random Access Memory), a magnetic disk or an optical disk, and various media that can store program codes.

[0076] In another aspect, the present application also provides a computer program product, which comprises a computer program, the computer program being stored on a non-transitory computer readable storage medium, and the computer program being executable by a processor to cause a computer to perform a deep learning-based shrimp body grading method, the method comprising: obtaining a target image containing a shrimp body to be detected, inputting the target image into a trained target detection network model, outputting trunk information of the shrimp body, and determining a trunk area of the shrimp body according to the trunk information, wherein the trunk is a part of the shrimp body after the head and tail are removed; extracting a local hyperspectral image of the trunk part according to a hyperspectral image containing the shrimp body and the trunk information, inputting the local hyperspectral image into a trained HybridSN network model, and obtaining a place of origin type of the shrimp body to be detected; extracting local near-infrared spectral data of the trunk part according to near-infrared spectral data containing the shrimp body and the trunk information, inputting the near-infrared spectral data into a trained SAE-FNN network model, and outputting a texture index of the shrimp body; and determining a quality grade of the shrimp body according to the trunk area, the place of origin type, and the texture index; wherein the target detection network model is trained according to target images of shrimp body samples with determined trunk information; the HybridSN network model is trained according to local hyperspectral images of shrimp body samples with determined place of origin types; and the SAE-FNN network model is trained according to local near-infrared spectral data of shrimp body samples with determined texture indexes.

[0077] In yet another aspect, the present application also provides a non-transitory computer-readable storage medium having stored thereon a computer program, which, when executed by a processor, implements a deep learning-based shrimp body grading method provided by any of the above methods, the method comprising: obtaining a target image containing a shrimp body to be detected, inputting the target image into a trained target detection network model, outputting trunk information of the shrimp body, and determining a trunk area of the shrimp body according to the trunk information, wherein the trunk is a part of the shrimp body after decapitation and tailing; extracting a local hyperspectral image of the trunk part according to a hyperspectral image containing the shrimp body and the trunk information, and inputting the local hyperspectral image into a trained HybridSN network model to obtain a place of origin type of the shrimp body to be detected; extracting local near-infrared spectral data of the trunk part according to near-infrared spectral data containing the shrimp body and the trunk information, and inputting the near-infrared spectral data into a trained SAE-FNN network model to output a texture index of the shrimp body; and determining a quality grade of the shrimp body according to the trunk area, the place of origin type, and the texture index; wherein the target detection network model is trained according to shrimp body sample target images with determined trunk information; the HybridSN network model is trained according to shrimp body sample local hyperspectral images with determined place of origin types; and the SAE-FNN network model is trained according to shrimp body sample local near-infrared spectral data with determined texture indexes.

[0078] The device embodiments described above are merely illustrative, wherein the units described as separate components can or can not be physically separated, and the components displayed as units can or can not be physical units, i.e., can be located in one place or distributed on multiple network units. Part or all of the modules can be selected to achieve the purpose of the embodiment scheme according to actual needs. Those skilled in the art can understand and implement without creative labor.

[0079] From the above description of the embodiments, those skilled in the art can clearly understand that the embodiments can be implemented by means of software plus necessary general hardware platforms, and of course can also be implemented by hardware. Based on such understanding, the above technical solutions, essentially or in terms of the contribution to the prior art, can be embodied in the form of a software product. The computer software product can be stored in a computer readable storage medium, such as a ROM / RAM, a magnetic disk, an optical disk, etc., and includes a plurality of instructions to make a computer device (which can be a personal computer, a server, or a network device, etc.) execute the methods described in each embodiment or some parts of the embodiments.

[0080] It should be pointed out finally that the above embodiments are only used to illustrate the technical solutions of the present application, but not to limit the same; and although the present application has been described in detail with reference to the foregoing embodiments, it should be appreciated by those skilled in the art that the technical solutions recorded in the foregoing embodiments can be modified, or some technical features thereof can be replaced equivalently; and these modifications or replacements do not make the essence of the corresponding technical solutions deviate from the spirit and scope of the technical solutions of the embodiments of the present application.

Claims

1. A shrimp body grading method based on deep learning, characterized in that, The method comprises the following steps: obtaining a target image containing a shrimp to be detected, inputting the target image into a trained target detection network model, outputting trunk information of the shrimp, and determining a trunk area of the shrimp according to the trunk information, wherein the trunk is a part of the shrimp after the head and tail are removed; extracting a local hyperspectral image of the trunk part according to a hyperspectral image containing the shrimp and the trunk information, inputting the local hyperspectral image into a trained HybridSN network model, and obtaining a place of origin type of the shrimp to be detected; extracting local near-infrared spectral data of the trunk part according to near-infrared spectral data containing the shrimp and the trunk information, and inputting the near-infrared spectral data into a trained SAE-FNN network model to output a texture index of the shrimp; determining a quality level of the shrimp according to the trunk area, the place of origin type and the texture index; wherein the target detection network model is trained according to shrimp sample target images with determined trunk information; the HybridSN network model is trained according to shrimp sample local hyperspectral images with determined place of origin types; and the SAE-FNN network model is trained according to shrimp sample local near-infrared spectral data with determined texture indexes; the trunk information is the position of the detection box containing the trunk in the image, and correspondingly, the determination of the trunk area of the shrimp according to the trunk information comprises: inputting the detection box containing the trunk into a trained DeepLabv3+ detection model to obtain a detection result that each pixel of the detection box is a trunk or a background; determining the trunk area of the shrimp according to the pixel area of the detection result that is the trunk; wherein the DeepLabv3+ detection model is trained according to sample images with trunk and background as labels.

2. The deep learning-based shrimp body grading method according to claim 1, wherein, The target detection network model is an SSD network model in which a backbone network is replaced by a Mobilenet network, an RFB network structure is added after a prediction layer and an FC7 layer, and an NMS module of the SSD network model is replaced by a Soft-NMS module.

3. The deep learning-based shrimp body grading method according to claim 1, wherein, Before the target image containing the shrimp to be detected is obtained, the following steps are further included: obtaining first shrimp sample hyperspectral images of different known place of origin types; inputting local hyperspectral images of trunk parts in the first shrimp sample hyperspectral images into a constructed HybridSN network model, and training the HybridSN network model in combination with corresponding known place of origin types as labels; and obtaining near-infrared spectral data of a plurality of second shrimp samples, and training the SAE-FNN network model in combination with texture index values of each second shrimp sample measured by experiments as training samples.

4. The deep learning-based shrimp body grading method according to claim 1, wherein, The HybridSN network model is obtained by adding a 3D multi-head self-attention mechanism after a 3D convolution layer, and the 3D multi-head self-attention mechanism is used to: input the local hyperspectral image into three convolution kernels to obtain three feature maps of HxWxCxD respectively; wherein H, W, C and D are height, width, channel number and depth of the feature map respectively. After the three feature maps are reshaped, three matrices Q, K and V of NxD / NxHWC are obtained; wherein, N is the number of feature maps; Three position matrices of NxD / NxHx1x1, NxD / Nx1xWx1 and NxD / Nx1x1xC are initialized, and the three position matrices are added to obtain a matrix of NxD / NxHxWxC; The added matrix is reshaped and multiplied by the matrix Q to obtain a final position encoding matrix P; The matrix Q is multiplied by the matrix K to obtain an attention matrix of NxHWCxHWC, the final position encoding matrix P is added to the attention matrix, and after a Softmax activation function, the matrix V is multiplied to output a matrix of NxD / NxHWC. After reshaping, a feature map of HxWxCxD is output.

5. A shrimp body grading device based on deep learning, characterized in that, Comprise: A body positioning module is configured to obtain a target image containing a shrimp body to be detected, input the target image into a trained target detection network model, output trunk information of the shrimp body, and determine a trunk area of the shrimp body according to the trunk information, wherein the trunk is a part of the shrimp body after decapitation and tail removal; A producing area detection module is configured to extract a local hyperspectral image of a trunk part according to a hyperspectral image containing the shrimp body and the trunk information, and input the local hyperspectral image into a trained HybridSN network model to obtain a producing area type of the shrimp body to be detected; A texture detection module is configured to extract local near-infrared spectral data of a trunk part according to near-infrared spectral data containing the shrimp body and the trunk information, and input the near-infrared spectral data into a trained SAE-FNN network model to output a texture index of the shrimp body; A quality detection module is configured to determine a quality level of the shrimp body according to the trunk area, the producing area type and the texture index; The target detection network model is trained according to a shrimp sample target image with determined trunk information; the HybridSN network model is trained according to a shrimp sample local hyperspectral image with determined producing area type; and the SAE-FNN network model is trained according to a shrimp sample local near-infrared spectral data with determined texture index. The trunk information is a position of a detection frame containing a trunk in an image, and accordingly, the determination of the trunk area of the shrimp body according to the trunk information comprises: Inputting the detection frame containing the trunk into a trained DeepLabv3+ detection model to obtain a detection result that each pixel of the detection frame is a trunk or a background; Determining the trunk area of the shrimp body according to a pixel area detected as a trunk; The DeepLabv3+ detection model is trained according to a sample image with trunk and background distinguished as labels.

6. An electronic device comprising a memory, a processor, and a computer program stored on the memory and executable on the processor, characterized in that, The processor implements the deep learning-based shrimp body grading method of any one of claims 1 to 4 when executing the program.

7. A deep learning based shrimp body grading system, characterized in that, The deep learning-based shrimp body grading device of claim 5, wherein: The screening device comprises a support that can rotate relatively, and a shell; N+2 baffles are installed on the inner wall of the shell at intervals, a first panel is installed vertically on the support below the baffles, the N+2 baffles divide the first panel into N+2 areas, wherein N areas correspond to quality levels one by one, and each area is provided with a falling hole, an automatically opening and closing door plate is installed at each falling hole, and the other two areas are a feeding port and an image acquisition area respectively; wherein N is a positive integer greater than 2; A camera is arranged above the image acquisition area for shooting target images and spectrum images containing shrimp bodies and sending them to the shrimp body grading device based on deep learning; a door plate control mechanism is arranged below the first panel for controlling the opening and closing of the corresponding door plate in the N areas according to the quality grade obtained by the shrimp body grading device based on deep learning.

8. The deep learning-based shrimp body grading system of claim 7, wherein, A second panel is further included, the second panel is below the first panel, and N shrimp falling holes corresponding to the falling holes are arranged on the second panel; The support and the shell are connected by a gear transmission mechanism, the gear transmission mechanism comprises meshing gears and a gear ring, the gear transmission mechanism is installed on the second panel, and the gear ring is installed on the inner wall of the shell. 9.A non-transitory computer-readable storage medium having stored thereon a computer program, characterized in that, The computer program is executed by the processor to realize the shrimp body grading method based on deep learning according to any one of claims 1 to 4.

Images