A grouper image classification and recognition method

Abstract

The application discloses a grouper image classification and recognition method, which combines a Transformer and a CNN, and in steps one and two, on the basis of collecting images of grouper in different states in different environments, data augmentation is applied to expand the data set; in step three of model construction, the characteristics that the CNN is good at extracting the relationship of adjacent pixel points and the characteristics that the Transformer is good at extracting the relationship of pixels far apart are utilized respectively, so that the spot distribution of the grouper can be better acquired to resist the influence of body color on recognition when the grouper changes color under stress. In addition, in the local enhanced feedforward network module, after dimension reduction is performed on the parameter matrix, convolution is applied to sampling, so that more information can be obtained, and the characteristics of different varieties can be extracted. The application can more accurately recognize different varieties of grouper, and provides a reliable method for grouper identification, germplasm identification, protection and improvement.

Description

Technical Field

[0001] This invention relates to the technical field of aquatic animal classification research, specifically to a method for classifying and recognizing grouper images. Background Technology

[0002] Grouper is a world-renowned and prized marine fish, belonging to the order Perciformes, family Serranidae, and subfamily Epinephelinae. It comprises 15 genera and 159 species, of which approximately 46 are distributed along the mainland coast. Grouper is widely distributed in the tropical and subtropical waters of the Pacific and Indian Oceans. Its flesh is delicious and nutritious, making it highly popular with consumers and a significant marine aquaculture species in coastal areas of my country, thus possessing substantial economic value.

[0003] Currently, the main types of grouper farmed in coastal areas of my country include the oblique-banded grouper (Epinephelus coioides), brown-spotted grouper (Epinephelus fuscoguttatus), saddle-banded grouper (Epinephelus lanceolatus), red-spotted grouper (Epinephelus akaara), cloud-patterned grouper (Epinephelus moara), leopard-gill sea bass (Plectropomus leopardus), humpback sea bass (Cromileptesaltivelis), and some hybrid grouper varieties, such as the tiger-dragon hybrid grouper (brown-spotted grouper ♀ × saddle-banded grouper ♂), commonly known as the pearl grouper, the golden tiger hybrid grouper (brown-spotted grouper ♀ × blue-bodied large-spotted grouper ♂), and the cedar tiger hybrid grouper (brown-spotted grouper ♀ × clear-water grouper ♂), etc.

[0004] Due to the large number of grouper species, and the fact that some varieties are quite similar in appearance, making them difficult to distinguish, especially with the rise of hybrid grouper farming in recent years, consumers often struggle to identify different types. Furthermore, the price difference between different grouper species is significant; for example, brown-spotted grouper costs approximately twice as much per kilogram as pearl grouper, despite their very similar appearance. Some vendors, seeking exorbitant profits, intentionally label pearl grouper as brown-spotted grouper, a practice very common in seafood markets along my country's coast and inland. Because consumers have limited knowledge of grouper species, they are frequently deceived. Therefore, developing a grouper species image recognition system is highly relevant to the market, helping consumers identify common grouper varieties. In addition, such a system would also benefit researchers engaged in grouper classification and genetic breeding.

[0005] Under stress, grouper easily change color and their spot features become less prominent, further reducing the differences between different species. Therefore, for many grouper species, the spatial relationships between pixels are indispensable. However, convolutional neural network methods commonly used in image recognition, even with residual connections, can only obtain limited spatial contextual information, and their application to grouper recognition is not very effective. Therefore, more effective methods are needed to better distinguish different grouper species based on their morphological characteristics. Summary of the Invention

[0006] The technical problem to be solved by this invention is to provide a grouper image classification and recognition method that can quickly and conveniently help users obtain grouper species information.

[0007] To solve the above-mentioned technical problems, the technical solution adopted by the present invention is as follows:

[0008] A method for classifying and recognizing grouper images, characterized by comprising the following steps:

[0009] Step 1: Collect images of grouper and construct a dataset;

[0010] Step 2: Expand the number of grouper images using data augmentation. By augmenting a single image into multiple image copies, the number of training samples is increased, thereby improving the network's generalization ability and reducing overfitting.

[0011] Step 3: Construct the model, which includes the following data processing procedures:

[0012] The image augmented in step two is processed by the computing module. The output of the computing module is divided into four equal parts of 2x2, and then converted into patch token vectors (first converting the matrix into a vector, and then concatenating each converted vector together), ready to be used as input to Transformer (a deep learning model).

[0013] The obtained patch tokens vector is processed through three sets of data processing modules. Each set of data processing modules includes, in sequence, a normalization processing module, a multi-head self-attention module, a residual connection and normalization processing module, and a local enhancement feedforward network module.

[0014] Step 4: Train the model from Step 3 and deploy the trained model as a backend service.

[0015] Preferably, in step one, the image will be scaled to a size of 384x384 pixels.

[0016] Preferably, in step two, the data augmentation methods include: horizontal / vertical flipping of the image, random cropping, random scaling, random rotation transformation, and random gamma transformation.

[0017] The upper and lower limits of random scaling are 0.5-3, the angle range of random rotation transformation is 0-60 degrees, and the upper and lower limits of γ for gamma transformation are 0 and 1.5.

[0018] Preferably, the computation module in step three consists of convolution, batch normalization, and max pooling.

[0019] The formula is as follows:

[0020] x' = MaxPool(BN(Conv(x)))

[0021] In the formula: Conv is convolution, BN is batch normalization, and MaxPool is max pooling.

[0022] Preferably, in step three, the structure of the local enhancement feedforward network module is as follows:

[0023] Results obtained from multi-head self-attention module Divided into representative class names (generally called "class", such as Tiger Dragon Grouper, Golden Tiger Grouper) (h is a custom name, taken from the first letter of head, indicating it comes from the multi-head self-attention module in the previous step) and represents the patch token. (h is the same as above, p is taken from the first letter of patch token), where Extending to higher dimensions through linear mapping yields... (p is the same as above, l1 is taken from Linear1, indicating that it is the result of applying Linear1), and then it is restored to a two-dimensional image through dimension recovery. (p is the same as above, s represents the result after Spatial Restore), then apply convolution to obtain (p is the same as above, c indicates the result after convolution), then flatten to get (p is the same as above, f represents the result after flattening), then linearly map it back to the original dimension to obtain... (p is the same as above, l2 indicates the result after Lienar2), finally, and representing class name information. Combined; after each linear mapping and convolution, batch normalization is applied, and the GELU activation function is used, as shown in the following formula:

[0024]

[0025]

[0026]

[0027]

[0028]

[0029]

[0030]

[0031] In the formula, Linear1 and Linear2 represent the first and second linear mappings, respectively; SpatialRestore is image restoration; Concat is the stitching operation; Conv is the convolution operation; Flatten is the matrix flattening operation; BN is batch normalization; and Split is vector splitting.

[0032] Combine the three Transformer modules The output is fed into a module consisting of a multi-head self-attention module and a feedforward network to obtain the final result.

[0033] Preferably, in step three, AdamW is selected as the optimizer function, a piecewise decay learning rate is applied, ReLU is selected as the activation function, and a Dropout layer (a type of neural network layer used to reduce overfitting) is connected after Dense (a densely connected layer). The coefficient is 0.25.

[0034] Preferably, in step four, the model is sized and trimmed to reduce its volume and speed up the inference process.

[0035] Preferably, in step one: images of different species of grouper are collected in different environments and under different conditions.

[0036] Compared with the prior art, the present invention has the following beneficial effects:

[0037] 1. This invention develops a grouper species image recognition system. By taking a picture and uploading it to the system, the corresponding grouper species can be identified, which can quickly and conveniently help users obtain grouper species information.

[0038] 2. This invention combines Transformer (a deep learning model using self-attention mechanism, commonly used in natural language processing and computer vision) and CNN (a deep learning model using convolutional kernels, commonly used in computer vision and speech recognition) for image recognition of grouper.

[0039] In steps one and two of this invention, based on the acquisition of grouper images in different environments and under different conditions, including normal underwater conditions, underwater stress conditions, normal out-of-water conditions, and out-of-water stress conditions, data augmentation is applied to expand the dataset. In step three, in model construction, the advantages of CNN in extracting relationships between neighboring pixels and Transformer in extracting relationships between distant pixels are utilized to better obtain the distribution of grouper spots, thus counteracting the influence of body color on recognition when grouper undergoes stress-induced color changes. Furthermore, in the local enhancement feedforward network module, after dimensionality reduction of the parameter matrix, convolution is applied for sampling to obtain more information, which is beneficial for extracting the characteristics of different species.

[0040] Among them, if CNN is used alone, it cannot handle tasks with complex body patterns such as grouper. For example, both the oblique-banded grouper and the brown-spotted grouper have large striped spots. The difference lies in the spatial relationship of these striped spots. It is difficult for CNN to learn these spatial features when used alone. On the other hand, if Transformer is used alone, the amount of data is not large enough, and Transformer has difficulty learning the local features of the spots.

[0041] 3. This invention can more accurately identify different grouper species, providing a reliable method for grouper identification, germplasm assessment, protection and improvement. Attached Figure Description

[0042] Figure 1 This is a schematic diagram illustrating the operating principle of the computing module of the present invention;

[0043] Figure 1 The English explanations are as follows: Convolution: Convolution; Max Pooling: Max Pooling; Split: 2x2 splitting; Embedding: Embedding layer; Patch tokens: Patch tokens, which are the sequence vectors of input Transformer.

[0044] Figure 2 This is a schematic diagram of the locally enhanced feedforward network module of the present invention;

[0045] Here, token represents a vector, and class token is a 0,1 vector encoded as the class name;

[0046] Figure 3 This is a schematic diagram showing the result displayed on the user interface of the software after the user inputs an image of a red-spotted grouper during the use of the software constructed in this invention.

[0047] Figure 4This is a schematic diagram showing the results displayed on the user interface of the software after the user inputs the image of the tiger-dragon hybrid spot into the software during use.

[0048] Figure 5 This is a schematic diagram showing the result displayed on the user interface of the software after the user inputs a golden tiger stripe image into the software during use. Detailed Implementation

[0049] The present invention will be further described below with reference to embodiments.

[0050] Example:

[0051] The grouper image classification and recognition method in this embodiment includes the following steps:

[0052] Step 1: Collect images of different species of grouper (slanted grouper, brown-spotted grouper, saddle-banded grouper, red-spotted grouper, leopard-gill spiny perch, humpback perch, pearl grouper, golden tiger hybrid grouper, and fir tiger hybrid grouper) in different environments and under different conditions. The environments include aquaculture ponds, vegetable markets, aquatic product markets, restaurant tanks, etc. The conditions include normal conditions in water, under stress conditions in water, normal conditions out of water, and under stress conditions out of water. Filter out unclear images, resize the images to 384x384, and construct a dataset.

[0053] Step 2: Since the number of grouper images is relatively small compared to the training sample size required by the deep neural network, data augmentation is used to expand the number of grouper images. By augmenting a single image into multiple image copies, the number of training samples is greatly increased, thereby improving the generalization of the network and reducing overfitting.

[0054] Data augmentation methods specifically include: horizontal / vertical flipping of images, random cropping, random scaling, random rotation transformation, and random gamma transformation;

[0055] The upper and lower limits of random scaling are 0.5-3, the angle range of random rotation transformation is 0-60 degrees, and the upper and lower limits of γ for gamma transformation are 0 and 1.5.

[0056] Step 3: Construct the model, which includes the following data processing procedures:

[0057] The image augmented in step two is processed by a computation module, which consists of convolution (Conv), batch normalization (BN), and max-pooling (MaxPool). The formula is:

[0058] x' = MaxPool(BN(Conv(x)))

[0059]

[0060] In the formula: Conv is convolution, BN is batch normalization, and MaxPool is max pooling;

[0061] The output of the computation module is then divided into four equal parts of 2x2, and then converted into a patch tokens vector, ready to be used as input to the Transformer, as shown in Figure 1.

[0062] The obtained patch tokens vector is processed through three sets of data processing modules. Each set of data processing modules includes, in sequence, a normalization processing module, a multi-head self-attention module, a residual connection and normalization processing module, and a local enhancement feedforward network module.

[0063] The aforementioned standardization processing module, residual connection, and standardization processing module employ conventional processing methods. Among them, the self-attention module has three basic variables: Q (Query), K (Key), and V (Value). Attention mimics the human brain's attention mechanism when viewing content, assigning weights to each part of the content to represent the degree of importance that needs attention. Self-attention is a variant of attention, adept at extracting correlations within data or features. The self-attention calculation formula is:

[0064] q i =W Q a i

[0065] k i =W K a i

[0066] v i =W V a i

[0067] a i =Wx i

[0068]

[0069]

[0070] In the formula, W represents the trainable parameters (Weight), Attention is the common name for the calculated result, Q is the Query at the current position, K is the Key at a certain position in the entire sequence, and V is the Value at the current position. First, the transposes of Q and K are multiplied by a dot product, then divided by... Here, dk is the first dimension of the sequence. The result is processed by softmax, multiplied by V, and the attention values ​​at each position are summed up as the self-attention value at that position.

[0071] Multi-head self-attention, building upon self-attention, changes the single Q, K, V parameter to multiple Q, K, V parameters. The number of parameters is the number of heads, calculated using the following formula:

[0072] MultiHead(Q,K,V)=Concat(head1,...,head h W O

[0073]

[0074]

[0075] In the formula: MultiHead is multi-head attention, Concat is directly concatenating the first and last parts of the matrix, and W O It is a parameter matrix of a linearly transformed fully connected layer. The parameters representing Q, K, and V respectively, hi i The first 'i' represents the position of the token in the sequence, and the second 'i' represents which group in the long position.

[0076] The structure of the locally enhanced feedforward network is as follows:

[0077] Results obtained from multi-head self-attention module Divided into representative class names (generally called "class", such as Tiger Dragon Grouper, Golden Tiger Grouper) (h is a custom name, taken from the first letter of head, indicating it comes from the multi-head self-attention module in the previous step) and represents the patch token. (h is the same as above, p is taken from the first letter of patch token), where Extending to higher dimensions through linear mapping yields... (p is the same as above, l1 is taken from Linear1, indicating that it is the result of applying Linear1), and then it is restored to a two-dimensional image through dimension recovery. (p is the same as above, s represents the result after Spatial Restore), then apply convolution to obtain (p is the same as above, c indicates the result after convolution), then flatten to get (p is the same as above, f represents the result after flattening), then linearly map it back to the original dimension to obtain... (p is the same as above, l2 indicates the result after Lienar2), finally, and representing class name information. Combined; after each linear mapping and convolution, batch normalization is applied, and the GELU activation function is used, as shown in the following formula:

[0078]

[0079]

[0080]

[0081]

[0082]

[0083]

[0084]

[0085] In the formula, Linear1 and Linear2 represent the first and second linear mappings, respectively; SpatialRestore is image restoration; Concat is the stitching operation; Conv is the convolution operation; Flatten is the matrix flattening operation; BN is batch normalization; and Split is vector splitting; as shown in Figure 2.

[0086] Three Transformers (a type of deep learning model) The output is fed into a module consisting of a multi-head self-attention module and a feedforward network (this feedforward network is a regular feedforward network) to obtain the final result.

[0087] Step 4: Train the model from Step 3 and deploy the trained model as a backend service. It can serve as the backend interface for software and mini-programs, allowing common programming techniques to be used to build a software system that runs on the corresponding user end (such as a mobile phone or computer) to create the software interface.

[0088] In step three above, AdamW is selected as the optimizer function, a piecewise decay learning rate is applied, ReLU is selected as the activation function, and a Dropout layer (a type of neural network layer used to reduce overfitting) is connected after Dense (a densely connected layer). The coefficient is 0.25.

[0089] The hyperparameters used are as follows:

[0090] Hyperparameter name hyperparameter values filter size 3x3 filter count 64 stride 2 head 3 metrics accuracy warmup ratio 0.1 optimizer AdamW learning rate 5e-3 weight decay 1.5e-3

[0091] In step four above, the model is sized and unnecessary branches and leaves are pruned to reduce the model volume and speed up the inference process.

[0092] In the subsequent process, this grouper image classification and recognition method can be developed into software. Users can upload images or take photos with a camera and upload them. The software receives the uploaded images, converts them, and then inputs them into the software's neural network. Finally, the recognition results are displayed on the software's user interface.

[0093] The software will display the species with the highest confidence scores during runtime, such as: saddle-banded grouper = 0.85, brown-spotted grouper = 0.1, and golden tiger grouper = 0.05. Figures 3 to 5 The image shows a diagram illustrating the results displayed on the user interface after a user inputs several grouper images into the software. Below each grouper image, the identified species and their corresponding score are displayed.

[0094] The above embodiments of the present invention are not intended to limit the scope of protection of the present invention. The implementation of the present invention is not limited thereto. All other modifications, substitutions or alterations made to the above structure of the present invention based on the above content of the present invention, in accordance with ordinary technical knowledge and common practice in the field, without departing from the basic technical idea of ​​the present invention, shall fall within the scope of protection of the present invention.

Claims

1. A method for classifying and recognizing grouper images, characterized in that, Includes the following steps: Step 1: Collect images of grouper and construct a dataset; Step 2: Expand the number of grouper images using data augmentation. By augmenting a single image into multiple image copies, the number of training samples is increased, thereby improving the network's generalization ability and reducing overfitting. Step 3: Construct the model, which includes the following data processing procedures: The image that has undergone data augmentation and amplification in step two is processed by the computing module. The output of the computing module is divided into four equal parts of 2x2, and then converted into patch tokens vectors, which are ready to be used as input to the Transformer. The obtained patch tokens vector is processed by three sets of data processing modules. Each set of data processing modules includes, in sequence, a normalization processing module, a multi-head self-attention module, a residual connection and normalization processing module, and a local enhancement feedforward network module. The structure of the locally enhanced feedforward network module is as follows: (Results obtained from the multi-head self-attention module) Divided into class names and representing the patch token in Extending to higher dimensions through linear mapping yields... Then, the image is restored to a two-dimensional image using dimensional recovery. Then apply convolution to obtain Then flatten it to get Then linearly map back to the initial dimension to obtain Finally, and the information representing the class name. Combined; after each linear mapping and convolution, batch normalization is applied, and the GELU activation function is used to combine the three Transformers. The output is fed into a module consisting of a multi-head self-attention module and a feedforward network to obtain the final result. Step 4: Train the model from Step 3 and deploy the trained model as a backend service.

2. The grouper image classification and recognition method according to claim 1, characterized in that: In step one, the image will be scaled to 384x384 pixels.

3. The grouper image classification and recognition method according to claim 1, characterized in that: In step two, data augmentation methods include: horizontal / vertical flipping of the image, random cropping, random scaling, random rotation transformation, and random gamma transformation.

4. The grouper image classification and recognition method according to claim 3, characterized in that: in, The upper and lower limits of random scaling are 0.5-3, the angle range of random rotation transformation is 0-60 degrees, and the upper and lower limits of γ for gamma transformation are 0 and 1.

5.

5. The grouper image classification and recognition method according to claim 1, characterized in that: The computation module in step three consists of convolution, batch normalization, and max pooling.

6. The grouper image classification and recognition method according to claim 1, characterized in that: In step three, AdamW is selected as the optimizer function, a piecewise decay learning rate is applied, ReLU is selected as the activation function, and a Dropout layer with a coefficient of 0.25 is connected after Dense.

7. The grouper image classification and recognition method according to claim 1, characterized in that: In step four, the model is sized and trimmed to reduce its volume and speed up inference.

8. The grouper image classification and recognition method according to claim 1, characterized in that: Step one involves collecting images of different species of grouper in different environments and under different conditions.

Images