Automatic aquatic product sorting system based on machine vision

Abstract

The invention relates to the field of aquatic product sorting, and particularly discloses an automatic aquatic product sorting system based on machine vision, which comprises the following steps: firstly, acquiring aquatic product appearance image data and aquatic product basic data acquired by a high-definition camera, and then, performing feature extraction and correlation analysis on the two data by using a deep learning technology to obtain an aquatic product sorting result; and finally, aquatic product types are generated through a generator, so that manual intervention is reduced, and the processing speed and accuracy of aquatic product sorting are improved.

Description

Technical Field

[0001] The present application relates to the field of aquatic product sorting, and more specifically, to an automatic aquatic product sorting system based on machine vision. Background Art

[0002] Aquatic products refer to edible organisms from water bodies such as oceans, lakes, and rivers, including fish, shellfish, crustaceans, and mollusks. They are an important source of nutrition, rich in high-quality protein, vitamins, minerals, and Omega-3 fatty acids, and are extremely beneficial to human health. Different types of aquatic products have their own unique flavors and nutritional components, and are widely used in different food cultures.

[0003] Traditional aquatic product sorting usually relies on manual operation, which is not only time-consuming and labor-intensive, but also prone to sorting errors. As the demand for aquatic product processing increases, the demand for efficient and accurate automated sorting systems is also growing.

[0004] Therefore, an automatic sorting system for aquatic products based on machine vision is desired. Summary of the invention

[0005] In order to solve the above technical problems, this application is proposed. The embodiment of this application provides an automatic sorting system for aquatic products based on machine vision, which first obtains aquatic product appearance image data and basic data of aquatic products collected by a high-definition camera, then uses deep learning technology to perform feature extraction and correlation analysis on the two, and finally generates aquatic product types through a generator, thereby reducing manual intervention and improving the processing speed and accuracy of aquatic product sorting.

[0006] According to one aspect of the present application, there is provided a machine vision-based automatic sorting system for aquatic products, comprising:

[0007] Aquatic product data acquisition module, used to acquire aquatic product appearance image data and basic data of aquatic products collected by a high-definition camera;

[0008] Aquatic product data extraction module, used to extract optimized aquatic product appearance-related feature vectors and aquatic product basic semantics-related feature vectors from the aquatic product appearance image data collected by the high-definition camera and the basic data of the aquatic product;

[0009] The aquatic product category generation module is used to generate aquatic product categories based on the optimized aquatic product appearance associated feature vector and the aquatic product basic semantic associated feature vector.

[0010] Compared with the prior art, the present application provides an automatic sorting system for aquatic products based on machine vision, which first obtains the appearance image data of aquatic products and the basic data of aquatic products collected by a high-definition camera, and then uses deep learning technology to perform feature extraction and correlation analysis on the two, and finally generates aquatic product types through a generator, thereby reducing manual intervention and improving the processing speed and accuracy of aquatic product sorting. BRIEF DESCRIPTION OF THE DRAWINGS

[0011] By describing the embodiments of the present application in more detail in conjunction with the accompanying drawings, the above and other purposes, features and advantages of the present application will become more apparent. The accompanying drawings are used to provide a further understanding of the embodiments of the present application and constitute a part of the specification. Together with the embodiments of the present application, they are used to explain the present application and do not constitute a limitation of the present application. In the accompanying drawings, the same reference numerals generally represent the same components or steps.

[0012] Figure 1 It is a block diagram of an automatic aquatic product sorting system based on machine vision according to an embodiment of the present application.

[0013] Figure 2 It is a block diagram of aquatic product data extraction module in a machine vision-based automatic sorting system for aquatic products according to an embodiment of the present application.

[0014] Figure 3 It is a block diagram of a unit for extracting the appearance features of aquatic products in a machine vision-based automatic sorting system for aquatic products according to an embodiment of the present application.

[0015] Figure 4 It is a block diagram of a subunit for encoding the appearance target features of aquatic products in a machine vision-based automatic sorting system for aquatic products according to an embodiment of the present application.

[0016] Figure 5 This is a block diagram of a module for generating aquatic product types in a machine vision-based automatic aquatic product sorting system according to an embodiment of the present application. DETAILED DESCRIPTION

[0017] Various exemplary embodiments, features and aspects of the present application will be described in detail below with reference to the accompanying drawings. The same reference numerals in the accompanying drawings represent elements with the same or similar functions. Although various aspects of the embodiments are shown in the accompanying drawings, the drawings are not necessarily drawn to scale unless otherwise specified.

[0018] The word “exemplary” is used exclusively herein to mean “serving as an example, example, or illustration.” Any embodiment described herein as “exemplary” is not necessarily to be construed as preferred or advantageous over other embodiments.

[0019] In addition, in order to better illustrate the present application, numerous specific details are given in the following specific embodiments. It should be understood by those skilled in the art that the present application can also be implemented without certain specific details. In some examples, methods, means, components and circuits well known to those skilled in the art are not described in detail in order to highlight the subject matter of the present application.

[0020] In addition, the terms "first" and "second" are used for descriptive purposes only and should not be understood as indicating or implying relative importance or implicitly indicating the number of the indicated technical features. Thus, a feature defined as "first" or "second" may explicitly or implicitly include one or more of the features. In the description of this application, the meaning of "plurality" is two or more, unless otherwise clearly and specifically defined.

[0021] Figure 1 Schematic diagram of the automatic sorting system of aquatic products based on machine vision in the embodiment of the present application. Figure 1 As shown, according to the automatic sorting system 100 of aquatic products based on machine vision according to the embodiment of the present application, it includes: an aquatic product data acquisition module 110, which is used to acquire the appearance image data of the aquatic product collected by the high-definition camera and the basic data of the aquatic product; an aquatic product data extraction module 120, which is used to extract the optimized aquatic product appearance association feature vector and the aquatic product basic semantic association feature vector from the aquatic product appearance image data collected by the high-definition camera and the basic data of the aquatic product; an aquatic product type generation module 130, which is used to generate the aquatic product type based on the optimized aquatic product appearance association feature vector and the aquatic product basic semantic association feature vector.

[0022] In the above-mentioned aquatic product automatic sorting system 100 based on machine vision, the aquatic product data acquisition module 110 is used to obtain the aquatic product appearance image data and the basic data of the aquatic product collected by the high-definition camera. It should be understood that aquatic products refer to edible organisms that can be caught or cultivated from water bodies such as oceans, lakes and rivers, including various types of fish, shellfish, crustaceans and mollusks. They are not only an important source of nutrition for humans, but also rich in high-quality protein, vitamins, minerals and Omega-3 fatty acids, which play an important role in promoting human health. Each aquatic product has its own unique taste and nutritional components and is widely used in different food cultures. Traditional aquatic product sorting mainly relies on manual operation, which is not only time-consuming and labor-intensive, but also prone to errors. With the increase in the demand for aquatic product processing, people's demand for systems that can perform automated sorting efficiently and accurately is also growing. Therefore, in the technical solution of the present application, by acquiring the aquatic product appearance image data and the basic data of aquatic products collected by the high-definition camera, and combining deep learning technology, the types of aquatic products are generated to significantly reduce manual intervention and improve the processing speed and accuracy of aquatic product sorting.

[0023] Specifically, the purpose of acquiring the appearance image data of aquatic products collected by high-definition cameras and the basic data of aquatic products is to achieve efficient and accurate automatic sorting of aquatic products. The core of this process is to use multi-dimensional data to comprehensively analyze and identify the types of aquatic products. Among them, the appearance image data provided by the high-definition camera has high resolution and details, which can clearly capture the visual characteristics of aquatic products, such as shape, color, texture, etc., which are key visual information for judging the types of aquatic products. At the same time, the basic data of aquatic products, including information such as their type, size, weight, and origin, provides additional semantic information. By comprehensively utilizing high-definition image data and basic data, the system can obtain a more comprehensive and accurate description of the characteristics of aquatic products by fusing information from these different sources. This multi-level and multi-dimensional data fusion enables the aquatic product sorting system to better cope with complex situations in practical applications, thereby achieving a more efficient and accurate automatic sorting process.

[0024] In the above-mentioned aquatic product automatic sorting system 100 based on machine vision, the aquatic product data extraction module 120 is used to extract optimized aquatic product appearance-related feature vectors and aquatic product basic semantic-related feature vectors from the aquatic product appearance image data collected by the high-definition camera and the basic data of the aquatic product. It should be understood that the comprehensive processing of the appearance feature vector and the basic semantic feature vector can more comprehensively describe the properties of the aquatic product, thereby improving the overall performance of the sorting system. This data fusion method can process and eliminate errors that may be caused by the ambiguity of image data or the insufficiency of basic data.

[0025] Figure 2 FIG. 1 is a block diagram of a water product data extraction module in a water product automatic sorting system based on machine vision according to an embodiment of the present application. Figure 2 As shown, in a specific embodiment of the present application, the aquatic product data extraction module 120 includes: an aquatic product appearance feature extraction unit 121, used to perform feature extraction on the aquatic product appearance image data collected by the high-definition camera to obtain the optimized aquatic product appearance related feature vector; an aquatic product basic data feature extraction unit 122, used to perform feature extraction on the basic data of the aquatic product to obtain the aquatic product basic semantic related feature vector.

[0026] It should be understood that the images captured by high-definition cameras contain rich visual features, such as the shape, color, texture and other details of aquatic products, which are essential for distinguishing different types of aquatic products. However, the original image data is too complex and large, and direct use may lead to inefficient processing and information redundancy. The process of feature extraction aims to extract the most representative and discriminative features from the image through computer vision techniques, such as convolutional neural networks (CNN). These features can be local, such as edges, textures, etc., or global, such as overall shape and color distribution. Through feature extraction, the system can quickly convert aquatic product image data into a digital representation that can be used for decision-making and operation while collecting and processing it in real time. This conversion not only helps to improve the performance and response speed of the sorting system, but also can cope with the challenges of appearance changes and lighting conditions of different aquatic products, thereby ensuring the stability and reliability of the entire automatic sorting process.

[0027] Furthermore, the basic data of aquatic products include key information such as species, weight, size, and origin, which plays an important role in identifying and classifying aquatic products. However, the original basic data is usually unstructured and irregular, and direct use may lead to difficult and inefficient information processing. The process of feature extraction uses data processing techniques, such as semantic encoders or feature extraction models, to convert basic data into a set of meaningful numerical features. These features can be numerical descriptions, such as mapping different types of aquatic products to numerical labels, or structured attribute descriptions, such as converting the weight and size of aquatic products into a unified measurement unit. In this way, the system is able to more easily compare and analyze the differences and similarities between different aquatic products during processing, thereby improving the accuracy of classification and identification.

[0028] Figure 3 FIG. 1 is a block diagram of a unit for extracting the appearance features of aquatic products in a machine vision-based automatic sorting system for aquatic products according to an embodiment of the present application. Figure 3 As shown, in a specific embodiment of the present application, the aquatic product appearance feature extraction unit 121 includes: an aquatic product appearance multi-scale feature encoding subunit 1211, used to perform multi-scale feature encoding on the aquatic product appearance image data collected by the high-definition camera to obtain a plurality of aquatic product appearance feature maps; an aquatic product appearance target feature encoding subunit 1212, used to perform target feature encoding on the plurality of aquatic product appearance feature maps to obtain the optimized aquatic product appearance associated feature vector.

[0029] It should be understood that high-definition cameras can capture the appearance of aquatic products with high resolution and details, but a single feature extraction may not fully reflect the complex visual characteristics of aquatic products. Multi-scale feature encoding processes image data at different scales and spatial resolutions to obtain multiple levels of visual information. For example, by using multiple convolutional layers and pooling layers of a convolutional neural network (CNN), local and global features of aquatic products can be extracted at different scales. These feature maps can include low-level edge and texture features to high-level overall shape and color distribution, and feature maps at each scale help capture different levels of image information. This multi-scale feature encoding method effectively extracts the appearance features of aquatic products from the original image and represents them in a quantifiable and comparable way. Such a multi-level representation not only helps the system to distinguish different types of aquatic products more accurately, but also maintains stable recognition capabilities in the face of lighting changes, viewing angle changes, or variability in the aquatic products themselves.

[0030] Furthermore, target feature encoding is performed on multiple aquatic product appearance feature maps to obtain optimized aquatic product appearance-related feature vectors in order to transform the complex and multidimensional visual information extracted from the image into a compact and operable digital representation, thereby enhancing the classification and recognition capabilities of the automatic sorting system. The feature maps obtained by multi-scale feature encoding cover different levels of visual information, such as edges, textures, shapes, and colors. However, these feature maps are often high-dimensional and lengthy, and their direct use may lead to increased computational complexity and difficulty in information processing. The process of target feature encoding aims to extract the most representative and recognizable features from multiple feature maps and integrate them into a unified feature vector. Through this encoding, the system can simplify the complex information of the original image into a structured, high-dimensional feature vector, which not only has comprehensive visual features, but also has high efficiency for subsequent analysis and decision-making. Among them, feature encoding usually involves dimensionality reduction, feature selection, and aggregation techniques. The optimized aquatic product appearance-related feature vector obtained in the end can effectively represent the overall appearance characteristics of aquatic products and provide reliable decision support in different sorting and classification tasks. This encoding method not only reduces the feature dimension and computational burden, but also enhances the discriminability and robustness of the features, enabling the system to maintain high performance under various conditions.

[0031] In a specific embodiment of the present application, the aquatic product appearance multi-scale feature encoding subunit 1211 includes: dividing the aquatic product appearance image data collected by the high-definition camera into multiple aquatic product appearance images; and passing the multiple aquatic product appearance images through a aquatic product appearance convolutional neural network model with multi-feature extraction capabilities to obtain the multiple aquatic product appearance feature maps.

[0032] It should be understood that dividing the image can reduce the amount of data for each processing unit, making the consumption of computing resources more efficient. The processing of large image data may cause excessive computing burden, especially in real-time sorting systems, which may cause delays or decreased processing speed. By dividing the image into multiple small aquatic product appearance images, local processing and analysis can be carried out more easily, thereby improving the response speed and processing power of the system. Dividing the image helps to improve resolution and recognition accuracy. Each small aquatic product appearance image can focus on the characteristics of a single aquatic product, thereby capturing and analyzing its appearance details more carefully. This localized processing can effectively reduce background noise and interference, allowing the system to more accurately identify and classify each aquatic product, which is particularly important in complex or crowded scenes.

[0033] Furthermore, multiple aquatic product appearance images are processed by a convolutional neural network (CNN) model with multi-feature extraction capabilities to obtain multiple aquatic product appearance feature maps in order to comprehensively and accurately capture the visual features of aquatic products and improve the recognition and classification capabilities of the automatic sorting system. Aquatic product appearance images contain rich visual information, such as shape, color, texture, and details, but this information often has a complex spatial structure and hierarchical relationship. Traditional feature extraction methods may not be able to fully mine and integrate this information, while the multi-feature extraction capability of convolutional neural networks can effectively solve this problem. Convolutional neural networks have multiple convolutional layers and pooling layers, which can automatically learn and extract local and global features in images. Specifically, the convolutional layer can detect low-level features such as edges, textures, and shapes in the image through filters (convolution kernels), and as the network level goes deeper, these low-level features will gradually combine into more complex high-level features, such as the overall shape and color pattern of aquatic products. The pooling layer further extracts the most significant feature information in the image by reducing the dimension and compressing the feature map, while reducing the redundancy and noise of the data. Through this multi-feature extraction method, the convolutional neural network can generate multiple aquatic product appearance feature maps, which not only contain visual information at different levels, but also retain important details and patterns in the image. The generation of such multi-feature maps can help the system understand the appearance characteristics of aquatic products more comprehensively, thereby improving the accuracy and robustness of classification. Specifically, each hybrid convolution layer of the aquatic product appearance convolutional neural network model with multi-feature extraction capability includes a parallel first convolution branch structure, a second convolution branch structure, a third convolution branch structure, and a fourth convolution branch structure, as well as a multi-scale fusion structure connected to the first to fourth convolution branch structures, wherein the first convolution branch uses a first convolution kernel with a first size, the second convolution branch uses a second convolution kernel with a first size and a first void rate, the third convolution branch uses a third convolution kernel with a first size and a second void rate, and the fourth convolution branch uses a fourth convolution kernel with a first size and a third void rate.More specifically, each mixed convolution layer of the aquatic product appearance convolutional neural network model with multi-feature extraction capability performs the following operations on the input data in the forward pass of the layer: using a first convolution kernel with a first size to convolutionally encode the multiple aquatic product appearance images to obtain a first scale feature map; using a second convolution kernel with a first void ratio to convolutionally encode the multiple aquatic product appearance images to obtain a second scale feature map; using a third convolution kernel with a second void ratio to convolutionally encode the multiple aquatic product appearance images to obtain a third scale feature map; using a fourth convolution kernel with a third void ratio to convolutionally encode the multiple aquatic product appearance images to obtain a fourth scale feature map. , wherein the first convolution kernel, the second convolution kernel, the third convolution kernel and the fourth convolution kernel have the same size, and the second convolution kernel, the third convolution kernel and the fourth convolution kernel have different void rates; the first scale feature map, the second scale feature map, the third scale feature map and the fourth scale feature map are aggregated along the channel dimension to obtain an aggregated feature map; the aggregated feature map is pooled to generate a pooled feature map; and the pooled feature map is activated to generate an activated feature map; wherein the output of the last layer of the aquatic product appearance convolutional neural network model with multi-feature extraction capability is the multiple aquatic product appearance feature maps.

[0034] Figure 4 FIG. 1 is a block diagram of a subunit for encoding the appearance target features of aquatic products in a machine vision-based automatic sorting system for aquatic products according to an embodiment of the present application. Figure 4 As shown, in a specific embodiment of the present application, the aquatic product appearance target feature encoding subunit 1212 includes: a secondary subunit 1212-1 for aquatic product appearance target detection, which is used to pass the multiple aquatic product appearance feature maps through aquatic product appearance target detection network to obtain multiple aquatic product appearance regions of interest; a secondary subunit 1212-2 for aquatic product appearance salient target detection, which is used to pass the multiple aquatic product appearance regions of interest through aquatic product appearance salient target detector to obtain aquatic product appearance associated feature map; a secondary subunit 1212-3 for aquatic product appearance feature pooling, which is used to perform global mean pooling on the aquatic product appearance associated feature map to obtain aquatic product appearance associated feature vector; a secondary subunit 1212-4 for aquatic product appearance feature optimization, which is used to perform a transformer-like structural displacement correction based on hierarchical precipitation expression information on the aquatic product appearance associated feature vector based on the aquatic product basic semantic associated feature vector to obtain an optimized aquatic product appearance associated feature vector.

[0035] It should be understood that the aquatic product appearance feature map contains various visual features extracted from the image, such as shape, color and texture, but does not provide the precise location information of each aquatic product in the image. The role of the object detection network is to automatically identify and locate the region of interest of the aquatic product by learning effective feature representation and region proposal based on the feature map. The aquatic product appearance object detection network is usually based on deep learning models, such as region-based convolutional neural network (R-CNN), fast R-CNN (Fast R-CNN), region generation network (Region Proposal Network, RPN) and YOLO (You On ly Look Once). These network structures can extract candidate regions from the feature map or directly predict the bounding box of the aquatic product in an end-to-end manner to accurately grasp the position and size of the aquatic product. Through the object detection network, the system can effectively correspond the appearance features of each aquatic product to its actual position in the image. This precise positioning not only helps to improve the classification accuracy of the sorting system, but also reduces missed detection and false detection during real-time processing.

[0036] Furthermore, multiple regions of interest of aquatic product appearance are processed by a salient object detector of aquatic product appearance to obtain a feature map associated with the appearance of aquatic products, in order to further improve the accuracy of aquatic product recognition and classification in the image. After the region of interest (ROI) is accurately located by the object detection network, it contains detailed information of each aquatic product, but the visual features of these regions still need to be further processed by the salient object detector to extract more distinguishing and relevant features. The role of the salient object detector is to identify and extract the most significant visual features from a specific region of interest. These features are usually the most distinguishing parts of the target object and can help the system better understand the specific properties and appearance of aquatic products. The salient object detector emphasizes key features in the image, such as contours, color distribution or texture patterns, through algorithms, thereby generating a feature map associated with the appearance of aquatic products, which contains the most important visual information related to each aquatic product. In this way, the salient object detector can extract and strengthen the most recognizable features from multiple regions of interest, so that the final feature map can not only accurately represent the appearance characteristics of aquatic products, but also reduce the interference of background noise and irrelevant information. Specifically, each layer of the aquatic product appearance salient object detector is used to perform the following operations on the input data in the forward pass of the layer: convolution processing is performed on the input data using a first convolution kernel to obtain a first convolution feature map; convolution processing is performed on the first convolution feature map using a second convolution kernel to obtain a second convolution feature map, wherein the size of the first convolution kernel is larger than the size of the second convolution kernel; and, pooling processing is performed on the second convolution feature map to obtain a pooled feature map; and, activation processing is performed on the pooled feature map to obtain an activation feature map; wherein the output of the last layer of the aquatic product appearance salient object detector is the aquatic product appearance associated feature map, and the input of the first layer of the aquatic product appearance salient object detector is the multiple aquatic product appearance regions of interest.

[0037] Furthermore, global mean pooling is performed on the appearance-related feature maps of aquatic products to obtain the appearance-related feature vectors of aquatic products, aiming to transform the complex appearance features of each aquatic product into a feature vector with a fixed length and uniform expression, so as to facilitate the subsequent classification, recognition and comparison tasks. The appearance-related feature maps of aquatic products usually contain important visual information enhanced by the salient object detector, but the size and dimension of these feature maps may vary due to different aquatic products and their locations. Global mean pooling compresses each channel of the entire feature map by averaging it into a fixed-length feature vector, which can retain key feature information while reducing dimensional complexity. Specifically, global mean pooling can effectively extract the average value of each channel in the feature map, thereby capturing the overall statistical characteristics of each aquatic product appearance-related feature map. This method can not only reduce the feature dimension, but also retain important local information, especially those salient features that are emphasized in the salient object detection stage. Since the pooling operation is performed on the entire feature map, the resulting feature vector has a fixed length and stable expression ability, which is suitable for subsequent machine learning model input or feature comparison tasks.

[0038] In particular, in the technical solution of the present application, the aquatic product appearance-related feature vector is extracted from the image data collected by the high-definition camera, focusing on the spatial information and visual details of the image, fully reflecting the appearance characteristics and shape details of the aquatic product, and having strong spatial resolution and local detail information. In contrast, the basic data of aquatic products (such as species, weight, source, etc.) is processed by the semantic encoder of the basic data of aquatic products, and the generated basic semantic feature vector mainly focuses on the semantic information and classification characteristics of the data. These data are usually structured and relatively static in the time dimension, reflecting the basic information of aquatic products, such as variety and quality, but lacking the dynamic expression of spatial details and visual changes. This difference in feature processing leads to the inconsistency of the fine-grained distribution of the aquatic product appearance-related feature vector and the aquatic product basic semantic-related feature vector in the feature space, resulting in the abnormal stacking phenomenon of the fused aquatic product type generation feature vector on the boundary of the feature manifold. Therefore, in the technical solution of the present application, the aquatic product appearance-related feature vector is subjected to the simulated converter structure displacement correction based on the hierarchical precipitation expression information based on the aquatic product basic semantic-related feature vector to obtain the optimized aquatic product appearance-related feature vector.

[0039] Among them, based on the basic semantic association feature vector of aquatic products, the aquatic product appearance association feature vector is subjected to a simulated converter structure displacement correction based on hierarchical precipitation expression information to obtain an optimized aquatic product appearance association feature vector, including: calculating the position difference between the aquatic product appearance association feature vector and the aquatic product basic semantic association feature vector to obtain an aquatic product sorting shift position difference representation vector; after cascading the aquatic product appearance association feature vector and the aquatic product basic semantic association feature vector into an aquatic product sorting joint feature vector, the aquatic product sorting joint feature vector is input into a one-dimensional convolution layer to obtain an aquatic product sorting shift convolution coding representation vector; after using the first aquatic product sorting linear embedding weight matrix to perform aquatic product sorting linear embedding modulation coding on the aquatic product appearance association feature vector to obtain a linear modulation vector of the aquatic product appearance association feature vector, the aquatic product sorting joint feature vector is calculated. The linear modulation vector of the aquatic product appearance associated feature vector and the basic semantic associated feature vector of the aquatic product are differentially calculated by position to obtain the aquatic product sorting shift modulation differential representation vector; the aquatic product sorting shift differential representation vector is used as the value vector, the aquatic product sorting shift convolutional coding representation vector and the aquatic product sorting shift modulation differential representation vector are used as the query vector and the key vector, and the aquatic product sorting shift differential representation vector, the aquatic product sorting shift convolutional coding representation vector and the aquatic product sorting shift modulation differential representation vector are input into the shift corrector based on the simulated converter structure to obtain the aquatic product sorting multi-scale hierarchical fusion shift modulation weight vector; the position point multiplication between the aquatic product sorting multi-scale hierarchical fusion shift modulation weight vector and the aquatic product appearance associated feature vector is calculated to obtain the optimized aquatic product appearance associated feature vector.

[0040] The following optimization formula is used to represent that the aquatic product appearance associated feature vector is subjected to a transformer-like structural displacement correction based on hierarchical precipitation expression information based on the aquatic product basic semantic associated feature vector to obtain an optimized aquatic product appearance associated feature vector;

[0041] Wherein, the optimization formula is:

[0042]

[0043] V 2d =Conv 1D ([V1; V2])

[0044]

[0045]

[0046] V1′=V1⊙V d

[0047] Among them, V 1drepresents the vector of aquatic product sorting displacement according to position difference, V1 represents the feature vector associated with the appearance of aquatic products, V2 represents the feature vector associated with the basic semantics of aquatic products, V 2d represents the aquatic product sorting shift convolutional coding representation vector, V 3d represents the aquatic product sorting shift modulation differential representation vector, W1 represents the first aquatic product sorting linear embedding weight matrix, b1 represents the aquatic product sorting modulation bias vector, V d represents the multi-scale hierarchical fusion shift modulation weight vector for aquatic product sorting, V 3d T represents the transposed vector of the aquatic product sorting shift modulation differential representation vector, d represents the length of the aquatic product sorting shift convolutional coding representation vector, Indicates position difference, Conv 1D represents a one-dimensional convolutional layer, [V1; V2] represents the concatenation of the aquatic product appearance-related feature vector and the aquatic product basic semantic-related feature vector, Relu represents a linear rectification function, represents matrix multiplication, Softmax represents the normalized exponential function, ⊙ represents the position point multiplication, and V1' represents the optimized aquatic product appearance related feature vector.

[0048] In the technical solution of the present application, in the process of fusing the aquatic product appearance-related feature vector and the aquatic product basic semantic-related feature vector, the fine-grained inconsistency of the feature distribution between the aquatic product appearance-related feature vector and the aquatic product basic semantic-related feature vector is taken into consideration, which may cause the fused aquatic product type generation feature vector to have abnormal stacking on the boundary of the feature manifold. Based on this, in the technical solution of the present application, before fusing the aquatic product appearance-related feature vector and the aquatic product basic semantic-related feature vector to obtain the aquatic product type generation feature vector, the aquatic product appearance-related feature vector is subjected to a simulated transformer structure displacement correction based on hierarchical precipitation expression information based on the aquatic product basic semantic-related feature vector.

[0049] In the technical solution of the present application, a converter-like structural displacement correction method based on hierarchical precipitation expression information is proposed to address the problem of fine-grained inconsistency in feature distribution that occurs in the process of fusing aquatic product appearance-related feature vectors and aquatic product basic semantic-related feature vectors. The method aims to reduce the abnormal stacking of feature vectors generated by aquatic product types on the boundary of the feature manifold, so as to improve the robustness and classification performance of the model.

[0050] Specifically, the method first calculates the element-by-element difference between the aquatic product appearance-related feature vector and the aquatic product basic semantics-related feature vector to obtain an aquatic product sorting shift element-by-element differential representation vector, thereby capturing the difference between the two feature vectors. Then, the aquatic product appearance-related feature vector and the aquatic product basic semantics-related feature vector are cascaded into an aquatic product sorting joint feature vector and input into a one-dimensional convolutional layer to generate an aquatic product sorting shift convolutional coding representation vector. That is, the deep-level shift implicit information between the aquatic product appearance-related feature vector and the aquatic product basic semantics-related feature vector is captured by one-dimensional convolutional coding. At the same time, the aquatic product appearance-related feature vector is subjected to aquatic product sorting linear embedding modulation coding using the first aquatic product sorting linear embedding weight matrix to be a linear modulation vector of the aquatic product appearance-related feature vector, and the element-by-element difference between the linear modulation vector of the aquatic product appearance-related feature vector and the feature vector of the aquatic product basic semantics-related feature vector is calculated to obtain an aquatic product sorting shift modulation differential representation vector. The aquatic product sorting shift modulation differential representation vector is used to represent the positional expression of the shift information of the aquatic product appearance-related feature vector and the aquatic product basic semantics-related feature vector in the linear space. That is, in the technical solution of the present application, the aquatic product sorting shift position difference, the aquatic product sorting shift convolutional coding representation vector and the aquatic product sorting shift modulation differential representation vector respectively represent the explicit expression of the fine-grained distribution difference shift information between the aquatic product appearance-related feature vector and the aquatic product basic semantics-related feature vector in different feature spaces.

[0051] Next, the aquatic product sorting shift element-by-element differential representation vector is used as the value vector, the aquatic product sorting shift convolutional coding representation vector and the aquatic product sorting shift modulation differential representation vector are used as the query vector and the key vector respectively, and these vectors are input into the shift corrector based on the simulated transformer structure to generate the aquatic product sorting multi-scale hierarchical fusion shift modulation weight vector. In this way, the attention mechanism is used to evaluate the importance of different scale features, that is, the attention mechanism is used to simulate the hierarchical precipitation expression of the fine-grained difference feature distribution between the aquatic product appearance association feature vector and the aquatic product basic semantic association feature vector in different feature spaces. Finally, the element-by-element product of the aquatic product sorting multi-scale hierarchical fusion shift modulation weight vector and the aquatic product appearance association feature vector is calculated to obtain the optimized aquatic product appearance association feature vector, that is, the weighted operation is used to assign weights to each element in the aquatic product appearance association feature vector, which reduces the influence of local disturbances and improves the quality of the feature vector.

[0052] In summary, in the method, the position difference of the aquatic product sorting shift, the aquatic product sorting shift convolutional coding representation vector and the aquatic product sorting shift modulation difference representation vector respectively represent the explicit expression of the fine-grained distribution difference shift information between the aquatic product appearance-related feature vector and the aquatic product basic semantic-related feature vector in different feature spaces. The generation of the aquatic product sorting multi-scale hierarchical fusion shift modulation weight vector uses the attention mechanism to evaluate the importance of features at different scales and provide weight information for feature shift correction. Ultimately, the generation of the optimized aquatic product appearance-related feature vector reduces the impact of disturbances, improves the model's robustness to local disturbances in the input data, enhances the model's adaptability to noisy or variant data sets, improves the fusion quality of the feature vector, and improves the overall classification performance and training efficiency of the model.

[0053] In a specific embodiment of the present application, the aquatic product basic data feature extraction unit 122 includes: passing the basic data of the aquatic product through an aquatic product basic data semantic encoder to obtain multiple aquatic product basic semantic feature vectors; and cascading the multiple aquatic product basic semantic feature vectors into an aquatic product basic semantic association feature vector.

[0054] It should be understood that the basic data of aquatic products may include various attributes such as the type, size, weight, color, texture, and source of aquatic products. Although these attributes are independent and heterogeneous in the original data, they can be mapped into a unified, high-dimensional semantic space through the semantic encoder of the basic data of aquatic products. In this space, the basic data of each aquatic product is converted into semantically meaningful feature vectors that can comprehensively reflect the key characteristics and relationships of aquatic products. Semantic encoders are usually based on deep learning models, such as neural network architectures, which can effectively learn and capture complex patterns and intrinsic structures in data. The encoder fuses different features in the original data through multiple layers of nonlinear transformations to extract more representative and discriminative feature representations. This transformation can not only integrate heterogeneous data into feature vectors with the same dimension, but also maintain the semantic information of the data, so that these feature vectors can better express the actual attributes and categories of aquatic products. Specifically, the basic data of the aquatic product is segmented to obtain a basic data word sequence; the embedding layer of the aquatic product basic data semantic encoder is used to map each basic data word in the basic data word sequence into a word embedding vector to obtain a sequence of basic data word embedding vectors; the converter-based Bert model of the aquatic product basic data semantic encoder is used to perform global contextual semantic encoding on the sequence of basic data word embedding vectors to obtain multiple aquatic product basic semantic feature vectors.

[0055] Furthermore, multiple basic semantic feature vectors of aquatic products are cascaded into basic semantic association feature vectors of aquatic products, aiming to integrate the independent features of each aquatic product into a more comprehensive and integrated feature vector, so as to more effectively describe and compare the relationship and similarity between different aquatic products. This cascade operation can organically combine the feature vectors of each aquatic product to form a feature representation with higher dimensions and richer semantics. The basic semantic feature vector of each aquatic product usually contains key attributes and features extracted by the semantic encoder. These vectors may include information such as the category, size, color, texture, etc. of the aquatic product, and each vector corresponds to an abstract representation of an aquatic product in the semantic space. By cascading these vectors, the system can integrate the feature information of multiple aquatic products into a larger feature vector, which reflects the common characteristics and differences of all aquatic products. The cascade operation usually includes simple vector connection or more complex splicing and aggregation methods to ensure that the contribution of each aquatic product is fully reflected in the final association feature vector. This approach not only helps to provide a more comprehensive description of aquatic products, but also supports subsequent classification, cluster analysis or decision support tasks.

[0056] In the above-mentioned aquatic product automatic sorting system 100 based on machine vision, the aquatic product category generation module 130 is used to generate aquatic product categories based on the optimized aquatic product appearance-related feature vector and the aquatic product basic semantic-related feature vector. It should be understood that the process of generating aquatic product categories usually involves the application of machine learning or deep learning models, which can learn and infer the classification of aquatic products from feature vectors. Among them, the optimized aquatic product appearance-related feature vector characterizes the diversity and distinctiveness of aquatic products in appearance through the extraction and encoding of visual features. The aquatic product basic semantic-related feature vector provides a semantic description of the types, attributes and characteristics of aquatic products. These feature vectors can be converted through semantic encoders and include information such as the category, source, and growth environment of aquatic products, which is crucial for identifying and distinguishing different types of aquatic products. Combining these two types of feature vectors, the system can use their rich visual and semantic information to infer and classify aquatic product categories through classification algorithms or pattern recognition technologies.

[0057] Figure 5 FIG. 1 is a block diagram of a module for generating aquatic product types in a machine vision-based automatic aquatic product sorting system according to an embodiment of the present application. Figure 5As shown, in a specific embodiment of the present application, the aquatic product type generation module 130 includes: an aquatic product feature fusion unit 131, used to fuse the optimized aquatic product appearance association feature vector and the aquatic product basic semantic association feature vector to obtain an aquatic product type generation feature vector; an aquatic product type feature generation unit 132, used to pass the aquatic product type generation feature vector through a generator to generate an aquatic product type.

[0058] It should be understood that by fusing these two types of feature vectors, the system can combine appearance features with semantic features to generate a comprehensive feature vector for aquatic product types. This fusion operation can be achieved through feature concatenation, weighted averaging, or other combination methods. The fused feature vector not only retains the details of the appearance features, but also integrates the depth of semantic information, thereby being able to more comprehensively describe the comprehensive characteristics of aquatic products.

[0059] Furthermore, the feature vector generated by the aquatic product species combines appearance features and semantic features. Through this comprehensive feature representation, the key attributes and classification information of aquatic products can be effectively captured. However, the feature vector itself is still an abstract, numerical representation, and directly extracting specific aquatic product species from it may not be intuitive or accurate. At this time, the role of the generator model is particularly important. Generator models, such as generative adversarial networks (GANs) or variational autoencoders (VAEs), are good at generating high-quality data outputs from complex feature spaces. In this case, the generator is trained to accept the feature vector generated by the aquatic product species as input and generate the corresponding aquatic product species label. By learning the features and category distribution in the training data, the generator model can map the feature vector to the actual category label, thereby achieving efficient classification tasks.

[0060] In summary, the embodiment of the present application first obtains the appearance image data of aquatic products and the basic data of aquatic products collected by a high-definition camera, and then uses deep learning technology to perform feature extraction and correlation analysis on the two, and finally generates aquatic product types through a generator, thereby reducing manual intervention and improving the processing speed and accuracy of aquatic product sorting.

[0061] As described above, the automatic sorting system 100 for aquatic products based on machine vision according to the embodiment of the present application can be implemented in various terminal devices. In one example, the automatic sorting system 100 for aquatic products based on machine vision can be integrated into the terminal device as a software module and / or a hardware module. For example, the automatic sorting system 100 for aquatic products based on machine vision can be a software module in the operating system of the terminal device, or can be an application developed for the terminal device; of course, the automatic sorting system 100 for aquatic products based on machine vision can also be one of the many hardware modules of the terminal device.

[0062] Alternatively, in another example, the machine vision-based automatic sorting system 100 for aquatic products and the terminal device may also be separate devices, and the machine vision-based automatic sorting system 100 for aquatic products may be connected to the terminal device via a wired and / or wireless network, and transmit interactive information in accordance with an agreed data format.

[0063] In the several embodiments provided by the present invention, it should be understood that the disclosed devices, apparatuses and methods can be implemented in other ways. For example, the device embodiments described above are only schematic, for example, the division of the modules is only a logical function division, and there may be other division methods in actual implementation.

[0064] The modules described as separate components may or may not be physically separated, and the components shown as modules may or may not be physical units, that is, they may be located in one place or distributed on multiple network units. Some or all of the modules may be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0065] In addition, each functional module in each embodiment of the present invention may be integrated into one processing unit, each unit may exist physically separately, or two or more units may be integrated into one unit. The above-mentioned integrated unit may be implemented in the form of hardware or in the form of hardware plus software functional modules.

[0066] It is obvious to those skilled in the art that the present invention is not limited to the details of the above exemplary embodiments, and that the present invention can be implemented in other specific forms without departing from the spirit or essential characteristics of the present invention.

[0067] Therefore, no matter from which point of view, the embodiments should be regarded as illustrative and non-restrictive, and the scope of the present invention is limited by the appended claims rather than the above description, so it is intended that all changes falling within the meaning and scope of the equivalent elements of the claims are included in the present invention. Any attached figure mark in the claims should not be regarded as limiting the claims involved.

[0068] In addition, it is clear that the word "comprising" does not exclude other units or steps, and the singular does not exclude the plural. Multiple units or devices stated in the system claim can also be implemented by one unit or device through software or hardware. The second and other words are used to indicate names, but not to indicate any particular order.

[0069] Finally, it should be noted that the above embodiments are only used to illustrate the technical solution of the present invention rather than to limit it. Although the present invention has been described in detail with reference to the preferred embodiments, those skilled in the art should understand that the technical solution of the present invention can be modified or replaced by equivalents without departing from the spirit of the technical solution of the present invention.

Claims

1. An automatic sorting system for aquatic products based on machine vision, characterized in that: include: Aquatic product data acquisition module, used to acquire aquatic product appearance image data and basic data of aquatic products collected by a high-definition camera; Aquatic product data extraction module, used to extract optimized aquatic product appearance-related feature vectors and aquatic product basic semantics-related feature vectors from the aquatic product appearance image data collected by the high-definition camera and the basic data of the aquatic product; The aquatic product category generation module is used to generate aquatic product categories based on the optimized aquatic product appearance associated feature vector and the aquatic product basic semantic associated feature vector.

2. The automatic aquatic product sorting system based on machine vision according to claim 1 is characterized in that: The aquatic product data extraction module comprises: A water product appearance feature extraction unit, used for extracting features from the water product appearance image data collected by the high-definition camera to obtain the optimized water product appearance related feature vector; The aquatic product basic data feature extraction unit is used to extract features from the basic data of the aquatic product to obtain the basic semantic association feature vector of the aquatic product.

3. The automatic aquatic product sorting system based on machine vision according to claim 2 is characterized in that: The aquatic product appearance feature extraction unit comprises: Aquatic product appearance multi-scale feature encoding subunit, used for performing multi-scale feature encoding on the aquatic product appearance image data collected by the high-definition camera to obtain a plurality of aquatic product appearance feature maps; The aquatic product appearance target feature encoding subunit is used to perform target feature encoding on the multiple aquatic product appearance feature images to obtain the optimized aquatic product appearance associated feature vector.

4. The automatic aquatic product sorting system based on machine vision according to claim 3 is characterized in that: The aquatic product appearance multi-scale feature encoding subunit comprises: Dividing the aquatic product appearance image data collected by the high-definition camera into a plurality of aquatic product appearance images; The multiple aquatic product appearance images are passed through an aquatic product appearance convolutional neural network model with multi-feature extraction capabilities to obtain the multiple aquatic product appearance feature maps.

5. The automatic aquatic product sorting system based on machine vision according to claim 4 is characterized in that: The aquatic product appearance target feature coding subunit includes: Aquatic product appearance target detection secondary subunit, used for passing the plurality of aquatic product appearance feature images through an aquatic product appearance target detection network to obtain a plurality of aquatic product appearance regions of interest; A secondary subunit for detecting salient objects in the appearance of aquatic products, used for passing the plurality of aquatic product appearance regions of interest through an aquatic product appearance salient object detector to obtain an aquatic product appearance associated feature map; Aquatic product appearance feature pooling secondary subunit, used for performing global mean pooling on the aquatic product appearance associated feature map to obtain an aquatic product appearance associated feature vector; A secondary subunit for optimizing the appearance features of aquatic products; based on the basic semantic association feature vector of aquatic products, the aquatic product appearance association feature vector is subjected to a simulated converter structure displacement correction based on hierarchical precipitation expression information to obtain the optimized aquatic product appearance association feature vector.

6. The automatic aquatic product sorting system based on machine vision according to claim 5 is characterized in that: Based on the basic semantic association feature vector of aquatic products, the aquatic product appearance association feature vector is subjected to a transformer-like structural displacement correction based on hierarchical precipitation expression information to obtain the optimized aquatic product appearance association feature vector, including: Calculating the position difference between the aquatic product appearance-related feature vector and the aquatic product basic semantics-related feature vector to obtain an aquatic product sorting displacement position difference representation vector; After concatenating the aquatic product appearance-related feature vector and the aquatic product basic semantics-related feature vector into an aquatic product sorting joint feature vector, the aquatic product sorting joint feature vector is input into a one-dimensional convolutional layer to obtain an aquatic product sorting shift convolutional coding representation vector; After performing aquatic product sorting linear embedding modulation encoding on the aquatic product appearance associated feature vector using the first aquatic product sorting linear embedding weight matrix to obtain an aquatic product appearance associated feature vector linear modulation vector, a position difference between the aquatic product appearance associated feature vector linear modulation vector and the aquatic product basic semantic associated feature vector is calculated to obtain an aquatic product sorting shift modulation difference representation vector; Taking the aquatic product sorting shift position differential representation vector as the value vector, the aquatic product sorting shift convolutional coding representation vector and the aquatic product sorting shift modulation differential representation vector as the query vector and key vector, the aquatic product sorting shift position differential representation vector, the aquatic product sorting shift convolutional coding representation vector and the aquatic product sorting shift modulation differential representation vector are input into a shift corrector based on an imitation transformer structure to obtain aquatic product sorting multi-scale hierarchical fusion shift modulation weight vector; The optimized aquatic product appearance-related feature vector is obtained by calculating the point-by-point multiplication between the aquatic product sorting multi-scale hierarchical fusion shift modulation weight vector and the aquatic product appearance-related feature vector.

7. The automatic aquatic product sorting system based on machine vision according to claim 6 is characterized in that: The aquatic product basic data feature extraction unit comprises: Passing the basic data of aquatic products through an aquatic product basic data semantic encoder to obtain a plurality of aquatic product basic semantic feature vectors; The multiple aquatic product basic semantic feature vectors are cascaded into an aquatic product basic semantic association feature vector.

8. The automatic aquatic product sorting system based on machine vision according to claim 7 is characterized in that: The aquatic product category generation module includes: Aquatic product feature fusion unit, used for fusing the optimized aquatic product appearance-related feature vector and the aquatic product basic semantics-related feature vector to obtain an aquatic product type generation feature vector; The aquatic product type feature generation unit is used to generate the aquatic product type feature vector through a generator to generate an aquatic product type.

Images