A mixed fruit picking method based on multi-attention image recognition
Patent Information
- Application Number
- CN202410713063.5
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Filing Date
- 2024-06-04
- Publication Date
- 2024-12-03
- Estimated Expiration
- 2044-06-04
Smart Images

Figure CN118476392B_ABST
Abstract
Description
Technical Field
[0001] The present invention relates to the technical field of intelligent agriculture, and in particular to a mixed fruit picking method based on multi-attention image recognition. Background Art
[0002] Recent advances in computer vision and machine learning, such as large-scale image recognition with convolutional neural networks, have greatly impacted multimedia systems. People get a lot of information from them in their daily lives, but the amount of information such as pictures and texts generated by the Internet is growing exponentially every day, which requires us to extract data related to user needs from a large amount of fragmented information and accurately classify it. However, the task of image recognition is still challenging for current models, and the differences between different categories exist in subtle details that may be overwhelmed by differences in posture, clutter, and deformation in natural images. Summary of the invention
[0003] The present invention discloses a mixed fruit picking method based on multi-attention image recognition, and the specific method is as follows:
[0004] Take images of mixed fruits ready to be picked;
[0005] Preprocess the mixed fruit images;
[0006] The convolution kernel is used to extract the features of the mixed fruit image and obtain several convolution feature maps;
[0007] Pooling is performed on several convolutional feature maps respectively to obtain several pooled feature maps;
[0008] Use multiple attention modules to process several pooled feature maps one by one, output several spatial heat maps containing important features of each region, and weight the spatial heat maps according to the characteristics of the mixed fruits;
[0009] Use several attention heads of the attention module to make local predictions on the spatial heatmap;
[0010] Adaptively focus on local predictions and take a weighted average of each local prediction to get the prediction of each attention module;
[0011] Combine the predictions of each attention module and the gating value of the attention head to generate the final recognition result;
[0012] Based on the final recognition results, the picking robot arm is controlled to complete the picking of the target fruit.
[0013] Furthermore, the mixed fruit image is preprocessed, specifically including:
[0014] Remove salt and pepper noise, speckle noise and Gaussian noise from the image, and perform grayscale processing on the image after noise removal.
[0015] Furthermore, several convolution feature maps are obtained. The specific method is as follows:
[0016] Convolution operations are performed on the preprocessed image through multiple convolution kernels to extract local features in the preprocessed image and obtain several convolution feature maps.
[0017] Furthermore, several pooled feature maps are obtained. The specific method is as follows:
[0018] Define a 2×2 pooling window, perform average pooling on the convolution feature map, and obtain several pooling feature maps.
[0019] Furthermore, several attention heads of the attention module are used to make local predictions on the spatial heat map. The specific method is as follows:
[0020] Each attention head makes a local prediction based on the selected information. The specific formula is:
[0021]
[0022] in, is the output head convolution weight, which refers to the weight parameter of the convolution layer used for the head layer; k corresponds to different attention heads, ⊙ is the product of each element; each is a vector, which represents its position in space i∈{1…h},j∈{1…w}, with a depth of l. are the local outputs of different attention heads.
[0023] Furthermore, when adaptively focusing on local prediction, a 1*1 convolutional layer is used to model each local output. The specific formula is:
[0024]
[0025] S l is a vector containing the attention gate value with depth l, W s ∈R AW*c*h*w is the convolution kernel.
[0026] Furthermore, each local prediction is weighted averaged to obtain the prediction of each attention module. The specific formula is as follows:
[0027]
[0028] In the formula is the prediction of a single attention module.
[0029] Furthermore, the calculation formula of the gate value is as follows:
[0030] g l =softmax(Wg *Z l )
[0031]
[0032] Among them, w g ∈R AW*c*h*w are the weights of linear operations, which determine the contribution of each feature to the final output; g l is pooling Z l The gate value of different attention depths is then used, depths is the depth set where attention is placed, and output is the final recognition result corrected by attention prediction.
[0033] Furthermore, after generating the final recognition result, the verification method is as follows:
[0034] Load validation set and test set data; extract features and classify validation data through the test model; compare the predicted results with the actual results, calculate the accuracy of the validation set, and modify the setting parameters;
[0035] The modifiable setting parameters include: c, number of input channels, h, feature map height, w, feature map width, and AW, attention width.
[0036] Furthermore, the mixed fruits are apples and oranges, and the specific method for weighting the spatial heat map is as follows:
[0037] Calculate the gradient G(l,i,j) of the target category l for each position (i,j) in the feature map x, where y l is the gradient value of the target category l, represents partial derivative;
[0038] Calculate the weight α(l,i,j) of each position (i,j) in the feature map x, α(l,i,j) = GlobalAverragePooling(ΩG(l,i,j)); where GlobalAverragePooling performs an average pooling operation on the gradient G(l,i,j) and converts it into a scalar value; Ω is the adaptive coefficient, which is adjusted according to the gradient error of adjacent points Δg = α(,i,j) - α(l,i-1,j-1), that is, Ω = e Δg ;
[0039] Compute weighted gradient map
[0040] Where ReLU represents the rectified linear unit activation function, and Σ represents the summation of all positions (i, j);
[0041] For H lNormalize it so that its value range is between 0 and 1. To generate the attention head H l The convolution kernel of the corresponding attention mask, that is, the convolution weight ratio of this layer, extracts the image features by multiplying and summing the elements at different positions of the input image, which can capture the local features in the image. * represents the convolution operator, Z l represents the convolutional feature map, which is the feature map extracted from the input image through the convolution operation;
[0042] In the convolution operation, an output value is generated by multiplying the convolution kernel by different positions of the input image element by element and summing them. By sliding the convolution kernel on the input image, different features can be extracted at different positions.
[0043] AW is the attention width, which corresponds to the number of attention heads in each module and refers to the size of the window or range used to calculate the attention weight in the attention mechanism;
[0044] H l ∈R AW*c*h*w The output matrix of the attention head, i.e., the spatial heat map, is used to represent the spatial distribution of a specific feature in the image. It represents the intensity or importance of the position by displaying a color or grayscale value at each position on the image.
[0045] Due to the adoption of the above technical solution, the present invention has the following beneficial effects:
[0046] 1. The invention can improve the efficiency of fruit picking. The robotic arm can improve the efficiency of fruit picking through fast, precise and continuous operation. Compared with manual picking, the robotic arm can complete more picking tasks in a shorter time, thereby increasing the yield and picking efficiency of crops; it can also reduce the pressure of manual labor. Crop picking is usually a heavy physical labor task that requires a lot of manpower input.
[0047] 2. The adaptability and learning ability of the neural network enables the robot arm to adapt and adjust according to different fruit types and environmental conditions. Through continuous learning and feedback, the neural network can improve the picking skills and decision-making ability of the robot arm, and gradually improve the effect and quality of picking.
[0048] Other advantages, objectives and features of the present invention will be described in the following description to some extent, and to some extent, will be obvious to those skilled in the art based on the following examination and study, or can be taught from the practice of the present invention. The objectives and other advantages of the present invention can be achieved and obtained through the following description. BRIEF DESCRIPTION OF THE DRAWINGS
[0049] The accompanying drawings of the present invention are as follows.
[0050] Figure 1 This is a schematic diagram of the three-dimensional structure of the picking robot arm.
[0051] Figure 2 Schematic diagram of the process of the embodiment.
[0052] Figure 3 Schematic diagram of the attention module structure.
[0053] Figure 4 Outputs the schematic diagram as a whole. DETAILED DESCRIPTION
[0054] The present invention will be further described below in conjunction with the accompanying drawings and embodiments.
[0055] A mixed fruit picking method based on multi-attention image recognition, the mixed fruits are apples and oranges, and a picking robot arm is used to automatically classify and pick the mixed fruits. Figure 1 The specific steps are as shown in Figure 2 As shown:
[0056] S1. Take an image of mixed fruits to be picked and remove the noise in the image, such as salt and pepper noise, speckle noise, and Gaussian noise. After acquiring and processing the image dataset, divide the processed dataset into a training set, a validation set, and a test set according to a certain ratio. Perform grayscale preprocessing on the image, such as resizing, cropping, and standardization, so that it can be input into the machine learning model.
[0057] S2. Represent the grayscale image as a two-dimensional matrix, where each element represents the grayscale value of the image pixel. Perform convolution operations by sliding multiple convolution kernels on the input image, perform element-wise product of the convolution kernel and the local area of the input image, and sum them to calculate the feature map. Local features in the image can be extracted. The size and parameters of the convolution kernel are learned during the model training process, and the weight parameters of the convolution kernel are updated through the back-propagation algorithm so that the network can extract and learn useful features in the input data.
[0058] S3. Perform an average pooling operation on the feature map to reduce the spatial dimension of the feature map. Define the pooling window size as a 2×2 pooling window, which can halve the spatial dimension of the feature map while retaining relatively important features. Slide the pooling window and select a stride of 2. Then perform an average pooling operation. For each pooling window, calculate the average value of the feature value in the window as the pooled output. Repeat the above two steps until the pooling window is applied to the entire input feature map. A new pooled feature map will be generated, and its spatial dimension is reduced relative to the input feature map.
[0059] S4, using 1*1 convolutional layer for modeling, applying softmax to the information at different abstract levels to normalize the output in the spatial dimension, each attention module contains multiple attention heads, which receive a convolutional feature map as input and output a spatial heat map containing the importance of each input area, such as Figure 3 shown.
[0060] The calculation formula is as follows:
[0061]
[0062] R b*c*h*w →R 2*1*h*w
[0063] Where x is the input training set, b is the processing dimension. In this example, a two-dimensional image is processed, that is, b = 2, c is the number of input channels, here the number of channels is 1, that is, each attention head receives a convolutional feature map as input. h is the feature map height, and w is the feature map width, which is selected here according to the input image size.
[0064] This embodiment is mainly used to distinguish between apples and oranges, so the generated spatial heat map focuses on the weighting of color features. Apples are usually red or green, while oranges are orange. In the spatial heat map, warm colors represent higher intensity or heat, so accurate classification and identification cannot be achieved by color alone; the present invention also takes into account the difference in appearance between apples and oranges. The surface of apples is smoother than that of oranges, so in the corresponding spatial heat map, the spatial heat map distribution of apples is more uniform, so there will be a greater weight distribution on the uniform distribution of spatial heat maps. If the gradient values of adjacent positions are similar, it means that they are relatively smooth, so the spatial heat maps of apples and oranges can be distinguished. The specific algorithm is as follows:
[0065] First, calculate the gradient of the target category l for each position (i, j) in the feature map x where y l is the gradient value of the target category l, represents partial derivative.
[0066] Calculate the weight α(l,i,j) of each position (i,j) in the feature map x, α(l,i,j) = GlobalAverragePooling(ΩG(l,i,j)); where GlobalAverragePooling performs an average pooling operation on the gradient G(l,i,j) to convert it into a scalar value. Ω is an adaptive coefficient, which adjusts its size according to the gradient value error of adjacent points Δg = α(l,i,j) - α(l,i-1,j-1), that is, Ω = e ΔgSpecifically, for apples and oranges, relatively smooth apples will have a greater weight on the feature map gradient.
[0067] Next, for each position (i, j) in the feature map x, calculate the weighted gradient map Among them, ReLU represents the rectified linear unit activation function, and Σ represents the summation of all positions (i, j).
[0068] Normalize Hl so that its value range is between 0 and 1:
[0069]
[0070] Z l =α(l,i,j
[0071] To generate the attention head H l The convolution kernel of the corresponding attention mask, that is, the convolution weight ratio of this layer, extracts the image features by multiplying and summing the elements at different positions of the input image, which can capture the local features in the image. * represents the convolution operator, Z l Represents the convolutional feature map, which is a feature map extracted from the input image through the convolution operation. In the convolution operation, an output value is generated by element-wise multiplication of the convolution kernel with different positions of the input image and summing them. By sliding the convolution kernel on the input image, different features can be extracted at different positions. AW is the attention width, which corresponds to the number of attention heads in each module and refers to the size of the window or range used to calculate the attention weight in the attention mechanism. It determines whether the model focuses on the local or global range of the input sequence when calculating attention.
[0072] H l ∈R AW*c*h*w The output matrix of the attention head, i.e., the spatial heat map, is used to represent the spatial distribution of a specific feature in the image. It represents the intensity or importance of the position by displaying a color or grayscale value at each position on the image.
[0073] S5. After outputting the spatial heat map, each attention head makes a local prediction based on the selected information to obtain the local output, and applies a 1*1 convolutional layer to the local output. The calculation formula is as follows:
[0074]
[0075] in is the output head convolution weight, which refers to the weight parameter of the convolution layer used for the head layer. k corresponds to different attention heads, and ⊙ is the product of each element. is a vector, which represents its position in space i∈{1…h},j∈{1…w}, with a depth of l. It is the local output of different attention heads, and the output is the local hotspot of the image.
[0076] S6, such as Figure 4 As shown, in this embodiment, since each attention head is forced to focus on different areas of the image, it is possible that the areas focused on by some attention heads provide more information than the areas focused on by other attention heads. In order to retain the part with the largest amount of information and discard the part with the smallest amount of information, adaptive attention is applied to each local output head, and a 1*1 convolution layer is used to model each local output.
[0077]
[0078] S l is a vector containing the attention gate value with depth l, W s ∈R AW*c*h*w is the convolution kernel. After using adaptive attention, the differences between varieties can be magnified and the similarities can be reduced, so that different varieties can be distinguished more accurately.
[0079] S7. After completing the adaptive attention of each attention head, this method performs weighted averaging on each local output to obtain the prediction of a single attention module.
[0080]
[0081] In the formula is the prediction of a single attention module, and the prediction result is the spatial heat map of the respective attention area.
[0082] The final output of the network consists of the gated average of the outputs of each attention module. The gated value is calculated based on the average pooling Z l The last feature map is generated after . The calculation formula is as follows:
[0083] g l =softmax(W g *Z l )
[0084]
[0085] w g ∈R AW*c*h*w are the weights of linear operations, and these weight parameters determine the contribution of each feature to the final output. l is pooling Z lThe gate values of different attention depths are then used, depths is the set of depths where attention is placed, and output is the final classification corrected by the attention prediction. Therefore, the model selects the appropriate region and abstraction level to produce the correct class prediction by comparing it with the different types of fruits learned in the training set.
[0086] For each training sample (x i ,y i ), the back propagation algorithm randomly initializes the weights w and bias b, performs forward propagation, and calculates the weighted input of each layer And activation value a. The formula is as follows:
[0087]
[0088]
[0089] Get the predicted value of the output layer
[0090] Next, calculate the error of the output layer, the formula is as follows:
[0091]
[0092] in, Represents the loss function The gradient of , activation_function represents the activation function, and activation_derivative represents the derivative of the activation function.
[0093] Then calculate the back propagation error, parameter gradient, average gradient and network parameter update
[0094] For each layer l=L-1,L-1,…,2,
[0095]
[0096] dw[l]=Δ[l]*a[l-1] T
[0097] db[l]=Δ[l]
[0098]
[0099]
[0100]
[0101]
[0102] dw[l] represents the weight coefficient of the lth layer, db[l] represents the bias gradient of the lth layer, Δ[l] is the error term of the layer, a[l-1] is the activation value of the previous layer, and ^T represents the transposition operation. earning_rate is a hyperparameter in the backpropagation algorithm, which is used to control the step size of each parameter update or the speed of learning.
[0103] S8. Optimize the image classification model through parameter optimization to obtain the best model, load the validation set and test set data; extract features and classify the validation data through the test model; compare the predicted results with the actual results, calculate the accuracy of the validation set, modify the parameters that can be set, and debug within a reasonable range. If the accuracy is found to be improved, repeat this operation and calculate the accuracy until the optimal result is reached. The parameters that can be set are: c, number of input channels, h, feature map height, w, feature map width, AW, attention width, etc.; input the test set into the optimal model to obtain the final classification result and accuracy.
[0104] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention rather than to limit it. Although the present invention has been described in detail with reference to the above embodiments, ordinary technicians in the relevant field should understand that the specific implementation methods of the present invention can still be modified or replaced by equivalents. Any modification or equivalent replacement that does not depart from the spirit and scope of the present invention should be covered within the scope of protection of the claims of the present invention.
Claims
1. A mixed fruit picking method based on multi-attention image recognition, characterized in that: The specific method is as follows: Take images of mixed fruits ready to be picked; Preprocess the mixed fruit images; The convolution kernel is used to extract the features of the mixed fruit image and obtain several convolution feature maps; Pooling is performed on several convolutional feature maps respectively to obtain several pooled feature maps; Use multiple attention modules to process several pooled feature maps one by one, output several spatial heat maps containing important features of each region, and weight the spatial heat maps according to the characteristics of the mixed fruits; Use several attention heads of the attention module to make local predictions on the spatial heatmap; Adaptively focus on local predictions and take a weighted average of each local prediction to get the prediction of each attention module; Combine the predictions of each attention module and the gating value of the attention head to generate the final recognition result; According to the final recognition result, the picking robot arm is controlled to complete the picking of the target fruit; The attention module uses several attention heads to make local predictions on the spatial heat map. The specific method is as follows: Each attention head makes a local prediction based on the selected information. The specific formula is: in, is the output head convolution weight, which refers to the weight parameter of the convolution layer used for the head layer, where AW is the attention width, c is the number of input channels, h is the feature map height, and w is the feature map width; k corresponds to different attention heads, and Z l Represents the convolution feature map, which extracts the feature map from the input image through the convolution operation; * represents the convolution operator, and ⊙ is the product of each element; is the output matrix of different attention heads, i.e., the spatial heat map; each is a vector, which represents its position in space i∈{1…h},j∈{1…w}, with a depth of l. are the local outputs of different attention heads.
2. The mixed fruit picking method based on multi-attention image recognition as claimed in claim 1, characterized in that: Preprocess the mixed fruit image, including: Remove salt and pepper noise, speckle noise and Gaussian noise from the image, and perform grayscale processing on the image after noise removal.
3. The mixed fruit picking method based on multi-attention image recognition as claimed in claim 1, characterized in that: Get several convolution feature maps. The specific method is as follows: Convolution operations are performed on the preprocessed image through multiple convolution kernels to extract local features in the preprocessed image and obtain several convolution feature maps.
4. The mixed fruit picking method based on multi-attention image recognition as claimed in claim 1, characterized in that: Get several pooled feature maps. The specific method is as follows: Define a 2×2 pooling window, perform average pooling on the convolution feature map, and obtain several pooling feature maps.
5. The mixed fruit picking method based on multi-attention image recognition as claimed in claim 1, characterized in that: The gate value is calculated as follows: g l =softmax(W g *Z l ) Among them, w g ∈R AW*c*h*w are the weights of the linear operation. These weight parameters determine the contribution of each feature to the final output, where AW is the attention width, c is the number of input channels, h is the feature map height, and w is the feature map width; Z l represents the convolutional feature map; g l is pooling Z l The gate values of different attention depths are is the prediction of a single attention module, depths is the depth set where attention is placed, and output is the final recognition result corrected by the attention prediction.
6. The mixed fruit picking method based on multi-attention image recognition as claimed in claim 1, characterized in that: After generating the final recognition result, the verification method is as follows: Load validation set and test set data; extract features and classify validation data through the test model; compare the predicted results with the actual results, calculate the accuracy of the validation set, and modify the setting parameters; The modifiable setting parameters include: c, number of input channels, h, feature map height, w, feature map width, and AW, attention width.
7. The mixed fruit picking method based on multi-attention image recognition as claimed in claim 1, characterized in that: The mixed fruits are apples and oranges. The specific method for weighting the spatial heat map is as follows: Calculate the gradient G(l,i,j) of the target category l for each position (i,j) in the feature map x, where y l is the gradient value of the target category l, represents partial derivative; Calculate the weight α(l,i,j) of each position (i,j) in the feature map x, α(l,i,j) = GlobalAverragePooling(ΩG(l,i,j)); where GlobalAverragePooling performs an average pooling operation on the gradient G(l,i,j) and converts it into a scalar value; Ω is the adaptive coefficient, which is adjusted according to the gradient value error of adjacent points Δg = α(l,i,j) - α(l,i-1,j-1), that is, Ω = e Δg ; Compute weighted gradient map Where ReLU represents the rectified linear unit activation function, and Σ represents the summation of all positions (i, j); For H l Normalize it so that its value range is between 0 and 1. Z l =α(l,i,j) To generate the attention head H l The convolution kernel of the corresponding attention mask, that is, the weight ratio of the convolution layer, extracts the image features by multiplying and summing the elements at different positions of the input image, capturing the local features in the image. * represents the convolution operator, Z l represents the convolutional feature map, which is the feature map extracted from the input image through the convolution operation; In the convolution operation, an output value is generated by multiplying the convolution kernel by different positions of the input image element by element and summing them. Different features can be extracted at different positions by sliding the convolution kernel on the input image. AW is the attention width, which corresponds to the number of attention heads in each module and refers to the size of the window or range used to calculate the attention weight in the attention mechanism; H l ∈R AW*c*h*w The output matrix of the attention head, i.e., the spatial heat map, is used to represent the spatial distribution of a specific feature in the image. It represents the intensity or importance of the position by displaying a color or grayscale value at each position on the image.
