A detection method and device for litchi images

Abstract

This application relates to a lychee detection method and device based on a graph neural network-based improved visual Transformer. The method comprises: obtaining a lychee image to be detected; dividing the lychee image to be detected into image blocks, and constructing edges based on the similarity between the image blocks to form graph data; inputting the constructed graph data into a visual Transformer model improved based on the graph neural network to obtain lychee category data and bounding box position and size; wherein, the constructed graph data is input into the graph neural network module to obtain a block embedding vector for each image block; the block embedding vector plus the position embedding vector is input into the Transformer encoder, and then predicted by the detection head to obtain the detection result of the lychee image. This method can achieve high-precision lychee detection in complex environments and under occlusion.

Description

Technical Field

[0001] The present application relates to the field of machine vision technology, and in particular to a method and device for detecting litchi images. Background Art

[0002] With the development of artificial intelligence and computer vision, visual inspection technology has significantly advanced. Due to its contactless, efficient, and low-cost nature, visual inspection technology is widely used in various fields, including lychee harvesting. Lychees are difficult to harvest due to their irregular shape, smaller size compared to other fruits, and easily obscured by trees. Traditional manual harvesting is inefficient and requires climbing trees, making automated lychee harvesting a necessity.

[0003] Existing automated picking methods require the detection and positioning of lychees. Traditional lychee visual inspection technology often detects lychees by constructing manual features such as shape and color. This method of detection based on manual features is slow, has low accuracy, and is easily affected by environmental factors such as lighting. Such models often have poor detection accuracy in complex environments. To further improve the accuracy of lychee visual inspection, the Transformer was migrated to the visual task. However, because the visual Transformer converts image data into sequence data by slicing, it ignores the connection between image blocks, resulting in the loss of local information. This can lead to missed detection when the lychee fruit is partially occluded, ultimately causing large deviations in the calculation of the picking point. Summary of the Invention

[0004] Based on this, it is necessary to provide a method and device for detecting litchi images that can ensure detection accuracy in complex backgrounds and in the presence of occlusions in order to address the above technical problems.

[0005] A method for detecting a litchi image, the method comprising:

[0006] Obtaining a litchi image to be detected;

[0007] Dividing the litchi image to be detected into image blocks, and constructing edges based on similarities between the image blocks to form graph data including nodes and edges;

[0008] The constructed graph data is input into the visual Transformer model improved based on the graph neural network to obtain the category data of litchi and the position and size of the bounding box. The constructed graph data is input into the graph neural network module to obtain the block embedding vector of each image block. The block embedding vector plus the position embedding vector are input into the Transformer encoder, and then predicted by the detection head to obtain the detection result of the litchi image.

[0009] In one embodiment, dividing the litchi image to be detected into image blocks and constructing edges according to similarities between the image blocks to form graph data includes:

[0010] The litchi image to be detected is divided into N×N image blocks, the size of each image block is P1×P2×3, each image block is regarded as a node in the graph data, and the node set V={v i |i=1,2,..,N 2}; where N is a positive integer, i={1,2,…,N 2} is the number of the image block;

[0011] Each image block is convolved through a one-dimensional convolutional layer to obtain a vector representation of each image block. Where d is the dimension of the image block vector representation, and the vector representation X={x i |i=1,2,..,N 2} is considered as a node attribute in the graph data;

[0012] By calculating the vector representation x i The similarity between them is measured, and for each image block, the top k image blocks with the highest similarity are found, which are regarded as the neighbor nodes of the image block. Directed edges are constructed between these vector blocks to obtain the edge set E; where k is the number of neighbors of each node.

[0013] In one embodiment, inputting the constructed graph data into a visual Transformer model improved based on a graph neural network to obtain the category data and bounding box location of litchi includes:

[0014] Input the constructed graph data G = (V, E, X) into the graph neural network module to obtain the block embedding vector of each image block;

[0015] The block embedding vector plus the position embedding vector is used as the input of the Transformer encoder to obtain the representation vector Z = {z i |i=1,2,..,N 2};

[0016] The image representation vector Z = {z i |i=1,2,..,N 2}Input the detection head for prediction to obtain the detection result of the litchi image; wherein the detection result includes the category data of the litchi in the image and the position and size of the bounding box.

[0017] In one embodiment, inputting the constructed graph data into a graph neural network module to obtain a block embedding vector for each image block includes:

[0018] According to the constructed graph data, the aggregation function is used to calculate the node v i Aggregate messages from neighbor nodes and get the aggregated neighbor node vector representation a i , the message is the vector representation x of the neighbor node j ;

[0019] Splicing node v i The vector representation of x i And the aggregated neighbor node vector represents a i ;

[0020] Use the update function to update the node v i The vector representation is updated to get the updated v i The vector representation of x' i ;

[0021] Use the updated vector to represent x' i , repeat the above aggregation, splicing, and updating operations to obtain the block embedding vector x" of each node after being processed by the graph neural network i ; where the block embedding vector x" i The dimension is d';

[0022] The above process is expressed as follows:

[0023] a i =aggregate({x j :v j ∈N(v i )})

[0024] x' i =update([x i ,a i ])

[0025] a' i =aggregate({x' j :v j ∈N(v i )})

[0026] x” i =update([x' i ,a' i ])

[0027] Among them, aggregate(·) is the aggregation function; update(·) is the update function, and [·,·] is the concatenation operation.

[0028] In one embodiment, the block embedding vector plus the position embedding vector is used as the input of the Transformer encoder to obtain the representation of the entire image, including:

[0029] Add the embedding vector of each block to the embedding vector of the position with the same dimension to obtain the concatenated embedding vector;

[0030] The concatenated embedding vector is input into the Transformer encoder, and the features are normalized by layer normalization to obtain the normalized embedding vector.

[0031] The normalized embedding vector is input into the attention module to obtain the vector representation b after attention module processing i , and the vector before layer normalization and the vector after attention module processing are represented as b i Add together to get the representation b' after the initial residual connection i ;

[0032] The representation b' after the initial residual connection i After layer normalization, the result is input into the multi-layer perceptron, and the processed result is connected with the initial residual to form the representation b' i Add them together to get the representation b after residual connection again i ;

[0033] Repeat the above operation L times to obtain the final representation Z of the entire image.

[0034] In one embodiment, the normalized embedding vector is input into the attention module to obtain the vector representation b after processing by the attention module. i ,include:

[0035] Multiply the normalized embedding vector of each image block by the learnable transformation matrix w q 、w k 、w v Then get q i 、k i 、v i ;

[0036] q i k with other image blocks i Perform inner product operation and softmax operation to obtain the attention score α between each image block;

[0037] Multiply the attention score by v of each image block i , get the vector representation b of each image block after attention module processing i ;

[0038] The above process can be expressed as the following formula:

[0039]

[0040] Where B={b i |i=1,2,...,N 2} is the processed image block representation b i The set of Q, K, and V are q i 、k i 、v i A collection of .

[0041] In one embodiment, the image representation vector Z={z i |i=1,2,..,N 2 Input detection head for prediction, including:

[0042] The representation vector Z of each image block is i |i=1,2,..,N 2} perform separate predictions, using three anchor boxes of preset sizes for each image block to obtain the prediction results corresponding to each image block; the prediction results include the coordinates of the center point of the bounding box of the litchi x box 、y box , the width w and height h of the bounding box, the category class of litchi; the anchor box is used to assist in the prediction of the position and size of the litchi bounding box;

[0043] The prediction results corresponding to all image blocks are screened using non-maximum suppression to obtain the final litchi detection result; the image block where the center point of the litchi bounding box is located is responsible for detecting the litchi.

[0044] A lychee image detection device, comprising:

[0045] An image acquisition module to be detected is used to acquire the grape image to be detected;

[0046] An image data preprocessing module is used to divide the litchi image to be detected into image blocks, and construct edges based on the similarity between the image blocks to form graph data containing nodes and edges;

[0047] The object detection module is used to input the constructed graph data into the visual Transformer model improved based on the graph neural network to obtain the category data and bounding box position of the litchi; wherein, the constructed graph data is input into the graph neural network module to obtain the block embedding vector of each image block; the block embedding vector plus the position embedding vector is input into the Transformer encoder, and then predicted by the detection head to obtain the detection result of the litchi image.

[0048] A computer device includes a memory and a processor, wherein the memory stores a computer program, and when the processor executes the computer program, the following steps are implemented:

[0049] Obtaining a litchi image to be detected;

[0050] Dividing the litchi image to be detected into image blocks, and constructing edges based on similarities between the image blocks to form graph data including nodes and edges;

[0051] The constructed graph data is input into the visual Transformer model improved based on the graph neural network to obtain the category data of litchi and the position and size of the bounding box. The constructed graph data is input into the graph neural network module to obtain the block embedding vector of each image block. The block embedding vector plus the position embedding vector are input into the Transformer encoder, and then predicted by the detection head to obtain the detection result of the litchi image.

[0052] A computer-readable storage medium stores a computer program, which, when executed by a processor, implements the following steps:

[0053] Obtaining a litchi image to be detected;

[0054] Dividing the litchi image to be detected into image blocks, and constructing edges based on similarities between the image blocks to form graph data including nodes and edges;

[0055] The constructed graph data is input into the visual Transformer model improved based on the graph neural network to obtain the category data of litchi and the position and size of the bounding box. The constructed graph data is input into the graph neural network module to obtain the block embedding vector of each image block. The block embedding vector plus the position embedding vector are input into the Transformer encoder, and then predicted by the detection head to obtain the detection result of the litchi image.

[0056] The above-mentioned detection method, device, computer equipment and storage medium based on lychee images, after the improvement of the visual Transformer by the graph neural network, can enhance the connection between image blocks and the local information of features through the graph neural network compared to other visual Transformer models, and then extract the image representation through the Transformer model to ensure global information. The image representation containing global and local information is input into the detection head to obtain the bounding box position, size and category information of all lychees in the image. This application combines local and global information to improve the detection accuracy of lychees in complex scenes and with occlusion. BRIEF DESCRIPTION OF THE DRAWINGS

[0057] Figure 11 is a flow chart of a method for detecting litchi images in one embodiment;

[0058] Figure 2 Schematic diagram of building a graph data node attribute in one embodiment

[0059] Figure 3 A schematic diagram of a framework of a method for detecting litchi images based on a graph neural network-based improved visual Transformer in one embodiment;

[0060] Figure 4 A schematic diagram of mean aggregation of a Transformer encoder and a graph neural network in one embodiment;

[0061] Figure 5 This is a structural block diagram of a litchi detection device based on an improved visual Transformer using a graph neural network in another embodiment;

[0062] Figure 6 FIG. 1 is a diagram showing the internal structure of a computer device in one embodiment. DETAILED DESCRIPTION

[0063] In order to make the purpose, technical solutions and advantages of this application more clear, the following further describes this application in detail with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain this application and are not intended to limit this application.

[0064] In one embodiment, Figure 1 As shown in the figure, a litchi detection method based on graph neural network to improve visual Transformer is provided, including the following steps:

[0065] S110, obtaining a litchi image to be detected;

[0066] The image of the lychees to be inspected is captured by a camera installed on the picking robot. The image of the grapes to be inspected captured by the camera is sent to the robot's built-in server or backend server for processing.

[0067] S120, dividing the litchi image to be detected into image blocks, and constructing edges based on similarities between the image blocks to form graph data including nodes and edges;

[0068] Graph data is a type of data. The robot's built-in server or backend server deploys an image segmentation algorithm and a similarity calculation algorithm. The image segmentation algorithm divides the image, and the similarity calculation algorithm calculates the similarity between image blocks to construct edges between them. In graph data, nodes consist of image blocks, and node attributes are vector representations of these blocks. Edges between nodes connect image blocks with high similarity.

[0069] S130, input the constructed graph data into the visual Transformer model improved based on the graph neural network to obtain the category data and bounding box position and size of the litchi; wherein, the constructed graph data is input into the graph neural network module to obtain the block embedding vector of each image block; the block embedding vector plus the position embedding vector is input into the Transformer encoder, and then predicted by the detection head to obtain the detection result of the litchi image.

[0070] Among them, an improved Transformer model based on graph neural network is deployed in the robot's built-in server or background server. The improved Transformer model is used to process the litchi image to obtain the litchi category data and the bounding box position and size.

[0071] In the above-mentioned litchi detection method based on the improved visual Transformer of graph neural network, after the improvement of visual Transformer by graph neural network, compared with other visual Transformer models, it can enhance the connection between image blocks and the local information of features through graph neural network, and then extract the image representation through the Transformer model to ensure global information. The image representation containing global and local information is input into the detection head to obtain the bounding box position, size and category information of all litchi in the image. This application combines local and global information to improve the detection accuracy of litchi in complex scenes and with occlusion.

[0072] In one embodiment, Figure 2 As shown in the figure, 1 represents the divided image block, 2 represents the pixel point, 3 represents the image block width, that is, P1, 4 represents the image block height, that is, P2, 5 represents the number of image block channels, that is, 3, 6 represents the one-dimensional convolution layer operation, and 7 represents the representation vector of the image block, that is, x i The method of dividing the litchi image to be detected into image blocks and constructing edges according to the similarity between the image blocks to form graph data includes: dividing the litchi image to be detected into N×N image blocks, each image block has a size of P1×P2×3, and each image block is regarded as a node in the graph data, and obtaining a set of nodes V={v i |i=1,2,..,N 2}; where N is a positive integer, i={1,2,…,N 2} is the number of the image block; each image block is convolved through a one-dimensional convolution layer to obtain a vector representation of each image block Where d is the dimension of the image block vector representation, and the vector representation X={x i|i=1,2,..,N 2} is considered as a node attribute in the Graph data; by calculating the vector representation x i The similarity between them is measured. For each image block, the top k image blocks with similarity are found and regarded as the neighbor nodes of the image block. Directed edges are constructed between these vector blocks to obtain the edge set E, where k is the number of neighbors of each node.

[0073] The image is divided into N×N image blocks by an image partitioning algorithm, where N is the number of image blocks divided along the width and height directions. For example, N can be set to 20, that is, the image is divided into 20×20 image blocks, a total of 400 image blocks. The obtained image blocks are regarded as nodes in the graph data.

[0074] Among them, the image block of size P1×P2×3 is converted into a d×1-dimensional vector representation x through a one-dimensional convolution layer. i , this vector representation will be regarded as the node attribute in the Graph data for subsequent aggregation and update operations of the graph neural network.

[0075] Among them, similarity calculation is used to capture the similarity between image blocks and set edges between image blocks with high similarity. Taking image block i as an example, its node attribute is x i , calculate the attributes of the remaining image block nodes and x i The image blocks with the top k similarities to image block i are regarded as neighbor nodes of image block i, and edges are constructed from neighbor nodes to image block i. The direction of the edge represents the message aggregation direction of the graph neural network.

[0076] Wherein, the vector represents x i The similarity calculation formula between them is as follows:

[0077]

[0078] Where D(·,·) is the cosine similarity function; ||·||2 is the L2 norm; x i is the vector representation of the target node; x j is the vector representation of other nodes, j = {1, 2, ..., N}; the nodes with the top k similarities to the target node will be regarded as neighbor nodes of the target node, and directed edges pointing to the target node will be constructed for the top k nodes; node v i The set of neighbor nodes is represented as v j ∈N(v i ).

[0079] In one embodiment, Figure 3 As shown, the constructed Graph data is input into the visual Transformer model improved based on the graph neural network to obtain the category data and bounding box position of the litchi, including: inputting the constructed Graph data G = (V, E, X) into the graph neural network module to obtain the block embedding vector of each image block; taking the block embedding vector plus the position embedding vector as the input of the Transformer encoder to obtain the representation Z = {z i |i=1,2,..,N 2 Inputting the representation vector of the image into the detection head for prediction to obtain a detection result for the litchi image, wherein the detection result includes the category data of the litchi in the image and the position and size of the bounding box.

[0080] Among them, the graph neural network module aggregates the representation vectors of neighbor nodes as part of the target node representation vector update through message aggregation, strengthening the connection between image blocks and the local information of the image.

[0081] The Transformer encoder combines the block embedding vectors obtained by the graph neural network module with the learnable position embedding vectors to process the image from a global perspective, generating an image representation. Due to the preprocessing by the graph neural network module, the image representation output by the Transformer encoder encompasses both global and local information, ensuring the richness of the representation vector.

[0082] The detection head takes the image representation as input and detects the litchi whose bounding box is located in each image block, obtaining the position, size, and category information of the litchi bounding box. In this embodiment, the graph neural network improves the visual Transformer, enhancing local information while preserving global information. This provides the detection head with a richer image representation, thereby improving the accuracy of litchi detection in complex environments and under occlusion.

[0083] In one embodiment, the step of inputting the constructed graph data into the graph neural network module to obtain the block embedding vector of each image block includes: according to the constructed graph data, using the aggregation function to embed the node v i Aggregate messages from neighbor nodes and get the aggregated neighbor node vector representation a i , the message is the vector representation x of the neighbor node j ; Splicing node v i The vector representation of x i And the aggregated neighbor node vector represents a i ; Use update function to update node v i The vector representation is updated to get the updated v i The vector representation of x i'; Use the updated vector to represent x i ', repeat the above aggregation, splicing, and updating operations to obtain the vector representation x of each node after being processed by the graph neural network i ”, that is, the block embedding vector, where the vector represents x i The dimension of "is d"; the above process is expressed as follows:

[0084] a i =aggregate({x j :v j ∈N(v i )})

[0085] x i '=update([x i ,a i ])

[0086] a i '=aggregate({x' j :v j ∈N(v i )})

[0087] x i ”=update([x i ',a i '])

[0088] Among them, aggregate(·) is the aggregation function; update(·) is the update function, and [·,·] is the concatenation operation.

[0089] Among them, Figure 4 As shown in Figure 2, the aggregation function uses mean aggregation, which calculates the average value of the representation vectors of neighboring nodes along the dimension direction. Mean aggregation can fully capture the distribution information of each attribute in the neighboring nodes, thereby strengthening the connection between similar nodes. The formula for mean aggregation is as follows:

[0090]

[0091] The update function is a linear layer with the following formula:

[0092] update([x i ',a i '])=w update [x i ',a i ']

[0093] In one embodiment, Figure 4As shown, the Transformer encoder includes: adding each block embedding vector to the position embedding vector of the same dimension, where the position embedding vector is learnable, to obtain a spliced embedding vector; inputting the spliced embedding vector into the Transformer encoder, first normalizing the features through a layer normalization operation, to obtain a normalized embedding vector; inputting the normalized embedding vector into the attention module to obtain a vector representation b after processing by the attention module i , and the vector before layer normalization and the vector after attention module processing are represented as b i Add, that is, residual connection, to get the representation b after the initial residual connection i '; The representation b after the initial residual connection i 'After layer normalization, input into the multilayer perceptron (MLP), and obtain the processed result and then connect it with the initial residual to form the representation b i 'Add and get the representation b after residual connection again" i ; Repeat the above operation L times to obtain the final representation Z of the entire image.

[0094] The position embedding vector is a learnable parameter that adjusts based on the loss during model training. It simply represents the position of each image block in the image as a vector, thus ensuring the order between image blocks.

[0095] Among them, the layer normalization operation can control the gradient size during training, thereby alleviating the problems of gradient disappearance and gradient explosion during training and improving the training speed and accuracy of the model.

[0096] Among them, the attention module calculates the attention scores between each image block, and then performs weighted processing on each image block, so that the network can better focus on the key image blocks.

[0097] Among them, the residual connection reduces the complexity of the model, reduces overfitting and prevents gradient disappearance.

[0098] In one embodiment, the normalized embedding vector is input into the attention module to obtain the vector representation b after processing by the attention module. i , including: multiplying the embedding vector of each image block by the learnable transformation matrix w q 、w k 、w v Then get q i 、k i 、v i ; q i k with other image blocks iPerform inner product operation and obtain the attention score α between each image block after softmax operation; multiply the attention score by v of each image block i , get the vector representation b of each image block after attention module processing i .

[0099] The above process can be expressed as the following formula:

[0100]

[0101] Where B = {b i |i=1,2,...,N 2} is the processed image block representation b i The set of Q, K, and V are q i 、k i 、v i A collection of .

[0102] In one embodiment, the inputting of the image representation vector into the detection head for prediction includes: predicting the representation vector of each image block separately, using three anchor boxes of preset sizes for prediction of each image block, and obtaining the prediction results corresponding to each image block; wherein the prediction results include the coordinates x of the center point of the bounding box of the litchi. box 、y box , the width w and height h of the bounding box, the category class of the litchi; the anchor box is used to assist in the prediction of the position and size of the litchi bounding box; the prediction results corresponding to all image blocks are screened using non-maximum suppression to obtain the final litchi detection result; the image block where the center point of the litchi bounding box is located is responsible for detecting the litchi.

[0103] Anchor boxes are pre-set boxes used to assist in lychee bounding box prediction. They are obtained by counting the sizes and proportions of common bounding boxes of lychee fruits in an image. The height and width of a lychee bounding box are expressed as a ratio relative to the bounding box. Setting anchor boxes transforms the lychee bounding box prediction problem into a regression problem, facilitating the setting of loss functions, model training, and model parameter updates.

[0104] The prediction results corresponding to all image blocks are screened using the non-maximum suppression method to obtain the litchi bounding box data. This process involves sorting all anchor box prediction results by confidence; taking the most confident litchi bounding box as the benchmark and removing all predictions whose IoU ratio with the benchmark is greater than a threshold; then taking the second most confident litchi bounding box from the remaining predictions as the benchmark and removing all predictions whose IoU ratio with the benchmark is greater than a threshold; and repeating this process by taking the third most confident litchi bounding box from the remaining predictions as the benchmark until the least confident prediction is used as the benchmark. Finally, all predictions selected as the benchmark constitute the litchi bounding box data. Confidence represents the model's confidence in the prediction, and IoU represents the ratio of the area of the intersection to the area of the union of two litchi bounding boxes.

[0105] It should be understood that although Figure 1 The steps in the flowchart are shown in sequence as indicated by the arrows, but these steps are not necessarily executed in the order indicated by the arrows. Unless otherwise specified in this document, there is no strict order restriction for the execution of these steps, and these steps can be executed in other orders. In addition, Figure 1 At least part of the steps may include multiple steps or multiple stages. These steps or stages are not necessarily performed at the same time, but can be performed at different times. The order of execution of these steps or stages is not necessarily one by one, but can be performed in turn or alternately with other steps or at least part of the steps or stages in other steps.

[0106] In one embodiment, Figure 5 As shown, a litchi image detection device is provided, comprising: an image acquisition module 210 to be detected, an image data preprocessing module 220 and a target detection module 230, wherein:

[0107] The image acquisition module 210 is used to acquire the grape image to be detected.

[0108] The image data preprocessing module 220 is used to divide the litchi image to be detected into image blocks, and construct edges according to the similarity between the image blocks to form graph data including nodes and edges.

[0109] The target detection module 230 is used to input the constructed graph data into the visual Transformer model improved based on the graph neural network to obtain the category data and bounding box position of the litchi; wherein, the constructed graph data is input into the graph neural network module to obtain the block embedding vector of each image block; the block embedding vector plus the position embedding vector is input into the Transformer encoder, and then predicted by the detection head to obtain the detection result of the litchi image.

[0110] In one embodiment, the image data preprocessing module 220 includes: an image division unit for dividing the litchi image to be detected into N×N image blocks, each image block has a size of P1×P2×3, and each image block is regarded as a node in the graph data, and a node set V={v i |i=1,2,..,N 2}; where N is a positive integer, i={1,2,…,N 2} is the number of the image block; the image conversion unit is used to perform convolution operation on each image block through a one-dimensional convolution layer to obtain a vector representation of each image block Where d is the dimension of the image block vector representation, and the vector representation X={x i |i=1,2,..,N 2} is considered as a node attribute in the Graph data; the edge construction unit is represented by the calculation vector x i The similarity between them is measured. For each image block, the top k image blocks with similarity are found and regarded as the neighbor nodes of the image block. Directed edges are constructed between these vector blocks to obtain the edge set E, where k is the number of neighbors of each node.

[0111] In one embodiment, the target detection module 230 includes: a feature preprocessing unit for inputting the constructed graph data G = (V, E, X) into the graph neural network module to obtain a block embedding vector for each image block; a feature encoding unit for adding the block embedding vector to the position embedding vector as the input of the Transformer encoder to obtain the representation (embedding) Z = {z i |i=1,2,..,N 2 A detection unit is configured to input the representation vector of the image into a detection head for prediction, thereby obtaining a detection result for the litchi image, wherein the detection result includes category data of the litchi in the image and a position and size of a bounding box.

[0112] In one embodiment, the feature preprocessing unit further includes: a neighbor node vector representation aggregation subunit for using an aggregation function to represent the node v according to the constructed graph data. i Aggregate messages from neighbor nodes and get the aggregated neighbor node vector representation a i , the message is the vector representation x of the neighbor node j ; Splicing node v i The vector representation of x i And the aggregated neighbor node vector represents a i; Update subunit, used to use the update function to update node v i The vector representation is updated to get the updated v i The vector representation of x i '; Use the updated vector to represent x' i , a repeating unit, used to repeat the above aggregation, splicing, and updating operations to obtain the vector representation x of each node after being processed by the graph neural network" i , which is a block embedding vector, where the vector represents x" i The dimension is d';

[0113] The above process is expressed as follows:

[0114] a i =aggregate({x j :v j ∈N(v i )})

[0115] x i '=update([x i ,a i ])

[0116] a i '=aggregate({x' j :v j ∈N(v i )})

[0117] x i ”=update([x i ',a i '])

[0118] Among them, aggregate(·) is the aggregation function; update(·) is the update function, and [·,·] is the concatenation operation.

[0119] In one embodiment, the feature encoding unit further includes: an embedding subunit for adding a position embedding vector of the same dimension to each block embedding vector, wherein the position embedding vector is learnable, to obtain a concatenated embedding vector; a normalization subunit for inputting the concatenated embedding vector into the Transformer encoder, first normalizing the features through a layer normalization operation, to obtain a normalized embedding vector; a residual connection subunit for inputting the normalized embedding vector into the attention mechanism, to obtain a vector representation b after the attention mechanism processing. i , and the vector before layer normalization and the vector after attention module processing are represented as b i Add, that is, residual connection, to get the representation b after the initial residual connectioni '; Multilayer perceptron processing subunit, used to represent b after the initial residual connection i 'After layer normalization, input into the multilayer perceptron (MLP), and obtain the processed result and then connect it with the initial residual to form the representation b i 'Add and get the representation b after residual connection again i "; A repeating unit is used to repeat the above operation L times to obtain the final representation Z of the entire image.

[0120] In one embodiment, the normalized embedding vector is input into the attention module to obtain the vector representation b after processing by the attention module. i , including: multiplying the normalized embedding vector of each image block by the learnable transformation matrix w q 、w k 、w v Then get q i 、k i 、v i ; q i k with other image blocks i Perform inner product operation and obtain the attention score α between each image block after softmax operation; multiply the attention score by v of each image block i , get the vector representation b of each image block after attention module processing i ;

[0121] The above process can be expressed as the following formula:

[0122]

[0123] Where B={b i |i=1,2,...,N 2} is the processed image block representation b i The set of Q, K, and V are q i 、k i 、v i A collection of .

[0124] In one embodiment, the detection unit includes: an anchor box prediction subunit, which is used to predict the representation vector of each image block separately, and each image block is predicted using three anchor boxes of preset sizes to obtain the prediction results corresponding to each image block; wherein the prediction results include the coordinates x of the center point of the bounding box of the litchi box 、y box, the width w and height h of the bounding box, the category class of the litchi; the anchor box is used to assist in the prediction of the position and size of the litchi bounding box; the screening sub-unit is used to filter the prediction results corresponding to all image blocks using non-maximum suppression to obtain the final litchi detection result; the image block where the center point of the litchi bounding box is located is responsible for detecting the litchi.

[0125] Regarding the specific limitations of the litchi detection device based on the improved visual Transformer of the graph neural network, please refer to the limitations of the litchi detection method based on the improved visual Transformer of the graph neural network above, which will not be repeated here. Each module in the above-mentioned litchi detection device based on the improved visual Transformer of the graph neural network can be implemented in whole or in part by software, hardware and a combination thereof. The above-mentioned modules can be embedded in or independent of the processor in the computer device in the form of hardware, or can be stored in the memory of the computer device in the form of software, so that the processor can call and execute the operations corresponding to the above modules.

[0126] In one embodiment, a computer device is provided. The computer device may be a server, and its internal structure diagram may be as follows: Figure 6 As shown. The computer device includes a processor, a memory and a network interface connected through a system bus. The processor of the computer device is used to provide computing and control capabilities. The memory of the computer device includes a non-volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system, a computer program and a database. The internal memory provides an environment for the operation of the operating system and the computer program in the non-volatile storage medium. The database of the computer device is used to store data of a visual Transformer model improved based on a graph neural network. The network interface of the computer device is used to communicate with an external terminal through a network connection. When the computer program is executed by the processor, a litchi detection method based on an improved visual Transformer of a graph neural network is implemented.

[0127] Those skilled in the art will understand that Figure 6 The structure shown in the figure is only a block diagram of a part of the structure related to the solution of the present application, and does not constitute a limitation on the computer device to which the solution of the present application is applied. The specific computer device may include more or fewer components than shown in the figure, or combine certain components, or have a different component arrangement.

[0128] In one embodiment, a computer device is further provided, including a memory and a processor. The memory stores a computer program, and the processor implements the steps in the above method embodiments when executing the computer program.

[0129] In one embodiment, a computer-readable storage medium is provided, on which a computer program is stored. When the computer program is executed by a processor, the steps in the above-mentioned method embodiments are implemented.

[0130] Those skilled in the art will appreciate that all or part of the processes in the above-mentioned embodiment methods can be implemented by instructing the relevant hardware through a computer program, and the computer program can be stored in a non-volatile computer-readable storage medium. When the computer program is executed, it can include the processes of the embodiments of the above-mentioned methods. Among them, any reference to memory, storage, database or other media used in the embodiments provided in this application may include at least one of non-volatile and volatile memory. Non-volatile memory may include read-only memory (ROM), magnetic tape, floppy disk, flash memory or optical memory, etc. Volatile memory may include random access memory (RAM) or external cache memory. As an illustration and not limitation, RAM can be in various forms, such as static random access memory (SRAM) or dynamic random access memory (DRAM).

[0131] The technical features of the above embodiments can be combined arbitrarily. To make the description concise, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0132] The above-described embodiments merely represent several implementation methods of the present application. While the descriptions are relatively specific and detailed, they should not be construed as limiting the scope of the present invention. It should be noted that a person skilled in the art could make various modifications and improvements without departing from the spirit of the present application, all of which fall within the scope of protection of the present application. Therefore, the scope of protection of the present patent application shall be determined by the appended claims.

Claims

1. A method for detecting litchi images, characterized in that: The method comprises: Obtaining a litchi image to be detected; Dividing the litchi image to be detected into image blocks, and constructing edges based on similarities between the image blocks to form graph data including nodes and edges; The constructed graph data is input into the visual Transformer model improved based on the graph neural network to obtain the category data of litchi and the position and size of the bounding box; The constructed graph data is input into the graph neural network module to obtain the block embedding vector of each image block. Each block embedding vector is added to the embedding vector of the position with the same dimension to obtain the spliced embedding vector. The spliced embedding vector is input into the Transformer encoder, and then predicted by the detection head to obtain the detection result of the litchi image. The step of inputting the constructed graph data into the graph neural network module to obtain a block embedding vector for each image block includes: According to the constructed graph data, the aggregation function is used to calculate the node v i Aggregate messages from neighbor nodes and get the aggregated neighbor node vector representation a i , the message is the vector representation x of the neighbor node j ; Splicing node v i The vector representation of x i And the aggregated neighbor node vector represents a i ; Use the update function to update the node v i The vector representation is updated to get the updated v i The vector representation x′ i ; Use the updated vector to represent x′ i Repeat the above aggregation, splicing, and updating operations to obtain the block embedding vector x″ of each node after the graph neural network processing i , where the block embedding vector x″ i The dimension is d″; The above process is expressed as follows: a i =aggregate({x j :v j ∈N(v i )}) Among them, aggregate(·) is the aggregation function; update(·) is the update function, and [·,·] is the concatenation operation.

2. The method according to claim 1, characterized in that The step of dividing the litchi image to be detected into image blocks and constructing edges according to similarities between the image blocks to form graph data includes: The litchi image to be detected is divided into N×N image blocks, the size of each image block is P1×P2×3, each image block is regarded as a node in the graph data, and the node set V={v i |i=1,2,...,N 2 }; where N is a positive integer, i = {1, 2, ..., N 2 } is the number of the image block; Each image block is convolved through a one-dimensional convolutional layer to obtain a vector representation of each image block. Where d is the dimension of the image block vector representation, and the vector representation X={x i |i=1,2,...,N 2 } is considered as a node attribute in the graph data; By calculating the vector representation x i The similarity between them is measured, and for each image block, the top k image blocks with the highest similarity are found, which are regarded as the neighbor nodes of the image block. Directed edges are constructed between these vector blocks to obtain the edge set E; where k is the number of neighbors of each node.

3. The method according to claim 1, characterized in that The constructed graph data is input into the visual Transformer model improved based on the graph neural network to obtain the category data and bounding box location of the litchi, including: Input the constructed graph data G = (V, E, X) into the graph neural network module to obtain the block embedding vector of each image block; The block embedding vector plus the position embedding vector is used as the input of the Transformer encoder to obtain the representation vector Z = {z i |i=1,2,...,N 2 }; The image representation vector Z = {z i |i=1,2,...,N 2 }Input the detection head for prediction to obtain the detection result of the litchi image; wherein the detection result includes the category data of the litchi in the image and the position and size of the bounding box.

4. The method according to claim 3, characterized in that The block embedding vector plus the position embedding vector is used as the input of the Transformer encoder to obtain the representation of the entire image, including: The concatenated embedding vector is input into the Transformer encoder, and the features are normalized by layer normalization to obtain the normalized embedding vector. The normalized embedding vector is input into the attention module to obtain the vector representation b after attention module processing i , and the vector before layer normalization and the vector after attention module processing are represented as b i Add together to get the representation b′ after the initial residual connection i ; The representation b′ after the initial residual connection i After layer normalization, the result is input into the multi-layer perceptron, and the processed result is connected with the initial residual to form the representation b′ i Add them together to get the representation b″ after residual connection again i ; Repeat the above operation L times to obtain the final representation Z of the entire image.

5. The method according to claim 4, characterized in that The normalized embedding vector is input into the attention module to obtain the vector representation b after processing by the attention module i ,include: Multiply the normalized embedding vector of each image block by the learnable transformation matrix w q 、w k 、w v Then get q i 、k i 、v i ; q i k with other image blocks i Perform inner product operation and softmax operation to obtain the attention score α between each image block; Multiply the attention score by v of each image block i , get the vector representation b of each image block after attention module processing i ; The above process can be expressed as the following formula: Where B={b i |i=1,2,...,N 2 } is the processed image block representation b i The set of Q, K, and V are q i 、k i 、v i A collection of .

6. The method according to claim 3, characterized in that The image representation vector Z={z i |i=1,2,...,N 2 Input detection head for prediction, including: The representation vector Z of each image block is i |i=1,2,...,N 2 } perform separate predictions, using three anchor boxes of preset sizes for each image block to obtain the prediction results corresponding to each image block; the prediction results include the coordinates of the center point of the bounding box of the litchi x box 、y box , the width w and height h of the bounding box, the category class of litchi; the anchor box is used to assist in the prediction of the position and size of the litchi bounding box; The prediction results corresponding to all image blocks are screened using non-maximum suppression to obtain the final litchi detection result; the image block where the center point of the litchi bounding box is located is responsible for detecting the litchi.

7. A litchi image detection device, characterized in that: The device comprises: The image acquisition module to be detected is used to obtain the litchi image to be detected; An image data preprocessing module is used to divide the litchi image to be detected into image blocks, and construct edges based on the similarity between the image blocks to form graph data containing nodes and edges; The object detection module is used to input the constructed graph data into a visual Transformer model improved by the graph neural network to obtain the category data and bounding box location of the litchi. The constructed graph data is input into the graph neural network module to obtain the block embedding vector of each image block. Each block embedding vector is added to the embedding vector of the position with the same dimension to obtain a spliced embedding vector. The spliced embedding vector is input into the Transformer encoder, and then predicted by the detection head to obtain the detection result of the litchi image. The step of inputting the constructed graph data into the graph neural network module to obtain a block embedding vector for each image block includes: According to the constructed graph data, the aggregation function is used to calculate the node v i Aggregate messages from neighbor nodes and get the aggregated neighbor node vector representation a i , the message is the vector representation x of the neighbor node j ; Splicing node v i The vector representation of x i And the aggregated neighbor node vector represents a i ; Use the update function to update the node v i The vector representation is updated to get the updated v i The vector representation x′ i ; Use the updated vector to represent x′ i Repeat the above aggregation, splicing, and updating operations to obtain the block embedding vector x″ of each node after the graph neural network processing i , where the block embedding vector x″ i The dimension is d″; The above process is expressed as follows: a i =aggregate({x j :v j ∈N(v i )}) Among them, aggregate(·) is the aggregation function; update(·) is the update function, and [·,·] is the concatenation operation.

8. A computer device comprising a memory and a processor, wherein the memory stores a computer program, wherein: When the processor executes the computer program, the steps of the method according to any one of claims 1 to 6 are implemented.

9. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the steps of the method according to any one of claims 1 to 6 are implemented.

Images