Blueberry picking system and method and computer medium

Abstract

The invention provides a blueberry picking system and method and a computer medium, an image acquisition module of the system obtains image information of at least two angles of a target fruit tree, a target detection module analyzes mature blueberry fruits in the fruit tree according to the image information by using a deep learning algorithm, and a control module sets a picking path and strategy of the blueberry fruits. The picking execution module executes the picking action of the blueberry fruits; the target detection module comprises an image division sub-module for dividing a blueberry image into a plurality of sub-regions, a first analysis sub-module for respectively capturing feature vectors of the plurality of sub-regions and determining a target sub-region where a fruit tree is located according to the feature vectors, and a second analysis sub-module for analyzing and outputting the position, size and maturity of the blueberry fruit in the target sub-region. According to the method, each sub-region is independently analyzed, the target region with the fruits in the acquired image is focused, the background part is weakened and removed, and the interference of background information in the pattern is reduced, so that the accuracy of blueberry fruit recognition is improved.

Description

Technical Field

[0001] The present invention belongs to the technical field of automated planting, specifically relates to the technical field of blueberry picking, and particularly relates to a blueberry picking system, method and computer-readable storage medium. Background Art

[0002] Blueberry fruits are small and grow densely. The traditional manual picking method is inefficient, labor-intensive, and with the continuous increase of labor costs, there is an urgent need for an efficient and accurate automated picking technology. Existing picking robots have problems such as insufficient accuracy and susceptibility to environmental interference in target recognition and positioning, resulting in the picking success rate and efficiency being difficult to meet the actual production requirements. For example, the background is complex and chaotic: the field background includes soil, weeds, other plants, etc., whose colors and textures are somewhat similar to blueberry fruits, easily interfering with the recognition of fruits by the detection algorithm and making it difficult for the algorithm to accurately distinguish fruits from the background. Summary of the Invention

[0003] To solve the above problems, the purpose of the present invention is to provide a blueberry picking method, system, and computer-readable storage medium. This method divides the blueberry fruit image through an image partitioning method and independently analyzes each sub-region, focusing on the target regions where fruits exist in the captured image, weakening and removing the background part, and reducing the interference of background information in the pattern, thereby improving the accuracy of blueberry fruit recognition.

[0004] Based on this, the present invention provides a blueberry picking system, which is characterized by including an image acquisition module, a target detection module, a picking execution module, and a control module. The image acquisition module, the target detection module, and the picking execution module are all electrically connected to the control module; wherein,

[0005] The image acquisition module is used to obtain image information of the target fruit tree from at least 2 angles. The target detection module is connected to the image acquisition module and uses a deep learning algorithm to analyze the position, size, and maturity information of blueberry fruits in the fruit tree from the image information. The control module sets the picking path and strategy according to the blueberry fruit information output by the target detection module, and the execution module executes the picking action of blueberry fruits according to the path and strategy output by the control module;

[0006] The target detection module includes:

[0007] An image partitioning sub-module, connected to the acquisition module, for partitioning the image information into multiple sub-regions,

[0008] A first analysis sub-module, connected to the image partitioning sub-module, for respectively capturing the feature vectors of multiple sub-regions and determining the target sub-regions where the fruit tree is located according to the feature vectors, weakening the background regions,

[0009] The second analysis sub-module, connected to the control module and the first analysis sub-module, is configured to analyze and output the position, size, and maturity of the blueberry fruits in the target sub-region.

[0010] Furthermore, the image acquisition module includes a high-definition camera and / or a multispectral camera.

[0011] Furthermore, the first analysis sub-module includes:

[0012] The first feature extraction unit is configured to extract the feature vectors of the sub-regions divided by each of the image division sub-modules respectively.

[0013] The attention mechanism unit is configured to calculate the attention weight of each of the sub-regions according to the extracted feature vectors for multiple sub-regions, and determine the target sub-region where the blueberry fruits are located according to the attention weight.

[0014] The first output unit outputs the regional position where the target sub-region calculated by the attention mechanism unit is located.

[0015] Furthermore, the first analysis sub-module further includes an image registration unit, configured to adjust the image angle of the blueberry fruits through image registration for the image information of at least two angles acquired by the image acquisition module.

[0016] Furthermore, the second analysis sub-module includes:

[0017] The second feature extraction module is configured to extract the global features of the target sub-region.

[0018] The recognition unit is configured to recognize and analyze the position, size, and maturity of the blueberry fruits in the target sub-region image according to the global features.

[0019] The second output unit is configured to output the position information of the identified ripe blueberry fruits.

[0020] Furthermore, the picking execution module includes a fixture and a cutting device. The fixture is a flexible gripper, configured to grasp the blueberry fruits according to the path and strategy output by the control system, and the cutting device is configured to cut the fruit stalk.

[0021] The present invention also provides an automated blueberry picking method, including the following steps:

[0022] Obtain the image data of the blueberry fruits, and divide the image data into multiple sub-regions.

[0023] Obtain the feature vectors of the sub-regions and calculate the attention weights of each of the sub-regions, and obtain the target sub-region where the blueberry fruits are located.

[0024] Obtain the global features of the target sub-region, and analyze the three-dimensional coordinates, size, and maturity information of the blueberry fruits in the target sub-region according to the global features.

[0025] Further, there are at least two target regions, and the position, size, and maturity information of the blueberry fruits in each target sub-region are calculated and analyzed independently.

[0026] Further, there are at least two target regions. After obtaining the target sub-regions, image fusion is performed on the target regions to form image data with higher recognition.

[0027] The present invention also provides a computer device, including a memory, a processor, and a network interface. The memory stores a computer program, and when the processor executes the computer program, the steps of the road scene recognition and query method are implemented.

[0028] The present invention also provides a computer-readable storage medium, on which a computer program is stored. When the computer program is executed by a processor, the steps of the road scene recognition method are implemented.

[0029] Beneficial effects:

[0030] The present invention provides a blueberry picking system, method and computer-readable storage medium. The system includes an image acquisition module, a target detection module, a picking execution module and a control module. The image acquisition module, the target detection module and the picking execution module are all electrically connected to the control module. Among them, the image acquisition module is used to obtain image information of at least two angles of a target fruit tree. The target detection module is connected to the image acquisition module and uses a deep learning algorithm to analyze the image information to obtain the position, size and maturity information of blueberry fruits in the fruit tree. The control module sets a picking path and strategy according to the blueberry fruit information output by the target detection module, and the execution module executes the picking action of the blueberry fruits according to the path and strategy output by the control module. The target detection module includes: an image partitioning sub-module, connected to the acquisition module, for partitioning the image information into multiple sub-regions; a first analysis sub-module, connected to the image partitioning sub-module, for respectively capturing the feature vectors of the multiple sub-regions and determining the target sub-region where the fruit tree is located, weakening the background region; and a second analysis sub-module, connected to the control module and the first analysis sub-module, for analyzing and outputting the position, size and maturity of the blueberry fruits in the target sub-region. In this application, the collected blueberry fruit images are partitioned into multiple independent images through image partitioning, and each sub-region is analyzed independently, focusing on the target regions with fruits in the collected images, weakening and removing the background part, and reducing the interference of background information in the pattern, thereby improving the accuracy of blueberry fruit recognition. BRIEF DESCRIPTION OF THE DRAWINGS

[0031] In order to more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the following will briefly introduce the drawings required for use in the description of the embodiments or the prior art. Obviously, the following drawings are only some embodiments of the present invention. For those of ordinary skill in the art, without creative efforts, other drawings can also be obtained based on these drawings.

[0032] Figure 1 is an exemplary system architecture diagram to which the embodiments of the present invention can be applied;

[0033] Figure 2 is a schematic structural diagram of the blueberry picking system provided by the embodiments of the present invention;

[0034] Figure 3 is a schematic diagram of a blueberry picking method provided by the embodiments of the present invention;

[0035] Figure 4 Schematic structural diagram of an embodiment of another computer device according to the present invention. DETAILED DESCRIPTION OF THE EMBODIMENTS

[0036] Next, the technical solutions in the embodiments of the present invention will be clearly and completely described in conjunction with the accompanying drawings in the embodiments of the present invention. Obviously, the described embodiments are only a part of the embodiments of the present invention, rather than all the embodiments. All other embodiments obtained by those of ordinary skill in the art based on the embodiments of the present invention without creative efforts shall fall within the protection scope of the present invention.

[0037] As Figure 1 shown, the blueberry picking system 100 may include a picking end 101, an image acquisition device 102, a control center 103, a network 104, and a server 105. The network 104 is used to provide a medium for communication links among the picking end 101, the image acquisition device 102, the control center 103, and the server 105. The network 104 may include various connection types, such as wired or wireless communication links, etc.

[0038] The image acquisition device 102 includes image acquisition devices such as scanners and cameras, and is used to acquire image data of blueberry fruits. Then, it interacts with the server 105 through the network 104 to complete the interaction content of data reading. It sends the fruit image data of blueberry fruits and task requests such as picking requests to the control center 103 and the server 105 through the network 104. The picking end 101 includes a fixture and a robotic arm, and picks the target blueberry fruits according to the picking path and picking strategy set by the control center 103 based on the image data.

[0039] The image acquisition device 102 and the control center 103 may be various electronic devices with a display screen, including but not limited to user devices, network devices, or devices formed by integrating a user device and a network device through a network. The user device includes but is not limited to any intelligent terminal electronic product that can perform human-computer interaction with a user through a touchpad or buttons, such as an intelligent scanner, a smart phone, a tablet computer, etc. The recognition terminal includes a data acquisition module and a recognition module, and the acquisition module includes image collectors such as scanners and cameras.

[0040] The control center 103 may be integrated and loaded on the acquisition module or may be a product independent of the acquisition module. The mobile electronic product may adopt any operating system, such as the android operating system, the IOS operating system, etc. Among them, the network device includes an electronic device that can automatically perform numerical calculations and information processing according to pre-set or stored instructions, and its hardware includes but is not limited to a microprocessor, an application specific integrated circuit (ASIC), a field programmable gate array (FPGA), a digital signal processor (DSP), an embedded device, etc.

[0041] The network 104 device includes, but is not limited to, a computer, a network host, a single network server, a set of multiple network servers, or a cloud composed of multiple servers; here, the cloud is composed of a large number of computers or network servers based on cloud computing, where cloud computing is a type of distributed computing and consists of a virtual supercomputer formed by a group of loosely coupled computers. The network includes, but is not limited to, the Internet, a wide area network, a metropolitan area network, a local area network, a VPN network, a wireless ad hoc network (Ad Hoc network), etc. Of course, those skilled in the art should understand that the above terminal devices are only examples, and other existing or future terminal devices that can be applied to this application should also be included within the protection scope of this application and are hereby incorporated by reference.

[0042] The server 105 is the server side of the blueberry picking system application and can communicate with the picking end 101, the image acquisition device 102, and the control center 103 through the network 104. The picking end 101, the image acquisition device 102, and the control center 103 can also be connected and communicate with each other in pairs or in multiple parties. The server 105 can be a single server, or a server cluster composed of several servers, or a cloud computing service center. It can also be a server that provides various services, such as a background server that supports the pages displayed on the identification terminals 101, 102, and 103.

[0043] It should be understood that Figure 1 the numbers of the picking end 101, the image acquisition device 102, the control center 103, the network 104, and the server 105 in

[0044] are only illustrative. According to the implementation requirements, there can be any number of picking ends 101, image acquisition devices 102, control centers 103, networks 104, and servers 105. Figure 2 As

[0045] shown, the present invention provides a blueberry picking system 200, including an image acquisition module 201, a target detection module 202, a picking execution module 203, and a control module 204. The image acquisition module 201, the target detection module 202, and the picking execution module 203 are all electrically connected to the control module 204. Among them

[0046] Target detection module 202: Using deep learning algorithms, it performs real-time analysis and processing on the collected images. Through training with a large amount of blueberry fruit image data, this module can accurately identify information such as the position, size, and maturity of blueberry fruits. Even when part of the fruit is blocked, it can accurately locate, effectively filter out interference factors such as the background and branches and leaves, and output the three-dimensional coordinate information of mature blueberry fruits.

[0047] Harvesting execution module 203: According to the coordinate information provided by the target detection module, the robotic arm quickly and accurately moves to the position of the target fruit. The harvesting actuator adopts a flexible gripper design, which can automatically adjust the clamping force according to the fruit size, gently grasp the fruit, avoid damaging the fruit, and is also equipped with a cutting device to cut off the fruit stalk at the moment of grasping the fruit, completing the harvesting action.

[0048] Control module 204: Integrates functions such as motion control, image processing, and data communication, and coordinates the work processes among the image acquisition, target detection, and harvesting execution modules. According to the actual situation at the harvesting site, it adjusts the moving path and harvesting strategy of the robot in real time to ensure the efficient and stable operation of the harvesting process.

[0049] Specifically, the image acquisition module 201 is used to obtain image information of the blueberry fruit tree from at least 2 angles. The target detection module 202 is connected to the image acquisition module 201, uses deep learning algorithms to analyze the collected blueberry image information, and determines the position, size, and maturity information of the blueberry fruits. The control module 204 sets the moving path of the robotic arm according to the position coordinates of the blueberry fruits output by the target detection module 202, and sets the clamping width and clamping force of the end fixture according to the fruit size and maturity information. The harvesting execution module 203 performs the harvesting action of the blueberry fruits according to the path and strategy output by the control module 204.

[0050] It should be noted that the target detection module 202 can fuse the image information of the blueberry fruit tree from at least 2 angles obtained by the image acquisition module 201 to obtain a new image with a higher scale and level, obtain richer image information, thereby extracting image features with higher recognition, and then perform blueberry fruit detection; it can also compare the obtained image information and then select an image with better image angle and image effect for target detection.

[0051] In the embodiment of the present invention application, the target detection module 202 includes an image division sub-module, a first analysis sub-module, and a second analysis sub-module, wherein the image division sub-module is connected to the image acquisition module, and the second analysis sub-module is connected to the control module 204.

[0052] Specifically, the image division sub-module is used to divide the image information into multiple sub-regions. The first analysis sub-module is used to capture the feature vectors of multiple sub-regions respectively and determine the target sub-region where the blueberry fruit trees are located in the fruit trees according to the feature vectors, weakening the background region and focusing on the key region where the blueberry fruits are located. The second analysis sub-module is used to analyze and output the size, maturity and three-dimensional coordinates of the target blueberry object in the target sub-region.

[0053] Exemplarily, in the embodiment of the present invention application, the image division sub-module divides the original image collected by the image acquisition module 201 based on the region growing algorithm. First, a "seed" is selected from the original image. Starting from this "seed" pixel point, according to the preset similarity criterion, the surrounding qualified pixel points are continuously merged into the same region, and a complete region is gradually grown. In this way, different growing regions constitute the segmented sub-images. For example, a threshold range is set for the gray value of the image pixels. First, a known pixel point inside the tissue is selected as the seed, and then the surrounding image is traversed to determine whether the adjacent pixels meet the set similarity conditions, and it is continuously expanded according to the similarity in gray features with the surrounding pixels until all the pixels within the corresponding threshold range in the image are included.

[0054] In other embodiments, the image division sub-module may perform image segmentation through other algorithms, such as threshold segmentation, clustering analysis and other methods or algorithms.

[0055] It should be noted that the first analysis sub-module includes a first feature extraction unit, an attention mechanism unit and a first output unit. Among them, the first feature extraction unit is connected to the image division sub-module and is used to extract the feature vectors of the sub-regions divided by each image division sub-module respectively. The attention mechanism unit is used to calculate the attention weights of each sub-region respectively or in parallel according to the extracted feature vectors of multiple sub-regions, and determine the target sub-region where the blueberry fruit trees are located according to the attention weights. The first output unit outputs the region position of the target sub-region calculated by the attention mechanism unit.

[0056] It should be understood that the attention mechanism unit analyzes each sub-region divided by each image division sub-module separately, calculates the region where the blueberry fruit trees exist, and ignores the background or other irrelevant regions. On the one hand, independent analysis can reduce the calculation intensity of the data. On the other hand, each sub-region can be analyzed targeted, the region where the blueberry fruit trees are located can be more accurately located, and the accuracy of image analysis is higher.

[0057] Exemplarily, the attention mechanism unit is a model trained based on deep learning. By collecting a large number of image samples with annotations (annotating the blueberry fruit tree area and non-blueberry fruit tree area), taking the extracted features as input data and the area category as the output label, and using machine learning algorithms (such as support vector machines, decision trees, etc.) for training, the trained model can automatically learn the importance degree of different features for distinguishing the blueberry fruit tree area from other areas, that is, obtain the weights corresponding to the features extracted by the first feature extraction unit 301. For example, training with a large number of images containing blueberry fruit trees (including fruits) and backgrounds, the trained model will determine the appropriate weights of features such as color, shape, and texture for identifying the blueberry fruit tree area, and then this model can be used to process new images to screen out the target sub-image area where the blueberry fruit tree is located.

[0058] Furthermore, the Scikit-Learn library can be used to conveniently implement the training and application processes of these machine learning algorithms. By defining feature vectors, label vectors, and selecting appropriate models for training, prediction, etc. Specifically, the training process includes the following steps:

[0059] Step 1: Data preparation and data partitioning.

[0060] (1). Data preparation, the Scikit-Learn library can directly load and organize various blueberry fruit tree image data in a qualified format for training and testing. The image data includes feature data and label data. The feature data are all features extracted from the images, including but not limited to feature vectors composed of features such as color and shape, and the area corresponding to each image is well annotated, where the blueberry fruit tree area is annotated as 1 and the non-blueberry fruit tree area is annotated as 0.

[0061] (2). Partition the dataset: Take the prepared original images and the corresponding marked data together as a trainable data, and form the original dataset with all the trainable data; randomly split the original data into 2 subsets in a ratio of 8:2, which are used as the training set and the test set respectively; the training set and the test set are based on the original dataset, and the two subsets are independently distributed in the image space and have no duplicates.

[0062] Step 2: Feature selection and initial weight assignment. Preliminary screening of features is performed before training starts, or initial attention weights are assigned to the features. These initial weights are based on prior knowledge or simple statistical analysis. For example, the shape, color, etc. of the blueberry fruit tree are highly relevant to the blueberry fruit tree, and higher initial attention weights are assigned.

[0063] Step 3: Construct a decision tree (node splitting). It should be understood that the construction of a decision tree is a recursive process. At each node, a feature needs to be selected to split the node in order to separate samples of different classes to the greatest extent. The splitting node can be selected by calculating metrics such as the information gain and Gini impurity of the blueberry fruit tree features. Specifically, let the dataset prepared in Step 1 be D, the number of class labels be k, and Ck represent the subset of samples belonging to the k-th class. The feature A has v different values {a1, a2, a3, …, av}, and Dv represents the subset of samples where the value of the feature A is av. Then the empirical entropy of the data D is: Among them, where |Ck| is the number of samples in class k, |D| is the total number of samples in the dataset D, and pk is the probability of class k in the dataset D, then:

[0064] The empirical conditional entropy of the feature A with respect to the dataset D is:

[0065] The information gain is: g(D, A) = H(D) - H(D|A).

[0066] It should be understood that the role of a feature in node splitting can be regarded as a manifestation of its attention weight. When splitting a node, the feature with the largest information gain can be selected, and the attention weight (importance) of this feature in this split is relatively high. Exemplarily, when calculating the information gain, the feature that can maximize the information gain is selected to split the node and is given a higher attention weight. In the classification of the blueberry fruit tree area, the information gain of the feature "blueberry fruit tree" is calculated as follows: First, calculate the entropy of the blueberry fruit tree, then calculate the entropy under the condition of the blueberry fruit tree, and subtract the two to obtain the information gain.

[0067] On the other hand, calculate the Gini impurity of the feature. Among them, the Gini impurity of the data D is The Gini impurity after the feature A is partitioned is: When splitting a decision tree node, the feature with the smallest Gini impurity is selected as the splitting feature. For example, the Gini impurity of the blueberry plant color feature is calculated as follows: First, calculate the Gini impurity of this feature, then compare the Gini impurities of different features, and select the feature with the smallest Gini impurity.

[0068] It should be understood that the splitting node splits the node into multiple child nodes according to the selected features. For example, for the blueberry fruit tree area, if the feature "shape" is selected and this feature has multiple values, such as the shapes of different parts of the blueberry fruit tree (such as fruits and leaves), then the node is split into the corresponding number of child nodes, each corresponding to a value. It should be noted that during the splitting process, the samples are assigned to the corresponding child nodes. For example, those containing blueberry fruits are assigned to one child node, and those containing leaf shapes are assigned to another child node.

[0069] Step 4: Recursively construct the tree structure, repeating the process of node splitting in Step 3 until the stopping condition is met. The stopping condition may include the maximum depth of the tree, the minimum number of samples in a node, etc. During this process, the role of each feature in different node splittings will accumulate continuously, thus forming the comprehensive attention weight of each feature for the entire classification process. Specifically, each node contains information about the feature, the splitting condition, and the child nodes. For example, the root node contains the root feature, and the child nodes of the root node contain the corresponding splitting features and splitting conditions. During the construction process, information such as the depth, parent node, and child nodes of each node is recorded for subsequent analysis and processing.

[0070] Step 5: Model evaluation and adjustment

[0071] (1). Model evaluation, using the test dataset to evaluate the trained decision tree classifier. Calculate metrics such as accuracy, precision, and recall. Among them:

[0072] The formula for accuracy is

[0073] where TP (True Positive) is the true positive, that is, the number of samples predicted as positive and actually positive; TN (True Negative) is the true negative, that is, the number of samples predicted as negative and actually negative; FP (False Positive) is the false positive, that is, the number of samples predicted as positive but actually negative; FN (False Negative) is the false negative, that is, the number of samples predicted as negative but actually positive. In the classification of the blueberry fruit tree area, this formula comprehensively considers the ratio of the number of samples with correct predictions for all categories to the total number of samples.

[0074] Precision: The formula for macro-average precision is where

[0075] The precision for each category i is calculated separately and then averaged.

[0076] Recall: The formula for macro-average recall is

[0077] Among them

[0078] Similarly, the recall rate is calculated for each category i separately, and then the average value is obtained.

[0079] Evaluate the performance of the model, such as whether there are problems such as overfitting and underfitting. The performance of the model can be evaluated by methods such as drawing learning curves and cross-validation.

[0080] (2). Adjust the attention weights

[0081] According to the evaluation results, adjust the attention weights of the features. For example, if it is found that a certain feature performs poorly in the model, its attention weight can be reduced.

[0082] Retrain the model with the adjusted attention weights. For example, after adjusting the weights, retrain the decision tree classifier and observe whether the performance of the model improves.

[0083] Continuously repeat the evaluation and adjustment process until the model reaches satisfactory performance.

[0084] It should be noted that the number of the target sub-regions is at least one. During the prediction process, multiple sub-regions with attention weights exceeding the preset threshold can be used as the target regions where blueberry fruit trees exist. The first output unit outputs the regional positions where one or more of the target sub-regions calculated by the attention mechanism unit are located.

[0085] Furthermore, the second analysis sub-module receives the regional positions output by the first output unit to obtain the images and global features of the target sub-regions, and then further analyzes each target sub-region to detect blueberry fruits based on the trained blueberry recognition model, analyzes the maturity of the blueberry fruits, and calculates and analyzes the mature blueberry fruits, their positions and sizes, and finally outputs information such as the size, maturity, and three-dimensional coordinates of the mature blueberry fruits.

[0086] Specifically, the second analysis sub-module includes a second feature extraction module, an identification unit, and a second output unit. Among them, the second feature extraction module is connected to the first identification module and is used to extract the global features of the target sub-regions.

[0087] The identification unit is used to identify and analyze the positions, sizes, and maturities of blueberry fruits in the target sub-region images according to the global features; the second output unit is used to output the position information, size, and maturity of the identified mature blueberry fruits. Among them, the identification unit is a blueberry recognition model trained based on the existing technology, and its training process and identification method will not be elaborated here.

[0088] It should be noted that the global features include, but are not limited to, color features, texture features, shape features, and size features. Exemplarily, the recognition unit can first calculate and analyze the presence of blueberry fruits through texture features and shape features, calculate the positions of the blueberry fruits, and then, based on the analyzed blueberry fruits, analyze the maturity and size of the blueberry fruits corresponding to the positions according to color features and size features. Exemplarily, since blueberry fruits are generally round, the recognition unit can assist in judging the positions, sizes, etc. of blueberry fruits by calculating whether there are contours of similar blueberry fruits in the image. Texture analysis, such as the gray-level co-occurrence matrix, is used to extract texture features, and by comparing the standard parameters of blueberry fruits, it is judged whether there are blueberry fruits. Based on the comprehensive analysis results, if it is judged that there are blueberry fruits, the position information of the blueberry fruits is recorded. Then, the color histogram of the target sub-region is analyzed according to color features, the color distribution range is statistically analyzed, and the maturity and size information of the blueberry fruits determined in the image are obtained according to the typical color regions of blueberry fruits at different maturities.

[0089] In another embodiment of the present application, after receiving the area positions output by the first output unit and obtaining the corresponding target sub-regions, the second analysis sub-module can further fuse the obtained multiple target sub-regions to form image data with higher recognition accuracy, so as to obtain richer feature expressions, merge the features of multiple target sub-regions into a feature with stronger discriminative ability than the input features, thereby improving the recognition accuracy and precision.

[0090] Specifically, before fusing the target sub-regions, preprocessing of the target sub-region graphics can be performed, such as grayscale conversion, filtering, histogram equalization, normalization, image scaling, denoising, contrast enhancement, image rotation, and flipping, etc., to improve the image quality of the sub-regions, obtain standardized images, and extract more accurate features. Further, the preprocessed images are weighted and fused based on the multi-scale and multi-level features of the target sub-regions to obtain an image to be recognized with richer information, enhance the visual effect, provide more dimensional information of blueberry fruits, and improve the recognition accuracy at the same time.

[0091] In other embodiments of the present application, the second analysis sub-module can also independently analyze each of the obtained target sub-regions, and separately analyze the three-dimensional coordinates, fruit sizes, and maturity information of blueberry fruits within each target sub-region. On the one hand, it can reduce the computational intensity, improve the processing speed, and enhance the recognition efficiency. On the other hand, it can focus on the local features of blueberry fruits in the target sub-regions to improve the recognition accuracy of blueberry fruits.

[0092] In the embodiment of the present invention application, it is also possible to reduce the interference of the field background on the recognition result by adjusting the shooting angle of the blueberry fruit tree during the image acquisition stage for the image acquisition module 201. Specifically, the image acquisition module includes a high-definition camera, a multispectral camera, a light compensation unit, an automatic focusing device, etc. Among them, the high-definition camera and the multispectral camera collect image data of the fruits in the orchard in multiple spectral bands, and capture the appearance, color, and texture features of the fruits through the combination of different spectral bands. The light compensation unit dynamically adjusts the light source intensity under insufficient or uneven light conditions to ensure that the brightness and contrast of the collected images meet the high-precision requirements. The automatic focusing device adjusts the camera focal length in real time according to the distance and size of the fruits to obtain clear fruit image data.

[0093] It should be noted that the collected images include at least images from more than 2 angles. The blueberry plants can be photographed from different angles, such as the top and the side, to obtain the complete information of the blueberry plants. For example, the blueberry stalk part is easier to identify when photographed from the side, and multi-angle shooting can avoid missing these key features.

[0094] Furthermore, image acquisition devices such as high-definition cameras and multispectral cameras adjust the appropriate shooting angle during image acquisition to obtain an appropriate acquisition range, reducing the proportion of the field background while also avoiding losing the spatial connection between blueberries due to too small a shooting angle.

[0095] Furthermore, the first analysis sub-module further includes an image registration unit for adjusting the image angle of the blueberry fruit through image registration for the image information of at least 2 angles obtained by the image acquisition module, and precisely aligning the collected images spatially to ensure that the collected images are in the same coordinate system, so as to fuse the image information between multiple images and eliminate errors caused by different shooting conditions (such as viewing angle, light, etc.), thereby improving the accuracy of blueberry region location and blueberry fruit analysis.

[0096] In another embodiment of the present invention, as Figure 3 shown, an automated blueberry picking method is provided, including the following steps:

[0097] 31: Obtain the image data of the blueberry fruit and divide the image data into multiple sub-regions.

[0098] 32: Obtain the feature vectors of the sub-regions and calculate the attention weights of each sub-region to obtain the target sub-region where the blueberry fruit is located.

[0099] 33: Obtain the global features of the target sub-region and analyze the three-dimensional coordinates, size, and maturity information of the blueberry fruit in the target sub-region according to the global features.

[0100] It should be noted that the automated blueberry picking method in this embodiment belongs to the same concept as the embodiment of the picking system. The specific implementation process can be found in detail in the embodiment of the picking system, and the technical features in the embodiment of the picking system are correspondingly applicable in this embodiment, so they will not be elaborated here.

[0101] To solve the above technical problems, an embodiment of the present application also provides a computer device 400. Specifically, please refer to Figure 4 , Figure 4 which is the basic structural block diagram of the computer device in this embodiment.

[0102] The computer device 400 includes a memory 401, a processor 402, and a network interface 403 that are communicatively connected to each other through a system bus. It should be noted that only the computer device 400 with components 401 - 403 is shown in the figure, but it should be understood that it is not required to implement all the shown components, and more or fewer components can be alternatively implemented. Among them, those skilled in the art of the present technology can understand that a computer device here is a device that can automatically perform numerical calculations and / or information processing according to pre-set or stored instructions, and its hardware includes but is not limited to microprocessors, application specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), digital signal processors (DSPs), embedded devices, etc.

[0103] The computer device can be a desktop computer, a notebook, a palm computer, a cloud server, or other computing devices. The computer device can interact with users through a keyboard, a mouse, a remote control, a touchpad, a voice control device, or other means.

[0104] The memory 401 includes at least one type of readable storage medium, which includes flash memory, hard disk, multimedia card, card-type memory (such as SD or DX memory, etc.), random access memory (RAM), static random access memory (SRAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), programmable read-only memory (PROM), magnetic memory, magnetic disk, optical disk, etc. In some embodiments, the memory 401 may be an internal storage unit of the computer device 400, such as the hard disk or memory of the computer device 400. In other embodiments, the memory 401 may also be an external storage device of the computer device 4, such as a plug-in hard disk, Smart Media Card (SMC), Secure Digital (SD) card, Flash Card, etc. equipped on the computer device 400. Of course, the memory 401 may also include both the internal storage unit and the external storage device of the computer device 4. In this embodiment, the memory 401 is generally used to store the operating system and various application software installed in the computer device 4, such as the program code of the textile defect recognition method. In addition, the memory 401 may also be used to temporarily store various data that have been output or will be output.

[0105] In some embodiments, the processor 402 may be a central processing unit (CPU), a controller, a microcontroller, a microprocessor, or other data processing chips. The processor 402 is generally used to control the overall operation of the computer device 4. In this embodiment, the processor 402 is used to run the program code stored in the memory 401 or process data, such as running the program code of the textile defect recognition method.

[0106] The network interface 403 may include a wireless network interface or a wired network interface, and this network interface 403 is generally used to establish a communication connection between the computer device 400 and other electronic devices.

[0107] The above are only the preferred embodiments of the present invention. It should be noted that for those of ordinary skill in the art, without departing from the technical principle of the present invention, several improvements and substitutions can be made, and these improvements and substitutions should also be regarded as the protection scope of the present invention.

Claims

1. A blueberry picking system, characterized in that: It includes an image acquisition module, a target detection module, a picking execution module and a control module, wherein the image acquisition module, the target detection module and the picking execution module are all electrically connected to the control module; wherein, The image acquisition module is used to obtain image information of at least two angles of the target fruit tree, the target detection module is connected to the image acquisition module, and the image information is analyzed by a deep learning algorithm to obtain the position, size and maturity information of the blueberry fruits in the fruit tree, the control module sets a picking path and strategy according to the blueberry fruit information output by the target detection module, and the execution module executes the picking action of the blueberry fruits according to the path and strategy output by the control module; The target detection module comprises: An image division submodule, connected to the acquisition module, is used to divide the image information into multiple sub-areas, The first analysis submodule is connected to the image division submodule and is used to capture the feature vectors of the plurality of subregions respectively and determine the target subregion where the fruit tree is located according to the feature vectors, thereby weakening the background region. The second analysis submodule is connected to the control module and the first analysis submodule, and is used to analyze and output the position, size and maturity of the blueberry fruit in the target subregion.

2. The blueberry picking system according to claim 1, characterized in that: The image acquisition module includes a high-definition camera and / or a multi-spectral camera.

3. The blueberry picking system according to claim 1, characterized in that: The first analysis submodule comprises: a first feature extraction unit, configured to respectively extract feature vectors of the sub-regions divided by each of the image division sub-modules; an attention mechanism unit, for calculating the attention weight of each of the sub-regions according to the extracted feature vectors, and determining the target sub-region where the blueberry fruit is located according to the attention weights, The first output unit outputs the region position of the target sub-region calculated by the attention mechanism unit.

4. The blueberry picking system according to claim 1, characterized in that: The first analysis submodule further includes an image registration unit, which is used to adjust the image angle of the blueberry fruit through image registration of the image information of at least two angles acquired by the image acquisition module.

5. The blueberry picking system according to claim 1, characterized in that: The second analysis submodule includes: A second feature extraction module, used to extract global features of the target sub-region; an identification unit, configured to identify and analyze the position, size and maturity of blueberry fruits in the target sub-region image according to the global features; The second output unit is used to output the position information of the identified ripe blueberry fruit.

6. The blueberry picking system according to claim 1, characterized in that: The picking execution module includes a clamp and a cutting device, wherein the clamp is a flexible clamp used to grasp the blueberry fruit according to the path and strategy output by the control system, and the cutting device is used to cut off the fruit stalk.

7. An automated blueberry picking method, characterized in that: The following steps are involved: Acquire image data of blueberry fruits, and divide the image data into a plurality of sub-areas; Acquire the feature vector of the sub-region and calculate the attention weight of each sub-region to obtain the target sub-region where the blueberry fruit is located; The global features of the target sub-region are acquired, and the three-dimensional coordinates, size and maturity information of the blueberry fruits in the target sub-region are analyzed according to the global features.

8. The automated blueberry picking method according to claim 7, characterized in that: The target areas include at least two, and the position, size and maturity information of the blueberry fruits in each target sub-area are independently calculated and analyzed.

9. The automated blueberry picking method according to claim 7, characterized in that: The target area includes at least two, and after acquiring the target sub-area, image fusion is performed on the target sub-area to form image data with higher recognition degree.

10. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the steps of the textile defect identification method described in any one of claims 7 to 9 are implemented.

Images