Fruit grading method, device, electronic device, sorting system and storage medium

Abstract

The present invention provides a fruit grading method, device, electronic device, sorting system and storage medium. The method includes: performing maturity detection based on the local features, global features and fruit region features of the multispectral image to obtain the maturity level of the fruit to be graded; performing damage detection based on the image features of the fruit image and the feature maps of each channel in the multispectral image to obtain the damage level of the fruit to be graded; performing fruit grade classification based on the volume and weight of the fruit to be graded, as well as the maturity level and / or damage level, to obtain the fruit grade of the fruit to be graded, realizing the refinement and comprehensiveness of the fruit grading process, overcoming the defects of low accuracy and credibility of the fruit grade in the traditional solution, and an automated fruit grading process based on comprehensive information, reducing the manpower requirement, lowering the fruit grading cost, and realizing the double improvement of the accuracy and credibility of the fruit grade.

Description

Technical Field

[0001] The present invention relates to the technical field of fruit detection, and in particular to a fruit grading method, device, electronic device, sorting system and storage medium. Background Art

[0002] Fruit grading, as an important step before fruit packaging and selling, can screen out fruits with excellent quality and eliminate fruits with poor quality, and then grade them according to fruit size for the packaging, transportation and selling of fruits with different qualities.

[0003] Currently, when grading fruits, most of them directly measure the volume or weight of fruits and then classify the fruit grades according to the volume or weight, that is, directly grade them according to the fruit size on the sorting machine. However, the fruit size is only one of the indicators in the fruit grading standard. Only grading the fruit grades based on it, the accuracy and reliability of the obtained fruit grades are worrying. Further, in the fine fruit grading, currently it mainly relies on the workers on the sorting conveyor belt or the staff in the fruit store. This way of grading fruits manually not only has high costs but also low efficiency and poor practicability. Summary of the Invention

[0004] The present invention provides a fruit grading method, device, electronic device, sorting system and storage medium to solve the defects in the prior art that the fruit grading is too limited and rough, resulting in low accuracy and credibility of the obtained fruit grades, and to achieve the refinement and comprehensiveness of the fruit grading process, as well as the double improvement of the accuracy and credibility of the fruit grades.

[0005] The present invention provides a fruit grading method, including:

[0006] Obtain a fruit image and a multispectral image of the fruit to be graded;

[0007] Based on the local features, global features and fruit region features of the multispectral image, perform maturity detection to obtain the maturity grade of the fruit to be graded;

[0008] Based on the image features of the fruit image and the feature maps of each channel in the multispectral image, perform damage detection to obtain the damage grade of the fruit to be graded;

[0009] Based on the volume and weight of the fruit to be graded, and the maturity grade and / or the damage grade, perform fruit grade classification to obtain the fruit grade of the fruit to be graded.

[0010] According to the fruit grading method provided by the present invention, the steps for determining the volume and weight of the fruit to be graded include:

[0011] Downsample the fused point cloud image of the fruit to be graded, extract features based on multiple sampling points obtained from the downsampling to obtain sampling point features;

[0012] Perform voxel division on the fused point cloud image, and perform voxel feature encoding and sparse feature extraction on the point cloud image after voxel division respectively to obtain voxel features and depth voxel features, project the depth voxel features to obtain bird's-eye view features of different scales;

[0013] Based on the voxel features, the sampling point features, and the bird's-eye view features of different scales, perform segmentation detection, volume estimation, and weight estimation respectively to obtain a segmented point cloud image, as well as the volume and weight of the fruit to be graded.

[0014] According to a fruit grading method provided by the present invention, the fruit grading is performed based on the volume and weight of the fruit to be graded, as well as the maturity level and / or the damage level to obtain the fruit grade of the fruit to be graded, including:

[0015] Based on the segmented point cloud image, perform fruit shape fitting to obtain the maximum fruit shape contour of the fruit to be graded on a two-dimensional plane and a three-dimensional fruit shape contour;

[0016] Based on the maximum fruit shape contour and the three-dimensional fruit shape contour, determine the fruit shape grade of the fruit to be graded;

[0017] Based on the volume and weight of the fruit to be graded, the fruit shape grade, as well as the maturity level and / or the damage level, perform fruit grading to obtain the fruit grade of the fruit to be graded.

[0018] According to a fruit grading method provided by the present invention, the damage detection is performed based on the image features of the fruit image and the feature maps of each channel in the multispectral image to obtain the damage level of the fruit to be graded, including:

[0019] Based on the image features of the fruit image, perform epidermal damage detection to obtain the epidermal damage condition of the fruit to be graded;

[0020] Based on the feature maps of each channel in the multispectral image, perform subcutaneous damage detection to obtain the subcutaneous damage condition of the fruit to be graded;

[0021] Based on the epidermal damage condition and the subcutaneous damage condition, determine the damage level of the fruit to be graded.

[0022] According to a fruit grading method provided by the present invention, the epidermal damage detection is performed based on the image features of the fruit image to obtain the epidermal damage condition of the fruit to be graded, including:

[0023] Extract features from the fruit image to obtain the image features of the fruit image, and decode the image features to obtain the fruit feature map of the fruit image;

[0024] Reduce the number of channels of the fruit feature map to obtain a single-channel feature map, and perform semantic segmentation based on the single-channel feature map to obtain the regional image features of each segmentation region;

[0025] Fuse the regional image features of each region and the image features, and perform epidermal damage detection based on the fused features and the regional image features of each region to obtain the epidermal damage condition of the fruit to be graded.

[0026] According to a fruit grading method provided by the present invention, the method for detecting subcutaneous damage based on the feature maps of each channel in the multispectral image to obtain the subcutaneous damage condition of the fruit to be graded includes:

[0027] Extract features from the multispectral image to obtain multispectral image features, and decode the multispectral image features to obtain the multispectral feature map of the multispectral image;

[0028] Based on the correlation between each channel in the multispectral feature map, perform channel separation to obtain the feature map of each channel, and perform subcutaneous damage detection based on each segmentation region in the feature map of each channel to obtain the subcutaneous damage condition of the fruit to be graded.

[0029] According to a fruit grading method provided by the present invention, the method for detecting the maturity based on the local features, global features and fruit region features of the multispectral image to obtain the maturity level of the fruit to be graded includes:

[0030] Extract the local features and global features of the multispectral image respectively;

[0031] Perform semantic segmentation on the multispectral image, and extract features based on the fruit region in the multispectral image obtained by semantic segmentation to obtain fruit region features;

[0032] Fuse the local features, the global features and the fruit region features, and perform maturity detection based on the fused features to obtain the maturity level of the fruit to be graded.

[0033] The present invention also provides a fruit grading device, including:

[0034] An image acquisition unit for acquiring a fruit image and a multispectral image of the fruit to be graded;

[0035] An index detection unit, configured to perform maturity detection based on local features, global features, and fruit region features of the multi-spectral image to obtain the maturity level of the fruit to be graded; and perform damage detection based on image features of the fruit image and feature maps of each channel in the multi-spectral image to obtain the damage level of the fruit to be graded.

[0036] A fruit grading unit, configured to perform fruit grading based on the volume and weight of the fruit to be graded, as well as the maturity level and / or the damage level, to obtain the fruit grade of the fruit to be graded.

[0037] The present invention also provides an electronic device, including a memory, a processor, and a computer program stored on the memory and executable on the processor. When the processor executes the program, the fruit grading method described in any one of the above is implemented.

[0038] The present invention also provides a sorting system, including: an image acquisition device, a conveyor belt, a processor, and a robotic arm;

[0039] The image acquisition device is configured to acquire a fruit image, a multi-spectral image, and point cloud images from different perspectives of the fruit to be graded on the conveyor belt, and transmit the fruit image, the multi-spectral image, and the point cloud images from different perspectives to the processor;

[0040] The processor is configured to determine the fruit grade of the fruit to be graded based on the fruit image, the multi-spectral image, and the fused point cloud image, and transmit the fruit grade to the robotic arm; the fused point cloud image is determined based on the point cloud images from different perspectives;

[0041] The robotic arm is configured to sort the fruit to be graded based on the fruit grade.

[0042] The present invention also provides a non-transitory computer-readable storage medium, on which a computer program is stored. When the computer program is executed by a processor, the fruit grading method described in any one of the above is implemented.

[0043] The fruit grading method, device, electronic device, sorting system and storage medium provided by the present invention start from the size specifications and appearance quality of fruits, determine the maturity level and damage level of the fruits to be graded through fruit images and multispectral images, and combine the volume and weight of the fruits to be graded, as well as the maturity level and / or damage level, to conduct comprehensive fruit grading to obtain fruit grades. Through precise calculation of indicators at different levels, the refinement and comprehensiveness of the fruit grading process are realized, as well as the maximization of fruit economic benefits. It overcomes the defects that the fruit grading in the traditional solution is too limited and rough, resulting in low accuracy and credibility of the obtained fruit grades. Based on the corresponding fruit grading standards, automated fruit grading is carried out with the help of comprehensive information, which not only reduces the manpower requirement, lowers the fruit grading cost, improves the fruit grading efficiency, but also realizes the double improvement of the accuracy and credibility of the fruit grades. BRIEF DESCRIPTION OF THE DRAWINGS

[0044] In order to more clearly illustrate the technical solutions in the present invention or the prior art, the following will briefly introduce the drawings required for use in the description of the embodiments or the prior art. Obviously, the following drawings are some embodiments of the present invention. For those of ordinary skill in the art, without creative efforts, other drawings can also be obtained based on these drawings.

[0045] Figure 1 is a flowchart of the fruit grading method provided by the present invention;

[0046] Figure 2 is a schematic diagram of the determination process of volume and weight provided by the present invention;

[0047] Figure 3 is a framework diagram of the determination process of volume and weight provided by the present invention;

[0048] Figure 4 is a flowchart of step 140 in the fruit grading method provided by the present invention;

[0049] Figure 5 is a flowchart of step 130 in the fruit grading method provided by the present invention;

[0050] Figure 6 is a flowchart of step 131 in the fruit grading method provided by the present invention;

[0051] Figure 7 is a framework schematic diagram of step 131 in the fruit grading method provided by the present invention;

[0052] Figure 8 is a flowchart of step 132 in the fruit grading method provided by the present invention;

[0053] Figure 9 It is a schematic framework diagram of step 132 in the fruit grading method provided by the present invention;

[0054] Figure 10 It is a schematic flowchart of step 120 in the fruit grading method provided by the present invention;

[0055] Figure 11 It is a schematic framework diagram of step 120 in the fruit grading method provided by the present invention;

[0056] Figure 12 It is the overall framework diagram of the fruit grading method provided by the present invention;

[0057] Figure 13 It is a schematic structural diagram of the fruit grading device provided by the present invention;

[0058] Figure 14 It is a schematic structural diagram of the electronic device provided by the present invention;

[0059] Figure 15 It is a schematic structural diagram of the sorting system provided by the present invention. Detailed implementation manners

[0060] To make the objectives, technical solutions and advantages of the present invention clearer, the technical solutions in the present invention will be clearly and completely described below with reference to the accompanying drawings in the present invention. Obviously, the described embodiments are some but not all of the embodiments of the present invention. All other embodiments obtained by those of ordinary skill in the art based on the embodiments in the present invention without creative efforts shall fall within the protection scope of the present invention.

[0061] Currently, there are some differences in the fruit grading standards in different regions. However, generally, the fruit grading standards in different regions measure the quality of fruits from the following two aspects:

[0062] One is the size specification of fruits, that is, the fruit diameter, volume, weight, etc. of fruits. According to the different sizes of fruits, the quality of fruits can be divided into 3L, 2L, L, M, S, etc. However, it should be noted that there may be differences in the fruit grading standards for different types of fruits.

[0063] The other is the external quality of fruits, which includes maturity, integrity, excellent degree, etc., that is, the ripening situation of fruits, the integrity of the fruit surface, and the excellent degree of color and luster, etc.

[0064] Most current fruit grading methods rely on the volume or weight of the fruit, that is, directly measuring the volume or weight of the fruit, and then classifying the fruit grades according to the volume or weight. In short, it is directly grading according to the fruit size on the sorting machine. However, the fruit size is only one of the indicators in the fruit grading standard. Using only it as the basis for fruit grading will not only make the fruit grading scheme too thin and one-sided, with extremely strong limitations, but also lead to low accuracy and poor credibility of the fruit grades obtained by grading.

[0065] Furthermore, in the fine grading of fruits, currently it mainly relies on the workers on the sorting conveyor belt or the staff in the fruit store. This way of relying on manual labor for fruit grading is not only costly but also inefficient and has poor practicability.

[0066] In summary, although there are relevant standards for fruit grading currently, but specifically in implementation, due to various problems (such as site problems, equipment problems, cost problems, etc.), in order to save costs and improve efficiency, the grading process is often simplified. In this way, the fruit grading process becomes like a formality, with little meaning, and the fruit grading effect is also extremely poor.

[0067] In response to this, the present invention provides a fruit grading method, aiming to use computer vision technology, starting from the size specifications and external quality of the fruit, and through indicators such as the volume and weight of the fruit, the maturity level, and the damage level, to conduct comprehensive evaluation to obtain the fruit grades, realizing the refinement and comprehensiveness of the fruit grading process. By grading fruits with comprehensive information, it can greatly improve the accuracy and credibility of the fruit grades. At the same time, the automated process reduces the manpower requirement, lowers the fruit grading cost, and improves the fruit grading efficiency. Figure 1 It is a schematic flow diagram of the fruit grading method provided by the present invention. As Figure 1 shown, this method includes:

[0068] Step 110, obtaining a fruit image and a multispectral image of the fruit to be graded;

[0069] Specifically, before fruit grading, it is first necessary to determine the fruit to be graded and obtain an image of the fruit to be graded. Given that the traditional fruit grading method based on a single indicator in the traditional scheme will lead to an extremely rough grading process and the low accuracy and poor credibility of the fruit grades obtained by grading, in the embodiments of the present invention, evaluations are made from multiple levels to obtain refined fruit grades, thereby realizing the dual improvement of the credibility and accuracy of the fruit grades.

[0070] Therefore, when obtaining the image of the fruit to be graded here, an image dataset can be established, which includes images for detecting the maturity and damage conditions in the external quality of the fruit to be graded. Specifically, it can be the fruit image and the multispectral image of the fruit to be graded.

[0071] Among them, the fruit image is the RGB image of the fruit to be graded, which can be used to detect the damage condition of the fruit to be graded; since the hyperspectral image contains the spectra of multiple bands to which the fruit to be graded is more sensitive, the maturity detection and damage detection can be performed based on it to obtain the maturity condition and damage condition of the fruit to be graded.

[0072] Step 120: Based on the local features, global features, and fruit region features of the hyperspectral image, perform maturity detection to obtain the maturity level of the fruit to be graded.

[0073] Specifically, in Step 110, after obtaining the hyperspectral image of the fruit to be graded, the hyperspectral image can be used for maturity detection to obtain the maturity level of the fruit to be graded. The specific process includes the following steps:

[0074] After determining the type of the fruit to be graded, spectral query can be performed according to the type of the fruit to know which part of the spectral band the fruit to be graded is more sensitive to, so as to obtain the sensitive spectral band of the fruit to be graded. After selecting the sensitive band, the image of the fruit to be graded can be collected by a hyperspectral camera to obtain a hyperspectral image; and since different maturities correspond to different presentation forms on the hyperspectral image, the hyperspectral image can be used for maturity detection to obtain the maturity level of the fruit to be graded.

[0075] In view of this, for the hyperspectral image of the fruit to be graded, a hybrid detection algorithm can be adopted in the embodiments of the present invention to improve the accuracy of maturity detection. Specifically, it can be performed in three branches. One branch is used to extract CNN (Convolutional Neural Network) features locally, that is, the local features of the hyperspectral image. Another branch is used to extract the features of the image globally, that is, the global features of the hyperspectral image. The middle branch is a semantic segmentation branch, which is used to perform semantic segmentation on the hyperspectral image to extract the fruit region of the fruit to be graded and extract the CNN features of the fruit region, that is, the fruit region features.

[0076] After that, based on these local features, global features, and fruit region features, the maturity detection of the fruit to be graded can be performed to determine the maturity level of the fruit to be graded. Specifically, the three can be fused to use the fused features for maturity detection, so as to obtain the maturity level of the fruit to be graded. Here, the maturity of the fruit to be graded can be divided into several levels in advance. For example, 4, 5, 6, etc. The detected maturity level corresponds to one of these levels.

[0077] It should be noted that, for the case of the integration of the above three, local features can make up for the missing detailed information in the global features, while global features can supplement the global semantic information lost in the extraction process of local features. The fruit region features can simultaneously supplement the fruit information of the fruit to be graded that is missing in both of the above, and the mutual supplementation of the three can make the resulting features more completely reflect the maturity information of the fruit to be graded, so as to be able to perform maturity detection more accurately to ensure the accuracy of the maturity level.

[0078] Here, the integration method of the three can be splicing, addition, weighted integration, etc., and the embodiments of the present invention do not make specific limitations on this.

[0079] Step 130, based on the image features of the fruit image and the feature maps of each channel in the multispectral image, perform damage detection to obtain the damage level of the fruit to be graded;

[0080] Specifically, after obtaining the fruit image and the multispectral image of the fruit to be graded through step 110, the fruit image and the multispectral image can be used for damage detection to obtain the damage level of the fruit to be graded. The specific process includes the following steps:

[0081] Since the damage of the fruit is divided into epidermal damage and subcutaneous damage, and the visible-light fruit image cannot directly detect the damage hidden under the epidermis, therefore, in the embodiments of the present invention, the fruit image and the multispectral image are simultaneously used to perform damage detection on the fruit to be graded to know the damage conditions of its epidermis and subcutaneous part, so as to clarify its damage level.

[0082] In view of this, the fruit image of the fruit to be graded can be used to detect its epidermal damage. Specifically, first, the fruit image is input into the encoding network (Encoder) to extract the image features of the fruit image, and then these image features can be used to perform epidermal damage detection to determine its epidermal damage situation, that is, based on the image features of the fruit image, the epidermal damage detection is performed on the fruit to be graded, so as to obtain its subcutaneous damage situation.

[0083] At the same time, the multispectral image of the fruit to be graded can be used to detect its subcutaneous damage. Since the multispectral image contains multiple channels and each channel corresponds to a waveband, the feature maps of each channel in the multispectral image can be used to detect the subcutaneous damage of the fruit to be graded to determine its subcutaneous damage situation. Specifically, first, the multispectral image is encoded and decoded to obtain its multispectral feature map, and then this multispectral feature map can be channel-separated to obtain the feature maps of each channel. After that, the subcutaneous damage detection can be performed based on the feature maps of each channel, so as to obtain the subcutaneous damage situation of the fruit to be graded.

[0084] After that, the damage level of the fruit to be graded can be determined by combining the epidermal damage condition and the subcutaneous damage condition. Specifically, based on the epidermal damage condition, the subcutaneous damage condition can be combined to jointly evaluate the damage condition of the fruit to be graded, and its damage level can be determined accordingly.

[0085] Step 140: Based on the volume and weight of the fruit to be graded, as well as the maturity level and / or damage level, perform fruit grading to obtain the fruit grade of the fruit to be graded.

[0086] Specifically, after obtaining the maturity level and damage level of the fruit to be graded, the volume and weight of the fruit to be graded, as well as the maturity level and / or damage level, can be combined to perform fruit grading to determine the fruit grade of the fruit to be graded. The specific process includes:

[0087] First, a fused point cloud image of the fruit to be graded needs to be obtained. The fused point cloud image here is determined based on the point cloud images from different perspectives. That is, millimeter-wave radar devices or industrial-grade 3D (3-Dimension) cameras deployed on both sides of the conveyor belt can be used to collect images of the fruit to be graded on the conveyor belt to obtain point cloud images of the fruit to be graded from two different perspectives, and then these two point cloud images from different perspectives are registered and fused to obtain the fused point cloud image of the fruit to be graded.

[0088] Here, the process of registration and fusion can be completed with the help of a 3D point cloud registration model. Specifically, the two point cloud images from different perspectives are input into the 3D point cloud registration model, and the 3D point cloud registration model performs registration and fusion on the input point cloud images from different perspectives, and finally obtains the fused point cloud image output by the 3D point cloud registration model. Before performing registration and fusion with the help of the 3D point cloud registration model, a 3D point cloud registration model can also be pre-trained using sample point cloud images from different perspectives and the fruit position labels in the sample point cloud images.

[0089] Subsequently, the volume and weight of the fruit to be graded can be determined using this fused point cloud image. Specifically, for the fused point cloud image, first, it can be aligned to perform voxel grid division to obtain the point cloud image after voxel division. Then, voxel features (Volume Pixel features) and Points features (Points features, sampling point features) in the point cloud can be extracted from this point cloud image after voxel division. Here, Points are multiple sampling points obtained by downsampling the fused point cloud image. The more sampling points, the stronger the expression ability of the sampling point features, and vice versa, the weaker the feature expression ability. It should be noted that the fused feature of the voxel feature and the sampling point feature here can be regarded as the image feature of the fused point cloud image.

[0090] Meanwhile, for the fused point cloud image, convolution functions of different scales can be utilized to obtain feature maps of different scales. Then, the feature maps of different scales can be fused and upsampled, and finally, BEV features (Bird's Eye View features) of different scales can be obtained.

[0091] After that, the voxel features, sampling point features, and BEV features of different scales of the fused point cloud image can be used for weight estimation and volume estimation to determine the weight and volume of the fruit to be graded. That is, the voxel features, sampling point features, and BEV features of different scales can be regarded as overall features, and they are used for prediction at the volume level and weight level to obtain the volume and weight of the fruit to be graded through regression estimation.

[0092] Subsequently, from all the fruit grading standards in the corresponding area, the fruit grading standard corresponding to the type of the fruit to be graded is determined. That is, with the type of the fruit to be graded as the index, the fruit grading standard corresponding to this type is searched and determined from all the fruit grading standards in the corresponding area.

[0093] After that, based on the volume and weight of the fruit to be graded, as well as the maturity level and / or damage level, the fruit grading standard corresponding to the type of the fruit to be graded is compared to determine its fruit grade. That is, the various indicators of the fruit to be graded can be directly compared with the corresponding indicators in the corresponding fruit grading standard, and the fruit grade is determined by combining the comparison results of various indicators.

[0094] Specifically, it can be to compare the volume and weight, as well as the maturity level, of the fruit to be graded with the volume grading standard, weight grading standard, and maturity grading standard in the corresponding fruit grading standard, and determine the fruit grade of the fruit to be graded based on this comparison result; it can also be to compare the volume and weight, as well as the damage level, of the fruit to be graded with the volume grading standard, weight grading standard, and damage grading standard in the corresponding fruit grading standard, and determine the fruit grade of the fruit to be graded based on this comparison result; it can also be to compare the volume and weight, maturity level, and damage level of the graded fruit with the volume grading standard, weight grading standard, maturity grading standard, and damage grading standard in the corresponding fruit grading standard, and comprehensively evaluate the fruit grade of the fruit to be graded based on this comparison result.

[0095] The fruit grading method provided by the present invention starts from the size specifications and appearance quality of fruits. Through fruit images and multispectral images, the maturity level and damage level of the fruits to be graded are determined. Combining the volume and weight of the fruits to be graded, as well as the maturity level and / or damage level, a comprehensive fruit grade division is carried out to obtain the fruit grade. Through the accurate calculation of indicators at different levels, the refinement and comprehensiveness of the fruit grading process are achieved, as well as the maximization of the economic benefits of fruits. It overcomes the defects that the fruit grading in the traditional scheme is too limited and rough, resulting in low accuracy and credibility of the fruit grades obtained by grading. Based on the corresponding fruit grading standards, automated fruit grading is carried out with the help of comprehensive information, which not only reduces the manpower requirement, lowers the fruit grading cost, improves the fruit grading efficiency, but also realizes the double improvement of the accuracy and credibility of the fruit grades.

[0096] Based on the above embodiments, Figure 2 is a schematic diagram of the determination process of the volume and weight provided by the present invention, as Figure 2 shown, the steps for determining the volume and weight of the fruits to be graded are as follows:

[0097] Step 210, downsample the fused point cloud image of the fruits to be graded, and extract features based on multiple sampling points obtained by downsampling to obtain sampling point features;

[0098] Step 220, perform voxel division on the fused point cloud image, and respectively perform voxel feature encoding and sparse feature extraction on the point cloud image after voxel division to obtain voxel features and depth voxel features, and project the depth voxel features to obtain bird's-eye view features at different scales;

[0099] Step 230, based on the voxel features, sampling point features, and bird's-eye view features at different scales, perform segmentation detection, volume estimation, and weight estimation respectively to obtain a segmented point cloud image, as well as the volume and weight of the fruits to be graded.

[0100] Specifically, Figure 3 is a framework diagram of the determination process of the volume and weight provided by the present invention. Referring to Figure 3 it can be seen that the determination process of the volume and weight of the fruits to be graded includes the following steps:

[0101] Step 210. First, the fused point cloud image of the fruit to be graded can be obtained, and the fused point cloud image is downsampled to obtain multiple sampling points, and feature extraction can be performed on these multiple sampling points to obtain sampling point features. Specifically, the point cloud of the fused point cloud image is downsampled to obtain different sampling points, then a query radius is set for the sampling points, and the set sampling points are input into a two-layer 3D convolutional neural network for feature extraction. The features obtained at this time are then passed through a Multilayer Perceptron (MLP) and a Max Pooling layer, and finally sampling point features are obtained.

[0102] Among them, the query radius can directly affect the fine-grainedness of the sampling point features, and the features based on the sampling points (sampling point features) show the fine-grainedness degree of the fused point cloud image. Moreover, the more sampling points there are, the stronger the expression ability of the sampling point features and the larger the corresponding memory; on the contrary, the fewer sampling points there are, the weaker the expression ability of the sampling point features and the smaller the corresponding memory.

[0103] Step 220. At the same time, the fused point cloud image can be voxelized to obtain the point cloud image after voxelization, and voxel feature encoding can be performed on this point cloud image to obtain voxel features. Specifically, if the dimension size of the point cloud is (D, H, W) and the size of each voxel is (V d , V h , V w ), then the point cloud is divided in three dimensions according to a certain size. After division, the dimension of the point cloud is (D / V d , D / V h , D / V w ). In the embodiments of the present invention, the fused point cloud image is divided according to the voxels to which the point cloud belongs. The number of points in each voxel is different, and then the points in each voxel are uniformly sampled to obtain multiple sampling points (T sampling points). These T sampling points represent the corresponding voxels. These T sampling points are input into multiple consecutive Voxel Feature Encoding (VFE) layers for voxel encoding. Subsequently, the features output by the voxel encoding layer are input into the MLP and Max Pooling layers, and finally voxel features can be obtained;

[0104] Here, the voxel features of the fused point cloud image reflect the correlation relationship between voxels in three-dimensional space, and it retains the feature expression ability along the Z axis; and for empty voxels, only the position information, that is, the coordinate position of the voxel, is retained here.

[0105] Furthermore, since the bird's-eye view feature can be regarded as the projection of a three-dimensional object onto a two-dimensional space, it has a large receptive field and can capture rich context information, which is beneficial for the detection of three-dimensional objects. Therefore, after obtaining the point cloud image after voxel division, sparse feature extraction can be performed on it to obtain depth voxel features, and these depth voxel features can be projected to obtain bird's-eye view features at different scales. Specifically, first, the point cloud image after voxel division is input into a sparse three-dimensional convolutional neural network to obtain the depth voxel features output by the network. Then, taking these depth voxel features as two-dimensional features, through two-dimensional convolution and upsampling, bird's-eye view features at different scales can be obtained.

[0106] Step 230, the voxel features, sampled point features, and bird's-eye view features at different scales can be fused to obtain the fused features of each sampled point. Specifically, the voxel feature is F v , the sampled point feature is F p , and the bird's-eye view feature is F b . Considering that the attributes corresponding to different features are different, here, taking the sampled point feature as the benchmark, the number of sampled points is M, the number of voxels is N, M > N. The sampled points are assigned to their respective voxels, and N is expanded to make N the same size as M. Then, the product of F p and the transpose of F v is calculated to obtain an M * M matrix. Softmax is performed along the last dimension of the matrix to normalize it between 0 and 1, obtaining the similarity between the feature (F ip ) of each sampled point and different voxel features;

[0107] Next, the similarities are sorted in descending order, that is, sorted in the order from high to low similarity. The voxel features corresponding to the top pre-set number (k) of similarities are selected from this similarity sequence, and a weighted average is performed based on these k voxel features to obtain a new voxel feature F iv . At the same time, the corresponding bird's-eye view feature F ib can be determined based on the position of the sampled point. Then, these three are fused (Concat) to obtain the fused feature F i = [F ip , F iv , F ib ; This fused feature is based on the feature of one sampled point, and by combining the fused features of all sampled points, the fused feature F can be obtained.

[0108] After that, based on this fused feature, segmentation detection, volume estimation, and weight estimation can be performed to obtain the segmented point cloud image, as well as the volume and weight of the fruit to be graded. Specifically, using the fused feature F iTaking [ ] as a reference, three branches are used to perform regression estimation on the 3D boxes, confidence scores, volumes, and weights of each sampling point. In other words, segmentation detection, volume estimation, and weight estimation are respectively performed through three MLP branches, so as to obtain the segmented point cloud image, as well as the volume and weight of the fruit to be graded.

[0109] Here, for the segmentation detection branch, instead of directly regressing the center point, the position offset from the current sampling point to the object center point is predicted, and the smooth L1 loss is used to constrain the per-point bounding box regression. For the confidence score, the IOU (Intersection over Union) between the detected 3D boxes and the true 3D boxes is used as the corresponding score, and its loss function also uses the smooth L1 loss.

[0110] For the volume estimation branch and the weight estimation branch, the corresponding volume and weight are directly regressed, and the MAE (Mean Absolute Error) loss is respectively used for constraint. In addition, the losses of the two branches can also be used for joint constraint, that is, according to the density calculation formula, the density is obtained from the weight and volume and used as an additional constraint. For the loss function of the density, the MSE loss is also used. In short, a density loss function is added to constrain the volume and weight.

[0111] Based on the above embodiments, Figure 4 is a schematic flowchart of step 140 in the fruit grading method provided by the present invention, as Figure 4 shown, step 140 includes:

[0112] Step 141, based on the segmented point cloud image, perform fruit shape fitting to obtain the maximum fruit shape contour of the fruit to be graded on the two-dimensional plane, as well as the three-dimensional fruit shape contour;

[0113] Step 142, based on the maximum fruit shape contour and the three-dimensional fruit shape contour, determine the fruit shape grade of the fruit to be graded;

[0114] Step 143, based on the volume and weight, fruit shape grade, maturity grade, and / or damage grade of the fruit to be graded, perform fruit grade division to obtain the fruit grade of the fruit to be graded.

[0115] Specifically, in step 140, the process of performing fruit grade division based on the volume and weight, maturity grade, and / or damage grade of the fruit to be graded to obtain the fruit grade may specifically include the following steps:

[0116] Step 141: First, the segmentation point cloud image obtained through segmentation detection can be utilized to perform fruit shape fitting to determine the maximum fruit shape contour of the fruit to be graded on a two-dimensional plane and its three-dimensional fruit shape contour. Specifically, the segmentation point cloud image can be projected to determine its maximum fruit shape contour on the two-dimensional plane. In short, the maximum fruit shape contour of the fruit to be graded can be directly fitted through its projection on the two-dimensional plane. At the same time, based on the segmentation point cloud image, the three-dimensional shape of the fruit to be graded, that is, the three-dimensional fruit shape contour, can also be fitted.

[0117] Step 142: Subsequently, the fruit shape grade of the fruit to be graded can be determined by combining the maximum fruit shape contour and the three-dimensional fruit shape contour of the fruit to be graded. Specifically, by comparing the maximum fruit shape contour of the fruit to be graded on the two-dimensional plane and the three-dimensional fruit shape contour, a comprehensive evaluation can be carried out to determine the quality level of the fruit to be graded in terms of fruit shape, that is, its fruit shape grade.

[0118] Step 143: Thereafter, based on the various indexes of the fruit to be graded determined above, combined with its fruit shape grade, fruit grading can be carried out to determine its fruit grade. That is, according to the volume and weight, fruit shape grade, and maturity grade and / or damage grade, the fruit to be graded is graded to obtain its fruit grade. Specifically, the above indexes of the fruit to be graded are compared with the corresponding indexes in the corresponding fruit grading standard, and combined with the comparison results of various indexes, the fruit grade of the fruit to be graded is determined.

[0119] Specifically, it can be to compare the volume and weight, fruit shape grade, and maturity grade of the fruit to be graded with the volume grading standard, weight grading standard, fruit shape grading standard, and maturity grading standard in the corresponding fruit grading standard, and determine the fruit grade of the fruit to be graded according to the comparison results; it can also be to compare the volume and weight, fruit shape grade, and damage grade of the fruit to be graded with the volume grading standard, weight grading standard, fruit shape grading standard, and damage grading standard in the corresponding fruit grading standard, and determine the fruit grade of the fruit to be graded according to the comparison results; it can also be to compare the volume and weight, fruit shape grade, maturity grade, and damage grade of the graded fruit with the volume grading standard, weight grading standard, fruit shape grading standard, maturity grading standard, and damage grading standard in the corresponding fruit grading standard, and comprehensively evaluate the fruit grade of the fruit to be graded according to the comparison results.

[0120] Based on the above embodiments, Figure 5 is a schematic flow chart of step 130 in the fruit grading method provided by the present invention, as Figure 5 shown, step 130 includes:

[0121] Step 131: Based on the image features of the fruit image, perform epidermal damage detection to obtain the epidermal damage condition of the fruit to be graded.

[0122] Step 132: Based on the feature maps of each channel in the multispectral image, perform subcutaneous damage detection to obtain the subcutaneous damage condition of the fruit to be graded.

[0123] Step 133: Determine the damage level of the fruit to be graded based on the epidermal damage condition and the subcutaneous damage condition.

[0124] Specifically, since fruit damage is divided into two types. One is the epidermal damage that can be directly seen, such as cracked peel, epidermal ulceration, etc. The other is the subcutaneous damage hidden under the epidermis, such as pulp bruising, pulp separation, pulp deterioration caused by collision, extrusion, etc. This kind of damage cannot be directly seen from the epidermis. In view of this, in the embodiments of the present invention, it is necessary to detect these two types of damage. And because it is difficult to detect subcutaneous damage caused by collision, extrusion, etc. through visible light images, multispectral images are used for detection instead.

[0125] Therefore, in step 130, the process of performing damage detection based on the image features of the fruit image and the feature maps of each channel in the multispectral image to obtain the damage level of the fruit to be graded can specifically include the following steps:

[0126] Step 131: First, the image features of the fruit image can be used to detect the epidermal damage condition of the fruit to be graded. Specifically, the fruit image of the fruit to be graded can be input into the encoding network to extract its features, so as to obtain its image features. Then, these image features can be used to perform epidermal damage detection to determine its epidermal damage condition, that is, on the basis of the fruit feature map decoded from this image feature, semantic segmentation can be performed to obtain the segmentation result (the regional image features of each segmentation region). Then, by combining the segmentation result and the image features, the epidermal damage condition of the fruit to be graded can be detected.

[0127] It should be noted that the epidermal damage detection through the segmentation result is actually aimed at the damage classification of different segmentation regions, rather than the damage detection of the entire fruit image, that is, the segmentation result of semantic segmentation is used to assist classification to clarify the epidermal damage condition.

[0128] Step 132: Meanwhile, the feature maps of each channel in the multispectral image can be utilized to detect the subcutaneous damage condition of the fruit to be graded. Since the multispectral image contains multiple channels and each channel corresponds to a band, channel separation can be performed on it to obtain the images of each channel. Specifically, based on the multispectral feature maps obtained by multispectral image encoding and decoding, channel separation is performed according to the correlation between the channels in the multispectral feature maps, thereby obtaining the feature maps of each channel. Then, based on the feature maps of each channel, the subcutaneous damage condition of the fruit to be graded can be detected.

[0129] Step 133: Subsequently, the damage level can be determined based on the epidermal damage condition and the subcutaneous damage condition of the fruit to be graded, that is, by combining the epidermal damage condition and the subcutaneous damage condition, jointly evaluating the damage condition of the fruit to be graded, and correspondingly determining its damage level.

[0130] Based on the above embodiments, Figure 6 is the flow schematic diagram of step 131 in the fruit grading method provided by the present invention. As Figure 6 shown, step 131 includes:

[0131] Step 131-1: Extract features from the fruit image to obtain the image features of the fruit image, and decode the image features to obtain the fruit feature map of the fruit image;

[0132] Step 131-2: Perform channel dimensionality reduction on the fruit feature map to obtain a single-channel feature map, and perform semantic segmentation based on the single-channel feature map to obtain the regional image features of each segmentation region;

[0133] Step 131-3: Fuse the regional image features and the image features, and perform epidermal damage detection based on the fused features and the regional image features to obtain the epidermal damage condition of the fruit to be graded.

[0134] Specifically, in step 131, the process of using the image features of the fruit image to perform epidermal damage detection to obtain the epidermal damage condition of the fruit to be graded includes the following steps: Figure 7 is the framework schematic diagram of step 131 in the fruit grading method provided by the present invention. Referring to Figure 7 it can be seen that the detection process of the epidermal damage condition of the fruit to be graded can specifically include:

[0135] Step 131-1: First, extract features from the fruit image to obtain its image features, and then decode the image features to obtain the fruit feature map of the fruit image. Specifically, the fruit image can be input into the encoding network for feature extraction to obtain its image features, and then a semantic segmentation branch and a classification branch are connected. In the semantic segmentation branch, first decode the image features through the decoding network (Decoder) to obtain a fruit feature map with the same size as the fruit image;

[0136] Step 131-2: Then, reduce the number of channels of the fruit feature map through a 1*1 convolutional layer to reduce it to a single channel, thereby obtaining a single-channel feature map, and then use the single-channel feature map for semantic segmentation to obtain the regional image features of each segmentation region, that is, perform semantic segmentation based on the single-channel feature map, and obtain the segmentation result through the sigmod function, that is, the regional image features of each segmentation region;

[0137] Step 131-3: Subsequently, compress the segmentation result and fuse it with the image features output by the encoding network, and on the basis of the fused features, perform further feature extraction through the convolutional layer. After that, the features output by the convolutional layer can be aggregated with the segmentation result, that is, pass the segmentation result through the Max Pooling layer and the 1*1 convolutional layer, and then multiply the features output at this time with the features output by the convolutional layer to obtain the classification result of the classification branch, that is, perform epidermis damage detection on the fruit to be graded and obtain the epidermis damage situation.

[0138] It should be noted that the classification result here is for different segmentation regions, rather than the entire fruit image. The segmentation result of semantic segmentation is used to assist classification, so as to realize the epidermis damage detection of the fruit to be graded and clarify its epidermis damage situation.

[0139] Based on the above embodiments, Figure 8 is the flow chart of step 132 in the fruit grading method provided by the present invention. As Figure 8 shown, step 132 includes:

[0140] Step 132-1: Extract features from the multispectral image to obtain multispectral image features, and then decode the multispectral image features to obtain the multispectral feature map of the multispectral image;

[0141] Step 132-2: Based on the correlation between the channels in the multispectral feature map, perform channel separation to obtain the feature maps of each channel, and based on each segmentation region in the feature maps of each channel, perform subcutaneous damage detection to obtain the subcutaneous damage situation of the fruit to be graded.

[0142] Specifically, in step 132, the process of using the feature maps of each channel in the multispectral image to perform subcutaneous damage detection to obtain the subcutaneous damage condition of the fruit to be graded includes the following steps: Figure 9 is a schematic framework diagram of step 132 in the fruit grading method provided by the present invention. As Figure 9 shown, the detection process of the subcutaneous damage condition of the fruit to be graded specifically includes:

[0143] Step 132-1: First, feature extraction can be performed on the multispectral image to determine its multispectral image features, and the multispectral image features can be decoded to obtain the multispectral feature maps of the multispectral image. Specifically, the multispectral image features can be obtained by performing feature extraction on the multispectral image through an encoding network, and then the multispectral image features can be decoded through a decoding network to obtain multispectral feature maps that are the same size as the multispectral image;

[0144] Step 132-2: Then, the relationship between different channels can be modeled through SENet to obtain the importance degree of each channel, thereby enhancing the extraction of relevant channel features and weakening the extraction of irrelevant channel features. Then, an attention mechanism is adopted for different channels to obtain the correlation between each channel, and channel separation is performed based on this correlation to obtain the feature maps of each channel. Then, the subcutaneous damage detection can be performed using each segmentation region in the feature maps of each channel to obtain the subcutaneous damage condition of the fruit to be graded, that is, the segmentation regions of the feature maps of each channel are extracted respectively, and the final classification result, that is, the subcutaneous damage condition of the fruit to be graded, is obtained through connected component and score combination.

[0145] Based on the above embodiments, Figure 10 is a schematic flowchart of step 120 in the fruit grading method provided by the present invention. As Figure 10 shown, step 120 includes:

[0146] Step 121: Extract the local features and global features of the multispectral image respectively;

[0147] Step 122: Perform semantic segmentation on the multispectral image, and perform feature extraction based on the fruit region in the multispectral image obtained by semantic segmentation to obtain fruit region features;

[0148] Step 123: Fuse the local features, global features, and fruit region features, and perform maturity detection based on the fused features to obtain the maturity level of the fruit to be graded.

[0149] Specifically, in step 120, the process of using the local features, global features, and fruit region features of the multispectral image to perform maturity detection to obtain the maturity level of the fruit to be graded includes the following steps: Figure 11It is a schematic framework diagram of step 120 in the fruit grading method provided by the present invention. As Figure 11 shown, the process of detecting the maturity of fruits to be graded includes:

[0150] Step 121, first, local features and global features of the multispectral image can be extracted respectively. Specifically, local features of the multispectral image can be extracted through a convolutional neural network to obtain its local features. At the same time, global features of the multispectral image can be extracted by using multiple consecutive Swin Transformer Blocks;

[0151] Step 122, at the same time, semantic segmentation of the multispectral image can be performed, and features of the fruit region in the multispectral image obtained by semantic segmentation can be extracted to obtain fruit region features. Specifically, a semantic segmentation network can be used to perform semantic segmentation on the multispectral image, directly setting to zero each region other than the fruit region in the segmentation result while retaining the fruit region, and then extracting features of the fruit region to obtain fruit region features;

[0152] Step 123, then, the local features, global features, and fruit region features of the multispectral image can be fused, and the maturity of the fruits to be graded can be detected by using the fused features, thereby obtaining the maturity level of the fruits to be graded. Specifically, the above three can be fused, then dimensionality reduction is performed through a fully connected layer, and finally the maturity level of the fruits to be graded is classified and output.

[0153] It should be noted that for the fusion of the above three, local features can make up for the missing detailed information in global features, global features can supplement the global semantic information lost in the extraction process of local features, and fruit region features can simultaneously supplement the fruit information of the fruits to be graded missing in both of the above. The mutual supplementation of the three can enable the resulting features to more completely reflect the maturity information of the fruits to be graded, thereby enabling more accurate maturity detection to ensure the accuracy of the maturity level.

[0154] Here, the fusion method of the three can be splicing, addition, weighted fusion, etc. As a preference, the fusion method is selected as splicing in the embodiments of the present invention.

[0155] Figure 12 It is the overall framework diagram of the fruit grading method provided by the present invention. As Figure 12 shown, the method includes:

[0156] First, obtain the fruit image and multispectral image of the fruits to be graded;

[0157] Subsequently, based on the local features, global features, and fruit region features of the multispectral image, maturity detection is performed to obtain the maturity level of the fruit to be graded. Specifically, the local features and global features of the multispectral image can be extracted respectively; semantic segmentation is performed on the multispectral image, and feature extraction is performed based on the fruit region in the multispectral image obtained by semantic segmentation to obtain fruit region features; the local features, global features, and fruit region features are fused, and maturity detection is performed based on the fused features to obtain the maturity level of the fruit to be graded;

[0158] Meanwhile, based on the image features of the fruit image and the feature maps of each channel in the multispectral image, damage detection is performed to obtain the damage level of the fruit to be graded. Specifically, based on the image features of the fruit image, epidermis damage detection is performed to obtain the epidermis damage situation of the fruit to be graded; based on the feature maps of each channel in the multispectral image, subcutaneous damage detection is performed to obtain the subcutaneous damage situation of the fruit to be graded; based on the epidermis damage situation and the subcutaneous damage situation, the damage level of the fruit to be graded is determined;

[0159] Among them, based on the image features of the fruit image, epidermis damage detection is performed to obtain the epidermis damage situation of the fruit to be graded, including: performing feature extraction on the fruit image to obtain the image features of the fruit image, decoding the image features to obtain the fruit feature map of the fruit image; performing channel dimensionality reduction on the fruit feature map to obtain a single-channel feature map, and performing semantic segmentation based on the single-channel feature map to obtain the regional image features of each segmentation region; fusing each regional image feature and the image features, and performing epidermis damage detection based on the fused features and each regional image feature to obtain the epidermis damage situation of the fruit to be graded.

[0160] Based on the feature maps of each channel in the multispectral image, subcutaneous damage detection is performed to obtain the subcutaneous damage situation of the fruit to be graded, including: performing feature extraction on the multispectral image to obtain multispectral image features, and decoding the multispectral image features to obtain the multispectral feature map of the multispectral image; based on the correlation between each channel in the multispectral feature map, performing channel separation to obtain the feature maps of each channel, and performing subcutaneous damage detection based on each segmentation region in the feature maps of each channel to obtain the subcutaneous damage situation of the fruit to be graded.

[0161] Thereafter, based on the volume and weight of the fruit to be graded, as well as the maturity level and / or damage level, fruit grading is performed to obtain the fruit grade of the fruit to be graded. Specifically, based on the segmented point cloud image, fruit shape fitting is performed to obtain the maximum fruit shape contour of the fruit to be graded on a two-dimensional plane and the three-dimensional fruit shape contour; based on the maximum fruit shape contour and the three-dimensional fruit shape contour, the fruit shape grade of the fruit to be graded is determined; based on the volume and weight of the fruit to be graded, the fruit shape grade, and the maturity level and / or damage level, fruit grading is performed to obtain the fruit grade of the fruit to be graded.

[0162] Among them, the steps for determining the volume and weight of the fruit to be graded include: downsampling the fused point cloud image of the fruit to be graded, extracting features based on multiple sampling points obtained by downsampling to obtain sampling point features; performing voxel division on the fused point cloud image, and respectively performing voxel feature encoding and sparse feature extraction on the point cloud image after voxel division to obtain voxel features and depth voxel features, projecting the depth voxel features to obtain bird's-eye view features at different scales; based on the voxel features, sampling point features, and bird's-eye view features at different scales, performing segmentation detection, volume estimation, and weight estimation respectively to obtain the segmented point cloud image, as well as the volume and weight of the fruit to be graded.

[0163] The method provided in the embodiments of the present invention starts from the size specifications and appearance quality of fruits, determines the maturity level and damage level of the fruits to be graded through fruit images and multi-spectral images, combines the volume and weight of the fruits to be graded, as well as the maturity level and / or damage level, performs comprehensive fruit grading to obtain the fruit grade. Through precise calculation of indicators at different levels, it realizes the refinement and comprehensiveness of the fruit grading process, as well as the maximization of fruit economic benefits, overcomes the defects that fruit grading in traditional solutions is too limited and rough, resulting in low accuracy and credibility of the fruit grades obtained by grading, and performs automated fruit grading with the help of comprehensive information based on the corresponding fruit grading standards, which not only reduces the manpower requirement, lowers the fruit grading cost, improves the fruit grading efficiency, but also realizes the double improvement of the accuracy and credibility of the fruit grades.

[0164] The fruit grading device provided by the present invention will be described below. The fruit grading device described below can be mutually referred to with the fruit grading method described above.

[0165] Figure 13 is a schematic structural diagram of the fruit grading device provided by the present invention, as Figure 13 shown, the device includes:

[0166] An image acquisition unit 1310, configured to acquire a fruit image and a multi-spectral image of the fruit to be graded;

[0167] The index detection unit 1320 is configured to perform maturity detection based on the local features, global features, and fruit region features of the multi-spectral image to obtain the maturity level of the fruit to be graded; and perform damage detection based on the image features of the fruit image and the feature maps of each channel in the multi-spectral image to obtain the damage level of the fruit to be graded.

[0168] The fruit grading unit 1330 is configured to perform fruit grading based on the volume and weight of the fruit to be graded, as well as the maturity level and / or the damage level, to obtain the fruit grade of the fruit to be graded.

[0169] The fruit grading device provided by the present invention starts from the size specifications and appearance quality of fruits, determines the maturity level and damage level of the fruit to be graded through fruit images and multi-spectral images, combines the volume and weight of the fruit to be graded, as well as the maturity level and / or the damage level, to perform comprehensive fruit grading to obtain the fruit grade. Through precise calculation of indicators at different levels, it realizes the refinement and comprehensiveness of the fruit grading process, as well as the maximization of fruit economic benefits, overcomes the defects that fruit grading in traditional solutions is too limited and rough, resulting in low accuracy and credibility of the obtained fruit grades, and performs automated fruit grading based on comprehensive information on the basis of corresponding fruit grading standards, which not only reduces the manpower requirement, lowers the fruit grading cost, improves the fruit grading efficiency, but also realizes the double improvement of the accuracy and credibility of the fruit grade.

[0170] Based on the above embodiments, the index detection unit 1320 is further configured to:

[0171] Downsample the fused point cloud image of the fruit to be graded, and perform feature extraction based on multiple sampling points obtained by the downsampling to obtain sampling point features;

[0172] Perform voxel division on the fused point cloud image, and respectively perform voxel feature encoding and sparse feature extraction on the point cloud image after voxel division to obtain voxel features and depth voxel features, and project the depth voxel features to obtain bird's-eye view features at different scales;

[0173] Based on the voxel features, the sampling point features, and the bird's-eye view features at different scales, perform segmentation detection, volume estimation, and weight estimation respectively to obtain a segmented point cloud image, as well as the volume and weight of the fruit to be graded.

[0174] Based on the above embodiments, the fruit grading unit 1330 is configured to:

[0175] Perform fruit shape fitting based on the segmented point cloud image to obtain the maximum fruit shape contour of the fruit to be graded on a two-dimensional plane and a three-dimensional fruit shape contour;

[0176] Determine the fruit shape grade of the fruit to be graded based on the maximum fruit shape contour and the three-dimensional fruit shape contour;

[0177] Perform fruit grading based on the volume and weight of the fruit to be graded, the fruit shape grade, and the maturity grade and / or the damage grade to obtain the fruit grade of the fruit to be graded.

[0178] Based on the above embodiments, the index detection unit 1320 is used for:

[0179] Perform epidermal damage detection based on the image features of the fruit image to obtain the epidermal damage condition of the fruit to be graded;

[0180] Perform subcutaneous damage detection based on the feature maps of each channel in the multispectral image to obtain the subcutaneous damage condition of the fruit to be graded;

[0181] Determine the damage grade of the fruit to be graded based on the epidermal damage condition and the subcutaneous damage condition.

[0182] Based on the above embodiments, the index detection unit 1320 is used for:

[0183] Extract features from the fruit image to obtain the image features of the fruit image, and decode the image features to obtain the fruit feature map of the fruit image;

[0184] Reduce the channels of the fruit feature map to obtain a single-channel feature map, and perform semantic segmentation based on the single-channel feature map to obtain the regional image features of each segmentation region;

[0185] Fuse the regional image features of each region and the image features, and perform epidermal damage detection based on the fused features and the regional image features of each region to obtain the epidermal damage condition of the fruit to be graded.

[0186] Based on the above embodiments, the index detection unit 1320 is used for:

[0187] Extract features from the multispectral image to obtain multispectral image features, and decode the multispectral image features to obtain the multispectral feature map of the multispectral image;

[0188] Perform channel separation based on the correlation between the channels in the multispectral feature map to obtain the feature maps of each channel, and perform subcutaneous damage detection based on each segmentation region in the feature maps of each channel to obtain the subcutaneous damage condition of the fruit to be graded.

[0189] Based on the above embodiments, the index detection unit 1320 is used for:

[0190] Extract the local features and global features of the multi-spectral image respectively;

[0191] Perform semantic segmentation on the multi-spectral image, and extract features based on the fruit region in the multi-spectral image obtained by semantic segmentation to obtain fruit region features;

[0192] Fuse the local features, the global features and the fruit region features, and perform maturity detection based on the fused features to obtain the maturity level of the fruit to be graded.

[0193] Based on the above embodiments, the present invention further provides a sorting system, Figure 15 is a schematic structural diagram of the sorting system provided by the present invention, as Figure 15 shown, the system includes an image acquisition device 1510, a conveyor belt 1520, a processor 1410 and a robotic arm 1530;

[0194] The image acquisition device 1510 is used to acquire fruit images, multi-spectral images and point cloud images from different perspectives of the fruit to be graded on the conveyor belt 1520, and transmit the fruit images, the multi-spectral images and the point cloud images from different perspectives to the processor 1410;

[0195] The processor 1410 is used to determine the fruit grade of the fruit to be graded based on the fruit image, the multi-spectral image, and the fused point cloud image, and transmit the fruit grade to the robotic arm 1530; the fused point cloud image is determined based on the point cloud images from different perspectives;

[0196] The robotic arm 1530 is used to sort the fruit to be graded based on the fruit grade.

[0197] In the sorting system provided by the present invention, the processor applies the images acquired by the image acquisition device to realize the automatic grading of the fruit to be graded, improves the fruit grading process, realizes the comprehensiveness and refinement of fruit grading, and at the same time improves the accuracy and credibility of the fruit grade; in addition, by connecting the processor and the robotic arm, the robotic arm can sort the fruits according to quality and batches, realizing the automation of the whole process from fruit picking and grading to packaging and selling, and at the same time maximizing the economic benefits of the fruits.

[0198] Figure 14 Illustrates a schematic physical structure diagram of an electronic device, as Figure 14As shown, the electronic device may include: a processor 1410, a communications interface 1420, a memory 1430, and a communication bus 1440. Among them, the processor 1410, the communication interface 1420, and the memory 1430 complete mutual communication through the communication bus 1440. The processor 1410 may call the logical instructions in the memory 1430 to execute a fruit grading method, which includes: obtaining a fruit image and a multispectral image of the fruit to be graded; performing maturity detection based on the local features, global features, and fruit region features of the multispectral image to obtain the maturity level of the fruit to be graded; performing damage detection based on the image features of the fruit image and the feature maps of each channel in the multispectral image to obtain the damage level of the fruit to be graded; and performing fruit grade division based on the volume and weight of the fruit to be graded, as well as the maturity level and / or the damage level, to obtain the fruit grade of the fruit to be graded.

[0199] In addition, when the logical instructions in the above-mentioned memory 1430 are implemented in the form of software functional units and sold or used as independent products, they may be stored in a computer-readable storage medium. Based on such an understanding, the technical solution of the present invention, in essence, or the part that contributes to the prior art, or a part of this technical solution, may be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions for causing a computer device (which may be a personal computer, a server, or a network device, etc.) to execute all or part of the steps of the methods described in various embodiments of the present invention. The foregoing storage medium includes: various media such as USB flash drives, mobile hard disks, read-only memories (ROMs), random access memories (RAMs), magnetic disks, or optical discs that can store program codes.

[0200] On the other hand, the present invention also provides a computer program product, which includes a computer program stored on a non-transitory computer-readable storage medium. The computer program includes program instructions, and when the program instructions are executed by a computer, the computer can execute the fruit grading method provided by each of the above methods. The method includes: obtaining a fruit image and a multispectral image of the fruit to be graded; performing maturity detection based on the local features, global features, and fruit region features of the multispectral image to obtain the maturity level of the fruit to be graded; performing damage detection based on the image features of the fruit image and the feature maps of each channel in the multispectral image to obtain the damage level of the fruit to be graded; and performing fruit grade classification based on the volume and weight of the fruit to be graded, as well as the maturity level and / or the damage level, to obtain the fruit grade of the fruit to be graded.

[0201] In another aspect, the present invention also provides a non-transitory computer-readable storage medium, on which a computer program is stored. When the computer program is executed by a processor, it realizes the fruit grading method provided by each of the above methods. The method includes: obtaining a fruit image and a multispectral image of the fruit to be graded; performing maturity detection based on the local features, global features, and fruit region features of the multispectral image to obtain the maturity level of the fruit to be graded; performing damage detection based on the image features of the fruit image and the feature maps of each channel in the multispectral image to obtain the damage level of the fruit to be graded; and performing fruit grade classification based on the volume and weight of the fruit to be graded, as well as the maturity level and / or the damage level, to obtain the fruit grade of the fruit to be graded.

[0202] The device embodiments described above are merely illustrative. The units described as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed to multiple network units. Some or all of the modules can be selected according to actual needs to achieve the purpose of the solution of this embodiment. Those of ordinary skill in the art can understand and implement it without creative efforts.

[0203] Through the description of the above embodiments, those skilled in the art can clearly understand that each embodiment can be implemented by means of software plus a necessary general hardware platform, and of course, it can also be implemented by hardware. Based on such an understanding, the essence of the above technical solution or the part that contributes to the prior art can be embodied in the form of a software product. This computer software product can be stored in a computer-readable storage medium, such as ROM / RAM, magnetic disk, optical disk, etc., and includes several instructions to enable a computer device (which can be a personal computer, a server, or a network device, etc.) to execute the methods described in each embodiment or some parts of the embodiments.

[0204] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit them; although the present invention has been described in detail with reference to the foregoing embodiments, those of ordinary skill in the art should understand that they can still modify the technical solutions described in the foregoing embodiments, or perform equivalent replacements for some of the technical features; and these modifications or replacements do not make the essence of the corresponding technical solutions deviate from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. A fruit grading method, characterized in that, Including: Obtaining a fruit image and a multispectral image of the fruit to be graded; Performing maturity detection based on local features, global features, and fruit region features of the multispectral image to obtain the maturity level of the fruit to be graded; Performing damage detection based on the image features of the fruit image and the feature maps of each channel in the multispectral image to obtain the damage level of the fruit to be graded; Performing fruit grading based on the volume and weight of the fruit to be graded, as well as the maturity level and / or the damage level, to obtain the fruit grade of the fruit to be graded; The determining steps for the volume and weight of the fruit to be graded include: Downsampling the fused point cloud image of the fruit to be graded, and extracting features based on multiple sampling points obtained by downsampling to obtain sampling point features; Performing voxel division on the fused point cloud image, and respectively performing voxel feature encoding and sparse feature extraction on the point cloud image after voxel division to obtain voxel features and depth voxel features, and projecting the depth voxel features to obtain bird's-eye view features of different scales; Performing segmentation detection, volume estimation, and weight estimation based on the voxel features, the sampling point features, and the bird's-eye view features of different scales to obtain a segmented point cloud image, as well as the volume and weight of the fruit to be graded.

2. The fruit grading method according to claim 1, wherein, The performing fruit grading based on the volume and weight of the fruit to be graded, as well as the maturity level and / or the damage level, to obtain the fruit grade of the fruit to be graded includes: Performing fruit shape fitting based on the segmented point cloud image to obtain the maximum fruit shape contour of the fruit to be graded on a two-dimensional plane and a three-dimensional fruit shape contour; Determining the fruit shape grade of the fruit to be graded based on the maximum fruit shape contour and the three-dimensional fruit shape contour; Performing fruit grading based on the volume and weight of the fruit to be graded, the fruit shape grade, as well as the maturity level and / or the damage level, to obtain the fruit grade of the fruit to be graded.

3. The fruit grading method according to claim 1 or 2, characterized in that, The performing damage detection based on the image features of the fruit image and the feature maps of each channel in the multispectral image to obtain the damage level of the fruit to be graded includes: Performing epidermal damage detection based on the image features of the fruit image to obtain the epidermal damage condition of the fruit to be graded; Performing subcutaneous damage detection based on the feature maps of each channel in the multispectral image to obtain the subcutaneous damage condition of the fruit to be graded; Determining the damage level of the fruit to be graded based on the epidermal damage condition and the subcutaneous damage condition.

4. The fruit grading method according to claim 3, wherein The performing epidermal damage detection based on the image features of the fruit image to obtain the epidermal damage condition of the fruit to be graded includes: Extracting features from the fruit image to obtain the image features of the fruit image, and decoding the image features to obtain the fruit feature map of the fruit image; Reducing the number of channels of the fruit feature map to obtain a single-channel feature map, and performing semantic segmentation based on the single-channel feature map to obtain the regional image features of each segmentation region; Fuse the image features of each region and the said image features, and based on the fused features and the image features of each region, perform epidermal damage detection to obtain the epidermal damage condition of the fruit to be graded.

5. The fruit grading method according to claim 3, characterized in that, The subcutaneous damage detection based on the feature maps of each channel in the multispectral image to obtain the subcutaneous damage condition of the fruit to be graded includes: Extract features from the multispectral image to obtain multispectral image features, and decode the multispectral image features to obtain the multispectral feature map of the multispectral image; Based on the correlation between each channel in the multispectral feature map, perform channel separation to obtain the feature map of each channel, and based on each segmentation region in the feature map of each channel, perform subcutaneous damage detection to obtain the subcutaneous damage condition of the fruit to be graded.

6. The fruit grading method according to claim 1 or 2, characterized in that, The maturity detection based on the local features, global features and fruit region features of the multispectral image to obtain the maturity level of the fruit to be graded includes: Extract the local features and global features of the multispectral image respectively; Perform semantic segmentation on the multispectral image, and based on the fruit region in the multispectral image obtained by semantic segmentation, extract features to obtain fruit region features; Fuse the local features, the global features and the fruit region features, and perform maturity detection based on the fused features to obtain the maturity level of the fruit to be graded.

7. A fruit grading device, characterized in that, Including: An image acquisition unit for acquiring the fruit image and multispectral image of the fruit to be graded; An index detection unit for performing maturity detection based on the local features, global features and fruit region features of the multispectral image to obtain the maturity level of the fruit to be graded; and performing damage detection based on the image features of the fruit image and the feature maps of each channel in the multispectral image to obtain the damage level of the fruit to be graded; A fruit grading unit for performing fruit grading based on the volume and weight of the fruit to be graded, and the maturity level and / or the damage level to obtain the fruit grade of the fruit to be graded; The determination steps of the volume and weight of the fruit to be graded include: Perform downsampling on the fused point cloud image of the fruit to be graded, and extract features based on the multiple sampling points obtained by downsampling to obtain sampling point features; Perform voxel division on the fused point cloud image, and perform voxel feature encoding and sparse feature extraction on the point cloud image after voxel division respectively to obtain voxel features and depth voxel features, and project the depth voxel features to obtain bird's-eye view features of different scales; Based on the voxel features, the sampling point features and the bird's-eye view features of different scales, perform segmentation detection, volume estimation and weight estimation respectively to obtain the segmented point cloud image, and the volume and weight of the fruit to be graded.

8. An electronic device, comprising a memory, a processor, and a computer program stored on the memory and executable on the processor, characterized in that, When the processor executes the program, it implements the fruit grading method according to any one of claims 1 to 6.

9. A sorting system, characterized in that, Including: An image acquisition device, a conveyor belt, a processor and a robotic arm; The image acquisition device is used to acquire fruit images, multispectral images, and point cloud images of the fruit to be graded on the conveyor belt, and transmit the fruit images, the multispectral images, and the point cloud images from different perspectives to the processor; The processor is used to perform maturity detection based on the multispectral image to obtain the maturity level of the fruit to be graded, perform damage detection based on the fruit image and the multispectral image to obtain the damage level of the fruit to be graded, and determine the volume and weight of the fruit to be graded based on the fused point cloud image. Then, based on the volume and weight of the fruit to be graded, as well as the maturity level and / or the damage level, perform fruit grading to obtain the fruit grade of the fruit to be graded, and transmit the fruit grade to the robotic arm; The fused point cloud image is determined based on the point cloud images from different perspectives; Among them, determining the volume and weight of the fruit to be graded based on the fused point cloud image includes: Performing downsampling on the fused point cloud image, and extracting features based on multiple sampling points obtained by the downsampling to obtain sampling point features; Performing voxel division on the fused point cloud image, and respectively performing voxel feature encoding and sparse feature extraction on the point cloud image after voxel division to obtain voxel features and depth voxel features, and projecting the depth voxel features to obtain bird's-eye view features at different scales; Based on the voxel features, the sampling point features, and the bird's-eye view features at different scales, respectively perform volume estimation and weight estimation to obtain the volume and weight of the fruit to be graded; The robotic arm is used to sort the fruit to be graded based on the fruit grade; 10. A non-transitory computer-readable storage medium having a computer program stored thereon, characterized in that, The computer program, when executed by the processor, implements the fruit grading method according to any one of claims 1 to 6.

Images