Fruit and vegetable detection method, device and equipment, picking robot and storage medium

Abstract

The invention relates to the field of artificial intelligence, and provides a fruit and vegetable detection method and device, electronic equipment, a picking robot and a storage medium, and the method comprises the steps: obtaining a fruit and vegetable image of a to-be-detected fruit and vegetable; performing multi-scale feature extraction based on the fruit and vegetable image, and performing semantic fusion on multi-scale fruit and vegetable features obtained by feature extraction to obtain fruit and vegetable fusion features; on the basis of the fruit and vegetable fusion features, maturity detection and defect disease detection are performed on the to-be-detected fruits and vegetables respectively to obtain fruit and vegetable detection results of the to-be-detected fruits and vegetables, so that the defects of high detection difficulty, low accuracy, high manual intervention judgment cost and long period after picking in a traditional fruit and vegetable detection scheme are overcome, efficient and accurate fruit and vegetable detection is realized, and the fruit and vegetable detection efficiency is improved. And data support is provided for automatic and accurate picking of fruits and vegetables, and powerful support is provided for improvement of the intelligent level of the picking robot and intelligent transformation of agricultural production.

Description

Technical Field

[0001] The present invention relates to the field of artificial intelligence technology, and in particular to a fruit and vegetable detection method, device, equipment, picking robot and storage medium. Background Art

[0002] With the rapid development of modern agriculture, the scale of fruit and vegetable cultivation is expanding, especially the large-scale cultivation of common vegetables such as tomatoes has become an important part of the agricultural industry. However, the traditional manual picking method faces challenges such as low efficiency, high cost and difficulty in ensuring picking quality when facing large-scale and high-density fruit and vegetable fields.

[0003] In recent years, artificial intelligence technology, especially deep learning technology, has shown great potential in fruit and vegetable detection, but it still faces many challenges in practical applications. On the one hand, fruits and vegetables mature at different stages and are usually hidden in dense leaves, which increases the difficulty and cost of detection; on the other hand, the maturity judgment standards of fruits and vegetables of different varieties and different growth environments are different, and after detection and picking, manual intervention is usually required to reduce the risk of deterioration during transportation and storage, which invisibly increases costs. Summary of the invention

[0004] The present invention provides a fruit and vegetable detection method, device, equipment, picking robot and storage medium, which are used to solve the defects in the prior art that the maturity difference is subtle, and the detection is difficult and accurate due to susceptibility to obstruction, and the risk needs to be manually judged after picking, which is costly and inefficient. The method realizes efficient and accurate fruit and vegetable detection, and provides strong support for the intelligent transformation of agricultural production.

[0005] The present invention provides a fruit and vegetable detection method, comprising: Acquire images of fruits and vegetables to be detected; Performing multi-scale feature extraction based on the fruit and vegetable images, and performing semantic fusion on the multi-scale fruit and vegetable features obtained by the feature extraction to obtain fruit and vegetable fusion features; Based on the fruit and vegetable fusion features, maturity detection and defect and disease detection are performed on the fruits and vegetables to be detected, respectively, to obtain fruit and vegetable detection results of the fruits and vegetables to be detected.

[0006] According to a fruit and vegetable detection method provided by the present invention, multi-scale feature extraction is performed based on the fruit and vegetable image, and the multi-scale fruit and vegetable features obtained by feature extraction are semantically fused to obtain fruit and vegetable fusion features; based on the fruit and vegetable fusion features, maturity detection and defect and disease detection are performed on the fruits and vegetables to be detected respectively to obtain fruit and vegetable detection results of the fruits and vegetables to be detected, including: Applying the feature extraction network in the fruit and vegetable detection model, performing multi-scale feature extraction based on the fruit and vegetable images, and semantically fusing the multi-scale fruit and vegetable features obtained by the feature extraction to obtain fruit and vegetable fusion features; Applying the defect detection network and maturity detection network in the fruit and vegetable detection model, based on the fruit and vegetable fusion features, respectively performing defect disease detection and maturity detection on the fruits and vegetables to be detected, and obtaining the fruit and vegetable detection results of the fruits and vegetables to be detected; The fruit and vegetable detection model is trained on the basis of a lightweight target detection model using sample fruit and vegetable images and their corresponding maturity labels and defect and disease labels.

[0007] According to a fruit and vegetable detection method provided by the present invention, based on the fruit and vegetable fusion features, defect disease detection and maturity detection are performed on the fruits and vegetables to be detected, respectively, to obtain the fruit and vegetable detection results of the fruits and vegetables to be detected, including: Based on the fruit and vegetable fusion features, the defect and disease detection is performed on the fruit and vegetable to be detected, so as to obtain the defect and disease detection results of the fruit and vegetable to be detected; If the defect and disease detection result indicates that the fruits and vegetables to be detected have defects and diseases, then the maturity detection of the fruits and vegetables to be detected is not performed; Otherwise, performing maturity detection on the fruits and vegetables to be detected based on the fruit and vegetable fusion features to obtain maturity detection results of the fruits and vegetables to be detected; The fruit and vegetable detection result is determined based on the defect and disease detection result, or the defect and disease detection result and the maturity detection result.

[0008] According to a fruit and vegetable detection method provided by the present invention, the fruit and vegetable detection model is determined based on the following steps: Determine a first sample image set, a second sample image set, and an initial detection model; the first sample image set includes first sample fruit and vegetable images and their corresponding maturity labels; the second sample image set includes second sample fruit and vegetable images and their corresponding defect and disease labels; the initial detection model includes an initial feature extraction network, an initial maturity detection network, and an initial defect detection network, and the initial feature extraction network is constructed based on the lightweight target detection model; Based on the initial feature extraction network, extract a first fruit and vegetable fusion feature of the first sample fruit and vegetable image, and based on the first fruit and vegetable fusion feature, obtain a predicted maturity detection result corresponding to the first sample fruit and vegetable image through the initial maturity detection network; Based on the predicted maturity detection result and the maturity label, performing parameter iteration on the initial feature extraction network and the initial maturity detection network in the initial detection model to obtain a feature extraction network and a maturity detection network; Based on the feature extraction network, extract the second fruit and vegetable fusion feature of the second sample fruit and vegetable image, and based on the second fruit and vegetable fusion feature, obtain the predicted defect and disease detection result corresponding to the second sample fruit and vegetable image through the initial defect detection network detection; Based on the predicted defect and disease detection result and the defect and disease label, performing parameter iteration on the initial defect detection network in the initial detection model to obtain a defect detection network; A fruit and vegetable detection model is constructed based on the feature extraction network, the maturity detection network and the defect detection network.

[0009] According to a fruit and vegetable detection method provided by the present invention, the initial feature extraction network is determined based on the following steps: Determine the target detection model; Lightweighting parameters of the neck network of the target detection model to obtain a lightweight target detection model, wherein the target detection model includes a backbone network, a neck network, and a head network; Based on the lightweight target detection model, an initial feature extraction network is constructed.

[0010] According to a fruit and vegetable detection method provided by the present invention, the first sample image set and the second sample image set are determined based on the following steps: Obtaining a first initial fruit and vegetable image and its corresponding maturity label, and a second initial fruit and vegetable image and its corresponding defect and disease label; performing data enhancement processing on the first initial fruit and vegetable image and the second initial fruit and vegetable image respectively to obtain a plurality of first enhanced fruit and vegetable images and a plurality of second enhanced fruit and vegetable images; Determine a first sample fruit and vegetable image based on the first initial fruit and vegetable image and the first enhanced fruit and vegetable image, and determine a second sample fruit and vegetable image based on the second initial fruit and vegetable image and the second enhanced fruit and vegetable image; Based on the first sample fruit and vegetable images and the maturity labels, a first sample image set is constructed; based on the second sample fruit and vegetable images and the defect and disease labels, a second sample image set is constructed; The data enhancement processing includes at least one of rotation transformation, scale transformation, Gaussian noise addition, displacement blur, and Poisson blur.

[0011] The present invention also provides a fruit and vegetable detection device, comprising: An acquisition unit, used for acquiring images of fruits and vegetables to be detected; The detection unit is used to perform multi-scale feature extraction based on the fruit and vegetable image, and semantically fuse the multi-scale fruit and vegetable features obtained by the feature extraction to obtain fruit and vegetable fusion features; based on the fruit and vegetable fusion features, perform maturity detection and defect and disease detection on the fruits and vegetables to be detected, respectively, to obtain fruit and vegetable detection results of the fruits and vegetables to be detected.

[0012] The present invention also provides a picking robot, comprising: an image acquisition device, a processor and a picking mechanical arm; The image acquisition device is used to acquire images of fruits and vegetables to be detected, and transmit the images of fruits and vegetables to the processor; The processor is used to perform multi-scale feature extraction based on the fruit and vegetable image, and semantically fuse the multi-scale fruit and vegetable features obtained by feature extraction to obtain fruit and vegetable fusion features; based on the fruit and vegetable fusion features, respectively perform maturity detection and defect and disease detection on the fruits and vegetables to be detected to obtain fruit and vegetable detection results of the fruits and vegetables to be detected, and transmit the fruit and vegetable detection results to the picking robot arm; The picking robot arm is used to pick or discard the fruits and vegetables to be detected based on the fruit and vegetable detection results.

[0013] The present invention also provides an electronic device, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein when the processor executes the program, any of the above-mentioned fruit and vegetable detection methods is implemented.

[0014] The present invention also provides a non-transitory computer-readable storage medium having a computer program stored thereon, and when the computer program is executed by a processor, the fruit and vegetable detection method as described in any one of the above is implemented.

[0015] The fruit and vegetable detection method, device, equipment, picking robot and storage medium provided by the present invention obtain fruit and vegetable images of fruits and vegetables to be detected; perform multi-scale feature extraction based on the fruit and vegetable images to extract accurate fruit and vegetable features, and perform semantic fusion on the extracted multi-scale fruit and vegetable features to capture semantic information of different levels of the fruits and vegetables to be detected to obtain fruit and vegetable fusion features; based on the fruit and vegetable fusion features, perform maturity detection and defect and disease detection on the fruits and vegetables to be detected, respectively, to obtain fruit and vegetable detection results of the fruits and vegetables to be detected, thereby overcoming the defects of traditional fruit and vegetable detection schemes such as high detection difficulty and low accuracy, and high cost and long cycle of human intervention judgment after picking, thereby achieving efficient and accurate fruit and vegetable detection, providing data support for the automated and accurate picking of fruits and vegetables, and providing strong support for improving the intelligence level of picking robots and the intelligent transformation of agricultural production. BRIEF DESCRIPTION OF THE DRAWINGS

[0016] In order to more clearly illustrate the technical solutions in the present invention or the prior art, a brief introduction will be given below to the drawings required for use in the embodiments or the description of the prior art. Obviously, the drawings described below are some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative work.

[0017] Figure 1 It is a schematic diagram of the process of the fruit and vegetable detection method provided by the present invention; Figure 2 It is a structural schematic diagram of the fruit and vegetable detection model provided by the present invention; Figure 3 It is a structural schematic diagram of a lightweight target detection model provided by the present invention; Figure 4 It is a structural schematic diagram of the fruit and vegetable detection device provided by the present invention; Figure 5 It is a structural schematic diagram of the picking robot provided by the present invention; Figure 6 It is a structural schematic diagram of the electronic device provided by the present invention. DETAILED DESCRIPTION

[0018] In order to make the purpose, technical solution and advantages of the present invention clearer, the technical solution of the present invention will be clearly and completely described below in conjunction with the drawings of the present invention. Obviously, the described embodiments are part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0019] At present, although artificial intelligence technology, especially deep learning technology, has shown great potential in fruit and vegetable detection, it still faces many challenges in practical applications. On the one hand, fruits and vegetables, especially tomatoes, have subtle differences in adjacent maturity characteristics and are easily blocked by leaves or other fruits, making it difficult for traditional image processing methods to accurately judge the maturity of the fruit. On the other hand, there are differences in the maturity judgment standards of tomatoes of different varieties and different growth environments, which further increases the difficulty of detection. In addition, after using artificial intelligence technology for detection and picking, manual intervention is often required to make judgments in order to eliminate defective and diseased fruits and reduce the risk of deterioration during transportation and storage, which invisibly increases labor costs and prolongs the "output" time of orchards and vegetable fields.

[0020] In this regard, the present invention provides a fruit and vegetable detection method, which aims to extract accurate fruit and vegetable features under a complex planting environment according to storage and transportation requirements, and fuse the extracted fruit and vegetable features to capture semantic information of different levels of fruits and vegetables to be detected, thereby obtaining semantically accurate and complete fruit and vegetable fusion features, and maturity detection and defect and disease detection can be performed based on the fruit and vegetable fusion features to obtain fruit and vegetable detection results of the fruits and vegetables to be detected, thereby achieving efficient and accurate fruit and vegetable detection, providing data support for the automated picking of fruits and vegetables, avoiding the problems of high detection difficulty, low accuracy, high cost and long time for manual judgment after detection, and providing strong support for improving the intelligence level of picking robots and the intelligent transformation of agricultural production.

[0021] Figure 1 It is a schematic diagram of the process of the fruit and vegetable detection method provided by the present invention, such as Figure 1 As shown, the execution subject of the method can be a picking robot or a processor built into the picking robot, and the embodiment of the present invention does not specifically limit this. The method includes: Step 110, obtaining images of fruits and vegetables to be detected; Step 120, performing multi-scale feature extraction based on the fruit and vegetable images, and semantically fusing the multi-scale fruit and vegetable features obtained by the feature extraction to obtain fruit and vegetable fusion features; based on the fruit and vegetable fusion features, performing maturity detection and defect and disease detection on the fruits and vegetables to be detected, respectively, to obtain fruit and vegetable detection results for the fruits and vegetables to be detected.

[0022] Specifically, considering that the current fruit and vegetable detection is difficult and has low accuracy due to the subtle differences in adjacent maturity characteristics and easy obstruction, as well as the differences in judgment standards for different varieties and different growth environments, and that defects and diseases still need to be manually judged after picking, which is costly and time-consuming, an efficient, accurate and adaptable fruit and vegetable detection method is proposed in an embodiment of the present invention. The method has extremely high accuracy and timeliness in the detection of common fruits and vegetables such as tomatoes, and provides benefits for improving the intelligence level of picking robots in agricultural production and promoting the process of agricultural modernization.

[0023] It is understandable that in the actual agricultural production process, before testing and picking fruits and vegetables, the first thing to be determined is the fruits and vegetables to be tested, that is, the fruits and vegetables to be tested. The fruits and vegetables to be tested here can be any type of fruits, vegetables, etc., for example, tomatoes, apples, pomegranates, peppers, eggplants, etc., and the embodiments of the present invention do not specifically limit this.

[0024] After the fruits and vegetables to be detected are determined, their images can be obtained to perform fruit and vegetable detection based on them. The images can be called fruit and vegetable images, that is, fruit and vegetable images of the fruits and vegetables to be detected are obtained. The fruit and vegetable images here can be obtained by using close-range image acquisition equipment (high-definition cameras, cameras, etc.) installed in orchards and vegetable fields, or they can be directly collected by using picking robots deployed in orchards, vegetable fields, etc. during the fruit and vegetable ripening season. The picking robots here are provided with image acquisition modules, which can directly collect images of fruits and vegetables planted in orchards, vegetable fields, etc., so as to facilitate the next step of fruit and vegetable detection.

[0025] Furthermore, after obtaining the fruit and vegetable images of the fruits and vegetables to be detected, in the embodiment of the present invention, feature extraction can be performed based on this to extract features that can accurately represent the fruits and vegetables, that is, the fruit and vegetable features of the fruits and vegetables to be detected. In order to be able to comprehensively and accurately capture the information about the fruits and vegetables to be detected in the fruit and vegetable images, so that this information can be fully applied in the subsequent detection process, thereby ensuring the precision and accuracy of the fruit and vegetable detection, in the embodiment of the present invention, at this time, multi-scale feature extraction can be performed on the fruit and vegetable images first, so as to extract the information of the fruits and vegetables to be detected in the fruit and vegetable images from multiple scales, thereby obtaining multi-scale fruit and vegetable features.

[0026] Specifically, a feature extraction network may be used here to extract features from multiple scales of the fruit and vegetable images to be detected, and obtain the extracted fruit and vegetable features at multiple scales, wherein the fruit and vegetable features at each scale contain the semantic information of the fruit and vegetables to be detected, and the integration of the semantic information at different scales can more completely and comprehensively reflect the semantic features of the fruit and vegetables to be detected, thereby ensuring the accuracy and reliability of subsequent detection. Here, the feature extraction network used can be constructed on the basis of the target detection model, for example, a feature extraction network for multi-scale feature extraction of fruit and vegetable images constructed on the basis of the YOLOv8 (You Only LookOnce version 8) model, or a feature extraction network constructed on the basis of the improved YOLOv8 model.

[0027] Furthermore, after obtaining the fruit and vegetable features, in order to fully apply the semantic information of the fruits and vegetables to be detected in the subsequent detection process and to reduce the amount of calculation as much as possible, in an embodiment of the present invention, the extracted multi-scale fruit and vegetable features can be fused to fully reflect the semantics of the fruits and vegetables to be detected, with a smaller amount of calculation and more accurate fruit and vegetable fusion features.

[0028] Specifically, the extracted multi-scale fruit and vegetable features can be fused at the semantic level, that is, semantic fusion is performed to obtain the semantic information in the fruit and vegetable features of different scales and fuse them, so as to obtain the fruit and vegetable fusion features containing semantic information of different levels; fruit and vegetable detection based on this feature can more accurately distinguish adjacent maturity levels and obscured fruits and vegetables, which greatly improves the accuracy of fruit and vegetable detection and reduces the difficulty of detection.

[0029] After that, fruit and vegetable detection can be performed based on the fruit and vegetable fusion features obtained by semantic fusion to obtain the fruit and vegetable detection results of the fruits and vegetables to be detected. However, considering that in the current large-scale agricultural production, after detection and picking by automated technology (such as picking robots), human intervention is usually required to remove defective and diseased fruits and vegetables, reduce the risk of deterioration during transportation and storage, and the problems of high cost and long cycle, in the embodiment of the present invention, when performing fruit and vegetable detection, in addition to the maturity of the fruits and vegetables to be detected, the defects and diseases of the fruits and vegetables to be detected can also be considered, so that they can be detected as detection targets, so that the conclusion of whether it is mature and whether there are defects and diseases can be directly drawn in the detection link, and then targeted picking can be carried out in the subsequent picking link, and no secondary judgment is required after picking to remove bad fruits and rotten vegetables, which reduces labor costs and shortens the "output" cycle.

[0030] Specifically, the fusion features of fruits and vegetables can be used to perform maturity detection and defect disease detection respectively to detect the maturity of the fruits and vegetables to be detected and whether there are defects and diseases, so as to obtain the fruit and vegetable detection results of the fruits and vegetables to be detected. The fruit and vegetable detection results here can include maturity detection results and defect disease detection results, or only defect disease detection results. The reason is that in order to save computing resources and reduce costs, after obtaining the results of defects and diseases, the subsequent process can be omitted and maturity detection is no longer performed, that is, only mature fruits and vegetables without defects and diseases are picked, and fruits and vegetables with defects and diseases are discarded. In this way, the problem of manual judgment and elimination one by one after picking in the whole range can be avoided, which improves the pertinence and effectiveness of picking and greatly reduces costs.

[0031] It should be noted that when performing maturity detection and defect disease detection, there may be slight differences in the detection of different types of fruits and vegetables. For example, for tomatoes, there are usually three levels of maturity, mature, green-ripe and unripe, while for fruits such as bananas and oranges, there are usually only two levels of maturity, mature and unripe. Correspondingly, for different types of fruits and vegetables, the apparent manifestations of defects and diseases may be different. Therefore, when performing fruit and vegetable detection on specific fruits and vegetables to be detected, the detection process can be adaptively adjusted according to their growth performance to ensure the precision and accuracy of the detection.

[0032] Furthermore, after completing the fruit and vegetable inspection and obtaining the fruit and vegetable inspection results of the fruits and vegetables to be inspected, in the embodiment of the present invention, fruits and vegetables can be picked based on this, that is, fruits and vegetables can be picked based on the fruit and vegetable inspection results by a picking robot to pick mature fruits and vegetables without defects and diseases, thereby achieving fast and effective fruit and vegetable picking, which can not only greatly reduce the intensity of manual labor and save a lot of labor costs, but also greatly improve the accuracy and work efficiency of picking, adapt to the needs of the modern fruit and vegetable planting industry for efficient and accurate picking, and effectively promote the development of agricultural modernization.

[0033] The fruit and vegetable detection method provided by the present invention obtains fruit and vegetable images of fruits and vegetables to be detected; performs multi-scale feature extraction based on the fruit and vegetable images to extract accurate fruit and vegetable features, and performs semantic fusion on the extracted multi-scale fruit and vegetable features to capture semantic information of different levels of the fruits and vegetables to be detected to obtain fruit and vegetable fusion features; based on the fruit and vegetable fusion features, maturity detection and defect and disease detection are performed on the fruits and vegetables to be detected to obtain fruit and vegetable detection results of the fruits and vegetables to be detected, thereby overcoming the defects of traditional fruit and vegetable detection schemes such as high detection difficulty and low accuracy, and high cost and long cycle of human intervention judgment after picking, thereby achieving efficient and accurate fruit and vegetable detection, providing data support for automatic and accurate picking of fruits and vegetables, and providing strong support for improving the intelligence level of picking robots and the intelligent transformation of agricultural production.

[0034] Based on the above embodiment, step 120 includes: The feature extraction network in the fruit and vegetable detection model is applied to perform multi-scale feature extraction based on fruit and vegetable images, and the multi-scale fruit and vegetable features obtained by feature extraction are semantically fused to obtain fruit and vegetable fusion features; the defect detection network and maturity detection network in the fruit and vegetable detection model are applied to perform defect and disease detection and maturity detection on the fruits and vegetables to be detected based on the fruit and vegetable fusion features, respectively, to obtain the fruit and vegetable detection results of the fruits and vegetables to be detected; The fruit and vegetable detection model is trained on the basis of a lightweight target detection model using sample fruit and vegetable images and their corresponding maturity labels and defect and disease labels.

[0035] Specifically, in the above step 120, the process of performing fruit and vegetable detection according to the fruit and vegetable images and obtaining the fruit and vegetable detection results can be implemented by a fruit and vegetable detection model.

[0036] Figure 2 It is a structural schematic diagram of the fruit and vegetable detection model provided by the present invention, such as Figure 2As shown in the figure, the fruit and vegetable detection model includes a feature extraction network, a maturity detection network and a defect detection network. Among them, the feature extraction network is used to perform multi-scale feature extraction on fruit and vegetable images, and semantically fuse the multi-scale fruit and vegetable features obtained by feature extraction to obtain fruit and vegetable fusion features. The maturity detection MD (Maturity Detect) network is used to perform maturity detection on the fruits and vegetables to be detected based on the fruit and vegetable fusion features, and the maturity detection results of the fruits and vegetables to be detected can be obtained. Correspondingly, the defect detection DD (DefectDetect) network performs defect and disease detection on the fruits and vegetables to be detected based on the fruit and vegetable fusion features, and the defect and disease detection results of the fruits and vegetables to be detected can be obtained.

[0037] In detail, for the fruit and vegetable detection model, the fruit and vegetable images of the fruits and vegetables to be detected are taken as input. After the fruit and vegetable images are input, the feature extraction network in the fruit and vegetable detection model can be first applied to perform multi-scale feature extraction on the fruit and vegetable images to be detected, so as to obtain multi-scale fruit and vegetable features. Then, the multi-scale fruit and vegetable features can be semantically fused to obtain fruit and vegetable fusion features; then, the defect detection network and maturity detection network in the fruit and vegetable detection model can be applied, and the fruit and vegetable fusion features can be used to perform defect and disease detection and maturity detection respectively, so as to obtain the fruit and vegetable detection results of the fruits and vegetables to be detected; the fruit and vegetable detection results here can include maturity detection results and defect and disease detection results, or can only include defect and disease detection results.

[0038] It is worth noting that before inputting fruit and vegetable images into the fruit and vegetable detection model, it is necessary to pre-train the fruit and vegetable detection model, that is, the fruit and vegetable detection model can be pre-trained using sample fruit and vegetable images and their corresponding labels. However, considering that in the process of model-based fruit and vegetable detection, high-performance prediction models require strong computing power support, and the harvesting robot is limited by factors such as cost and power consumption, and its computing power is limited, making it difficult to directly deploy a high-performance model. The model is usually complex in architecture, has high computational cost, is time-consuming, and has poor real-time detection.

[0039] In view of this, in an embodiment of the present invention, when constructing a fruit and vegetable detection model for fruit and vegetable detection, factors such as the performance and computing power requirements of the model are taken into consideration to construct a model with high performance, low computing power requirements, high detection accuracy, and a wide range of adaptability. Specifically, here, it can be improved on the basis of the current mature target detection model, such as the YOLOv8 model, to perform lightweight processing on it, reduce the number of model parameters, and reduce the computing power resource requirements, so as to obtain a lightweight target detection model, that is, the initial model for fruit and vegetable detection; then, the sample fruit and vegetable images and their maturity labels and defect disease labels can be used to train the initial model, and finally a trained fruit and vegetable detection model can be obtained.

[0040] Among them, the sample fruit and vegetable images used for model training can be collected in advance, and their corresponding maturity labels and defect and disease labels can be manually annotated or obtained together with the sample fruit and vegetable images when they are obtained. The embodiment of the present invention does not make specific limitations on this.

[0041] For example, images of fruits and vegetables can be directly crawled from public websites on the Internet as sample fruit and vegetable images, and then manually labeled to obtain maturity labels and defect and disease labels of the sample fruit and vegetable images.

[0042] For example, when obtaining sample fruit and vegetable images, you can specifically search for images with maturity and defect labels from public data sets and websites, and crawl such images as sample fruit and vegetable images. In this way, you can determine their maturity labels and defect and disease labels while obtaining the sample fruit and vegetable images.

[0043] For another example, images of fruits and vegetables can be collected on-site (orchards, vegetable fields, etc.) through image acquisition equipment as sample fruit and vegetable images. These small number of images can be annotated by experts to obtain maturity labels and defect and disease labels.

[0044] In addition, it should be noted that after the training is completed, in order to test the effect of the training and verify the performance of the model, it is usually necessary to verify and test the trained model. That is, when collecting sample image sets (sample fruit and vegetable images and their labels), the sample image sets can be divided into three parts in advance, namely, training set, verification set and test set. The training set is first used to train the initial model. After the training is completed, the verification set and test set are used to verify and test the trained model. Finally, a fruit and vegetable detection model with excellent performance can be obtained.

[0045] In the embodiment of the present invention, on the basis of the traditional target detection model, a lightweight target detection model is constructed, which is used as the initial model in the training process, and the model is trained using a sample image set to obtain a fruit and vegetable detection model. Based on this fruit and vegetable detection model, maturity detection and defect disease detection are performed on the fruits and vegetables to be detected. While improving the accuracy of detection and reducing labor costs, the computing resources and computing costs required for detection are greatly reduced, and a significant improvement in detection accuracy and real-time performance is achieved. In addition, the fruit and vegetable detection model obtained based on this training in the embodiment of the present invention has good generalization ability and robustness in the actual agricultural production environment, and can cope with complex and changeable field environments.

[0046] Based on the above embodiment, in step 120, based on the fruit and vegetable fusion features, defect disease detection and maturity detection are performed on the fruits and vegetables to be detected, respectively, to obtain the fruit and vegetable detection results of the fruits and vegetables to be detected, including: Based on the fusion features of fruits and vegetables, defects and diseases of the fruits and vegetables to be tested are detected to obtain the defect and disease detection results of the fruits and vegetables to be tested; If the defect and disease detection results indicate that the fruits and vegetables to be tested have defects and diseases, the maturity test will not be conducted on the fruits and vegetables to be tested; Otherwise, the maturity of the fruits and vegetables to be detected is detected based on the fruit and vegetable fusion features to obtain the maturity detection results of the fruits and vegetables to be detected; The fruit and vegetable inspection results are determined based on the defect and disease inspection results, or the defect and disease inspection results and the maturity inspection results.

[0047] Specifically, in step 120, the process of performing defect disease detection and maturity detection on the fruits and vegetables to be detected according to the fruit and vegetable fusion characteristics to obtain the fruit and vegetable detection results of the fruits and vegetables to be detected may specifically include: First, based on the fruit and vegetable fusion features, defect and disease detection can be performed on the fruits and vegetables to be detected to detect whether they have defects and diseases, so as to obtain the defect and disease detection results of the fruits and vegetables to be detected; here, specifically, on the basis of the fruit and vegetable fusion features, the defect detection DD network in the fruit and vegetable detection model can be used to perform defect and disease detection, so as to obtain the defect and disease detection results of the fruits and vegetables to be detected output by the defect detection DD network.

[0048] Next, if the defect and disease detection results show that the fruits and vegetables to be tested have defect and disease problems, that is, they have defects and / or pests and diseases, in short, the corresponding fruits and vegetables to be tested are bad fruits and rotten vegetables, and there is no need to perform maturity detection on them at this time to facilitate subsequent picking, that is, there is no need to perform maturity detection on the fruits and vegetables to be tested. Correspondingly, if the defect and disease detection results show that the fruits and vegetables to be tested do not have defect and disease problems, that is, they have no defects and / or pests and diseases, then maturity detection can continue to be performed on them, that is, the fruit and vegetable fusion features are used to perform maturity detection on the fruits and vegetables to be tested, and the maturity detection results of the fruits and vegetables to be tested are obtained; here, specifically, on the basis of the fruit and vegetable fusion features, the maturity detection MD network in the fruit and vegetable detection model is used to perform maturity disease detection on the fruits and vegetables to be tested, thereby obtaining the maturity detection results of the fruits and vegetables to be tested output by the maturity detection MD network.

[0049] Then, the fruit and vegetable detection results of the fruits and vegetables to be detected can be determined based on the defect and disease detection results, or the defect and disease detection results and the maturity detection results. That is, when the defect and disease detection results show that the fruits and vegetables to be detected have defect and disease problems, the defect and disease detection results can be directly used as the fruit and vegetable detection results of the fruits and vegetables to be detected; and when the defect and disease detection results show that the fruits and vegetables to be detected do not have defect and disease problems, the fruit and vegetable detection results of the fruits and vegetables to be detected can be determined based on the defect and disease detection results and the maturity detection results.

[0050] Based on the above embodiment, the fruit and vegetable detection model is determined based on the following steps: Determine a first sample image set, a second sample image set, and an initial detection model; the first sample image set includes first sample fruit and vegetable images and their corresponding maturity labels; the second sample image set includes second sample fruit and vegetable images and their corresponding defect and disease labels; the initial detection model includes an initial feature extraction network, an initial maturity detection network, and an initial defect detection network, and the initial feature extraction network is constructed based on a lightweight target detection model; Based on the initial feature extraction network, extract the first fruit and vegetable fusion feature of the first sample fruit and vegetable image, and based on the first fruit and vegetable fusion feature, obtain the predicted maturity detection result corresponding to the first sample fruit and vegetable image through the initial maturity detection network detection; Based on the predicted maturity detection results and maturity labels, the initial feature extraction network and the initial maturity detection network in the initial detection model are iterated to obtain a feature extraction network and a maturity detection network; Based on the feature extraction network, extract the second fruit and vegetable fusion features of the second sample fruit and vegetable image, and based on the second fruit and vegetable fusion features, obtain the predicted defect and disease detection results corresponding to the second sample fruit and vegetable image through the initial defect detection network detection; Based on the predicted defect disease detection results and defect disease labels, the initial defect detection network in the initial detection model is iterated on parameters to obtain a defect detection network; A fruit and vegetable detection model is constructed based on feature extraction network, maturity detection network and defect detection network.

[0051] Specifically, the training process of the above-mentioned fruit and vegetable detection model may include the following steps: First, it is necessary to determine the data set required for model training, that is, the sample image set. Since the maturity detection and defect disease detection of fruits and vegetables belong to different tasks, their focus is slightly different. Therefore, in order to ensure the performance of the model on the two tasks and improve the accuracy of detection, in the embodiment of the present invention, the two detection networks can be trained separately and their respective training data sets, that is, the first sample image set and the second sample image set, can be constructed. Among them, the first sample image set contains the first sample fruit and vegetable images and their corresponding maturity labels, and the second sample image set contains the second sample fruit and vegetable images and their corresponding defect disease labels.

[0052] At the same time, it is also necessary to determine the initial model in the training process, that is, the initial detection model; the initial detection model includes an initial feature extraction network, an initial maturity detection network, and an initial defect detection network. Among them, the initial feature extraction network is constructed based on a lightweight target detection model.

[0053] Then, the initial feature extraction network in the initial detection model can be used to perform multi-scale feature extraction on the first sample fruit and vegetable images in the first sample image set to obtain multi-scale first sample fruit and vegetable features. Then, the extracted multi-scale first sample fruit and vegetable features can be semantically fused to obtain first fruit and vegetable fusion features. Afterwards, based on the first fruit and vegetable fusion features obtained by semantic fusion, the initial maturity detection network in the initial detection model can be used to perform maturity detection to obtain the predicted maturity detection result corresponding to the first sample fruit and vegetable images.

[0054] Subsequently, the loss of the initial detection model on the maturity detection task can be measured based on the predicted maturity detection result and the maturity label corresponding to the first sample fruit and vegetable image, and the initial feature extraction network and the initial maturity detection network in the initial detection model can be trained based on this loss, that is, the parameters of the initial feature extraction network and the initial maturity detection network are adjusted according to the measured loss, so that the predicted maturity detection result output by the adjusted model is as close as possible to the corresponding maturity label, and finally the feature extraction network and maturity detection network trained on the task can be obtained.

[0055] Afterwards, the parameters of the feature extraction network and the maturity detection network can be fixed, and the initial defect detection network in the initial detection model can be trained using the second sample image set to obtain a trained defect detection network; here, specifically, the feature extraction network obtained by the previous step of training can be used to perform multi-scale feature extraction on the second sample fruit and vegetable images in the second sample image set to obtain multi-scale second sample fruit and vegetable features, and then the extracted multi-scale second sample fruit and vegetable features can be semantically fused to obtain second fruit and vegetable fusion features; then, based on the second fruit and vegetable fusion features obtained by semantic fusion, the initial defect detection network in the initial detection model can be used to perform defect and disease detection to obtain a predicted defect and disease detection result corresponding to the second sample fruit and vegetable image; further, based on the predicted defect and disease detection result and the defect and disease label corresponding to the second sample fruit and vegetable image, the loss of the initial detection model in the defect and disease detection task can be measured, and the initial defect detection network can be trained based on this loss, that is, the parameters of the initial defect detection network are adjusted according to the measured loss, so that the predicted defect and disease detection result output by the adjusted model is as close as possible to the corresponding defect and disease label, and finally a trained defect detection network can be obtained.

[0056] Thereafter, a fruit and vegetable detection model for fruit and vegetable detection can be constructed based on the feature extraction network, maturity detection network and defect detection network obtained through the above training.

[0057] In an embodiment of the present invention, an initial feature extraction network is constructed based on a lightweight target detection model, and an initial detection model is constructed in combination with an initial maturity detection network and an initial defect detection network. The initial detection model is trained using a first sample image set and a second sample image set to obtain a fruit and vegetable detection model. Based on this model, multi-task discrimination of maturity detection, appearance defects, and disease classification can be achieved, which effectively improves the utilization rate of features extracted by the feature extraction network and overall reduces model parameters and the time cost required for model training.

[0058] Based on the above embodiment, the initial feature extraction network is determined based on the following steps: Determine the target detection model; The parameters of the neck network of the target detection model are lightweighted to obtain a lightweight target detection model, which includes a backbone network, a neck network and a head network; Based on the lightweight target detection model, an initial feature extraction network is constructed.

[0059] Specifically, the construction process of the initial feature extraction network includes the following steps: First, the initial model used to construct the initial feature extraction network can be determined. Since the feature extraction network is mainly used for feature extraction and semantic fusion of images, in the embodiment of the present invention, a target detection model with strong information capture capability is selected as the initial model in the model construction process. The target detection model here can be various types of relatively mature models for target detection, such as the target detection model of the YOLO series, Faster R-CNN (Fast Regional Convolutional Neural Network), SSD (Single Shot MultiBox Detector), etc. Preferably, the YOLO V8 model in the YOLO series is selected as the target detection model in the embodiment of the present invention.

[0060] Next, the target detection model can be processed to reduce the number of model parameters while ensuring the performance of the model, thereby achieving model lightweight and reducing computing power requirements, thereby obtaining a lightweight target detection model. Figure 3 It is a schematic diagram of the structure of the lightweight target detection model provided by the present invention. Figure 3As shown, the target detection model (YOLO V8 model) includes three parts: the backbone network (Backbone), the neck network (Neck) and the head network (Head). When performing parameter lightweighting, the last three C2f modules (numbered 13, 16 and 19 in the figure) in the neck network can be lightweighted, that is, these three C2f modules are replaced with VoVGSCSP (VoVNet-based Generalized and Simplified CSP) modules, so that the accuracy of detection can be guaranteed while reducing the number of model parameters. In addition, in the embodiment of the present invention, a detection head part is added to merge features to capture semantic information at different levels, thereby helping the model to better detect complex scenes and small-sized fruits and vegetables.

[0061] Figure 3 The MHSA in the backbone network is Multi-Head Self-Attention, which can capture information from different subspaces (i.e., "heads"), thereby enhancing the model's understanding and processing capabilities of complex inputs.

[0062] After that, the initial feature extraction network can be constructed based on the lightweight target detection model obtained by lightweighting the parameters. Here, the lightweight target detection model can be directly used as the initial feature extraction network, or the lightweight target detection model can be further optimized and the parameters can be lightweighted to finally obtain the initial feature extraction network.

[0063] Based on the above embodiment, the first sample image set and the second sample image set are determined based on the following steps: Obtaining a first initial fruit and vegetable image and its corresponding maturity label, and a second initial fruit and vegetable image and its corresponding defect and disease label; Performing data enhancement processing on the first initial fruit and vegetable image and the second initial fruit and vegetable image respectively to obtain a plurality of first enhanced fruit and vegetable images and a plurality of second enhanced fruit and vegetable images; Determine a first sample fruit and vegetable image based on the first initial fruit and vegetable image and the first enhanced fruit and vegetable image, and determine a second sample fruit and vegetable image based on the second initial fruit and vegetable image and the second enhanced fruit and vegetable image; Based on the first sample fruit and vegetable images and maturity labels, a first sample image set is constructed, and based on the second sample fruit and vegetable images and defect and disease labels, a second sample image set is constructed; The data enhancement processing includes at least one of rotation transformation, scale transformation, Gaussian noise addition, displacement blur, and Poisson blur.

[0064] Specifically, the process of determining the first sample image set and the second sample image set includes: First, the initial images, i.e., the first initial fruit and vegetable images and the second initial fruit and vegetable images, can be determined, and their corresponding labels, i.e., the maturity label corresponding to the first initial fruit and vegetable images, and the defect and disease label corresponding to the second initial fruit and vegetable images, can be determined; the initial fruit and vegetable images and their corresponding labels here can be collected in advance, or collected and labeled.

[0065] Next, the first initial fruit and vegetable image and the second initial fruit and vegetable image may be subjected to data enhancement processing to obtain a plurality of first enhanced fruit and vegetable images and a plurality of second enhanced fruit and vegetable images; the data enhancement processing here includes at least one of rotation transformation, scale transformation, Gaussian noise addition, displacement blur, and Poisson blur. That is, the first initial fruit and vegetable image and the second initial fruit and vegetable image may be subjected to any one or more of rotation transformation, scale transformation, Gaussian noise addition, displacement blur, and Poisson blur to obtain the processed first enhanced fruit and vegetable image and the second enhanced fruit and vegetable image.

[0066] Then, the first sample fruit and vegetable image can be determined based on the first initial fruit and vegetable image and the first enhanced fruit and vegetable image, and the second sample fruit and vegetable image can be determined based on the second initial fruit and vegetable image and the second enhanced fruit and vegetable image; that is, the first initial fruit and vegetable image and the first enhanced fruit and vegetable image can be directly used as the first sample fruit and vegetable image, or a screening can be performed on the basis of the first initial fruit and vegetable image and the first enhanced fruit and vegetable image to screen out some high-quality images as the first sample fruit and vegetable image. Similarly, the second initial fruit and vegetable image and the second enhanced fruit and vegetable image can be directly used as the second sample fruit and vegetable image, or a screening can be performed on the basis of the second initial fruit and vegetable image and the second enhanced fruit and vegetable image to screen out some high-quality images as the second sample fruit and vegetable image.

[0067] Afterwards, a first sample image set can be constructed based on the first sample fruit and vegetable images and the corresponding maturity labels, and a second sample image set can be constructed based on the second sample fruit and vegetable images and the corresponding defect and disease labels. It should be noted that the first enhanced fruit and vegetable images and the second enhanced fruit and vegetable images obtained by performing data enhancement processing on the basis of the original first initial fruit and vegetable images and the second initial fruit and vegetable images have the same corresponding labels as the original first initial fruit and vegetable images and the second initial fruit and vegetable images. For example, for the first initial fruit and vegetable image A, after performing data enhancement processing on it, multiple first enhanced fruit and vegetable images A1, A2, A3, A4 and A5 can be obtained, and the maturity labels corresponding to A1, A2, A3, A4 and A5 are the maturity labels corresponding to the first initial fruit and vegetable image A.

[0068] In the embodiment of the present invention, data enhancement processing is performed on the initial fruit and vegetable images, sample fruit and vegetable images are determined based on the enhanced fruit and vegetable images and the initial fruit and vegetable images, and a sample data set is constructed based on the sample fruit and vegetable images and their corresponding labels. This achieves simple and fast data expansion, ensures sufficient diversity of training data, and provides support for improving model performance.

[0069] The fruit and vegetable detection device provided by the present invention is described below. The fruit and vegetable detection device described below and the fruit and vegetable detection method described above can be referred to each other.

[0070] Figure 4 Schematic diagram of the structure of the fruit tree detection device provided by the present invention. Figure 4 As shown, the device comprises: The acquisition unit 410 is used to acquire the fruit and vegetable images to be detected; The detection unit 420 is used to perform multi-scale feature extraction based on the fruit and vegetable image, and semantically fuse the multi-scale fruit and vegetable features obtained by the feature extraction to obtain fruit and vegetable fusion features; based on the fruit and vegetable fusion features, perform maturity detection and defect and disease detection on the fruits and vegetables to be detected, respectively, to obtain fruit and vegetable detection results of the fruits and vegetables to be detected.

[0071] The fruit and vegetable detection device provided by the present invention obtains fruit and vegetable images of fruits and vegetables to be detected; performs multi-scale feature extraction based on the fruit and vegetable images to extract accurate fruit and vegetable features, and performs semantic fusion on the extracted multi-scale fruit and vegetable features to capture semantic information of different levels of the fruits and vegetables to be detected to obtain fruit and vegetable fusion features; based on the fruit and vegetable fusion features, maturity detection and defect and disease detection are performed on the fruits and vegetables to be detected to obtain fruit and vegetable detection results of the fruits and vegetables to be detected, thereby overcoming the defects of traditional fruit and vegetable detection schemes such as high detection difficulty and low accuracy, and high cost and long cycle of human intervention judgment after picking, thereby realizing efficient and accurate fruit and vegetable detection, providing data support for automatic and accurate picking of fruits and vegetables, and providing strong support for improving the intelligence level of picking robots and the intelligent transformation of agricultural production.

[0072] Based on the above embodiment, the detection unit 420 is used for: Applying the feature extraction network in the fruit and vegetable detection model, performing multi-scale feature extraction based on the fruit and vegetable images, and semantically fusing the multi-scale fruit and vegetable features obtained by the feature extraction to obtain fruit and vegetable fusion features; Applying the defect detection network and maturity detection network in the fruit and vegetable detection model, based on the fruit and vegetable fusion features, respectively performing defect disease detection and maturity detection on the fruits and vegetables to be detected, and obtaining the fruit and vegetable detection results of the fruits and vegetables to be detected; The fruit and vegetable detection model is trained on the basis of a lightweight target detection model using sample fruit and vegetable images and their corresponding maturity labels and defect and disease labels.

[0073] Based on the above embodiment, the detection unit 420 is used for: Based on the fruit and vegetable fusion features, the defect and disease detection is performed on the fruit and vegetable to be detected, so as to obtain the defect and disease detection results of the fruit and vegetable to be detected; If the defect and disease detection result indicates that the fruits and vegetables to be detected have defects and diseases, then the maturity detection of the fruits and vegetables to be detected is not performed; Otherwise, performing maturity detection on the fruits and vegetables to be detected based on the fruit and vegetable fusion features to obtain maturity detection results of the fruits and vegetables to be detected; The fruit and vegetable detection result is determined based on the defect and disease detection result, or the defect and disease detection result and the maturity detection result.

[0074] Based on the above embodiment, the device further includes a training unit, which is used to: Determine a first sample image set, a second sample image set, and an initial detection model; the first sample image set includes first sample fruit and vegetable images and their corresponding maturity labels; the second sample image set includes second sample fruit and vegetable images and their corresponding defect and disease labels; the initial detection model includes an initial feature extraction network, an initial maturity detection network, and an initial defect detection network, and the initial feature extraction network is constructed based on the lightweight target detection model; Based on the initial feature extraction network, extract a first fruit and vegetable fusion feature of the first sample fruit and vegetable image, and based on the first fruit and vegetable fusion feature, obtain a predicted maturity detection result corresponding to the first sample fruit and vegetable image through the initial maturity detection network; Based on the predicted maturity detection result and the maturity label, performing parameter iteration on the initial feature extraction network and the initial maturity detection network in the initial detection model to obtain a feature extraction network and a maturity detection network; Based on the feature extraction network, extract the second fruit and vegetable fusion feature of the second sample fruit and vegetable image, and based on the second fruit and vegetable fusion feature, obtain the predicted defect and disease detection result corresponding to the second sample fruit and vegetable image through the initial defect detection network detection; Based on the predicted defect and disease detection result and the defect and disease label, performing parameter iteration on the initial defect detection network in the initial detection model to obtain a defect detection network; A fruit and vegetable detection model is constructed based on the feature extraction network, the maturity detection network and the defect detection network.

[0075] Based on the above embodiment, the training unit is used for: Determine the target detection model; Lightweighting parameters of the neck network of the target detection model to obtain a lightweight target detection model, wherein the target detection model includes a backbone network, a neck network, and a head network; Based on the lightweight target detection model, an initial feature extraction network is constructed.

[0076] Based on the above embodiment, the training unit is used for: Obtaining a first initial fruit and vegetable image and its corresponding maturity label, and a second initial fruit and vegetable image and its corresponding defect and disease label; performing data enhancement processing on the first initial fruit and vegetable image and the second initial fruit and vegetable image respectively to obtain a plurality of first enhanced fruit and vegetable images and a plurality of second enhanced fruit and vegetable images; Determine a first sample fruit and vegetable image based on the first initial fruit and vegetable image and the first enhanced fruit and vegetable image, and determine a second sample fruit and vegetable image based on the second initial fruit and vegetable image and the second enhanced fruit and vegetable image; Based on the first sample fruit and vegetable images and the maturity labels, a first sample image set is constructed; based on the second sample fruit and vegetable images and the defect and disease labels, a second sample image set is constructed; The data enhancement processing includes at least one of rotation transformation, scale transformation, Gaussian noise addition, displacement blur, and Poisson blur.

[0077] Figure 5 Schematic diagram of the structure of the picking robot provided by the present invention. Figure 5 As shown, the picking robot includes: an image acquisition device 510, a processor 520 and a picking mechanical arm 530; The image acquisition device 510 is used to acquire images of fruits and vegetables to be detected, and transmit the images of fruits and vegetables to the processor 520; The processor 520 is used to perform multi-scale feature extraction based on the fruit and vegetable image, and semantically fuse the multi-scale fruit and vegetable features obtained by feature extraction to obtain fruit and vegetable fusion features; based on the fruit and vegetable fusion features, perform maturity detection and defect disease detection on the fruits and vegetables to be detected, respectively, to obtain fruit and vegetable detection results of the fruits and vegetables to be detected, and transmit the fruit and vegetable detection results to the picking robot arm 530; The picking robot arm 530 is used to pick or discard the fruits and vegetables to be detected based on the fruit and vegetable detection results.

[0078] Figure 6 An example of a physical structure diagram of an electronic device is shown in FIG. Figure 6As shown, the electronic device may include: a processor 610, a communication interface 620, a memory 630 and a communication bus 640, wherein the processor 610, the communication interface 620 and the memory 630 communicate with each other through the communication bus 640. The processor 610 may call the logic instructions in the memory 630 to execute the fruit and vegetable detection method, which includes: obtaining fruit and vegetable images of fruits and vegetables to be detected; performing multi-scale feature extraction based on the fruit and vegetable images, and semantically fusing the multi-scale fruit and vegetable features obtained by the feature extraction to obtain fruit and vegetable fusion features; based on the fruit and vegetable fusion features, performing maturity detection and defect disease detection on the fruits and vegetables to be detected, respectively, to obtain fruit and vegetable detection results of the fruits and vegetables to be detected.

[0079] In addition, the logic instructions in the above-mentioned memory 630 can be implemented in the form of a software functional unit and can be stored in a computer-readable storage medium when it is sold or used as an independent product. Based on such an understanding, the technical solution of the present invention, in essence, or the part that contributes to the prior art or the part of the technical solution, can be embodied in the form of a software product, and the computer software product is stored in a storage medium, including a number of instructions for a computer device (which can be a personal computer, a server, or a network device, etc.) to perform all or part of the steps of the method described in each embodiment of the present invention. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (ROM, Read-Only Memory), random access memory (RAM, Random Access Memory), disk or optical disk, etc. Various media that can store program codes.

[0080] On the other hand, the present invention also provides a computer program product, which includes a computer program stored on a non-transitory computer-readable storage medium, and the computer program includes program instructions. When the program instructions are executed by a computer, the computer can execute the fruit and vegetable detection method provided by the above methods, and the method includes: obtaining fruit and vegetable images of fruits and vegetables to be detected; performing multi-scale feature extraction based on the fruit and vegetable images, and semantically fusing the multi-scale fruit and vegetable features obtained by the feature extraction to obtain fruit and vegetable fusion features; based on the fruit and vegetable fusion features, performing maturity detection and defect and disease detection on the fruits and vegetables to be detected, respectively, to obtain fruit and vegetable detection results of the fruits and vegetables to be detected.

[0081] On the other hand, the present invention also provides a non-transitory computer-readable storage medium having a computer program stored thereon. When the computer program is executed by a processor, the method for detecting fruits and vegetables provided by the above methods is implemented, and the method includes: obtaining fruit and vegetable images of fruits and vegetables to be detected; performing multi-scale feature extraction based on the fruit and vegetable images, and semantically fusing the multi-scale fruit and vegetable features obtained by the feature extraction to obtain fruit and vegetable fusion features; based on the fruit and vegetable fusion features, performing maturity detection and defect and disease detection on the fruits and vegetables to be detected, respectively, to obtain fruit and vegetable detection results of the fruits and vegetables to be detected.

[0082] The device embodiments described above are merely illustrative, wherein the units described as separate components may or may not be physically separated, and the components displayed as units may or may not be physical units, that is, they may be located in one place, or they may be distributed on multiple network units. Some or all of the modules may be selected according to actual needs to achieve the purpose of the scheme of this embodiment. Ordinary technicians in this field can understand and implement it without paying creative labor.

[0083] Through the description of the above implementation methods, those skilled in the art can clearly understand that each implementation method can be implemented by means of software plus a necessary general hardware platform, and of course, can also be implemented by hardware. Based on this understanding, the above technical solution is essentially or the part that contributes to the prior art can be embodied in the form of a software product, and the computer software product can be stored in a computer-readable storage medium, such as ROM / RAM, a disk, an optical disk, etc., including a number of instructions for a computer device (which can be a personal computer, a server, or a network device, etc.) to execute the methods described in each embodiment or some parts of the embodiments.

[0084] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit it. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or make equivalent replacements for some of the technical features therein. However, these modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. A method for detecting fruits and vegetables, characterized in that: include: Acquire images of fruits and vegetables to be detected; Performing multi-scale feature extraction based on the fruit and vegetable images, and performing semantic fusion on the multi-scale fruit and vegetable features obtained by the feature extraction to obtain fruit and vegetable fusion features; Based on the fruit and vegetable fusion features, maturity detection and defect and disease detection are performed on the fruits and vegetables to be detected, respectively, to obtain fruit and vegetable detection results of the fruits and vegetables to be detected.

2. The fruit and vegetable detection method according to claim 1, characterized in that: The multi-scale feature extraction is performed based on the fruit and vegetable images, and the multi-scale fruit and vegetable features obtained by the feature extraction are semantically fused to obtain fruit and vegetable fusion features; Based on the fruit and vegetable fusion features, the fruits and vegetables to be detected are respectively subjected to maturity detection and defect and disease detection, and the fruit and vegetable detection results of the fruits and vegetables to be detected are obtained, including: Applying the feature extraction network in the fruit and vegetable detection model, performing multi-scale feature extraction based on the fruit and vegetable images, and semantically fusing the multi-scale fruit and vegetable features obtained by the feature extraction to obtain fruit and vegetable fusion features; Applying the defect detection network and maturity detection network in the fruit and vegetable detection model, based on the fruit and vegetable fusion features, respectively performing defect disease detection and maturity detection on the fruits and vegetables to be detected, and obtaining the fruit and vegetable detection results of the fruits and vegetables to be detected; The fruit and vegetable detection model is trained on the basis of a lightweight target detection model using sample fruit and vegetable images and their corresponding maturity labels and defect and disease labels.

3. The fruit and vegetable detection method according to claim 1 or 2, characterized in that: Based on the fruit and vegetable fusion features, defect and disease detection and maturity detection are respectively performed on the fruits and vegetables to be detected to obtain the fruit and vegetable detection results of the fruits and vegetables to be detected, including: Based on the fruit and vegetable fusion features, the defect and disease detection is performed on the fruit and vegetable to be detected, so as to obtain the defect and disease detection results of the fruit and vegetable to be detected; If the defect and disease detection result indicates that the fruits and vegetables to be detected have defects and diseases, then the maturity detection of the fruits and vegetables to be detected is not performed; Otherwise, performing maturity detection on the fruits and vegetables to be detected based on the fruit and vegetable fusion features to obtain maturity detection results of the fruits and vegetables to be detected; The fruit and vegetable detection result is determined based on the defect and disease detection result, or the defect and disease detection result and the maturity detection result.

4. The fruit and vegetable detection method according to claim 2, characterized in that: The fruit and vegetable detection model is determined based on the following steps: Determine a first sample image set, a second sample image set, and an initial detection model; the first sample image set includes first sample fruit and vegetable images and their corresponding maturity labels; the second sample image set includes second sample fruit and vegetable images and their corresponding defect and disease labels; the initial detection model includes an initial feature extraction network, an initial maturity detection network, and an initial defect detection network, and the initial feature extraction network is constructed based on the lightweight target detection model; Based on the initial feature extraction network, extract a first fruit and vegetable fusion feature of the first sample fruit and vegetable image, and based on the first fruit and vegetable fusion feature, obtain a predicted maturity detection result corresponding to the first sample fruit and vegetable image through the initial maturity detection network; Based on the predicted maturity detection result and the maturity label, performing parameter iteration on the initial feature extraction network and the initial maturity detection network in the initial detection model to obtain a feature extraction network and a maturity detection network; Based on the feature extraction network, extract the second fruit and vegetable fusion feature of the second sample fruit and vegetable image, and based on the second fruit and vegetable fusion feature, obtain the predicted defect and disease detection result corresponding to the second sample fruit and vegetable image through the initial defect detection network detection; Based on the predicted defect and disease detection result and the defect and disease label, performing parameter iteration on the initial defect detection network in the initial detection model to obtain a defect detection network; A fruit and vegetable detection model is constructed based on the feature extraction network, the maturity detection network and the defect detection network.

5. The method for detecting fruits and vegetables according to claim 4, characterized in that: The initial feature extraction network is determined based on the following steps: Determine the target detection model; Lightweighting parameters of the neck network of the target detection model to obtain a lightweight target detection model, wherein the target detection model includes a backbone network, a neck network, and a head network; Based on the lightweight target detection model, an initial feature extraction network is constructed.

6. The method for detecting fruits and vegetables according to claim 4, characterized in that: The first sample image set and the second sample image set are determined based on the following steps: Obtaining a first initial fruit and vegetable image and its corresponding maturity label, and a second initial fruit and vegetable image and its corresponding defect and disease label; performing data enhancement processing on the first initial fruit and vegetable image and the second initial fruit and vegetable image respectively to obtain a plurality of first enhanced fruit and vegetable images and a plurality of second enhanced fruit and vegetable images; Determine a first sample fruit and vegetable image based on the first initial fruit and vegetable image and the first enhanced fruit and vegetable image, and determine a second sample fruit and vegetable image based on the second initial fruit and vegetable image and the second enhanced fruit and vegetable image; Based on the first sample fruit and vegetable images and the maturity labels, a first sample image set is constructed; based on the second sample fruit and vegetable images and the defect and disease labels, a second sample image set is constructed; The data enhancement processing includes at least one of rotation transformation, scale transformation, Gaussian noise addition, displacement blur, and Poisson blur.

7. A fruit and vegetable detection device, characterized in that: include: An acquisition unit, used for acquiring images of fruits and vegetables to be detected; A detection unit, configured to perform multi-scale feature extraction based on the fruit and vegetable image, and semantically fuse the multi-scale fruit and vegetable features obtained by the feature extraction to obtain a fruit and vegetable fusion feature; Based on the fruit and vegetable fusion features, maturity detection and defect and disease detection are performed on the fruits and vegetables to be detected, respectively, to obtain fruit and vegetable detection results of the fruits and vegetables to be detected.

8. A picking robot, characterized in that: include: Image acquisition equipment, processors, and harvesting robotic arms; The image acquisition device is used to acquire images of fruits and vegetables to be detected, and transmit the images of fruits and vegetables to the processor; The processor is used to perform multi-scale feature extraction based on the fruit and vegetable image, and perform semantic fusion on the multi-scale fruit and vegetable features obtained by feature extraction to obtain fruit and vegetable fusion features; Based on the fruit and vegetable fusion features, respectively perform maturity detection and defect and disease detection on the fruits and vegetables to be detected, obtain fruit and vegetable detection results of the fruits and vegetables to be detected, and transmit the fruit and vegetable detection results to the picking robot arm; The picking robot arm is used to pick or discard the fruits and vegetables to be detected based on the fruit and vegetable detection results.

9. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein: When the processor executes the computer program, the fruit and vegetable detection method according to any one of claims 1 to 6 is implemented.

10. A non-transitory computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the fruit and vegetable detection method according to any one of claims 1 to 6 is implemented.

Images