A strawberry defect detection method and device, computer equipment and storage medium

Abstract

This application belongs to the field of computer vision technology and discloses a method, apparatus, computer device, and storage medium for strawberry defect detection. The method includes: acquiring an original image containing a strawberry to be detected; preprocessing the original image to obtain an image to be detected; inputting the image to be detected into a pre-trained strawberry defect detection model to obtain surface defect detection information and overall contour detection information of the strawberry to be detected, the strawberry defect detection model including an object detection and semantic segmentation network model; and obtaining the defect detection result of the strawberry to be detected based on the surface defect detection information and overall contour detection information. This application can achieve accurate and efficient defect detection of strawberries in industrial sorting scenarios.

Description

Technical Field

[0001] This application relates to the field of computer vision technology, and in particular to a method, apparatus, computer equipment, and storage medium for detecting defects in strawberries. Background Technology

[0002] Currently, there are two main methods for detecting defects in strawberries: one is manual screening, which involves visually identifying strawberries with deformities, rotten surfaces, or bruises. This method is not only labor-intensive and time-consuming, but also prone to errors and inefficiency due to insufficient prior knowledge and reduced concentration of screening personnel. The other method is machine vision, which combines artificial intelligence algorithms with visual sensors to improve detection efficiency. Existing machine vision methods are primarily used in outdoor and greenhouse cultivation areas, typically for detecting strawberry diseases and their location, and monitoring strawberry growth. However, in industrial sorting scenarios, it is often necessary to inspect the quality of strawberries on the production line. Existing technologies neither consider the potential quality problems caused by inadequate preservation during industrial sorting nor acknowledge that strawberry shape is also part of the quality screening process. Therefore, existing technologies suffer from the problem of inaccurate and inefficient detection of strawberry defects in industrial sorting scenarios. Summary of the Invention

[0003] This application provides a method, apparatus, computer equipment, and storage medium for detecting defects in strawberries, which can accurately and efficiently detect defects in strawberries in industrial sorting scenarios.

[0004] In a first aspect, embodiments of this application provide a method for detecting defects in strawberries, the method comprising:

[0005] Obtain the original image containing the strawberry to be detected;

[0006] The original image is preprocessed to obtain the image to be detected;

[0007] The image to be detected is input into a pre-trained strawberry defect detection model to obtain surface defect detection information and overall contour detection information of the strawberry to be detected. The strawberry defect detection model includes a target detection and semantic segmentation network model.

[0008] Based on the surface defect detection information and overall contour detection information of the strawberry to be tested, the defect detection results of the strawberry to be tested are obtained.

[0009] In one embodiment, the image to be detected is input into a pre-trained strawberry defect detection model to obtain surface defect detection information and overall contour detection information of the strawberry to be detected, including:

[0010] The image to be detected is input into a pre-trained object detection and semantic segmentation network model, which outputs the overall contour detection information of the strawberry to be detected and the prediction results of multiple surface defects.

[0011] The optimal surface defect prediction result is selected from multiple surface defect prediction results using a non-maximum suppression algorithm, thus obtaining the surface defect detection information of the strawberry to be detected.

[0012] In one embodiment, the original image is preprocessed to obtain the image to be detected, including:

[0013] Scale the original image to a preset resolution using a preset ratio;

[0014] The image at a preset resolution is normalized to obtain the image to be detected.

[0015] In one embodiment, the strawberry defect detection model further includes a classification network model; based on the surface defect detection information and overall contour detection information of the strawberry to be detected, the defect detection result of the strawberry to be detected is obtained, including:

[0016] Based on the original image, surface defect detection information of the strawberry to be detected, and overall contour detection information, a defect detection image of the strawberry to be detected is generated.

[0017] The overall contour detection information of the strawberry to be detected is input into a pre-trained classification network model to obtain the size defect detection result of the strawberry to be detected.

[0018] Based on the defect detection images and size defect detection results of the strawberries to be tested, the defect detection results of the strawberries to be tested are obtained.

[0019] In one embodiment, the surface defect detection information of the strawberry to be detected is a surface defect detection image, and the overall contour detection information of the strawberry to be detected is an overall contour detection image.

[0020] Based on the original image, surface defect detection information of the strawberry to be detected, and overall contour detection information, a defect detection image of the strawberry to be detected is generated, including:

[0021] The surface defect detection image is converted into a surface defect image according to a preset ratio, and the overall contour detection image is converted into an overall contour image according to a preset ratio. The surface defect image and the overall contour image are the same size as the original image.

[0022] The surface defect image, the overall contour image, and the original image are stitched together to generate a defect detection image.

[0023] In one embodiment, the pre-trained strawberry defect detection model is deployed in an embedded device.

[0024] In one embodiment, before acquiring the original image containing the strawberry to be detected, the method further includes:

[0025] A strawberry defect detection model is constructed, which includes an object detection and semantic segmentation network model and a classification network model. The object detection and semantic segmentation network model includes an object detection network and a semantic segmentation network.

[0026] Multiple sample images containing strawberries from an industrial sorting line were acquired, and defect annotations and contour mapping were performed on the strawberries in each sample image to obtain a training image set.

[0027] The strawberry defect detection model is trained based on the training image set to obtain a pre-trained strawberry defect detection model;

[0028] Deploy the pre-trained strawberry defect detection model on an embedded device.

[0029] Secondly, embodiments of this application provide a strawberry defect detection device, which includes:

[0030] The image acquisition module is used to acquire the original image containing the strawberry to be detected;

[0031] The image preprocessing module is used to preprocess the original image to obtain the image to be detected;

[0032] The model detection module is used to input the image to be detected into the pre-trained strawberry defect detection model to obtain surface defect detection information and overall contour detection information of the strawberry to be detected. The strawberry defect detection model includes a target detection and semantic segmentation network model.

[0033] The result output module is used to obtain the defect detection results of the strawberry to be detected based on the surface defect detection information and the overall contour detection information of the strawberry to be detected.

[0034] Thirdly, embodiments of this application provide a computer device, including a memory and a processor. The memory stores a computer program, and when the computer program is executed by the processor, the processor performs the steps of the strawberry defect detection method as described in any of the above embodiments.

[0035] Fourthly, embodiments of this application provide a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the steps of the strawberry defect detection method as described in any of the above embodiments.

[0036] In summary, compared with the prior art, the beneficial effects of the technical solution provided in this application include at least the following:

[0037] This application provides a strawberry defect detection method, which involves acquiring an original image containing a strawberry to be detected; preprocessing the original image to obtain an image to be detected; inputting the image to be detected into a pre-trained strawberry defect detection model to obtain surface defect detection information and overall contour detection information of the strawberry. The strawberry defect detection model includes a target detection and semantic segmentation network model, which can perform target detection and semantic segmentation of the image to be detected, thereby detecting surface defects and segmenting the overall shape of the strawberry, thus obtaining surface defect detection information and overall contour detection information. Based on the surface defect detection information and overall contour detection information of the strawberry to be detected, the defect detection result of the strawberry is obtained. This method can detect both surface defects and shape defects in strawberries, and is suitable for detecting strawberry defects in industrial sorting scenarios. The above method can detect images containing strawberries using a pre-trained strawberry defect detection model, accurately detecting both surface defects and the overall contour of the strawberry to determine whether shape defects exist, enabling accurate and efficient defect detection of strawberries in industrial sorting scenarios. Attached Figure Description

[0038] Figure 1 A flowchart of a strawberry defect detection method provided as an exemplary embodiment of this application.

[0039] Figure 2 An example diagram of a strawberry defect detection process provided as an exemplary embodiment of this application.

[0040] Figure 3 A flowchart of model training steps provided for an exemplary embodiment of this application.

[0041] Figure 4 This is a structural diagram of a strawberry defect detection model provided as an exemplary embodiment of this application.

[0042] Figure 5 An example diagram of data annotation provided for an exemplary embodiment of this application.

[0043] Figure 6 This is an application scenario diagram of a strawberry defect detection method provided as an exemplary embodiment of this application.

[0044] Figure 7 This is a structural diagram of a strawberry defect detection device provided as an exemplary embodiment of this application. Detailed Implementation

[0045] The technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, and not all embodiments. Based on the embodiments of this application, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application.

[0046] Please see Figure 1 This application provides a method for detecting defects in strawberries. This method uses a terminal device as the executing entity for application in industrial sorting scenarios. The method specifically includes the following steps:

[0047] Step S1: Obtain the original image containing the strawberry to be detected.

[0048] The strawberries to be tested can be strawberries transported on an industrial production line; the original images are strawberry images captured by a video acquisition device, and each original image can contain one or more strawberries; in specific implementation, in order to more accurately detect the quality of strawberries, the video acquisition device can be configured to take pictures of only one strawberry at a time, so that each original image contains only one strawberry; the above-mentioned video acquisition device can be installed directly above the production line. When setting the installation angle, in order to obtain clear and accurate strawberry images, the shooting angle of the video acquisition device can be set to be perpendicular to the production line, that is, directly facing the strawberry position.

[0049] Specifically, in an industrial sorting scenario, a video capture device acquires images of individual strawberries transported on the assembly line and sends these images to a terminal device. The terminal device then uses the image containing the strawberries as the original image. The terminal device can be connected to the video capture device either via a data cable or wirelessly.

[0050] Step S2: Preprocess the original image to obtain the image to be detected.

[0051] The image to be detected is the original image after preprocessing. The size of the image to be detected is usually smaller than that of the original image, which helps to reduce the amount of data to be processed. At the same time, the image to be detected meets the input requirements of the strawberry defect detection model in step S3.

[0052] Specifically, the original image is preprocessed to transform it into an image to be detected that meets the input requirements of the strawberry defect detection model. In practice, mature image preprocessing techniques such as scaling, cropping, and grayscale conversion can be used. For example, the original image can be processed according to a preprocessing flow of grayscale conversion, geometric transformation, and image enhancement to simplify the data to the maximum extent, thereby improving the reliability of subsequent object detection and semantic segmentation. Besides the preprocessing methods listed in this embodiment, other common image preprocessing methods can also be used.

[0053] Step S3: Input the image to be detected into the pre-trained strawberry defect detection model to obtain surface defect detection information and overall contour detection information of the strawberry to be detected. The strawberry defect detection model includes a target detection and semantic segmentation network model.

[0054] The object detection and semantic segmentation network model can be a combination of object detection and semantic segmentation networks. The object detection network can employ R-CNN, SSD, or YOLO series networks, while the semantic segmentation network can use FCN, SegNet, U-Net, or DeepLab series networks. This strawberry defect detection model is typically deployed on a terminal device. Besides the object detection and semantic segmentation networks listed above, other common object detection and semantic segmentation network models can be combined to construct the strawberry defect detection model, simultaneously achieving object detection and semantic segmentation functions. It should be noted that any content not described in detail in this specification is prior art known to those skilled in the art.

[0055] Specifically, the pre-trained object detection and semantic segmentation network model can perform object detection on the input image to obtain surface defect detection information of the strawberry through the object detection network, and can also perform semantic segmentation on the image to obtain overall contour detection information of the strawberry through the semantic segmentation network. In specific implementation, the surface defect detection information can include the location and defect category of the strawberry surface defects, and the overall contour detection information can include the overall shape contour of the strawberry. For example, different pixel values ​​can be used to distinguish the foreground and background of the strawberry to form a binary grayscale image representing the strawberry contour, and this binary grayscale image can be used as the overall contour detection information.

[0056] Step S4: Based on the surface defect detection information and overall contour detection information of the strawberry to be detected, the defect detection results of the strawberry to be detected are obtained.

[0057] The defect detection results may include a visualized defect detection image obtained based on surface defect detection information and overall contour detection information of the strawberry to be detected, and may also include dimensional defect detection results. The dimensional defect detection results can be obtained from the overall contour detection information. In the defect detection image, all surface defects of the strawberry are outlined and their corresponding defect types are displayed. Specifically, the terminal device can obtain the defect detection results of the strawberry to be detected based on the surface defect detection information and overall contour detection information. In some embodiments, the terminal device can visualize the defect detection results of the strawberry to be detected through a display device. The visualized defect detection results can be a defect detection image or a defect detection image marked with dimensional defect detection results.

[0058] The strawberry defect detection method provided in the above embodiments can acquire an original image containing strawberries to be detected; preprocess the original image to obtain an image to be detected; input the image to be detected into a pre-trained strawberry defect detection model to obtain surface defect detection information and overall contour detection information of the strawberry to be detected. The strawberry defect detection model includes a target detection and semantic segmentation network model, which can realize target detection and semantic segmentation of the image to be detected, thereby detecting surface defects of the strawberry and segmenting the overall shape of the strawberry, i.e., obtaining surface defect detection information and overall contour detection information of the strawberry; based on the surface defect detection information and overall contour detection information of the strawberry to be detected, the defect detection result of the strawberry to be detected is obtained. This method can detect both surface defects and shape defects of strawberries, and is suitable for detecting strawberry defects in industrial sorting scenarios. The above method uses a pre-trained strawberry defect detection model to detect images containing strawberries, accurately detecting both surface defects and the overall contour of the strawberry to determine whether there are shape defects, enabling accurate and efficient defect detection of strawberries in industrial sorting scenarios.

[0059] In some embodiments, step S3 specifically includes the following steps:

[0060] The image to be detected is input into a pre-trained object detection and semantic segmentation network model, which outputs the overall contour detection information of the strawberry to be detected and the prediction results of multiple surface defects.

[0061] The optimal surface defect prediction result is selected from multiple surface defect prediction results using a non-maximum suppression algorithm, thus obtaining the surface defect detection information of the strawberry to be detected.

[0062] The surface defect prediction results can be multiple candidate boxes corresponding to each defect in the strawberry image output by the object detection network. Since a large number of candidate boxes are generated at the same target location during object detection, and these candidate boxes may overlap, a non-maximum suppression algorithm is used to find the optimal candidate box, eliminate redundant candidate boxes, and use the resulting image containing only the optimal candidate boxes for each surface defect as the surface defect detection information for the strawberry to be detected. The non-maximum suppression algorithm, abbreviated as NMS, is used to search for local maxima and suppress maxima.

[0063] The above embodiments can eliminate redundant surface defect prediction results and output the best surface defect prediction results, thereby improving the accuracy of surface defect detection information.

[0064] In the specific implementation process, the quality of the image to be detected can directly affect the detection effect of the model. Therefore, before analyzing the image (feature extraction, segmentation, etc.), it is necessary to preprocess the original image to obtain an image that meets the input requirements of the strawberry defect detection model in step S3.

[0065] In some embodiments, step S2 specifically includes the following steps:

[0066] Scale the original image to a preset resolution using a preset ratio.

[0067] The image at a preset resolution is normalized to obtain the image to be detected.

[0068] The preset resolution is the input image resolution that meets the requirements of the strawberry defect detection model. It is usually smaller than the resolution of the original image. For example, the preset resolution is 640x640.

[0069] The above embodiments can process the original image into an image to be detected by scaling and normalization. First, the original image is scaled down to a smaller resolution image to reduce the amount of data to be processed. Then, the smaller resolution image is normalized to obtain the image to be detected. This can eliminate irrelevant information in the image, restore useful real information, enhance the detectability of relevant information, thereby simplifying image data and helping to improve the reliability of subsequent feature extraction, segmentation, etc.

[0070] In some embodiments, the strawberry defect detection model further includes a classification network model; step S4 specifically includes the following steps:

[0071] Based on the original image, surface defect detection information of the strawberry to be detected, and overall contour detection information, a defect detection image of the strawberry to be detected is generated.

[0072] The overall contour detection information of the strawberry to be detected is input into a pre-trained classification network model to obtain the size defect detection result of the strawberry to be detected.

[0073] Based on the defect detection images and size defect detection results of the strawberries to be tested, the defect detection results of the strawberries to be tested are obtained.

[0074] Among them, the defect detection image can be an image obtained by combining the original image, surface defect detection information and overall contour detection information. The defect detection image can be used to mark the location and type of defects on the surface of the strawberry, and blur the background of the image other than the strawberry to a certain extent, which can reduce the amount of data during subsequent storage or display processing; the size defect detection result can be either normal or abnormal. Abnormal strawberry size can include deformity, being too small or too large.

[0075] The above embodiments can use a classification network model to classify the shape of strawberries based on the overall contour detection information of strawberries, and can also visualize the defect detection images to facilitate the observation of strawberry defects.

[0076] In some embodiments of the above examples, the surface defect detection information of the strawberry to be detected is a surface defect detection image, and the overall contour detection information of the strawberry to be detected is an overall contour detection image. To generate a defect detection image that is easy to observe, the step of generating a defect detection image of the strawberry to be detected based on the original image, the surface defect detection information, and the overall contour detection information specifically includes the following steps:

[0077] The surface defect detection image is converted into a surface defect image according to a preset ratio, and the overall contour detection image is converted into an overall contour image according to a preset ratio. The surface defect image and the overall contour image are the same size as the original image.

[0078] The surface defect image, the overall contour image, and the original image are stitched together to generate a defect detection image.

[0079] Among them, the surface defect detection image is the target detection result output by the pre-trained strawberry defect detection model, and the overall contour detection image is the semantic segmentation result output by the pre-trained strawberry defect detection model.

[0080] Specifically, please see Figure 2 An example of the strawberry defect detection process is as follows: after the original image is preprocessed in step S2, it is input into a pre-trained model. The model outputs the target detection result and the semantic segmentation result. After processing the target detection result and the semantic segmentation result respectively, the generated defect detection image is visualized.

[0081] In some embodiments, in order to make the pre-trained strawberry defect detection model applicable to real-world industrial scenarios, the pre-trained strawberry defect detection model is deployed in an embedded device.

[0082] In some embodiments, a pre-trained strawberry defect detection model needs to be obtained before step S1. Please refer to [link to relevant documentation]. Figure 3 The method also includes the following steps:

[0083] Step S5: Construct a strawberry defect detection model. The strawberry defect detection model includes an object detection and semantic segmentation network model and a classification network model. The object detection and semantic segmentation network model includes an object detection network and a semantic segmentation network.

[0084] In the above embodiment, the strawberry defect detection model comprises two components: an object detection and semantic segmentation network model, and a classification network model. Since most machine learning algorithms are computationally inefficient and slow, they cannot meet the real-time detection requirements of industrial production lines. Therefore, in specific implementations, the object detection and semantic segmentation network model can employ an improved YOLOv5 network model. The YOLOv5 network is a single-stage detection algorithm, offering faster detection speeds and greater lightweight design compared to two-stage algorithms like Faster-RCNN, making it suitable for deployment in various scenarios. This embodiment needs to address the location and shape analysis of surface rot in strawberries. However, the YOLOv5 network can only locate the rotten area. Therefore, the YOLOv5 network needs to be improved by adding a semantic segmentation network to extract foreground information from the strawberry. Finally, this foreground information is input into the classification network model to classify the strawberry's shape (e.g., deformed, oversized, undersized, and normal) to obtain the size defect detection result. Among them, the foreground information of strawberries is the overall outline detection information of strawberries; the original YOLOv5 network model includes a feature extraction network, a feature fusion network and a detection network. The feature fusion network can input the feature map with larger fused features into the semantic segmentation network.

[0085] Please see below. Figure 4The YOLOv5 network consists of three main parts. The first part is the feature extraction network backbone, which extracts the feature information of the image to obtain the corresponding feature map F. The second part is the feature fusion network head, which fuses the feature information from different layers in the feature map F to obtain a feature map F' with richer feature information, thereby improving model performance. The third part is the detection network detect, which transforms the final feature map F' into feature vectors through a three-branch structure for model training. Its output vector contains three parts, namely output1, output2, and output3, with output sizes of 1 / 8, 1 / 16, and 1 / 32 of the input size, respectively. Output1 has a smaller receptive field and can mainly identify smaller objects, while output3 has the largest receptive field and can identify larger objects, and output2 is in between. Figure 4 The Mask network in this paper is an improved part of the YOLOv5 network. It refers to the segmentation head module of the Deeplabv3 network. It takes the feature map with a large feature map after fusion of the head part as input. First, it goes through two convolutional layers to adjust the number of output channels to the number of categories (the number of categories in this invention is 2, namely strawberry and background). Then, it goes through an 8x upsampling layer of bilinear interpolation to restore the size of the input image and obtain the semantic segmentation result. The specific process is shown in equations (1) and (2). The segmentation result is used as the input of the classification network. The classification network judges the shape of the strawberry based on the foreground information. During the training process, the Mask network calculates the cross-entropy loss with the original image pixel by pixel and adds it to the loss obtained from detection for backpropagation.

[0086] x l =Conv2d(W l ,x l-1 )+b l (1)

[0087] o l =Upsample(x l ,s) (2)

[0088] Where, x l This represents the two convolutional operations in the Mask network, x l-1 ∈R C×H×W For the output of module C3, x l ∈R 2×H×W W is the output after convolution. l b l ∈R represents the convolution parameters. l This represents the upsampling operation of the Mask network, where s represents the scaling factor, and o l ∈R 2×sH×sW This represents the output mask.

[0089] Step S6: Acquire multiple sample images containing strawberries from the industrial sorting line, and perform defect annotation and contour calibration on the strawberries in each sample image to obtain a training image set.

[0090] Please see below. Figure 5 In the example of data annotation, defect detection annotation is the bounding coordinates and defect category of surface defects of strawberries in the sample image; contour annotation is the annotation of the contour of strawberries in the sample image.

[0091] Specifically, in the data annotation process of sample images, it is necessary to annotate detection data and segmentation data. In practical implementation, for detection data, since the main source of defects during sorting is rot caused by insufficient preservation, this embodiment can focus only on locating rot defects. Generally, damage and pests are less common in strawberry inspection and can be disregarded here, but adjustments can be made according to the actual situation. For semantic segmentation data, the outline of the strawberry can be labeled to form a binary grayscale image, with the foreground and background of the strawberry distinguished by different pixel values. Using the same set of sample images as the defect localization helps with parallel training of the model and feature extraction.

[0092] Step S7: Train the strawberry defect detection model based on the training image set to obtain a pre-trained strawberry defect detection model.

[0093] Step S8: Deploy the pre-trained strawberry defect detection model in an embedded device.

[0094] The above embodiments can obtain a pre-trained strawberry defect detection model. This pre-trained strawberry defect detection model can not only locate surface rot defects in strawberries, but also analyze the shape of strawberries, which can increase the functionality of the model and improve detection efficiency.

[0095] Currently, detection models are generally used for inference computation based on CPUs or GPUs on computers or servers. However, due to the large size of computers and servers, they are inconvenient and costly as edge computing devices in real-world scenarios, making them unsuitable for practical applications. In some embodiments, to make the improved YOLOv5 model applicable to real-world industrial scenarios, step S104 specifically includes the following steps:

[0096] The pre-trained strawberry defect detection model was deployed on an embedded device using the OpenVINO toolkit.

[0097] Specifically, the pre-trained strawberry defect detection model can be converted into the ONNX representation format; this ONNX representation model can then be converted into the TensorRT format using the OpenVINO tool library; and the resulting TensorRT model can be deployed in an embedded device. For example... Figure 6 As shown, the pre-trained strawberry defect detection model (improved YOLOv5 model) can be ported to embedded devices and implemented in specific application scenarios using the OpenVINO tool library.

[0098] The strawberry defect detection model is built on the PyTorch framework, a deep learning framework. The ONNX representation format is the Open Neural Network Exchange (ONNX) format, a standard for representing deep learning models that allows models to be transferred between different frameworks. The TensorRT format is a common model format. The OpenVINO toolkit is a tool suite developed by Intel based on its existing hardware platform to accelerate the development of high-performance computer vision and deep learning vision applications. It supports deep learning on various Intel platform hardware accelerators and allows direct heterogeneous execution.

[0099] The above embodiments can deploy the model to embedded devices using the OpenVINO tool library, which can improve the detection speed of the deployed model and achieve real-time detection.

[0100] Please see Figure 7 Another embodiment of this application provides a strawberry defect detection device, which includes:

[0101] Image acquisition module 101 is used to acquire the original image containing the strawberry to be detected.

[0102] The image preprocessing module 102 is used to preprocess the original image to obtain the image to be detected.

[0103] The model detection module 103 is used to input the image to be detected into the pre-trained strawberry defect detection model to obtain surface defect detection information and overall contour detection information of the strawberry to be detected. The strawberry defect detection model includes a target detection and semantic segmentation network model.

[0104] The result output module 104 is used to obtain the defect detection results of the strawberry to be detected based on the surface defect detection information and the overall contour detection information of the strawberry to be detected.

[0105] In some embodiments, the model detection module 103 is specifically used to: input the image to be detected into a pre-trained object detection and semantic segmentation network model, so that the pre-trained object detection and semantic segmentation network model outputs the overall contour detection information of the strawberry to be detected and multiple surface defect prediction results; and select the optimal surface defect prediction result from the multiple surface defect prediction results through a non-maximum suppression algorithm to obtain the surface defect detection information of the strawberry to be detected.

[0106] In some embodiments, the image preprocessing module 102 is specifically used to: scale the original image to a preset resolution by a preset ratio; and normalize the image at the preset resolution to obtain the image to be detected.

[0107] In some embodiments, the strawberry defect detection model further includes a classification network model; the result output module 104 is specifically used for:

[0108] Based on the original image, surface defect detection information, and overall contour detection information of the strawberry to be detected, a defect detection image of the strawberry to be detected is generated; the overall contour detection information of the strawberry to be detected is input into a pre-trained classification network model to obtain the size defect detection result of the strawberry to be detected; based on the defect detection image and size defect detection result of the strawberry to be detected, the defect detection result of the strawberry to be detected is obtained.

[0109] In some embodiments, the surface defect detection information of the strawberry to be detected is a surface defect detection image, and the overall contour detection information of the strawberry to be detected is an overall contour detection image; the result output module 104 is further specifically used to: convert the surface defect detection image into a surface defect image according to a preset ratio, and convert the overall contour detection image into an overall contour image according to a preset ratio, wherein the surface defect image and the overall contour image are the same size as the original image; and stitch the surface defect image, the overall contour image and the original image together to generate a defect detection image.

[0110] In some embodiments, the pre-trained strawberry defect detection model is deployed in an embedded device.

[0111] In some embodiments, the apparatus further includes a model training module, which is specifically used for:

[0112] A strawberry defect detection model is constructed, which includes an object detection and semantic segmentation network model and a classification network model. The object detection and semantic segmentation network model includes an object detection network and a semantic segmentation network. Multiple sample images containing strawberries on an industrial sorting line are acquired, and defects in the strawberries in each sample image are labeled and contours are defined to obtain a training image set. The strawberry defect detection model is trained based on the training image set to obtain a pre-trained strawberry defect detection model. The pre-trained strawberry defect detection model is deployed in an embedded device.

[0113] The specific limitations of the strawberry defect detection device provided in this embodiment can be found in the embodiments of the strawberry defect detection method described above, and will not be repeated here. Each module in the above-described strawberry defect detection device can be implemented entirely or partially through software, hardware, or a combination thereof. Each module can be embedded in or independent of the processor in a computer device in hardware form, or stored in the memory of a computer device in software form, so that the processor can call and execute the operations corresponding to each module.

[0114] This application provides a computer device that may include a processor, memory, network interface, and database connected via a system bus. The processor provides computing and control capabilities. The memory includes a non-volatile storage medium and internal memory. The non-volatile storage medium stores an operating system, computer programs, and a database. The internal memory provides an environment for the operation of the operating system and computer programs in the non-volatile storage medium. The network interface communicates with external terminals via a network connection. When the computer program is executed by the processor, it causes the processor to perform the steps of the strawberry defect detection method as described in any of the above embodiments.

[0115] The working process, working details, and technical effects of the computer equipment provided in this embodiment can be found in the embodiment of the strawberry defect detection method described above, and will not be repeated here.

[0116] This application provides a computer-readable storage medium storing a computer program thereon. When the computer program is executed by a processor, it implements the steps of the strawberry defect detection method as described in any of the above embodiments. The computer-readable storage medium refers to a data storage medium, which may include, but is not limited to, floppy disks, optical disks, hard disks, flash memory, USB flash drives, and / or memory sticks. The computer may be a general-purpose computer, a special-purpose computer, a computer network, or other programmable devices.

[0117] The working process, working details, and technical effects of the computer-readable storage medium provided in this embodiment can be found in the embodiments of the strawberry defect detection method described above, and will not be repeated here.

[0118] Those skilled in the art will understand that all or part of the processes in the methods of the above embodiments can be implemented by a computer program instructing related hardware. The computer program can be stored in a non-volatile computer-readable storage medium, and when executed, it can include the processes of the embodiments of the above methods. Any references to memory, storage, databases, or other media used in the embodiments provided in this application can include non-volatile and / or volatile memory. Non-volatile memory can include read-only memory (ROM), programmable ROM (PROM), electrically programmable ROM (EPROM), electrically erasable programmable ROM (EEPROM), or flash memory. Volatile memory can include random access memory (RAM) or external cache memory. By way of illustration and not limitation, RAM is available in various forms, such as static RAM (SRAM), dynamic RAM (DRAM), synchronous DRAM (SDRAM), dual data rate SDRAM (DDRSDRAM), enhanced SDRAM (ESDRAM), synchronous link DRAM (SLDRAM), Rambus direct RAM (RDRAM), direct memory bus dynamic RAM (DRDRAM), and memory bus dynamic RAM (RDRAM), etc.

[0119] The technical features of the above embodiments can be combined in any way. For the sake of brevity, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0120] The embodiments described above are merely illustrative of several implementation methods of this application, and while the descriptions are relatively specific and detailed, they should not be construed as limiting the scope of the invention patent. It should be noted that those skilled in the art can make various modifications and improvements without departing from the concept of this application, and these all fall within the protection scope of this application. Therefore, the protection scope of this patent application should be determined by the appended claims.

Claims

1. A method for detecting defects in strawberries, characterized in that, The method includes: Obtain the original image containing the strawberry to be detected; The original image is preprocessed to obtain the image to be detected; A strawberry defect detection model is constructed, which includes an object detection and semantic segmentation network model and a classification network model. The object detection and semantic segmentation network model includes an object detection network and a semantic segmentation network. The target detection network is built based on the YOLOv5 network. It is used to receive the image to be detected, extract the features of the image to be detected, locate the rotten position of the strawberry, obtain surface defect detection information, and send the surface defect detection information to the semantic segmentation network. The semantic segmentation network performs semantic segmentation on the surface defect detection information to obtain overall contour detection information, and sends the overall contour detection information to the classification network model. The classification network model classifies the strawberry shape based on the overall contour detection information to obtain the size defect detection result of the strawberry to be detected. The size defect detection result includes normal and abnormal cases. The abnormal cases include deformity, being too small or too large. The image to be detected is input into a pre-trained strawberry defect detection model to obtain surface defect detection information and overall contour detection information of the strawberry to be detected. Based on the original image, the surface defect detection information of the strawberry to be detected, and the overall contour detection information, a defect detection image of the strawberry to be detected is generated. The defect detection image is used to mark the location and type of defects on the surface of the strawberry, and to blur the background of the image other than the strawberry. Based on the defect detection image of the strawberry to be detected and the size defect detection result, the defect detection result of the strawberry to be detected is obtained.

2. The method according to claim 1, characterized in that, The step of inputting the image to be detected into a pre-trained strawberry defect detection model to obtain surface defect detection information and overall contour detection information of the strawberry to be detected includes: The image to be detected is input into the pre-trained target detection and semantic segmentation network model, so that the pre-trained target detection and semantic segmentation network model outputs the overall contour detection information of the strawberry to be detected and the prediction results of multiple surface defects. The optimal surface defect prediction result is selected from the multiple surface defect prediction results using a non-maximum suppression algorithm to obtain the surface defect detection information of the strawberry to be detected.

3. The method according to claim 1, characterized in that, The preprocessing of the original image to obtain the image to be detected includes: The original image is scaled to a preset resolution using a preset ratio; The image at the preset resolution is normalized to obtain the image to be detected.

4. The method according to claim 3, characterized in that, The surface defect detection information of the strawberry to be detected is a surface defect detection image, and the overall contour detection information of the strawberry to be detected is an overall contour detection image. The step of generating a defect detection image of the strawberry to be detected based on the original image, the surface defect detection information of the strawberry to be detected, and the overall contour detection information includes: The surface defect detection image is converted into a surface defect image according to the preset ratio, and the overall contour detection image is converted into an overall contour image according to the preset ratio. The surface defect image and the overall contour image are the same size as the original image. The surface defect image, the overall contour image, and the original image are stitched together to generate a defect detection image.

5. The method according to any one of claims 1 to 4, characterized in that, The pre-trained strawberry defect detection model is deployed on an embedded device.

6. The method according to any one of claims 1 to 4, characterized in that, Before acquiring the original image containing the strawberry to be detected, the method further includes: A strawberry defect detection model is constructed, which includes an object detection and semantic segmentation network model and a classification network model. The object detection and semantic segmentation network model includes an object detection network and a semantic segmentation network. Multiple sample images containing strawberries from an industrial sorting line were acquired, and defect annotation and contour mapping were performed on the strawberries in each sample image to obtain a training image set. The strawberry defect detection model is trained based on the training image set to obtain a pre-trained strawberry defect detection model. The pre-trained strawberry defect detection model is deployed on an embedded device.

7. A strawberry defect detection device, characterized in that, The device includes: The image acquisition module is used to acquire the original image containing the strawberry to be detected; The image preprocessing module is used to preprocess the original image to obtain the image to be detected; A model building module is used to construct a strawberry defect detection model. This model includes an object detection and semantic segmentation network model and a classification network model. The object detection and semantic segmentation network model comprises an object detection network and a semantic segmentation network. The object detection network, built on the YOLOv5 network, receives the image to be detected, extracts features from the image, locates the rotten area of ​​the strawberry, obtains surface defect detection information, and sends this information to the semantic segmentation network. The semantic segmentation network performs semantic segmentation on the surface defect detection information to obtain overall contour detection information, which is then sent to the classification network model. The classification network model classifies the strawberry shape based on the overall contour detection information to obtain the size defect detection result of the strawberry to be detected. The size defect detection result includes normal and abnormal cases, with abnormal cases including deformed, too small, or too large. The model detection module is used to input the image to be detected into a pre-trained strawberry defect detection model to obtain surface defect detection information and overall contour detection information of the strawberry to be detected. The image fusion module is used to generate a defect detection image of the strawberry to be detected based on the original image, the surface defect detection information of the strawberry to be detected, and the overall contour detection information. The defect detection image is used to mark the location and type of defects on the surface of the strawberry and to blur the background of the image other than the strawberry. The result output module is used to obtain the defect detection result of the strawberry to be detected based on the defect detection image of the strawberry to be detected and the size defect detection result.

8. A computer device, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that, When the processor executes the computer program, it implements the steps of the method as described in any one of claims 1 to 6.

9. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by a processor, it implements the steps of the method as described in any one of claims 1 to 6.

Images