A method and system for detecting defects in shelf-grown fruit based on image recognition

Abstract

This invention discloses a method and system for detecting defects in fruit on shelves based on image recognition. The method includes image acquisition, image processing, and defect analysis steps, and incorporates an image acquisition unit, an image processing unit, a defect analysis unit, a control unit, and a timing unit. This invention relates to the field of fruit defect detection technology. This image recognition-based method and system for detecting defects in fruit on shelves accurately solves the problem of machine vision inspection failing to distinguish between fruit and shelf defects, effectively improving the accuracy and efficiency of fruit defect detection. Through timed detection, it addresses the issue of undetected defects caused by changes in the fruit's own quality. By identifying and detecting moving fruit, the workload for detecting fruit defects is simplified, the execution speed is faster, and it is easy to implement in practice. It overcomes the problem of not being able to promptly address defective fruit when its position on the shelf changes.

Description

Technical Field

[0001] This invention relates to the field of fruit defect detection technology, specifically to a method and system for detecting defects in shelf-grown fruit based on image recognition. Background Technology

[0002] Fruits have their own nutritional characteristics and are an important part of the human diet. Before being sold, fruit manufacturers classify and grade the fruit and remove defective fruit. However, when fruit enters fruit shops or supermarkets, these stores are often too small or have a wide variety of goods to be equipped with sorting machinery. Furthermore, they lack existing systems for detecting fruit defects in these stores. Therefore, the inspection is done visually by employees, which is inefficient and doesn't benefit the store's sales. Customers may return or exchange their chosen fruit, and among the returned fruit, there may be defective pieces. Inspecting each piece is time-consuming, and individual fruits may also develop defects due to their own quality changes. If not handled promptly, the presence of one defective fruit can contaminate the entire batch, leading to significant economic losses.

[0003] Therefore, this invention proposes a method and system for detecting fruit defects on shelves based on image recognition, which can be used on shelves in fruit shops or supermarkets. It eliminates the need for large sorting machines, saves space, and effectively solves the problem of low detection efficiency of fruit defects in distribution stores. Summary of the Invention

[0004] To address the shortcomings of existing technologies, this invention provides a method and system for detecting defects in fruit on shelves based on image recognition, which solves the problem of low detection efficiency for fruit defects in retail stores.

[0005] To achieve the above objectives, the present invention provides the following technical solution: a method and system for detecting defects in shelf-grown fruit based on image recognition, comprising the following steps;

[0006] Step 1: Image Acquisition

[0007] The background image is obtained by taking pictures of the target scene with a camera.

[0008] Several target objects are placed in the target scene, and the target scene containing the target objects is photographed by a camera to obtain a panoramic image;

[0009] Step 2: Image Processing

[0010] Image segmentation is performed by comparing the background image and the panoramic image to remove background image elements from the panoramic image and obtain a foreground image containing only the target object.

[0011] Image analysis is performed on the foreground image to obtain the two-dimensional coordinate information and appearance information of several target objects;

[0012] Step 3: Defect Analysis

[0013] Convert the foreground image to RGB mode and extract the R component image;

[0014] The R component image is brightness corrected to make the brightness of the R component image uniform. Based on the difference between the gray value of the defect and the gray value of the surrounding normal area on the brightness corrected R component image, the gray image is converted into a binary surface defect image. The defect is marked on the corresponding two-dimensional coordinate information and appearance information of the defect area.

[0015] Preferably, if the target object has a defect, the controller will issue a defect signal; if the target object has no defect, the controller will not issue a signal, and the target object will be re-detected for defects at the set time interval.

[0016] Preferably, the foreground image is divided into several foreground sub-images based on the appearance information of the target object, and one foreground sub-image corresponds to one target object;

[0017] If a target object is removed, its corresponding foreground sub-image is automatically deleted, and no further defect detection is performed;

[0018] If the position of a target object changes, the foreground sub-image corresponding to it is automatically deleted, and the foreground sub-images corresponding to other target objects are combined with the background image to form a background sub-image. The camera then takes another picture of the target scene containing several target objects to obtain a panoramic sub-image.

[0019] The background sub-image and the panoramic sub-image are processed by image segmentation. The background sub-image elements in the panoramic sub-image are removed, and a new foreground sub-image containing only the changed position of the target object is obtained.

[0020] Preferably, based on the two-dimensional coordinate information and appearance information of the new foreground sub-image, defect detection is performed on the target object whose position has changed, and then the third step is used to determine whether the target object whose position has changed has any defects.

[0021] If a target object whose position has changed has a defect, the defect is marked on its corresponding two-dimensional coordinate information and appearance information, and the defect signal is prompted by the controller.

[0022] If the target object whose position has changed has no defects, the control terminal will not give any signal prompts. The new foreground sub-image obtained will be combined with the foreground sub-images corresponding to other target objects to form a new foreground image. Then, the target object whose position has changed and other target objects will be subjected to defect detection again at the set time interval.

[0023] A shelf fruit defect detection system based on image recognition is disclosed. The system specifically employs shelf fruit defect detection methods to detect defects in shelf fruit.

[0024] Beneficial effects

[0025] This invention provides a method and system for detecting defects in shelf-grown fruit based on image recognition. Compared with existing technologies, it has the following advantages:

[0026] This invention uses image segmentation to remove the background color of the shelf, obtaining an image containing only fruit. This accurately solves the problem of machine vision inspection being unable to distinguish between fruit and shelf defects, effectively improving the accuracy and efficiency of fruit defect detection. Furthermore, by using timed detection, it addresses the issue of undetected defects caused by changes in the fruit's own quality. By identifying moving fruit and combining stationary fruit with the shelf as a background, and then detecting the moving fruit, the workload for detecting fruit defects is simplified, the execution speed is faster, and it is easier to implement in practice. This overcomes the problem of not being able to promptly address defective fruit when its position on the shelf changes. Attached Figure Description

[0027] Figure 1 This is a system block diagram of an image recognition-based fruit shelf defect detection system. Detailed Implementation

[0028] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0029] Please see Figure 1 The present invention provides a technical solution:

[0030] A method for detecting defects in fruit on shelves based on image recognition, characterized by comprising the following steps;

[0031] Step 1: Image Acquisition

[0032] The background image is obtained by taking pictures of the target scene with a camera.

[0033] Several target objects are placed in the target scene, and the target scene containing the target objects is photographed by a camera to obtain a panoramic image;

[0034] Step 2: Image Processing

[0035] Image segmentation is performed by comparing the background image and the panoramic image to remove background image elements from the panoramic image and obtain a foreground image containing only the target object.

[0036] Image segmentation processing utilizes the grabcut function from the grabcut algorithm. This technology is currently in use. Its working principle involves passing the image to be segmented and the mask image as parameters to this function. Specifically, grabCut assumes the background image is the background and sets the mask to 0, while patterns different from the background image are considered foreground and have a mask of 2. Then, according to the grabCut algorithm, the regions occupied by patterns different from the background image are identified as background areas, and their masks are changed from 2 to 0. Finally, after processing, patterns in the background image whose masks remain at 2 are the identified foreground elements and are marked as foreground images.

[0037] Image analysis is performed on the foreground image to obtain the two-dimensional coordinate information and appearance information of several target objects;

[0038] The foreground image is divided into several foreground sub-images based on the appearance information of the target object, and each foreground sub-image corresponds to one target object;

[0039] Assuming there are m fruits arranged horizontally and n fruits arranged vertically on a shelf, where m and n are both greater than 0 and are integers, the image processing unit sequentially confirms the two-dimensional coordinate information of each fruit and labels them according to their coordinates as (X1,Y1)(X1,Y2), ..., (X2,Y1)(X2,Y2), ..., (Xm,Yn). Taking the foreground sub-image with two-dimensional coordinate information (Xm,Yn) as an example;

[0040] If the fruit at coordinate point (Xm,Yn) is removed, the monitoring unit, in conjunction with the coordinate acquisition unit, determines that the main color of the shelf area (Xm,Yn) changes to the original color of the shelf. The foreground sub-image of (Xm,Yn) is then deleted by the image processing unit, and no further defect analysis is performed.

[0041] If the fruit at coordinate point (Xm, Yn) is removed and then put back, the principle is the same as when the fruit at coordinate point (Xm, Yn) is removed. The subsequent monitoring unit, in conjunction with the coordinate acquisition unit, determines that the main color of the shelf area (Xm, Yn) has changed. The control unit then controls the image acquisition unit to re-acquire images of the fruit in the shelf area (Xm, Yn). The specific method is as follows:

[0042] By combining the foreground sub-image with the shelf background where the fruit coordinates remain unchanged to form the background sub-image, the camera takes another picture of the target scene containing several target objects to obtain a panoramic sub-image.

[0043] The background sub-image and the panoramic sub-image are processed by image segmentation. The background sub-image elements in the panoramic sub-image are removed, and a new foreground sub-image containing only the changed position of the target object is obtained.

[0044] Based on the two-dimensional coordinate information and appearance information of the new foreground sub-image, the target object whose position has changed is subjected to defect analysis in the third step, and it is determined whether the target object whose position has changed has any defect problems.

[0045] Step 3: Defect Analysis

[0046] Convert the foreground image or new foreground sub-image to RGB mode and extract the R component image;

[0047] The R-component image is brightness corrected to make its brightness uniform. Based on the difference between the grayscale value of the defect and the grayscale value of the surrounding normal area in the brightness-corrected R-component image, the grayscale image is converted into a binary surface defect image. The region props function in Matlab software is used to find the region coordinates of possible defect holes. This technique is well known to those skilled in the art, so it will not be described in detail here. The defect is marked on its corresponding two-dimensional coordinate information and appearance information in the defect area, and the defect signal is prompted by the controller.

[0048] If the target object is defect-free, the controller will not give any signal prompts, and the new foreground sub-image obtained will be combined with the foreground sub-images corresponding to other target objects to form a new foreground image. Then, the target object whose position has changed will be subjected to defect detection again with other target objects at the set time interval.

[0049] A shelf fruit defect detection system based on image recognition, which uses the aforementioned shelf fruit defect detection method to detect defects in shelf fruit, includes:

[0050] The monitoring unit is used to compare and analyze changes in the main color of the shelf, determine whether the main color has changed, and transmit the signal of the change in the main color of the shelf to the coordinate acquisition unit.

[0051] The coordinate acquisition unit, connected to the monitoring unit, is used to acquire the two-dimensional coordinate information of the fruit on the shelf based on the color change signal of the shelf body, and transmit it to the control unit.

[0052] The control unit is connected to the image acquisition unit, image processing unit and defect analysis unit. It is used to control the image acquisition unit to acquire images of the fruit on the shelf based on the two-dimensional coordinate information. It is also used to plan the signal prompt path based on the appearance information and two-dimensional coordinate information of the fruit on the shelf, and control the defect analysis unit to mark the defects of the fruit on the shelf according to the signal prompt path.

[0053] The image acquisition unit is used to acquire the shelf background image, panoramic image and panoramic sub-image in the above-mentioned shelf fruit defect detection method, and send them to the image processing unit connected to the image acquisition unit for processing of the acquired images;

[0054] The image processing unit is used to perform image segmentation processing on the image acquired by the image acquisition unit to obtain a foreground image containing only the fruit and several foreground sub-images, and further obtain the appearance information of the fruit. The obtained appearance information is transmitted to the defect analysis unit through the control unit.

[0055] The defect analysis unit is used to adjust the appearance information of the fruit on the shelf in RGB mode, and then analyze it to obtain the defect area.

[0056] The timing unit, connected to the control unit, is used to set the foreground image as the initial signal and output a timing detection signal to indicate the time information for the next detection;

[0057] This invention uses image segmentation to remove the background color of the shelf, obtaining an image containing only fruit. This accurately solves the problem of machine vision inspection being unable to distinguish between fruit and shelf defects, effectively improving the accuracy and efficiency of fruit defect detection. Furthermore, by using timed detection, it addresses the issue of undetected defects caused by changes in the fruit's own quality. By identifying moving fruit and combining stationary fruit with the shelf as a background, and then detecting the moving fruit, the workload for detecting fruit defects is simplified, the execution speed is faster, and it is easier to implement in practice. This overcomes the problem of not being able to promptly address defective fruit when its position on the shelf changes.

[0058] Furthermore, any content not described in detail in this specification is existing technology known to those skilled in the art.

[0059] It should be noted that, in this document, relational terms such as "first" and "second" are used only to distinguish one entity or operation from another, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Furthermore, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such process, method, article, or apparatus.

[0060] Although embodiments of the invention have been shown and described, it will be understood by those skilled in the art that various changes, modifications, substitutions and alterations can be made to these embodiments without departing from the principles and spirit of the invention, the scope of which is defined by the appended claims and their equivalents.

Claims

1. A method for detecting defects in shelf-grown fruit based on image recognition, characterized in that, Includes the following steps; Step 1: Image Acquisition The background image is obtained by taking pictures of the target scene with a camera. Several target objects are placed in the target scene, and the target scene containing the target objects is photographed by a camera to obtain a panoramic image; Step 2: Image Processing Image segmentation is performed by comparing the background image and the panoramic image to remove background image elements from the panoramic image and obtain a foreground image containing only the target object. Image analysis is performed on the foreground image to obtain the two-dimensional coordinate information and appearance information of several target objects; Step 3: Defect Analysis Convert the foreground image to RGB mode and extract the R component image; Brightness correction is performed on the R component image to make the brightness of the R component image uniform. Based on the difference between the gray value of the defect and the gray value of the surrounding normal area on the brightness-corrected R component image, the gray image is converted into a binary surface defect image. The defect is marked on the corresponding two-dimensional coordinate information and appearance information in the defect area. The foreground image is divided into several foreground sub-images based on the appearance information of the target object, and each foreground sub-image corresponds to one target object; If a target object is removed, its corresponding foreground sub-image is automatically deleted, and no further defect detection is performed; If the position of a target object changes, the foreground sub-image corresponding to it is automatically deleted, and the foreground sub-images corresponding to other target objects are combined with the background image to form a background sub-image. The camera then takes another picture of the target scene containing several target objects to obtain a panoramic sub-image. The background sub-image and the panoramic sub-image are processed by image segmentation. The background sub-image elements in the panoramic sub-image are removed, and a new foreground sub-image containing only the changed position of the target object is obtained.

2. The method for detecting defects in shelf-grown fruit based on image recognition according to claim 1, characterized in that: If the target object has defects, the controller will send a defect signal to indicate the defect. If the target object is defect-free, the controller will not issue a signal, and the target object will undergo a second defect detection at the set time interval.

3. The method for detecting defects in shelf-grown fruit based on image recognition according to claim 1, characterized in that: Based on the two-dimensional coordinate information and appearance information of the new foreground sub-image, defect detection is performed on the target object whose position has changed. Then, according to the third step, it is determined whether the target object whose position has changed has any defects.

4. A shelf-mounted fruit defect detection system based on image recognition, characterized in that, The system specifically employs the shelf fruit defect detection method as described in any one of claims 1-3 to detect defects in shelf fruit.

Images