Produce evaluation system, produce evaluation method, and produce evaluation program

Abstract

The objective of this invention is to propose a novel technique for evaluating fresh produce. [Solution] A system for evaluating fruits and vegetables, It comprises an acquisition unit, a detection unit, and an evaluation unit, The acquisition unit acquires an image of the produce using infrared light. The detection unit detects anomalies that are subject to aggregation, which are anomalies detected based on a first threshold related to the size of the anomaly in the captured image. The evaluation unit is a produce evaluation system that evaluates the produce based on the number of abnormal points to be aggregated.

Description

Technical Field

[0001] The present invention relates to a fresh produce evaluation system, a fresh produce evaluation method, and a fresh produce evaluation program for evaluating fresh produce from a captured image of fresh produce using infrared light.

Background Art

[0002] Conventionally, a technique for investigating the internal quality of fresh produce using infrared light is known. Patent Document 1 discloses a technique for investigating the internal quality of fresh produce having seeds at the center of fruits such as avocados and mangoes using near-infrared light. Specifically, a light source installed in an obliquely facing state irradiates the fruit so as to avoid the internal seeds from below a tray provided with slits, and a light receiving unit evaluates the internal quality based on spectroscopic information such as a spectrum signal by spectroscopically analyzing the transmitted light. The technique is described.

[0003] Patent Document 2 describes a technique for acquiring a captured image using transmitted light of infrared rays and ultraviolet rays, and determining the site and range of browning on the surface and inside of fresh produce by comparing the captured image of a normal product photographed in advance with the captured image of the evaluation target. Specifically, light is irradiated from a light projecting device, and a light receiving device installed around detects the transmitted light. Thereafter, detection information such as the amount of transmitted light and scattered light is stored, and the site and range of the deteriorated part are determined by comparing the stored normal detection information with the detection information of the inspection target. The technique is described.

[0004] Also, Patent Document 2 discloses a method for measuring the size and shape of deteriorated, discolored, damaged, and deteriorated parts on the fruit skin and inside using light receiving information based on a captured image of reflected light irradiated from a light source as a conventional technique.

Prior Art Documents

Patent Documents

[0005] <l

Patent Document 1

Patent Document 2

[0006] It is desirable for fruits and vegetables to maintain appropriate freshness (ripeness) during transportation and sale. Fruits and vegetables that have lost too much freshness will rot, and mold will grow inside and on the surface. Such fruits and vegetables cause disadvantages by accelerating the spoilage of fresh fruits and vegetables transported at the same time and by contributing to unnecessary waste at the time of sale.

[0007] Traditionally, the ripeness of fruits and vegetables during transportation and sales has been sorted by hand based on factors such as color and hardness, but this method relied heavily on subjective judgment and was inherently unreliable. Furthermore, manual sorting required considerable experience.

[0008] In light of the current situation described above, the present invention aims to propose a novel technology for evaluating fresh produce. [Means for solving the problem]

[0009] [1] A produce evaluation system comprising an acquisition unit, a detection unit, and an evaluation unit, wherein the acquisition unit acquires an image of produce using infrared light, the detection unit detects aggregated abnormal points among the abnormal points present in the image, which are abnormal points detected based on a first threshold relating to the magnitude of the abnormal points, and the evaluation unit evaluates the produce based on the number of aggregated abnormal points. [2] The fruit and vegetable evaluation system described in [1], wherein the fruit and vegetable is an avocado. [3] The acquisition unit acquires the captured images of the produce taken from at least two or more sides, in the produce evaluation system according to [1] or [2]. [4] The fruit and vegetable evaluation system according to any one of [1] to [3], wherein the acquisition unit acquires the captured images consisting of three different sides of the fruit and vegetable rotated by 120 degrees each. [5] A fruit and vegetable evaluation system according to any one of [1] to [4], further comprising an imaging device, the imaging device comprising a base, a light source, and an imaging unit, wherein the fruit and vegetable is placed on the base, the light source emits infrared light onto the fruit and vegetable, and the imaging unit images the side of the fruit and vegetable irradiated with infrared light by the light source. [6] The fruit and vegetable evaluation system according to any one of [1] to [5], wherein at least a portion of the base that includes the portion on which the fruit and vegetable touches the ground is rotatable in the horizontal direction, and the fruit and vegetable rotate in the horizontal direction as the portion of the base rotates. [7] The light source is a surface light source that projects infrared light onto the same side surface of the produce from above and below, the produce evaluation system according to any one of [1] to [6]. [8] A fruit and vegetable evaluation system according to any one of [1] to [7], further comprising a determination unit, wherein the determination unit places a bounding rectangle based on the endpoints of the abnormal points in the captured image, and the detection unit sets the first threshold to the length of the long side of the bounding rectangle. [9] The first threshold is set by the detection unit to prevent false detection of the abnormal point, as described in any of [1] to [8], for the fruit and vegetable evaluation system.

[10] A fruit and vegetable evaluation system according to any one of [1] to [9], further comprising a mask processing unit, wherein the mask processing unit creates a mask image from the captured image, and the detection unit detects the aggregated abnormal points in the mask image.

[11] The fruit and vegetable evaluation system described in any of [1] to

[10] , wherein the mask image is created by an image obtained by inverting the hue of an image that has been masked for its contours and an image obtained by masking for its background.

[12] A method for evaluating fruits and vegetables using a computer, comprising an acquisition step, a detection step, and an evaluation step, wherein the acquisition step acquires an image of the fruits and vegetables using infrared light, the detection step detects aggregated abnormal points among the abnormal points present in the image, which are abnormal points determined based on a first threshold relating to the size of the abnormal points, and the evaluation step evaluates the fruits and vegetables based on the number of aggregated abnormal points. A produce evaluation program that causes a computer to execute the produce evaluation method described in

[13]

[12] .

[0010] The inventions described in [1],

[12] and

[13] prevent false detections caused by the condition of the skin of fruits and vegetables or the amount of infrared light irradiated onto them, and allow evaluation to be performed by focusing on abnormal points of a specific size that contribute to the quality of fruits and vegetables.

[0011] The invention described in [2] makes it possible to evaluate avocados, which have a black skin and are difficult to detect visually, based on the presence of abnormalities.

[0012] [3] The invention makes it possible to detect abnormalities even when abnormalities are found in areas other than the side that was photographed, and to perform an appropriate evaluation.

[0013] The invention described in [4] makes it possible to detect abnormalities in the entire produce with the fewest number of captured images, enabling rapid evaluation of produce.

[0014] The invention described in [5] makes it possible to reduce the influence of other light sources and differences in mounting conditions when acquiring captured images.

[0015] The invention described in [6] makes it possible to rotate fruits and vegetables when taking multiple images of them, and to easily take images of different sides. In addition, since there is no interference between devices compared to when multiple light source devices and imaging devices are used, it is possible to reduce false detections.

[0016] The invention described in [7] makes it possible to irradiate fruits and vegetables with a uniform amount of light when imaging them, thereby reducing false detections and enabling accurate evaluation.

[0017] The invention described in [8] makes it possible to accurately determine the magnitude of anomalies and efficiently detect anomalies in the data collection.

[0018] According to the invention according to [9], it is possible to prevent false detection and detect only abnormal points of a specific size that contribute to evaluation.

[0019] According to the invention according to

[10] , it is possible to prevent false detection of abnormal points in a captured image and perform accurate evaluation.

[0020] According to the invention according to

[11] , it is possible to create an image subjected to a process of preventing false detection based on a captured image and perform accurate evaluation.

Effect of the Invention

[0021] According to the present invention, it is possible to provide a novel technique capable of detecting abnormal points of a specific size and evaluating fruits and vegetables based on the number of the abnormal points.

Brief Description of the Drawings

[0022] [Figure 1] Flow of evaluation in the present embodiment. [Figure 2] Block configuration diagram in the present embodiment. [Figure 3] Hardware configuration diagram in the present embodiment. [Figure 4] Schematic diagram of an imaging device in the present embodiment. [Figure 5] Schematic diagram of mask image generation in the present embodiment. [Figure 6] Schematic diagram of the arrangement of circumscribed rectangles of abnormal points in the present embodiment. [Figure 7] Example of screen display of evaluation results of unripe fruits and vegetables in the present embodiment. [Figure 8] Example of screen display of evaluation results of overripe fruits and vegetables in the present embodiment. [Figure 9] Flowchart in the present embodiment. [Figure 10] Transition of the number of abnormal points according to the number of days elapsed after purchasing an avocado. [Figure 11] State of change of an avocado according to the number of days elapsed in a visible light image and an infrared light image. [Modes for carrying out the invention]

[0023] In this invention, "fresh produce" refers to organs other than leaves, stems, and roots in woody and herbaceous plants that are usable as food by humans. Specifically, fresh produce includes both true fruits, which consist of the ovary, such as peaches and persimmons, and false fruits, which consist of parts other than the ovary, such as apples and strawberries. In this embodiment, we will describe an example in which avocado is used as the fresh produce.

[0024] In this invention, "abnormalities" refer to characteristics that are not found in normal produce and are unique to overripe (slow-ripening) produce. Specifically, abnormalities include blemishes on the surface of produce, browning on the surface and inside of produce, and discoloration due to disease or spoilage. Browning refers to a localized change in color caused by the destruction of cells in fruits and vegetables.

[0025] Ripeness is an indicator of whether a fruit or vegetable is ready to eat. If the fruit or vegetable has just been harvested and is not yet suitable for eating, it is considered unripe; if it is ready to eat and suitable for consumption, it is considered ripe; and if it is past its prime and showing signs of spoilage, it is considered overripe.

[0026] In this invention, infrared light refers to both visible and invisible light classified as red, comprising infrared light, near-infrared light, and far-infrared light. In this embodiment, an embodiment using near-infrared light with a wavelength of 780 nm to 2500 nm will be described, but similar effects can be achieved using light other than near-infrared light.

[0027] In this invention, a user refers to a person who uses this invention to evaluate fruits and vegetables. Specifically, it is envisioned that wholesalers, retailers, importers, and transporters involved in the production of fruits and vegetables will use this invention as users. In other embodiments, consumers may also be included as users.

[0028] <1. Overview of this embodiment> First, we will explain the evaluation flow of the produce evaluation system in this embodiment using Figure 1.

[0029] The produce evaluation system assesses whether produce is unripe, ripe, or overripe using the following procedure. (1) The user places the produce on the base inside the imaging device. (2) The imaging device has a light source device and an imaging device, and images of fruits and vegetables are captured by the various devices mounted on it. (3) Images captured by the imaging device are acquired by the evaluation device. (4) The evaluation device detects anomalies of a specific size in the images of fruits and vegetables acquired by (3). (5) Based on the number of detected anomalies, the evaluation device evaluates the imaged fruits and vegetables. (6) The evaluation results described above are output to the user terminal for the user to review.

[0030] <2. System Configuration> Further details will be described below with reference to the attached drawings. While preferred embodiments are shown in the drawings, many different forms are possible and the embodiments are not limited to those described herein.

[0031] For example, in this embodiment, the configuration and operation of the produce evaluation system 1 will be described, but similar effects can be achieved by devices having similar functions, methods performed by such devices, or computer programs that cause a computer device to perform the methods. The program may be provided as a non-transient recording medium readable by a computer, or it may be provided so that it can be downloaded from an external server.

[0032] In the following embodiments, "part" may include, for example, hardware resources implemented by a circuit in a broad sense, and information processing of software that can be specifically realized by these hardware resources. In this embodiment, "information" can be represented, for example, by the physical value of a signal value representing voltage or current, the high or low value of a signal value as a set of binary bits composed of 0s or 1s, or by a quantum superposition (so-called qubit), and communication and calculations can be performed on a circuit in a broad sense.

[0033] In a broad sense, a circuit is a set of circuits (Circuitry) that are realized by appropriately combining circuits, processors, and memory. For example, it includes circuits that include any of the following: CPU (Central Processing Unit), GPU (Graphics Processing Unit), LSI (Large Scale Integration), ASIC (Application Specific Integrated Circuit), FPGA (Field-Programmable Gate Array), etc.

[0034] Figure 2 is a block diagram representing one embodiment. As shown in Figure 2, the produce evaluation system 1 comprises an evaluation device 2, a user terminal 3, and an imaging device 4, which are configured to communicate via a network NW. Although only one user terminal 3 is shown in Figure 2, there may be multiple user terminals 3.

[0035] The evaluation device 2 acquires images captured by the imaging device 4 using the acquisition unit 10, and detects abnormal points in the acquired images using the detection unit 11. Subsequently, the evaluation unit 12 evaluates the fruits and vegetables according to the number of abnormal points detected.

[0036] The acquisition unit 10 acquires images of fruits and vegetables captured by the imaging device 4 using infrared light.

[0037] The detection unit 11 detects anomalies in the captured image acquired by the acquisition unit 10. After detecting all anomalies, the determination unit 14, described later, detects anomalies of a specific size as target anomalies based on the length of the longer side of the circumscribing rectangle it has placed.

[0038] The evaluation unit 12 counts the number of abnormal points to be aggregated from the detection results of the detection unit 11, and evaluates the ripeness of the imaged fruits and vegetables according to that number.

[0039] Here, the mask processing unit 13 creates a mask image to exclude the contour portions of the produce and the background in the captured image from the range of anomaly detection by the detection unit 11. The detection unit 11 and the evaluation unit 12 may be configured to detect anomalies and evaluate the ripeness of the produce based on the mask image.

[0040] The determination unit 14 determines the size of the abnormal point in the mask image created by the mask processing unit 13 or the captured image acquired by the acquisition unit 10, and places a bounding box around the abnormal point. Specifically, the determination unit 14 determines the endpoints of the abnormal point and places a bounding box (circumscribing rectangle) related to that abnormal point in the captured image based on those endpoints.

[0041] The output unit 15 outputs the evaluation results from the evaluation unit 12 to the user terminal 3. Alternatively, the system may be configured to output not only the evaluation results, but also the captured images acquired by the acquisition unit 10 and the mask images generated by the mask processing unit 13, along with the evaluation results, to the user terminal 3.

[0042] The user terminal 3 receives output from the output unit 15 and presents it to the user, as well as issuing imaging instructions to the imaging device 4.

[0043] The imaging device 4 captures infrared light images of fruits and vegetables placed inside it. Specifically, the light source unit 41 irradiates infrared light onto the produce placed on the base unit 40 by the user, and the imaging unit 42 acquires the reflected light emitted from the light source unit 41 and reflected by the produce, thereby capturing an infrared light image of the placed produce.

[0044] The base portion 40 is a platform for placing an avocado inside the imaging device 4. It is preferable that the base portion 40 is equipped with a motor device and that a portion of it is rotatable in the horizontal direction.

[0045] The light source unit 41 projects infrared light onto the fruits and vegetables placed on the base unit 40.

[0046] The imaging unit 42 receives the infrared light reflected from the produce from the light source unit 41, thereby imaging the produce with respect to the reflected infrared light.

[0047] In Figure 2, the evaluation device 2, user terminal 3, and imaging device 4 are connected to each other via a network NW. However, they may also be connected by wired or wireless communication without using a network NW, and some of the functional components of the evaluation device 2 may be provided in the user terminal 3, imaging device 4, or both.

[0048] Furthermore, the produce evaluation system 1 is composed of one or more information processing devices. In this embodiment, the imaging device 4 captures images of the produce, the evaluation device 2 evaluates the produce, and the user terminal 3 receives and displays the evaluation results to the user. However, the system may be composed of a single housing device equipped with information processing devices having functions equivalent to these functions.

[0049] Furthermore, the fruit and vegetable evaluation system 1 may consist of one terminal device and one housing device, in which all of the functional components of the evaluation device 2 are contained within the user terminal 3 or imaging device 4, and an information processing device is provided.

[0050] Furthermore, the produce evaluation system 1 is composed of one information processing device or a plurality of communicationable information processing devices, which will be described later. At least one of the one or more information processing devices includes an acquisition unit 10, at least one a detection unit 11, at least one an evaluation unit 12, at least one a mask processing unit 13, at least one a determination unit 14, and at least one an output unit 15.

[0051] <3. Hardware Configuration> Next, the hardware configuration of the fruit and vegetable evaluation system 1 in this embodiment will be described using Figure 3. A terminal device 200 (computer device) such as a personal computer, smartphone, or tablet can be used as the user terminal 3. In this embodiment, the evaluation device 2 is an information processing device 20 on which a computer program (fruit and vegetable evaluation program) that executes the fruit and vegetable evaluation method is installed. Alternatively, one or more information processing devices 20 (computer devices) such as a general-purpose server or personal computer can be used as the evaluation device 2. A housing device 210 can be used as the imaging device 4.

[0052] Figure 3(a) is a hardware configuration diagram of the information processing device 20. As shown in Figure 3(a), the information processing device 20 has a communication unit 21, a control unit 22, and a storage unit 23, which are used to perform the functions of each unit and each process.

[0053] The control unit 22 has a processor such as a CPU capable of executing instruction sets and runs the OS and programs. The storage unit 23 has volatile memory such as RAM capable of storing instruction sets, and non-volatile recording media such as HDDs, SSDs, and flash memory capable of storing the OS, produce evaluation programs, DBMS, etc. The communication unit 21 has an interface for physically connecting to or wirelessly communicating with the network NW, and performs communication control with the network NW to input and output information.

[0054] Figure 3(b) is a hardware configuration diagram of the terminal device 200. As shown in Figure 3(b), the terminal device 200 has a communication unit 201, a storage unit 202, an input unit 203, an output unit 204, and a control unit 205, which are used to perform the functions of each unit and each process.

[0055] The control unit 205 has a processor such as a CPU capable of executing instruction sets and executes the OS, application programs, etc. The storage unit 202 has volatile memory such as RAM capable of storing instruction sets, and non-volatile recording media such as HDDs, SSDs, flash memory, etc., capable of recording the OS, arbitrary application programs, etc. The communication unit 201 has an interface for physically connecting to or wirelessly communicating with a network NW, and performs communication control with the network NW to input and output information. The input unit 203 has an operation input device capable of input processing such as a touch panel or keyboard, and an audio input device capable of voice input such as a microphone. The output unit 204 has a display device capable of display processing such as a display, and an audio output device such as a speaker.

[0056] Figure 3(c) is a hardware configuration diagram of the housing device 210. As shown in Figure 3(c), the housing device 210 includes a communication unit 211, a storage unit 212, a drive control unit 213, a drive circuit unit 214, a drive unit 215, a camera unit 216, a lighting unit 217, and a control unit 218, which are used to enable the operation of each unit and each process.

[0057] The control unit 218 has a processor such as a CPU capable of executing instruction sets and executes the OS, application programs, etc. The storage unit 212 has volatile memory such as RAM capable of storing instruction sets, and non-volatile recording media such as HDD, SSD, flash memory, etc., capable of recording the OS, arbitrary application programs, etc. The communication unit 211 has an interface for physically connecting to or wirelessly communicating with the network NW, and performs communication control with the network NW to input and output information.

[0058] The drive control unit 213 is a control circuit that inputs control instructions such as rotational speed and stopping position to the drive unit 215. The drive circuit unit 214 is an inverter that adjusts the current and voltage input to the drive unit 215 based on the control instructions input by the drive control unit 213. The drive unit 215 is a motor device that is driven based on the adjusted current and voltage input by the drive circuit unit 214. The camera unit 216 receives reflected light from the illumination unit 217 and performs imaging. The illumination unit 217 emits light onto the object to be imaged, which is placed inside the housing device 210. In Figure 3(c), only one camera unit 216 and one illumination unit 217 are shown, but the housing device 210 may be configured to have multiple units.

[0059] <4. Imaging of fruits and vegetables> Next, we will explain how to image fruits and vegetables using the imaging device 4. Note that the base unit 40, light source unit 41, and imaging unit 42 in Figure 4 are the same as the base unit 40, light source unit 41, and imaging unit 42 in Figure 1.

[0060] Figure 4(a) is a transmission cross-sectional view of the imaging device 4. The imaging device 4 is covered by a housing 48, and imaging of fruits and vegetables is performed inside the housing 48. The housing 48 is light-shielding, so no light enters the interior during imaging.

[0061] The housing 48 is equipped with a base 40 for placing fruits and vegetables, an imaging unit 42, and a light source unit 41.

[0062] The base portion 40 has a recess at its top, and the user places fruits and vegetables 45 in the recess. The light source unit 41 emits light 46 onto the placed fruits and vegetables 45, and the imaging unit 42 receives the reflected light 47 from the light 46 emitted by the light source unit 41 that is reflected by the fruits and vegetables 45, and takes an image of the fruits and vegetables 45.

[0063] The light 46 emitted by the aforementioned light source unit 41 is infrared light, and the imaging unit 42 receives the infrared light and captures it as an IR (Infrared Ray) image. The light 46 is preferably near-infrared light, and it is preferable to project light with a wavelength of 700 nm to 2500 nm. However, the light source 41 may be configured to project not only near-infrared light, but also infrared light with a wavelength of 1 mm to 700 nm, or far-infrared light with a wavelength of 3 micrometers to 1 mm.

[0064] Furthermore, the light source unit 41 is preferably a rectangular surface light source, and the light 46 is projected by the surface light source, but it is also possible to use a ring-shaped light source or the like.

[0065] In different embodiments, the light source unit 41 may be capable of emitting ultraviolet light and visible light in addition to infrared light. In that case, the imaging unit 42 will image the fruits and vegetables 45 using ultraviolet light and visible light in addition to infrared light. The image captured using light other than infrared light will be acquired by the acquisition unit 10 and may be output to the user terminal 3 by the output unit 15 along with the configuration used for evaluation and the evaluation results described later.

[0066] Furthermore, multiple light source units 41 are provided to project light 46 onto the same side of the produce 45. In this embodiment, a configuration in which light 46 is projected from above and below the produce is described as shown in Figure 4, but a configuration in which light source units 41 are installed in the left-right direction to project light onto the same side of the produce 45, or a configuration in which light is projected onto the same side of the produce 45 from the up, down, left, and right directions, is also possible.

[0067] The imaging unit 42 receives reflected light 47 from the side of the produce 45, which is part of the light 46 emitted from the light source unit 41, and takes an image of the side of the produce 45. Since imaging is performed using infrared light, it is preferable that the imaging unit 42 uses a camera device capable of receiving infrared light, such as an IR camera. The side view images of the produce 45 captured by the imaging unit 42 are stored in the storage unit 212 and can be acquired by the acquisition unit 10 connected via the network NW. Alternatively, the captured images may be stored in the storage unit 23, and the acquisition unit 10 may acquire the captured images from the storage unit 23.

[0068] Furthermore, it is preferable that the imaging unit 42 captures images of fruits and vegetables from multiple sides, and stores the images of multiple sides of the same fruit and vegetable in the storage unit 212. In this case, the acquisition unit 10 acquires the aforementioned images of multiple sides. Specifically, the base portion 40, which is equipped with a motor device (not shown), is divided into a rotating portion 43 that includes the part on which the produce 45 makes contact with the ground, and a support portion 44 other than the rotating portion. The motor device rotates the rotating portion 43 in the horizontal direction, and as the rotating portion 43 rotates, the produce 45 rotates in the horizontal direction, and it is preferable that the imaging unit 42 is able to image multiple sides of the produce 45.

[0069] Preferably, the side view images of the produce captured by the imaging unit 42 are three images rotated 120 degrees horizontally. Specifically, the rotating unit 43 rotates 120 degrees horizontally, and the imaging unit 42 takes images of three different sides of the produce 45 and stores them in the storage unit 212. In this case, the acquisition unit 10 acquires images of the produce from three different sides rotated 120 degrees horizontally. Alternatively, the acquired images may be stored in the storage unit 23, and the acquisition unit 10 may acquire the images from the storage unit 23.

[0070] Figure 4(b) is a transmission cross-sectional view of the imaging device 4 taken along the line A-A'. The fruits and vegetables 45 and the base 40 are shown as dotted lines for illustrative purposes. The imaging unit 42 is located between two vertically arranged light source units 41. The light source units 41 irradiate the produce 45 with infrared light from above and below. The light source unit 41 is a rectangular surface light source equipped with light-emitting elements such as infrared light-emitting diodes on the surface that emits light. It is preferable that the light 46 is projected by a surface light source as shown in Figure 4(b), but a configuration using a ring-shaped light source or the like may also be used.

[0071] <5. Image processing for anomaly detection> Next, we will explain the evaluation using evaluation device 2. Images of the side view of the produce, captured by the imaging device 4 and stored in the storage unit 212, are acquired by the acquisition unit 10. If the imaging unit 42 has captured images of multiple sides of the produce, the acquisition unit 10 acquires all images of the same produce from the storage unit 212.

[0072] A mask image is created from the acquired image by the mask processing unit 13. The mask image is created through two types of processing: contour masking and anomaly detection, and background masking. Masking is a process that excludes areas from processing by the detection unit 11 and the determination unit 14 in order to suppress false detection of anomalies. Anomaly detection and other operations are not performed within the masked area.

[0073] Figure 5 is a schematic diagram of the mask image generation process in this embodiment. Image 50 is an infrared image (IR image) captured by the imaging unit 42. The mask processing unit 13 removes the background and contour portions from image 50 acquired by the acquisition unit 10, and creates image 53, which is an anomaly detection image in which only the anomalies are extracted.

[0074] Next, we will explain the process of creating image 53, which is an image used for detecting anomalies. The mask processing unit 13 performs adaptive thresholding and median filtering on the IR image as preprocessing. This reduces the effect of uneven illumination in the captured image and clarifies the contours.

[0075] Next, the procedure for masking contours and extracting anomalies by the mask processing unit 13 will be explained. The mask processing unit 13 creates a grayscale image from the pre-processed image 50 described above. Subsequently, the mask processing unit 13 calculates the gradient rate of brightness between each pixel and extracts areas with a gradient rate greater than or equal to a threshold, as well as the area enclosed by such areas.

[0076] Image 51 is an image extracted based on the brightness gradient ratio described above. Contour 511 represents the contour of the produce in Image 50, and part 512 represents an anomaly on the produce in Image 50. The mask processing unit 13 performs the above extraction process on the entire image to extract abnormal points related to the entire produce and the contours of the produce.

[0077] The mask processing unit 13 performs mask processing on the extracted contours. The aforementioned mask processing may be performed only on the portion extracted as a contour in the image 51, such as contour 511, or it may be configured so that the mask processing is applied to a predetermined width starting from the extraction point of the outermost contour on the extracted contour 511 and extending inward. Even if the extracted region of contour 511 is interrupted, the contour 511 can be determined based on the length of continuous dots, etc., and the entire contour 511 can be masked. Furthermore, since images 50 and 51 do not have background masking applied, anomalies may be falsely detected on the background. In such cases, the false detection of anomalies on the background can be eliminated by performing the background masking process described later.

[0078] Next, the procedure for masking the background by the mask processing unit 13 will be explained. First, the mask processing unit 13 converts image 50 into a black and white binary image to create image 52. Then, the mask processing unit 13 performs a masking process on the area outside the outermost contour of the contours representing fruits and vegetables in the converted binary image (the black area in image 52) to create image 52.

[0079] Image 53 is an anomaly detection image created by combining the contour-processed image 51 and the background-processed image 52, both created by the mask processing unit 13. The anomaly detection image is created by combining an image (Image 51) with the hue inverted from the contour-masked image and an image (Image 52) with the background-masked image. In Image 53, area 531 is not displayed because the contours of the produce and the background have been removed by the aforementioned masking process, and only the area corresponding to the anomaly point portion 512 extracted in Image 51 is displayed as portion 532 in Image 53.

[0080] Furthermore, the area 521 in image 52 overlaps with the area inside the outline of the fruit and vegetable to which the portion 512 in image 51 belongs. In image 53, this area is displayed in black because it is superimposed on the color-inverted image 51.

[0081] <6. Detection and Evaluation of Anomalies> Next, we will explain the procedure for detecting anomalies in the captured image. The determination unit 14 acquires image 53, which is an image for detecting anomalies created by the mask processing unit 13, and places a bounding rectangle for measuring the size of the anomalies based on the endpoints of the extracted anomalies. Specifically, the determination unit 14 places a bounding rectangle related to the abnormal point on the grayscale image based on the endpoint of the abnormal point in the acquired abnormal point detection image.

[0082] Figure 6 is a schematic diagram of the bounding rectangle positioned by the determination unit 14. In this embodiment, abnormalities are broadly classified into three types. Figure 6(a) shows the state in which the circumscribing rectangle is arranged for S-size abnormalities, Figure 6(b) for M-size abnormalities, and Figure 6(c) for L-size abnormalities, respectively, showing abnormalities observed in the same fruit or vegetable. The definitions and rationale for the contributions of each size will be explained later.

[0083] In this embodiment, a procedure for detecting M-sized abnormalities, which contribute most to the ripeness of fruits and vegetables, will be described. The detection unit 11 acquires information about the bounding rectangle related to the anomaly point located in the grayscale image. Specifically, it acquires the lengths of two perpendicularly intersecting sides of the bounding rectangle, which is represented as a rectangle.

[0084] The detection unit 11 detects M-size anomalies to be aggregated based on the length of the longer side and a preset threshold among the acquired lengths of the two sides. Specifically, the lengths of sides 62 and 63 in rectangle 61 are compared, and the length of the longer side, side 63, is compared with a threshold value to detect M-sized anomalies.

[0085] The aforementioned threshold for M size is set to prevent false detections. The abnormal points included in the abnormal point detection image created by the mask processing unit 13 include false detections caused by shadows during imaging, etc. Since the S-size defects in Figure 6(a) and the L-size defects in Figure 6(b) do not contribute to the ripeness of the produce, they are not included in the aggregation for evaluation as false detections in this embodiment.

[0086] In this embodiment, it is preferable that the threshold is configured to detect an M-size abnormality when the actual dimensions of the longest part of the abnormality present on the produce, either horizontally or vertically, are within the range of 2.4 mm to 16 mm. However, since one of the purposes of this threshold is to prevent false detections, it is not limited to a certain range as long as it suppresses false detections and contributes to both the ripeness of the produce and the absence of false detections. It should be noted that the threshold for detecting abnormalities to be aggregated for evaluation by the evaluation unit 12 will differ depending on the origin and variety of the avocado used as produce in this embodiment.

[0087] The evaluation unit 12 compiles the number of M-sized anomalies detected by the detection unit 11 and evaluates the produce based on that number. Although Figure 6 illustrates the procedure for detecting M-size anomalies on one side, it is preferable to perform M-size anomaly detection on images of three sides of the produce rotated 120 degrees each, and evaluate the produce based on the total number of detected anomalies.

[0088] Specifically, the evaluation unit 12 receives an output from the detection unit 11 indicating that it has detected an M-sized anomaly, and counts the number of times this output has been received. The evaluation of the produce is performed using this count as the number of detected M-sized anomalies. For example, an evaluation based on the number of M-size anomalies is performed such that if the total number of M-size anomalies is less than 10, it is considered immature; if it is between 10 and 120, it is considered moderately ripe; and if it is 120 or more, it is considered overripe.

[0089] The output unit 15 outputs the evaluation results from the evaluation unit 12 to the user terminal 3. At the same time, the system may also output the captured image acquired by the acquisition unit 10, and an image in which only the M-sized anomalies related to the captured image with a bounding rectangle placed on it (for example, Figure 6(b)) which has been converted to grayscale by the determination unit 14.

[0090] Figures 7 and 8 show examples of screen displays of evaluation results from the output unit 15. Figure 7 shows the evaluation results for unripe produce, and Figure 8 shows the evaluation results for overripe produce. The screen display includes a display 71 showing the evaluation results from the evaluation unit 12, an image 72 showing the image of the produce acquired by the acquisition unit 10, and an image 73 which shows the bounding rectangle related to the M-size anomaly (display 71, image 72, and image 73 are all denoted by the same reference numerals in Figures 7 and 8). The screen display from the output unit 15 may also include a form for inputting information such as the place of origin and inspection date, and operation buttons for storing information in the storage unit 23, allowing the inspection results and associated information to be stored together.

[0091] <7. Flowchart in this embodiment> Next, the flowchart in this embodiment will be explained using Figure 9.

[0092] First, the user places the produce onto the base 40 inside the imaging device (S1). Subsequently, the light source unit 41 inside the imaging device 4 projects infrared light onto the produce, and the imaging unit 42 takes images of the sides of the produce (S2). At this time, it is preferable that the rotating unit 43 provided on the base unit 40 rotates to take images of three sides of the produce rotated by 120 degrees each.

[0093] The acquisition unit 10 acquires the images captured by the imaging unit 42. If, at this time, three images of the produce are not acquired (No in S3), an instruction to take the images again is output to the imaging device 4. If three images are acquired (Yes in S3), the mask processing unit 13 masks the outlines of the fruits and vegetables and the background from the images acquired by the acquisition unit 10, and creates an anomaly detection image by combining these images (S4).

[0094] The determination unit 14 determines the endpoints of the abnormal points in the abnormal point detection image created by the mask processing unit 13 and places a bounding rectangle related to the abnormal points (S5).

[0095] The detection unit 11 obtains the lengths of two orthogonal sides of the circumscribing rectangle positioned by the determination unit 14, and compares the length of the longer side with a preset threshold. If the longer side of the bounding rectangle is within the threshold (Yes in S6), the anomaly point related to that bounding rectangle is detected as an M-size anomaly point, and the aggregation is performed by outputting a message to the evaluation unit 12 indicating that an M-size anomaly point to be aggregated has been detected (S7). If the longer side of the bounding rectangle is not within the threshold (No in S6), the abnormal point related to that bounding rectangle is not considered an abnormal point to be aggregated, and no output is sent to the evaluation unit 12 (S8).

[0096] If there are any anomalies extracted in the anomaly detection image for which processing in S6, S7, and S8 is incomplete (No in S9), processing in S6 is performed on the unprocessed anomalies. If all anomalies have been compared with a threshold (Yes in S9), the evaluation unit 12 evaluates the produce based on the number of detected M-size anomalies, and the output unit 15 outputs the evaluation result to the user terminal 3 (S10).

[0097] <Example of experiment> The inventors investigated the relationship between the number of days since purchase and the number of detected anomalies using an apparatus similar to that of evaluation apparatus 2 and imaging apparatus 4. Specifically, they detected anomalies in an image captured using imaging apparatus 4 of one side of an avocado.

[0098] Figure 10(a) is a graph showing the relationship between the total number of anomalies and the number of days elapsed since purchase. From Figure 10(a), the total number of anomalies increases as the number of days elapsed since purchase. This suggests that there is a positive correlation between the number of anomalies in infrared light-based imaging and the number of days elapsed, and that these anomalies can be a factor in determining the ripeness of avocados.

[0099] Figure 10(b) is a graph that aggregates the anomalies in Figure 10(a) by size and shows the relationship between the number of anomalies of each size and the number of days elapsed since purchase. Graph S shows the trend in the number of anomalies of size S (less than 2.4 mm), graph M shows the trend in the number of anomalies of size M (2.4 mm to 16 mm), and graph L shows the trend in the number of anomalies of size L (greater than 16 mm).

[0100] Graphs S and L show that the number of anomalies of each size did not increase as the number of days passed since purchase, exhibiting a trend without significant fluctuations (Graph S) or large changes (Graph L). In contrast, Graph M shows an increase as the number of days passed since purchase.

[0101] This suggests a positive correlation between the number of abnormalities in medium-sized avocados and the number of days elapsed, indicating that this significantly contributes to determining the ripeness of the avocados. Furthermore, the S size showed fluctuations according to the number of days elapsed, suggesting it was a false detection unrelated to the ripeness of the avocado. Similarly, the L size did not show an increase according to the number of days elapsed, suggesting it was also a false detection unrelated to ripeness.

[0102] The inventors performed the same experiment on a total of 10 avocados, and in all cases, the results were similar, showing an increase in the number of medium-sized abnormalities over time.

[0103] Figure 11 shows images of the external condition of the avocado captured with visible light (top row) and the detection results of M-sized anomalies captured using infrared light (bottom row), taken at different times since purchase. The indicator 112 in the bottom row image indicates the detected M-sized anomaly, and the indicators are similar in the images taken on day 3 and day 7. The portion 110 in the visible light image and the portion 111 in the infrared light image are mold that has grown on the surface of the avocado, respectively, and are not related to any abnormalities. Figure 11 shows that as the number of days since purchase increases and the fruit approaches over-ripening, the number of medium-sized defects increases. This increase in the number of medium-sized defects is not affected by surface foreign matter such as mold.

[0104] Based on the above, it is possible to suppress false detections by focusing on abnormalities between 2.4 mm and 16 mm in size present in images captured using infrared light, and there is a positive correlation between the number of abnormalities of this size and the number of days elapsed. Therefore, it is possible to determine the ripeness of fruits and vegetables by focusing on abnormalities between 2.4 mm and 16 mm in size.

[0105] In this embodiment, anomalies are classified into three types: S size, M size, and L size, and an evaluation is performed based on M-sized anomalies. However, this classification may have two or more types.

[0106] Specifically, the detection unit 11 may be configured to classify abnormal points on fruits and vegetables into two categories: those to be measured and those not to be measured, based on thresholds that are the same as or different from the thresholds mentioned above. In that case, the evaluation unit 12 performs an evaluation based on the number of abnormal points that are measured.

[0107] Furthermore, if the abnormalities are to be classified into four or more types, the evaluation unit 12 may be configured to perform evaluations by expanding or subdividing the thresholds to include classifications such as S size, M size, L size, and XL size, and focusing on classifications that particularly contribute to the evaluation of fruits and vegetables.

[0108] In this embodiment, the classification of anomalies is described based on their vertical or horizontal length as the magnitude of the anomaly; however, the classification may also be based on the area of ​​the anomaly.

[0109] Specifically, the system may be configured such that the determination unit 14 determines the area of ​​an anomaly point based on the number of consecutive white dots present in the anomaly point detection image, and the detection unit 11 classifies the anomaly points that contribute to the evaluation based on this area. [Explanation of Symbols]

[0110] 1. Produce Evaluation System 2. Evaluation device 10 Acquisition Department 11 Detection Unit 12 Evaluation Department 13 Mask Processing 14 Judgment section 15 Output section 3. User terminals 4. Imaging device 40 Base 41 Light source section 42 Imaging Department 20 Information Processing Devices 21 Communications Department 22 Control Unit 23 Memory section 200 terminal devices 201 Communications Department 202 Storage section 203 Input section 204 Output section 205 Control Unit 210 Enclosure device 211 Communications Department 212 Storage section 213 Drive control unit 214 Drive circuit section 215 Drive unit 216 Camera Section 217 Lighting Section 218 Control Unit 43 Rotating part 44 Support part 45 Fruits and Vegetables 46 light 47 Reflected light 48 cabinets

Claims

1. A system for evaluating fruits and vegetables, It comprises an acquisition unit, a detection unit, and an evaluation unit, The acquisition unit acquires an image of the avocado using infrared light. The detection unit detects anomalies that are to be aggregated, which are anomalies detected based on the fact that the size of the anomaly present in the captured image falls within a predetermined first numerical range. The evaluation unit is a produce evaluation system that evaluates whether the avocado is ripe when there are multiple abnormal points to be aggregated and the number of such points falls within a specified second numerical range.

2. The produce evaluation system according to Claim 1, wherein the detection unit places a bounding rectangle around the abnormal point, sets the length of the long side of the bounding rectangle as the first numerical range, and detects the abnormal point to be aggregated when the first numerical range is a numerical range corresponding to the actual size of the abnormal point present in the avocado being 2.4 mm or more and 16 mm or less.

3. The produce evaluation system according to claim 1 or 2, wherein the evaluation unit evaluates that the produce is ripe when the number of abnormal points to be aggregated is 10 or more and less than 120.

4. The produce evaluation system according to Claim 1, wherein the evaluation unit evaluates whether the quantity is unripe if it falls within a numerical range smaller than the second numerical range, ripe if it falls within the second numerical range, or overripe if it falls within a numerical range greater than the upper limit of the second numerical range.

5. The acquisition unit acquires the captured images of the avocado taken from at least two or more sides, as described in claim 1 of the fruit and vegetable evaluation system.

6. The fruit and vegetable evaluation system according to claim 5, wherein the acquisition unit acquires the image of the avocado from three different sides rotated by 120 degrees each.

7. It is further equipped with an imaging device, The imaging device comprises a base, a light source, and an imaging unit. The avocado is placed on the base, The light source unit projects infrared light onto the avocado, The imaging unit images the side of the avocado that has been irradiated with the infrared light by the light source unit. The fruit and vegetable evaluation system according to claim 1, wherein the imaging device is covered by a light-shielding housing.

8. The base portion is such that at least a part of it, including the portion on which the avocado makes contact with the ground, is rotatable in the horizontal direction. The fruit and vegetable evaluation system according to claim 7, wherein the avocado rotates in the horizontal direction as a part of the base rotates.

9. It further includes a determination unit, The determination unit places a bounding rectangle based on the endpoints of the abnormal point in the captured image, The fruit and vegetable evaluation system according to claim 1, wherein the detection unit compares the first numerical range with the length of the longer side of the circumscribing rectangle.

10. Further equipped with a mask processing unit, The mask processing unit creates a mask image from the captured image, The fruit and vegetable evaluation system according to claim 1, wherein the detection unit detects the abnormal points to be aggregated in the mask image.

11. The fruit and vegetable evaluation system according to claim 10, wherein the mask image is created by combining a contour mask image obtained by inverting the hue of an image obtained by masking a predetermined width from the extraction point located at the outermost part of the range indicating the avocado in the captured image, out of the contours extracted by the mask processing unit, toward the interior of the range indicating the avocado in the captured image, and a background mask image obtained by masking the range outside the range indicating the avocado in the captured image from the extraction point.

12. A computer-based method for evaluating fruits and vegetables, It comprises an acquisition process, a detection process, and an evaluation process. In the acquisition process described above, an image of the avocado is acquired using infrared light. In the detection step, anomalies to be aggregated are detected, which are anomalies determined based on the fact that the size of the anomaly present in the captured image falls within a specified first numerical range. The evaluation step involves evaluating whether the avocado is ripe when there are multiple abnormal points to be aggregated and the number of such points falls within a specified second numerical range.

13. A produce evaluation program that causes a computer to execute the produce evaluation method described in claim 12.

Images