Method and apparatus for detecting foreign matter contained in an inspection object

Abstract

This method for detecting a specific foreign substance contained in an inspection target includes: acquiring first image data of the inspection target, the first image data having pixels each having a pixel value of a first band group including one or more wavelength bands (S110, S210); determining, using the first image data, one or more pixel regions satisfying a first condition as a first foreign substance region (S120, S230); acquiring second image data of a region including the first foreign substance region, the second image data having pixels each having a pixel value of a second band group including more wavelength bands than the first band group (S152, S260); determining, using the second image data, one or more pixel regions satisfying a second condition different from the first condition as a second foreign substance region where the specific foreign substance exists (S154, S270); and outputting information about the second foreign substance region (S180, S300).

Description

[Technical Field]

[0001] The present disclosure relates to a method and apparatus for detecting foreign matter contained in an object to be inspected. [Background technology]

[0002] By utilizing spectral information from multiple narrow wavelength bands, e.g., ten or more bands, it becomes possible to grasp the detailed physical properties of an object, which was not possible with conventional RGB images that only have information from three bands. A camera that acquires such multi-wavelength information is called a "hyperspectral camera." Hyperspectral cameras are used in a variety of fields, including food inspection, biological testing, pharmaceutical development, and mineral component analysis. In this specification and drawings, wavelength bands may be referred to as "bands."

[0003] Patent Document 1 discloses a system and method for determining the quality of food and detecting foreign substances in food based on hyperspectral images of the food.

[0004] Patent Document 2 discloses a method for recognizing objects using machine learning from hyperspectral images generated using compressed sensing.

[0005] Patent Document 3 discloses a method for recognizing and analyzing foreign matter in an object to be measured, such as a sample, by using a microspectrometer that irradiates the object to be measured with light and measures the spectrum of transmitted light, reflected light, scattered light, or fluorescence. [Prior art documents] [Patent documents]

[0006] [Patent Document 1] Special Publication No. 2020-524326 [Patent Document 2] International Publication No. 2020 / 080045 [Patent Document 3] Patent No. 6778451 Summary of the Invention

[0007] The present disclosure provides a technique for reducing the processing load for detecting foreign matter contained in an inspection object.

[0008] A method according to one aspect of the present disclosure is a method for detecting a specific foreign substance contained in an inspection object, the method including: acquiring first image data of the inspection object, each pixel of which has a pixel value for a first band group including one or more wavelength bands; determining, from the first image data, one or more pixel regions that satisfy a first condition as a first foreign substance region; acquiring second image data of a region including the first foreign substance region, each pixel of which has a pixel value for a second band group that includes more wavelength bands than the first band group; determining, from the second image data, one or more pixel regions that satisfy a second condition different from the first condition as a second foreign substance region in which the specific foreign substance is present; and outputting information about the second foreign substance region.

[0009] A general or specific aspect of the present disclosure may be realized as a system, an apparatus, a method, an integrated circuit, a computer program, or a computer-readable recording medium, or as any combination of a system, an apparatus, a method, an integrated circuit, a computer program, and a recording medium. A computer-readable recording medium includes, for example, a non-volatile recording medium such as a CD-ROM (Compact Disc-Read Only Memory). An apparatus may be composed of one or more devices. When an apparatus is composed of two or more devices, the two or more devices may be located in a single device or may be located separately in two or more separate devices. In this specification and claims, the term "apparatus" may refer not only to a single device but also to a system consisting of multiple devices.

[0010] According to one aspect of the present disclosure, it is possible to reduce the data processing load in a system for detecting foreign matter contained in an inspection object. [Brief explanation of the drawings]

[0011] [Figure 1A] FIG. 1 is a diagram schematically illustrating an example configuration of a hyperspectral imaging system. [Figure 1B] FIG. 10 is a diagram schematically illustrating another configuration example of a hyperspectral imaging system. [Figure 1C] FIG. 10 is a diagram schematically illustrating yet another configuration example of a hyperspectral imaging system. [Figure 1D] FIG. 10 is a diagram schematically illustrating yet another configuration example of a hyperspectral imaging system. [Figure 2A] FIG. 1 is a diagram schematically illustrating an example of a filter array. [Figure 2B] FIG. 10 is a diagram illustrating an example of the spatial distribution of the transmittance of light in each of a plurality of wavelength bands included in a target wavelength range. [Figure 2C] FIG. 10 is a diagram showing an example of the spectral transmittance of one region included in the filter array. [Figure 2D] 10A and 10B are diagrams illustrating examples of the spectral transmittance of other regions included in the filter array. [Figure 3A] 1 is a diagram for explaining the relationship between a target wavelength range W and a plurality of wavelength bands W1, W2, . . . , WN included therein. [Figure 3B] 10 is another diagram for explaining the relationship between the target wavelength range W and the multiple wavelength bands W1, W2, . . . , WN included therein. FIG. [Figure 4A] FIG. 10 is a diagram for explaining the characteristics of spectral transmittance in a certain region of the filter array. [Figure 4B] 4B is a diagram showing the results of averaging the spectral transmittance shown in FIG. 4A for each of wavelength band W1, wavelength band W2, . . . , wavelength band WN. [Figure 5] FIG. 1 is a diagram schematically illustrating an example of the configuration of an inspection system according to a first embodiment. [Figure 6] FIG. 1 is a block diagram illustrating an example of the configuration of an inspection system. [Figure 7] 10 is a flowchart showing an example of a two-stage foreign object detection operation by a processing circuit. [Figure 8]10 is a flowchart showing details of the foreign object detection process under the second condition in step S150. [Figure 9] FIG. 2 is a diagram illustrating an example of information stored in a storage device. [Figure 10] FIG. 10 is a diagram showing an example of the reflectance spectra and first band group of two types of metals and a background. [Figure 11] FIG. 10 is a diagram showing an example of an image generated for each band by a restoration calculation using a restoration table corresponding to the first band group. [Figure 12] 12 is a diagram showing an example of a first foreign substance region cut out in the example shown in FIG. 11. FIG. [Figure 13] FIG. 10 is a diagram showing examples of the reflectance spectra and second band group of two types of metals and a background. [Figure 14] FIG. 10 is a block diagram showing an example of the configuration of an inspection system according to a second embodiment. [Figure 15] 10 is a flowchart showing an example of the operation of two-stage foreign object detection in the second embodiment. [Figure 16] FIG. 1 is a diagram showing coordinate axes and coordinates. DETAILED DESCRIPTION OF THE INVENTION

[0012] The embodiments described below are all comprehensive or specific examples. The numerical values, shapes, materials, components, component arrangements, positions and connection forms, steps, and step orders shown in the following embodiments are merely examples and are not intended to limit the technology of the present disclosure. Among the components in the following embodiments, components that are not described in the independent claims that represent the highest concepts are described as optional components. Each figure is a schematic diagram and is not necessarily an exact illustration. Furthermore, in each figure, substantially identical or similar components are assigned the same reference numerals. Duplicate descriptions may be omitted or simplified.

[0013] In this disclosure, all or part of a circuit, unit, device, component, or part, or all or part of a functional block in a block diagram, may be implemented by one or more electronic circuits, including, for example, a semiconductor device, a semiconductor integrated circuit (IC), or an LSI (large scale integration). An LSI or IC may be integrated on a single chip or may be configured by combining multiple chips. For example, functional blocks other than memory elements may be integrated on a single chip. While the terms LSI and IC are used here, the term may be changed depending on the degree of integration, and may be referred to as a system LSI, a VLSI (very large scale integration), or an ULSI (ultra large scale integration). A field programmable gate array (FPGA), which is programmable after LSI fabrication, or a reconfigurable logic device, which can reconfigure connections within an LSI or set up circuit partitions within an LSI, may also be used for the same purpose.

[0014] Furthermore, all or part of the functions or operations of a circuit, unit, device, component, or section can be implemented by software processing. In this case, the software is recorded on one or more non-transitory recording media such as ROMs, optical disks, hard disk drives, etc., and when the software is executed by a processor, the functions specified in the software are performed by the processor and peripheral devices. A system or device may include one or more non-transitory recording media on which the software is recorded, a processor, and necessary hardware devices, such as interfaces.

[0015] First, an overview of the embodiment of the present disclosure will be described.

[0016] A method according to an embodiment of the present disclosure is a method for detecting a specific foreign substance contained in an inspection object, the method including: (a) acquiring first image data of the inspection object, each pixel having a pixel value for a first band set including one or more wavelength bands; (b) determining, from the first image data, one or more pixel regions that satisfy a first condition as a first foreign substance region; (c) acquiring second image data of a region including the first foreign substance region, each pixel having a pixel value for a second band set that includes more wavelength bands than the first band set; (d) determining, from the second image data, one or more pixel regions that satisfy a second condition different from the first condition as a second foreign substance region in which the specific foreign substance is present; and (e) outputting information regarding the second foreign substance region.

[0017] The above method is executed by a computer. According to the above method, one or more first foreign substance regions that satisfy a first condition are first determined from first image data containing information on a relatively small number of bands. The first foreign substance region may be a region estimated to be highly likely to contain a specific foreign substance. Once the first foreign substance region is determined, second image data containing information on more bands is acquired for the region containing the first foreign substance region. From the second image data, one or more pixel regions that satisfy the second condition are determined as second foreign substance regions containing a specific foreign substance. Through this operation, in many cases where a specific foreign substance is not present, the inspection is completed based on a determination made only on images of a relatively small number of bands. Only when there is a possibility that a specific foreign substance is present is the inspection narrowed down to a smaller region and performed based on images of a larger number of bands. This two-stage foreign substance inspection reduces the computational load and significantly shortens the time required for the entire inspection process.

[0018] The first condition may be that the pixel region is composed of multiple consecutive pixels whose pixel values ​​for the first band group satisfy a specific condition, and that the size of the pixel region exceeds a predetermined size. For example, if the first band group includes one band, a pixel region consisting of multiple consecutive pixels whose pixel values ​​are within a predetermined range may be detected as a first foreign body region if the group of consecutive pixels in the band exceeds a predetermined size. Alternatively, if the first band group includes two bands, a pixel region consisting of multiple pixels may be detected as a first foreign body region if the ratio or difference between the pixel values ​​of the two bands exceeds a predetermined size. "Contiguous" pixels means that the pixels are adjacent or close to each other in the image. The "predetermined size" may be, for example, a threshold value related to the number of pixels in the pixel region, the diameter of the circumscribing circle, or the diameter of the inscribing circle.

[0019] The second condition may be that the pixel region is classified into one of a predetermined classification list based on pixel values ​​of the second band group. For example, if a combination (e.g., ratio) of pixel values ​​of multiple bands included in the second band group satisfies a predetermined condition, it may be determined that the pixel region is classified into one of the predetermined classification list. The classification may be performed according to a trained model that has been trained in advance using training data.

[0020] Acquiring the first image data may include acquiring compressed image data in which image information for each of a plurality of wavelength bands including the second band group is compressed into a single two-dimensional image, and generating the first image data from the compressed image data. Acquiring the second image data may include extracting the region including the first foreign substance region from the compressed image data, and generating the second image data based on data of the extracted region. The region including the first foreign substance region may be the same region as the first foreign substance region.

[0021] Generating the first image data may include restoring the first image data from the compressed image data using a first restoration table corresponding to the first band group. Generating the second image data may include restoring the second image data from the data of the extracted region using a second restoration table corresponding to the second band group. This configuration allows the first image data and the second image data to be restored with higher accuracy.

[0022] The compressed image may be generated by an imaging device including a filter array and an image sensor. The filter array may include multiple types of filters having different transmission spectra. The first restoration table and the second restoration table may be generated based on the distribution of the transmission spectra in the multiple filter arrays.

[0023] The first image data may be acquired by a first imaging operation by an imaging device, and the second image data may be acquired by a second imaging operation by the imaging device. In this case, the imaging device is not limited to an imaging device including the filter array described above, but may be any hyperspectral imaging device.

[0024] The method may further include outputting a warning to an output device when the second foreign object region is detected. The output device may be, for example, one or more devices selected from the group consisting of a display, a speaker, a buzzer, and a lamp. The warning may include, for example, one or more pieces of information selected from the group consisting of light, sound, an image, text, and vibration.

[0025] The method may further include storing the positions of the first foreign substance region and the second foreign substance region in a storage device, where the positions are positions in an image and can be specified by two-dimensional coordinate values.

[0026] According to another embodiment of the present disclosure, an apparatus for detecting foreign matter contained in an inspection object includes a processor and a storage medium storing a computer program. The processor executes the computer program to (a) acquire first image data of the inspection object, in which each pixel has a pixel value for a first band group including one or more wavelength bands; (b) determine, from the first image data, one or more pixel regions that satisfy a first condition as a first foreign matter region; (c) acquire second image data of a region including the first foreign matter region, in which each pixel has a pixel value for a second band group that includes more wavelength bands than the first band group; (d) determine, from the second image data, one or more pixel regions that satisfy a second condition different from the first condition as a second foreign matter region in which the specific foreign matter exists; and (e) output information regarding the second foreign matter region.

[0027] According to yet another embodiment of the present disclosure, there is provided a computer program for detecting foreign matter contained in an inspection object, the computer program causing a computer to (a) acquire first image data of the inspection object, in which each pixel has a pixel value for a first band group including one or more wavelength bands, (b) determine from the first image data one or more pixel regions that satisfy a first condition as a first foreign matter region, (c) acquire second image data of a region including the first foreign matter region, in which each pixel has a pixel value for a second band group that includes more wavelength bands than the first band group, (d) determine from the second image data one or more pixel regions that satisfy a second condition different from the first condition as a second foreign matter region in which the specific foreign matter is present, and (e) output information regarding the second foreign matter region.

[0028] Exemplary embodiments of the present disclosure will be described in more detail below.

[0029] (Embodiment 1) First, a configuration example of a hyperspectral imaging system used in the first embodiment of the present disclosure will be described, followed by a description of an inspection system that uses the hyperspectral imaging system.

[0030] <Hyperspectral imaging system> FIG. 1A is a diagram schematically illustrating an example configuration of a hyperspectral imaging system. This system includes an imaging device 100 and a processing device 200. The imaging device 100 has a configuration similar to that of the imaging device disclosed in Patent Document 2. The imaging device 100 includes an optical system 140, a filter array 110, and an image sensor 160. The optical system 140 and the filter array 110 are disposed on the optical path of light incident from an object 70, which is the subject of the imaging. The filter array 110 is disposed between the optical system 140 and the image sensor 160.

[0031] FIG. 1A illustrates an apple as an example of the object 70. The object 70 is not limited to an apple, but may be any object that can be an inspection target. The image sensor 160 generates compressed image 120 data in which information of multiple wavelength bands is compressed as a two-dimensional monochrome image. The processing device 200 generates image data for each of the multiple wavelength bands included in the target wavelength range based on the compressed image 120 data generated by the image sensor 160. This generated image data of the multiple wavelength bands is referred to in this specification as "hyperspectral image data." Here, the number of wavelength bands included in the target wavelength range is N (N is an integer equal to or greater than 4). In the following description, the generated image data of the multiple wavelength bands is referred to as hyperspectral images 250W1, 250W2, ..., 250W N These are collectively referred to as a hyperspectral image 250. In this specification, a signal representing an image, that is, a set of signals representing the pixel values ​​of each pixel, may be simply referred to as an "image."

[0032] The filter array 110 is an array of multiple light-transmitting filters arranged in rows and columns. The multiple filters include multiple types of filters with different spectral transmittances, i.e., wavelength-dependence of light transmittance. The filter array 110 modulates the intensity of incident light for each wavelength and outputs the modulated light. This process performed by the filter array 110 is referred to as "encoding" in this specification.

[0033] 1A, the filter array 110 is disposed near or directly above the image sensor 160. Here, "near" means close enough that a reasonably clear image of light from the optical system 140 is formed on the surface of the filter array 110. "Directly above" means that the two are so close that there is almost no gap between them. The filter array 110 and the image sensor 160 may be integrated.

[0034] 1A, optical system 140 includes at least one lens. Although optical system 140 is shown as a single lens in FIG. 1A, optical system 140 may be a combination of multiple lenses. Optical system 140 forms an image on the imaging surface of image sensor 160 through filter array 10.

[0035] The filter array 110 may be located away from the image sensor 160. FIGS. 1B to 1D are diagrams showing configuration examples of the imaging device 100 in which the filter array 110 is located away from the image sensor 160. In the example of FIG. 1B, the filter array 110 is located between the optical system 140 and the image sensor 160 and at a position away from the image sensor 160. In the example of FIG. 1C, the filter array 110 is located between the object 70 and the optical system 140. In the example of FIG. 1D, the imaging device 100 includes two optical systems 140A and 140B, and the filter array 110 is located between them. As in these examples, an optical system including one or more lenses may be located between the filter array 110 and the image sensor 160. In FIGS. 1A to 1D, an image corresponding to wavelength band W1, i.e., an image of wavelength band W1, is represented by 250W1, ~, wavelength band W NThe image corresponding to the wavelength band W N The image is 250W N is.

[0036] The image sensor 160 is a monochrome photodetector having a plurality of photodetecting elements (also referred to herein as "pixels") arranged two-dimensionally. The image sensor 160 may be, for example, a charge-coupled device (CCD), a complementary metal oxide semiconductor (CMOS), or an infrared array sensor. The photodetecting elements include, for example, photodiodes. The image sensor 160 does not necessarily have to be a monochrome sensor. For example, a color sensor having R / G / B, R / G / B / IR, or R / G / B / W filters may be used. Using a color sensor can increase the amount of wavelength-related information and improve the accuracy of generating the hyperspectral image 250. The wavelength range to be acquired may be determined arbitrarily and may be not limited to the visible wavelength range but may also be ultraviolet, near-infrared, mid-infrared, or far-infrared.

[0037] The processing device 200 is a computer including a processor and a storage medium such as a memory. The processing device 200 generates a plurality of images 250W1, 250W2, ..., 250W3, each including information of a plurality of wavelength bands, based on the compressed image 120 acquired by the image sensor 160. N Generate data.

[0038] FIG. 2A is a diagram schematically illustrating an example of a filter array 110. The filter array 110 has a plurality of regions arranged two-dimensionally. In this specification, these regions may be referred to as "cells." An optical filter having an individually set spectral transmittance is disposed in each region. The spectral transmittance is expressed by a function T(λ), where λ is the wavelength of incident light. The spectral transmittance T(λ) can take a value between 0 and 1.

[0039] 2A, the filter array 110 has 48 rectangular regions arranged in 6 rows and 8 columns. This is merely an example, and in actual applications, more regions may be provided. The number of regions may be approximately the same as the number of pixels in the image sensor 160, for example. The number of filters included in the filter array 110 is determined depending on the application and may range from several tens to several tens of millions, for example.

[0040] FIG. 2B shows multiple wavelength bands W1, W2, . . . , W included in the target wavelength range. N 2B is a diagram showing an example of the spatial distribution of light transmittance for each wavelength band. In the example shown in FIG. 2B, the difference in shading in each region represents the difference in transmittance. The lighter the region, the higher the transmittance, and the darker the region, the lower the transmittance. As shown in FIG. 2B, the spatial distribution of light transmittance differs depending on the wavelength band.

[0041] 2C and 2D are diagrams showing examples of the spectral transmittance of region A1 and region A2 included in the filter array 110 shown in FIG. 2A, respectively. The spectral transmittance of region A1 and the spectral transmittance of region A2 are different from each other. In this way, the spectral transmittance of the filter array 110 varies depending on the region. However, it is not necessary for all regions to have different spectral transmittances. In the filter array 110, the spectral transmittances of at least some of the multiple regions are different from each other. The filter array 110 includes two or more filters with different spectral transmittances. In one example, the number of spectral transmittance patterns of the multiple regions included in the filter array 110 may be equal to or greater than the number N of wavelength bands included in the target wavelength range. The filter array 110 may be designed so that the spectral transmittances of more than half of the regions are different.

[0042] 3A and 3B show a target wavelength range W and multiple wavelength bands W1, W2, . . . , W NThis is a diagram for explaining the relationship between the wavelengths of the visible light and the ultraviolet light. The target wavelength range W can be set to various ranges depending on the application. The target wavelength range W can be, for example, the visible light wavelength range of about 400 nm to about 700 nm, the near-infrared wavelength range of about 700 nm to about 2500 nm, or the near-ultraviolet wavelength range of about 10 nm to about 400 nm. Alternatively, the target wavelength range W can be a radio wave range such as mid-infrared or far-infrared. In this specification, the wavelength range used is not limited to the visible light range. In this specification, the term "light" refers to radiation in general, including not only visible light but also infrared and ultraviolet light.

[0043] In the example shown in FIG. 3A, N is an arbitrary integer equal to or greater than 4, and the target wavelength range W is divided into N equal wavelength ranges, each of which is designated as wavelength band W1, wavelength band W2, . . . , wavelength band W N However, the present invention is not limited to this example. The multiple wavelength bands included in the target wavelength range W may be set arbitrarily. For example, the bandwidth may be non-uniform depending on the wavelength band. There may be a gap or overlap between adjacent wavelength bands. In the example shown in FIG. 3B, the bandwidth differs depending on the wavelength band, and there is a gap between two adjacent wavelength bands. In this way, the multiple wavelength bands may be determined arbitrarily as long as they are different from each other.

[0044] 4A is a diagram illustrating the characteristics of the spectral transmittance in a certain region of the filter array 110. In the example shown in FIG. 4A, the spectral transmittance has multiple maximum values ​​P1 to P5 and multiple minimum values ​​for wavelengths in the target wavelength band W. In the example shown in FIG. 4A, the optical transmittance in the target wavelength band W is normalized so that the maximum value is 1 and the minimum value is 0. In the example shown in FIG. 4A, the optical transmittance in wavelength band W2 and wavelength band W3 is normalized so that the maximum value is 1 and the minimum value is 0. N-1 In this way, the spectral transmittance of each region is expressed as a multiple of wavelength bands W1 to W2. N In the example of Fig. 4A, the maximum values ​​P1, P3, P4, and P5 are 0.5 or more.

[0045] As described above, the light transmittance of each region varies with wavelength. Therefore, the filter array 110 transmits a larger proportion of the components in a certain wavelength range of the incident light and less of the components in other wavelength ranges. For example, for the light in k of the N wavelength bands, the transmittance may be greater than 0.5, and for the light in the remaining N - k wavelength ranges, the transmittance may be less than 0.5. k is an integer that satisfies 2 ≤ k < N. If the incident light is white light that evenly contains all visible light wavelength components, the filter array 110 modulates the incident light into light having a plurality of discrete intensity peaks with respect to wavelength for each region, and superimposes and outputs these multi-wavelength lights.

[0046] FIG. 4B shows, as an example, the spectral transmittance shown in FIG. 4A averaged for each of the wavelength bands W1, wavelength band W2, ···, wavelength band W N This is a diagram showing the result. The averaged transmittance is obtained by integrating the spectral transmittance T(λ) for each wavelength band and dividing by the bandwidth of that wavelength band. In this specification, the value of the transmittance averaged for each wavelength band in this way is defined as the transmittance in that wavelength band. In this example, the transmittance is prominently high in three wavelength ranges with maximum values P1, P3, and P5. In particular, in the two wavelength ranges with maximum values P3 and P5, the transmittance exceeds 0.8.

[0047] In the example shown in FIGS. 2A to 2D, a grayscale transmittance distribution is assumed in which the transmittance of each region can take any value between 0 and 1. However, it is not necessarily a grayscale transmittance distribution. For example, a binary scale transmittance distribution in which the transmittance of each region can take either a value close to 0 or a value close to 1 may be adopted. In the binary scale transmittance distribution, each region transmits most of the light in at least two of the plurality of wavelength ranges included in the target wavelength range and does not transmit most of the light in the remaining wavelength ranges. Here, "most" generally refers to 80% or more.

[0048] A portion of the cells, for example half of the cells, may be replaced with a transparent region. Such a transparent region may cover all wavelength bands W1 to W2 included in the wavelength range W of interest. N The filter array 110 transmits light of various wavelengths at a similarly high transmittance, for example, 80% or more. In such a configuration, the transparent regions may be arranged, for example, in a checkerboard pattern. That is, in two arrangement directions of the regions in the filter array 110, regions whose light transmittance varies depending on the wavelength and transparent regions may be arranged alternately.

[0049] Such data indicating the spatial distribution of the spectral transmittance of the filter array 110 is acquired in advance based on design data or actual measurement calibration, and is stored in a storage medium provided in the processing device 200. This data is used in the calculation processing described below.

[0050] The filter array 110 can be constructed using, for example, a multilayer film, an organic material, a diffraction grating structure, or a microstructure containing metal. When a multilayer film is used, for example, a dielectric multilayer film or a multilayer film containing a metal layer can be used. In this case, at least one of the thickness, material, and stacking order of each multilayer film is different for each cell. This allows different spectral characteristics to be achieved for each cell. The use of a multilayer film allows for sharp rises and falls in the spectral transmittance. A configuration using organic materials can be achieved by using different pigments or dyes in each cell or by stacking different materials. A configuration using a diffraction grating structure can be achieved by providing a diffraction structure with a different diffraction pitch or depth for each cell. When a microstructure containing metal is used, it can be fabricated using plasmon effect-based spectral analysis.

[0051] Next, an example of signal processing by the processing device 200 will be described. The processing device 200 generates a multi-wavelength hyperspectral image 250 based on the compressed image 120 output from the image sensor 160 and the spatial distribution characteristics of the transmittance for each wavelength of the filter array 110. Here, "multi-wavelength" refers to a wavelength range that is greater than the wavelength ranges of the three colors RGB captured by a typical color camera. The number of wavelength ranges can be, for example, between 4 and 100. This number of wavelength ranges is referred to as the number of bands. Depending on the application, the number of bands may exceed 100.

[0052] Before describing the "two-stage foreign substance inspection operation" of the present disclosure, a method for generating data for a hyperspectral image 250 from a compressed image 120 acquired by an image sensor 160 will be described. The data for the hyperspectral image 250 is represented by f. If the number of bands is N, f is the image data f1 of wavelength band W1, image data f2 of wavelength band W2, ..., wavelength band W N Image data of N Here, as shown in Figure 1A, the horizontal direction of the image is the x direction, and the vertical direction of the image is the y direction. If the number of pixels in the x direction of the image data to be obtained is n, and the number of pixels in the y direction is m, then image data f1, image data f2, ..., image data f N Each of the elements f is two-dimensional data with n×m pixels. Therefore, data f is three-dimensional data with n×m×N elements. This three-dimensional data is called "hyperspectral image data" or "hyperspectral data cube." Meanwhile, data g of compressed image 120 obtained by encoding and multiplexing using filter array 110 has n×m elements. Data g can be expressed by the following equation (1).

[0053]

number

[0054] where f1, f2, . . . , f NEach of these is data with n × m elements. Therefore, the vector on the right side is a one-dimensional vector with n × m × N rows and 1 column. The vector g is converted into a one-dimensional vector with n × m rows and 1 column, and then expressed and calculated. The matrix H is the sum of the components f1, f2, ..., f of the vector f. N is encoded and intensity-modulated with different encoding information (hereinafter also referred to as "mask information") for each wavelength band, and then added together. Therefore, H is a matrix with n × m rows and n × m × N columns.

[0055] Given a vector g and a matrix H, it seems possible to calculate f by solving the inverse problem of equation (1). However, because the number of elements n×m×N of the desired data f is greater than the number of elements n×m of the acquired data g, this problem is ill-posed and cannot be solved as is. Therefore, the processing device 200 utilizes the image redundancy contained in the data f to find a solution using a compressed sensing technique. Specifically, the desired data f is estimated by solving the following equation (2).

[0056]

number

[0057] Here, f' represents the estimated data of f. The first term in the parentheses in the above equation represents the amount of deviation between the estimation result Hf and the acquired data g, the so-called residual term. Here, the sum of squares is used as the residual term, but the absolute value or the square root of the sum of squares, etc., may also be used as the residual term. The second term in the parentheses is a regularization term or stabilization term. Equation (2) means to find f that minimizes the sum of the first and second terms. The processing device 200 can converge the solution by recursive iterative calculations and calculate the final solution f'.

[0058] The first term in the parentheses in equation (2) represents the sum of squares of the difference between the acquired data g and Hf, which is the result of transforming the estimation process f by the matrix H. The second term, Φ(f), is a constraint for regularizing f and is a function that reflects the sparsity information of the estimation data. This function has the effect of smoothing or stabilizing the estimation data. The regularization term can be expressed, for example, by the discrete cosine transform (DCT) of f, the wavelet transform, the Fourier transform, or the total variation (TV) of f. For example, using the total variation transform can obtain stable estimation data that suppresses the influence of noise in the observation data g. The sparsity of the object 70 in the space of each regularization term varies depending on the texture of the object 70. A regularization term that makes the texture of the object 70 sparser in the regularization term space can be selected. Alternatively, multiple regularization terms can be included in the calculation. τ is a weighting coefficient. The larger the weighting coefficient τ, the greater the amount of redundant data reduction and the higher the compression rate. The smaller the weighting factor τ, the weaker the convergence to a solution. The weighting factor τ is set to an appropriate value that allows f to converge to a certain extent but does not result in over-compression.

[0059] In the configurations of FIGS. 1B and 1C , the image encoded by the filter array 110 is acquired in a blurred state on the imaging plane of the image sensor 160. Therefore, by storing this blur information in advance and reflecting the blur information in the aforementioned matrix H, a hyperspectral image 250 can be generated. Here, the blur information is expressed by a point spread function (PSF). The PSF is a function that defines the degree of spread of a point image to surrounding pixels. For example, if a point image corresponding to a pixel on an image spreads to a k×k pixel region around the pixel due to blurring, the PSF can be defined as a group of coefficients, i.e., a matrix, that indicates the effect on the brightness of each pixel within that region. The hyperspectral image 250 can be generated by reflecting the effect of blurring of the encoding pattern by the PSF in the matrix H. The filter array 110 may be positioned at any position, but a position that does not cause the encoding pattern of the filter array 110 to be lost due to excessive diffusion can be selected.

[0060] Through the above processing, a hyperspectral image 250 can be generated from the compressed image 120 acquired by the image sensor 160. In the above example, the processing device 200 applies an algorithm utilizing the principle of compressed sensing to all wavelength bands included in the target wavelength range to generate the hyperspectral image 250. In this case, if the resolution of the compressed image 120 is high, the computational load for generating the hyperspectral image 250 increases, and the time required for inspection increases.

[0061] Therefore, in this embodiment, the computational load and time required for the entire inspection are reduced by performing the following two-stage restoration and inspection. In the first stage, the restoration calculation is performed only on a first band group including a relatively small number of bands, rather than on all bands, to generate images of each band from the compressed image. Based on the images of the relatively small number of bands, a first foreign substance region in the image where a specific foreign substance is likely to exist is identified. In the second stage, the restoration calculation is performed only on a relatively small region including the identified first foreign substance region, on a second band group including more bands than the first band group. This identifies a second foreign substance region in which the specific foreign substance to be detected exists from within the first foreign substance region. Information about the identified second foreign substance region is output to an output device such as a display.

[0062] This method makes it possible to detect specific foreign particles contained in an object to be inspected with a smaller amount of calculation, thereby significantly reducing the time required for the inspection process.

[0063] An example of the configuration and operation of an inspection system based on the above method will now be described in more detail.

[0064] <Inspection system> 5 is a diagram schematically illustrating an example of the configuration of an inspection system according to this embodiment. The inspection system includes an imaging device 100, a processing device 200, an output device 300, and an actuation device 400. The output device 300 may include devices such as a display, a speaker, and a lamp. The actuation device 400 may include devices such as a belt conveyor and a picking device.

[0065] The object 70 to be inspected is placed on a conveyor belt and transported. The object 70 may be any item, such as an industrial product or food. The inspection system detects foreign objects mixed in the object 70 based on a compressed image of the object 70. The detected foreign object may be any object, such as a specific metal, plastic, insect, dust, or hair. The foreign object is not limited to these objects and may also be a part of the object 70 where quality has deteriorated. For example, if the object 70 is food, a spoiled part of the food may be detected as a foreign object. When the inspection system detects a foreign object, it can output information indicating that a foreign object has been detected to the output device 300, or remove the object 70 containing the foreign object using a picking device.

[0066] The imaging device 100 is a camera capable of the aforementioned hyperspectral imaging. The imaging device 100 generates the aforementioned compressed image by capturing an image of the object 70 continuously moving on a conveyor. The processing device 200 is any computer, such as a personal computer, a server computer, or a laptop computer. The processing device 200 generates images for each of a plurality of bands by performing the aforementioned restoration calculation based on the compressed image generated by the imaging device 100. The processing device 200 detects foreign objects contained in the object 70 based on the images of those bands and outputs the detection results to the output device 300.

[0067] 6 is a block diagram showing an example configuration of an inspection system. The processing device 200 includes a processing circuit 210 and a storage device 220. The output device 300 includes a display 310, a speaker 320, and a lamp 330. The operating device 400 includes a conveyor 410 and a picking device 420.

[0068] 1A to 1D, the imaging device 100 includes an image sensor, a filter array, and an optical system such as a lens. The imaging device 100 captures an image of an object 70, generates compressed image data, and sends the compressed image data to the processing device 200.

[0069] The processing device 200 generates an image for each band based on the compressed image generated by the imaging device 100. The processing circuit 210 of the processing device 200 includes a processor such as a CPU or a GPU. The processing circuit 210 determines whether or not a specific foreign object is included in the target object 70 based on the compressed image generated by the imaging device 100, and outputs information indicating the determination result.

[0070] The processing circuit 210 performs a two-stage restoration process on the compressed image acquired from the imaging device 100. In the first restoration stage, restoration is performed on only a relatively small number of bands included in the target wavelength range for the entire compressed image. This relatively small number of bands is referred to as a first band group. The number of bands included in the first band group is any number equal to or greater than one, and in some examples, is between two and five. In this restoration process, the processing circuit 210 uses a first restoration table that contains only information on matrix elements corresponding to the first band group from the matrix H in the above-described equations (1) and (2). In this case, a composite restoration table that combines information on matrix elements corresponding to bands other than the bands included in the first band group may be used to generate a composite restoration image corresponding to bands other than the bands included in the first band group. The processing circuit 210 restores the image of each band in the first band group based on the compressed image and the first restoration table in accordance with the above-described equation (2). The processing circuit 210 first identifies a first foreign substance region, which is likely to contain a foreign substance, based on the pixel values ​​of multiple pixels included in the restored image corresponding to each band, according to a discrimination model based on a first condition. The first condition may be, for example, that the pixel region is composed of multiple consecutive pixels whose pixel values ​​for the first band group satisfy a specific condition and that the size of the pixel region exceeds a predetermined size. The processing circuit 210 stores the first foreign substance region in the storage device 220 and performs restoration for each band of the second band group, which is greater than the number of bands in the first band group, in a relatively narrow region including the first foreign substance region. In this restoration process, the processing circuit 210 uses a second restoration table that contains only information on matrix elements corresponding to the second band group, out of the matrix H in the above-mentioned equations (1) and (2). The second band group may include all bands in the target wavelength range. Based on the compressed image and the second restoration table, the processing circuit 210 calculates pixel values ​​of multiple pixels in the region corresponding to the first foreign substance region included in the image of each band in the second band group according to the above-mentioned equation (2).The processing circuit 210 identifies a second foreign substance region containing a specific foreign substance based on the pixel values ​​of multiple pixels in a region corresponding to the first foreign substance region included in each band image according to a discrimination model based on a second condition, and stores information indicating the identified second foreign substance region as a detection result in the storage device 220. The second condition may be, for example, that the region is classified into one of a predetermined classification list based on the pixel values ​​of the second band group. When a foreign substance satisfying the second condition is detected, the processing circuit 210 transmits a control signal to the external output device 300 and the actuation device 400. In response to the received control signal, the output device 300 outputs a warning using light, images, text, a beep, or sound from at least one of the display 310, speaker 320, and lamp 330. In response to the received control signal, the actuation device 400 may switch the path of the conveyor 410 or remove the object 70 in which the picking device 420 detected the foreign substance from the conveyor 410.

[0071] The storage device 220 includes any storage medium, such as a semiconductor memory, a magnetic storage device, an optical storage device, etc. The storage device 220 stores computer programs executed by the processing circuit 210, data used by the processing circuit 210 in the process, and data generated by the processing circuit 210 in the process. The storage device 220 stores, for example, compressed image data generated by the imaging device 100, restoration tables corresponding to each combination of bands such as the first band group and the second band group, discriminant models corresponding to each combination of bands, information indicating the positions on the image of the detected first foreign substance region and second foreign substance region, and information indicating the foreign substance determination results.

[0072] FIG. 7 is a flowchart showing an example of a two-stage foreign object detection operation by the processing circuit 210. In this example, the processing circuit 210 executes the operations of steps S100 to S180. First, the processing circuit 210 acquires a compressed image of the object 70 generated by the imaging device 100 using compressed sensing imaging (step S100). Next, the processing circuit 210 performs a restoration calculation based on the above-described equation (2) using a first restoration table corresponding to the first band group, and generates one or more images of the object 70 for the first band group from the compressed image (step S110). The processing circuit 210 applies a first discriminant model based on a first condition to the generated one or more images, and detects a first foreign object region that satisfies the first condition (step S120). Here, one or more types of foreign objects are detected according to the first discriminant model. The processing circuit 210 may detect two or more different types of foreign objects according to the first discriminant model. The processing circuit 210 determines whether a first foreign substance region that satisfies the first condition exists (step S130). If a first foreign substance region exists, the process proceeds to step S140. If a first foreign substance region does not exist, the process ends. The processing circuit 210 stores information indicating the detected first foreign substance region in the storage device 220 (step S140). Next, the processing circuit 210 performs foreign substance detection processing based on two conditions for the detected first foreign substance region, based on information from a second band group that includes more bands than the first band group (step S150). The number of types of foreign substances that satisfy the second condition detected here is one or more, and may be multiple. The number of types of foreign substances that satisfy the second condition is fewer than the number of types of foreign substances that satisfy the first condition. The number of types of foreign substances that satisfy the first condition but do not satisfy the second condition may be one or multiple. The processing circuit 210 determines whether a second foreign substance region that satisfies the second condition exists (step S160). If a second foreign substance region exists, the process proceeds to step S109. If the second foreign object region does not exist, the process ends. If the second foreign object region exists, the processing circuit 210 stores information about the second foreign object region in the storage device 220 (step S170). Then, the processing circuit 210 outputs a control signal to the output device 300 and the actuation device 400 (step S180).The output device 300 receives a control signal from the processing circuit 210 and outputs a warning display on the display 310, etc. The actuation device 400 receives a control signal from the processing circuit 210 and controls the conveyor 410, the picking device 420, etc.

[0073] FIG. 8 is a flowchart showing details of the foreign substance detection process under the second condition in step S150. Step S150 includes steps S151 to S155 shown in FIG. 8. One or more first foreign substance regions can be detected from the compressed image according to the discriminant model under the first condition. After the positions of the first foreign substance regions on the image are recorded in the storage device 220 in step S140, the processing circuit 210 cuts out the first foreign substance regions from the compressed image (step S151). The processing circuit 210 performs a reconstruction calculation based on the above-mentioned equation (2) for the cut-out first foreign substance regions using the second reconstruction table corresponding to the second band group, thereby generating a second image for the second band group (step S152). Next, the processing circuit 210 selects one unprocessed region from the first foreign substance regions (step S153). The processing circuit 210 applies the second discriminant model under the second condition to the selected first foreign substance region and determines whether the second condition is satisfied (step S154). Processing circuit 210 stores the determination result in storage device 220. Next, processing circuit 210 determines whether processing has been completed for all of the first foreign substance regions (step S155). If an unprocessed region remains, processing returns to step S153. Steps S153 to S155 are repeated until processing has been completed for all of the first foreign substance regions. When processing has been completed for all of the first foreign substance regions, processing proceeds to step S160.

[0074] FIG. 9 is a diagram showing an example of information stored in the storage device 220. In this example, the processing circuit 210 detects particles of two types of metals (metal 1 and metal 2) as foreign matter, and causes the output device 300 to issue a warning if a particle of metal 1 is detected. The compressed image data obtained by compressed sensing imaging is assigned a number (test sample No.) for each test sample and recorded together with information on the date and time the compressed image was acquired. In this example, metal 1 and metal 2 are detected as foreign matter based on a discrimination model using a first condition. Metal 1 and metal 2 are detected separately based on a discrimination model using a second condition. In this example, a warning is issued if metal 1 is detected.

[0075] FIG. 10 is a diagram illustrating an example of a first band group used in foreign substance detection under the first condition. In FIG. 10, two wavelength ranges representing the first band group are illustrated in gray. FIG. 10 also illustrates an example of the reflectance spectrum of two types of foreign substances (i.e., metal 1 and metal 2) and the reflectance spectrum of the background. In this example, the processing circuit 210 distinguishes between the two types of foreign substances and the background by comparing the reflectance in the two illustrated bands for each pixel. Note that in this example, the first band group includes two bands, 350 nm ± 10 nm and 600 nm ± 10 nm, but this is merely an example. The bands to be included in the first band group are appropriately selected depending on the object to be inspected. The number of bands included in the first band group may be any number greater than or equal to one, and may be three or more. While FIG. 10 illustrates a wavelength range from 350 nm to 850 nm, other wavelength ranges may also be used.

[0076] FIG. 11 shows an example of an image of a band (600 nm ± 10 nm) near the center of the graph shown in FIG. 10 , among images generated for each band by a restoration calculation using a restoration table corresponding to the first band group. This image reflects the reflectance distribution of the subject, with brightness varying depending on the reflectance. Areas with higher reflectance appear whiter, while areas with lower reflectance appear blacker. In this example, the black granular areas in the image represent foreign objects, and the bright areas represent the background. The discrimination model based on the first condition detects foreign objects based on the size of a continuous pixel area whose pixel values ​​in the first band group satisfy a specific condition. When the first band group includes two bands as shown in FIG. 10 , the specific condition may be, for example, that the ratio of the pixel values ​​of the two bands falls within a predetermined range. In the example shown in FIG. 10 , the reflectances of the two types of foreign objects are similar in all bands, but the reflectance of the background differs significantly between the two bands. Therefore, whether an object is a foreign object or background can be determined by whether the ratio of the pixel values ​​of the two bands is within a range of, for example, 0.8 to 1.2. When the first band group includes one band, the specific condition may be whether the pixel value of the one band is above or below a certain threshold. When the first band group includes three or more bands, the specific condition may be that, when a reference band is determined from among the three or more bands, ratios to the reference band are calculated for each of multiple bands other than the reference band, and each of the calculated ratio values ​​falls within a predetermined range. In this way, a region of consecutive pixels in which a value (e.g., a ratio) calculated based on the pixel values ​​of corresponding pixels in each band included in the first band group falls within a predetermined range can be detected as a region in which a foreign substance may exist. The first condition may be, for example, a condition that the diameter of a circumscribing circle of the consecutive pixel region detected in this manner is equal to or greater than a threshold. In the example shown in FIG. 9, a pixel region in which the diameter of a circumscribing circle is 1 mm or greater is detected as a region in which a foreign substance (i.e., metal 1 or metal 2) exists. The condition for discrimination is not limited to a condition based on the ratio of reflectance between wavelength bands, but may be a condition based on the difference, or a condition derived by machine learning.

[0077] The processing circuit 210 assigns an ID to each foreign substance region detected according to the first condition and records its position on the image in the storage device 220. The recorded position of the foreign substance region may be a representative position such as the center or center of gravity of the foreign substance region. A foreign substance region detected in this manner is referred to as a "first foreign substance region." A first foreign substance region may be detected in multiple locations depending on the number of foreign substances.

[0078] The processing circuit 210 extracts one or more detected first foreign substance regions from the compressed image, and performs foreign substance determination processing for each extracted foreign substance region, determining whether it is a metal 1 or a metal 2, in accordance with the discrimination model under the second condition, and stores the determination results (e.g., the type of foreign substance) in the storage device 220.

[0079] FIG. 12 is a diagram showing an example of first foreign substance regions cut out in the example shown in FIG. 11. In FIG. 12, of the cut-out first foreign substance regions, only two regions corresponding to metal 1 are represented by circles. The foreign substance region corresponding to metal 2 is similarly cut out. In this example, circular regions are cut out, but rectangular regions may also be cut out. For each of the cut-out first foreign substance regions, the processing circuitry 210 generates an image for each band using a restoration table corresponding to the second band group.

[0080] FIG. 13 is a diagram showing an example of the second band group in foreign object detection under the second condition. In FIG. 13, the wavelength ranges displayed in gray are the second band group. In this example, the second band group includes nine bands. Metal 1 and Metal 2 can be distinguished by comparing the reflectance in the nine bands. The condition for the distinction may be, for example, whether the ratio or difference in pixel values ​​between the bands is within a predetermined range, or a condition derived by machine learning.

[0081] Specifically, if the second band group includes nine bands, the band with the shortest wavelength is designated as band 1 and the band with the longest wavelength is designated as band 9, the ratio of each band from band 1 to band 8 to the reference pixel value of band 9 is calculated, and discrimination can be made based on this ratio. In this example, the conditions for discrimination can be that the ratio of metal 1 to metal 2 is equal in band 2, and the ratio of metal 2 is lower than metal 1 in bands 4 to 7.

[0082] More specifically, if the second band group includes nine bands, with the band on the shortest wavelength side being band 1 and the band on the longest wavelength side being band 9, metal 1 and metal 2 may be distinguished based on the ratio of the pixel value of band 1 to the pixel value of band 9 (={(pixel value of band 1) / (pixel value of band 9)}), to the ratio of the pixel value of band 8 to the pixel value of band 9 (={(pixel value of band 8) / (pixel value of band 9)}).

[0083] In this example, {(pixel value of band 2 for metal 1) / (pixel value of band 9 for metal 1)} = {(pixel value of band 2 for metal 2) / (pixel value of band 9 for metal 2)}, {(pixel value of band 4 for metal 1) / (pixel value of band 9 for metal 1)} > {(pixel value of band 4 for metal 2) / (pixel value of band 9 for metal 2)}, {(pixel value of band 5 for metal 1) / (pixel value of band 9 for metal 1)} > {(pixel value of band 5 for metal 2) / (pixel value of band 9 for metal 2)}, {(pixel value of band 6 for metal 1) / (pixel value of band 9 for metal 1)} > {(pixel value of band 6 for metal 2) / (pixel value of band 9 for metal 2)}, {(pixel value of band 7 for metal 1) / (pixel value of band 9 for metal 1)} > {(pixel value of band 7 for metal 2) / (pixel value of band 9 for metal 2)} The condition for determination may be:

[0084] When derived by machine learning, a model for estimating the type of foreign matter is created from the information on pixel values ​​for each band based on information on pixel values ​​for each band and information indicating the type of foreign matter, and the foreign matter is derived based on this model.

[0085] If the determination result shows that metal 1 is detected, the processing circuit 210 causes the output device 300 to output a warning. As shown in Fig. 9, the processing circuit 210 may cause the storage device 220 to store the fact that a warning has been output.

[0086] In the above example, two types of metals are assumed to be the foreign objects to be detected, but other objects may also be detected. For example, in a food inspection system, a technology similar to that of this embodiment may be applied to distinguish between insects and burnt food. The foreign objects for which a warning is issued are not limited to one type, but may be two or more types. For example, of the first foreign object regions detected based on the pixel values ​​of each band of the first band group, an area classified into one of multiple predetermined classification lists based on a combination of pixel values ​​for the second band group may be determined to be a second foreign object region in which a specific foreign object is present.

[0087] As described above, according to this embodiment, a first-stage detection operation using a limited number of bands first detects one or more regions where one or more foreign substances may exist as first foreign substance regions. Then, a second-stage detection operation using more bands detects a second foreign substance region containing a specific foreign substance from within the first foreign substance region. This operation significantly reduces the computational load and inspection time compared to detecting foreign substances based on information from all bands within the entire image.

[0088] The method for determining the first foreign substance region may take into consideration the following: In the first stage of foreign substance inspection operation, the first band group (i.e., one or more wavelength bands) may include wavelength band W1, wavelength band W2, N This will be explained assuming that:

[0089] In S110, the processing circuit 210 calculates image data f1 of the wavelength band W1 from the compressed image 120, generates an image 250W1 including the image data f1, and N Image data of N Calculate the image data f N Images containing 250W N The first image is image 250W1 and image 250W N Includes.

[0090] In S110, the processing circuit 210 extracts image data f2, . . . , image data f3, . . . of wavelength band W2 from the compressed image 120. N-1 That is, in S110, the processing circuit 210 does not calculate the image 250W2, ∼, image 250W from the compressed image 120. N-1 does not generate.

[0091] In S110, the inverse problem of the following equation (3) is solved instead of the above equation (1) to obtain the image data f1 and the image data f N Ask for.

[0092]

number

[0093]

number

[0094]

number

[0095] The pixel values ​​of the pixels included in the compressed image 120 are

[0096]

number

[0097] It can also be expressed as P(g ij) is the pixel g included in the compressed image 120 ij The pixel values ​​are i=1 to m and j=1 to n. ij is located at coordinates (i, j) in the compressed image 120. The coordinate axes and coordinates may be as shown in FIG.

[0098] The data g of compressed image 120 is g=(P(g 11 )…P(g 1n )…P(g m1 )…P(g mn )) T It can also be expressed as:

[0099] Wavelength band W k Corresponding image 250W k (k=1~N) is the image data f k It can be considered that it has. Image 250W k The pixel values ​​of the pixels included in

[0100]

number

[0101] It can also be expressed as P(f kij ) is the image 250W k Pixels f included in kij (i=1 to m, j=1 to n) kij The image is 250W k In this case, it is located at coordinates (i,j).

[0102] Image 250W k Image data of k teeth f k =(P(f k11 )…P(f k1n )…P(f km1 )…P(f kmn )) T It can also be expressed as:

[0103] Image data pThe pixel value P(f pij ) and image data f q The pixel value P(f qij ) is the pixel value of the same part of the object. This is the pixel value of pixel f pij and pixel value f qij H shown below can be determined so that it corresponds to the same position of the subject.

[0104] The detection of the first foreign body region in S120 is performed by r 11 =P(f 111 ) / P(f N11 ), ···, r mn =P(f 1mn ) / (f Nmn ) and from these, 0.8≦r ij Find all coordinates (i,j) that satisfy ≦1.2.

[0105] And 0.8≦r rs ≦1.2 and 0.8≦r (r+1)s ≦1.2 or 0.8≦r rs ≦1.2 and 0.8≦r (r-1)s ≦1.2 or 0.8≦r rs ≦1.2 and 0.8≦r r(s+1) ≦1.2 or 0.8≦r rs ≦1.2 and 0.8≦r r(s-1) If ≦1.2, the pixel located at coordinates (r, s) is determined to be included in the region of consecutive pixels (1≦r−1, r+1≦m, 1≦s−1, s+1≦n).

[0106] 0.8≦r rs Even if ≦1.2, 0.8≦r (r+1)s Not ≦1.2 and 0.8≦r (r-1)s Not ≦1.2 and 0.8≦r r(s+1) Not ≦1.2 and 0.8≦r r(s-1) If not ≦1.2, the pixel located at coordinates (r, s) may be determined not to be included in the region of contiguous pixels.

[0107] The first foreign object area determined in the above manner is assumed to be an area having coordinates (r, s), coordinates (r+1, s), coordinates (r, s+1), and coordinates (r+1, s+1), and the following explanation will be given.

[0108] In S140, the coordinates of the first foreign substance region, ie, coordinates (r, s), coordinates (r+1, s), coordinates (r, s+1), and coordinates (r+1, s+1), are recorded in the storage device 220.

[0109] The method for determining the second foreign substance region may take into consideration the following: In the second stage foreign substance inspection operation, the second band group (i.e., the plurality of second wavelength bands) is a wavelength band W t , wavelength band W u , wavelength band W v , wavelength band W w The number of second wavebands (four in this example) is greater than the number of one or more wavebands (two in the above example). From compressed image 120 and matrix H, image 250W t The pixel values ​​of some of the pixels in the image are calculated. t The pixel values ​​of the other pixels of the pixel 10 are not calculated.

[0110] Compressed image 120 and matrix H produce image 250W u The pixel values ​​of some of the pixels in the image are calculated. u The pixel values ​​of the other pixels of the pixel 10 are not calculated.

[0111] Compressed image 120 and matrix H produce image 250W v The pixel values ​​of some of the pixels in the image are calculated. v The pixel values ​​of the other pixels of the pixel 10 are not calculated.

[0112] Compressed image 120 and matrix H produce image 250W w The pixel values ​​of some of the pixels in the image are calculated. w The pixel values ​​of the other pixels of the pixel 10 are not calculated.

[0113] That is, (1) Image 250W t Image data of t =(P(f t11 )…P(f t1n )…P(f tm1 )…P(f tmn )) T P(f trs ), P(f t(r+1)s ), P(f tr(s+1) ), P(f t(r+1)(s+1) ) is calculated, and P(f t11 ), ~, P(f tmn ), P(f trs ), P(f t(r+1)s ), P(f tr(s+1) ), P(f t(r+1)(s+1) ) is not calculated.

[0114] (2) Image 250W u Image data of u =(P(f u11 )…P(f u1n )…P(f um1 )…P(f umn )) T P(f urs ), P(f u(r+1)s ), P(f ur(s+1) ), P(f u(r+1)(s+1) ) is calculated, and P(f u11 ), ~, P(f umn ), P(f urs ), P(f u(r+1)s ), P(f ur(s+1) ), P(f u(r+1)(s+1) ) is not calculated.

[0115] (3) Image 250W v Image data of v =(P(f v11 )…P(f v1n )…P(f vm1 )…P(f vmn )) T P(f vrs ), P(f v(r+1)s ), P(f vr(s+1) ), P(f v(r+1)(s+1) ) is calculated, and P(f v11 ), ~, P(f vmn), P(f vrs ), P(f v(r+1)s ), P(f vr(s+1) ), P(f v(r+1)(s+1) ) is not calculated.

[0116] (4) Image 250W w Image data of w =(P(f w11 )…P(f w1n )…P(f wm1 )…P(f wmn )) T P(f wrs ), P(f w(r+1)s ), P(f wr(s+1) ), P(f w(r+1)(s+1) ) is calculated, and P(f w11 ), ~, P(f wmn ), P(f wrs ), P(f w(r+1)s ), P(f wr(s+1) ), P(f w(r+1)(s+1) ) is not calculated.

[0117] From (1) to (4), of the mn×4 pixel values, 4×4=16 image values ​​are calculated, and (mn−4)×4 pixel values ​​are not calculated.

[0118] For S150, image 250W t The pixel value P(f trs ), Image 250W t The pixel value P(f t(r+1)s ), Image 250W t The pixel value P(f tr(s+1) ), Image 250W t The pixel value P(f t(r+1)(s+1) ) is used to perform foreign object detection under the second condition. t The four pixel values ​​contained in the image are used. t The other (m×n−4) pixel values ​​involved are not used.

[0119] For S150, image 250W u The pixel value P(f urs ), Image 250W u The pixel value P(f u(r+1)s ), Image 250W u The pixel value P(f ur(s+1) ), Image 250W u The pixel value P(f u(r+1)(s+1) ) is used to perform foreign object detection under the second condition. u The four pixel values ​​contained in the image are used. u The other (m×n−4) pixel values ​​involved are not used.

[0120] For S150, image 250W v The pixel value P(f vrs ), Image 250W v The pixel value P(f v(r+1)s ), Image 250W v The pixel value P(f vr(s+1) ), Image 250W v The pixel value P(f v(r+1)(s+1) ) is used to perform foreign object detection under the second condition. t The four pixel values ​​contained in the image are used. v The other (m×n−4) pixel values ​​involved are not used.

[0121] For S150, image 250W w The pixel value P(f wrs ), Image 250W w The pixel value P(f w(r+1)s ), Image 250W w The pixel value P(fwr(s+1) ), Image 250W w The pixel value P(f w(r+1)(s+1) ) is used to perform foreign object detection under the second condition. w The four pixel values ​​contained in the image are used. w The other (m×n−4) pixel values ​​involved are not used.

[0122] (Embodiment 2) Next, a description will be given of an inspection system according to embodiment 2. Fig. 14 is a block diagram showing an example of the configuration of the inspection system according to this embodiment.

[0123] In the first embodiment, the imaging device 100 and the processing circuit 210 restore an image for each band from a compressed image using compressed sensing. In contrast, in the present embodiment, the imaging device 100 captures images for each band and combines the multiple images acquired by capturing the images to generate image data including information on a desired group of bands. Instead of including a filter array 110 as shown in FIGS. 1A to 1D, the imaging device 100 in this embodiment includes a mechanism for capturing images for each band and generating an image for each band. In this embodiment, the imaging device 100 directly generates an image for each band. For this reason, the storage device 220 does not store a restoration table used to generate an image for each band.

[0124] In this embodiment, the imaging device 100 first captures images of each band included in the first band group and generates multiple images corresponding to those bands. The processing circuit 210 detects a first foreign object region from the images of the first band group generated by the imaging device 100 in accordance with a discrimination model based on a first condition. If a first foreign object region is detected, the imaging device 100 captures images of each band in a second band group, which includes more bands than the first band group, and generates multiple images corresponding to those bands. The processing circuit 210 performs foreign object detection processing on those images in accordance with a discrimination model based on a second condition and stores the results in the storage device 220. If a foreign object region satisfying the second condition is detected, the processing circuit 210 sends a control signal to the output device 300 and the actuation device 400. Upon receiving the control signal, the output device 300 outputs a warning to the display 310 or the like. Upon receiving the control signal, the actuation device 400 controls the conveyor 410 or the like.

[0125] FIG. 15 is a flowchart showing an example of the operation of two-stage foreign substance detection using multiple image capture in this embodiment. In this example, the processing circuit 210 instructs the image capture device 100 to capture an image for each band belonging to the first band group (step S200). As a result, the image capture device 100 generates image data for each band in the first band group. For example, if the first band group is composed of a band around 350 nm and a band around 600 nm, image capture is performed using an illumination light wavelength of 350 nm, and then image capture is performed using an illumination light wavelength of 600 nm. The illumination light wavelength may be adjusted by the user for each capture, or the image capture device 100 may be equipped with an illumination device and the wavelength may be adjusted automatically. The processing circuit 210 acquires image data for the generated first band group (step S210). The processing circuit 210 performs processing to detect a first foreign substance region from the acquired image data in accordance with a discrimination model based on a first condition (step S230). The processing circuit 210 determines whether a first foreign substance region that satisfies the first condition exists (step S240). If the first foreign substance region is not detected, the process ends. If the first foreign substance region is detected, the processing circuit 210 causes the imaging device 100 to capture an image of a relatively narrow region including the first foreign substance region for each band belonging to the second band group and generate image data for each band (step S250). The processing circuit 210 acquires image data for the second band group generated for the relatively narrow region including the first foreign substance region (step S260). The processing circuit 210 performs processing to detect a second foreign substance region from the acquired image data in accordance with a discrimination model based on the second condition (step S270). The processing circuit 210 determines whether a second foreign substance region that satisfies the second condition exists (step S280). If the second foreign substance region is not detected, the process ends. If the second foreign substance region is detected, the processing circuit 210 stores information about the second foreign substance region in the storage device 220 (step S290). The processing circuit 210 then outputs control signals to the output device 300 and the actuation device 400 (step S300). The output device 300 receives a control signal from the processing circuit 210 and causes a display 310 or the like to display a warning.The operating device 400 receives control signals from the processing circuit 210 and controls the conveyor 410, the picking device 420, and the like.

[0126] In this embodiment, imaging for each band is performed by switching the wavelength of the illumination light and performing imaging multiple times, but imaging for each band may be performed by other methods. For example, the imaging device 100 may perform imaging multiple times while switching between multiple filters with different transmission wavelength ranges. The imaging device 100 may also be a line-scan hyperspectral camera equipped with a prism or a diffraction grating.

[0127] As described above, in this embodiment, imaging is performed multiple times for each band. If there are a large number of bands to be imaged, the inspection will take a long time. In this embodiment, imaging is performed for a relatively small number of first band groups, and imaging is performed for a larger number of second band groups only if a foreign object is detected. Therefore, in most cases where no foreign object is detected, the inspection is completed without requiring a long time for imaging. Because detailed inspection of the second band group is performed only if a foreign object is detected, the time required for the entire inspection process can be significantly reduced. [Industrial Applicability]

[0128] The technology disclosed herein is useful, for example, in cameras and measuring instruments that acquire images of multiple wavelengths. The technology disclosed herein can be used, for example, in applications for detecting foreign matter in articles such as industrial products or food. [Explanation of symbols]

[0129] 70 Objects 100 Imaging device 110 Filter Array 120 images 140 Optical system 160 image sensors 200 Processing Equipment 210 Processing circuit 220 Storage device 300 Output Device 310 Display 320 Speaker 330 Lamp 400 Actuator 410 Conveyor 420 Picking device

Claims

1. A method for detecting a foreign substance contained in an inspection object, comprising: acquiring first image data for the test object, each pixel having a pixel value for a first set of bands including one or more wavelength bands; determining one or more pixel regions that satisfy a first condition as a first foreign substance region from the first image data; acquiring second image data for an area including the first foreign substance area, each pixel having a pixel value for a second band group that includes more wavelength bands than the first band group; determining, from the second image data, one or more pixel regions that satisfy a second condition different from the first condition as a second foreign substance region in which the foreign substance exists; outputting information about the second foreign substance region; A method comprising:

2. 2. The method of claim 1, wherein the first condition is that the pixel region is composed of a plurality of consecutive pixels whose pixel values ​​for the first band group satisfy a specific condition, and the size of the pixel region exceeds a predetermined size.

3. The method of claim 1 or 2, wherein the second condition is that the pixel region is classified into one of a predetermined classification list based on pixel values ​​for the second set of bands.

4. acquiring the first image data includes acquiring compressed image data in which image information for each of a plurality of wavelength bands including the second band group is compressed as a single two-dimensional image, and generating the first image data from the compressed image data; obtaining the second image data includes extracting the region including the first foreign substance region from the compressed image data, and generating the second image data based on data of the extracted region; The method according to any one of claims 1 to 3.

5. generating the first image data includes restoring the first image data from the compressed image data using a first restoration table corresponding to the first band group; generating the second image data includes restoring the second image data from data of the extracted region using a second restoration table corresponding to the second band group; The method of claim 4.

6. the compressed image data is generated by an imaging device comprising a filter array and an image sensor; the filter array includes a plurality of types of filters having different transmission spectra; the first restoration table and the second restoration table are generated based on a distribution of the transmission spectrum in the filter array. The method of claim 5.

7. the first image data is acquired by a first imaging operation by an imaging device; the second image data is acquired by a second imaging operation by the imaging device; The method according to any one of claims 1 to 3.

8. and outputting a warning to an output device when the second foreign object region is detected.

8. The method according to any one of claims 1 to 7.

9. The method of claim 1 , further comprising storing the location of the first foreign body region and the location of the second foreign body region in a storage device.

10. An apparatus for detecting foreign matter contained in an inspection object, a processor; a storage medium storing a computer program; Equipped with The processor executes the computer program, acquiring first image data for the test object, each pixel having a pixel value for a first set of bands including one or more wavelength bands; determining one or more pixel regions that satisfy a first condition as a first foreign substance region from the first image data; acquiring second image data for an area including the first foreign substance area, each pixel having a pixel value for a second band group that includes more wavelength bands than the first band group; determining, from the second image data, one or more pixel regions that satisfy a second condition different from the first condition as a second foreign substance region in which the foreign substance exists; outputting information about the second foreign substance region; A device that performs the following:

11. A computer program for detecting foreign matter contained in an inspection object, the computer comprising: acquiring first image data for the test object, each pixel having a pixel value for a first set of bands including one or more wavelength bands; determining one or more pixel regions that satisfy a first condition as a first foreign substance region from the first image data; acquiring second image data for an area including the first foreign substance area, each pixel having a pixel value for a second band group that includes more wavelength bands than the first band group; determining, from the second image data, one or more pixel regions that satisfy a second condition different from the first condition as a second foreign substance region in which the foreign substance exists; outputting information about the second foreign substance region; A computer program that executes

12. A method for detecting a foreign substance contained in an inspection object, comprising: acquiring one or more first images, the one or more first images corresponding to one or more first wavelength bands; determining a region that satisfies a first condition as a first foreign substance region based on the one or more first images; acquiring a plurality of second images, each of the plurality of second images including an area corresponding to the first foreign substance area, the plurality of second images corresponding to a plurality of second wavelength bands, the number of the plurality of second wavelength bands being greater than the number of the one or more first wavelength bands; determining, based on the plurality of second images, one or more regions that satisfy a second condition different from the first condition as a second foreign substance region in which the foreign substance exists; outputting information about the second foreign substance region; method.

13. a plurality of first pixel values ​​of a plurality of first pixels included in each of the one or a plurality of first images are calculated based on a third image obtained by capturing an image of the inspection object with a camera, and the plurality of first pixels are included in an area corresponding to the first foreign substance area and an area other than the first foreign substance area; a plurality of pixel values ​​of a plurality of pixels included in a region corresponding to the first foreign substance region and included in each of the plurality of second images is calculated based on the third image; The pixel values ​​of the pixels included in each of the second images and included in a region other than the region corresponding to the first foreign substance region are not calculated. The method of claim 12.

Images