Information processing method, information processing device, and program

Abstract

To provide an information processing method, etc., capable of acquiring the surface shape of a movable object without applying removal correction.SOLUTION: An information processing method includes the steps of: irradiating a surface of an object with a plurality of illumination lights corresponding to each of N wavelength bands (N is a natural number of four or more) using a light source (S10); generating M images (M is a natural number of four or more) corresponding to each of the plurality of wavelength bands by capturing the surface of the object with a camera in a state in which the plurality of illumination lights corresponding to the respective N wavelength bands are incident on the surface of the object from mutually different directions (S20, S30); and generating information relating to a surface shape of the object on the basis of the M images (S40).SELECTED DRAWING: Figure 5

Description

[Technical Field]

[0001] The present disclosure relates to an information processing method, an information processing device, and a program. [Background technology]

[0002] Patent Document 1 describes an image inspection device based on photometric stereo and multispectral imaging. Patent Document 2 describes the calculation of image data in multiple wavelength bands using a compressed sensing technique. Patent Document 3 describes a hyperspectral imaging system. [Prior art documents] [Patent documents]

[0003] [Patent Document 1] Japanese Patent Application Publication No. 2018-189561 [Patent Document 2] Japanese Patent Application Laid-Open No. 2016-156801 [Patent Document 3] Japanese Patent Application Publication No. 2017-201317 Summary of the Invention [Problem to be solved by the invention]

[0004] However, the technology disclosed in Patent Document 1 is a technology in which light is irradiated onto a subject from four directions in sequence, and four images are taken to obtain information about the surface shape of the subject. Therefore, if the subject is movable, the subject will move during imaging, and correction for the movement of the subject is required to obtain information about the subject's surface shape. Furthermore, if the correction for the movement is not performed accurately, accurate information about the surface shape may not be obtained. Note that Patent Documents 2 and 3 do not disclose obtaining information about the surface shape of a movable subject.

[0005] Therefore, the present disclosure provides an information processing method, an information processing device, and a program that can acquire the surface shape of a movable object without applying movement correction. [Means for solving the problem]

[0006] An information processing method according to one embodiment of the present disclosure includes using a light source to irradiate a surface of an object with a plurality of illumination lights corresponding to N wavelength bands (N is a natural number greater than or equal to 4); capturing an image of the surface of the object with an imaging device while the plurality of illumination lights corresponding to the N wavelength bands are irradiated onto the surface of the object from mutually different directions, thereby generating M images (M is a natural number greater than or equal to 4) corresponding to each of the plurality of wavelength bands; and generating information related to the surface shape of the object based on the M images.

[0007] An information processing device according to one embodiment of the present disclosure includes a control unit that uses a light source to irradiate a surface of an object with a plurality of illumination lights corresponding to N wavelength bands (N is a natural number greater than or equal to 4), a first processing unit that uses an imaging device to capture an image of the surface of the object while the plurality of illumination lights corresponding to the N wavelength bands are irradiated onto the surface of the object from mutually different directions, thereby generating M images (M is a natural number greater than or equal to 4) corresponding to each of the plurality of wavelength bands, and a second processing unit that generates information regarding the surface shape of the object based on the M images.

[0008] A program according to one aspect of the present disclosure is a program for causing a computer to execute the above-described information processing method. [Effects of the Invention]

[0009] According to one aspect of the present disclosure, it is possible to realize an information processing method or the like that can acquire the surface shape of a movable object without applying movement correction. [Brief explanation of the drawings]

[0010] [Figure 1] FIG. 1 is a block diagram showing a functional configuration of an information processing system according to the first embodiment. [Figure 2A] FIG. 2A is a diagram illustrating an example of the configuration of a light-emitting unit according to the first embodiment. [Figure 2B] FIG. 2B is a diagram showing another example of the configuration of the light-emitting section according to the first embodiment. [Figure 3] FIG. 3 is a schematic diagram showing the configuration of the imaging device according to the first embodiment. [Figure 4A] FIG. 4A is a schematic diagram showing a filter array according to the first embodiment. [Figure 4B] FIG. 4B is a diagram illustrating an example of a transmission spectrum of the filter according to the first embodiment. [Figure 4C] FIG. 4C is a diagram illustrating an example of a transmission spectrum of another filter according to the first embodiment. [Figure 4D] FIG. 4D is a diagram illustrating an example of the transmittance of the first wavelength band of the filter array according to the first embodiment. [Figure 4E] FIG. 4E is a diagram illustrating an example of the transmittance of the second wavelength band of the filter array according to the first embodiment. [Figure 5] FIG. 5 is a flowchart showing the operation of the information processing system according to the first embodiment. [Figure 6] FIG. 6 is a schematic diagram for explaining how the lighting device according to the first embodiment irradiates an object with light. [Figure 7] FIG. 7 is a schematic diagram showing an example of a multi-wavelength image according to the first embodiment. [Figure 8] FIG. 8 is a schematic diagram for explaining generation of information about the surface shape of the object according to the first embodiment. [Figure 9] FIG. 9 is a diagram showing an example of display of information relating to the surface condition according to the first embodiment. [Figure 10] FIG. 10 is a schematic diagram for explaining the overlap of spectra according to the second embodiment. [Figure 11]FIG. 11 is a flowchart showing the operation of the information processing system according to the second embodiment. [Figure 12] FIG. 12 is a flowchart showing the detailed operation of step S60 shown in FIG. [Figure 13] FIG. 13 is a schematic diagram illustrating an example of a multi-wavelength image including a low-contrast image according to the second embodiment. DETAILED DESCRIPTION OF THE INVENTION

[0011] (Background to this disclosure) Before describing the information processing method and the like according to the present disclosure, the background to the present disclosure will be described.

[0012] As described in the "Background Art" above, Patent Document 1 discloses an image inspection device that illuminates an inspection object according to the photometric stereo method. Specifically, Patent Document 1 discloses an image inspection device that can acquire images for photometric stereo by irradiating the object with light from four directions. In Patent Document 1, the surface shape of a single object is acquired by photographing it while changing the direction of illumination.

[0013] Here, if the object is moving, the object will move during the four captures, and therefore correction for the movement (i.e., change in the object's position relative to the imaging device) must be performed. This raises concerns that the amount of processing by the information processing device that generates information about the surface shape will increase. Furthermore, if the correction for the movement is not performed accurately, there is a risk that the accuracy of the generated information about the surface shape will decrease. In other words, there are concerns about the accuracy of the information about the surface shape obtained by capturing images of a moving object.

[0014] As described above, there are concerns about the amount of processing by the information processing device and the accuracy of the information relating to the surface shape in the technology of Patent Document 1. Furthermore, Patent Documents 2 and 3 do not disclose any technology that solves these concerns.

[0015] Therefore, the inventors of the present application have conducted extensive research into information processing methods and the like that can acquire the surface shape of a movable object without applying movement correction, and have devised the information processing methods and the like described below. Note that a movable object may be an object that is attached to or placed on a device having a movement function and moved by the device (for example, an object that is placed on a conveyor and moves), or an object that has a movement function and is self-propelled. Furthermore, the movement may be a translational movement or a rotational movement.

[0016] An information processing method according to a first aspect of the present disclosure includes using a light source to irradiate a surface of an object with a plurality of illumination lights corresponding to N wavelength bands (N is a natural number greater than or equal to 4); capturing an image of the surface of the object with an imaging device while the plurality of illumination lights corresponding to the N wavelength bands are irradiated onto the surface of the object from mutually different directions, thereby generating M images (M is a natural number greater than or equal to 4) corresponding to each of the plurality of wavelength bands; and generating information related to the surface shape of the object based on the M images.

[0017] As a result, since the image is captured while the surface of the object is irradiated with a plurality of illumination lights corresponding to N wavelength bands from different directions, M images can be generated from a single image capture. In other words, there is no need to capture images multiple times as shown in Patent Document 1. Furthermore, since the surface of the object is irradiated with a plurality of illumination lights from different directions, the surface shape of the object can be acquired by photometric stereo. Therefore, according to one aspect of the present disclosure, an information processing method can be realized that can acquire the surface shape of a movable object without applying movement correction.

[0018] Furthermore, for example, an information processing method according to a second aspect may be the information processing method according to the first aspect, in which the number of the M images to be generated is greater than the N images.

[0019] This makes it possible to generate information about the surface condition with higher accuracy, since information about the surface shape can be generated from M images, which is greater than N. For example, if the M images are images in which the colors of the irradiated light are different from one another, the accuracy of the information about the surface condition can be effectively improved.

[0020] Also, for example, an information processing method according to a third aspect may be an information processing method according to the second aspect, further including determining a plurality of first object images corresponding to each of the N wavelength bands from among the M images, and information regarding the surface shape may be generated based on the plurality of first object images.

[0021] This allows images corresponding to N wavelength bands to be selected from M images, thereby preventing the selection of images that are not suitable for generating information about surface shape, such as images with low brightness.

[0022] Also, for example, an information processing method according to a fourth aspect may be the information processing method according to the third aspect, further including acquiring illumination data including information on the N wavelength bands associated with the plurality of irradiation lights, and the plurality of first target images may be determined based on the illumination data.

[0023] This makes it possible to prevent the use of illumination data from leading to the selection of an image that is not suitable for generating information about the surface shape.

[0024] Furthermore, for example, an information processing method according to a fifth aspect is an information processing method according to any one of the first to fourth aspects, wherein the M images include N first images corresponding to the N wavelength bands and one or more second images corresponding to wavelength bands of overlapping light in which at least two irradiation lights corresponding to at least two wavelength bands among the N wavelength bands are overlapped, and the information regarding the surface shape may be generated based on the N first images and the one or more second images.

[0025] This increases the number of images that can be used to generate information about the surface shape, thereby improving the accuracy of the information about the surface shape.

[0026] Furthermore, for example, an information processing method according to a sixth aspect may be the information processing method according to the fifth aspect, wherein irradiating the surface of the object includes irradiating the plurality of irradiation lights onto the surface of the object from the mutually different directions using an illumination device that does not have a light source corresponding to the wavelength band of the overlapping light.

[0027] This makes it possible to improve the accuracy of information regarding the surface shape without increasing the number of lights in the lighting device.

[0028] Furthermore, for example, an information processing method according to a seventh aspect may be an information processing method according to any one of the first to sixth aspects, and may further include deriving an index value for each of the M images based on pixel values ​​contained in the image, and determining a plurality of second target images from among the M images to be used for generating information related to the surface shape based on the index value for each of the M images.

[0029] This allows determining an image suitable for generating information about the surface shape in terms of index values ​​based on pixel values.

[0030] Also, for example, an information processing method according to an eighth aspect may be the information processing method according to the seventh aspect, wherein the index value indicates the variation in distribution of the pixel values ​​and the number of pixels, and determining the plurality of second target images to be used for generating information related to the surface shape may include determining, from among the M images, images whose index value is equal to or greater than a predetermined threshold as the plurality of second target images.

[0031] This allows the second object image to be determined from which images that are prone to quantization errors are excluded, making it possible to generate information about the surface condition with even greater accuracy.

[0032] Also, for example, an information processing method according to a ninth aspect is an information processing method according to any one of the first to eighth aspects, and the information about the surface shape may include at least one of the curvature, contour, unevenness, reflectance distribution, and dimensions of the object.

[0033] This makes it possible to acquire at least one of the curvature, contour, unevenness, reflectance distribution, and dimensions of a movable object without applying movement correction.

[0034] Also, for example, an information processing method according to a 10th aspect may be an information processing method according to any one of the 1st to 9th aspects, in which the imaging device includes an optical filter including a plurality of regions, each of which has a different transmission spectrum from the other, and the imaging device outputs a compressed image including information of the N wavelength bands by detecting light that has passed through the optical filter.

[0035] This makes it possible to realize an information processing method that can acquire the surface shape of a moving object without applying movement correction, using an imaging device that can acquire compressed images.

[0036] Also, for example, an information processing method according to an eleventh aspect may be an information processing method according to any one of the first to tenth aspects, wherein the imaging device includes an optical filter including a plurality of regions, and the transmission spectrum of each of the plurality of regions has a peak corresponding to each of the N wavelength bands, and the imaging device outputs an image including a plurality of pixel values ​​corresponding to each of the N wavelength bands.

[0037] This makes it possible to realize an information processing method that can acquire the surface shape of a moving body without applying movement correction using an imaging device that outputs an image containing multiple pixel values ​​corresponding to each of N wavelength bands (i.e., an imaging device that outputs N spectral images).

[0038] Furthermore, for example, an information processing method according to a twelfth aspect may be an information processing method according to any one of the first to eleventh aspects, and may include using an illumination device having a plurality of light source units for irradiating the plurality of illumination lights, the illumination device having the plurality of light source units arranged to surround the object in a planar view, to irradiate the plurality of illumination lights onto the surface of the object from the mutually different directions, and the mutually different directions may include directions toward the object from each of two light source units arranged opposite each other in the planar view.

[0039] This makes it possible to obtain the surface shape of a movable object without applying movement correction by using images obtained when light is irradiated from each of two light source units arranged opposite each other in a planar view in a direction toward the object.

[0040] Also, for example, an information processing method according to a 13th aspect is an information processing method according to any one of the 1st to 12th aspects, and the photographing of the surface of the object using the imaging device may be performed while the object is moving relative to the imaging device.

[0041] This makes it possible to acquire the surface shape of the object without applying movement correction to an image captured while the object is moving relative to the imaging device.

[0042] An information processing device according to a fourteenth aspect of the present disclosure includes a control unit that uses a light source to irradiate a surface of an object with a plurality of illumination lights corresponding to each of N wavelength bands (N is a natural number greater than or equal to four), a first processing unit that uses an imaging device to capture an image of the surface of the object while the plurality of illumination lights corresponding to each of the N wavelength bands are irradiated onto the surface of the object from mutually different directions, thereby generating M images (M is a natural number greater than or equal to four) corresponding to each of the plurality of wavelength bands, and a second processing unit that generates information regarding the surface shape of the object based on the M images.

[0043] This provides the same effect as the above-described information processing method.

[0044] A program according to a fifteenth aspect of the present disclosure is a program for causing a computer to execute the information processing method according to any one of the first to thirteenth aspects.

[0045] This provides the same effect as the above-described information processing method.

[0046] These general or specific aspects may be realized as a system, a method, an integrated circuit, a computer program, or a non-transitory recording medium such as a computer-readable CD-ROM, or as any combination of the system, method, integrated circuit, computer program, or recording medium. The program may be pre-stored in the recording medium, or may be supplied to the recording medium via a wide area communication network including the Internet.

[0047] Hereinafter, the embodiments will be specifically described with reference to the drawings.

[0048] The embodiments described below are all comprehensive or specific examples. The numerical values, shapes, components, component placement and connection configurations, steps, and step order shown in the following embodiments are merely examples and are not intended to limit the present disclosure. Furthermore, among the components in the following embodiments, components not described in independent claims are described as optional components.

[0049] Furthermore, in this specification, terms indicating relationships between elements such as coincidence, terms indicating the shapes of elements such as rectangle or circle, as well as numerical values ​​and numerical ranges are not expressions that express only the strict meaning, but are expressions that also include a substantially equivalent range, for example, a difference of about a few percent (or about 10%).

[0050] Furthermore, in this specification, ordinal numbers such as "first" and "second" do not refer to the number or order of components unless otherwise specified, but are used for the purpose of avoiding confusion and distinguishing between components of the same type.

[0051] (Embodiment 1) The information processing system according to this embodiment will be described below with reference to FIGS.

[0052] [1-1. Information Processing System Configuration] First, the configuration of an information processing system according to this embodiment will be described with reference to Figures 1 to 4E. Figure 1 is a block diagram showing the functional configuration of an information processing system 100 according to this embodiment.

[0053] 1, the information processing system 100 includes a lighting device 10, an imaging device 20, and an information processing device 30. The lighting device 10, the imaging device 20, and the information processing device 30 are connected to each other so as to be able to communicate with each other.

[0054] Illumination device 10 uses a light source to irradiate the surface of an object, which is a subject, with multiple beams of illumination light (multispectral light) corresponding to N wavelength bands (N is a natural number equal to or greater than 4). Illumination device 10 is configured to be able to simultaneously irradiate the object with light of the N wavelength bands from different directions. Illumination device 10 includes a light-emitting unit 11. Note that the light source is implemented as an LED (Light Emitting Diode) element, but may be implemented with any configuration that lights up with supplied power and emits light of the desired wavelength band.

[0055] The light emitting unit 11 is configured to be able to simultaneously emit a plurality of irradiation light beams corresponding to the N wavelength bands, respectively.

[0056] FIG. 2A is a diagram showing an example of the configuration of light-emitting unit 11a according to the present embodiment. FIG. 2B is a diagram showing another example of the configuration of light-emitting unit 11b according to the present embodiment. FIGS. 2A and 2B are diagrams showing the light-emitting portion of light-emitting unit 11 when viewed from the object side. For ease of identification, the light-emitting unit shown in FIG. 2A is denoted by the reference symbol 11a, and the light-emitting unit shown in FIG. 2B is denoted by the reference symbol 11b.

[0057] As shown in Fig. 2A, the light-emitting unit 11a has a plurality of blocks 11a1 to 11a4. Each of the plurality of blocks 11a1 to 11a4 is arranged, for example, in a ring shape (a polygonal (quadrilateral in the example of Fig. 2A) ring). Each of the plurality of blocks 11a1 to 11a4 is formed, for example, in a rectangular shape, and each includes one or more light sources.

[0058] Block 11a1 is a portion of light-emitting unit 11a that includes one or more light sources capable of emitting irradiation light with a peak wavelength λ1. When block 11a1 includes multiple light sources, the multiple light sources may each be light sources that emit light with a peak wavelength λ1 (for example, light sources of the same type), or at least one light source may include a light source that has a peak wavelength different from the other light sources. When at least one light source includes a light source that has a peak wavelength different from the other light sources, the peak wavelength of the light obtained by mixing the light from the multiple light sources is λ1.

[0059] Blocks 11a2 to 11a4 are similarly configured. Specifically, block 11a2 is configured to include one or more light sources capable of emitting irradiation light with a peak wavelength λ2, block 11a3 is configured to include one or more light sources capable of emitting irradiation light with a peak wavelength λ3, and block 11a4 is configured to include one or more light sources capable of emitting irradiation light with a peak wavelength λ4.

[0060] The peak wavelengths λ1 to λ4 are different from each other. The blocks 11a1 to 11a4 may be arranged in a regular pattern, such as in ascending order of peak wavelength, or may be arranged randomly. The peak wavelengths may be different from each other by, for example, the wavelength resolution of the imaging device 20 or more.

[0061] Each of the plurality of blocks 11a1 to 11a4 is arranged so as to be able to irradiate the entire surface of the object with irradiation light from different directions. Although the object is not shown in Fig. 2A, the plurality of blocks 11a1 to 11a4 are arranged to surround the object in a plan view of the surface of the object. Blocks 11a1 and 11a3 are arranged so as to be able to irradiate the object with irradiation light from opposite directions in a plan view. In the example of Fig. 2A, block 11a1 is arranged so as to be able to irradiate the object with irradiation light from above, and block 11a3 is arranged so as to be able to irradiate the object with irradiation light from below. Blocks 11a2 and 11a4 are positioned in a plan view so that they can irradiate the object with light from opposite directions relative to the object. In the example of Figure 2A, block 11a2 is positioned so that it can irradiate the object with light from the left side, and block 11a4 is positioned so that it can irradiate the object with light from the right side.

[0062] The plurality of blocks 11a1 to 11a4 are arranged spaced apart from one another, but may also be arranged adjacent to one another.

[0063] As shown in Fig. 2B, the light-emitting unit 11b has a plurality of blocks 11b1 to 11b8. Each of the plurality of blocks 11b1 to 11b8 is arranged, for example, in a ring shape (in the example of Fig. 2B, a circular ring shape). Each of the plurality of blocks 11b1 to 11b8 is formed, for example, in a fan shape, and each includes one or more light sources. Each of the plurality of blocks 11b1 to 11b8 is arranged so that it can irradiate the entire surface of the object with irradiation light from mutually different directions.

[0064] Block 11b1 includes one or more light sources capable of emitting irradiation light with a peak wavelength λ1, block 11b2 includes one or more light sources capable of emitting irradiation light with a peak wavelength λ2, block 11b3 includes one or more light sources capable of emitting irradiation light with a peak wavelength λ3, and block 11b4 includes one or more light sources capable of emitting irradiation light with a peak wavelength λ4. Block 11b5 includes one or more light sources capable of emitting irradiation light with a peak wavelength λ5, block 11b6 includes one or more light sources capable of emitting irradiation light with a peak wavelength λ6, block 11b7 includes one or more light sources capable of emitting irradiation light with a peak wavelength λ7, and block 11b8 includes one or more light sources capable of emitting irradiation light with a peak wavelength λ8. Note that the peak wavelengths λ1 to λ8 are all different wavelengths. The plurality of blocks 11b1 to 11b8 may be arranged in a regular manner, such as in ascending order of peak wavelength, or may be arranged randomly.

[0065] 2A and 2B show an example in which the blocks are arranged at equal intervals in the circumferential direction of the ring (every 90 degrees in FIG. 2A, and every 45 degrees in FIG. 2B), but the blocks are not limited to being arranged at equal intervals. Each block is an example of a light source unit.

[0066] The spectral width of the irradiated light emitted from the light-emitting unit 11 is not particularly limited, and may be determined, for example, according to the sensitivity of the image sensor 21a (see FIG. 3) of the photographing unit 21. The spectrum of the irradiated light may include, for example, ultraviolet light to near-infrared light. For example, the spectrum of the irradiated light may include, but is not limited to, a spectrum of 400 nm or more and 1600 nm or less.

[0067] As shown in FIG. 2A or 2B, the light-emitting unit 11 of the lighting device 10 includes a plurality of light source units (blocks) arranged to surround the object 200 in a planar view. This allows the lighting device 10 to simultaneously irradiate the surface of the object 200 with multispectral light from different directions. Intermediate colors, which are formed by overlapping blocks of adjacent wavelength bands, can also be used to generate information about the surface shape. The peak wavelengths of the intermediate colors are different from the peak wavelengths λ1 to λ8. The "different directions" may include directions from two light source units arranged opposite each other in a planar view toward the object.

[0068] In the following, an example in which light-emitting section 11 has four blocks shown in FIG. 2A will be described.

[0069] 1 again, imaging device 20 captures an image of the surface of an object using imaging device 20 in a state where a plurality of illumination lights corresponding to N wavelength bands are irradiated onto the surface of the object from mutually different directions, thereby generating M images (M is a natural number equal to or greater than 4) corresponding to each of the plurality of wavelength bands. It can also be said that imaging device 20 captures an image of an object using imaging device 20 in a state where a plurality of illumination lights are projected onto the object at mutually different positions.

[0070] The imaging device 20 has a configuration similar to that of the imaging device disclosed in Patent Document 2, and is capable of capturing a compressed image including a plurality of pixels. Each of the plurality of pixels included in the compressed image includes information of four or more wavelength bands. The imaging device 20 includes an imaging unit 21 and a communication unit 22. The imaging device 20 may also include a control circuit (not shown) that controls the imaging unit 21 to cause the imaging unit 21 to generate a compressed image.

[0071] The photographing unit 21 has an image sensor 21a. The image sensor 21a is a monochrome photodetector having a plurality of photodetection elements arranged in a matrix. For example, a CCD (Charge-Coupled Device) image sensor, a CMOS (Complementary Metal Oxide Semiconductor) image sensor, an infrared array image sensor, a terahertz array image sensor, or a millimeter-wave array image sensor can be used as the image sensor 21a. Note that the image sensor 21a does not have to be a monochrome photodetector and may be a color photodetector. The wavelength range detectable by the image sensor 21a is not particularly limited and may be, for example, visible light, ultraviolet light, infrared light, terahertz waves, or any combination thereof.

[0072] The communication unit 22 is a communication interface for the imaging device 20 to communicate with the information processing device 30. The communication between the imaging device 20 and the information processing device 30 may be performed by wire or wirelessly. Furthermore, there are no particular limitations on the communication standard.

[0073] 1, the imaging device 20 includes an image sensor 21a, a filter array 21b, and an optical system 23. The image sensor 21a and the filter array 21b constitute the photographing unit 21, but are not limited to this. The configuration of the imaging device 20 including the image sensor 21a, the filter array 21b, and the optical system 23 will be described later with reference to FIG.

[0074] The information processing device 30 is communicably connected to the lighting device 10 and the imaging device 20 via wired and / or wireless communication. The information processing device 30 includes a communication unit 31, a processing unit 32, a control unit 33, a storage unit 34, and a display unit 35.

[0075] The communication unit 31 is a communication interface that enables the information processing device 30 to communicate with each of the lighting device 10 and the imaging device 20. The communication unit 31 includes, for example, a communication circuit (or a communication module).

[0076] The communication unit 31 can acquire compressed images from the imaging device 20 and store them in the storage unit 34. The communication unit 31 functions as an image acquisition unit that acquires compressed images. Note that the communication unit 31 does not have to acquire compressed images directly from the imaging device 20, and may acquire compressed images via another device or the like.

[0077] The processing unit 32 is a component (e.g., a circuit) that performs information processing to acquire information about the surface shape of the surface of the object. For example, the processing unit 32 generates M restored image signals corresponding to multiple wavelength bands by performing a restoration operation on the compressed image signal. Here, the M restored images are M spectral images, and the M restored image signals are M spectral image signals. The M restored images are an example of M images. M may be the same as N or may be greater than N. In other words, M is greater than or equal to N.

[0078] The M restored images represented by the M restored image signals may constitute a hyperspectral image represented by information of four or more wavelength bands. For example, a hyperspectral image is composed of four or more spectral images corresponding to four or more wavelength bands. A hyperspectral image is also called a multi-wavelength image.

[0079] The restoration calculation may be the same as the restoration calculation described in Patent Document 2. Specifically, four or more spectral images may be generated as a hyperspectral image based on the following equation (1).

[0080]

number

[0081] Here, g is data representing the compressed image and is expressed, for example, as a one-dimensional array (i.e., a vector). If the compressed image is an image of n×m pixels, the data g is expressed as a one-dimensional array having n×m elements. f is data representing w spectral images that correspond one-to-one to w wavelength bands and is expressed, for example, as a one-dimensional array. When w spectral images constitute a hyperspectral image, w is an integer greater than or equal to 4.

[0082] Also, f1, f2, . . . f w are the spectral image data corresponding to the first wavelength band W1, the spectral image data corresponding to the second wavelength band W2, ..., the w-th wavelength band W w The spectral image data corresponds to the

[0083] Data f1, f2, . . ., f w Each of these is represented by, for example, a one-dimensional array. If each spectral image is an image of n × m pixels, the data f1, f2, ..., f w are represented by a one-dimensional array with n × m elements, and data f is represented by a one-dimensional array with n × m × w elements. H is a matrix with n × m rows and n × m × w columns, called the system matrix, and corresponds to the mask data.

[0084] The matrix H is the transmission spectrum of the first wavelength band W1, the second wavelength band W2, ..., the wth wavelength band W w The transmittance may be determined based on the transmittance spectrum of the light emitting element.

[0085] The data f that satisfies equation (1) can be estimated using a compressed sensing technique, specifically, by equation (2).

[0086]

number

[0087] Equation (2) expresses the search for data f that minimizes the sum of the first and second terms in the parentheses. Data f can be calculated as the final calculation result data by converging the calculation result data through recursive iteration.

[0088] The first term in the parentheses in equation (2) represents the sum of squares of the difference between data Hf obtained by transforming data f in the estimation process using matrix H and data g, which is a so-called residual term. Here, the sum of squares is used, but instead of the sum of squares, the sum of absolute values, the square root of the sum of squares, etc. may be used.

[0089] The sum of squares of the difference between data Hf and data g is (g1-r1)×(g1-r1)++(g n×m -r n×m )×(g n×m -r n×m ), where g1, , g n×m is g=(g1 g n×m ) T are the elements of data g expressed as r1, . . . , r n×m is Hf=(r1···r n×m ) T is an element of data Hf expressed as

[0090] The second term in the parentheses in equation (2) is a regularization term, sometimes called a stabilization term. Φ(f) represents a constraint on the regularization of f and is a function that reflects the sparse information of the data f. This function has the effect of smoothing or stabilizing the data f. Φ(f) can be expressed, for example, by the discrete cosine transform (DCT), wavelet transform, Fourier transform, total variation (TV), or any combination thereof.

[0091] τ is a weighting coefficient for the regularization term, and corresponds to the influence of regularization in the reconstruction calculation. The larger the value of τ, the greater the influence of regularization, and the stronger the convergence of the solution in the iterative calculation. Conversely, the smaller the value of τ, the less the influence of regularization, and the weaker the convergence of the solution in the iterative calculation.

[0092] Note that if the restoration process refers to a process of generating M restored images corresponding to multiple wavelength bands by performing restoration calculations on a compressed image, the restoration process may be performed while the illumination device 10 is emitting multiple irradiation lights, or may be performed after the illumination device 10 has finished emitting multiple irradiation lights, that is, after the restoration process is performed. When the restoration process is performed after the restoration process is performed, the compressed image may be stored in the storage unit 34, and the compressed image may be read out from the storage unit 34 after the restoration process is performed.

[0093] The control unit 33 is a component (circuit) that controls each component of the information processing device 30. The control unit 33 may further control at least one of light emission by the lighting device 10 and image capture by the imaging device 20. The control unit 33 may output a control signal to the lighting device 10 to cause the lighting device 10 to irradiate a target with a plurality of illumination lights, or may output a control signal to the imaging device 20 to execute image capture.

[0094] The storage unit 34 can store compressed images and / or four or more spectral images (restored images). For example, when the processing unit 32 subsequently performs restoration processing, the storage unit 34 may store compressed images until the restoration processing is performed. The storage unit 34 can also store a restoration table for generating a hyperspectral image from a compressed image. The storage unit 34 can be implemented using, for example, a hard disk drive and / or a solid state drive.

[0095] The display unit 35 is a user interface for displaying information. A compressed image, a monochrome image, an RGB image, a hyperspectral image, or the like is displayed on the display unit 35 by the processing unit 32. The display unit 35 may be, for example, a liquid crystal display (LCD) or an organic light-emitting diode (OLED) display.

[0096] The display unit 35 may display a graphical user interface (GUI) for acquiring information about the surface condition. The display unit 35 may also be a touch panel. A user may input information to the display unit 35. The processing unit 32 may acquire information from the user via the display unit 35. That is, the display unit 35 may be an input / output device. Alternatively, the processing unit 32 may acquire information from the user via an input device different from the display unit 35.

[0097] Note that FIG. 1 illustrates an exemplary functional configuration of the information processing system 100, and the functional configuration of the information processing system 100 is not limited to that illustrated in FIG. 1. For example, part or all of the lighting device 10 and the imaging device 20 may be included in the information processing device 30. Also, for example, the display unit 35 may not be included in the information processing device 30. For example, the display unit 35 may be realized as a separate device (display device) communicatively connected to the information processing device 30. Also, for example, the information processing device 30 may be divided into multiple devices and may be realized by, for example, a cloud server. Also, the imaging device 20 may include a processing unit 32, and the imaging device 20 may be capable of generating a hyperspectral image from a compressed image (for example, a compressed image P shown in FIG. 3, which will be described later).

[0098] Here, the configuration of the imaging device 20 will be further described with reference to Figures 3 to 4E. Figure 3 is a schematic diagram showing the configuration of the imaging device 20 according to this embodiment.

[0099] The imaging device 20 has a configuration similar to that of the imaging device disclosed in Patent Document 2. Specifically, the imaging device 20 includes a control circuit (not shown), an image sensor 21a, a filter array 21b, and an optical system 23.

[0100] The filter array 21b is disposed on the optical path of light incident from the object 200, which is the subject, and in FIG. 3, it is disposed between the optical system 23 and the image sensor 21a. The filter array 21b functions as the encoding element of Patent Document 2. The filter array 21b may be integrated with the image sensor 21a.

[0101] The arrangement of filter array 21b is not limited to the arrangement shown in Fig. 3. For example, filter array 21b may be arranged between optical system 23 and image sensor 21a, but away from image sensor 21a. Alternatively, filter array 21b may be arranged between object 200 and optical system 23. Alternatively, filter array 21b may be arranged within optical system 23.

[0102] The optical system 23 is disposed on the optical path of light incident from the object 200, and is disposed between the object 200 and the filter array 21b in Fig. 3. The optical system 23 includes at least one lens and can form an image of the object 200 on the imaging surface of the image sensor 21a via the filter array 21b.

[0103] The configuration and arrangement of the optical system 23 are not limited to those shown in FIG. 3. For example, the optical system 23 may be disposed between the filter array 21b and the image sensor 21a. Furthermore, for example, the optical system 23 may include a plurality of lenses arranged on the optical path. In this case, the filter array 21b may be disposed between adjacent lenses of the plurality of lenses.

[0104] The filter array 21b includes a plurality of filters. The number of the plurality of filters may be n×m. The n×m filters are 11 ,···,filter nm Contains filters 11 The transmission spectrum S in the wavelength band W1 to Ww 11 ,..., the transmission spectrum S in wavelength band W1 to wavelength band Ww of the filter nm nm may all be different, or the transmission spectrum S11 , ···, transmission spectrum S nm may be the same. Here, the transmission spectrum may refer to the light transmittance spectrum.

[0105] filter 11 The transmittance S of the wavelength band W1 11W1 ,···,filter 11 Transmittance S of the wavelength band Ww 11Ww ,···,filter nm The transmittance S of the wavelength band W1 nmW1 ,..., wavelength band W of filter nm w Transmittance S nmWw may all be different, or the transmittance S 11W1 ,..., transmittance S 11Ww ,..., transmittance S nmW1 ,..., transmittance S nmWw may be the same. w may be an integer of 4 or more. In the present disclosure, transmittance may mean light transmittance.

[0106] Wavelength band W α The transmittance of the filter β in the above formula (3) may be expressed as follows:

[0107]

number

[0108] Wαmin is the wavelength band W α is the minimum wavelength value of the wavelength band W α is the maximum wavelength value, h(λ) is a function indicating the transmission spectrum, and λ is the wavelength.

[0109] In addition, the wavelength band W α The transmittance of the filter β in the wavelength band W is not limited to the formula (3). α The transmittance of the filter β in the wavelength band W may be the transmittance obtained by dividing the formula (3) by (Wαmax-Wαmin). αThe transmittance of the filter β in the wavelength band W α The frequency λ that represents α0 Transmittance h(λ α0 ) may be used. α0 is Wαmin≦λ α0 ≦Wαmax. For example, the wavelength band W α The center frequency may be ((Wαmax−Wαmin) / 2).

[0110] The configuration of filter array 21b will now be described with reference to FIGS. 4A to 4E. FIG. 4A is a schematic diagram showing filter array 21b according to the present embodiment. Filter array 21b includes filters 21b1 and 21b2. FIG. 4B is a diagram showing an example of the transmission spectrum of filter 21b1 according to the present embodiment. FIG. 4C is a diagram showing an example of the transmission spectrum of filter 21b2 according to the present embodiment. FIG. 4D is a diagram showing an example of the transmittance of a first wavelength band W1 of filter array 21b according to the present embodiment. FIG. 4E is a diagram showing an example of the transmittance of a second wavelength band W2 of filter array 21b according to the present embodiment. In FIGS. 4D and 4E, the shading of each region represents the transmittance of the filter, with lighter regions representing higher transmittance and darker regions representing lower transmittance.

[0111] Filter array 21b includes a plurality of filters arranged in a matrix. In the example shown in FIG. 4A, filter array 21b includes 48 filters arranged in 6 rows and 8 columns. Filter 21b1 is the filter arranged in the upper left of the 48 filters, and filter 21b2 is the filter arranged in the lower right of the 48 filters. Note that the number of filters included in filter array 21b is not limited to 48. For example, the number of filters included in filter array 21b may be approximately the same as the number of pixels of image sensor 21a, and may be determined depending on the application within a range from several tens to several tens of millions, for example.

[0112] The wavelength dependencies of the transmittance of the multiple filters included in filter array 21b are different from one another. For example, in filter 21b1, the transmittance of the first wavelength band W1 is significantly lower than the transmittance of the second wavelength band W2. On the other hand, in filter 21b2, the transmittance of the first wavelength band W1 is approximately the same as the transmittance of the second wavelength band W2. In other words, the wavelength dependency of the transmittance of filter 21b1 is different from the wavelength dependency of the transmittance of filter 21b2. Note that only the transmittances of two of the four or more wavelength bands, the first wavelength band W1 and the second wavelength band W2, are illustrated and described here, and the transmittances of the other wavelength bands included in the four or more wavelength bands are not illustrated or described.

[0113] In this way, the imaging device 20 includes an optical filter including multiple regions, each having a different transmission spectrum, and is configured to be able to output a compressed image including information of N wavelength bands by detecting light that has passed through the optical filter. The imaging device 20 may be a hyperspectral camera that can capture spectral images in at least four or more wavelength bands.

[0114] Note that the present invention is not limited to generating an image of four or more wavelength bands using a compressed sensing method with a hyperspectral camera such as the imaging device 20 described above. For example, an image including information on four or more wavelength bands may be generated using a snapshot method in which a single shot is taken. For example, the imaging device 20 may be a camera that detects light corresponding to each of four or more wavelength bands for each pixel of the image sensor 21a and outputs an image including pixels corresponding to each of the four or more wavelength bands. For example, the imaging device 20 may have a configuration similar to that of the imaging device disclosed in Patent Document 3. For example, the imaging device 20 may be configured to include an optical filter (color filter) including multiple regions, each of which has a transmission spectrum corresponding to each of N wavelength bands, and to output an image including multiple pixel values ​​corresponding to each of the N wavelength bands. This allows the imaging device 20 to generate a hyperspectral image.

[0115] [1-2. Operation of information processing system] Next, the operation of the information processing system 100 configured as above will be described with reference to Fig. 5 to Fig. 9. Fig. 5 is a flowchart showing the operation (information processing method) of the information processing system 100 according to this embodiment. Fig. 5 shows the processing executed by the information processing device 30.

[0116] As shown in FIG. 5, first, the control unit 33 of the information processing device 30 outputs an instruction to the light-emitting unit 11 to emit light (S10). The control unit 33 outputs a control signal to the light-emitting unit 11 to cause it to emit multiple illumination lights. The instruction includes an instruction to simultaneously emit multiple illumination lights. For example, when the lighting device 10 has the configuration shown in FIG. 2A, the instruction includes an instruction to simultaneously emit each of the multiple blocks 11a1 to 11a4. Note that the instruction may cause the multiple blocks 11a1 to 11a4 to start emitting light simultaneously, or may cause them to start emitting light at different times. The instruction may include an instruction to cause the multiple blocks 11a1 to 11a4 to emit light so that there is a period during which they are simultaneously emitting light.

[0117] FIG. 6 is a schematic diagram illustrating the irradiation of the object 200 with irradiation light by the lighting device 10 according to the present embodiment. FIG. 6 is a schematic diagram illustrating the surface of the object 200, the surface shape of which is to be measured, viewed from above the surface. The dot hatching within the arrows shown in FIG. 6 indicates the color of the irradiation light, and different dot hatching indicates different colors (or peak wavelengths) of the irradiation light. FIG. 6 also illustrates an example of spectral data of the irradiation light from each block. The surface shape of the object 200 is circular. In plan view, each irradiation light is irradiated from the outside of the object 200 toward (inward of) the object 200.

[0118] 6, four (or more) irradiation lights with different spectra are irradiated from different directions (four directions, top, bottom, left, and right, in FIG. 6) onto the object 200. The different spectra may mean, for example, that the peak wavelengths at which the intensity shown in the spectral data has a maximum value are different from each other, as shown in the spectral data, or that the center wavelengths are different.

[0119] Note that, as long as the direction of irradiation of the irradiation light is known in advance, it is not limited to the directions of up, down, left, and right with respect to the object 200. For example, one of the irradiation lights may be irradiated onto the object 200 from an oblique direction (for example, diagonally upward and to the right) in a plan view.

[0120] 5 again, next, the control unit 33 of the information processing device 30 outputs an instruction to the photographing unit 21 to photograph (S20). The control unit 33 outputs a control signal to the photographing unit 21, thereby causing the photographing unit 21 to photograph the surface of the object 200 while the plurality of irradiation lights are irradiated onto the object 200. The photographing timing is not particularly limited as long as it is a timing when the plurality of irradiation lights are irradiated onto the object 200.

[0121] For example, the control unit 33 may cause the photographing unit 21 to photograph the object 200 while it is moving relative to the imaging device 20 by outputting a control signal to the photographing unit 21 including an instruction to photograph the object 200 while it is moving relative to the imaging device 20. The control unit 33 may also cause the photographing unit 21 to photograph the object 200 while it is not moving relative to the imaging device 20 by outputting a control signal to the photographing unit 21 including an instruction to photograph the object 200 while it is not moving relative to the imaging device 20.

[0122] Next, the processing unit 32 of the information processing device 30 acquires a multi-wavelength image based on the image capture by the imaging device 20 (S30). When the processing unit 32 acquires a compressed image from the imaging device 20, the processing unit 32 may acquire the multi-wavelength image by generating the multi-wavelength image as four or more spectral images based on the compressed image and a restoration table stored in the storage unit 34. The processing unit 32 functions as a first processing unit that generates M spectral images.

[0123] Furthermore, when the imaging device 20 is configured to be able to generate a multi-wavelength image, the processing unit 32 may acquire the multi-wavelength image obtained by photographing from the imaging device 20 via the communication unit 31. Furthermore, the processing unit 32 may acquire the multi-wavelength image based on the imaging device 20 via the communication unit 31 from another device different from the imaging device 20 (for example, a device capable of generating a multi-wavelength image from a compressed image).

[0124] In this way, the processing unit 32 may acquire the multi-wavelength image as a result of its own information processing, or may acquire the multi-wavelength image itself from an external device.

[0125] 6 can also be said to be a state in which multiple illumination lights are projected onto different positions on the object 200. "Different positions on the object 200" means that the multiple illumination lights are irradiated onto the entire surface of the object 200, and the optical axes of the multiple illumination lights are incident on different positions on the object 200. The optical axis is a straight line that coincides with the main emission direction of the light emitted from each block. In this case, the imaging device 20 captures an image of the object in this state.

[0126] Fig. 7 is a schematic diagram showing an example of a multi-wavelength image according to the present embodiment. In Fig. 7, brightness is indicated by dot hatching density, with lower dot hatching density indicating higher brightness. The surface shape of the object 200 is circular (see Fig. 6), and the brightness distribution is also circular.

[0127] As shown in FIG. 7, the processing unit 32 acquires, as multi-wavelength images, a spectral image P1 (FIG. 7(a)) corresponding to illumination light of a peak wavelength λ1, a spectral image P2 (FIG. 7(b)) corresponding to illumination light of a peak wavelength λ2, a spectral image P3 (FIG. 7(c)) corresponding to illumination light of a peak wavelength λ3, and a spectral image P4 (FIG. 7(d)) ​​corresponding to illumination light of a peak wavelength λ4. The spectral image P1 is, for example, an image obtained by extracting a luminance image of the peak wavelength λ1 from a compressed image. The same is true for the spectral images P2 to P4. The processing unit 32 acquires spectral images corresponding to illumination light irradiated from each direction in this way. Note that the distributions of the spectral images P1 to P4 are different because the light is irradiated in different directions.

[0128] Here, a comparison between the spectral images P1 to P4 and spectral images generated by irradiating light sequentially from each direction (spectral images generated by a conventional method) will be described. In the conventional method, it is assumed that irradiating light is also irradiated from above, below, left, and right in a planar view. An example of the conventional method is the method disclosed in Patent Document 1.

[0129] Spectral image P1 is equivalent to an image captured using a conventional method with light shining from above, spectral image P2 is equivalent to an image captured using a conventional method with light shining from the left, spectral image P3 is equivalent to an image captured using light shining from below, and spectral image P4 is equivalent to an image captured using light shining from the right. Therefore, information about the surface shape of object 200 obtained using spectral images P1 to P4 is equivalent to information about the surface condition of object 200 obtained using images using the conventional method. Note that "equivalent" means, for example, that the luminance distributions in the two images are equivalent or similar.

[0130] 5 again, the processing unit 32 generates information about the surface shape of the object 200 based on the multi-wavelength images (M images) acquired in step S30 (S40). The information about the surface shape includes at least one of the curvature, contour, unevenness, reflectance distribution, and dimensions of the surface of the object 200. The method by which the processing unit 32 generates the information about the surface shape of the object 200 is not particularly limited, and any known method may be used. The processing unit 32 functions as a second processing unit that generates information about the surface shape of the object 200.

[0131] An example of a known method for generating information about the surface shape of object 200 will now be described with reference to FIG. 8. FIG. 8 is a schematic diagram for explaining generation of information about the surface shape of object 200 according to this embodiment. FIG. 8 shows how surface 210 of object 200 is imaged by imaging device 20 while being illuminated by light from four different directions. References l1 to l4 indicate the irradiation directions of the irradiated light. For example, direction l1 indicates the direction of the irradiated light from block 11a1, direction l2 indicates the direction of the irradiated light from block 11a2, direction l3 indicates the direction of the irradiated light from block 11a3, and direction l4 indicates the direction of the irradiated light from block 11a4. Note that in FIG. 8, for convenience, the shape of object 200 is shown as a rectangular parallelepiped.

[0132] 8, in a state where the object 200 is irradiated with illumination light beams having different illumination directions, the imaging device 20 captures an image of the object 200 from above. The processing unit 32 calculates the normal vector ("n" in FIG. 8) of the surface 210 of the object 200 by calculation from the spectral images (spectral images P1 to P4 in this case) corresponding to the illumination light beams obtained by the imaging.

[0133] The unit vector indicating one light source direction (illumination direction) is as follows:

[0134]

number

[0135] Also, the unknown normal vector is as follows:

[0136]

number

[0137] The pixel values ​​(luminance values) of four pixels whose coordinates match in the four spectral images P1 to P4 are as follows:

[0138]

number

[0139] Here, as shown in the following equation (4), the pixel value (observation value) can be calculated by the inner product of the unit vector and the normal vector.

[0140]

number

[0141] If the unit vector in the light source direction is known, the normal vector can be calculated by multiplying both sides by the inverse matrix. For example, methods for predicting the unit vector in the light source direction include a method using sparsity and a method using deep learning.

[0142] The processing unit 32 uses equation (4) to calculate the normal vector of each pixel that forms the surface 210 of the object 200. Then, the processing unit 32 performs image processing operations based on the calculated normal vectors, thereby generating information about the surface shape of the object 200.

[0143] As the image processing operation, an appropriate operation may be selected according to the information on the surface shape to be generated. For example, a method using the mean curvature may be used to obtain a gently curved surface shape, and for example, a method using the Gaussian curvature may be used to obtain a surface shape of a curved surface that changes significantly, such as an edge portion.

[0144] 5 again, the display unit 35 displays the information relating to the surface shape generated by the processing unit 32 (S50), thereby making it possible to present the measurement results of the information relating to the surface shape to the user.

[0145] 9 is a diagram showing an example of displaying information about the surface condition according to this embodiment. Fig. 9 shows the results when the peak wavelength λ1 is 470 nm, the peak wavelength λ2 is 550 nm, the peak wavelength λ3 is 620 nm, and the peak wavelength λ4 is 740 nm.

[0146] As shown in FIG. 9, the display unit 35 may display two result images, result images 1 and 2, which are measurement results obtained using image processing operations. Because there are multiple image processing operation methods for obtaining information about the surface shape, the display unit 35 may display two result images showing effective results. Result image 1 represents the curvature of a sphere and is calculated, for example, using a method that uses mean curvature as an image processing operation. Result image 2 represents the contour of the object 200 and is calculated, for example, using a method that uses Gaussian curvature as an image processing operation. Note that the number of images displayed as results is not limited to two, and may be one, or three or more. The user may also set in advance which result of the image processing operation to display. When multiple result images are displayed, the user can select the optimal image that suits their purpose.

[0147] The display unit 35 may also display the spectral images used to calculate the result images 1 and 2 (the four spectral images shown in the used bands in FIG. 9) side by side on the same screen. In the example of FIG. 9, spectral images of wavelength bands of 470 nm, 550 nm, 620 nm, and 740 nm (i.e., spectral images showing the luminance distribution of the same wavelength as the peak wavelength of the irradiated light) are displayed. This allows the user to find any defects in any of the spectral images (any of the wavelength bands).

[0148] (Embodiment 2) The information processing system according to this embodiment will be described below with reference to Figures 10 to 13. Note that the following description will focus on differences from the first embodiment, and descriptions of the same or similar aspects as those in the first embodiment will be omitted or simplified.

[0149] The configuration of the information processing system according to this embodiment is the same as that of the information processing system 100 according to the first embodiment, and therefore a description thereof will be omitted. In the following description, the reference numerals of the information processing system 100 according to the first embodiment will be used.

[0150] [2-1. Spectral overlap] First, the overlap of spectra in the information processing system 100 according to this embodiment will be described with reference to Fig. 10. Fig. 10 is a schematic diagram for explaining the overlap of spectra according to this embodiment. This embodiment will also be described assuming that the lighting device 10 including the light-emitting unit 11a shown in Fig. 2A is used. Fig. 10 shows how illumination hits the object, including the overlap of adjacent spectra.

[0151] It is known that the accuracy of information about the surface shape improves as the number of illumination colors of the illumination light used to generate the information increases. Furthermore, there are illumination colors that cannot be used depending on the color of the object 200, such as when the color of the illumination light is similar to the color of the object 200. Therefore, it is preferable for the illumination device 10 to have a large number of illumination colors. Therefore, in this embodiment, an example will be described in which, without increasing the number of blocks (number of wavelength bands) of the illumination device 10, images are acquired under conditions in which illumination light is irradiated from more different directions than the number of illumination light to be irradiated, and information about the surface shape is generated using spectral images that are greater than the number of illumination light. The number of illumination light, the number of blocks, and the number of different directions are synonymous.

[0152] As shown in FIG. 10, this embodiment assumes a state in which illumination light having different wavelength bands is irradiated from seven directions. Block 11a5 is a block that assumes that light (overlapping light) obtained by mixing light with peak wavelength λ1 and light with peak wavelength λ2 is irradiated onto object 200 from the upper left diagonal direction. The spectral data of the illumination light from block 11a5 is data obtained by adding together the spectral data of light with peak wavelength λ1 and the spectral data of light with peak wavelength λ2 (adding together the intensities for each wavelength). The illumination direction of light from block 11a5 is based on the illumination directions of blocks 11a1 and 11a2. Note that illumination device 10 does not include block 11a5 (i.e., a component that emits light in the wavelength band indicated by λ1+λ2).

[0153] Similarly, block 11a6 is a block that assumes that light obtained by mixing light of peak wavelength λ2 and light of peak wavelength λ3 (overlapping light) is irradiated onto object 200 from the diagonally lower left, and the spectral data is data obtained by adding together the spectral data of the light of peak wavelength λ2 and the spectral data of the light of peak wavelength λ3. Block 11a7 is a block that assumes that light obtained by mixing light of peak wavelength λ3 and light of peak wavelength λ4 (overlapping light) is irradiated onto object 200 from the diagonally lower right, and the spectral data is data obtained by adding together the spectral data of the light of peak wavelength λ3 and the spectral data of the light of peak wavelength λ4. Note that lighting device 10 does not include blocks 11a6 and 11a7.

[0154] By assuming the above, for example, the illumination device 10 does not need to be equipped with a light source corresponding to the wavelength band of overlapping light, which is a mixture of light with a peak wavelength λ1 and light with a peak wavelength λ2. In other words, it is possible to augment the spectral images by image processing while still using the current illumination device 10. By augmenting, for example, it is possible to increase the number of multi-wavelength images obtained from one compressed image to a number greater than the number of illuminations. In the example of FIG. 10, seven spectral images can be generated.

[0155] In this way, the number of spectral images generated for information about the surface shape may be greater than the number of illuminations (e.g., N) of the illumination device 10. For example, the generated spectral images may include spectral images corresponding to wavelengths between peak wavelengths λ1 and λ2.

[0156] [2-2. Operation of information processing system] Next, the operation of the information processing system 100 configured as described above will be described with reference to FIGS. 11 to 13. FIG. 11 is a flowchart showing the operation (information processing method) of the information processing system 100 according to this embodiment. The flowchart shown in FIG. 11 includes steps S60 and S70 in addition to the flowchart shown in FIG. 5. When a spectral image corresponding to the color of the object 200 is used, the contrast of the spectral image is low, which can reduce the accuracy in generating information about the surface shape of the object 200. The flowchart shown in FIG. 11 solves this problem. The contrast corresponds to an index value that indicates the variation in distribution between the pixel values ​​of the spectral image and the number of pixels having that pixel value.

[0157] As shown in FIG. 11, the processing unit 32 acquires a multi-wavelength image (S30). The multi-wavelength image includes M (M>N) spectral images. Here, the M spectral images include N first images corresponding to N wavelength bands (for example, four wavelength bands, i.e., a wavelength band including λ1, a wavelength band including λ2, a wavelength band including λ3, and a wavelength band including λ4, as shown in FIG. 10), and one or more second images (three second images in this case) that are one or more spectral images corresponding to wavelength bands of overlapping light in which at least two irradiating lights corresponding to at least two wavelength bands of the N wavelength bands overlap (for example, three wavelength bands, i.e., a wavelength band including λ1+λ2, a wavelength band including λ2+λ3, and a wavelength band including λ3+λ4, as shown in FIG. 10). The N first images and one or more second images are an example of M images.

[0158] Next, the processing unit 32 determines whether or not there are any low-contrast images among the multi-wavelength images (here, N first images and one or more second images) (S60), and if there are any low-contrast images (Yes in S60), excludes the low-contrast spectroscopic images from the spectroscopic images used to generate information about the surface shape, i.e., spectroscopic images that may reduce the accuracy of generating information about the surface shape (S70). On the other hand, if there are no low-contrast images (No in S60), the processing unit 32 proceeds to step S40. This makes it possible to prevent low-contrast images from being used to generate information about the surface condition.

[0159] The spectroscopic image excluded in step S70 may be stored in the storage unit 34 with information indicating that the contrast is low added thereto, or may be deleted.

[0160] Fig. 12 is a flowchart showing the detailed operation (information processing method) of step S60 shown in Fig. 11. Fig. 12 shows an example of determining whether an image has low contrast based on the distribution of a luminance histogram (i.e., luminance variation).

[0161] As shown in FIG. 12 , the processing unit 32 acquires a luminance histogram of the object 200 for each of the multiple spectral images included in the multi-wavelength image (S61). When acquiring the luminance histogram of the object 200, the processing unit 32 may identify the region of the object 200 in the spectral image. For example, the processing unit 32 may identify the region of the object 200 in the spectral image by applying a mask to regions other than the object 200. A method for generating the mask may involve, for example, extracting the contour of the object 200 by binarization and transmitting only the portion of the spectral image surrounded by the contour (using the pixel values ​​of the surrounded portion of the spectral image), thereby acquiring the luminance histogram. This makes it possible to acquire the luminance histogram using only the pixel values ​​of the region of the object 200. The luminance histogram is an example of information indicating the distribution of pixel values ​​and the number of pixels.

[0162] Note that sudden changes in contrast can result in noise, so when processing images (e.g., acquiring a brightness histogram), pixel values ​​may be normalized to align the pixel values ​​of the multiple spectral images used in the processing.

[0163] Next, the processing unit 32 determines whether or not there is a uniform distribution (luminance distribution) based on the luminance histogram (S62). For example, the processing unit 32 may derive a standard deviation from the luminance histogram for each of the M images based on pixel values ​​contained in the image, compare the standard deviation with a predetermined threshold stored in memory, and determine that there is no uniform distribution (there is little variation in luminance) if the standard deviation is less than the predetermined threshold. The standard deviation is an example of an index value.

[0164] The predetermined threshold may be, for example, 0.2, a value between 0.2 and 0.5, or a value equal to or greater than 0.5. The predetermined threshold may be, for example, acquired in advance and stored in the storage unit 34. The predetermined threshold may be different for each image (for example, for each peak wavelength), or a common value may be used. When the processing unit 32 acquires illumination data, the processing unit 32 may determine the predetermined threshold for each image (for example, for each peak wavelength) based on a table in which peak wavelengths are associated with predetermined thresholds. The table may be set in advance and stored in the storage unit 34. The table may be set based on characteristics of the object 200, such as the color and shape of the object 200. The processing unit 32 may receive input of the predetermined threshold from the user each time.

[0165] The index used to determine whether or not there is a certain distribution is not limited to the standard deviation, and may be another index that indicates the variation in brightness, such as variance.

[0166] Next, if the processing unit 32 determines that there is a certain distribution (Yes in S62), it determines that the contrast of the spectroscopic image is not low (S63), and if it determines that there is no certain distribution (No in S62), it determines that the contrast of the spectroscopic image is low (S64).

[0167] Next, the processing unit 32 determines whether the determination of whether the contrast is low or not has been completed for all images (all spectral images) (S65). If the determination of whether the contrast is low or not has been completed for all images (Yes in S65), the processing unit 32 proceeds to step S66. If the determination of whether the contrast is low or not has not been completed for all images (No in S65), the processing unit 32 proceeds to step S61 and performs the processing from step S61 onwards for the next spectral image.

[0168] Next, the processing unit 32 determines whether there is an image (spectral image) that is determined to have low contrast (S66), and if it determines that there is an image with low contrast (Yes in S66), proceeds to Yes in step S60 shown in Figure 11, and if it determines that there is no image with low contrast (No in S66), proceeds to No in step S60 shown in Figure 11.

[0169] In this way, the processing unit 32 determines, from among the M images, a plurality of images (one example of second target images) to be used for generating information related to the surface shape, based on the index values ​​of each of the M images. The plurality of second target images includes four or more of the M images. The determined plurality of second target images can also be said to be images among the M images that have index values ​​equal to or greater than a predetermined threshold. In other words, the processing unit 32 may determine, from among the M images, images whose index values ​​are equal to or greater than a predetermined threshold as the plurality of second target images.

[0170] In steps S62 to S64, it is determined whether the contrast is low or not, but it may be determined which of three or more levels of contrast (for example, high, medium, low contrast) the contrast is.

[0171] 13 is a schematic diagram showing an example of a multi-wavelength image including a low-contrast image according to the present embodiment, which is acquired when the color of the irradiating light including the peak wavelength λ2 is similar to the color of the object 200.

[0172] As shown in FIG. 13, only the spectral image P2a has a higher reflectance of the irradiated light than the other spectral images, so the entire object 200 appears bright and has low contrast. In FIG. 13, the brightness and low contrast of the spectral image P2a are indicated without dot hatching. Images with low contrast are more likely to introduce quantization errors, which may result in low accuracy of information regarding the surface shape. In step S70 shown in FIG. 11, the processing unit 32 excludes the spectral image P2a shown in FIG. 13, thereby making it possible to select a spectral image that is advantageous for obtaining accurate information regarding the surface shape. Furthermore, because contrast is used, the spectral image P2a can be excluded even if the color of the object 200 is not known in advance.

[0173] The method of determining whether an image has low contrast is not limited to the method shown in FIG. 12 . For example, the determination may be performed based on an image (color image) of the object 200 captured by an imaging device other than the imaging device 20. The processing unit 32 may determine wavelengths (or wavelength bands) at which the image contrast becomes low based on an image of the object 200 generated by the other imaging device, and exclude spectral images corresponding to the determined wavelengths (or wavelength bands) from the multi-wavelength image captured by the imaging device 20. An example of the other imaging device is, but is not limited to, an RGB camera. The processing unit 32 may also receive, from the user, via the input UI, a selection of images to be used for generating information about the surface shape. In other words, the processing unit 32 may receive, from the user, via the input UI, a selection of images to be excluded from images to be used for generating information about the surface shape.

[0174] The method of extracting the spectral images used to generate information about the surface shape from the multi-wavelength image is not limited to using contrast. The processing unit 32 may extract the spectral images used to generate information about the surface shape, for example, using illumination data indicating the wavelengths of the illumination light emitted by the illumination device 10. For example, in the case shown in FIG. 10, the processing unit 32 may extract four spectral images from the seven spectral images based on the illumination data for each of blocks 11a1 to 11a7. The illumination data includes information about N wavelength bands associated with multiple illumination lights.

[0175] The processing unit 32 acquires, for example, spectral data and irradiation direction of each irradiated light as the illumination data. Here, it is assumed that 450 nm, 500 nm, 550 nm, and 600 nm are acquired as center wavelengths. The processing unit 32 may acquire the illumination data from the illumination device 10 or a device that manages illumination data via the communication unit 31. Alternatively, the illumination data may be acquired in advance and stored in the storage unit 34.

[0176] Then, the processing unit 32 extracts spectral images to be used for generating information about the surface shape from the acquired multi-wavelength image based on the illumination data. For example, if the multi-wavelength image includes multiple spectral images at 10-nm intervals, such as 450 nm, 460 nm, 470 nm, 480 nm, 490 nm, 500 nm, ... 700 nm, the processing unit 32 references the illumination data and extracts spectral images at the central wavelengths of the irradiated light, 450 nm, 500 nm, 550 nm, and 600 nm, from the acquired multi-wavelength image, and generates information about the surface shape based on the extracted spectral images. The extracted spectral images are an example of first target images. In this case, in step S40, information about the surface shape is generated based on the multiple first target images.

[0177] If the central wavelength of the illumination and the wavelength of the spectral image do not match, the processing unit 32 may extract a spectral image having a wavelength closest to the central wavelength. For example, if the central wavelength of the illumination and the wavelength of the spectral image do not match, the processing unit 32 may extract a spectral image having the smallest difference between the central wavelength and the wavelength of the spectral image.

[0178] It should be noted that it may be determined in advance which wavelengths to use from the multi-wavelength image. For example, the storage unit 34 may store a table in which illumination data of the illumination device 10 is associated with information indicating which wavelengths of spectral images to use. The processing unit 32 may also acquire, in advance, an input from the user via the input UI regarding which wavelengths of spectral images to use from the multi-wavelength image, and perform settings regarding the spectral images to be extracted. In this way, the processing unit 32 may extract multiple first target images corresponding to each of the N wavelength bands from M images.

[0179] It can also be said that in step S40, information about the surface shape is generated based on N first images and one or more second images. The information about the surface shape may be generated only from the N first images, only from one or more second images, or from four or more images that are a mixture of first images and second images.

[0180] The processing unit 32 may generate information about the surface shape using, for example, all seven spectroscopic images.

[0181] (Other embodiments) The information processing method according to one or more aspects has been described above based on Embodiments 1 and 2, but the present disclosure is not limited to Embodiments 1 and 2. As long as it does not deviate from the spirit of the present disclosure, various modifications that a person skilled in the art can conceive of to the present embodiment and embodiments constructed by combining components of different embodiments may also be included in the present disclosure.

[0182] For example, in the above-described first and second embodiments, a plurality of illumination lights corresponding to N wavelength bands are irradiated onto the surface of an object from different directions, and M images corresponding to the plurality of wavelength bands are generated by capturing an image of the surface of the object using an imaging device while the object is moving. However, the present disclosure is also effective when the object is stationary. In other words, the technology of the present disclosure may be used to generate M images corresponding to the plurality of wavelength bands by capturing an image of the surface of the object using an imaging device while the object is stationary, and a plurality of illumination lights corresponding to N wavelength bands are irradiated onto the surface of the object from different directions, and the image of the surface of the object is captured. This allows M images to be generated in one shot, thereby shortening the inspection time for the surface shape.

[0183] Furthermore, in the above-described first and second embodiments, in the process of performing judgment or display using the peak wavelength, the center wavelength may be used instead of the peak wavelength. Furthermore, in the above-described first and second embodiments, in the process of performing judgment or display using the center wavelength, the peak wavelength may be used instead of the center wavelength.

[0184] In the first and second embodiments, each component may be configured with dedicated hardware, or may be realized by executing a software program suitable for each component. Each component may be realized by a program execution unit such as a CPU or processor reading and executing a software program recorded on a recording medium such as a hard disk or semiconductor memory.

[0185] The order in which the steps in the flowchart are executed is merely an example for specifically explaining the present disclosure, and an order other than the above may be used. Also, some of the steps may be executed simultaneously (in parallel) with other steps, or some of the steps may not be executed.

[0186] The division of functional blocks in the block diagram is an example, and multiple functional blocks may be realized as a single functional block, one functional block may be divided into multiple blocks, or some functions may be moved to another functional block.Furthermore, the functions of multiple functional blocks having similar functions may be processed in parallel or time-shared by a single piece of hardware or software.

[0187] Furthermore, the information processing device according to the above-described embodiments may be realized as a single device or may be realized by multiple devices. When the information processing device is realized by multiple devices, the components of the device may be distributed in any manner among the multiple devices. When the information processing device is realized by multiple devices, the communication method between the multiple devices is not particularly limited, and may be wireless communication or wired communication. Furthermore, wireless communication and wired communication may be combined between the devices.

[0188] Furthermore, each of the components described in the first and second embodiments above may be implemented as software or, typically, as an LSI, which is an integrated circuit. These components may be individually integrated into a single chip, or some or all of them may be integrated into a single chip. While the term "LSI" is used here, it may also be referred to as an IC, system LSI, super LSI, or ultra LSI depending on the level of integration. Furthermore, the integration method is not limited to LSI; it may also be implemented using a dedicated circuit (a general-purpose circuit that executes a dedicated program) or a general-purpose processor. It is also possible to use a field programmable gate array (FPGA), which can be programmed after LSI fabrication, or a reconfigurable processor, which allows the connection or settings of circuit cells within an LSI to be reconfigured. Furthermore, if an integrated circuit technology that can replace LSI emerges due to advances in semiconductor technology or a derivative technology, that technology may naturally be used to integrate the components.

[0189] A system LSI is an ultra-multifunctional LSI manufactured by integrating multiple processing units on a single chip, and is specifically a computer system consisting of a microprocessor, ROM (Read Only Memory), RAM (Random Access Memory), etc. Computer programs are stored in the ROM. The system LSI achieves its functions when the microprocessor operates in accordance with the computer program.

[0190] Furthermore, one aspect of the present disclosure may be a computer program that causes a computer to execute each of the characteristic steps included in the information processing method shown in any one of FIG. 5, FIG. 11, and FIG.

[0191] Furthermore, for example, the program may be a program to be executed by a computer. Another aspect of the present disclosure may be a computer-readable non-transitory recording medium on which such a program is recorded. For example, such a program may be recorded on a recording medium and distributed or circulated. For example, the distributed program may be installed in a device having another processor, and the program may be executed by the processor, thereby causing the device to perform each of the above processes. [Industrial Applicability]

[0192] The present disclosure is applicable to an information processing method for restoring an image, and can be used in image processing systems, imaging systems, camera systems, analysis systems, recognition systems, and the like. [Explanation of symbols]

[0193] 10. Lighting equipment 11, 11a, 11b Light-emitting part 11a1, 11a2, 11a3, 11a4, 11a5, 11a6, 11a7, 11b1, 11b2, 11b3, 11b4, 11b5, 11b6, 11b7, 11b8 Blocks (light source section) 20 Imaging device 21 Photography Department 21a Image sensor 21b Filter Array 21b1, 21b2 filters 22, 31 Communications Department 23 Optical system 30 Information processing equipment 32 Processing section (first processing section, second processing section) 33 Control Unit 34 Storage section 35 Display section 100 Information Processing Systems 200 objects 210 Surface P compressed image P1, P2, P2a, P3, P4 Spectral images (images) λ1, λ2, λ3, λ4, λ5, λ6, λ7, λ8 peak wavelengths

Claims

1. Irradiating a surface of an object with a plurality of irradiation lights corresponding to N wavelength bands (N is a natural number equal to or greater than 4) using a light source; capturing an image of the surface of the object using an imaging device in a state in which the plurality of illumination lights corresponding to the N wavelength bands are irradiated onto the surface of the object from mutually different directions, thereby generating M images (M is a natural number equal to or greater than 4) corresponding to the plurality of wavelength bands; generating information about the surface shape of the object based on the M images; Information processing methods.

2. The number of the M images to be generated is greater than the N images. The information processing method according to claim 1 .

3. determining a plurality of first target images from the M images corresponding to each of the N wavelength bands; the information about the surface shape is generated based on the plurality of first object images; The information processing method according to claim 2 .

4. acquiring illumination data including information on the N wavelength bands associated with the plurality of illumination lights; the plurality of first target images are determined based on the illumination data. The information processing method according to claim 3 .

5. The M images include N first images corresponding to the N wavelength bands and one or more second images corresponding to wavelength bands of overlapping light in which at least two illumination lights corresponding to at least two wavelength bands among the N wavelength bands are overlapped, the information about the surface shape is generated based on the N first images and the one or more second images; The information processing method according to any one of claims 1 to 4.

6. Irradiating the surface of the object includes irradiating the surface of the object with the plurality of illumination lights from the different directions using an illumination device that does not have a light source corresponding to the wavelength bands of the overlapping light. The information processing method according to claim 5 .

7. deriving an index value for each of the M images based on pixel values ​​contained in the image; determining, from the M images, a plurality of second target images to be used for generating information about the surface shape based on the index values ​​of the M images; The information processing method according to any one of claims 1 to 4.

8. the index value indicates a variation in distribution of the pixel values ​​and the number of pixels, determining the plurality of second target images to be used for generating the information about the surface shape includes determining, from the M images, images whose index values ​​are equal to or greater than a predetermined threshold value as the plurality of second target images; The information processing method according to claim 7.

9. The information about the surface shape includes at least one of the curvature, contour, unevenness, reflectance distribution, and size of the object. The information processing method according to any one of claims 1 to 4.

10. the imaging device includes an optical filter including a plurality of regions; each of the plurality of regions has a different transmission spectrum from the others; the imaging device detects light that has passed through the optical filter and outputs a compressed image including information on the N wavelength bands. The information processing method according to any one of claims 1 to 4.

11. the imaging device includes an optical filter including a plurality of regions; a transmission spectrum of each of the plurality of regions having a peak corresponding to each of the N wavelength bands; the imaging device outputs an image including a plurality of pixel values ​​corresponding to each of the N wavelength bands. The information processing method according to any one of claims 1 to 4.

12. an illumination device including a plurality of light source units for irradiating the plurality of illumination lights, the plurality of light source units being arranged so as to surround the object in a plan view, and irradiating the surface of the object with the plurality of illumination lights from the mutually different directions; the different directions include directions from each of two light source units arranged opposite to each other in the plan view toward the object; The information processing method according to any one of claims 1 to 4.

13. The imaging of the surface of the object using the imaging device is performed while the object is moving relative to the imaging device. The information processing method according to any one of claims 1 to 4.

14. a control unit that uses a light source to irradiate a surface of the object with a plurality of irradiation lights corresponding to N wavelength bands (N is a natural number equal to or greater than 4); a first processing unit that generates M images (M is a natural number equal to or greater than 4) corresponding to the plurality of wavelength bands by capturing an image of the surface of the object using an imaging device in a state in which the plurality of illumination lights corresponding to the N wavelength bands are irradiated onto the surface of the object from mutually different directions; a second processing unit that generates information about the surface shape of the object based on the M images, Information processing device.

15. A program for causing a computer to execute the information processing method according to any one of claims 1 to 4.

Images