Imaging system and control method thereof

Abstract

To provide a technology for stably extracting polarization information regardless of subject or situation.SOLUTION: A processing device comprises: control means for controlling one or both of an illumination device and a polarization camera in order to image a transparent object under different imaging conditions; image acquisition means for acquiring a plurality of original images captured under different imaging conditions; polarization extraction means for executing a polarization extraction process on each of the plurality of original images to extract polarization information for each pixel, which is information relating to polarized light originating from specular reflection on the transparent object; and image generation means for synthesizing multiple pieces of polarization information extracted from each of the plurality of original images by the polarization extraction process, and generating a polarization information image by converting the polarization information into an image.SELECTED DRAWING: Figure 7

Description

[Technical Field]

[0001] The present invention relates to techniques for imaging transparent objects. [Background technology]

[0002] There is a known technology for imaging transparent objects using a polarization camera that has an imaging element in which polarizers with different transmission axis directions are regularly arranged (see Patent Document 1). This technology is expected to be applied to, for example, the detection and inspection of objects made of glass or transparent resin. [Prior art documents] [Patent documents]

[0003] [Patent Document 1] Japanese Patent Publication No. 2020-17688 Summary of the Invention [Problem to be solved by the invention]

[0004] When light is shone on a transparent object, some of the light is specularly reflected from the surface of the object. Because specularly reflected light has polarization properties, it can be observed with a polarization camera, and the surface (reflective surface) of the transparent object can be imaged based on the observation results.

[0005] However, polarization cameras are subject to not only specular reflections but also diffuse reflections from the subject and background, which can cause noise. Furthermore, if the distance between the light source and the subject is too great, the specular reflections can be very weak, making it difficult to extract polarization information. Conversely, if the light source and the subject are too close, the specular reflections can be too bright, resulting in overexposure (reflected light exceeding the dynamic range of the polarization camera), making it impossible to extract polarization information. However, because the shape and reflective properties of transparent objects, the background conditions, and the relative distance and relative position between the transparent object and the light source vary depending on the equipment placement and subject selection, it is difficult to develop a system that can reliably extract polarization information for all subjects and all situations. Therefore, reliable detection and inspection using polarization cameras typically requires adjusting (finishing) the imaging conditions depending on the target subject and situation. However, such adjustments require advanced skills and knowledge, as well as trial and error, and the manual effort hinders labor-saving and automation of product detection and inspection. Furthermore, adjusting the imaging conditions can be extremely difficult. For example, adjusting the brightness of the lighting to suit one part of the field of view can result in overexposure or underexposure in other parts. This type of problem can become more pronounced when there are multiple subjects in the field of view of the polarization camera, and each subject is at a different distance or position relative to the lighting.

[0006] The present invention has been made in view of the above-mentioned circumstances, and an object of the present invention is to provide a technique that can stably extract polarization information regardless of the subject or situation. [Means for solving the problem]

[0007] The present invention provides an imaging system for imaging a transparent object, comprising an illumination device for illuminating the transparent object, a polarization camera having an imaging element in which polarizers with different transmission axis directions are regularly arranged, and a processing device, wherein the processing device comprises a control means for controlling either or both of the illumination device and the polarization camera in order to photograph the transparent object under different photographing conditions, an image acquisition means for acquiring a plurality of original images photographed under different photographing conditions, a polarization extraction means for executing a polarization extraction process for each pixel of each of the plurality of original images to extract polarization information, which is information regarding polarization resulting from specular reflection on the transparent object, and a polarization extraction means for extracting polarization information, which is information regarding polarization resulting from specular reflection on the transparent object, from each pixel of the plurality of original images. and an image generation means for synthesizing a plurality of pieces of polarization information extracted from each of the plurality of original images through processing, and generating a polarization information image by visualizing the polarization information.

[0008] Depending on the arrangement of equipment and the selection of the subject, the shape and reflective properties of transparent objects, the background conditions, and the relative distance and relative positional relationship between the transparent object and the lighting vary. Therefore, the imaging system according to the present invention captures multiple images of the same transparent object under different shooting conditions to obtain multiple original images. If any of the shooting conditions match, it is expected that at least one original image suitable for extracting polarization information of the transparent object will be obtained. Therefore, by using the polarization information extracted from each of the multiple original images, it becomes possible to stably extract polarization information regardless of the subject or situation.

[0009] The image generating means may perform an operation of selecting, for each pixel, the polarization information with the highest reliability from the plurality of polarization information extracted from each of the plurality of original images, and generate the polarization information image using the polarization information selected for each pixel.

[0010] A high-quality polarization information image can be generated by collecting reliable polarization information from multiple source images. However, the polarization information extracted from the same source image may not be selected for every pixel. This is because the shape, orientation, and reflective properties of the reflective surface, the intensity of the illumination light reaching the reflective surface, and the influence of diffuse reflection vary depending on the position within the field of view of the polarization camera. Therefore, by selecting the optimal polarization information for each pixel and using it to generate the final polarization information image, a highly reliable polarization information image can be obtained regardless of the position within the field of view. Furthermore, this method can accurately extract the polarization information of all objects even when there are multiple objects within the field of view of the polarization camera (or even when multiple objects with different shapes, reflective properties, or postures are mixed together).

[0011] The image generating means may consider the strongest degree of polarization among the plurality of pieces of polarization information to be the most reliable polarization information. "Strongest degree of polarization" means that the polarization component is most clearly observed. The stronger the degree of polarization (the more clearly the polarization component is observed), the more reliable the polarization information can be considered. Therefore, by selecting the strongest degree of polarization from the plurality of pieces of polarization information obtained for one pixel, it is possible to obtain the most reliable polarization information for that pixel (i.e., the area on the reflective surface of the transparent object corresponding to that pixel).

[0012] The control means may control either or both of the lighting device and the polarization camera so as to obtain multiple original images with different brightness levels. If the specular reflected light is too weak, it becomes difficult to extract polarization information, and if the specular reflected light is too strong, it becomes impossible to extract polarization information due to overexposure. Therefore, in order to accurately extract polarization information, it is essential to observe specular reflected light of an appropriate intensity. By capturing multiple original images with different brightness levels as in the present invention, it is expected that specular reflected light of an appropriate intensity can be observed in one of the original images, and that highly reliable polarization information can be extracted from the original image.

[0013] The control means may perform control to vary the "illumination intensity of the lighting device" as the imaging condition, or may perform control to vary the "exposure time of the polarization camera" as the imaging condition, or may perform control to vary the "gain of the polarization camera" as the imaging condition. The control means may simultaneously control two or more of the "illumination intensity of the lighting device," "exposure time of the polarization camera," and "gain of the polarization camera."

[0014] The control means may be configured to obtain a plurality of original images with different lighting directions for the transparent object. The lighting device may be controlled in accordance with the lighting direction. Changing the lighting direction changes the state of diffuse reflection in the background, etc. Therefore, by selecting from multiple original images captured with different lighting directions the image that has the least influence of the diffuse reflection component (i.e., noise) and in which the polarization component is most clearly visible, it becomes possible to extract more reliable polarization information.

[0015] The lighting device may have a plurality of light sources that can be turned on independently, and the control means may change the lighting direction of the transparent object by switching the light sources that are turned on. By using such a lighting device, the lighting direction can be switched quickly with simple control, thereby shortening the takt time of photography.

[0016] The polarization extraction process may include a noise reduction process for removing or reducing noise contained in the calculation result after calculating the polarization information for each pixel of the original image. By performing the noise reduction process, high-quality polarization information can be extracted.

[0017] The noise removal process may include a process of comparing the polarization information of a target pixel and its surrounding pixels to determine whether the polarization information of the target pixel and the surrounding pixels is similar or dissimilar, and deleting the polarization information of the target pixel if it is determined that the polarization information is dissimilar. This process makes it possible to remove unreliable polarization information.

[0018] The noise removal process may include a process of comparing the polarization information of a target pixel and its surrounding pixels to determine whether the polarization information of the target pixel and the surrounding pixels is similar or dissimilar, and if it is determined that the polarization information of the target pixel is similar, replacing the polarization information of the target pixel with the polarization information of a pixel selected from the target pixel and the surrounding pixels. This process smooths the polarization information, thereby further improving the signal-to-noise ratio of the polarization information.

[0019] The present invention may also provide a control method for an imaging system comprising an illumination device, a polarization camera having an imaging element in which polarizers with different transmission axis directions are regularly arranged, and a processing device, the control method comprising the steps of: photographing a transparent object under different shooting conditions by controlling either or both of the illumination device and the polarization camera; importing a plurality of original images photographed under different shooting conditions into the processing device; and executing a polarization extraction process by the processing device for each pixel of each of the plurality of original images to extract polarization information, which is information regarding polarization resulting from specular reflection on the transparent object; synthesizing the plurality of pieces of polarization information extracted from each of the plurality of original images by the polarization extraction process; and generating a polarization information image in which the polarization information is visualized.

[0020] The present invention may provide a program for causing a processor to execute each step of the above control method.

[0021] The present invention may be understood as an imaging system having at least some of the above means, or as a processing device, control device, or image processing device. The present invention may also be understood as a device that detects or inspects transparent objects using polarization information images obtained by an imaging system. The present invention may also be understood as a control method, image processing method, detection method, or inspection method for an imaging system that includes at least some of the above processes, or as a program for realizing such a method or a recording medium on which the program is recorded. The above means and processes can be combined with each other as much as possible to constitute the present invention. [Effects of the Invention]

[0022] According to the present invention, polarization information can be extracted stably regardless of the subject or situation. [Brief explanation of the drawings]

[0023] [Figure 1] FIG. 1 is a diagram schematically showing the overall configuration of an imaging system according to an embodiment of the present invention. [Figure 2] FIG. 2 is a diagram showing a schematic structure of a polarization camera. [Figure 3] FIG. 3A is a diagram schematically illustrating the structure of a lighting device, and FIGS. 3B and 3C are diagrams illustrating other configuration examples of the lighting device. [Figure 4] FIG. 4 is a block diagram showing the logical configuration (functional configuration) of the processing device. [Figure 5] 5A and 5B are diagrams illustrating specular reflection. [Figure 6] FIG. 6 is a diagram illustrating a method for extracting polarization components from an image captured by a polarization camera. [Figure 7] FIG. 7 is a flowchart showing an example of the operation of the imaging system. [Figure 8] FIG. 8 shows an example of a user interface for inputting image processing parameters. [Figure 9] FIG. 9 is a flowchart of the polarization information calculation process. [Figure 10] 10A to 10D are diagrams showing examples of neighboring pixels. [Figure 11] FIG. 11A is a diagram showing an example of specular reflection at a curved surface portion, and FIG. 11B is a diagram showing an example of selecting a mask size when the range into which polarized light is incident is narrow. [Figure 12] FIG. 12 is a diagram showing an example in which a large mask size is suitable. [Figure 13] FIG. 13 is a flowchart of the noise removal process. [Figure 14] FIG. 14 is a diagram showing an example of a comparison range. [Figure 15] FIG. 15 is a flowchart of the image generation process. [Figure 16] FIG. 16A is a diagram showing a state in which a syringe placed in a nest is observed by a zenith camera, and FIG. 16B is a diagram for explaining non-uniformity in the intensity of illumination light depending on the position within the field of view. [Figure 17] FIG. 17 shows an example of the difference between the polarization intensity images obtained when the illumination intensity is changed stepwise, and a polarization intensity image obtained by combining the images. [Figure 18] 18A and 18B are diagrams for explaining the second embodiment. DETAILED DESCRIPTION OF THE INVENTION

[0024] First Embodiment (System Configuration) 1 shows a schematic diagram of the overall configuration of an imaging system according to a first embodiment of the present invention. This imaging system 1 is a system for capturing images of transparent objects using a polarization camera 10, and is used, for example, to detect and inspect transparent objects on a factory production line.

[0025] The imaging system 1 mainly comprises a polarization camera 10, an illumination device 11, a processing device 12, and a stage 13. The polarization camera 10 is an imaging means having an imaging element with an array of polarizers. The illumination device 11 is a light source for irradiating illumination light L onto a subject (transparent object) W placed on the stage 13. The processing device 12 is a device that controls the entire imaging system 1 and performs information processing using images captured by the polarization camera 10. The stage 13 is a device for placing or holding the subject W.

[0026] The polarization camera 10 is positioned so as to capture an image of the subject W placed on the stage 13 from the zenith direction. If we consider an XYZ coordinate system with the X and Y axes parallel to the stage 13 and the Z axis perpendicular to the stage 13, the optical axis of the polarization camera 10 is parallel to the Z axis. In this system, the objective is to have the polarization camera 10 capture the light specularly reflected by the subject W, so it is desirable that the angle of incidence of the illumination light L with respect to the surface of the subject is as close to the Brewster's angle as possible. Therefore, in this embodiment, the illumination device 11 is positioned at approximately the same height as the stage 13 (in practice, the illumination light L is The lighting device 11 is positioned so that its lower end is at approximately the same height as the upper end of the stage 13 (so that it is not blocked by the stage 13), and the illumination light L is incident on the subject W from almost directly beside it (perpendicular to the Z axis). This type of lighting arrangement is also called low-angle lighting. In the combination of a zenith camera and low-angle lighting as in this embodiment, the specular reflection R of the illumination light L, which has an incident angle of approximately 45 degrees on the subject surface, is captured by the polarization camera 10. Note that, to bring the incident angle closer to the Brewster's angle, a lighting arrangement may be adopted in which the illumination light L is irradiated from a position lower than the subject W, or the optical axis of the polarization camera 10 may be tilted opposite to the lighting device 11.

[0027] (polarized camera) An example of the configuration of the polarization camera 10 will be described with reference to Fig. 2. Fig. 2 is a diagram showing a schematic diagram of the structure of the polarization camera 10.

[0028] The polarization camera 10 has a structure in which an image sensor 20 is combined with a polarizer array 21. The image sensor 20 is a device in which photoelectric conversion elements (also called pixels), such as a CCD (Charge Coupled Device) sensor or a CMOS (Complementary Metal Oxide Semiconductor) sensor, are two-dimensionally arranged, and is also called an image sensor. The polarizer array 21 is a device in which a large number of polarizers are two-dimensionally arranged, and the position and size of each polarizer 210 are designed so that each polarizer 210 corresponds to one pixel (light receiving element) 200 of the image sensor 20. The polarizer 210 is an optical element that has the property of passing only linearly polarized light in a specific direction (the vibration direction of the linearly polarized light transmitted by the polarizer 210 is called the transmission axis direction of the polarizer 210). As shown in FIG. 2 , the polarizer array 21 of this embodiment has a structure in which polarizers 210 with four different transmission axis directions (0°, 45°, 90°, and 135°) are regularly arranged. Specifically, an array pattern is adopted in which linearly polarized light components having different transmission axis directions (0 degrees, 45 degrees, 90 degrees, 135 degrees) are incident on four 2×2 pixels of the image sensor 20.

[0029] The resolution (pixel pitch) and size (number of pixels) of the image sensor 20 and polarizer array 21 can be appropriately designed depending on the subject and the intended use of the image. The polarizer array 21 can be realized using a wire grid, a photonic crystal, or any other method. While FIG. 2 shows four types of polarizers with different transmission axis directions combined into four 2x2 pixels, other variations in the transmission axis directions and arrangements of the polarizers may be used.

[0030] (Lighting equipment) An example of the configuration of illumination device 11 will be described with reference to Fig. 3A. Fig. 3A is a diagram schematically illustrating the structure of illumination device 11 of this embodiment, showing the state when stage 13 is viewed from the polarization camera 10 side.

[0031] The lighting device 11 is composed of four rod-shaped lights 30 arranged to surround the stage 13. When the four rod-shaped lights 30 are turned on simultaneously, the subject W on the stage 13 can be illuminated from all four directions (X-positive direction, X-negative direction, Y-positive direction, and Y-negative direction). Furthermore, the lighting direction can be switched by selectively turning on only one of the rod-shaped lights 30. The rod-shaped lights 30 are composed, for example, of multiple LED light sources arranged on a substrate and a diffusion plate arranged to cover them.

[0032] The structure of the illumination device 11 is not limited to that shown in FIG. 3A. For example, a circular illumination device 11 as shown in FIG. 3B may be used. With this structure, the subject W can be illuminated simultaneously from all directions. Also, by selectively turning on the LED light source, it is possible to arbitrarily switch the illumination direction. Also, instead of omnidirectional illumination, a configuration in which the subject W is illuminated only from a specific direction as shown in FIG. 3C may be used. For example, if the range in which the subject W exists is limited or the angle of the subject surface to be detected is known or limited, it is possible to illuminate from the required direction. Alternatively, even when using the lighting device 11 as shown in Fig. 3C, omnidirectional photography may be performed by changing the relative positions of the lighting device 11 and the subject W (for example, by rotating the subject W on the stage).

[0033] (Processing device) An example of the configuration of the processing device 12 will be described with reference to Fig. 4. Fig. 4 is a block diagram showing the logical configuration (functional configuration) of the processing device 12 of this embodiment.

[0034] The processing device 12 mainly comprises a camera control unit 40, an illumination control unit 41, and an image processing unit 42. The camera control unit 40 controls the polarization camera 10. For example, the camera control unit 40 controls the imaging conditions (exposure time, gain, etc.) of the polarization camera 10, captures image data from the polarization camera 10, and calibrates the polarization camera 10. The illumination control unit 41 controls the illumination device 11. For example, the illumination control unit 41 controls the illumination conditions (emission intensity, emission time, etc.) and controls the on / off of each light source. The image processing unit 42 processes the image captured from the polarization camera 10.

[0035] The image processing unit 42 of this embodiment has functions such as an image acquisition unit 420, a polarization extraction unit 421, a polarization information image generation unit 422, a display unit 423, and a parameter reception unit 424. The image acquisition unit 420 acquires an image captured by the polarization camera 10 (hereinafter also referred to as the "original image") via the camera control unit 40. The polarization extraction unit 421 performs polarization extraction processing to extract polarization information from the original image. Polarization information is information related to polarization resulting from specular reflection on a transparent object. The polarization information image generation unit 422 generates an image representing polarization information (hereinafter also referred to as the "polarization information image") based on the extraction result of the polarization extraction processing. The display unit 423 performs processing to display the generated polarization information image on a display device. The parameter reception unit 424 has a function of receiving changes to various condition settings (parameters) that determine the operation of the imaging system 1. Details of each process performed by the image processing unit 42 will be described later.

[0036] The processing device 12 may be configured by a computer equipped with, for example, a processor such as a CPU or GPU, a memory as a main memory, a storage as an auxiliary memory, a display device, an input device such as a mouse or a touch panel, a network I / F, etc. The computer may be a general-purpose computer such as a personal computer, a tablet terminal, a smartphone, or a field computer, or may be an embedded computer or a dedicated device, or may be other devices such as the polarization camera 10 or a PLC (Programmable Logic Controller). The configuration shown in Fig. 4 may utilize the computer resources of a device. The configuration shown in Fig. 4 is realized by loading a program stored in a storage device or the like into memory and executing it with a processor. However, part or all of the configuration shown in Fig. 4 may be configured with dedicated devices such as an ASIC (Application Specific Integrated Circuit) or an FPGA (Field Programmable Gate Array). Also, part of the configuration or part of the processing shown in Fig. 4 may be executed by another device using cloud computing or distributed computing.

[0037] (Transparent object detection using a polarization camera) This section explains the basic principles of transparent object detection using a polarization camera.

[0038] As shown in Figure 5A, when light 50 is incident obliquely on a transparent object W such as glass or transparent resin, most of the incident light 50 becomes refracted light 51 and passes through the transparent object W, but a portion of the incident light 50 is specularly reflected on the surface (interface) of the transparent object W and becomes reflected light 52. This specular reflection causes a difference in reflectance between the p-wave (polarized component whose electric field vibration direction is parallel to the plane of incidence) and the s-wave (polarized component whose electric field vibration direction is perpendicular to the plane of incidence (i.e., parallel to the reflection plane)) contained in the incident light 50. Figure 5B shows an example of the difference in reflectance between the s-wave and p-wave, with the angle of incidence on the horizontal axis and the reflectance on the vertical axis. As shown in this example, the reflectance of the s-wave increases monotonically according to the angle of incidence α. However, the reflectivity of p-waves gradually decreases as the angle of incidence α increases from 0 degrees, and at a certain angle (called the Brewster angle), the reflectivity becomes 0. Therefore, the reflected light 52 observed in the case of specular reflection is dominated by s-waves, i.e., polarized components that vibrate in a direction parallel to the surface of the transparent object W, which is the reflective surface. By utilizing this property and capturing polarized light with a polarization camera, it is possible to image the surface (reflective surface) of the transparent object W.

[0039] Incidentally, not only polarized light due to specular reflection but also light diffusely reflected from the subject and background enters a polarization camera. When imaging a transparent object, the diffusely reflected light component is unnecessary and can cause artifacts, so it is first necessary to extract only the polarized light component from the image captured by the polarization camera. To achieve this, this embodiment utilizes the difference in transmission characteristics of polarizers.

[0040] With reference to Figure 6, we will explain how to extract polarization components from images captured by a polarization camera. The intensity of the polarization components received by the light receiving element is maximum when the polarization direction θ coincides with the transmission axis direction of the polarizer. It decreases as the transmission axis direction deviates from the polarization direction θ and is minimum when the transmission axis direction and the polarization direction θ are perpendicular. In other words, as the transmission axis direction of the polarizer is changed from 0 to 180 degrees, the intensity of the polarization components received by the light receiving element changes like a sine wave. On the other hand, the intensity of the diffusely reflected light component contained in the incident light is constant, regardless of the transmission axis direction of the polarizer. Therefore, it is possible to extract only information about the polarization components contained in the incident light based on the change in light intensity when observed through multiple types of polarizers with different transmission axis directions (transmission axis direction dependence).

[0041] For example, as shown in FIG. 6, consider a case in which incident light containing a polarization component with a polarization direction θ is observed through four polarizers 210 with transmission axis directions of 0°, 45°, 90°, and 135°. In this case, the values ​​of the pixels 200 corresponding to each polarizer 210 (corresponding to the light intensity of the transmitted light) differ depending on the transmission axis direction. By fitting a sine wave to these four pixel values, the polarization direction θ and polarization intensity of the polarization component can be estimated. In FIG. 6, the angle at which the fitting curve is maximized represents the polarization direction θ of the polarization component, and the amplitude of the fitting curve represents the polarization intensity (also called the degree of polarization). By applying this polarization extraction process to each pixel of the image captured by the polarization camera 10, information (such as the polarization direction and polarization intensity) of polarized light specularly reflected by a transparent object W present within the field of view of the polarization camera 10 can be captured.

[0042] The polarization extraction process by the polarization extraction unit 421 of this embodiment also follows the same basic principle of extracting polarization information based on the change in light intensity (transmission axis direction dependency) when observed through multiple types of polarizers with different transmission axis directions. However, to further improve accuracy and usability, unique processing such as selection of mask size (a parameter that defines the range of neighboring pixels to be referenced when calculating polarization) and noise removal has been added. This will be described in more detail later.

[0043] (Example of imaging system operation) An example of the operation of the imaging system 1 will be described with reference to the flowchart of FIG.

[0044] In step S70, a transparent object to be the subject W is placed on the stage 13. The subject W may be transported and positioned using a robot or a transport device, or the subject W may be set on the stage 13 by an operator.

[0045] In step S71, the lighting control unit 41 of the processing device 12 sets the lighting conditions of the lighting device 11. The lighting device 11 of this embodiment can switch the lighting intensity between four levels, 1 to 4, and the lighting control unit 41 first sets the lighting intensity to 1 (the darkest state).

[0046] In step S72, an image of subject W is captured. Specifically, illumination control unit 41 turns on illumination device 11 in accordance with the given illumination conditions, and irradiates subject W with illumination light. Then, with subject W illuminated, camera control unit 40 controls polarization camera 10 in accordance with the given imaging conditions to capture an image.

[0047] In step S73, the image acquisition unit 420 of the processing device 12 captures the image (original image) captured in step S72 from the polarization camera 10. The captured original image data is stored in memory or storage and is subjected to processing by the image processing unit 42.

[0048] In step S74, the processing device 12 checks whether image capture under all shooting conditions has been completed, and if not, returns to step S71 to perform image capture under the next shooting condition. That is, in step S71, the illumination control unit 41 changes the illumination intensity of the illumination device 11 to 2, and in step S72, the subject W is captured again, and in step S73, an original image corresponding to the illumination intensity 2 is captured. In this embodiment, the processing of steps S71 to S74 is repeated four times, and four original images of the same subject W with different brightnesses are captured in the processing device 12. When image capture under all shooting conditions has been completed, the process proceeds to step S75.

[0049] In step S75, the user inputs image processing parameters to be used in the image processing unit 42 using a user interface provided by the parameter receiving unit 424. If there is no need to change the image processing parameters, the processing of step S74 may be skipped. Furthermore, the image processing parameters may be set before capturing an image.

[0050] FIG. 8 shows an example of a user interface (UI) for inputting image processing parameters provided by the parameter receiving unit 424. This UI screen includes a mask size setting 80, a noise reduction setting 81, a display image setting 82, a full image display area 83, a magnified image display area 84, and a magnification setting 85. The mask size setting 80 is a UI for setting the "mask size," which defines the range of neighboring pixels to be referenced when calculating polarization information from the original image. In the example of FIG. 8, the user is prompted to select a mask size from 2×2, 3×3, 4×4, or 5×5; however, the user may also be prompted to input an arbitrary size. The noise reduction setting 81 is a UI for setting conditions for the "noise reduction process," which removes or reduces noise contained in the polarization information calculation results. In the example of FIG. 8, the user can select whether or not to apply noise reduction. If "yes," the user can also input parameters to be used in the noise reduction process (comparison range, polarization angle difference threshold, and polarization intensity difference threshold).

[0051] The display image settings 82 are UIs for setting the type of image to be displayed in the entire image display area 83 and the enlarged image display area 84. In the example of FIG. 8, the original image, polarization intensity image, polarization direction image, and brightness image can be selected. The entire image display area 83 is an area where the entire image (low resolution) is displayed, and the enlarged image display area 84 is an area where a portion of the image (the portion indicated by a frame 86) is displayed at high resolution. The enlargement settings 85 are UIs for setting the magnification and range of the enlarged image to be displayed in the enlarged image display area 84.

[0052] In step S76, the polarization extraction unit 421 calculates polarization information from the multiple original images captured in step S73. The polarization information calculation process will be described in detail later.

[0053] In step S77, the polarization extraction unit 421 applies noise removal processing to the calculation result of step S76. Note that if noise removal is set to "no" in the UI of Fig. 8, the processing of step S77 is skipped. Details of the noise removal processing will be described later.

[0054] In step S78, the polarization information image generating unit 422 receives the plurality of polarization information images generated by the polarization extracting unit 421. The multiple pieces of polarization information extracted from each of the original images are combined to generate a polarization information image. Details of the image generation process will be described later. In this embodiment, three types of images are generated as the polarization information image: a polarization intensity image, a polarization direction image, and a brightness image. The polarization intensity image is an image that represents the polarization intensity (degree of polarization) using shades, and the polarization direction image is an image that represents the polarization direction using shades or pseudo-colors. The brightness image is an image that represents the light intensity of the polarized component using shades.

[0055] In step S79, the display unit 423 displays the image selected in the display image setting 82 of the UI of FIG. 8 in the whole image display area 83 and the enlarged image display area 84.

[0056] Here, when the user changes the display image setting 82 on the UI of Fig. 8, the images displayed in the whole image display area 83 and the enlarged image display area 84 change accordingly. Using this UI allows the user to visually compare the appearance of the original image with the polarization information (polarization intensity, polarization direction, brightness), facilitating tasks such as confirming whether the calculation of polarization information and noise removal have been performed appropriately, and checking the shape and state of transparent objects. Note that if the mask size setting 80 or noise removal setting 81 is changed on the UI of Fig. 8, the processes of steps S76 to S79 ​​may be executed again. Using such a UI allows the user to fine-tune image processing parameters (mask size, whether or not to perform noise removal, comparison range, threshold, etc.) while checking the calculation results (polarization information image).

[0057] In this embodiment, the lighting conditions of the lighting device 11 were changed as the shooting conditions, but images with different brightnesses can also be captured by changing the imaging conditions of the polarization camera 10. For example, the camera control unit 40 of the processing device 12 may control the exposure time of the polarization camera 10. The longer the exposure time, the greater the amount of light received by the imaging element, and therefore brighter images can be captured. Alternatively, the camera control unit 40 may control the gain of the polarization camera 10. The gain is the amplification factor of the imaging element, and the higher the gain, the brighter the image obtained. Of course, it is also possible to control both the lighting conditions of the lighting device 11 and the imaging conditions of the polarization camera 10.

[0058] (Polarization information calculation process) 9 and 10A to 10D, an implementation example of the polarization information calculation process (step S76 in FIG. 7) of the polarization extraction unit 421 will be described. FIG. 9 is a flowchart of the polarization information calculation process, and FIGS. 10A to 10D are diagrams showing examples of neighboring pixels. Here, the size (number of pixels) of the original image is assumed to be M rows x N columns, the pixel in the i-th row and j-th column is represented as (i, j), and the pixel value (light intensity) of that pixel is represented as I(i, j) (i = 1, 2, . . . , M, j = 1, 2, . . . , N). FIGS. 10A to 10D show an enlarged portion of the original image, with each rectangle representing a pixel, the arrow within the pixel representing the transmission axis direction, and the symbols in the margins representing the row and column numbers of the pixel.

[0059] In step S10, the polarization extraction unit 421 selects one original image to be processed from among a plurality of original images captured under different shooting conditions in steps S71 to S74 of Fig. 7. In this embodiment, in step S10, the original images captured at illumination intensity 1, the original image captured at illumination intensity 2, the original image captured at illumination intensity 3, and the original image captured at illumination intensity 4 are selected in this order.

[0060] In step S11, the polarization extraction unit 421 selects a pixel of interest to be calculated from the original image. In step S11, it is assumed that the pixels of interest are selected in order from (1,1) to (M,N).

[0061] In step S12, the polarization extraction unit 421 selects neighboring pixels to be referenced when calculating polarization information for the pixel of interest. The range of neighboring pixels to be selected here is determined by the mask size setting 80 in FIG. The mask size is determined by the mask size specified in . First, an example of the default mask size of 2 × 2 will be described. When the mask size is 2 × 2, as shown in FIG. 10A, three pixels (i-1, j-1), (i-1, j), and (i, j-1) adjacent to the pixel of interest (i, j) are selected as neighboring pixels. Note that if the pixel of interest is on the boundary of the original image and there are no neighboring pixels, the processing of steps S12 to S13 may be skipped, or an appropriate value may be padded.

[0062] In step S13, the polarization extraction unit 421 calculates polarization information for the pixel of interest based on the light intensities (pixel values) of the pixel of interest and its neighboring pixels. If the light intensities when the transmission axis directions are 0 degrees, 45 degrees, 90 degrees, and 135 degrees are I1, I2, I3, and I4, respectively, the brightness I0, polarization direction θ, and polarization intensity D of the pixel of interest can be calculated using the following formulas.

number

[0063] In the example of FIG. 10A, the transmission axis directions corresponding to the four pixels (i-1, j-1), (i-1, j), (i, j-1), and (i, j) are 90 degrees, 135 degrees, 45 degrees, and 0 degrees, respectively, and therefore I1 to I4 in equations (1) to (3) are given by the following equations. I1=I(i,j) I2=I(i,j-1) I3=I(i-1,j-1) I4=I(i-1,j)

[0064] The brightness I0 in equation (1) is an index that represents the intensity of the incident light that enters the pixel of interest. The polarization direction θ in equation (2) represents the principal axis direction of the polarization component contained in the incident light that enters the pixel of interest. The polarization intensity D in equation (3) is an index that represents the degree of polarization of the polarization component contained in the incident light that enters the pixel of interest.

[0065] The above formulas (1) to (3) are merely examples, and the indices corresponding to the brightness, polarization direction, and polarization intensity may be calculated using other formulas. For example, the coefficients in formulas (1) to (3) may be replaced with other coefficients, or formula (3) may be replaced with the following formula (4):

number

[0066] In step S14, the polarization extraction unit 421 checks whether the calculation process for polarization information has been completed for all pixels in the original image (i.e., whether i=M and j=N), and if there are any unprocessed pixels, returns to step S11 and performs processing on the next pixel of interest.

[0067] In step S15, the polarization extraction unit 421 checks whether the calculation process for polarization information has been completed for all original images, and if there are unprocessed original images, the process returns to step S10 to process the next original image.

[0068] Through the above-described process, three items of polarization information (polarization direction θ, polarization intensity D, and brightness I0) are calculated for every pixel in every original image. The results of the polarization information calculation process are saved in memory or storage in a data format that associates the coordinates (i,j) of each pixel with the polarization direction θ(i,j), polarization intensity D(i,j), and brightness I0(i,j), and are then used for subsequent processing.

[0069] Next, a processing example when the mask size is larger than 2×2 will be described with reference to FIGS. 10B to 10D.

[0070] As shown in FIG. 10B, when the mask size is 3×3, eight pixels (i-1, j-1), (i-1, j), (i-1, j+1), (i, j-1), (i, j+1), (i, j-1), (i, j+1), (i+1, j-1), (i+1, j), and (i+1, j+1) adjacent to the pixel of interest (i, j) are selected as neighboring pixels in step S12. In the example of FIG. 10B, the nine pixel groups including the pixel of interest and neighboring pixels include one pixel whose transmission axis direction is 0 degrees, two pixels whose transmission axis direction is 45 degrees, two pixels whose transmission axis direction is 135 degrees, and four pixels whose transmission axis direction is 90 degrees. When multiple pixels with the same transmission axis direction exist, the calculation in step S13 is performed by substituting representative values ​​of the pixels for each transmission axis direction into I1 to I4 in equations (1) to (3).

[0071] For example, when the average value is used as the representative value, I1 to I4 are given by the following formulas. I1=I(i,j) I2={I(i,j-1)+I(i,j+1)} / 2 I3={I(i-1,j-1)+I(i-1,j+1)+I(i+1,j-1)+I(i+1,j+1)} / 4 I4={I(i,j-1)+I(i,j+1)} / 2

[0072] The representative value may be the average value, the mode value, the median value, the maximum value, the minimum value, etc. Alternatively, without using the representative value, I1 to I4 may be estimated by fitting a sine wave to the values ​​of the nine pixels, or the brightness I0, the polarization direction θ, and the polarization intensity D may be calculated directly.

[0073] FIG. 10C shows an example where the mask size is 4×4. The selected range includes the eight neighboring pixels from the 3×3 mask size plus seven additional pixels: (i-2, j-2), (i-2, j-1), (i-2, j), (i-2, j+1), (i-1, j-2), (i, j-2), and (i+1, j-2). In the example of FIG. 10C, the selected range includes four pixels at angles of 0°, 45°, 90°, and 135°. In this case, as in the 3×3 mask size, either substitute representative values ​​for I1 to I4 for each transmission axis direction, or perform sinusoidal wave fitting on the values ​​of the 16 pixels.

[0074] FIG. 10D shows an example with a 5×5 mask size. The selected range includes the 15 neighboring pixels from a 4×4 mask, plus an additional nine pixels: (i-2, j+2), (i-1, j+2), (i, j+2), (i+1, j+2), (i+2, j-2), (i+2, j-1), (i+2, j), (i+2, j+1), and (i+2, j+2). In the example of FIG. 10D, the selected range includes nine pixels at 0 degrees, six pixels at 45 degrees, six pixels at 135 degrees, and four pixels at 90 degrees. In this case, as with the 3×3 mask, we can substitute representative values ​​for I1 to I4 for each transmission axis direction, or perform sinusoidal wave fitting on the values ​​of the 25 pixels.

[0075] In this embodiment, the mask size, that is, the pixel range to be referred to when calculating polarization information, can be set arbitrarily. The user can select an appropriate mask size depending on the shape and condition of the subject, or the purpose of the system.

[0076] For example, as shown in FIG. 11A, when the polarization camera 10 captures specular reflection from a curved surface of the subject W, the reflecting surface is extremely small, so the area where the polarized light is incident may be a narrow area of ​​only a few pixels wide (depending on the resolution of the polarization camera 10). In such a case, if the mask size is too large, pixels where no polarized light is incident may also be referenced, as shown in FIG. 11B, which may reduce the detectability of polarization information. Therefore, it is preferable to select a mask size smaller than the area where the polarized light is incident. This allows for appropriate extraction of polarization information from the narrow area.

[0077] On the other hand, as shown in Figure 12, when inspecting for surface contamination or foreign matter, it is expected that similar polarization characteristics will appear over a relatively wide area of ​​the image due to the reflectance characteristics (specular reflection, scattering strength) of a flat surface. In such cases, if the mask size is too small, the inspection may be sensitive to local differences in reflectance, resulting in unstable inspection. Therefore, it is preferable to set a somewhat larger mask size to average out disturbing factors such as local differences in reflectance. This can improve the stability of the inspection.

[0078] In this embodiment, four mask sizes, 2x2, 3x3, 4x4, and 5x5, are shown as examples of mask sizes, but the mask size is not limited to these. For example, a mask larger than 5x5 may be used, or a mask with a shape other than a square may be used. Furthermore, the user may be allowed to design the size and shape of the mask as desired.

[0079] (Noise removal processing) The polarization information calculation process described above utilizes the principle of detecting polarization by regarding local pixel value differences as differences in transmission due to differences in the direction of the polarizer's transmission axis. Therefore, for example, when a high-spatial-frequency brightness change occurs in an edge or texture area of ​​a subject, this may be mistaken for a pixel value difference due to polarization, resulting in a false detection of polarization. The noise removal process is a process for removing or reducing noise due to such false detection from the calculation results of the polarization information calculation process.

[0080] An implementation example of the noise removal process (step S77 in FIG. 7) will be described with reference to Fig. 13, Fig. 14A, and Fig. 14B. Fig. 13 is a flowchart of the noise removal process, and Fig. 14A and Fig. 14B are diagrams showing examples of comparison ranges.

[0081] In step S20, the polarization extraction unit 421 reads the parameters set in the noise removal setting 81 in Fig. 8. Here, four parameters are read: whether or not noise removal is enabled, the comparison range, the polarization angle difference threshold, and the polarization intensity difference threshold. If noise removal is set to "enabled," the subsequent processing is skipped (step S21). If noise removal is set to "enabled," the processing proceeds to step S22.

[0082] In step S22, the polarization extraction unit 421 reads the results of the polarization information calculation process from memory or storage. Here, data is obtained that associates the coordinates (i, j) of each pixel with the polarization direction θ(i, j), polarization intensity D(i, j), and brightness I0(i, j) extracted from each of the multiple original images.

[0083] In step S23, the polarization extraction unit 421 selects one original image to be processed from the plurality of original images. In this embodiment, the original images are selected in the order of the original image captured at illumination intensity 1, the original image captured at illumination intensity 2, the original image captured at illumination intensity 3, and the original image captured at illumination intensity 4. The selection shall be made.

[0084] In step S24, the polarization extraction unit 421 selects a target pixel to be processed. In step S24, the target pixels are selected in order from (1,1) to (M,N).

[0085] In step S25, the polarization extraction unit 421 compares the polarization directions of the target pixel and its surrounding pixels to determine whether the polarization directions are similar. As shown in FIG. 14A, when the comparison range is set to "1," a comparison is made between the target pixel (i, j) and the three surrounding pixels (i, j+1), (i+1, j), and (i+1, j+1). Also, as shown in FIG. 14B, when the comparison range is set to "2," a comparison is made between the target pixel (i, j) and the eight surrounding pixels (i-1, j-1), (i-1, j), (i-1, j+1), (i, j-1), (i, j+1), (i+1, j-1), (i+1, j), and (i+1, j+1). If any combination is found in which the difference in polarization direction is greater than the polarization angle difference threshold, the polarization extraction unit 421 determines that the polarization directions are "dissimilar," and proceeds to step S27. On the other hand, if it is determined to be "similar," the process proceeds to step S26.

[0086] In step S26, the polarization extraction unit 421 compares the polarization intensities of the target pixel and its surrounding pixels to determine whether the polarization intensities are similar. The method for selecting the comparison range is as shown in Figures 14A and 14B. If even one combination is found in which the difference in polarization intensities is greater than the polarization intensity difference threshold, it is determined to be "dissimilar" and the process proceeds to step S27. On the other hand, if it is determined to be "similar," the process proceeds to step S28.

[0087] If either the polarization direction or the polarization intensity is "dissimilar," the polarization information of the target pixel is an outlier compared to the surrounding pixels, and is likely to be noise. Therefore, in step S27, the polarization extraction unit 421 deletes the polarization information of the target pixel (polarization direction θ and polarization intensity D).

[0088] If both the polarization direction and the polarization intensity are "similar," in step S28, the polarization extraction unit 421 selects the pixel with the greatest polarization intensity among the target pixel and the pixels in the comparison range, and replaces the polarization direction and polarization intensity of the target pixel with the polarization direction and polarization intensity of the selected pixel. This process achieves a smoothing effect that reduces local variations in the polarization direction and polarization intensity.

[0089] In step S29, the polarization extraction unit 421 checks whether the noise removal process has been completed for all pixels, and if there are any unprocessed pixels, the process returns to step S24 to process the next target pixel.

[0090] In step S30, the polarization extraction unit 421 checks whether noise removal processing has been completed for all original images, and if there are unprocessed original images, the process returns to step S23 to process the next original image.

[0091] The above-described process removes noise (outliers) and smooths the polarization direction θ and polarization intensity D, which are the calculation results of the polarization information calculation process. The corrected data is then overwritten and saved in memory or storage.

[0092] (Image generation processing) An example of implementation of the image generation process (step S78 in FIG. 7) will be described with reference to the flowchart in FIG.

[0093] In step S40, the polarization information image generating unit 422 first selects a target pixel (i ,j). Each time the loop of steps S40 to S45 is repeated, a target pixel is selected in order from (1,1) to (M,N). The polarization information image generator 422 then obtains polarization information extracted from each of the multiple original images for the target pixel (i,j). Here, if the polarization information extracted from pixel (i,j) of the kth original image is denoted as PIk(i,j), in this example, polarization information PI1(i,j) to PI4(i,j) corresponding to each of the four original images captured at illumination intensities 1 to 4 is read from memory. Note that the polarization information PIk(i,j) includes three items of information: polarization intensity Dk(i,j), polarization direction θk(i,j), and brightness I0k(i,j) (the subscript k represents the original image number).

[0094] In step S41, the polarization information image generation unit 422 compares the multiple pieces of polarization information PI1(i,j) through PI4(i,j) acquired in step S40 and determines that the polarization information with the strongest polarization intensity Dk(i,j) is the most reliable polarization information and adopts it as the polarization information for the target pixel (i,j). Polarization intensity is an index that represents the degree of polarization; a larger value indicates that the polarization component is more clearly observed. Therefore, by adopting the piece of polarization information PI1(i,j) through PI4(i,j) with the strongest polarization intensity acquired under different shooting conditions, the most reliable polarization information for pixel (i,j) (i.e., the area on the reflective surface of the transparent object corresponding to pixel (i,j)) can be obtained. The following description will continue assuming that PIn(i,j), the polarization information for the nth original image, is adopted.

[0095] In step S42, the polarization information image generator 422 writes the polarization intensity value Dn(i,j) of the polarization information PIn(i,j) selected in step S41 to pixel (i,j) of the polarization intensity image. Similarly, in step S43, the polarization information image generator 422 writes the polarization direction value θn(i,j) of the polarization information PIn(i,j) selected in step S41 to pixel (i,j) of the polarization direction image. Also, in step S44, the polarization information image generator 422 writes the brightness value I0n(i,j) of the polarization information PIn(i,j) selected in step S41 to pixel (i,j) of the brightness image.

[0096] By performing the above-described steps S40 to S44 for each pixel (step S45), a polarization information image can be obtained that combines the good parts (i.e., highly reliable polarization information) of the multiple polarization information PI1 to PI4. In the above embodiment, the polarization intensity D itself is used as an index representing the reliability of the polarization information, but the reliability of the polarization information may also be determined using a different index. For example, (I3 - I1) 2 +(I2-I4) 2 The index obtained by (an index corresponding to the contrast strength of the four pixels) may be used as the reliability of the polarization information. In addition, since the dark areas of an image are generally more susceptible to the influence of noise, the brightness of the image may also be taken into consideration. For example, if MAX(I1, I2, I3, I4) is below a threshold (i.e., in the case of a dark area of ​​the image), it can be assumed that there is no polarization there, and the reliability may be set to 0 (the minimum value).

[0097] The advantages of the polarization information image of this embodiment will be described using an example in which the imaging system 1 described above is applied to detection and inspection in the manufacturing process of syringes filled with liquid medicine.

[0098] In the manufacturing process of pre-filled syringes, robots are usually used to pick and place the syringes and inject the liquid. If the syringe is in an incorrect position or lying on its side, it could be damaged or the manufacturing equipment could malfunction. Therefore, a mechanism is needed to monitor the syringe's position and orientation, and output an error or stop the equipment if a problem occurs.

[0099] FIG. 16A shows a state in which syringes 101 placed in nest 100 are observed by polarization camera 10 installed at the zenith. Two syringes 101b are not inserted into the holes in nest 100 and are lying on their side. However, since syringes 101 are colorless and transparent, they can be seen, for example, Even when photographed using normal transmitted illumination and an optical camera, the boundary between the syringe 101 and the background (white nest 100) is barely discernible in the image. Therefore, it is difficult to detect the overturned syringe 101b in a normal optical image.

[0100] 16B, in the imaging system 1 of this embodiment, low-angle lighting devices 11 (light sources 11r, 11l, 11t, and 11b) are installed around the nest 100, and the polarization camera 10 captures the specular reflection (polarized light) on the surface of the syringe 101. The orientation of the reflective surface of the syringe 101 (the orientation in the XY plane of the normal to the reflective surface) can take any angle between 0 and 360 degrees, but by illuminating the syringe 101 from all sides as in FIG. 16B, the polarization camera 10 can observe the specular reflection from the reflective surface in any orientation.

[0101] However, when comparing syringe 101r on the right side of the field of view with syringe 101l on the left side, for example, the intensity of the specularly reflected light observed from each syringe 101r, 101l will not be the same. This is because, although light from light source 11r, which is located on the right side of nest 100, is incident on the right-facing reflective surface of each syringe 101r, 101l, there is a difference in the intensity of the illumination light incident on the reflective surface between syringe 101r located closer to light source 11r and syringe 101l located farther from light source 11r. Therefore, it is not possible to capture all subjects within the field of view of polarization camera 10 under completely uniform lighting conditions. If the illumination intensity of light source 11r is set to match the proximal syringe 101r, the polarized component in the distal syringe 101l may be weak, making it difficult to extract polarization information. Conversely, if the illumination intensity of light source 11r is set to match the distal syringe 101l, whiteout may occur in the proximal syringe 101r, making it impossible to extract polarization information.

[0102] Therefore, imaging system 1 measures polarization multiple times under different imaging conditions. Images 104a to 104d in FIG. 17 are schematic examples of polarization intensity images obtained when the illumination intensity is gradually increased. It can be seen that the optimal illumination intensity varies depending on the position within the field of view. By combining and collecting high-polarization intensity portions from these images 104a to 104d, it is possible to obtain a high-quality polarization information image 105 in which the polarization information of all subjects present within the field of view has been appropriately extracted. In polarization information image 105, a clearly circular image of syringe 101a in the correct position appears, while a clearly linear band-like image of syringe 101b lying on its side appears. Using this polarization information image 105 makes it possible to stably inspect the position and orientation of syringes and detect syringes in an incorrect position or orientation.

[0103] Computer-based image recognition technology may be used to detect syringes from the polarization information image 105 and determine the syringe's posture. For example, a syringe in the correct posture will exhibit specular reflection from the inner circle and the flange surrounding it. By focusing on this image feature and detecting the shape of the circle and flange from the polarization information image using a method such as pattern matching, it becomes possible to automatically recognize syringes in the correct posture. In the case of a syringe lying on its side, for example, first, a set of adjacent pixels (pixel region) within a specific polarization direction range (e.g., 0° to 60°) is extracted, and then pixel regions with widths and lengths equal to or greater than thresholds are detected from the extracted pixel region. This process makes it possible to automatically recognize the strip-shaped pixel region of a syringe lying on its side.

[0104] In the above embodiment, multiple images were taken while switching the illumination conditions (illumination intensity) of the illumination device 11, but multiple images may be taken while switching the imaging conditions (exposure conditions or gain) of the polarization camera 10, or control may be exercised to switch both the illumination conditions of the illumination device 11 and the imaging conditions of the polarization camera 10. In other words, the imaging conditions may be changed in any way as long as multiple original images with different brightnesses can be obtained.

[0105] Second Embodiment In the first embodiment, the subject was photographed while illuminated from all directions. This method has the advantage of being able to capture reflective surfaces in all directions within the field of view in a single shot, thereby shortening the time required to capture the original image. However, simultaneous illumination from all directions can potentially cause noise when observing the specular reflection of illumination light from one direction due to diffuse reflection of illumination light from other directions. Figure 18A shows an example. Light incident from light source 11r is specularly reflected from the slope on the right side of subject W and observed by polarization camera 10. If strong light is incident from light source 11l in another direction, the diffuse reflection light in the background passes through the transparent subject W and is observed together with the specular reflection light. For example, in a situation where light source 11r is far from subject W and light source 11l is close to subject W, the strong diffuse reflection light may drown out the weak specular reflection light, significantly reducing the accuracy of extracting polarization information.

[0106] In order to reduce such diffuse reflection noise, in the second embodiment, multiple images are captured while switching the illumination direction relative to the subject W. For example, as shown in FIG. 18B, the first image is captured with only the light source 11l turned on, and then the second image is captured with only the light source 11r turned on. This allows the diffuse reflection of the illumination light from the light source 11l and the specular reflection of the illumination light from the light source 11r to be observed separately. This reduces the situation in which the weak specular reflection is drowned out by the strong diffuse reflection.

[0107] An example of the operation of the imaging system 1 of the second embodiment will now be described with reference to the flowchart of FIG.

[0108] In step S70, a transparent object to be the subject W is placed on the stage 13. The subject W may be transported and positioned using a robot or a transport device, or the subject W may be set on the stage 13 by an operator.

[0109] In step S71, the lighting control unit 41 of the processing device 12 sets the lighting conditions (lighting direction) of the lighting device 11. The lighting device 11 of this embodiment has four rod-shaped lights (light sources) 30 arranged to surround the stage 13, and each rod-shaped light 30 can be independently switched on / off and its lighting intensity can be independently controlled. The lighting control unit 41 sets the lighting conditions so that only one of the four rod-shaped lights 30 is turned on and the other three are turned off.

[0110] In step S72, the subject W is photographed. Specifically, the illumination control unit 41 turns on the illumination device 11 in accordance with the given illumination conditions, and irradiates the subject W with illumination light. At this time, the subject W on the stage 13 is illuminated from only one direction. Then, with the subject W illuminated, the camera control unit 40 controls the polarization camera 10 in accordance with the given imaging conditions to photograph the subject W.

[0111] In step S73, the image acquisition unit 420 of the processing device 12 captures the image (original image) captured in step S72 from the polarization camera 10. The captured original image data is stored in memory or storage and is subjected to processing by the image processing unit 42.

[0112] In step S74, the processing device 12 checks whether image capture under all shooting conditions has been completed, and if not, returns to step S71 to perform image capture under the next shooting condition. That is, in step S71, the lighting control unit 41 changes the lighting direction of the lighting device 11, in step S72, the subject W is captured again, and in step S73, a new original image is captured. In this embodiment, the processing of steps S71 to S74 is repeated four times, and four original images of the same subject W but with different lighting directions are captured by the processing device 12. When image capture under all shooting conditions has been completed, the process proceeds to step S75. The processing from this point onwards may be the same as in the first embodiment, and therefore a description thereof will be omitted.

[0113] As mentioned above, changing the lighting direction changes the diffuse reflection conditions in the background, etc. Therefore, by selecting from multiple original images taken with different lighting directions the one with the least influence of the diffuse reflection component (i.e., noise) and the clearest polarization component, it is possible to extract more reliable polarization information.

[0114] In this embodiment, the lighting direction is changed by switching the light source that is turned on, using the lighting device 11 that has multiple light sources that can be turned on independently. This method has the advantage that the lighting direction can be changed quickly with simple control, thereby shortening the tact time for photography.

[0115] (others) The above-described embodiment merely exemplifies the configuration of the present invention, and the present invention is not limited to the specific embodiment described above, and various modifications are possible within the scope of the technical concept thereof.

[0116] For example, the arrangement and configuration of the polarization camera and lighting device are not limited to those shown in Figure 1 and Figures 3A to 3C, and may be designed appropriately depending on the subject and application. In other words, it is sufficient that the specular reflection of the illumination light can be observed by the polarization camera. Furthermore, multiple polarization cameras may be provided, or movable polarization cameras and lighting devices may be used.

[0117] In the above embodiment, polarizers with 0-degree, 45-degree, 90-degree, and 135-degree transmission axis directions are used, but the setting of the transmission axis direction is not limited to this. It is sufficient that the imaging element can observe light that has passed through polarizers with different transmission axis directions and can extract polarization information contained in the incident light from the observation results.

[0118] The UI screen in FIG. 8 allows the user to set the mask size used in the polarization information calculation process and the parameters used in the noise removal process, but the configuration of the UI screen and the parameter items are merely examples. Any UI that allows the user to change the parameters used in the polarization extraction process may be used. In other words, it is sufficient if the user can change the polarization extraction logic as appropriate depending on the subject, application, etc. In addition, in the above embodiment, the parameter receiving unit 424 provides a user interface for setting parameters, but the parameter receiving unit 424 may also receive a parameter change instruction from an external device via a network from the user.

[0119] The image brightness switching control in the first embodiment and the illumination direction switching control in the second embodiment may be performed together, which makes it possible to both uniformize the illumination conditions across the entire field of view and reduce diffuse reflection noise, thereby enabling the extraction of more reliable polarization information.

[0120] In the second embodiment, the illumination direction is changed by switching the light source to be turned on, but the configuration of the illumination device is not limited to this. For example, the illumination direction may be changed by using an illumination device 11 that illuminates from one direction as shown in Fig. 3C and changing the relative positional relationship between the illumination device 11 and the subject W.

[0121] <Appendix 1> An imaging system (1) for imaging a transparent object (W), comprising: an illumination device (11) for illuminating a transparent object (W); a polarization camera (10) having an image pickup element (20) in which polarizers (210) with different transmission axis directions are regularly arranged; a processing device (12), The processing device (12) control means (40, 41) for controlling either or both of the illumination device (11) and the polarization camera (10) in order to photograph the transparent object (W) under different photographing conditions; image acquisition means (420) for acquiring a plurality of original images taken under different photographing conditions; a polarization extraction means (421) for executing a polarization extraction process to extract polarization information, which is information about polarization resulting from specular reflection on the transparent object (W), for each pixel of each of the plurality of original images; and an image generating means (422) for synthesizing the plurality of pieces of polarization information extracted from the plurality of original images by the polarization extraction process and generating a polarization information image by visualizing the polarization information. An imaging system (1).

[0122] <Appendix 2> A control method for an imaging system (1) including an illumination device (11), a polarization camera (10) having an imaging element (20) in which polarizers (210) having different transmission axis directions are regularly arranged, and a processing device (12), comprising: A step of photographing a transparent object (W) under different photographing conditions by controlling either or both of the lighting device (11) and the polarization camera (10); A step of importing a plurality of original images taken under different photographing conditions into the processing device (12); and performing a polarization extraction process by the processing device (12) for each pixel of each of the plurality of original images to extract polarization information, which is information about polarization resulting from specular reflection on the transparent object (W), and synthesizing the plurality of pieces of polarization information extracted from each of the plurality of original images by the polarization extraction process to generate a polarization information image in which the polarization information is visualized. A control method comprising: [Explanation of symbols]

[0123] 1: Imaging system 10: Polarized camera 11: Lighting equipment 12: Processing device 13: Stage 20: Image sensor 21: Polarizer array 30: Rod lighting

Claims

1. 1. An imaging system for imaging a transparent object, comprising: an illumination device for illuminating a transparent object; a polarization camera having an image sensor in which polarizers with different transmission axis directions are regularly arranged; a processing device, The processing device includes: a control means for controlling one or both of the illumination device and the polarization camera in order to photograph the transparent object under different photographing conditions; image acquisition means for acquiring a plurality of original images captured under different photographing conditions; a polarization extraction means for executing a polarization extraction process to extract, for each pixel of each of the plurality of original images, polarization information that is information regarding polarization resulting from specular reflection on the transparent object; an image generating means for synthesizing the plurality of pieces of polarization information extracted from the plurality of original images by the polarization extraction process and generating a polarization information image by visualizing the polarization information, The image generating means performs an operation of selecting, for each pixel, the polarization information with the highest reliability from the plurality of pieces of polarization information extracted from each of the plurality of original images, and generates the polarization information image using the polarization information selected for each pixel. An imaging system characterized by:

2. The image generating means considers the strongest degree of polarization among the plurality of pieces of polarization information to be the most reliable polarization information.

2. The imaging system according to claim 1, wherein:

3. The control means controls one or both of the lighting device and the polarization camera so as to obtain a plurality of original images with different brightness levels.

3. The imaging system according to claim 1, wherein the imaging system comprises: a first lens;

4. The control means controls the illumination intensity of the illumination device to vary as the photographing condition.

4. The imaging system according to claim 3.

5. The control means controls the polarization camera to vary the exposure time as the photographing condition.

5. The imaging system according to claim 3, wherein the imaging system comprises: a first optical fiber;

6. 6. The imaging system according to claim 3, wherein the control means controls the polarization camera to vary a gain as the imaging condition.

7. The control means controls the lighting device so as to obtain a plurality of original images with different lighting directions for the transparent object.

3. The imaging system according to claim 1, wherein the imaging system comprises: a first lens;

8. The lighting device has a plurality of light sources that can be turned on independently, The control means changes the illumination direction of the transparent object by switching the light source to be turned on.

8. The imaging system according to claim 7,

9. The polarization extraction process includes a noise removal process for removing or reducing noise contained in the calculation result after calculating the polarization information of each pixel of the original image.

9. The imaging system according to claim 1, wherein the imaging system comprises: a first lens;

10. The noise removal process includes a process of determining whether the polarization information of the target pixel and the surrounding pixels is similar or dissimilar by comparing the polarization information of the target pixel and the surrounding pixels, and deleting the polarization information of the target pixel if it is determined that the polarization information is dissimilar.

10. The imaging system according to claim 9.

11. The noise removal process includes a process of comparing polarization information between a target pixel and its surrounding pixels to determine whether the polarization information of the target pixel and the surrounding pixels is similar or dissimilar, and if it is determined that the polarization information of the target pixel is similar, replacing the polarization information of the target pixel with polarization information of a pixel selected from the target pixel and the surrounding pixels.

11. The imaging system according to claim 9, wherein the imaging system comprises:

12. A control method for an imaging system including an illumination device, a polarization camera having an imaging element in which polarizers with different transmission axis directions are regularly arranged, and a processing device, comprising: Photographing a transparent object under different photographing conditions by controlling one or both of the illumination device and the polarization camera; A step of importing a plurality of original images taken under different photographing conditions into the processing device; and performing a polarization extraction process by the processing device to extract, for each pixel of each of the plurality of original images, polarization information that is information about polarization resulting from specular reflection on the transparent object, and synthesizing the plurality of pieces of polarization information extracted from each of the plurality of original images by the polarization extraction process to generate a polarization information image that visualizes the polarization information, In the step of generating the polarization information image, an operation of selecting the polarization information with the highest reliability from the plurality of pieces of polarization information extracted from each of the plurality of original images is performed for each pixel, and the polarization information image is generated using the polarization information selected for each pixel. A control method comprising:

13. A program for causing a processor to execute each step of the control method according to claim 12.

Images