Imaging system and control method thereof

Abstract

To provide a new technology for extracting high-quality polarization information from an image of a polarization camera.SOLUTION: A processing device comprises: image acquisition means for acquiring an original image captured by a polarization camera while a transparent object is illuminated by an illumination device; polarized light extraction means for executing a polarization extraction process of extracting, for each pixel of the original image, polarization information, which is information pertaining to polarized light originating from specular reflection on the transparent object; image generation means for generating, on the basis of a result extracted in the polarization extraction process, a polarization information image obtained by converting the polarization information into an image; and reception means for receiving a change in parameter used in the polarization extraction process.SELECTED DRAWING: Figure 4

Description

[Technical field]

[0001] The present invention relates to techniques for imaging transparent objects. [Background technology]

[0002] There is a known technology for imaging transparent objects using a polarization camera having an imaging element in which polarizers with different transmission axis directions are regularly arranged (see Patent Document 1). This technology is expected to be applied to, for example, the detection and inspection of articles made of glass or transparent resin. [Prior art documents] [Patent documents]

[0003] [Patent Document 1] JP 2020-17688 A Summary of the Invention [Problem to be solved by the invention]

[0004] When light is shone on a transparent object, some of the light is specularly reflected from the surface of the object. Since specularly reflected light has polarized properties, it can be observed with a polarization camera, and the surface (reflective surface) of the transparent object can be imaged based on the observation results.

[0005] However, polarized cameras are not only subject to specular reflection, but also to light diffusely reflected from the subject and background, which can become a source of noise. In addition, the way polarization appears varies depending on the shape and reflective properties of the subject. For these reasons, it is difficult to prepare a versatile logic that can provide stable performance for all subjects and situations.

[0006] The present invention has been made in consideration of the above-mentioned circumstances, and an object of the present invention is to provide a novel technique for extracting high-quality polarization information from images taken by a polarization camera. [Means for solving the problem]

[0007] The present invention provides an imaging system for imaging a transparent object, comprising an illumination device for illuminating a transparent object, a polarization camera having an imaging element in which polarizers with different transmission axis directions are regularly arranged, and a processing device, wherein the processing device has an image acquisition means for acquiring an original image captured by the polarization camera while the transparent object is illuminated by the illumination device, a polarization extraction means for performing a polarization extraction process to extract polarization information, which is information regarding polarization resulting from specular reflection on the transparent object, for each pixel of the original image, an image generation means for generating a polarization information image that visualizes the polarization information based on the extraction result of the polarization extraction process, and a reception means for receiving changes to parameters used in the polarization extraction process.

[0008] According to this configuration, the user can change the parameters of the polarization extraction process as necessary. Therefore, for example, the user can adaptively adjust the parameters according to the shape and state of the transparent object that is the subject, the state of diffuse reflection, and the like, thereby making it possible to extract high-quality polarization information. Note that the accepting unit may provide a user interface for changing the parameters and accept the parameters changed by the user on the user interface, or the accepting unit may receive a parameter change instruction from the user from an external device via a network.

[0009] The polarization extraction process may include a polarization information calculation process for calculating polarization information of a pixel of interest by referring to pixel values ​​of the pixel of interest and its neighboring pixels. When the polarizer has n types of transmission axis directions (n≧2), it is advisable to select neighboring pixels so that the pixel group including the pixel of interest and neighboring pixels includes at least n types of pixels corresponding to each of the n types of transmission axis directions.

[0010] In this case, the parameters may include a mask size, which is an item that specifies the range of neighboring pixels to be referred to when calculating the polarization information in the polarization information calculation process. For example, by setting an appropriate mask size according to the size of the reflective surface of the transparent object that is the subject, it is possible to extract high-quality polarization information.

[0011] The receiving unit may include a user interface that allows a user to select a mask size to be used in the polarization information calculation process from a plurality of mask sizes having different sizes. By providing a selection format, usability can be improved.

[0012] The polarization extraction process may include a noise removal process for removing or reducing noise contained in the calculation result after the polarization information of each pixel of the original image is calculated by the polarization information calculation process. By performing the noise removal process, it is possible to extract high-quality polarization information.

[0013] The noise removal process may include a process of determining whether the polarization information of the target pixel and the surrounding pixels is similar or dissimilar by comparing the polarization information of the target pixel and the surrounding pixels, and deleting the polarization information of the target pixel when it is determined that the polarization information is dissimilar. By such a process, it is possible to remove polarization information with low reliability.

[0014] The noise removal process may include a process of determining whether the polarization information of the target pixel and the surrounding pixels is similar or dissimilar by comparing the polarization information between the target pixel and the surrounding pixels, and if it is determined that the polarization information is similar, replacing the polarization information of the target pixel with the polarization information of a pixel selected from the target pixel and the surrounding pixels. This process smoothes the polarization information, thereby further improving the signal-to-noise ratio of the polarization information.

[0015] The parameters may include an item that specifies a range of the surrounding pixels to be compared with the target pixel. The parameters may also include an item that specifies a criterion for determining whether the polarization information is similar or dissimilar. For example, by appropriately adjusting these parameters according to the level and range of noise contained in the result of the polarization information calculation process, the signal-to-noise ratio can be improved without overcorrection.

[0016] The parameters may include an item for specifying whether or not to apply the noise removal process to the calculation result of the polarization information calculation process. If the result of the polarization information calculation process is of sufficiently high quality, the noise removal process can be skipped to prevent overcorrection and reduce processing time.

[0017] The present invention may provide a control method for an imaging system comprising an illumination device, a polarization camera having an imaging element in which polarizers with different transmission axis directions are regularly arranged, and a processing device, the control method including the steps of illuminating a transparent object with the illumination device, photographing with the polarization camera, importing the original image photographed by the polarization camera into the processing device, and executing a polarization extraction process by the processing device to extract polarization information, which is information regarding polarization resulting from specular reflection on the transparent object, for each pixel of the original image, and generating a polarization information image in which the polarization information is visualized based on the extraction result of the polarization extraction process, and characterized in that the control method is capable of changing parameters used in the polarization extraction process.

[0018] The present invention may provide a program for causing a processor to execute each step of the above control method.

[0019] The present invention may be understood as an imaging system having at least a part of the above means, or as a processing device, a control device, or an image processing device. The present invention may also be understood as an apparatus for detecting or inspecting a transparent object using a polarization information image obtained by an imaging system. The present invention may also be understood as a control method, an image processing method, a detection method, or an inspection method for an imaging system including at least a part of the above processing, or as a program for realizing such a method or a recording medium on which the program is recorded. Note that the above means and processing can be combined with each other as much as possible to constitute the present invention. Effect of the Invention

[0020] According to the present invention, high-quality polarization information can be extracted from an image taken by a polarization camera. [Brief description of the drawings]

[0021] [Figure 1] FIG. 1 is a diagram showing a schematic overall configuration of an imaging system according to an embodiment of the present invention. [Diagram 2] FIG. 2 is a diagram showing a schematic structure of a polarization camera. [Diagram 3] FIG. 3A is a diagram showing a schematic structure of a lighting device, and FIGS. 3B and 3C are diagrams showing other configuration examples of the lighting device. [Figure 4] FIG. 4 is a block diagram showing a logical configuration (functional configuration) of the processing device. [Diagram 5] 5A and 5B are diagrams illustrating specular reflection. [Figure 6] FIG. 6 is a diagram illustrating a method for extracting a polarization component from an image captured by a polarization camera. [Figure 7] FIG. 7 is a flowchart showing an example of the operation of the imaging system. [Figure 8] FIG. 8 shows an example of a user interface for inputting image processing parameters. [Figure 9] FIG. 9 is a flowchart of the polarization information calculation process. [Figure 10] 10A to 10D are diagrams showing examples of neighboring pixels. [Figure 11] FIG. 11A is a diagram showing an example of specular reflection at a curved portion, and FIG. 11B is a diagram showing an example of selecting a mask size when the range in which polarized light is incident is narrow. [Figure 12] FIG. 12 is a diagram showing an example in which a large mask size is suitable. [Figure 13] FIG. 13 is a flowchart of the noise removal process. [Figure 14] FIG. 14 is a diagram showing an example of a comparison range. [Figure 15] FIG. 15 is a diagram showing an application example of the imaging system. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0022] (System Configuration) 1 is a schematic diagram showing the overall configuration of an imaging system according to an embodiment of the present invention. The imaging system 1 is a system for imaging a transparent object using a polarization camera 10, and is used, for example, to detect and inspect transparent objects on a factory production line.

[0023] The imaging system 1 mainly comprises a polarization camera 10, an illumination device 11, a processing device 12, and a stage 13. The polarization camera 10 is an imaging means having an imaging element on which polarizers are arranged. The illumination device 11 is a light source for irradiating illumination light L onto a subject (transparent object) W placed on the stage 13. The processing device 12 is a device that controls the entire imaging system 1 and executes information processing using images captured by the polarization camera 10. The stage 13 is a device for placing or holding the subject W.

[0024] The polarized camera 10 is disposed so as to photograph the subject W placed on the stage 13 from the zenith direction. In an XYZ coordinate system in which the X and Y axes are parallel to the stage 13 and the Z axis is perpendicular to the stage 13, the optical axis of the polarized camera 10 is parallel to the Z axis. In this system, the objective is to capture the light specularly reflected by the subject W with the polarized camera 10, so it is desirable that the angle of incidence of the illumination light L on the subject surface is as close to the Brewster angle as possible. Therefore, in this embodiment, the illumination device 11 is disposed at approximately the same height as the stage 13 (in practice, the lower end of the illumination device 11 is disposed at approximately the same height as the upper end of the stage 13 so that the illumination light L is not blocked by the stage 13), and the illumination light L is incident on the subject W from approximately the side (perpendicular to the Z axis). Such an illumination arrangement is also called low-angle illumination. In the combination of the zenith camera and low-angle illumination as in this embodiment, the specular reflection R of the illumination light L, which has an incident angle of approximately 45 degrees on the subject surface, is captured by the polarized camera 10. In order to bring the angle of incidence closer to Brewster's angle, a lighting arrangement may be adopted in which the illumination light L is applied from a position lower than the subject W, or the optical axis of the polarization camera 10 may be tilted to the opposite side to the illumination device 11.

[0025] (polarized camera) An example of the configuration of the polarization camera 10 will be described with reference to Fig. 2. Fig. 2 is a diagram showing a schematic diagram of the structure of the polarization camera 10.

[0026] The polarization camera 10 has a structure in which an imaging element 20 is combined with a polarizer array 21. The imaging element 20 is a device in which photoelectric conversion elements (also called pixels), such as a CCD (Charge Coupled Device) sensor or a CMOS (Complementary Metal Oxide Semiconductor) sensor, are arranged two-dimensionally, and is also called an image sensor. The polarizer array 21 is a device in which a large number of polarizers are arranged two-dimensionally, and the position and size are designed so that one polarizer 210 corresponds to one pixel (light receiving element) 200 of the imaging element 20. The polarizer 210 is an optical element that has the property of passing only linearly polarized light in a specific direction (the vibration direction of the linearly polarized light transmitted by the polarizer 210 is called the transmission axis direction of the polarizer 210). The polarizer array 21 of this embodiment has a structure in which polarizers 210 with four types of transmission axis directions (0 degrees, 45 degrees, 90 degrees, and 135 degrees) are regularly arranged, as shown in FIG. 2. Specifically, an array pattern is adopted in which linearly polarized light components having different transmission axis directions (0 degrees, 45 degrees, 90 degrees, 135 degrees) are incident on four 2×2 pixels of the imaging element 20.

[0027] The resolution (pixel pitch) and size (number of pixels) of the image sensor 20 and the polarizer array 21 may be appropriately designed according to the subject and the purpose of the image. The polarizer array 21 may be realized by a wire grid, a photonic crystal, or the like, and any of these may be used. In addition, in FIG. 2, four types of polarizers with transmission axis directions are combined into four pixels of 2×2, but the variation in the transmission axis directions and arrangement of the polarizers may be other configurations.

[0028] (Lighting equipment) An example of the configuration of illumination device 11 will be described with reference to Fig. 3A. Fig. 3A is a diagram showing a schematic structure of illumination device 11 of the present embodiment, showing the state when stage 13 is viewed from the polarization camera 10 side.

[0029] The lighting device 11 is composed of four rod-shaped lights 30 arranged to surround the stage 13. When the four rod-shaped lights 30 are turned on simultaneously, the subject W on the stage 13 can be illuminated from all four directions (X-positive direction, X-negative direction, Y-positive direction, and Y-negative direction). Furthermore, the lighting direction can be switched by selectively turning on only one of the rod-shaped lights 30. The rod-shaped lights 30 are composed of, for example, multiple LED light sources arranged on a substrate and a diffusion plate arranged to cover them.

[0030] The structure of the illumination device 11 is not limited to that shown in FIG. 3A. For example, a circular illumination device 11 as shown in FIG. 3B may be used. With this structure, the subject W can be illuminated simultaneously from all directions. Also, the illumination direction can be arbitrarily switched by selectively turning on the LED light source. Also, instead of omnidirectional illumination, a configuration in which the subject W is illuminated only from a specific direction as shown in FIG. 3C may be used. For example, if the range in which the subject W exists is limited or the angle of the subject surface to be detected is known or limited, it is sufficient to illuminate from a required direction. Alternatively, even if the illumination device 11 as shown in FIG. 3C is used, omnidirectional shooting may be performed by changing the relative position of the illumination device 11 and the subject W (for example, by rotating the subject W on the stage).

[0031] (Processing device) An example of the configuration of the processing device 12 will be described with reference to Fig. 4. Fig. 4 is a block diagram showing the logical configuration (functional configuration) of the processing device 12 of this embodiment.

[0032] The processing device 12 mainly comprises a camera control unit 40, an illumination control unit 41, and an image processing unit 42. The camera control unit 40 controls the polarization camera 10. For example, the camera control unit 40 controls the imaging conditions (exposure time, gain, etc.) of the polarization camera 10, captures image data from the polarization camera 10, calibrates the polarization camera 10, etc. The illumination control unit 41 controls the illumination device 11. For example, the illumination control unit 41 controls the illumination conditions (emission intensity, emission time, etc.) and controls the on / off of each light source. The image processing unit 42 processes the image captured from the polarization camera 10.

[0033] The image processing unit 42 of this embodiment has functions such as an image acquisition unit 420, a polarization extraction unit 421, a polarization information image generation unit 422, a display unit 423, and a parameter reception unit 424. The image acquisition unit 420 acquires an image (hereinafter also referred to as an "original image") captured by the polarization camera 10 via the camera control unit 40. The polarization extraction unit 421 performs a polarization extraction process to extract polarization information from the original image. The polarization information is information related to polarization resulting from specular reflection on a transparent object. The polarization information image generation unit 422 generates an image (hereinafter also referred to as a "polarization information image") representing the polarization information based on the extraction result of the polarization extraction process. The display unit 423 performs a process to display the generated polarization information image on a display device. The parameter reception unit 424 is a function to receive changes to various condition settings (parameters) that determine the operation of the imaging system 1. Details of each process of the image processing unit 42 will be described later.

[0034] The processing device 12 may be configured, for example, by a computer equipped with a processor such as a CPU or GPU, a memory as a main memory, a storage as an auxiliary memory, a display device, an input device such as a mouse or a touch panel, a network I / F, etc. The computer may be a general-purpose computer such as a personal computer, a tablet terminal, a smartphone, or a field computer, or may be an embedded computer or a dedicated device, or may be other equipment such as the polarization camera 10 or a PLC (Programmable Logic Controller). The configuration shown in Fig. 4 may utilize computer resources of a device. The configuration shown in Fig. 4 is realized by loading a program stored in a storage or the like into a memory and executing it with a processor. However, a part or all of the configuration shown in Fig. 4 may be configured with a dedicated device such as an ASIC (Application Specific Integrated Circuit) or an FPGA (Field Programmable Gate Array). Also, a part of the configuration or part of the processing shown in Fig. 4 may be executed by another device using cloud computing or distributed computing.

[0035] (Transparent object detection using a polarization camera) This section explains the basic principles of transparent object detection using a polarization camera.

[0036] As shown in FIG. 5A, when light 50 is obliquely incident on a transparent object W such as glass or transparent resin, In this case, most of the incident light 50 becomes refracted light 51 and passes through the transparent object W, but a part of the incident light 50 is specularly reflected at the surface (interface) of the transparent object W to become reflected light 52. In this specular reflection, a difference in reflectance occurs between the p-wave (polarized component whose electric field vibration direction is parallel to the incident surface) and the s-wave (polarized component whose electric field vibration direction is perpendicular to the incident surface (i.e., parallel to the reflecting surface)) contained in the incident light 50. FIG. 5B shows an example of the difference in reflectance between the s-wave and the p-wave, with the incident angle on the horizontal axis and the reflectance on the vertical axis. As shown in this example, the reflectance of the s-wave increases monotonically with the incident angle α, but the reflectance of the p-wave gradually decreases as the incident angle α increases from 0 degrees, and the reflectance becomes 0 at a certain angle (called the Brewster angle). Therefore, the reflected light 52 observed in the case of specular reflection is dominated by the s-wave, i.e., the polarized component vibrating in a direction parallel to the surface of the transparent object W, which is the reflecting surface. By taking advantage of this property and capturing polarized light with a polarization camera, it is possible to image the surface (reflective surface) of a transparent object W.

[0037] Incidentally, not only polarized light due to specular reflection but also light diffusely reflected from the subject and background enters a polarization camera. In imaging a transparent object, the diffuse reflected light component is unnecessary and can cause artifacts, so it is first necessary to extract only the polarized light component from the image captured by the polarization camera. To achieve this, in this embodiment, the difference in transmission characteristics due to the polarizer is utilized.

[0038] With reference to FIG. 6, a method for extracting polarized components from an image of a polarized camera will be described. The intensity of the polarized components received by the light receiving element is maximum when the polarization direction θ coincides with the transmission axis direction of the polarizer, decreases as the transmission axis direction deviates from the polarization direction θ, and is minimum when the transmission axis direction and the polarization direction θ are perpendicular to each other. In other words, when the transmission axis direction of the polarizer is changed from 0 degrees to 180 degrees, the intensity of the polarized components received by the light receiving element changes like a sine wave. On the other hand, the intensity of the diffuse reflection light component contained in the incident light is constant, regardless of the transmission axis direction of the polarizer. Therefore, it is possible to extract only the information of the polarized components contained in the incident light based on the change in light intensity (transmission axis direction dependency) when observed through multiple types of polarizers with different transmission axis directions.

[0039] For example, as shown in FIG. 6, a case is assumed in which incident light containing a polarization component with a polarization direction θ is observed through four types of polarizers 210 with transmission axis directions of 0 degrees, 45 degrees, 90 degrees, and 135 degrees. In this case, the values ​​of the pixels 200 corresponding to the polarizers 210 (corresponding to the light intensity of the transmitted light) have differences that depend on the transmission axis direction. The polarization direction θ and polarization intensity of the polarization component can be estimated by fitting a sine wave to the pixel values ​​of these four points. In FIG. 6, the angle at which the fitting curve is maximized represents the polarization direction θ of the polarization component, and the amplitude of the fitting curve represents the polarization intensity (also called the degree of polarization). By applying such a polarization extraction process to each pixel of the image obtained by the polarization camera 10, information on the polarized light (polarization direction, polarization intensity, etc.) specularly reflected by a transparent object W present in the field of view of the polarization camera 10 can be captured.

[0040] The polarization extraction process by the polarization extraction unit 421 of this embodiment also follows the same basic principle of extracting polarization information based on the change in light intensity (transmission axis direction dependency) when observed through multiple types of polarizers with different transmission axis directions. However, in order to further improve accuracy and usability, unique processes such as selection of mask size (a parameter that specifies the range of neighboring pixels to be referenced when calculating polarization) and noise removal are added. Details will be described later.

[0041] (Example of imaging system operation) An example of the operation of the imaging system 1 will be described with reference to the flowchart of FIG.

[0042] In step S70, a transparent object to be the subject W is placed on the stage 13. The subject W may be transported and positioned by a robot or a transport device, or an operator may transport and position the subject W on the stage 13. The subject W may be placed on top.

[0043] In step S71, the user utilizes a user interface provided by the parameter receiving unit 424 of the processing device 12 to input the imaging conditions of the polarization camera 10 and the illumination conditions of the illumination device 11. If there is no need to change the imaging conditions or the illumination conditions, the process of step S71 may be skipped.

[0044] In step S72, an image of the subject W is captured. Specifically, the illumination control unit 41 of the processing device 12 turns on the illumination device 11 in accordance with the given illumination conditions, and irradiates the subject W with illumination light. Then, with the subject W illuminated, the camera control unit 40 controls the polarization camera 10 in accordance with the given imaging conditions to capture the image.

[0045] In step S73, the image acquisition unit 420 of the processing device 12 captures the image (original image) captured in step S72 from the polarization camera 10. The captured original image data is stored in a memory or storage and is used for processing by the image processing unit .

[0046] In step S74, the user uses a user interface provided by parameter receiving unit 424 to input image processing parameters to be used in image processing unit 42. If there is no need to change the image processing parameters, step S74 may be skipped.

[0047] FIG. 8 is an example of a user interface (UI) for inputting image processing parameters provided by the parameter receiving unit 424. This UI screen is composed of a mask size setting 80, a noise removal setting 81, a display image setting 82, an entire image display area 83, an enlarged image display area 84, and an enlargement setting 85. The mask size setting 80 is a UI for setting a "mask size" that specifies the range of neighboring pixels to be referenced when calculating polarization information from an original image. In the example of FIG. 8, the mask size is selected from 2×2, 3×3, 4×4, and 5×5, but the user may input any size. The noise removal setting 81 is a UI for setting conditions related to a "noise removal process" that removes or reduces noise included in the calculation result of polarization information. In the example of FIG. 8, it is possible to select whether or not to apply the noise removal process, and if "yes", it is possible to further input parameters (comparison range, polarization angle difference threshold, polarization intensity difference threshold) used in the noise removal process. The display image setting 82 is a UI for setting the type of image to be displayed in the entire image display area 83 and the enlarged image display area 84. 8, the original image, the polarization intensity image, the polarization direction image, and the brightness image can be selected. The entire image display area 83 is an area where the entire image (low resolution) is displayed, and the enlarged image display area 84 is an area where a part of the image (part indicated by a frame 86) is displayed at high resolution. The enlargement setting 85 is a UI for setting the magnification and range of the enlarged image to be displayed in the enlarged image display area 84.

[0048] In step S75, the polarization extraction unit 421 calculates polarization information from the original image captured in step S73. The polarization information calculation process will be described in detail later.

[0049] In step S76, the polarization extraction unit 421 applies a noise removal process to the calculation result in step S75. Note that if the noise removal is set to "no" in the UI of Fig. 8, the process of step S76 is skipped. The details of the noise removal process will be described later.

[0050] In step S77, the polarization information image generator 422 generates a polarization information image based on the calculation result of the polarization extractor 421. In this embodiment, three types of images are generated as the polarization information images: a polarization intensity image, a polarization direction image, and a brightness image. The polarization intensity image is an image that represents the polarization intensity (degree of polarization) with shading, and the polarization direction image is an image that represents the polarization direction with shading or pseudo color. The brightness image is an image that represents the light intensity of the polarized light components by using shades of gray.

[0051] In step S78, the display unit 423 displays the image selected in the display image setting 82 of the UI of FIG.

[0052] Here, when the user changes the display image setting 82 on the UI of Fig. 8, the images displayed in the whole image display area 83 and the enlarged image display area 84 are switched accordingly. By using this UI, the appearance of the original image can be compared with the polarization information (polarization intensity, polarization direction, brightness), which makes it easy to check, for example, whether the calculation of polarization information and the removal of noise have been performed appropriately, and to check the shape and state of a transparent object. Note that, when the mask size setting 80 or the noise removal setting 81 is changed on the UI of Fig. 8, the processes of steps S75 to S78 may be executed again. By using such a UI, it is possible to adjust image processing parameters (mask size, whether or not noise removal is performed, comparison range, threshold value, etc.) while checking the calculation result (polarization information image).

[0053] (Polarization information calculation process) With reference to Fig. 9 and Figs. 10A to 10D, an implementation example of the polarization information calculation process (step S75 in Fig. 7) of the polarization extraction unit 421 will be described. Fig. 9 is a flowchart of the polarization information calculation process, and Figs. 10A to 10D are diagrams showing examples of neighboring pixels. Here, the size (number of pixels) of the original image is M rows x N columns, the pixel in the i-th row and j-th column is represented as (i, j), and the pixel value (light intensity) of that pixel is represented as I(i, j) (i = 1, 2,..., M, j = 1, 2,..., N). Figs. 10A to 10D show an enlarged portion of the original image, where each rectangle represents a pixel, the arrow in the pixel represents the transmission axis direction, and the symbols written in the margins represent the row number and column number of the pixel.

[0054] In step S10, the polarization extraction unit 421 selects a pixel of interest to be calculated from the original image. In step S10, it is assumed that the pixels of interest are selected in order from (1,1) to (M,N).

[0055] In step S11, the polarization extraction unit 421 selects neighboring pixels to be referenced when calculating the polarization information of the pixel of interest. The range of neighboring pixels selected here is determined by the mask size specified in the mask size setting 80 in FIG. 8. First, the default mask size of 2×2 is taken as an example for explanation. When the mask size is 2×2, as shown in FIG. 10A, three pixels (i-1, j-1), (i-1, j), and (i, j-1) adjacent to the pixel of interest (i, j) are selected as neighboring pixels. Note that if the pixel of interest is on the boundary of the original image and there are no neighboring pixels, the processes of steps S11 to S12 may be skipped, or an appropriate value may be padded.

[0056] In step S12, the polarization extraction unit 421 calculates polarization information of the pixel of interest based on the light intensities (pixel values) of the pixel of interest and the neighboring pixels. If the light intensities when the transmission axis directions are 0 degrees, 45 degrees, 90 degrees, and 135 degrees are I1, I2, I3, and I4, respectively, the brightness I0, polarization direction θ, and polarization intensity D of the pixel of interest can be calculated by the following formulas.

number

[0057] In the example of FIG. 10A, the transmission axis directions corresponding to the four pixels (i-1, j-1), (i-1, j), (i, j-1), and (i, j) are 90 degrees, 135 degrees, 45 degrees, and 0 degrees, respectively, and therefore I1 to I4 in equations (1) to (3) are given by the following equations. I1 = I(i,j) I2 = I(i,j-1) I3 = I(i-1,j-1) I4 = I(i-1,j)

[0058] The brightness I0 in formula (1) is an index that represents the intensity of the incident light that enters the pixel of interest. The polarization direction θ in formula (2) represents the principal axis direction of the polarization component contained in the incident light that enters the pixel of interest. The polarization intensity D in formula (3) is an index that represents the degree of polarization of the polarization component contained in the incident light that enters the pixel of interest.

[0059] The above formulas (1) to (3) are merely examples, and the indices corresponding to the brightness, polarization direction, and polarization intensity may be calculated using other formulas. For example, the coefficients in formulas (1) to (3) may be replaced with other ones, or formula (3) may be replaced with the following formula (4).

number

[0060] In step S13, the polarization extraction unit 421 checks whether the calculation process for polarization information has been completed for all pixels of the original image (i.e., whether i=M and j=N), and if there are unprocessed pixels, returns to step S10 to perform processing on the next pixel of interest.

[0061] Through the above-mentioned process, the polarization direction θ, polarization intensity D, and brightness I0 are calculated for all pixels of the original image. The calculation results of the polarization information calculation process are stored in memory or storage in a data format that associates, for example, the coordinates (i,j) of each pixel with the polarization direction θ(i,j), polarization intensity D(i,j), and brightness I0(i,j), and are used for subsequent processing.

[0062] Next, a processing example when the mask size is larger than 2×2 will be described with reference to FIGS. 10B to 10D.

[0063] As shown in FIG. 10B, when the mask size is 3×3, eight pixels (i-1, j-1), (i-1, j), (i-1, j+1), (i, j-1), (i, j+1), (i+1, j-1), (i+1, j), (i+1, j+1) adjacent to the pixel of interest (i, j) are selected as neighboring pixels in step S11. In the example of FIG. 10B, the nine pixel groups including the pixel of interest and the neighboring pixels include one pixel with a transmission axis direction of 0 degrees, two pixels with a transmission axis direction of 45 degrees and two pixels with a transmission axis direction of 135 degrees, and four pixels with a transmission axis direction of 90 degrees. In this way, pixels with the same transmission axis direction are selected as neighboring pixels in step S11. When there are a plurality of elements, in the calculation of step S12, the representative values ​​of the pixels in each transmission axis direction may be substituted for I1 to I4 in formulas (1) to (3).

[0064] For example, when the average value is used as the representative value, I1 to I4 are given by the following formula. I1 = I(i,j) I2={I(i,j-1)+I(i,j+1)} / 2 I3={I(i-1,j-1)+I(i-1,j+1)+I(i+1,j-1)+I(i+1,j+1)} / 4 I4={I(i,j-1)+I(i,j+1)} / 2

[0065] As the representative value, in addition to the average value, the mode, median, maximum value, minimum value, etc. may be used. Furthermore, without using the representative value, I1 to I4 may be estimated by fitting a sine wave to the values ​​of the nine pixels, or the brightness I0, polarization direction θ, and polarization intensity D may be calculated directly.

[0066] FIG. 10C shows an example where the mask size is 4×4. The range selected is the 8 neighboring pixels when the mask size is 3×3 plus 7 pixels (i-2, j-2), (i-2, j-1), (i-2, j), (i-2, j+1), (i-1, j-2), (i, j-2), (i+1, j-2). In the example of FIG. 10C, the selected range includes 4 pixels each at 0 degrees, 45 degrees, 90 degrees, and 135 degrees. In this case, as in the case of 3×3, the representative values ​​for each transmission axis direction can be substituted for I1 to I4, or a sine wave can be fitted to the values ​​of 16 pixels.

[0067] FIG. 10D shows an example of a mask size of 5×5. The range selected is the 15 neighboring pixels when the mask size is 4×4 plus 9 pixels (i-2,j+2), (i-1,j+2), (i,j+2), (i+1,j+2), (i+2,j-2), (i+2,j-1), (i+2,j), (i+2,j+1), (i+2,j+2). In the example of FIG. 10D, the selected range includes 9 pixels at 0 degrees, 6 pixels at 45 degrees and 6 pixels at 135 degrees, and 4 pixels at 90 degrees. In this case, as in the case of 3×3, the representative values ​​for each transmission axis direction can be substituted for I1 to I4, or a sine wave can be fitted to the values ​​of the 25 pixels.

[0068] In this embodiment, the mask size, i.e., the pixel range to be referred to when calculating polarization information, can be changed as desired. The user can select an appropriate mask size depending on the shape and state of the subject, or the purpose of the system.

[0069] For example, as shown in FIG. 11A, in the case where the polarized camera 10 captures specular reflection from a curved portion of the subject W, the reflecting surface is extremely narrow, so that the range in which the polarized light is incident may be a narrow region of a few pixels wide (although this depends on the resolution of the polarized camera 10). In such a case, if the mask size is too large, as shown in FIG. 11B, pixels in which no polarized light is incident may also be referenced, which may reduce the detectability of the polarization information. Therefore, it is preferable to select a mask size smaller than the range in which the polarized light is incident. This allows the polarization information in the narrow region to be appropriately extracted.

[0070] On the other hand, as shown in Figure 12, when inspecting for stains or foreign objects on a surface, it is expected that similar polarization characteristics will appear over a relatively wide range on the image due to the reflection characteristics of a flat surface (specular reflection, scattering strength). In such cases, if the mask size is too small, the inspection may be sensitive to local differences in reflectance, resulting in unstable inspection. Therefore, by setting a somewhat larger mask size, it is possible to eliminate local differences in reflectance. It is preferable to average out disturbance factors such as differences in reflectance, thereby improving the stability of the inspection.

[0071] In this embodiment, four mask sizes, 2×2, 3×3, 4×4, and 5×5, are shown as examples of mask sizes, but the mask size is not limited to these. For example, a mask larger than 5×5 may be used, or a mask having a shape other than a square may be used. In addition, the user may be allowed to design the size and shape of the mask as desired.

[0072] (Noise removal processing) The above-mentioned polarization information calculation process utilizes the principle that local pixel value differences are regarded as differences in the amount of transmission due to differences in the directions of the transmission axes of the polarizers, and polarization is detected based on this. Therefore, for example, when a high spatial frequency luminance change occurs in an edge part or texture part of a subject, this may be mistaken as a pixel value difference due to polarization, leading to erroneous detection of polarization. The noise removal process is a process for removing or reducing noise due to such erroneous detection from the calculation results of the polarization information calculation process.

[0073] An implementation example of the noise removal process (step S76 in FIG. 7) will be described with reference to FIG 13, FIG 14A, and FIG 14B. FIG 13 is a flowchart of the noise removal process, and FIG 14A and FIG 14B are diagrams showing examples of comparison ranges.

[0074] In step S20, the polarization extraction unit 421 reads the parameters set in the noise removal setting 81 in Fig. 8. Here, four parameters are read: with / without noise removal, comparison range, polarization angle difference threshold, and polarization intensity difference threshold. If the noise removal is set to "without," the subsequent processing is skipped (step S21). If the noise removal is set to "with," the processing proceeds to step S22.

[0075] In step S22, the polarization extraction unit 421 reads the calculation results of the polarization information calculation process from the memory or storage. Here, data is obtained in which the coordinates (i, j) of each pixel are associated with the polarization direction θ(i, j), the polarization intensity D(i, j), and the brightness I0(i, j).

[0076] In step S23, the polarization extraction unit 421 selects a target pixel to be processed. In step S23, the target pixels are selected in order from (1,1) to (M,N).

[0077] In step S24, the polarization extraction unit 421 compares the polarization directions of the target pixel and its surrounding pixels to determine whether the polarization directions are similar. As shown in FIG. 14A, when the comparison range is set to "1", a comparison is made between the target pixel (i, j) and the three surrounding pixels (i, j+1), (i+1, j), and (i+1, j+1). Also, as shown in FIG. 14B, when the comparison range is set to "2", a comparison is made between the target pixel (i, j) and the eight surrounding pixels (i-1, j-1), (i-1, j), (i-1, j+1), (i, j-1), (i, j+1), (i+1, j-1), (i+1, j), and (i+1, j+1). Then, when any combination in which the difference in the polarization direction is greater than the polarization angle difference threshold is found, it is determined to be "dissimilar" and the process proceeds to step S26. On the other hand, if it is determined to be "similar," the process proceeds to step S25.

[0078] In step S25, the polarization extraction unit 421 compares the polarization intensity between the target pixel and its surrounding pixels, and determines whether the polarization intensities are similar. The method of selecting the comparison range is as shown in Fig. 14A and Fig. 14B. If any combination is found in which the difference in polarization intensity is greater than the polarization intensity difference threshold, it is determined to be "dissimilar" and the process proceeds to step S26. On the other hand, if it is determined to be "similar," the process proceeds to step S27.

[0079] If either the polarization direction or the polarization intensity is "dissimilar," the polarization information of the target pixel is an unusual value (outlier) compared to the surrounding pixels, and is likely to be noise. Therefore, in step S26, the polarization extraction unit 421 deletes the polarization information of the target pixel (polarization direction θ and polarization intensity D).

[0080] If both the polarization direction and the polarization intensity are "similar," in step S27, the polarization extraction unit 421 selects the pixel with the greatest polarization intensity among the target pixel and the pixels in the comparison range, and replaces the polarization direction and polarization intensity of the target pixel with the polarization direction and polarization intensity of the selected pixel. This process provides a smoothing effect that reduces local variations in the polarization direction and polarization intensity.

[0081] In step S28, the polarization extraction unit 421 checks whether the noise removal process has been completed for all pixels, and if there are any unprocessed pixels, the process returns to step S23 to process the next target pixel.

[0082] By the above-mentioned process, noise (outliers) are removed and smoothing is performed on the polarization direction θ and polarization intensity D, which are the calculation results of the polarization information calculation process. The corrected data is overwritten and saved in the memory or storage.

[0083] (Application example) An example will be described in which the imaging system 1 described above is applied to detection and inspection in the manufacturing process of syringes filled with a liquid medicine.

[0084] In the manufacturing process of filled syringes, a robot is usually used to pick and place the syringe and inject the liquid. If the syringe is in an incorrect position or lying on its side, it may be damaged or the manufacturing equipment may break. Therefore, a mechanism is needed to monitor the position and posture of the syringe and output an error or stop the equipment if a problem occurs.

[0085] Fig. 15A shows a state in which syringes 101 arranged in nest 100 are observed by zenith camera 102. Two syringes 101b are not inserted into the holes in nest 100 and are lying on their sides. However, because syringes 101 are colorless and transparent, even if they are photographed using normal transmitted illumination and an optical camera, as shown in Fig. 15B, the boundary between syringe 101 and the background (white nest 100) is barely discernible in the image, making it difficult to stably detect lying syringes 101b.

[0086] In contrast, when polarization information is extracted using imaging system 1 according to an embodiment of the present invention, only the specular reflection on the surface of syringe 101 can be imaged, as shown in Fig. 15C. In the polarization information image, a circular image appears for syringe 101a in the correct position, and a linear band-like image appears for syringe 101b lying on its side. By using such polarization information images, it becomes possible to stably inspect the position and posture of the syringe, and detect a syringe in an incorrect position or posture.

[0087] Computer image recognition technology may be used to detect the syringe from the polarization information image and determine the posture of the syringe. For example, a syringe in the correct posture will have mirror reflections on the inner circle and the flange around it. By focusing on this image feature and detecting the shape of the circle and the flange from the polarization information image using a method such as pattern matching, it becomes possible to automatically recognize a syringe in the correct posture. In the case of a syringe lying on its side, for example, first, a set of adjacent pixels (pixel region) having a specific polarization direction range (e.g., 0° to 60°) is extracted, and then, from the extracted pixel region, a set of pixels having a width and length equal to or greater than a threshold value is extracted. By performing such processing, it is possible to automatically recognize the strip-shaped pixel area of ​​the syringe lying on its side.

[0088] (others) The above-described embodiment merely describes an exemplary configuration of the present invention. The present invention is not limited to the above-described specific embodiment, and various modifications are possible within the scope of the technical concept thereof.

[0089] For example, the arrangement and configuration of the polarization camera and the lighting device are not limited to those shown in Fig. 1 and Fig. 3A to Fig. 3C, and may be appropriately designed according to the subject and the purpose. In other words, it is sufficient that the specular reflection of the illumination light can be observed by the polarization camera. Furthermore, multiple polarization cameras may be provided, and movable polarization cameras and lighting devices may be used.

[0090] In the above embodiment, polarizers with 0 degree, 45 degree, 90 degree, and 135 degree are used, but the setting of the transmission axis direction is not limited to this. It is sufficient that the image sensor can observe the light that has passed through the polarizers with different transmission axis directions and can extract the polarization information contained in the incident light from the observation result.

[0091] In the UI screen of FIG. 8, the mask size used in the polarization information calculation process and the parameters used in the noise removal process can be set, but the configuration of the UI screen and the parameter items are merely examples. Any UI that allows the user to change the parameters used in the polarization extraction process may be used. In other words, it is sufficient if the user himself / herself can change the logic of the polarization extraction appropriately according to the subject, the purpose, etc. In addition, in the above embodiment, the parameter receiving unit 424 provides a user interface for setting parameters, but the parameter receiving unit 424 may receive a parameter change instruction from the user from an external device via a network.

[0092] <Appendix 1> An imaging system (1) for imaging a transparent object (W), comprising: An illumination device (11) for illuminating a transparent object (W); A polarization camera (10) having an image sensor (20) in which polarizers (210) having different transmission axis directions are regularly arranged; A processing device (12), The processing device (12) comprises: an image acquisition means (420) for acquiring an original image captured by the polarization camera (10) while the transparent object (W) is illuminated by the illumination device (11); a polarization extraction means (421) for performing a polarization extraction process to extract, for each pixel of the original image, polarization information that is information about polarization resulting from specular reflection on the transparent object (W); an image generating means (422) for generating a polarization information image by imaging the polarization information based on an extraction result of the polarization extraction process; and a reception means (424) for receiving a change in a parameter used in the polarization extraction process. An imaging system (1).

[0093] <Appendix 2> A method for controlling an imaging system (1) including an illumination device (11), a polarization camera (10) having an imaging element (20) in which polarizers (210) having different transmission axis directions are regularly arranged, and a processing device (12), comprising the steps of: Illuminating a transparent object (W) with the illumination device (11); Taking an image using the polarization camera (10); A step (S73) of importing an original image captured by the polarization camera (10) into the processing device (12); and executing a polarization extraction process (S75, S76) by the processing device (12) to extract polarization information, which is information regarding polarization resulting from specular reflection on the transparent object (W), for each pixel of the original image, and generating a polarization information image (S77) by imaging the polarization information based on the extraction result of the polarization extraction process (S75, S76), A control method characterized in that parameters used in the polarization extraction process (S75, S76) can be changed (S74). [Explanation of symbols]

[0094] 1: Imaging system 10: Polarized camera 11: Lighting equipment 12: Processing device 13: Stage 20: Image sensor 21: Polarizer array 30: Rod lighting

Claims

1. 1. An imaging system for imaging a transparent object, comprising: an illumination device for illuminating a transparent object; a polarization camera having an image sensor in which polarizers having different transmission axis directions are regularly arranged; A processing device, The processing device includes: an image acquisition means for acquiring an original image captured by the polarization camera while the transparent object is illuminated by the illumination device; a polarization extraction means for executing a polarization extraction process to extract, for each pixel of the original image, polarization information that is information regarding polarization resulting from specular reflection on the transparent object; an image generating means for generating a polarization information image by imaging the polarization information based on an extraction result of the polarization extraction process; and a reception unit that receives a change to a parameter used in the polarization extraction process.

2. The polarization extraction process includes a polarization information calculation process that calculates polarization information of a pixel of interest by referring to pixel values ​​of the pixel of interest and its neighboring pixels.

2. The imaging system according to claim 1 .

3. The parameters include a mask size, which is an item that defines the range of neighboring pixels to be referenced when calculating polarization information in the polarization information calculation process.

3. The imaging system according to claim 2.

4. The receiving means includes a user interface that allows a user to select a mask size to be used in the polarization information calculation process from a plurality of mask sizes having different sizes.

4. The imaging system according to claim 3.

5. The polarization extraction process includes a noise removal process for removing or reducing noise included in the calculation result after the polarization information of each pixel of the original image is calculated by the polarization information calculation process.

5. The imaging system according to claim 2, wherein the imaging device is a digital camera.

6. The noise removal process includes a process of determining whether the polarization information of the target pixel and the surrounding pixels is similar or dissimilar by comparing the polarization information between the target pixel and the surrounding pixels, and deleting the polarization information of the target pixel when it is determined that the polarization information is dissimilar.

6. The imaging system according to claim 5.

7. The noise removal process includes a process of determining whether the polarization information of the target pixel and the surrounding pixels is similar or dissimilar by comparing the polarization information between the target pixel and the surrounding pixels, and replacing the polarization information of the target pixel with the polarization information of a pixel selected from the target pixel and the surrounding pixels when it is determined that the polarization information is similar.

7. The imaging system according to claim 5, wherein the imaging device is a digital camera.

8. 8. The imaging system according to claim 6, wherein the parameters include an item that defines a range of the surrounding pixels to be compared with the target pixel.

9. The parameters include items that define criteria for determining whether the polarization information is similar or dissimilar.

9. The imaging system according to claim 6, wherein the imaging system comprises: a first imaging unit;

10. The parameters include an item for specifying whether or not to apply the noise removal process to the calculation result of the polarization information calculation process.

10. The imaging system according to claim 5, wherein the imaging system further comprises:

11. A control method for an imaging system including an illumination device, a polarization camera having an imaging element in which polarizers having different transmission axis directions are regularly arranged, and a processing device, comprising: illuminating a transparent object with the illumination device; Taking an image using the polarization camera; A step of inputting an original image captured by the polarization camera into the processing device; and executing a polarization extraction process by the processing device to extract polarization information, which is information regarding polarization resulting from specular reflection on the transparent object, for each pixel of the original image, and generating a polarization information image by imaging the polarization information based on an extraction result of the polarization extraction process. The parameters used in the polarization extraction process can be changed. A control method comprising:

12. A program for causing a processor to execute each step of the control method according to claim 11.

Images