Imaging system and control method thereof

Abstract

To provide a new technology for assisting visual confirmation or read of information obtained by a polarization camera.SOLUTION: A processing device includes: image acquiring means for acquiring an original image captured by means of a polarization camera while a transparent object is illuminated by an illuminating device; polarized light extracting means for executing polarization extraction processing to extract polarization information, which is information relating to polarized light originating from specular reflection on the transparent object, for each pixel in the original image; image generating means for generating a polarization information image formed by converting the polarization information into an image, on the basis of a result extracted by the polarization extraction processing; and display means for displaying the polarization information image on a screen. The polarization information includes information relating to a polarization direction. The polarization information image includes a polarization direction image representing the polarization direction, which is one item of the polarization information, using pseudo-colors. The display means displays, on the screen, the polarization direction image together with an auxiliary image indicating association between the pseudo-colors and the polarization direction.SELECTED DRAWING: Figure 16

Description

[Technical Field]

[0001] The present invention relates to techniques for imaging transparent objects. [Background technology]

[0002] There is a known technology for imaging transparent objects using a polarization camera that has an imaging element in which polarizers with different transmission axis directions are regularly arranged (see Patent Document 1). This technology is expected to be applied to, for example, the detection and inspection of objects made of glass or transparent resin. [Prior art documents] [Patent documents]

[0003] [Patent Document 1] Japanese Patent Publication No. 2020-17688 Summary of the Invention [Problem to be solved by the invention]

[0004] When light is shone on a transparent object, some of the light is specularly reflected from the surface of the object. Because specularly reflected light has polarization properties, it can be observed with a polarization camera, and the surface (reflective surface) of the transparent object can be imaged based on the observation results.

[0005] However, polarization cameras not only receive specularly reflected light, but also light diffusely reflected from the subject and background, which can become a source of noise. Furthermore, the way polarization appears varies depending on the subject's shape and reflection characteristics. Therefore, stable detection and inspection using a polarization camera often requires fine-tuning the image processing parameters, imaging conditions, and lighting conditions to suit the subject and the situation. However, even if users visually inspect the results obtained by a polarization camera, it is difficult to determine whether the polarization information has been correctly extracted or to distinguish between polarization and noise. Therefore, adjusting the image processing parameters and imaging and lighting conditions requires advanced skills, knowledge, and trial and error.

[0006] The present invention has been made in consideration of the above-mentioned circumstances, and its purpose is to provide a new technology for assisting in the visual confirmation and interpretation of information obtained by a polarization camera. [Means for solving the problem]

[0007] The present invention provides an imaging system for imaging a transparent object, comprising: an illumination device for illuminating the transparent object; a polarization camera having an imaging element in which polarizers with different transmission axis directions are regularly arranged; and a processing device, wherein the processing device comprises: an image acquisition means for acquiring an original image captured by the polarization camera while the transparent object is illuminated by the illumination device; a polarization extraction means for performing a polarization extraction process to extract, for each pixel of the original image, polarization information, which is information regarding polarization resulting from specular reflection on the transparent object; an image generation means for generating a polarization information image that visualizes the polarization information based on the extraction result of the polarization extraction process; and a display means for displaying the polarization information image on a screen, wherein the polarization information includes information on the polarization direction, and the polarization information image includes a polarization direction image that represents the polarization direction, which is one piece of the polarization information, in pseudo-color; and the display means displays, on the screen, an auxiliary image showing the correspondence between pseudo-color and polarization direction together with the polarization direction image.

[0008] According to this configuration, the polarization direction, which is one piece of polarization information, is displayed in pseudo-color, so the user can visually recognize differences in the orientation of the surface (reflective surface) of a transparent object as differences in color. Moreover, an auxiliary image showing the correspondence between pseudocolor and polarization direction is also displayed, making it possible to quantitatively grasp the polarization direction (i.e., the orientation of the reflective surface) from the color of the image. Therefore, providing such a display screen can assist in understanding the shape and structure of transparent objects from polarization information. Furthermore, since it becomes possible to discover noise and artifacts contained in the image using clues such as color distribution and discontinuities, it is expected that it will be easier to confirm whether polarization information has been correctly extracted and to evaluate the quality of polarization information images.

[0009] The processing device may further include an input unit that provides a user interface for allowing a user to specify an image to be displayed from among a plurality of images including the original image and the polarization information image, and the display unit may be configured to display the image specified by the user interface on the screen. This configuration allows the user to specify the image to be displayed on the screen (e.g., an image to be confirmed). Furthermore, since the original image can be displayed in addition to the polarization information image, it is possible to use the original image and the polarization information image to compare them visually to confirm whether the polarization information has been correctly extracted.

[0010] The polarization information may include multiple items of polarization information, and the polarization information image may include multiple types of polarization information images corresponding to the multiple items of polarization information. This enables, for example, visual comparison of the original image with the polarization information image, or visual comparison of different types of polarization information images, to confirm whether the polarization information has been correctly extracted. The multiple items of polarization information may include, for example, polarization direction, polarization intensity, and brightness. In this case, the polarization direction image may be an image in which the hue of pixels is changed according to the polarization direction, and the saturation or brightness of the pixels is changed according to the polarization intensity.

[0011] The user interface allows a user to alternatively select an image to be displayed from the plurality of images, and the display means may switch the image to be displayed on the screen in response to the user's selection. At this time, it is preferable that the image before switching and the image after switching are displayed in a state where they are aligned. This makes it possible to easily compare the same positions of the images by alternately switching between the original image of the subject and the polarization information image.

[0012] The user interface allows the user to select two or more images to be displayed from the plurality of images, and the display means may display the two or more images selected by the user on the screen in a manner that allows comparison. Examples of a "comparable manner" include a manner in which two or more images are displayed side by side on the screen, a manner in which two or more images are displayed superimposed, or a manner in which two or more images are automatically switched between and displayed like a slideshow. Such a display allows the images to be easily compared.

[0013] The polarization extraction process may include a noise reduction process for removing or reducing noise contained in the calculation result after calculating the polarization information for each pixel of the original image. By performing the noise reduction process, high-quality polarization information can be extracted.

[0014] The noise removal process may include a process of comparing the polarization information of a target pixel and its surrounding pixels to determine whether the polarization information of the target pixel and the surrounding pixels is similar or dissimilar, and deleting the polarization information of the target pixel if it is determined that the polarization information is dissimilar. This process makes it possible to remove unreliable polarization information.

[0015] The noise removal process may include a process of comparing the polarization information of a target pixel and its surrounding pixels to determine whether the polarization information of the target pixel and the surrounding pixels is similar or dissimilar, and if it is determined that the polarization information of the target pixel is similar, replacing the polarization information of the target pixel with the polarization information of a pixel selected from the target pixel and the surrounding pixels. This process smooths the polarization information, thereby further improving the signal-to-noise ratio of the polarization information.

[0016] The present invention may provide a control method for an imaging system comprising an illumination device, a polarization camera having an imaging element in which polarizers with different transmission axis directions are regularly arranged, and a processing device, the control method including the steps of: illuminating a transparent object with the illumination device; photographing with the polarization camera; importing the original image photographed by the polarization camera into the processing device; performing a polarization extraction process with the processing device to extract polarization information, which is information about polarization resulting from specular reflection on the transparent object, for each pixel of the original image; generating a polarization information image that visualizes the polarization information based on the extraction result of the polarization extraction process; and displaying the polarization information image on a screen, wherein the polarization information includes information about the polarization direction, and the polarization information image includes a polarization direction image that represents the polarization direction, which is one piece of the polarization information, in pseudo-color, and an auxiliary image showing the correspondence between the pseudo-color and the polarization direction is displayed on the screen together with the polarization direction image.

[0017] The present invention may provide a program for causing a processor to execute each step of the above control method.

[0018] The present invention may be understood as an imaging system having at least some of the above means, or as a processing device, control device, or image processing device. The present invention may also be understood as a device that detects or inspects transparent objects using polarization information images obtained by an imaging system. The present invention may also be understood as a control method, image processing method, detection method, or inspection method for an imaging system that includes at least some of the above processes, or as a program for realizing such a method or a recording medium on which the program is recorded. The above means and processes can be combined with each other as much as possible to constitute the present invention. [Effects of the Invention]

[0019] According to the present invention, it is possible to assist in visual confirmation and interpretation of information obtained by a polarization camera. [Brief explanation of the drawings]

[0020] [Figure 1] FIG. 1 is a diagram schematically showing the overall configuration of an imaging system according to an embodiment of the present invention. [Figure 2] FIG. 2 is a diagram showing a schematic structure of a polarization camera. [Figure 3] FIG. 3A is a diagram schematically illustrating the structure of a lighting device, and FIGS. 3B and 3C are diagrams illustrating other configuration examples of the lighting device. [Figure 4] FIG. 4 is a block diagram showing the logical configuration (functional configuration) of the processing device. [Figure 5] 5A and 5B are diagrams illustrating specular reflection. [Figure 6] FIG. 6 is a diagram illustrating a method for extracting polarization components from an image captured by a polarization camera. [Figure 7] FIG. 7 is a flowchart showing an example of the operation of the imaging system. [Figure 8] FIG. 8 shows an example of a user interface for inputting image processing parameters. [Figure 9] FIG. 9 is a flowchart of the polarization information calculation process. [Figure 10] 10A to 10D are diagrams showing examples of neighboring pixels. [Figure 11] FIG. 11A is a diagram showing an example of specular reflection at a curved surface portion, and FIG. 11B is a diagram showing an example of selecting a mask size when the range into which polarized light is incident is narrow. [Figure 12] FIG. 12 is a diagram showing an example in which a large mask size is suitable. [Figure 13] FIG. 13 is a flowchart of the noise removal process. [Figure 14] FIG. 14 is a diagram showing an example of a comparison range. [Figure 15] FIG. 15 is a diagram showing an application example of the imaging system. [Figure 16] FIG. 16 is a diagram showing an example of a display of a polarization direction image on the UI screen of FIG. [Figure 17] FIG. 17 is a diagram showing an example of displaying polarization information. [Figure 18] FIG. 18A is a diagram showing an example of noise and artifacts contained in a polarization direction image, FIG. 18B is a diagram showing another example of an auxiliary image, and FIG. 18C is a diagram showing another example of an auxiliary image. [Figure 19] FIG. 19 is a diagram showing another example of a UI screen for specifying an image to be displayed and checking the image. DETAILED DESCRIPTION OF THE INVENTION

[0021] (System Configuration) 1 shows a schematic diagram of the overall configuration of an imaging system according to an embodiment of the present invention. This imaging system 1 is a system for capturing images of transparent objects using a polarization camera 10, and is used, for example, to detect and inspect transparent objects on a factory production line.

[0022] The imaging system 1 mainly comprises a polarization camera 10, an illumination device 11, a processing device 12, and a stage 13. The polarization camera 10 is an imaging means having an imaging element with an array of polarizers. The illumination device 11 is a light source for irradiating illumination light L onto a subject (transparent object) W placed on the stage 13. The processing device 12 is a device that controls the entire imaging system 1 and performs information processing using images captured by the polarization camera 10. The stage 13 is a device for placing or holding the subject W.

[0023] The polarization camera 10 is positioned so as to capture an image of the subject W placed on the stage 13 from the zenith direction. In an XYZ coordinate system with the X and Y axes parallel to the stage 13 and the Z axis perpendicular to the stage 13, the optical axis of the polarization camera 10 is parallel to the Z axis. Since the objective of this system is to capture light specularly reflected by the subject W with the polarization camera 10, it is desirable for the angle of incidence of the illumination light L on the subject surface to be as close to the Brewster's angle as possible. Therefore, in this embodiment, the illumination device 11 is positioned at approximately the same height as the stage 13 (in practice, the lower end of the illumination device 11 is positioned at approximately the same height as the upper end of the stage 13 so that the illumination light L is not blocked by the stage 13), and the illumination light L is incident on the subject W from approximately directly beside it (perpendicular to the Z axis). This illumination arrangement is also referred to as low-angle illumination. With the combination of a zenith camera and low-angle illumination as in this embodiment, the polarization camera 10 captures the specular reflection R of the illumination light L, which is incident at an angle of approximately 45 degrees on the subject surface. In order to bring the angle of incidence closer to Brewster's angle, a lighting arrangement may be adopted in which illumination light L is applied from a position lower than subject W, or the optical axis of polarization camera 10 may be tilted to the opposite side from lighting device 11.

[0024] (polarized camera) An example of the configuration of the polarization camera 10 will be described with reference to Fig. 2. Fig. 2 is a diagram showing a schematic diagram of the structure of the polarization camera 10.

[0025] The polarization camera 10 has a structure in which an imaging element 20 is combined with a polarizer array 21. The imaging element 20 is a device in which photoelectric conversion elements (also called pixels), such as a CCD (Charge Coupled Device) sensor or a CMOS (Complementary Metal Oxide Semiconductor) sensor, are arranged two-dimensionally, and is also called an image sensor. The polarizer array 21 is a device in which a large number of polarizers are arranged two-dimensionally, and the position and size are designed so that one polarizer 210 corresponds to one pixel (light receiving element) 200 of the imaging element 20. The polarizer 210 is an optical element that has the property of passing only linearly polarized light in a specific direction (the vibration direction of the linearly polarized light transmitted by the polarizer 210 is called the transmission axis direction of the polarizer 210). As shown in FIG. 2 , the polarizer array 21 of this embodiment has four types of transmission axis directions (0 degree, 45 degrees, The polarizer 210 has a structure in which polarizers 210 with different transmission axis directions (0 degree, 45 degree, 90 degree, 135 degree) are regularly arranged. Specifically, the arrangement pattern is such that linearly polarized light components with different transmission axis directions (0 degree, 45 degree, 90 degree, 135 degree) are incident on four 2×2 pixels of the image sensor 20.

[0026] The resolution (pixel pitch) and size (number of pixels) of the image sensor 20 and polarizer array 21 can be appropriately designed depending on the subject and the intended use of the image. The polarizer array 21 can be realized using a wire grid, a photonic crystal, or any other method. While FIG. 2 shows four types of polarizers with different transmission axis directions combined into four 2x2 pixels, other variations in the transmission axis directions and arrangements of the polarizers may be used.

[0027] (Lighting equipment) An example of the configuration of illumination device 11 will be described with reference to Fig. 3A. Fig. 3A is a diagram schematically illustrating the structure of illumination device 11 of this embodiment, showing the state when stage 13 is viewed from the polarization camera 10 side.

[0028] The lighting device 11 is composed of four rod-shaped lights 30 arranged to surround the stage 13. When the four rod-shaped lights 30 are turned on simultaneously, the subject W on the stage 13 can be illuminated from all four directions (X-positive direction, X-negative direction, Y-positive direction, and Y-negative direction). Furthermore, the lighting direction can be switched by selectively turning on only one of the rod-shaped lights 30. The rod-shaped lights 30 are composed, for example, of multiple LED light sources arranged on a substrate and a diffusion plate arranged to cover them.

[0029] The structure of the illumination device 11 is not limited to that shown in FIG. 3A. For example, a circular illumination device 11 as shown in FIG. 3B may be used. This structure also allows the subject W to be illuminated simultaneously from all directions. The illumination direction can also be arbitrarily switched by selectively turning on the LED light sources. Instead of omnidirectional illumination, a configuration in which the subject W is illuminated only from a specific direction, as shown in FIG. 3C, may be used. For example, if the range in which the subject W exists is limited or the angle of the subject surface to be detected is known or limited, it is sufficient to illuminate from the required direction. Alternatively, even when the illumination device 11 as shown in FIG. 3C is used, omnidirectional imaging may be performed by changing the relative positions of the illumination device 11 and the subject W (for example, by rotating the subject W on a stage).

[0030] (Processing device) An example of the configuration of the processing device 12 will be described with reference to Fig. 4. Fig. 4 is a block diagram showing the logical configuration (functional configuration) of the processing device 12 of this embodiment.

[0031] The processing device 12 mainly comprises a camera control unit 40, an illumination control unit 41, and an image processing unit 42. The camera control unit 40 controls the polarization camera 10. For example, the camera control unit 40 controls the imaging conditions (exposure time, gain, etc.) of the polarization camera 10, captures image data from the polarization camera 10, and calibrates the polarization camera 10. The illumination control unit 41 controls the illumination device 11. For example, the illumination control unit 41 controls the illumination conditions (emission intensity, emission time, etc.) and controls the on / off of each light source. The image processing unit 42 processes the image captured from the polarization camera 10.

[0032] The image processing unit 42 of this embodiment has functions such as an image acquisition unit 420, a polarization extraction unit 421, a polarization information image generation unit 422, a display unit 423, and a parameter reception unit 424. The image acquisition unit 420 acquires an image captured by the polarization camera 10 (hereinafter also referred to as the "original image") via the camera control unit 40. The polarization extraction unit 421 performs polarization extraction processing to extract polarization information from the original image. Polarization information is information related to polarization resulting from specular reflection on a transparent object. The polarization information image generation unit 422 generates an image representing the polarization information based on the extraction results of the polarization extraction processing. The image processing unit 422 generates an image (hereinafter also referred to as a "polarization information image"). The display unit 423 performs processing to display the generated polarization information image on a display device. The parameter receiving unit 424 has a function to receive changes to various condition settings (parameters) that determine the operation of the imaging system 1. The details of each process of the image processing unit 42 will be described later.

[0033] The processing device 12 may be configured by a computer equipped with, for example, a processor such as a CPU or GPU, a memory as a main memory, a storage as an auxiliary memory, a display device, an input device such as a mouse or a touch panel, a network I / F, etc. The computer may be a general-purpose computer such as a personal computer, a tablet terminal, a smartphone, or a field computer, or may be an embedded computer or a dedicated device, or may be other devices such as the polarization camera 10 or a PLC (Programmable Logic Controller). The configuration shown in Fig. 4 may utilize the computer resources of a device. The configuration shown in Fig. 4 is realized by loading a program stored in a storage device or the like into memory and executing it with a processor. However, part or all of the configuration shown in Fig. 4 may be configured with dedicated devices such as an ASIC (Application Specific Integrated Circuit) or an FPGA (Field Programmable Gate Array). Also, part of the configuration or part of the processing shown in Fig. 4 may be executed by another device using cloud computing or distributed computing.

[0034] (Transparent object detection using a polarization camera) This section explains the basic principles of transparent object detection using a polarization camera.

[0035] As shown in Figure 5A, when light 50 is incident obliquely on a transparent object W, such as glass or transparent resin, most of the incident light 50 becomes refracted light 51 and passes through the transparent object W, but a portion of the incident light 50 is specularly reflected at the surface (interface) of the transparent object W and becomes reflected light 52. This specular reflection results in a difference in reflectance between the p-wave (polarized component whose electric field oscillation direction is parallel to the plane of incidence) and the s-wave (polarized component whose electric field oscillation direction is perpendicular to the plane of incidence (i.e., parallel to the reflecting surface)) contained in the incident light 50. Figure 5B shows an example of the difference in reflectance between the s-wave and p-wave, with the horizontal axis representing the angle of incidence and the vertical axis representing the reflectance. As shown in this example, the reflectance of the s-wave monotonically increases with the angle of incidence α, while the reflectance of the p-wave gradually decreases as the angle of incidence α increases from 0 degrees, reaching zero at a certain angle (called the Brewster angle). Therefore, the reflected light 52 observed in the case of specular reflection is dominated by the s-wave, i.e., the polarized component oscillating in a direction parallel to the surface of the transparent object W, which is the reflecting surface. By utilizing this property and capturing polarized light with a polarization camera, the surface (reflective surface) of the transparent object W can be imaged.

[0036] Incidentally, not only polarized light due to specular reflection but also light diffusely reflected from the subject and background enters a polarization camera. When imaging a transparent object, the diffusely reflected light component is unnecessary and can cause artifacts, so it is first necessary to extract only the polarized light component from the image captured by the polarization camera. To achieve this, this embodiment utilizes the difference in transmission characteristics of polarizers.

[0037] With reference to Figure 6, we will explain how to extract polarization components from images captured by a polarization camera. The intensity of the polarization components received by the light receiving element is maximum when the polarization direction θ coincides with the transmission axis direction of the polarizer. It decreases as the transmission axis direction deviates from the polarization direction θ and is minimum when the transmission axis direction and the polarization direction θ are perpendicular. In other words, as the transmission axis direction of the polarizer is changed from 0 to 180 degrees, the intensity of the polarization components received by the light receiving element changes like a sine wave. On the other hand, the intensity of the diffusely reflected light component contained in the incident light is constant, regardless of the transmission axis direction of the polarizer. Therefore, it is possible to extract only information about the polarization components contained in the incident light based on the change in light intensity when observed through multiple types of polarizers with different transmission axis directions (transmission axis direction dependence).

[0038] For example, as shown in Figure 6, there are four types of transmission axis angles: 0 degrees, 45 degrees, 90 degrees, and 135 degrees. Assume that incident light containing a polarization component with a polarization direction θ is observed through a polarizer 210. In this case, the values of the pixels 200 corresponding to each polarizer 210 (corresponding to the light intensity of the transmitted light) differ depending on the direction of the transmission axis. By fitting a sine wave to these four pixel values, the polarization direction θ and polarization intensity of the polarization component can be estimated. In Figure 6, the angle at which the fitting curve is maximized represents the polarization direction θ of the polarization component, and the amplitude of the fitting curve represents the polarization intensity (also called the degree of polarization). By applying this polarization extraction process to each pixel of the image acquired by the polarization camera 10, information (such as the polarization direction and polarization intensity) of the polarized light specularly reflected by a transparent object W present within the field of view of the polarization camera 10 can be captured.

[0039] The polarization extraction process by the polarization extraction unit 421 of this embodiment also follows the same basic principle of extracting polarization information based on the change in light intensity (transmission axis direction dependency) when observed through multiple types of polarizers with different transmission axis directions. However, to further improve accuracy and usability, unique processing such as selection of mask size (a parameter that defines the range of neighboring pixels to be referenced when calculating polarization) and noise removal has been added. This will be described in more detail later.

[0040] (Example of imaging system operation) An example of the operation of the imaging system 1 will be described with reference to the flowchart of FIG.

[0041] In step S70, a transparent object to be the subject W is placed on the stage 13. The subject W may be transported and positioned using a robot or a transport device, or the subject W may be set on the stage 13 by an operator.

[0042] In step S71, the user uses a user interface provided by the parameter receiving unit 424 of the processing device 12 to input the imaging conditions of the polarization camera 10 and the lighting conditions of the lighting device 11. Note that if there is no need to change the imaging conditions or lighting conditions, the processing of step S71 may be skipped.

[0043] In step S72, an image of the subject W is captured. Specifically, the illumination control unit 41 of the processing device 12 turns on the illumination device 11 in accordance with the given illumination conditions, and irradiates the subject W with illumination light. Then, with the subject W illuminated, the camera control unit 40 controls the polarization camera 10 in accordance with the given imaging conditions to capture the image.

[0044] In step S73, the image acquisition unit 420 of the processing device 12 captures the image (original image) captured in step S72 from the polarization camera 10. The captured original image data is stored in memory or storage and is subjected to processing by the image processing unit 42.

[0045] In step S74, the user uses a user interface provided by parameter receiving unit 424 to input image processing parameters to be used in image processing unit 42. Note that if there is no need to change the image processing parameters, the processing of step S74 may be skipped.

[0046] 8 is an example of a user interface (UI) for inputting image processing parameters provided by the parameter receiving unit 424. This UI screen is composed of a mask size setting 80, a noise reduction setting 81, a display image setting 82, a whole image display area 83, an enlarged image display area 84, and an enlargement setting 85. The mask size setting 80 is a UI for setting the "mask size" that defines the range of neighboring pixels to be referenced when calculating polarization information from the original image. In the example of FIG. 8, the mask size can be selected from 2x2, 3x3, 4x4, and 5x5, but the user may also be allowed to input any size. Noise reduction Settings 81 is a UI for setting conditions for "noise reduction processing" that removes or reduces noise contained in the calculation results of polarization information. In the example of Fig. 8, it is possible to select whether or not to apply noise reduction processing, and if "yes", it is also possible to input parameters to be used in the noise reduction processing (comparison range, polarization angle difference threshold, polarization intensity difference threshold).

[0047] The display image settings 82 are a UI for allowing the user to specify the type of image to be displayed in the entire image display area 83 and the enlarged image display area 84. In the example of FIG. 8, four types of images can be selected: the original image, the polarization intensity image, the polarization direction image, and the brightness image. The entire image display area 83 is an area where the entire image (low resolution) is displayed, and the enlarged image display area 84 is an area where a portion of the image (the portion indicated by the frame 86) is displayed at high resolution. The enlargement settings 85 are a UI for setting the magnification and range of the enlarged image to be displayed in the enlarged image display area 84.

[0048] In step S75, the polarization extraction unit 421 calculates polarization information from the original image captured in step S73. Details of the polarization information calculation process will be described later.

[0049] In step S76, the polarization extraction unit 421 applies noise removal processing to the calculation result of step S75. Note that if noise removal is set to "no" in the UI of Fig. 8, the processing of step S76 is skipped. Details of the noise removal processing will be described later.

[0050] In step S77, the polarization information image generation unit 422 generates a polarization information image based on the calculation result of the polarization extraction unit 421. In this embodiment, three types of polarization information images are generated: a polarization intensity image, a polarization direction image, and a brightness image. The polarization intensity image is an image that represents the polarization intensity (degree of polarization) using shades of gray, and the polarization direction image is an image that represents the polarization direction using pseudo-colors. The brightness image is an image that represents the light intensity of the polarized component using shades of gray.

[0051] In step S78, the display unit 423 displays the image selected in the display image setting 82 of the UI of FIG. 8 in the whole image display area 83 and the enlarged image display area 84.

[0052] Here, when the user changes the display image setting 82 on the UI of Fig. 8, the images displayed in the whole image display area 83 and the enlarged image display area 84 change accordingly. Using this UI allows the user to visually compare the appearance of the original image with the polarization information (polarization intensity, polarization direction, brightness), facilitating tasks such as confirming whether the calculation of polarization information and noise removal have been performed appropriately, and checking the shape and state of transparent objects. Note that if the mask size setting 80 or noise removal setting 81 is changed on the UI of Fig. 8, the processes of steps S75 to S78 may be executed again. Using such a UI allows the user to fine-tune image processing parameters (mask size, whether or not to perform noise removal, comparison range, threshold, etc.) while checking the calculation results (polarization information image).

[0053] (Polarization information calculation process) 9 and 10A to 10D, an implementation example of the polarization information calculation process (step S75 in FIG. 7) of the polarization extraction unit 421 will be described. FIG. 9 is a flowchart of the polarization information calculation process, and FIGS. 10A to 10D are diagrams showing examples of neighboring pixels. Here, the size (number of pixels) of the original image is assumed to be M rows x N columns, the pixel in the i-th row and j-th column is represented as (i, j), and the pixel value (light intensity) of that pixel is represented as I(i, j) (i = 1, 2,..., M, j = 1, 2,..., N). FIGS. 10A to 10D show enlarged portions of the original image, with each rectangle representing a pixel, the arrow within the pixel representing the transmission axis direction, and the symbols in the margins representing the row and column numbers of the pixel.

[0054] In step S10, the polarization extraction unit 421 selects a pixel of interest to be calculated from the original image. In step S10, the target pixels are selected in order from (1,1) to (M,N).

[0055] In step S11, the polarization extraction unit 421 selects neighboring pixels to be referenced when calculating polarization information for the pixel of interest. The range of neighboring pixels selected here is determined by the mask size specified in the mask size setting 80 in FIG. 8. First, an example of a 2×2 mask size, which is the default, will be described. When the mask size is 2×2, as shown in FIG. 10A, three pixels (i-1, j-1), (i-1, j), and (i, j-1) adjacent to the pixel of interest (i, j) are selected as neighboring pixels. Note that if the pixel of interest is on the boundary of the original image and no neighboring pixels exist, steps S11 and S12 may be skipped, or an appropriate value may be added for padding.

[0056] In step S12, the polarization extraction unit 421 calculates polarization information for the pixel of interest based on the light intensities (pixel values) of the pixel of interest and its neighboring pixels. If the light intensities when the transmission axis directions are 0 degrees, 45 degrees, 90 degrees, and 135 degrees are I1, I2, I3, and I4, respectively, the brightness I0, polarization direction θ, and polarization intensity D of the pixel of interest can be calculated using the following formulas.

number

[0057] In the example of FIG. 10A, the transmission axis directions corresponding to the four pixels (i-1, j-1), (i-1, j), (i, j-1), and (i, j) are 90 degrees, 135 degrees, 45 degrees, and 0 degrees, respectively, and therefore I1 to I4 in equations (1) to (3) are given by the following equations. I1=I(i,j) I2=I(i,j-1) I3=I(i-1,j-1) I4=I(i-1,j)

[0058] The brightness I0 in equation (1) is an index that represents the intensity of the incident light that enters the pixel of interest. The polarization direction θ in equation (2) represents the principal axis direction of the polarization component contained in the incident light that enters the pixel of interest. The polarization intensity D in equation (3) is an index that represents the degree of polarization of the polarization component contained in the incident light that enters the pixel of interest.

[0059] The above formulas (1) to (3) are merely examples, and the indices corresponding to the brightness, polarization direction, and polarization intensity may be calculated using other formulas. For example, the coefficients in formulas (1) to (3) may be replaced with other coefficients, or formula (3) may be replaced with the following formula (4):

number

[0060] In step S13, the polarization extraction unit 421 checks whether the calculation process for polarization information has been completed for all pixels of the original image (i.e., whether i=M and j=N), and if there are any unprocessed pixels, If so, the process returns to step S10 and the process for the next pixel of interest is executed.

[0061] Through the above-described process, three items of polarization information (polarization direction θ, polarization intensity D, and brightness I0) are calculated for all pixels in the original image. The results of the polarization information calculation process are saved in memory or storage in a data format that associates the coordinates (i,j) of each pixel with the polarization direction θ(i,j), polarization intensity D(i,j), and brightness I0(i,j), and are then used for subsequent processing.

[0062] Next, a processing example when the mask size is larger than 2×2 will be described with reference to FIGS. 10B to 10D.

[0063] As shown in FIG. 10B, when the mask size is 3×3, in step S11, eight pixels (i-1, j-1), (i-1, j), (i-1, j+1), (i, j-1), (i, j+1), (i, j-1), (i, j+1), (i+1, j-1), (i+1, j), and (i+1, j+1) adjacent to the pixel of interest (i, j) are selected as neighboring pixels. In the example of FIG. 10B, the nine pixel groups including the pixel of interest and neighboring pixels include one pixel whose transmission axis direction is 0 degrees, two pixels whose transmission axis direction is 45 degrees, two pixels whose transmission axis direction is 135 degrees, and four pixels whose transmission axis direction is 90 degrees. When there are multiple pixels with the same transmission axis direction, in the calculation of step S12, representative values of the pixels for each transmission axis direction are substituted for I1 to I4 in equations (1) to (3).

[0064] For example, when the average value is used as the representative value, I1 to I4 are given by the following formulas. I1=I(i,j) I2={I(i,j-1)+I(i,j+1)} / 2 I3={I(i-1,j-1)+I(i-1,j+1)+I(i+1,j-1)+I(i+1,j+1)} / 4 I4={I(i,j-1)+I(i,j+1)} / 2

[0065] The representative value may be the average value, the mode value, the median value, the maximum value, the minimum value, etc. Alternatively, without using the representative value, I1 to I4 may be estimated by fitting a sine wave to the values of the nine pixels, or the brightness I0, the polarization direction θ, and the polarization intensity D may be calculated directly.

[0066] FIG. 10C shows an example where the mask size is 4×4. The selected range includes the eight neighboring pixels from the 3×3 mask size plus seven additional pixels: (i-2, j-2), (i-2, j-1), (i-2, j), (i-2, j+1), (i-1, j-2), (i, j-2), and (i+1, j-2). In the example of FIG. 10C, the selected range includes four pixels at angles of 0°, 45°, 90°, and 135°. In this case, as in the 3×3 mask size, either substitute representative values for I1 to I4 for each transmission axis direction, or perform sinusoidal wave fitting on the values of the 16 pixels.

[0067] FIG. 10D shows an example with a 5×5 mask size. The selected range includes the 15 neighboring pixels from a 4×4 mask, plus an additional nine pixels: (i-2, j+2), (i-1, j+2), (i, j+2), (i+1, j+2), (i+2, j-2), (i+2, j-1), (i+2, j), (i+2, j+1), and (i+2, j+2). In the example of FIG. 10D, the selected range includes nine pixels at 0 degrees, six pixels at 45 degrees, six pixels at 135 degrees, and four pixels at 90 degrees. In this case, as with the 3×3 mask, we can substitute representative values for I1 to I4 for each transmission axis direction, or perform sinusoidal wave fitting on the values of the 25 pixels.

[0068] In this embodiment, the mask size, i.e., the pixel range to be referenced when calculating polarization information, can be changed as desired. The user can select an appropriate mask size depending on the shape and condition of the subject, the purpose of the system, etc.

[0069] For example, as shown in FIG. 11A, when the polarization camera 10 captures specular reflection from a curved surface of the subject W, the reflecting surface is extremely small, so the area where the polarized light is incident may be a narrow area of only a few pixels wide (depending on the resolution of the polarization camera 10). In such a case, if the mask size is too large, pixels where no polarized light is incident may also be referenced, as shown in FIG. 11B, which may reduce the detectability of polarization information. Therefore, it is preferable to select a mask size smaller than the area where the polarized light is incident. This allows for appropriate extraction of polarization information from the narrow area.

[0070] On the other hand, as shown in Figure 12, when inspecting for surface contamination or foreign matter, it is expected that similar polarization characteristics will appear over a relatively wide area of the image due to the reflectance characteristics (specular reflection, scattering strength) of a flat surface. In such cases, if the mask size is too small, the inspection may be sensitive to local differences in reflectance, resulting in unstable inspection. Therefore, it is preferable to set a somewhat larger mask size to average out disturbing factors such as local differences in reflectance. This can improve the stability of the inspection.

[0071] In this embodiment, four mask sizes, 2x2, 3x3, 4x4, and 5x5, are shown as examples of mask sizes, but the mask size is not limited to these. For example, a mask larger than 5x5 may be used, or a mask with a shape other than a square may be used. Furthermore, the user may be allowed to design the size and shape of the mask as desired.

[0072] (Noise removal processing) The polarization information calculation process described above utilizes the principle of detecting polarization by regarding local pixel value differences as differences in transmission due to differences in the direction of the polarizer's transmission axis. Therefore, for example, when a high-spatial-frequency brightness change occurs in an edge or texture area of a subject, this may be mistaken for a pixel value difference due to polarization, resulting in a false detection of polarization. The noise removal process is a process for removing or reducing noise due to such false detection from the calculation results of the polarization information calculation process.

[0073] An implementation example of the noise removal process (step S76 in FIG. 7) will be described with reference to Fig. 13, Fig. 14A, and Fig. 14B. Fig. 13 is a flowchart of the noise removal process, and Fig. 14A and Fig. 14B are diagrams showing examples of comparison ranges.

[0074] In step S20, the polarization extraction unit 421 reads the parameters set in the noise removal setting 81 in Fig. 8. Here, four parameters are read: whether or not noise removal is enabled, the comparison range, the polarization angle difference threshold, and the polarization intensity difference threshold. If noise removal is set to "enabled," the subsequent processing is skipped (step S21). If noise removal is set to "enabled," the processing proceeds to step S22.

[0075] In step S22, the polarization extraction unit 421 reads the calculation results of the polarization information calculation process from memory or storage. Here, data is acquired that associates the coordinates (i, j) of each pixel with the polarization direction θ(i, j), polarization intensity D(i, j), and brightness I0(i, j).

[0076] In step S23, the polarization extraction unit 421 selects a target pixel to be processed. In step S23, the target pixels are selected in order from (1,1) to (M,N).

[0077] In step S24, the polarization extraction unit 421 compares the polarization directions of the target pixel and its surrounding pixels to determine whether the polarization directions are similar. As shown in FIG. 14A, when the comparison range is set to "1," a comparison is made between the target pixel (i, j) and the three surrounding pixels (i, j+1), (i+1, j), and (i+1, j+1). Also, as shown in FIG. 14B, when the comparison range is set to "2," a comparison is made between the target pixel (i, j) and the eight surrounding pixels (i-1, j-1), (i-1, j), (i-1, j+1), (i, j-1), (i, j+1), (i+1, j-1), (i+1, j), and (i+1, j+1). If any combination is found in which the difference in polarization direction is greater than the polarization angle difference threshold, the polarization extraction unit 421 determines that the polarization directions are "dissimilar," and proceeds to step S26. On the other hand, if it is determined to be "similar," the process proceeds to step S25.

[0078] In step S25, the polarization extraction unit 421 compares the polarization intensities of the target pixel and its surrounding pixels to determine whether the polarization intensities are similar. The method for selecting the comparison range is as shown in Figures 14A and 14B. If even one combination is found in which the difference in polarization intensities is greater than the polarization intensity difference threshold, it is determined to be "dissimilar" and the process proceeds to step S26. On the other hand, if it is determined to be "similar," the process proceeds to step S27.

[0079] If either the polarization direction or the polarization intensity is "dissimilar," the polarization information of the target pixel is an outlier compared to the surrounding pixels, and is likely to be noise. Therefore, in step S26, the polarization extraction unit 421 deletes the polarization information of the target pixel (polarization direction θ and polarization intensity D).

[0080] If both the polarization direction and the polarization intensity are "similar," in step S27, the polarization extraction unit 421 selects the pixel with the greatest polarization intensity among the target pixel and the pixels in the comparison range, and replaces the polarization direction and polarization intensity of the target pixel with the polarization direction and polarization intensity of the selected pixel. This process achieves a smoothing effect that reduces local variations in the polarization direction and polarization intensity.

[0081] In step S28, the polarization extraction unit 421 checks whether the noise removal process has been completed for all pixels, and if there are any unprocessed pixels, the process returns to step S23 to process the next target pixel.

[0082] The above-described process removes noise (outliers) and smooths the polarization direction θ and polarization intensity D, which are the calculation results of the polarization information calculation process. The corrected data is then overwritten and saved in memory or storage.

[0083] (Application example) An example will be described in which the above-described imaging system 1 is applied to detection and inspection in the manufacturing process of syringes filled with liquid medicine.

[0084] In the manufacturing process of pre-filled syringes, robots are usually used to pick and place the syringes and inject the liquid. If the syringe is in an incorrect position or lying on its side, it could be damaged or the manufacturing equipment could malfunction. Therefore, a mechanism is needed to monitor the syringe's position and orientation, and output an error or stop the equipment if a problem occurs.

[0085] FIG. 15A shows a state in which syringes 101 placed in nest 100 are observed by zenith camera 102. Two syringes 101b are not inserted into the holes in nest 100 and are lying on their side. However, because syringes 101 are colorless and transparent, even if they are photographed using normal transmitted illumination and an optical camera, as shown in FIG. 15B, the syringes 101 and the background (white syringes) are not clearly distinguishable. The boundary of the syringe 101b is barely discernible on the image, making it difficult to stably detect the syringe 101b lying on its side.

[0086] In contrast, when polarization information is extracted using imaging system 1 according to an embodiment of the present invention, it is possible to image only the specular reflection on the surface of syringe 101, as shown in Fig. 15C. In the polarization information image, a circular image appears for syringe 101a in the correct position, and a linear band-like image appears for syringe 101b lying on its side. By using such polarization information images, it becomes possible to stably inspect the position and orientation of syringes and detect syringes in an incorrect position or orientation.

[0087] (Image display example) FIG. 16 shows an example of a display of a polarization direction image on the UI screen of FIG.

[0088] The display image setting 82 on this UI screen is a user interface for specifying the image to be displayed from among four images: the original image and three types of polarization information images: a polarization intensity image, a polarization direction image, and a brightness image. The UI for display image setting 82 is configured using so-called radio buttons, allowing the image to be displayed to be selected alternatively.

[0089] When the user selects "Polarization Direction Image" in the display image settings 82, the images displayed in the entire image display area 83 and the enlarged image display area 84 are switched to the polarization direction image accordingly. At this time, the polarization direction image is displayed using pseudo colors. The pseudo colors represent the polarization direction. For example, the polarization direction range from 0 degrees to 180 degrees is represented by a change in hue from blue to cyan to blue-green to green to yellow to orange to red to magenta. In this embodiment, as shown in FIG. 16 , an auxiliary image 90 indicating the correspondence between the pseudo colors and the polarization direction is displayed in the entire image display area 83 and the enlarged image display area 84. For convenience of illustration, FIG. 16 shows the change in hue of the pseudo colors as six different patterns. However, on an actual display screen, it is preferable to set the pseudo colors so that the hue changes continuously according to the angle of the polarization direction.

[0090] By viewing such a display, the user can visually recognize differences in the orientation of the surface (reflective surface) of the transparent object as differences in color. For example, in the enlarged image display area 84 of Fig. 16, a roughly V-shaped image is displayed, and since the left straight line portion 91 and the right straight line portion 93 have different colors, it can be seen that the orientation of the reflective surface corresponding to the left straight line portion 91 and the right straight line portion 93 is different. Furthermore, since the color gradually changes at the bent portion 92 surrounded by the dashed line, it can be seen that the orientation of the reflective surface is gradually changing.

[0091] Furthermore, the polarization direction of each reflecting surface (i.e., the orientation in the XY plane) can be quantitatively grasped by referring to auxiliary image 90. In the example of Figure 16, it can be seen that the left straight section 91 is approximately 135 degrees, the right straight section 93 is approximately 45 degrees, and the polarization direction gradually changes from approximately 135 degrees to approximately 45 degrees at bent section 92 in between.

[0092] Furthermore, polarization information such as polarization intensity, polarization direction, and brightness may be displayed numerically. FIG. 17 shows an example of a UI that displays polarization information numerically. In this UI, when a mouse cursor 88 is placed over an image displayed in an enlarged image display area 84 (or when the image is clicked with the mouse cursor 88), information such as the polarization intensity, polarization direction, and brightness at the pixel position of the mouse cursor 88 is displayed in an information display area 87. Furthermore, when a polarization direction image is displayed, a GUI component 89 such as a bar indicating the polarization direction value at the pixel position of the mouse cursor 88 may be superimposed on an auxiliary image 90. Providing such an information display function makes it easy to check the polarization information of a specific location within an image in detail.

[0093] Providing display screens such as those shown in Figures 16 and 17 can help users understand the shape, structure, or state of transparent objects from polarization information. For example, in the polarization direction image corresponding to the state in Figure 15A, a syringe 101a in the correct position appears as a circular image whose color changes circumferentially, while a syringe 101b lying on its side appears as a monochromatic band-like image. Moreover, the band-like image exhibits different colors depending on the orientation of syringe 101b. Therefore, by using such polarization direction images, it becomes possible to easily visually recognize and detect a syringe 101b lying on its side.

[0094] Computer-based image recognition technology may also be used to detect syringes from polarization information images and determine their orientation. For example, a syringe in the correct orientation will exhibit specular reflections from the inner circle and the flange surrounding it. Focusing on this image feature and detecting the shape of the circle and flange from the polarization information image using a method such as pattern matching enables automatic recognition of syringes in the correct orientation. In the case of a syringe lying on its side, for example, first, a set of adjacent pixels (pixel region) with a specific polarization direction range (e.g., 0° to 60°) is extracted, and then pixel regions with widths and lengths greater than or equal to a threshold are detected from the extracted pixel region. This type of processing makes it possible to automatically recognize the strip-shaped pixel region of a syringe lying on its side.

[0095] Furthermore, it is possible to detect noise and artifacts in the image using the color distribution and discontinuities in the polarization direction image as clues. An example is shown in FIG. 18A. In the polarization direction image in FIG. 18A, a long, narrow image 94 appears near the V-shaped image. However, the color of image 94 indicates that the polarization direction is approximately 180 degrees. Given the structure of the subject, it is unlikely that a reflective surface with a polarization direction of approximately 180 degrees would be located in close proximity to a reflective surface with a polarization direction of approximately 45 degrees. Therefore, it can be assumed that image 94 is noise or an artifact (e.g., due to diffuse reflection at the edge of the subject). If such noise or artifacts are discovered, it is recommended to reduce the noise or artifacts by adjusting the mask size and noise reduction parameters, capturing an image with a polarization camera, extracting polarization information, and rechecking the image. Using the display screen and UI of this embodiment is expected to facilitate checking whether polarization information is correctly extracted and evaluating the quality of polarization information images.

[0096] 18B and 18C show other examples of auxiliary images. These auxiliary images are annular images whose colors change in the circumferential direction, like a hue wheel. Pseudo colors are associated with the azimuth angles on the annulus, representing the polarization direction. Using such an auxiliary image allows users to more intuitively grasp the correspondence between pseudo colors and polarization direction.

[0097] The method of displaying the polarization direction image is not limited to the above example. For example, the correspondence between the polarization direction (angle) and hue may be designed arbitrarily. The hue may be changed continuously or stepwise depending on the angle (for example, by switching the hue every 15 degrees). Furthermore, not only the hue but also the saturation or brightness may be changed. For example, by displaying an image in which the hue is changed depending on the polarization direction (angle) and the saturation or brightness is changed depending on the polarization intensity, it is expected that the polarization direction and polarization intensity can be intuitively grasped.

[0098] FIG. 19 shows another example of a UI screen for specifying an image to be displayed and for checking the image. On this UI screen, two images to be displayed can be selected using pull-down menus 95a and 95b. The image specified using the pull-down menu 95a is displayed in a first display area 96a, and the image specified using the pull-down menu 95b is displayed in a second display area 96b. Displaying multiple images side-by-side on the screen makes it easy for the user to visually compare the images (in the example of FIG. 19, the original image and the polarization direction image), making it easier to check whether polarization information has been correctly extracted and to understand the shape and structure of the subject. 19 shows an example in which two images are displayed side by side, but three or more images may be displayed side by side. Also, instead of displaying the images to be compared side by side, the images to be compared may be displayed overlapping each other, or may be displayed by automatically switching between them like a slide show.

[0099] (others) The above-described embodiment merely exemplifies the configuration of the present invention, and the present invention is not limited to the specific embodiment described above, and various modifications are possible within the scope of the technical concept thereof.

[0100] For example, the arrangement and configuration of the polarization camera and lighting device are not limited to those shown in Figure 1 and Figures 3A to 3C, and may be designed appropriately depending on the subject and application. In other words, it is sufficient that the specular reflection of the illumination light can be observed by the polarization camera. Furthermore, multiple polarization cameras may be provided, or movable polarization cameras and lighting devices may be used.

[0101] In the above embodiment, polarizers with 0-degree, 45-degree, 90-degree, and 135-degree transmission axis directions are used, but the setting of the transmission axis direction is not limited to this. It is sufficient that the imaging element can observe light that has passed through polarizers with different transmission axis directions and can extract polarization information contained in the incident light from the observation results.

[0102] The UI screen in FIG. 8 allows the user to set the mask size used in the polarization information calculation process and the parameters used in the noise removal process, but the configuration of the UI screen and the parameter items are merely examples. Any UI that allows the user to change the parameters used in the polarization extraction process may be used. In other words, it is sufficient if the user can change the polarization extraction logic as appropriate depending on the subject, application, etc. In addition, in the above embodiment, the parameter receiving unit 424 provides a user interface for setting parameters, but the parameter receiving unit 424 may also receive a parameter change instruction from an external device via a network from the user.

[0103] <Appendix 1> An imaging system (1) for imaging a transparent object (W), comprising: an illumination device (11) for illuminating a transparent object (W); a polarization camera (10) having an image pickup element (20) in which polarizers (210) with different transmission axis directions are regularly arranged; a processing device (12), The processing device (12) an image acquisition means (420) for acquiring an original image captured by the polarization camera (10) while the transparent object (W) is illuminated by the illumination device (11); a polarization extraction means (421) for performing a polarization extraction process to extract polarization information, which is information about polarization resulting from specular reflection on the transparent object (W), for each pixel of the original image; an image generating means (422) for generating a polarization information image by visualizing the polarization information based on the extraction result of the polarization extraction process; and a display means (423) for displaying the polarization information image on a screen, the polarization information includes information on a polarization direction, the polarization information image includes a polarization direction image that represents the polarization direction, which is one of the polarization information, in pseudo-color; The display means (423) displays, on the screen, the polarization direction images (91, 92, 93) and an auxiliary image (90) showing the correspondence between the pseudo color and the polarization direction. An imaging system (1).

[0104] <Appendix 2> A control method for an imaging system (1) including an illumination device (11), a polarization camera (10) having an imaging element (20) in which polarizers (210) having different transmission axis directions are regularly arranged, and a processing device (12), comprising: Illuminating a transparent object (W) with the illumination device (11); Taking an image using the polarization camera (10); A step (S73) of importing the original image captured by the polarization camera (10) into the processing device (12); the processing device (12) executes a polarization extraction process (S75, S76) for extracting polarization information, which is information relating to polarization resulting from specular reflection on the transparent object (W), for each pixel of the original image, and generates a polarization information image (S77) that visualizes the polarization information based on the extraction results of the polarization extraction process (S75, S76); and displays the polarization information image on a screen (S78), the polarization information includes information on a polarization direction, the polarization information image includes a polarization direction image that represents the polarization direction, which is one of the polarization information, in pseudo-color; A control method characterized in that an auxiliary image (90) showing the correspondence between pseudocolor and polarization direction is displayed on the screen together with the polarization direction image (91, 92, 93). [Explanation of symbols]

[0105] 1: Imaging system 10: Polarized camera 11: Lighting equipment 12: Processing device 13: Stage 20: Image sensor 21: Polarizer array 30: Rod lighting

Claims

1. 1. An imaging system for imaging a transparent object, comprising: an illumination device for illuminating a transparent object; a polarization camera having an image sensor in which polarizers with different transmission axis directions are regularly arranged; a processing device, The processing device includes: an image acquisition means for acquiring an original image captured by the polarization camera while the transparent object is illuminated by the illumination device; a polarization extraction means for executing a polarization extraction process to extract polarization information, which is information about polarization resulting from specular reflection on the transparent object, for each pixel of the original image; an image generating means for generating a polarization information image by visualizing the polarization information based on the extraction result of the polarization extraction process; a display means for displaying the polarization information image on a screen, the polarization information includes information on a polarization direction, the polarization information image includes a polarization direction image that represents the polarization direction, which is one of the polarization information, in pseudo-color; The display means displays, on the screen, an auxiliary image indicating the correspondence between pseudo colors and polarization directions together with the polarization direction image. An imaging system characterized by:

2. the processing device further comprises an input means for providing a user interface for allowing a user to specify an image to be displayed from a plurality of images including the original image and the polarization information image; The display means displays an image designated by the user interface on the screen.

2. The imaging system according to claim 1, wherein:

3. The polarization information includes a plurality of items of polarization information, The polarization information image includes a plurality of types of polarization information images corresponding to the plurality of items of polarization information, respectively.

3. The imaging system according to claim 2.

4. The multiple items of polarization information include polarization direction, polarization intensity, and brightness.

4. The imaging system according to claim 3.

5. The polarization direction image is an image in which the hue of the pixel is changed according to the polarization direction, and the saturation or brightness of the pixel is changed according to the polarization strength.

5. The imaging system according to claim 4.

6. the user interface allows a user to alternatively select an image to be displayed from among the plurality of images; The display means switches the image displayed on the screen in response to a user selection.

6. The imaging system according to claim 2, wherein the imaging system comprises: a first lens;

7. the user interface allows a user to select two or more images to be displayed from the plurality of images; The display means displays the two or more images selected by the user on the screen in a manner that allows comparison.

6. The imaging system according to claim 2, wherein the imaging system comprises: a first lens;

8. The polarization extraction process includes a noise removal process for removing or reducing noise contained in the calculation result after calculating the polarization information of each pixel of the original image.

8. The imaging system according to claim 1, wherein the imaging system comprises: a first lens;

9. The noise removal process includes a process of determining whether the polarization information of the target pixel and the surrounding pixels is similar or dissimilar by comparing the polarization information of the target pixel and the surrounding pixels, and deleting the polarization information of the target pixel if it is determined that the polarization information is dissimilar.

9. The imaging system according to claim 8.

10. The noise removal process includes a process of comparing polarization information between a target pixel and its surrounding pixels to determine whether the polarization information of the target pixel and the surrounding pixels is similar or dissimilar, and if it is determined that the polarization information of the target pixel is similar, replacing the polarization information of the target pixel with polarization information of a pixel selected from the target pixel and the surrounding pixels.

10. The imaging system according to claim 8, wherein the imaging system comprises:

11. A control method for an imaging system including an illumination device, a polarization camera having an imaging element in which polarizers with different transmission axis directions are regularly arranged, and a processing device, comprising: illuminating a transparent object with the illumination device; taking an image using the polarization camera; A step of inputting an original image captured by the polarization camera into the processing device; the processing device executes a polarization extraction process to extract polarization information, which is information relating to polarization resulting from specular reflection on the transparent object, for each pixel of the original image, and generates a polarization information image that visualizes the polarization information based on the extraction result of the polarization extraction process; and displays the polarization information image on a screen, the polarization information includes information on a polarization direction, the polarization information image includes a polarization direction image that represents the polarization direction, which is one of the polarization information, in pseudo-color; An auxiliary image showing the correspondence between the pseudo color and the polarization direction is displayed on the screen together with the polarization direction image. A control method comprising:

12. A program for causing a processor to execute each step of the control method according to claim 11.

Images