Information processing apparatus, information processing method, and recording medium

Abstract

A polarization image capturing unit acquires time-series polarization images showing a target object. A polarization information calculating unit calculates polarization information for each polarization image from the time-series polarization images acquired by the polarization image capturing unit. A target object recognizing unit recognizes a target object image region based on a non-polarization image generated by a non-polarization image generating unit based on the polarization images. A state change detecting unit detects a change in a state of the target object based on a time-series change in the polarization information in the target object image region recognized by the target object recognizing unit. An output unit outputs a state change detection result obtained by the state change detecting unit in association with the non-polarization image generated by the non-polarization image generating unit. This makes it possible to easily detect a change in a state of the target object.

Description

TECHNICAL FIELD

[0001] The present technology relates to an information processing apparatus, an information processing method, and a program, and detecting a state change of an object. BACKGROUND

[0002] Conventionally, a state change of an object is determined based on temperature. For example, in Patent Literature 1, an atmospheric temperature and a temperature of a laundry as an object are compared with each other, and whether the laundry has been dried is determined as a state change of the object based on a comparison result.

[0003] LIST OF CITATIONS

[0004] PATENT LITERATURE

[0005] Patent Literature 1

[0006] JP H06-347428 A SUMMARY

[0007] TECHNICAL PROBLEM

[0008] If a state change of an object is determined based on temperature, a state change cannot be determined in a portion where temperature is not measured. Therefore, in order to determine a state change of an object with high resolution, an expensive high-resolution thermometer needs to be used, resulting in difficulty in easily determining a state change of an object with high resolution.

[0009] Therefore, an object of the present technology is to provide an information processing apparatus, an information processing method, and a program capable of easily detecting a state change of an object.

[0010] SOLUTION TO PROBLEM

[0011] A first aspect of the present technology is

[0012] An information processing apparatus including: a polarization information calculation unit that calculates polarization information of each polarization image from time-series polarization images of an object; and

[0013] a state change detection unit that detects a state change of the object based on a time-series change in the polarization information calculated by the polarization information calculation unit.

[0014] In the technology, a polarization information calculating unit calculates one or different polarization information for each polarization image from time-series polarization images of an object. A state change detecting unit calculates a state change index indicating a change in the time series of the polarization information calculated by the polarization information calculating unit for each pixel position, each polarization information, each object region, or each polarization image, and detects a state change of the object based on the state change index. For example, the state change detecting unit detects that a change in the polarization information becomes smaller than a predetermined change or a change in the polarization information becomes larger than a predetermined change over time. Further, the state change detecting unit merges the state change indices for each pixel in an image region of the object or each object, and detects a state change of the object based on the merged state change indices. In the merging of the state change indices, the state change detecting unit assigns a weight to the state change indices for each pixel in an image region of the object or for each image region of the object, and merges the state change indices to which the weights are assigned. The state change detecting unit increases the weight of the polarization information having high reliability.

[0015] An object specifying unit specifies an image region of an object in a polarization image. The object specifying unit performs recognition by using, for example, a non-polarization image generated by the non-polarization image generating unit, and specifies an image region of an object.

[0016] An information processing apparatus includes an output unit that outputs a detection result of a state change of an object, the detection result being obtained by a state change detecting unit. For example, the output unit associates the state change detection result obtained by the state change detecting unit with a non-polarization image generated by a non-polarization image generating unit, performs image processing on the non-polarization image in accordance with the detection result of the state change, and outputs an image.

[0017] A second aspect of the present technology is

[0018] An information processing method includes: calculating, by a polarization information calculating unit, polarization information for each polarization image from time-series polarization images of an object; and

[0019] detecting, by a state change detecting unit, a state change of the object based on a change in the time series of the polarization information calculated by the polarization information calculating unit.

[0020] A third aspect of the present technology is

[0021] A program that causes a computer to detect a state change of an object, the program causing the computer to perform the following operations:

[0022] calculating polarization information for each polarization image from time-series polarization images of an object; and

[0023] detect a state change of the object based on the time series change of the calculated polarization information. BRIEF DESCRIPTION OF DRAWINGS

[0024] [ Figure 1 ] Figure 1 A configuration of an embodiment is shown.

[0025] [ Figure 2 ] Figure 2 A configuration of a polarization imaging unit is shown.

[0026] [ Figure 3 ] Figure 3 A pixel configuration in multiple polarization directions is shown.

[0027] [ Figure 4 ] Figure 4 A pixel configuration in multiple polarization directions (three-color pixels and white pixels) is shown.

[0028] [ Figure 5 ] Figure 5 A pixel configuration in multiple polarization directions (non-polarization pixels) is shown.

[0029] [ Figure 6 ] Figure 6 is a flowchart showing the operation of an embodiment.

[0030] [ Figure 7 ] Figure 7 is a flowchart showing the detection of a state change.

[0031] [ Figure 8 ] Figure 8 Operation in a case where it is detected that the change in polarization information becomes smaller than a predetermined change over time is shown.

[0032] [ Figure 9 ] Figure 9 Operation in a case where it is detected that the change in polarization information becomes larger than a predetermined change over time is shown.

[0033] [ Figure 10 ] Figure 10 Setting of a state change index based on the result of a comparison between polarization information and a threshold value is shown.

[0034] [ Figure 11 ] Figure 11 Output of the detection result of a separate state change is shown.

[0035] [ Figure 12 ] Figure 12 The detection result of a state change and a non-polarization image that are output while being associated with each other are shown. DETAILED DESCRIPTION

[0036] An embodiment for implementing the present technology will be described below. Here, the description will be made in the following order.

[0037] 1. Configuration of Embodiment

[0038] 2. Operation of Embodiment

[0039] 3. Application Examples

[0040] <1. Configuration of Embodiment>

[0041] Figure 1 A configuration according to an embodiment of the present technology is shown. An information processing system 10 includes a polarization imaging unit 20 that captures polarization images, a polarization information calculation unit 30 that calculates polarization information of each image from a time series of polarization images captured by the polarization imaging unit 20, an object specifying unit 50 that specifies an object whose state change is to be detected, a state change detection unit 60 that detects a state change of the object based on a time series change in the polarization information calculated by the polarization information calculation unit 30, and an output unit 70 that outputs a detection result of the state change of the object. Alternatively, the information processing system 10 can include a non-polarization image generation unit 40 that generates a non-polarization image from a polarization image captured by the polarization imaging unit 20, the object specifying unit 50 can specify the object based on the non-polarization image, and the output unit 70 can output the detection result of the state change such that the detection result is associated with the non-polarization image. The non-polarization image generation unit 40 is provided in the following description.

[0042] Figure 2 A configuration of the polarization imaging unit is shown. The polarization imaging unit 20 performs imaging using a polarization element and captures a polarization image. For example, as shown in (a) of Figure 2 Fig. 1, the polarization imaging unit 20 captures a polarization image with a polarization filter 202 provided on an image sensor 201 of a CMOS (Complementary Metal Oxide Semiconductor) or a CCD (Charge-Coupled Device) or the like, the polarization filter 202 having a pixel configuration with a plurality of polarization directions. If linearly polarized light can be extracted from object light indicating an object, the polarization filter 202 can be, for example, a wire grid or a photonic crystal. The arrow of the polarization filter 202 as well as the polarization plates 204 and 212-1 to 212-4 to be described later indicates a polarization direction.

[0043] As shown in (b) of Figure 2 Fig. 1, the polarization imaging unit 20 can generate a plurality of polarization images of different polarization directions by using a configuration of a multi-lens array. For example, a plurality of lenses 203 (lens array) are provided in front of the image sensor 201, and a polarization plate 204 is provided in front of the lens array. The polarization plate 204 has a pixel configuration with a plurality of polarization directions. The polarization plate 204 can be, for example, a wire grid or a photonic crystal. The arrow of the polarization plate 204 indicates a polarization direction. Figure 2(b) There are four lenses 203, and each of the lenses 203 forms an optical image of the object on the imaging surface of the image sensor 201. Furthermore, a polarizing plate 204 is disposed in front of each lens 203, and multiple polarized images are generated in different polarization directions by means of the polarizing plates 204 having different polarization directions. The polarization imaging unit 20 configured in this way can capture multiple polarization images in a single shot, thereby enabling rapid object identification. Figure 2 As shown in (c), polarizing plates 212-1 to 212-4 with different polarization directions can be provided in front of imaging units 210-1 to 210-4 to generate multiple polarized images with different polarization directions from different viewpoints.

[0044] If the object moves slowly or is operated on step by step, such as Figure 2 As shown in (d), a polarizing plate 211 can be disposed in front of the imaging unit 210. In this case, the polarizing plate 211 is rotated to capture images in different polarization directions, thereby obtaining multiple polarized images in different polarization directions.

[0045] exist Figure 2 In cases (b) and (c), if the interval between lenses 203 or between imaging units 210-1 to 210-4 is negligible relative to the distance to the object, parallax in multiple polarized images with different polarization directions is ignored. Therefore, by averaging the brightness of polarized images with different polarization directions, an image equivalent to a non-polarized normal brightness image can be obtained. In cases where parallax cannot be ignored, polarized images with different polarization directions are aligned according to the parallax amount, and the brightness of the aligned images is averaged, so that an image equivalent to a non-polarized normal brightness image can be obtained. Figure 2 In case (d), the brightness of polarized images with different polarization directions is averaged for each pixel, so that an image equivalent to a non-polarized normal brightness image can be obtained.

[0046] Figure 3 , Figure 4 and Figure 5 Pixel configurations in multiple polarization directions are shown. The configurations shown are repeated in both the horizontal and vertical directions. Figure 3 (a) and (b) in the figure show the pixel configuration when capturing a black and white image. Figure 3 (a) shows a 2×2 pixel block of polarized pixels, including polarized pixels in polarization directions (polarization angles) of, for example, 0º, 45º, 90º and 135º. Figure 3 (b) shows a 4×4 pixel polarization pixel block, including polarization pixels in polarization directions such as 0º, 45º, 90º, and 135º, while 2×2 pixels are used as units in the polarization direction. If the unit of the polarization component of the polarization filter is a 2×2 pixel, such as...Figure 3 As shown in (b), the leakage ratio of polarization components from adjacent regions of different polarization component units relative to the polarization components obtained for that polarization component unit is less than Figure 3 The leakage ratio of a 1×1 pixel in (a). When a wire grid is used as a polarization filter, polarized light passes through, including a field component in the vertical direction relative to the grid direction (line direction). Transmittance increases with line length. Therefore, if the unit of the polarization component is a 2×2 pixel, the transmittance is higher than that of a 1×1 pixel. Thus, the 2×2 pixel, as the unit of the polarization component, has a higher transmittance than the 1×1 pixel, thereby improving the extinction ratio.

[0047] Figure 3 (c) through (g) show the pixel configuration when capturing a color image. Figure 3 (c) shows the three primary color pixels (red, green, and red pixels) in the Bayer layout, where Figure 3 In (a), the 2×2 pixel polarized pixel block is used as a color unit.

[0048] Figure 3 (d) in the figure shows the target Figure 3 In (b), each pixel block of 2×2 pixels in the same polarization direction is set with three primary color pixels in the Bayer layout.

[0049] Figure 3 (e) shows the tri-color pixels set in the Bayer layout for each pixel block of 2×2 pixels in the same polarization direction, while 2×2 pixel blocks in different polarization directions have pixels of the same color.

[0050] Figure 3 (f) shows a 2×2 pixel block in the same polarization direction in the Bayer layout, wherein the phase difference between the horizontal and vertical adjacent pixel blocks in the polarization direction is 90 degrees, and the phase difference between the vertical and horizontal adjacent pixel blocks in the polarization direction is ±45 degrees.

[0051] Figure 3 (g) in the Bayer layout shows a 2×2 pixel block in the same polarization direction, wherein the phase difference between the adjacent pixel block in the polarization direction and the adjacent pixel block in the vertical direction is 90 degrees, and the phase difference between the adjacent pixel block in the polarization direction and the adjacent pixel block in the horizontal direction is ±45 degrees.

[0052] Figure 4 The settings for the three primary color pixels and the white pixel are shown. For example, Figure 4 (a) shows the replacement Figure 3 The white pixel is one of the green pixels in a 2×2 pixel block in the same polarization direction in the Bayer layout shown in (d).

[0053] Figure 4 (b) shows the replacement Figure 3 The white pixel is one of the green pixels in a 2×2 pixel block in the same polarization direction in the Bayer layout shown in (e), while 2×2 pixel blocks in different polarization directions have pixels of the same color.

[0054] As disclosed in WO 2016 / 136085, the setting of white pixels can achieve a greater dynamic range when generating normal information compared to the absence of white pixels. Furthermore, white pixels have an appropriate signal-to-noise ratio and are therefore less susceptible to noise in calculations such as color difference.

[0055] Figure 5 The settings for non-polarized pixels are shown. Figure 5 (a) through (d) in the figure show the acquisition of black and white images, while Figure 5 (e) to (l) in the diagram illustrate the acquisition of the color image. The polarization direction and color pixels are shown below. Figure 3 As shown.

[0056] Figure 5 (a) shows the replacement of the diagonally placed Figure 3 In (b), the non-polarized pixels are polarized pixels in a 2×2 pixel block with the same polarization direction.

[0057] Figure 5 (b) shows a polarized pixel with a 45-degree phase difference diagonally arranged in a 2×2 pixel block, which has a 90-degree phase difference with the adjacent pixel block.

[0058] Figure 5 (c) shows polarized pixels arranged diagonally in the same polarization direction within a 2×2 pixel block, with a phase difference of 45 degrees from adjacent pixel blocks. The pixels are polarized in both directions with a 45-degree phase difference. Polarization information can be obtained from unpolarized pixels and polarized pixels in both polarization directions, for example, according to techniques disclosed in WO 2018 / 074064.

[0059] Figure 5 (d) in the figure shows a polarized pixel with a 45-degree phase difference set diagonally in a 2×2 pixel pixel block, which is polarized in two directions with a 45-degree phase difference.

[0060] Figure 5(e) in FIG. 1 shows the same color pixel blocks (2x2 pixels) in a Bayer layout, where the 4x4 pixel blocks are configured using two 2x2 pixel blocks in four different polarization directions and two 2x2 pixel blocks including non-polarized pixels, such that the pixel blocks of polarized pixels are set as green pixels and the pixel blocks of non-polarized pixels are set as red or blue pixels.

[0061] Figure 5 (f) in FIG. 1 shows the polarized pixels set as in (d) of FIG. 1 and the pixel blocks of the three primary colors in a Bayer layout, where the color cells are pixel blocks including two polarized images in different polarization directions and two non-polarized pixels. Figure 5

[0062] Figure 5 (g) in FIG. 1 shows the pixel blocks of the three primary colors in a Bayer layout, where the color cells are 2x2 pixel blocks. The pixel blocks of green pixels include two polarized pixels in different polarization directions.

[0063] Figure 5 (h) in FIG. 1 shows the polarized pixels set as in (d) of FIG. 1. The pixel blocks including two polarized images in different polarization directions and two non-polarized pixels have three green pixels and one red pixel as one of the non-polarized pixels. In the adjacent pixel blocks, one of the non-polarized pixels is a blue pixel. Figure 5

[0064] (i) and (j) in FIG. 1 show the pixels of the three primary colors in the 4x4 pixel blocks including non-polarized pixels as color pixels. Figure 5 (k) and (1) in FIG. 1 show the pixels of the three primary colors in the 4x4 pixel blocks including non-polarized pixels, where some of the non-polarized pixels are used as color pixels. Figure 5

[0065] The configurations in FIG. 1 are merely exemplary, and other configurations can be used instead. The configurations with mixed infrared (IR) pixels can be repeated to enable high-sensitivity imaging, for example, at night. Figures 3 to 5

[0066] ​​The polarization information calculated by the polarization information calculation unit 30 will be described below. When a polarization plate is disposed perpendicular to an observation direction to observe partially polarized light that passes through the polarization plate, the luminance of the transmitted light changes every time the polarization plate is rotated. When a two-dimensional coordinate system (x-axis and y-axis) is defined on a plane of the polarization plate that is rotated to have a maximum luminance Imax and a minimum luminance Imin, a polarization angle υ formed by the rotation of the polarization plate is defined as an angle formed by a polarization axis of the polarization plate and the x-axis, and is expressed as an angle formed from the x-axis to the y-axis. The polarization axis indicates a direction of polarization of light that passes through the polarization plate. When the polarization plate is rotated, the polarization direction has a periodicity of 180º, and the polarization angle ranges from 0º to 180º. In this case, it is known that when a polarization angle θpol in observation of the maximum luminance Imax is defined as a phase angle φ, a luminance I observed during the rotation of the polarization plate can be expressed by the following equation (1).

[0067] [Math. 1]

[0068] (1)

[0069] In equation (1), the polarization angle θpol is determined during generation of the polarization image, and the maximum luminance Imax, the minimum luminance Imin, and the phase angle φ are variables. Therefore, by fitting the polarization model formula in equation (1) by using luminances of polarization images in three or more polarization directions, a luminance of a desired phase angle φ can be estimated based on the polarization model formula indicating a relationship between the luminance and the polarization angle.

[0070] In this case, from the minimum luminance Imin and the maximum luminance Imax, a degree of polarization p can be calculated as polarization information based on equation (2). The degree of polarization indicates a ratio of polarization in the observed light.

[0071] [Math. 2]

[0072] (2)

[0073] The polarization model formula expressed in equation (1) can be transformed into equation (3). If the polarization directions of the polarization elements are disposed at intervals of 45 degrees, for example, if a polarization direction of 0 degrees has a luminance "I0", a polarization direction of 45 degrees has a luminance "I1", a polarization direction of 90 degrees has a luminance "I2", and a polarization direction of 135 degrees has a luminance "I3", the coefficient a in equation (3) is a value expressed in equation (4). The coefficients b and c in equation (3) are values expressed in equations (5) and (6).

[0074] [Math. 3]

[0075] (3)

[0076] (4)

[0077] (5)

[0078] (6)

[0079] At this time, the degree of polarization p can be calculated based on Equation (7). As the polarization information, the polarization phase φp can be calculated based on Equation (8).

[0080] [Math. 4]

[0081] (7)

[0082] (8)

[0083] The polarization information calculating unit 30 can calculate a Stokes vector as the polarization information. Equation (9) indicates the Stokes vector S. The observed values I0, I45, I90, and I135 indicate observed values in polarization directions of 0º, 45º, 90º, and 135º.

[0084] [Math. 5]

[0085] (9)

[0086] In the Stokes vector S, the component S0 indicates the brightness or average brightness of the non-polarized light expressed in Equation (10). The component S1 indicates the difference in intensity between the polarization directions of 0º and 90º expressed in Equation (11), and the component S2 indicates the difference in intensity between the polarization directions of 45º and 135º expressed in Equation (12).

[0087] [Math. 6]

[0088] (10)

[0089] (11)

[0090] (12)

[0091] The non-polarized image generating unit 40 generates a non-polarized image from the polarized image captured by the polarization imaging unit 20. The non-polarized image generating unit 40 performs the operation of Equation (10) to generate a non-polarized image. In the case where there is a non-polarized pixel as shown in Figure 5 Equation (10), the non-polarized image can be generated by using the pixel value generated by the non-polarized pixel. The non-polarized image generating unit 40 outputs the generated non-polarized image to the object specifying unit 50 and the output unit 70.

[0092] The object specifying unit 50 specifies a region of an object whose state change is to be detected from the polarimetric image captured by the polarimetric imaging unit 20 or the non-polarimetric image generated by the non-polarimetric image generating unit 40.

[0093] The region of the object can be specified by a user or automatically specified based on recognition. For example, if the region of the object is specified by the user, the image is divided into regions in advance, and then the region is specified so that the region corresponding to at least one object is specified by the user for each object. Alternatively, the user can specify a boundary between the region of the object and other regions, and specify at least one closed region as the region of the object.

[0094] If the region of the object is automatically specified based on recognition, recognition is performed by using the non-polarimetric image generated by the non-polarimetric image generating unit 40 as in a conventional technique, and a region in which a subject is expected to be recognized is specified as the region of the object. The object specifying unit 50 can automatically extract only a region having a high degree of polarization, or automatically extract only a portion in which the polarization information has changed based on a time series.

[0095] The state change detecting unit 60 detects a state change of the object based on a time series change in the polarization information by using the polarization information on the region of the object specified by the object specifying unit 50 in the polarization information calculated by the polarization information calculating unit 30. For example, the state change detecting unit 60 detects that a change in the polarization information becomes smaller than a predetermined change or a change in the polarization information becomes larger than a predetermined change as the state change over time. The state change detecting unit 60 outputs a detection result of the state change to the output unit 70.

[0096] The output unit 70 outputs a detection result of a state change of the object. The output unit 70 outputs a state change detection image in which the state change of the object is expressed as, for example, a binary value or a continuous value for each pixel based on the detection result of the state change, which is generated by the state change detecting unit 60. Alternatively, the output unit 70 can output a state change detection image in which the state change of the object is expressed as, for example, a binary value or a continuous value for each object, or output the state change as state change detection information including a binary value or a continuous value for each image including the object.

[0097] If the state change of the object is expressed as, for example, a binary value or a continuous value for each object, the output unit 70 can perform, for example, image processing on the non-polarimetric image according to the detection result of the state change, and output the image so that the non-polarimetric image generated by the non-polarimetric image generating unit 40 and the detection result of the state change generated by the state change detecting unit 60 are associated with each other.

[0098] <2. Operation of the Embodiment>

[0099] Figure 6 is a flowchart showing the operation of the embodiment. In step ST1, the information processing system starts capturing polarization images. The polarization imaging unit 20 of the information processing system 10 starts imaging to capture polarization images, and then proceeds to step ST2. The interval of imaging by the polarization imaging unit 20 can be set in advance at a predetermined time interval, or can be set according to a state change of the subject. For example, in the case of a short-time state change, the interval of imaging is shortened, while in the case of a long-time state change, the interval of imaging is lengthened.

[0100] In step ST2, the information processing system acquires polarization information. The polarization information calculating unit 30 of the information processing system 10 calculates polarization information of each image according to the polarization images captured sequentially by the polarization imaging unit 20. The polarization information calculating unit 30 calculates at least one polarization information that changes according to a state change of the subject. For example, the polarization imaging unit 20 calculates a degree of polarization, a polarization phase, and a Stokes vector, and then proceeds to step ST3.

[0101] In step ST3, the information processing system generates non-polarization images. The non-polarization image generating unit 40 of the information processing system 10 generates non-polarization images according to the polarization images captured in step ST1, and then proceeds to step ST4.

[0102] In step ST4, the information processing system specifies a subject. The subject specifying unit 50 of the information processing system 10 specifies a region of a subject in a polarization image based on a user instruction or recognition, etc., and then proceeds to step ST5.

[0103] In step ST5, the information processing system detects a state change. The state change detecting unit 60 of the information processing system 10 detects a state change of a subject based on, for example, polarization information about the subject specified in step ST4.

[0104] Figure 7 is a flowchart showing the detection of a state change. In step ST11, the state change detecting unit performs filtering. The state change detecting unit 60 performs filtering on polarization information about a region of a subject in a time direction or a spatial direction, and removes noise. For example, the state change detecting unit 60 performs filtering using an average filter, a median filter, and a Gaussian filter, etc., and then proceeds to step ST12. Since a delay is caused by filtering in the time direction, for example, the interval of imaging of polarization images can be adjusted to avoid the influence of the delay.

[0105] In step ST12, the state change detection unit calculates the state change index. The state change detection unit 60 calculates, for each pixel or each polarization information, a state change index indicating a change in the time series of the polarization information. The state change detection unit 60 calculates a slope s(x, y, p) of a change in the polarization information p(x, y) of the pixel (x, y) for which the state change index is to be calculated. Further, the state change detection unit 60 calculates a state change index c(t, x, y, p) at the time t based on the slope s(x, y, p).

[0106] If the change in the polarization information is detected to become smaller than the predetermined change over time, the state change detection unit 60 generates the state change index c(t, x, y, p) as a binary value based on the formula (13) and the formula (14). Further, the state change detection unit 60 can generate the state change index c(t, x, y, p) as a continuous value based on the formula (15).

[0107] [mathematical formula 7]

[0108] If (13)

[0109] If (14)

[0110] (15)

[0111] Figure 8 The operation in a case where the change in the polarization information is detected to become smaller than the predetermined change over time is shown. Figure 8 (a) and (b) in FIG. 12 show changes in the polarization information p(x, y) over time. Figure 8 (a) in FIG. 12 shows that a decrease in the value of the polarization information p(x, y) decreases over time. Figure 8 (b) in FIG. 12 shows that an increase in the polarization information p(x, y) decreases over time.

[0112] If the state change detection unit 60 detects that the absolute value of the slope s(x, y, p) of the change in the polarization information p(x, y) over time in (a) of FIG. 11 at the time t1 Figure 8 is equal to or lower than the threshold value Th1, or if the state change detection unit 60 detects that the absolute value of the slope s(x, y, p) of the change in the polarization information p(x, y) over time in (b) of FIG. 11 at the time t2 Figure 8 is equal to or lower than the threshold value Th1, the state change index c(x, y, p) changes from “0” to “1” based on the formula (13) and (14), as shown inFigure 8 As shown in (c).

[0113] If the parameter α in formula (15) is pre-set to the maximum value of the absolute value of the slope s(x, y, p), then the state change exponent c(x, y, p) changes continuously from "0" to "1" based on formula (15), as follows: Figure 8 As shown in (d).

[0114] If the detected change in polarization information becomes greater than a predetermined change over time, the state change detection unit 60 generates a state change exponent c(t, x, y, p) as a binary value based on formulas (16) and (17). Alternatively, the state change detection unit 60 can generate the state change exponent c(t, x, y, p) as a continuous value based on formula (18).

[0115] [Mathematical Expression 8]

[0116] if (16)

[0117] if (17)

[0118] (18)

[0119] Figure 9 The operation is shown when the detected change in polarization information becomes greater than a predetermined change over time. Figure 9 (a) and (b) in the figure show the change of polarization information p(x, y) over time. Figure 9 (a) shows that the value of polarization information p(x, y) increases over time. Figure 9 (b) shows that the decrease in polarization information p(x, y) increases over time.

[0120] If the state change detection unit 60 detects a change at time t3 Figure 9 The absolute value of the slope s(x, y, p) of the polarization information p(x, y) changing with time in (a) is greater than the threshold Th2, or if the state change detection unit 60 detects at time t4 Figure 9 If the absolute value of the slope s(x, y, p) of the polarization information p(x, y) changing over time in (b) is greater than the threshold Th2, then the state change exponent c(x, y, p) changes from “0” to “1” based on formulas (16) and (17), as shown below. Figure 9 As shown in (c).

[0121] If the parameter β of formula (18) is set in advance to the maximum value of the absolute value of the slope s(x, y, p), the state change index c(x, y, p) continuously changes from "0" to "1" based on formula (18) as shown in (d) of FIG. 10. Figure 9

[0122] Alternatively, the state change detection unit 60 can set the state change index based on the comparison result between the polarization information and the predetermined threshold value regardless of the change in the polarization information. Figure 10 The setting of the state change index based on the result of the comparison between the polarization information and the threshold value is shown. For example, when the polarization information is equal to or lower than the threshold value, the state change detection unit 60 sets the state change index to "0", and when the polarization information is greater than the threshold value, the state change detection unit 60 sets the state change index to "1".

[0123] Figure 10 (a) and (c) of FIG. 9 indicate the polarization information p(x, y) with respect to the pixel (x, y) at each time. For example, if the polarization information p(x, y) with respect to the pixel (x, y) changes as shown in (a) of FIG. 9, the state change index c(x, y, p) is "1" in the period during which the polarization information p(x, y) is greater than the threshold value Th3, and the state change index c(x, y, p) is "0" when the polarization information p(x, y) is equal to or lower than the threshold value Th3 as shown in (b) of FIG. 9. Figure 10 Figure 10 If the polarization information p(x, y) with respect to the pixel (x, y) changes as shown in (c) of FIG. 9, the state change index c(x, y, p) is "0" in the period during which the polarization information p(x, y) is equal to or lower than the threshold value Th4, and the state change index c(x, y, p) is "1" when the polarization information p(x, y) is greater than the threshold value Th4 as shown in (d) of FIG. 9.

[0124] If the polarization information p(x, y) with respect to the pixel (x, y) changes as shown in (c) of FIG. 9, the state change index c(x, y, p) is "0" in the period during which the polarization information p(x, y) is equal to or lower than the threshold value Th4, and the state change index c(x, y, p) is "1" when the polarization information p(x, y) is greater than the threshold value Th4 as shown in (d) of FIG. 9. Figure 10 Figure 10 In this way, in each polarization information and each target region (or on the image), the state change detection unit 60 calculates the state change index for each pixel, and calculates as many state change indices as "(the number of polarization information) x (the number of pixels in the region) x (the number of regions (for the entire image, "1"))".

[0125] In step ST13, the state change detection unit merges the state change indices. The state change detection unit 60 merges the state change indices calculated in step ST12. The state change indices are merged for each polarization information, each pixel, each object region, or each image.

[0126] In step ST13, the state change detection unit merges the state change indices. The state change detection unit 60 merges the state change indices calculated in step ST12. The state change indices are merged for each polarization information, each pixel, each object region, or each image.​​​

[0127] The state change detection unit 60 merges the state change indices according to any one of the following methods. The merged state change indices are used as the detection result of the state change of the object.

[0128] In the first method, the merging is performed according to the number of polarization information, and the merged state change index of each pixel in the region of the object is used as the detection result of the state change of the object. For example, the state change index of the pixel (xi, yi) of the region Rj at the time t is merged based on, for example, Equation (19). In merging the state change index of the pixel (xi, yi) of the region Rj at the time t, the maximum value can be calculated as shown in Equation (20), the minimum value can be calculated as shown in Equation (21), or the median value can be calculated as shown in Equation (22). Furthermore, in the merging of the state change indices, the average value or statistics such as the mode can be calculated. In Equation (19), the weight Wpi of the polarization information is the weight of the polarization information pi, and the weight Wpn is the weight of the polarization information pn. The polarization information having a high reliability has a large weight. For example, if the polarization information includes the polarization degree and the polarization phase, the polarization degree frequently changes according to the state change, and the polarization phase is susceptible to noise. Therefore, the weight Wpb of the polarization phase is expressed as "Lb = (1-La):(La>Lb)", where "La" is the weight Wpa of the polarization degree. If the Stokes vector is also used as the polarization information, the weight Wps1 is set for the component S1 representing the polarization, and the weight Wps2 is set for the component S2. For example, the weight Wpa = 0.7, the weight Wpb = 0.1, the weight Wps1 = 0.1, and the weight Wps2 = 0.1 are set. The weights are normalized in advance according to the number of polarization information.

[0129] [Equation 9]

[0130] (19)

[0131] (20)

[0132] (21)

[0133] (22)

[0134] In the second method, the merging is performed on (the number of polarization information) x (the number of pixels in a region), and the merged state change index of each region is used as the detection result of the state change of the object. For example, the state change index of the region Rj at the time t is merged based on the formula (23) or (24). As shown in the formula (24), in addition to the maximum value, the minimum value, the median value, the average value, or the statistics such as the mode can be calculated. In the formula (23), for example, the weight Wili is the weight of the pixel (xii, yii), and the weight Win is the weight of the pixel (xin, yin). The weight Win to the weight Wili refers to the inverse of the distance from the center of gravity of the region, the probability density function of the Gaussian distribution or the like, the degree of polarization, or other values of the polarization information. Since the number of pixels varies among the regions, the weights are normalized in advance.

[0135] [mathematical formula 10]

[0136] (23)

[0137] (24)

[0138] In the third method, the merging is performed on (the number of polarization information) x (the number of pixels in a region) x (the number of regions), and the single state change index of the image is used as the detection result of the state change. For example, the state change index of the entire image at the time t is merged based on the formula (25) or (26). As shown in the formula (26), in addition to the maximum value, the minimum value, the median value, the average value, or the statistics such as the mode can be calculated. In the formula (25), for example, the weight WR1 is the weight set for the region R1, and the weight WRn is the weight set for the region Rn. The weight WR1 to the weight WRn is set based on the number of pixels in the region, the average degree of polarization in the region, and other values of the polarization information. Since the number of regions varies among the images, the weights are normalized in advance.

[0139] [mathematical formula 11]

[0140] (25)

[0141] (26)

[0142] As described above, the state change index is merged in the order of (the number of polarization information) → (the number of pixels in a region) → (the number of regions). The order can be changed to (the number of pixels in a region) → (the number of regions) → (the number of polarization information).

[0143] Referring again to Figure 6The information processing system outputs the detection result of the state change in step ST6. The output unit 70 of the information processing system 10 outputs the detection result of the state change, which is generated by the state change detection unit 60.

[0144] When the detection result of the state change is output, the output unit 70 outputs the detection result of the state change alone. Alternatively, the output unit 70 can output the detection result in association with the unpolarized image. For example, the output unit 70 can perform image processing on the unpolarized image according to the detection result of the state change, change a display property (for example, at least any one of luminance, hue, and saturation) of the unpolarized image according to the detection result of the state change, and superimpose an image of the detection result of the state change on the unpolarized image.

[0145] If the detection result of the state change is obtained for each pixel, the output unit 70 changes a display property of a pixel corresponding to the unpolarized image according to the detection result of the state change, for example. If the detection result of the state change is obtained for each region, the output unit 70 can change a display property of a region corresponding to the unpolarized image according to the detection result of the state change, and superimpose an image corresponding to the detection result of the state change on the region corresponding to the polarized image. If the detection result of the state change is obtained for each image, the output unit 70 adds information about the detection result of the state change to the unpolarized image.

[0146] Figure 11 The output of the detection result of the state change alone is shown. In Figure 11 , two objects OB1 and OB2 are included in the polarized image. In Figure 11 , (a) and (b) show that the detection result of the state change is output for each pixel alone. In Figure 11 , (c) and (d) show that the detection result of the state change is output for each region of the objects OB1 and OB2 alone. In Figure 11 , (a) and (c) show that the detection result of the state change is "0" or "1" in a binary representation. In Figure 11 , (b) and (d) show that the detection result of the state change is a value ranging from "0" to "1".

[0147] Figure 12 The detection result of the state change and the unpolarized image, which are associated with each other, are shown. Figure 12 , (a) and Figure 12 , (d) each show the unpolarized image. The object OB is a shirt, for example. A wet shirt has a high degree of polarization, and a dry shirt has a low degree of polarization.

[0148] Figure 12 , (b) and Figure 12Each of (e) in FIG. 12 shows a detection result of a state change. In Figure 12 In (b) of FIG. 11, the detection result of the state change is "0" or "1" in binary representation. In Figure 12 In (e) of FIG. 12, the detection result of the state change is a value ranging from "0" to "1".

[0149] If the detection result of the state change is "0" or "1" in binary representation, the output unit 70 associates Figure 12 the detection result of the state change in (b) of FIG. 11 with Figure 12 the non-polarized image in (a) of FIG. 11, performs image processing, and outputs Figure 12 the image of (c) in FIG. 12 so that the detection result of the state change is associated with the non-polarized image.

[0150] If the detection result of the state change is a value ranging from "0" to "1", the output unit 70 associates Figure 12 the detection result of the state change in (e) of FIG. 12 with Figure 12 the non-polarized image in (d) of FIG. 12, performs image processing, and outputs Figure 8 the image of (f) in FIG. 12 so that the detection result of the state change is associated with the non-polarized image.

[0151] In this way, when the wet shirt is dried, the degree of polarization decreases so that the dry shirt has a low degree of polarization. As ​ indicated, the detection result of the state change changes from "0" to "1" or continuously from "0" to "1". Thus, the information processing system 10 can indicate the drying state of the shirt as a numerical value or a result of image processing on the non-polarized image.

[0152] The present technology can easily recognize a state change of an object based on polarization information. The detection result of the state change can be output while being associated with a non-polarized image, thereby easily recognizing a type of the object, a position of the state change on the object, and a type of the state change, such as a drying state of a clothing, and the like.

[0153] <3. Application Examples>

[0154] The operation of the foregoing embodiment shows the detection of a state change of a clothing. As long as a state change to be observed is accompanied by a change in polarization information, the embodiment can be applied to the observation of various objects. For example, the embodiment can detect a drying state of a shoe, a floor, or a surface painted or coated with an adhesive, and whether a clothing is wetted with a liquid by being in contact with rain or melted ice. In addition, if polarization information is changed by surface deterioration, corrosion, or friction, and the like, the embodiment can recognize deterioration, corrosion, friction, and metal fatigue, and the like of a facility or equipment from a remote location based on a polarization image of the facility or equipment, the polarization image being captured at a predetermined time interval.

[0155] By capturing a polarized image of a road or the like at predetermined time intervals, it is possible to recognize the state of the road from a dry state, a wet state, and a frozen state based on the polarized image. This makes it possible to accurately and easily manage the road condition.

[0156] Further, the information processing system of the present technology realizes output of a non-polarized image in addition to the detection of the state change, and thus is useful in the monitoring field, such as theft prevention, detection of suspicious persons, and monitoring of pets. Although the non-polarized image is generally output, the detection result of the state change is associated with the output non-polarized image only when the detection of the state change is required (for example, when a function of detecting the state change is enabled).

[0157] The series of processes described in the specification can be executed by a hardware configuration, a software configuration, or a combination thereof. When the processes are executed by software, after a program in which the process sequence is recorded is installed in a memory in a computer embedded in a dedicated hardware, the program is executed. Alternatively, the program can be executed after being installed in a general-purpose computer capable of executing various types of processes.

[0158] For example, the program can be recorded in advance on a hard disk, an SSD (Solid State Drive), or a ROM (Read Only Memory) as a recording medium. Alternatively, the program can be temporarily or permanently stored (recorded) in a removable recording medium such as a floppy disk, a CD-ROM (Compact Disc-Read Only Memory), an MO (Magnetic Optical) disk, a DVD (Digital Versatile Disc), a BD (Blu-Ray Disc (registered trademark)), a magnetic disk, or a semiconductor memory card. The removable recording medium can be provided as so-called packaged software.

[0159] In addition to being installed from the removable recording medium into the computer, the program can be transmitted to the computer wirelessly or wiredly via a WAN (Wide Area Network) such as a cellular system, a LAN (Local Area Network), or the Internet, from a download site. The computer can receive the program transmitted in this way, and install the program on a recording medium such as a built-in hard disk.

[0160] The effects described in the present specification are merely examples and are not limiting, and there can be additional effects not described. Furthermore, the present technology should not be interpreted as being limited to the embodiments described above. Embodiments of the present technology disclose the present technology in a form of examples, and it is obvious that those skilled in the art can modify or replace the embodiments without departing from the gist of the present technology. In other words, the gist of the present technology should be determined considering the claims.

[0161] The information processing apparatus of the present technology can also have the following configuration.

[0162] (1) An information processing apparatus comprising: a polarization information calculation unit that calculates polarization information of each polarization image from time-series polarization images of an object; and

[0163] a state change detection unit that detects a state change of the object based on a time-series change in the polarization information calculated by the polarization information calculation unit.

[0164] (2) The information processing apparatus according to (1), in which the state change detection unit calculates a state change index indicating the time-series change in the polarization information, and detects the state change of the object based on the state change index.

[0165] (3) The information processing apparatus according to (2), in which the state change detection unit calculates the state change index for each pixel position of the polarization image.

[0166] (4) The information processing apparatus according to (2) or (3), in which the polarization information calculation unit calculates a plurality of different polarization information, and

[0167] the state change detection unit calculates the state change index for each polarization information.

[0168] (5) The information processing apparatus according to any one of (2) to (4), in which the state change detection unit merges the state change indices, and detects the state change of the object based on the merged state change indices.

[0169] (6) The information processing apparatus according to (5), in which the state change detection unit assigns weights to the state change indices, and merges the state change indices to which weights are assigned.

[0170] (7) The information processing apparatus according to (6), in which the state change detection unit increases the weights of polarization information having high reliability.

[0171] (8) The information processing apparatus according to (6) or (7), in which the state change detection unit sets the weights for each pixel in an image region of the object or for each image region of the object.

[0172] (9) The information processing apparatus according to any one of (5) to (8), in which the state change detection unit merges the state change indices for each pixel position of the polarization image.

[0173] (10) The information processing apparatus according to any one of (5) to (9), wherein the state change detection unit merges the state change indices of each object in the polarization image.

[0174] (11) The information processing apparatus according to any one of (2) to (10), wherein the state change detection unit detects that a change in the polarization information becomes smaller than a predetermined change over time or the change in the polarization information becomes larger than the predetermined change as a detection of a state change of the object.

[0175] (12) The information processing apparatus according to any one of (2) to (11), further comprising an object specifying unit that specifies an image region of the object in the polarization image,

[0176] wherein a state change is detected based on a state change index of the image region of the object specified by the object specifying unit.

[0177] (13) The information processing apparatus according to (12), further comprising a non-polarization image generation unit that generates a non-polarization image from the polarization image,

[0178] wherein the object specifying unit performs recognition by using the non-polarization image generated by the non-polarization image generation unit and specifies the image region of the object.

[0179] (14) The information processing apparatus according to any one of (1) to (13), further comprising an output unit that outputs a detection result of a state change of the object, the detection result being obtained by the state change detection unit.

[0180] (15) The information processing apparatus according to (14), further comprising a non-polarization image generation unit that generates a non-polarization image from the polarization image,

[0181] wherein the output unit outputs the non-polarization image such that the detection result of the state change obtained by the state change detection unit is associated with the non-polarization image generated by the non-polarization image generation unit.

[0182] (16) The information processing apparatus according to (15), wherein the output unit performs image processing on the non-polarization image generated by the non-polarization image generation unit according to the detection result of the state change and outputs the non-polarization image.

[0183] LIST OF REFERENCE NUMERALS

[0184] 10 information processing system

[0185] 20 polarization imaging unit

[0186] 30 polarization information calculation unit

[0187] 40 unpolarized image generation unit

[0188] 50 object specification unit

[0189] 60 state change detection unit

[0190] 70 output unit

[0191] 201 image sensor

[0192] 202 polarization filter

[0193] 203 lens

[0194] 204, 211, 212-1 to 212-4 polarization plate

[0195] 210, 210-1 to 210-4 imaging unit

Claims

1. An information processing apparatus, comprising: A polarization information calculation unit calculates the polarization information of each polarization image based on the time-series polarization images of the object. as well as A state change detection unit detects a state change of the object based on a time-series change in polarization information calculated by the polarization information calculation unit. The state change detection unit calculates a state change index that indicates a time-series change in the polarization information, and detects state changes of the object based on the state change index.

2. The information processing apparatus according to claim 1, wherein, The state change detection unit calculates the state change index for each pixel position of the polarization image.

3. The information processing apparatus according to claim 1, wherein, The polarization information calculation unit calculates multiple different polarization information, and The state change detection unit calculates the state change index for each polarization information.

4. The information processing apparatus according to claim 1, wherein, The state change detection unit merges the state change indices and detects state changes of the object based on the merged state change indices.

5. The information processing apparatus according to claim 4, wherein, The state change detection unit assigns weights to the state change index and merges the weighted state change indices.

6. The information processing apparatus according to claim 5, wherein, The state change detection unit adds weights for polarization information with high reliability.

7. The information processing apparatus according to claim 5, wherein, The state change detection unit assigns a weight to each pixel in the image region indicating the object or to each image region indicating the object.

8. The information processing apparatus according to claim 4, wherein, The state change detection unit merges the state change index at each pixel location of the polarization image.

9. The information processing apparatus according to claim 4, wherein, The state change detection unit merges the state change index of each object in the polarization image.

10. The information processing apparatus according to claim 1, wherein, The state change detection unit detects whether the change in polarization information becomes less than a predetermined change or becomes greater than the predetermined change over time, as a state change detection of the object.

11. The information processing apparatus according to claim 1, further comprising an object designation unit, the object designation unit designating an image region of the object in the polarization image, in, State changes are detected based on the state change index of the image region of the object specified by the object designation unit.

12. The information processing apparatus according to claim 11 further includes a non-polarized image generation unit, wherein the non-polarized image generation unit generates a non-polarized image based on the polarized image. in, The object designation unit performs recognition by using the unpolarized image generated by the unpolarized image generation unit, and designates the image region of the object.

13. The information processing apparatus according to claim 1 further includes an output unit, the output unit outputting a detection result of a state change of the object, the detection result being obtained by the state change detection unit.

14. The information processing apparatus according to claim 13 further includes a non-polarized image generation unit, wherein the non-polarized image generation unit generates a non-polarized image based on the polarized image. in, The output unit outputs the unpolarized image, such that the detection result of the state change is associated with the unpolarized image generated by the unpolarized image generation unit, and the detection result is obtained by the state change detection unit.

15. The information processing apparatus according to claim 14, wherein, The output unit performs image processing on the unpolarized image generated by the unpolarized image generation unit based on the detection result of the state change, and outputs the unpolarized image.

16. An information processing method, comprising: The polarization information calculation unit calculates the polarization information of each polarization image based on the time-series polarization images of the object; as well as The state change detection unit detects a state change of the object based on a time-series change in the polarization information calculated by the polarization information calculation unit. The state change detection unit calculates a state change index that indicates a time-series change in the polarization information, and detects state changes of the object based on the state change index.

17. A recording medium containing a program that causes a computer to detect a change in the state of an object, said program causing the computer to perform the following operations: Calculate the polarization information of each polarization image based on the time-series polarization images of the object; Calculate the state change index indicating the time-series change of the polarization information; and The state change of the object is detected based on the state change index.

Images