Image object detection device, image object detection method, and program

Abstract

To detect an object in an image taken in an unfavorable weather condition with the same accuracy as in a favorable weather condition.SOLUTION: An image object detection device comprising a weather feature model generating part that generates a weather feature model that indicates a feature of weather displayed in an image, a feature map generating part that generates a feature map from image data of object detection targets, a coupling part that generates a feature map by conducting coupling that causally intervenes the weather feature model generated by the weather feature model generating part to the feature map generated by the feature map generating part, and an object detection part that detects an object from the feature map generated by the coupling part.SELECTED DRAWING: Figure 1

Description

Technical Field

[0001] The present invention relates to an image object detection device, an image object detection method, and a program.

Background Art

[0002] In recent years, many techniques for detecting objects from images using deep learning have been proposed. Such object detection techniques are expected to be used in the field of autonomous driving in the future. It has been reported that the object detection techniques proposed so far have excellent object detection capabilities for images taken in good weather conditions such as sunny days. However, in order to apply the technology to the field of autonomous driving, it is required to have the same object detection ability as in the case of good weather conditions in various weather conditions.

Prior Art Documents

Non-Patent Documents

[0003]

Non-Patent Document 1

Summary of the Invention

Problems to be Solved by the Invention

[0004] However, in an environment with severe weather conditions, in a place with little light, or at night, the sensor noise of the camera's imaging device increases, and in such cases, the quality of the captured image deteriorates. Therefore, when object detection is performed on an image captured under non-optimal weather conditions such as rain or fog using the object detection technologies proposed so far, there is a problem that the accuracy of object detection is lower than when the image is captured under optimal weather conditions (see, for example, Non-Patent Document 1).

[0005] An object of the present invention is to provide a technology capable of detecting an object with the same accuracy as in the case of good weather conditions from an image captured under non-optimal weather conditions.

Means for Solving the Problem

[0006] One aspect of the present invention includes a weather feature model generation unit that generates a weather feature model indicating the weather features displayed in an image, a feature map generation unit that generates a feature map from the image data of the object to be detected, and a combination that causally intervenes the weather feature model generated by the weather feature model generation unit with respect to the feature map generated by the feature map generation unit, thereby generating a feature map with the influence of weather suppressed, and an object detection unit that performs object detection from the feature map generated by the combination unit. It is an image object detection device provided with.

[0007] One aspect of the present invention includes a weather feature model generation step of generating a weather feature model indicating the weather features displayed in an image, a feature map generation step of generating a feature map from the image data of the object to be detected, and a combination that causally intervenes the weather feature model generated by the weather feature model generation step with respect to the feature map generated by the feature map generation step, thereby generating a feature map with the influence of weather suppressed, and an object detection step of performing object detection from the feature map generated by the combination step. It is an image object detection method including.

[0008] One aspect of the present invention is a program for causing a computer to function as weather feature model generation means for generating a weather feature model indicating weather features displayed in an image, feature map generation means for generating a feature map from image data of an object to be detected, and combining means for performing a combination that causally intervenes in the weather feature model generated by the weather feature model generation means on the feature map generated by the feature map generation means, thereby generating a feature map with the influence of weather suppressed, and object detection means for performing object detection from the feature map generated by the combining means.

Advantages of the Invention

[0009] According to the present invention, it becomes possible to detect an object with the same accuracy as in the case of good weather conditions from an image taken in a bad weather condition.

Brief Description of the Drawings

[0010]

Figure 1

Figure 2

Figure 3

Figure 4

Figure 5

Figure 6

Figure 7

Figure 8

Modes for Carrying Out the Invention

[0011] (First Embodiment) Hereinafter, embodiments of the present invention will be described with reference to the drawings. FIG. 1 is a block diagram showing the configuration of an image object detection device 1 according to the first embodiment. The image object detection device 1 includes a weather feature model generation unit 11, a weather feature model storage unit 12, an image data storage unit 13, a feature map generation unit 14, a combination unit 15, and an object detection unit 16.

[0012] The weather feature model generation unit 11 generates a weather feature model indicating the features of the weather displayed in the image. The weather feature model storage unit 12 stores the weather feature model generated by the weather feature model generation unit 11. The image data storage unit 13 stores image data that is the target of object detection. The feature map generation unit 14 generates a feature map from the image data stored in the image data storage unit 13. The combination unit 15 generates a feature map with the influence of the weather suppressed by combining the feature map generated by the feature map generation unit 14 and the weather feature model stored in the weather feature model storage unit 12. Here, the process of combining the feature map and the weather feature model performed by the combination unit 15 is a combination process by causal intervention, which is an intervention in a so-called structural causal model. By performing this process, when the image data corresponding to the feature map is, for example, image data taken in a rainy state, the influence of rain in the image data is suppressed. The object detection unit 16 performs an object detection process of detecting the type, position, and range of the object included in the image data corresponding to the feature map from the feature map generated by the combination unit 15.

[0013] FIG. 2 is a block diagram showing an example of the detailed configuration of the image object detection device 1, and shows an example in which the mask R-CNN (Region based Convolutional Neural Network) method is applied as a method for object detection. The image data storage unit 13 stores in advance, for example, image data that is the target of object detection. The image data is, for example, color image data, and is three-dimensional array data in which two-dimensional image data of each of the R (Red), G (Green), and B (Blue) channels, which are two-dimensional image data of the same size, are superimposed. Here, the size is the size determined by the number of vertical and horizontal pixels of the two-dimensional image data, in other words, the number of vertical and horizontal data of the two-dimensional array data.

[0014] The feature map generation unit 14 includes a CNN (Convolutional Neural Network) layer 21, and performs a convolution operation on the image data stored in the image data storage unit 13 by the CNN layer 21 to generate a feature map 50. Here, the feature map 50 is defined by the following formula (1).

[0015]

Equation

[0016] In formula (1), "F" represents the feature map 50, and "R" is a mathematical symbol representing the real number space. The superscript "d" of R is the number of dimensions, and "c" is the number of channels. That is, formula (1) shows that the feature map 50 is represented as an element of the d×c-dimensional real number space. Here, since the CNN layer 21 performs a convolution operation on data in which three two-dimensional image data stored in the image data storage unit 13 are superimposed in the channel direction, the number of dimensions "d" of the feature map 50 obtained by the convolution operation is d = 2. The number of channels "c" is a value corresponding to the number of channels of the filter applied in the CNN layer 21 and the like.

[0017] More specifically, as the CNN layer 21, a deep neural network such as an FCN (Fully Convolutional Network) that includes only CNNs may be applied, or a deep neural network that includes a CNN such as VGG (Visual Geometry Group) or ResNet (Residual Network) and a neural network other than CNNs may be applied. The feature map 50 obtained by the convolution operation by the CNN layer 21 is data of a three-dimensional array in which two-dimensional array data of the same size with the number of channels "c" are stacked in the channel direction. The two-dimensional array data to be stacked is data with the same number of vertical and horizontal data. The size of the two-dimensional array data is a size obtained by reducing the size of the image data read by the feature map generation unit 14 from the image data storage unit 13, and is a size corresponding to the size of the image data read by the feature map generation unit 14 from the image data storage unit 13 and the size of the filter of the CNN layer 21.

[0018] The object detection unit 16 includes an RPN (Region Proposal Network) layer 22 and a BoxHead unit 23. The RPN layer 22 is a neural network that detects the position and range where an object exists in the feature map 50. The BoxHead unit 23 includes an RoI (Region of Interest)-Align layer 31, fully connected layers 32, 33-1, 33-2, and CNN layers 34, 35. The RoI-Align layer 31 extracts the part where an object exists in the feature map 51, that is, the feature map 51 with the influence of weather suppressed, and the output of the RPN layer 22, performs Pooling processing on the data of the extracted part, and outputs data indicating the result of the Pooling processing.

[0019] The fully-connected layers 32, 33-1, and 33-2 are neural networks with full connections. When the data output by the RoI-Align layer 31 is given to the fully-connected layer 32, the fully-connected layers 32, 33-1 calculate data of a class indicating the type of the object from the given data. The output stage of the fully-connected layer 33-1 is a Softmax function, and the fully-connected layers 32, 33-1 calculate data that probabilistically indicates to which of a plurality of predetermined classes the object belongs as data indicating the class of the object. When the data output by the RoI-Align layer 31 is given to the fully-connected layer 32, the fully-connected layers 32, 33-2 calculate data indicating the position and range where the object exists, that is, data indicating the position and range of a so-called bounding box, from the given data.

[0020] The CNN layers 34, 35 are neural networks that perform convolution operations. When the data output by the RoI-Align layer 31 is given to the CNN layer 34, the CNN layers 34, 35 output data indicating the type of mask to be applied to the part of the object from the given data. Note that learned weights and biases are applied to the neurons of the neural networks constituting the CNN layer 21, the RPN layer 22, the fully-connected layers 32, 33-1, 33-2, and the CNN layers 34, 35 before the object detection process is performed.

[0021] The weather feature model generation unit 11 includes a weather image data storage unit 41, a feature map generation unit 42, a classification unit 43, a detection unit 44, and a synthesis unit 45. The weather image data storage unit 41 stores in advance a plurality of image data taken in various weather conditions. The image data stored in the weather image data storage unit 41 is RGB color image data, similar to the image data stored in the image data storage unit 13.

[0022] Similar to the feature map generation unit 14, the feature map generation unit 42 includes CNN layers such as FCN, VGG, ResNet, etc., and generates a feature map from each of the plurality of image data stored in the weather image data storage unit 41. The classification unit 43 clusters the plurality of feature maps generated by the feature map generation unit 42. The detection unit 44 detects the central feature vector of each cluster clustered by the classification unit 43. The synthesis unit 45 synthesizes the central feature vectors detected by the detection unit 44 to generate a weather feature model 70, which is a model that collectively represents the features of various weathers. Here, the weather feature model 70 is defined by the following formula (2).

[0023] [Number]

[0024] In formula (2), "W" represents the weather feature model 70, and formula (2) indicates that the weather feature model 70 is represented as an element in the d×h-dimensional real number space. In formula (2), the number of dimensions "d" is d = 2, the same as the feature map 50. The synthesis unit 45 synthesizes the central feature vectors so that the data of the two-dimensional array having the same size as the data of one channel of the feature map 50 is overlaid for "h" channels to form three-dimensional array data, and generates the weather feature model 70. The number of channels "h" is the number of clusters obtained when the classification unit 43 performs clustering, that is, the number of weather types.

[0025] (Processing by the weather feature model generation unit of the first embodiment) Referring to FIGS. 3 and 4, the processing by the weather feature model generation unit 11 of the first embodiment will be described. Before the processing shown in FIG. 3 is started, a plurality of image data captured in a plurality of predetermined types of weather conditions are written in advance in the weather image data storage unit 41. Here, as an example, as shown in FIG. 4, in the weather image data storage unit 41, rain image data 52-1, fog image data 52-2, snow image data 52-3, cloudy image data 52-4, light cloudy image data 52-5, … captured in each of the states of “rain”, “fog”, “snow”, “cloudy”, and “light cloudy” are assumed to be written in advance. In FIG. 4, one rain image data 52-1 is shown as the image data for rainy weather, but it is assumed that a plurality of image data are stored in the weather image data storage unit 41 for one type of weather.

[0026] The rain image data 52-1, fog image data 52-2, snow image data 52-3, cloudy image data 52-4, light cloudy image data 52-5, … are image data captured at arbitrary positions, and the more different the shooting positions are in a large number of image data, the more generalized the finally obtained weather feature model 70 will be. However, the image data captured at the same position may be included in some of the image data.

[0027] The feature map generation unit 42 reads out the rain image data 52-1, fog image data 52-2, snow image data 52-3, cloudy image data 52-4, light cloud image data 52-5,... stored in the weather image data storage unit 41 one by one, and performs a convolution operation on each of the read image data to generate a feature map. That is, as shown in FIG. 4, the feature map generation unit 42 generates a feature map 53-1 of the rain image data for the rain image data 52-1. The feature map generation unit 42 also generates corresponding feature maps 53-2, 53-3, 53-4, 53-5,... for each of the fog image data 52-2, snow image data 52-3, cloudy image data 52-4, light cloud image data 52-5,... other than the rain image data 52-1. The feature map generation unit 42 outputs the generated feature maps 53-1, 53-2, 53-3, 53-4, 53-5,... to the classification unit 43 (step Sa1).

[0028] The classification unit 43 takes in the feature maps 53-1, 53-2, 53-3, 53-4, 53-5,... output by the feature map generation unit 42, and clusters the taken-in feature maps 53-1, 53-2, 53-3, 53-4, 53-5,... by, for example, a Gaussian mixture model (GMM). For example, as shown in the scatter diagram 60 of FIG. 4, the classification unit 43 plots the feature maps 53-1, 53-2, 53-3, 53-4, 53-5,... in the vector space. In the scatter diagram 60, the marks of "〇", "☆", "◇", "□", "△" are the results of plotting the feature maps 53-1, 53-2, 53-3, 53-4, 53-5,...

[0029] As described above, since a plurality of image data are stored in the weather image data storage unit 41 for one type of weather, for example, there will be a plurality of rain feature maps in addition to the feature map 53-1. In the scatter diagram 60, the positions indicated by each of the rain feature maps including the feature map 53-1 are indicated by the mark of "〇". Similarly, the positions indicated by each of the fog feature maps including the feature map 53-2 are indicated by the mark of "☆". The positions indicated by each of the snow feature maps including the feature map 53-3 are indicated by the mark of "◇". The positions indicated by each of the cloudy feature maps including the feature map 53-4 are indicated by the mark of "□". The positions indicated by each of the slightly cloudy feature maps including the feature map 53-5 are indicated by the mark of "△".

[0030] When plotted in the vector space, each of the plotted points is not classified by weather type like "〇", "☆", "◇", "□", "△", but the classification unit 43 classifies each of the plotted points by weather type by performing clustering. Note that the scatter diagram 60 shown in FIG. 4 shows, as an example for easy understanding, an example in which clustering is performed in a two-dimensional vector space, but the feature maps 53-1, 53-2, 53-3, 53-4, 53-5,... to be clustered are data of a three-dimensional array. Therefore, the clustering by the classification unit 43 may be performed in a multi-dimensional vector space exceeding two dimensions. The classification unit 43 outputs data indicating the result of clustering to the detection unit 44 (step Sa2).

[0031] The detection unit 44 takes in the data indicating the clustered results output by the classification unit 43, and based on the taken-in data, detects the central feature vectors 54-1, 54-2, 54-3, 54-4, 54-5 of each cluster. The detection unit 44 outputs data indicating each of the detected central feature vectors 54-1, 54-2, 54-3, 54-4, 54-5 to the synthesis unit 45 (step Sa3). The synthesis unit 45 takes in the data indicating each of the central feature vectors 54-1, 54-2, 54-3, 54-4, 54-5 output by the detection unit 44, synthesizes the taken-in data, and generates a weather feature model 70 which is a model representing collectively the features of five types of weather, namely "rain", "fog", "snow", "cloudy", and "lightly cloudy", and which is data in a three-dimensional array. Here, the process of synthesizing the data indicating each of the central feature vectors 54-1, 54-2, 54-3, 54-4, 54-5 is, for example, a process of calculating the inner product of each of the central feature vectors 54-1, 54-2, 54-3, 54-4, 54-5 (step Sa4). The synthesis unit 45 writes and stores the data of the generated weather feature model 70 in the weather feature model storage unit 12 (step Sa5).

[0032] (Object detection process in the first embodiment) With reference to FIGS. 5 and 6, the object detection process by the image object detection device 1 of the first embodiment will be described. Before the process shown in FIG. 5 starts, the weather feature model generation unit 11 generates the weather feature model 70, and the data of the weather feature model 70 generated by the weather feature model generation unit 11 is written in the weather feature model storage unit 12. Learned weights and biases are applied to each neuron of the neural networks constituting the CNN layer 21, the RPN layer 22, the fully connected layers 32, 33-1, 33-2, and the CNN layers 34, 35.

[0033] The feature map generation unit 14 reads the image data from the image data storage unit 13, and performs a convolution operation on the read image data by the CNN layer 21 to generate a feature map 50. The feature map generation unit 14 outputs the generated feature map 50 to the combination unit 15 and the object detection unit 16 (step Sb1). The combination unit 15 takes in the feature map 50 output by the feature map generation unit 14. The combination unit 15 reads the data of the weather feature model 70 from the weather feature model storage unit 12. The combination unit 15 performs a process of combining the feature map 50 and the weather feature model 70, that is, a process of causal intervention combination represented by the following formula (3).

[0034]

Equation

[0035] In formula (3), the F with a circumflex on the left side represents the feature map 51 generated by the combination unit 15. The formula of the first term on the right side is a formula for applying the softmax function to the product of the transpose of the feature map 50 (F T ) and the weather feature model 70 (W), that is, the product of two 3D arrays. Here, the transpose of the feature map 50 is to swap the elements of the feature map 50 as follows. That is, assuming that the feature map 50 is data of a 3D array of X×Y×c, and the elements of each 2D array data of X and Y for each of the c channels are represented by (x, y). However, x = 1~X and y = 1~Y. The transpose of the feature map 50 is to swap the elements of (x, y) of the 2D array data for each channel to become the elements of (y, x) while maintaining the channel direction in the feature map 50.

[0036] According to the formula of the first term on the right side of formula (3), when h = c, in other words, when the number of channels h of the weather feature model 70 (i.e., the weather feature model W) is the same as the number of channels c of the feature map 50 (i.e., the feature map F), data of a three-dimensional array with the same size and the same number of channels as the feature map 50 can be obtained. The values of the elements whose quality has deteriorated due to the influence of weather are large in each channel. For example, as shown in FIG. 6, if the feature map 50 is data of a three-dimensional array obtained by stacking data of a 4×4 two-dimensional array by the number corresponding to the number of channels "c", and the weather feature model 70 is data of a three-dimensional array obtained by stacking data of a 4×4 two-dimensional array by the number corresponding to the number of channels "h".

[0037] When h = c, the combining unit 15 calculates the product of the data of the three-dimensional array obtained by transposing the feature map 50 and the data of the three-dimensional array of the weather feature model 70, thereby obtaining data of a three-dimensional array with the same size and the same number of channels as the feature map 50, that is, 4×4×c three-dimensional array data. The combining unit 15 applies the softmax function to the values of the 16 elements for each channel of the calculated three-dimensional array data, and sets each output value of the softmax function as the value of the 16 elements. As a result, the sum of the values of the 16 elements becomes "1", and data of c two-dimensional arrays in which the values of the elements whose quality has deteriorated due to the influence of weather in the feature map 50 are large can be obtained. The combining unit 15 generates three-dimensional array data 71 with the same size and the same number of channels as the feature map 50 by stacking the c two-dimensional array data in the channel direction.

[0038] The "×" on the right side of Equation (3) represents an operation indicating the Hadamard product for each channel. The combining unit 15 performs an operation of multiplying the values of the elements at corresponding positions for each channel between the data 71 of the three-dimensional array obtained as the result of the first equation and the data of the three-dimensional array of the feature map 50. In other words, the combining unit 15 multiplies the value of the element at the x-th row and y-th column in the z-th channel of the data 71 of the three-dimensional array obtained as the result of the first equation by the value of the element at the x-th row and y-th column in the z-th channel of the feature map 50, and sets the resulting value as the value of the pixel at the x-th row and y-th column in the z-th channel of the feature map 51. Here, x, y, and z are positive integers. In the case of the example shown in FIG. 6, x = 1 to 4, y = 1 to 4, and z = 1 to c. As a result, in the feature map 50, the values of the elements whose quality has deteriorated due to the influence of the weather are emphasized, and a feature map 51 in which the influence of the weather is suppressed is obtained.

[0039] On the other hand, when the number of channels h of the weather feature model 70 is not h = c, in the above procedure, the product of the feature map 50 and the weather feature model 70 cannot be calculated. When h ≠ c, the procedure for the combining unit 15 to generate the feature map 51 is as follows. Let WF be the part including the weather feature for each channel of the feature map 50 (hereinafter referred to as the weather feature part WF). Here, the weather feature part WF is the data of a two-dimensional array defined by the following equation (4), that is, the data of one channel of the data of a three-dimensional array. When the feature map 50 is the data of a 4×4×c three-dimensional array as shown in FIG. 6, the weather feature part WF is the data of a 4×4×1 three-dimensional array.

[0040]

Equation

[0041] The combining unit 15 calculates the weather feature part WF for c channels according to the following equation (5).

[0042]

Equation

[0043] In Equation (5), j = 1 to c, and WF j is the weather feature part WF of the j-th channel of the feature map 50 having c channels. i is 1 to h, and m j,i represents the j-th m i and m i can be calculated by the following Equation (6). In Equation (6) below, "F" on the right side represents the j-th channel of the feature map 50 (i.e., the feature map F).

[0044]

Number

[0045] In Equations (5) and (6), "W ·,i " is the feature amount of the i-th channel of the weather feature model 70 (i.e., the weather feature model W). m i is defined by the following Equation (7), so m j,i , that is, the matrix m, is defined by the following Equation (8).

[0046]

Number

[0047]

Number

[0048] As shown in Equation (8), the matrix m indicates the correlation between the h feature amounts of the weather feature model 70 and the c feature amounts of the feature map 50. That is, Equation (6) represents the correlation matrix m as the similarity between the j-th feature of the feature map 50 (i.e., the feature map F) of the image data of the object to be detected and the weather feature model 70 (i.e., the weather feature model W) obtained by synthesizing the central feature vectors of each type of all weather iIt will be an equation for calculation. By calculating this similarity for each of the c channels of the feature map 50, the correlation matrix m j,i will be obtained. This correlation matrix m j,i is used as the weight of the weather feature indicating the feature of the weather, and the weighted sum of the weather features is calculated by Equation (5), and WF j indicating the j-th feature of the feature map 50 affected by the weather is calculated. Since the number of channels of the feature map 50 is c, the combining unit 15 will calculate c WF j .

[0049] The combining unit 15 uses the feature map 50 (i.e., the feature map F) and the calculated c weather feature parts WF j (j = 1 to c) as the data of the three-dimensional array superposed in the channel direction, and performs the residual calculation shown on the right side of the following Equation (9) for each of the c channels, removes the influence of the weather from the feature map 50, and generates the feature map 51 shown on the left side.

[0050]

Equation

[0051] The combining unit 15 outputs the generated feature map 51 to the object detection unit 16 (step Sb2). Note that the above procedure when h ≠ c can also be applied when h = c.

[0052] The object detection unit 16 takes in the feature map 50 output by the feature map generation unit 14 and the feature map 51 output by the combining unit 15, and performs the following object detection process. The RPN layer 22 takes in the feature map 50 and detects the position where the object exists. The RoI - Align layer 31 takes in the feature map 51 and the output of the RPN layer 22, and extracts the part where the object exists in the feature map 51 based on the output of the RPN layer 22 that is taken in. The RoI - Align layer 31 performs Pooling processing on the data of the extracted part, and outputs the data indicating the result of the Pooling processing to the fully - connected layer 32 and the CNN layer 34.

[0053] When the fully-connected layer 32 captures the data output by the RoI-Align layer 31, the fully-connected layer 33-1 outputs data indicating the class of the object included in the region indicated by the captured data, and the fully-connected layer 33-2 outputs data indicating the position and range where the object included in the region indicated by the captured data exists. When the CNN layer 34 captures the data output by the RoI-Align layer 31, the CNN layer 35 outputs data indicating the type of mask to be applied to the part of the object included in the region indicated by the captured data (step Sb3).

[0054] By applying the data indicating the class of the object, the data indicating the position and range where the object exists, and the data indicating the type of mask to be applied to the object to the image data read from the image data storage unit 14 by the feature map generation unit 14, for example, each object displayed in the image data is surrounded by a bounding box, and the data indicating the class of the object associated with the bounding box is displayed. Furthermore, the parts of each object are masked with different colors.

[0055] In the image object detection device 1 of the first embodiment described above, the weather feature model generation unit 11 generates a weather feature model 70 indicating the features of the weather displayed in the image. The feature map generation unit 14 generates a feature map 50 from the image data of the object to be detected. The combining unit 15 performs a combination that causally intervenes the weather feature model 70 generated by the weather feature model generation unit 11 on the feature map 50 generated by the feature map generation unit 14, thereby generating a feature map 51 with the influence of the weather suppressed. The object detection unit 16 performs object detection from the feature map 51 generated by the combining unit 15. The weather feature model 70 generated by the weather feature model generation unit 11 is a model that collectively represents the information specific to the weather included in the feature amounts extracted from the images taken in various weather states, rather than for each type of weather. In other words, it can be said to be a general-purpose model that comprehensively represents various types of weather. By having the combining unit 15 perform a combination that causally intervenes such a weather feature model 70 on the feature map 50 of the image data of the object to be detected, a feature map 51 with the influence of the weather suppressed can be obtained. Therefore, the object detection unit 16 performs object detection processing on the feature map 51 with the influence of the weather suppressed, so that even if the image data of the object to be detected is image data taken in a non-optimal weather state, the object can be detected with the same accuracy as in the case of an optimal weather state. Since the weather feature model 70 is a general-purpose model for various types of weather, even if the image data of the object to be detected is damaged by the influence of any type of weather, regardless of the content and degree of the damage, the influence of any type of weather can be suppressed. Therefore, by applying the weather feature model 70, it is possible to improve the robustness of object detection.

[0056] In the first embodiment described above, the classification unit 43 performs clustering by a mixture Gaussian model. However, for example, clustering may be performed by another clustering method such as the K-means method.

[0057] In the above-described first embodiment, five types of weather, namely, "rain", "fog", "snow", "cloudy", and "lightly cloudy", are shown as examples of the types of weather of the image data stored in the weather image data storage unit 41. However, it is sufficient that at least two types of image data of different weather types are stored in the weather image data storage unit 41. Image data taken in weather conditions other than "rain", "fog", "snow", "cloudy", and "lightly cloudy" may be stored in the weather image data storage unit 41. In this case, the detection unit 44 detects each of the clusters generated by the clustering performed by the classification unit 43, that is, the central feature vectors 54-1, 54-2, 54-3, 54-4, 54-5,... corresponding to each of the weather types.

[0058] (Second Embodiment) FIG. 7 is a block diagram showing the configuration of the image object detection device 1a according to the second embodiment. In the second embodiment, the same components as those in the first embodiment are denoted by the same reference numerals, and different components will be described below. The image object detection device 1a includes a weather feature model generation unit 11a, a weather feature model storage unit 12, an image data storage unit 13, a feature map generation unit 14, a combining unit 15, and an object detection unit 16.

[0059] The weather feature model generation unit 11 of the first embodiment generated the weather feature model 70 from various types of weather image data, that is, rain image data 52-1, fog image data 52-2, snow image data 52-3, cloudy image data 52-4, lightly cloudy image data 52-5,.... By the way, when shooting is performed in weather conditions such as rain or fog, it is observed that in the image obtained by shooting, the influence of the weather is uniformly distributed among the pixels of the image. Based on this, instead of generating the weather feature model 70 from the actually shot weather image data, the weather feature model generation unit 11a of the second embodiment generates an image of random noise in which noise appears uniformly in each pixel, and generates a weather feature model from the generated random noise image. Here, for convenience of explanation, the weather feature model in the second embodiment is denoted by reference numeral 70a and hereinafter referred to as the weather feature model 70a.

[0060] (Processing by the Weather Feature Model Generation Unit of the Second Embodiment) FIG. 8 is a flowchart showing the processing flow by the weather feature model generation unit 11a of the second embodiment. The weather feature model generation unit 11a generates two-dimensional image data of random noise having the same size as the data of the two-dimensional array of each channel of the feature map 50, that is, "c" pieces, for the number of channels of the feature map 50 (step Sc1). The weather feature model generation unit 11a generates data of a three-dimensional array in which the generated two-dimensional image data of "c" pieces of random noise are superimposed in the channel direction as the data of the weather feature model 70a (step Sc2).

[0061] Note that the random noise needs to be uniformly distributed in one two-dimensional image. Therefore, the weather feature model generation unit 11a generates two-dimensional image data such that the pixel value of each pixel of the two-dimensional image data is a random number of a normal distribution. As a specific example of the random noise, for example, Gaussian noise which is irregular noise of a normal distribution can be applied, and in particular, it is desirable to use white Gaussian noise. The weather feature model 70a is defined by the following equation (10).

[0062] [Number]

[0063] As can be seen from Equation (10), both the weather feature model 70 of the first embodiment and the weather feature model 70a of the second embodiment are elements of a d-dimensional real number space having the same number of dimensions, but are data of a three-dimensional array having different numbers of channels. The weather feature model 70a of the second embodiment is data of a three-dimensional array represented as an element of a d×c-dimensional real number space having the same size and the same number of channels as the feature map 50. The weather feature model generation unit 11a writes the generated data of the weather feature model 70a into the weather feature model storage unit 12 and stores it (step Sc3).

[0064] In the second embodiment, the object detection process is the same as the object detection process of the first embodiment shown in FIG. 5. However, for the feature map 50 generated by the feature map generation unit 14, the weather feature model for which the combining unit 15 performs causal intervention is the weather feature model 70a of the second embodiment. As a result, also in the second embodiment, as in the first embodiment, it becomes possible to perform object detection after suppressing the influence of the weather in the feature map 50 obtained from the image data of the object to be detected.

[0065] In the above-described second embodiment, the weather feature model generation unit 11a may generate h random noises having a number different from c, and generate the weather feature model 70a from the generated h random noises. In this case, the combining unit 15 generates the feature map 51 from the feature map 50 and the weather feature model 70a according to the procedure in the case where h≠c described in the first embodiment.

[0066] In the above-described first and second embodiments, as a specific example of the object detection unit 16, an example in which Mask R-CNN is applied is shown. On the other hand, as the object detection unit 16, for example, other object detection methods using deep neural networks such as Faster R-CNN and YOLO (You Only Look Once) may be applied. Even when applying other object detection methods, as in the example where Mask R-CNN is applied, the combining unit 15 is inserted at a position where the feature map corresponding to the feature map 50 is obtained as an output, and the weather feature models 70 and 70a are combined with the feature map 50.

[0067] In the above-described first and second embodiments, it is assumed that the image data stored in the weather image data storage unit 41 and the image data storage unit 13 is RGB color image data. On the other hand, the image data stored in the weather image data storage unit 41 and the image data storage unit 13 may be CMYK color image data or grayscale image data. The image data stored in the weather image data storage unit 41 and the image data storage unit 13 may be image data captured by the same camera or image data captured by different cameras. However, it is necessary to capture the image data so that the domains of both image data are the same so that there is no domain difference such that one is RGB color image data and the other is grayscale image data.

[0068] In the above-described first and second embodiments, the number of vertical and horizontal pixels of the image data stored in the image data storage unit 13 may be the same or different. In the above-described first embodiment, the sizes of the image data stored in the weather image data storage unit 41 and the image data storage unit 13 may be the same or different. The number of vertical and horizontal pixels of the image data stored in the weather image data storage unit 41 may be the same or different. However, as described above, it is necessary that the size of the two-dimensional array data for each channel of the feature map 50 matches the size of the two-dimensional array data for each channel of the weather feature models 70, 70a. Therefore, it is necessary to configure the CNN layer 21 included in the feature map generation unit 14 and the weather feature model generation units 11, 11a so that the sizes of the two-dimensional array data for each channel of the feature map 50 and the weather feature models 70, 70a match.

[0069] The image object detection devices 1 and 1a in the above-described embodiments may be implemented by a computer. In that case, a program for realizing this function may be recorded on a computer-readable recording medium, and the program recorded on this recording medium may be read into a computer system and executed to realize it. Here, the “computer system” shall include hardware such as an OS and peripheral devices. Further, the “computer-readable recording medium” refers to a portable medium such as a flexible disk, a magneto-optical disk, a ROM, a CD-ROM, etc., and a storage device such as a hard disk built in a computer system. Furthermore, the “computer-readable recording medium” also includes, like a communication line when transmitting a program via a network such as the Internet or a communication line such as a telephone line, a medium that dynamically holds a program for a short period of time, and a volatile memory inside a computer system that serves as a server or a client in that case, which holds a program for a certain period of time. Also, the above program may be for realizing a part of the aforementioned functions, and may further be realizable in combination with a program already recorded in the computer system for the aforementioned functions, or may be realized using a programmable logic device such as an FPGA (Field Programmable Gate Array).

[0070] As described above, the embodiments of the present invention have been described in detail with reference to the drawings. However, the specific configuration is not limited to this embodiment, and designs and the like within the scope not departing from the gist of the present invention are also included.

Explanation of Reference Numerals

[0071] 1…Image object detection device, 11…Weather feature model generation unit, 12…Weather feature model storage unit, 13…Image data storage unit, 14…Feature map generation unit, 15…Combination unit, 16…Object detection unit

Claims

1. A weather feature model generation unit that generates a weather feature model indicating the features of the weather displayed in the image; A feature map generation unit that generates a feature map from the image data of the object to be detected; A combining unit that performs a combination that causally intervenes the weather feature model on the feature map generated by the feature map generation unit, thereby generating a feature map with the influence of the weather suppressed; An object detection unit that performs object detection from the feature map generated by the combining unit; An image object detection apparatus comprising the above.

2. The weather feature model generation unit: Generates a feature map from each of a plurality of image data in which the weather states corresponding to each of a plurality of weather types are photographed, clusters the generated feature maps, and synthesizes the central feature vectors of each of the clusters obtained by the clustering to generate the weather feature model. The image object detection apparatus according to claim 1, comprising the above.

3. The weather feature model generation unit: Generates the weather feature model from an image of random noise. The image object detection apparatus according to claim 1, comprising the above.

4. The random noise is white Gaussian noise. The image object detection apparatus according to claim 3.

5. The combining unit: Performs a combination that causally intervenes the weather feature model on the feature map by calculating the Hadamard product of the data obtained by applying the softmax function to the product of the transpose of the feature map and the weather feature model and the feature map. The image object detection apparatus according to any one of claims 1 to 4.

6. The combining unit: Calculates the correlation matrix between each of the channels of the feature map and each of the channels of the weather feature model, extracts the part including the weather feature for each channel of the feature map based on the calculated correlation matrix and the weather feature model, and performs a residual calculation to remove the weather feature from the feature map by using the extracted part including the weather feature and the feature map, thereby performing a combination that causally intervenes the weather feature model on the feature map. The image object detection apparatus according to any one of claims 1 to 4.

7. A weather feature model generation step of generating a weather feature model indicating the features of the weather displayed in the image; A feature map generation step of generating a feature map from the image data of the object to be detected; A combining step of generating a feature map with the influence of weather suppressed by performing a coupling that causally intervenes the weather feature model generated by the weather feature model generation step on the feature map generated by the feature map generation step; An object detection step of performing object detection from the feature map generated by the combining step; An image object detection method comprising the above.

8. A computer, Weather feature model generation means for generating a weather feature model indicating the features of the weather displayed in the image, Feature map generation means for generating a feature map from the image data of the object to be detected, Combining means for generating a feature map with the influence of weather suppressed by performing a coupling that causally intervenes the weather feature model generated by the weather feature model generation means on the feature map generated by the feature map generation means, Object detection means for performing object detection from the feature map generated by the combining means, A program for causing the computer to function as the above.

Images