Scene determination device, correlation coefficient calculation device, and scene determination method

Abstract

A scene determination device (1) comprises: a feature amount extraction unit (11) that extracts typical image feature amounts from a typical image group in which a scene to be determined appears; a correlation coefficient calculation unit (12) that calculates, from the typical image feature amounts, a correlation coefficient which indicates a correlation between the typical image feature amounts; a feature amount extraction unit (21) that extracts an input image feature amount from an input image; a distance calculation unit (22) that calculates, on the basis of the input image feature amount and the correlation coefficient, determination distance data indicating a difference between the feature of the input image and the feature of the typical image group; and a scene determination unit (23) that determines, on the basis of the determination distance data, a scene which appears in the input image.

Description

Scene discrimination device, correlation coefficient calculation device, and scene discrimination method

[0001] The present disclosure relates to a scene discrimination device, a correlation coefficient calculation device, and a scene discrimination method.

[0002] The popularity of electric vehicles is progressing, and the electrification of in-vehicle devices is accelerating. For example, as a requirement for electronic mirrors, there is a desire to improve visibility in bad weather such as fog or snow. In order to improve visibility, it is necessary to determine whether the scene depicted in the image is a bad weather scene such as fog or snow, and when it is determined that these scenes are bad weather scenes, the visibility can be improved by correcting the image according to the scene.

[0003] Patent Documents 1 to 5 disclose techniques for discriminating the scene depicted in an image.

[0004] Japanese Patent Application Laid-Open No. 2010-273144 International Publication No. 2015 / 190184 Japanese Patent Application Laid-Open No. 2012-244538 International Publication No. 2017 / 047494 Japanese Patent Application Laid-Open No. 2016-40961

[0005] As methods for discriminating the scene depicted in an image, there are methods that use other information such as distance information, methods that manually analyze the features of the image, or methods that analyze the features of the image by machine learning. However, in the method that uses other information such as distance information, sensors such as LiDAR (Light Detection And Ranging) are required, resulting in high costs. Also, in the method of manually analyzing the features of the image, the discrimination accuracy decreases. Further, in the method of analyzing the features of the image by machine learning, a high-performance CPU (Central Processing Unit) is required, resulting in high costs.

[0006] Therefore, the present disclosure provides a scene discrimination device and the like that can discriminate the scene depicted in an image at low cost and with high accuracy.

[0007] The scene discrimination device according to this disclosure is a scene discrimination device for discriminating a scene in an input image, comprising: a typical image feature extraction unit for extracting typical image features from a group of typical images in which the scene to be discriminated is depicted; a correlation coefficient calculation unit for calculating a correlation coefficient showing the correlation between the typical image features from the typical image features; an input image feature extraction unit for extracting input image features from the input image; a distance calculation unit for calculating discrimination distance data showing the difference between the features of the input image and the features of the group of typical images based on the input image features and the correlation coefficient; and a scene discrimination unit for discriminating a scene in the input image based on the discrimination distance data.

[0008] The correlation coefficient calculation device according to this disclosure comprises a typical image feature extraction unit that extracts typical image features from a group of typical images in which the scene to be discriminated is depicted, and a correlation coefficient calculation unit that calculates a correlation coefficient between the parameters constituting the typical image features from the typical image features.

[0009] The scene discrimination device according to this disclosure is a scene discrimination device for discriminating a scene in an input image, comprising: an input image feature extraction unit for extracting input image feature quantities from the input image; a distance calculation unit for calculating discrimination distance data indicating the difference between the features of the input image and the features of the typical image group, based on the input image feature quantities and correlation coefficients between parameters constituting the typical image feature quantities, which are calculated in advance from the input image feature quantities and typical image feature quantities extracted from a group of typical images in which the scene to be discriminated is depicted; and a scene discrimination unit for discriminating a scene in the input image based on the discrimination distance data.

[0010] The scene discrimination method relating to this disclosure is a scene discrimination method for discriminating a scene in an input image, and includes: a typical image feature extraction step of extracting typical image features from a group of typical images in which the scene to be discriminated is depicted; a correlation coefficient calculation step of calculating a correlation coefficient showing the correlation between the typical image features from the typical image features; an input image feature extraction step of extracting input image features from the input image; a distance calculation step of calculating discrimination distance data showing the difference between the features of the input image and the features of the group of typical images based on the input image features and the correlation coefficient; and a scene discrimination step of discriminating a scene in the input image based on the discrimination distance data.

[0011] These comprehensive or specific embodiments may be implemented as a system, method, integrated circuit, computer program, or recording medium such as a computer-readable CD-ROM, or as any combination of a system, method, integrated circuit, computer program, and recording medium.

[0012] According to one aspect of this disclosure, a scene discrimination device can discriminate scenes in an image at low cost and with high accuracy.

[0013] This is a block diagram showing an example of a scene discrimination device according to an embodiment. This is a diagram showing an example of the distribution of each scene when there are two features. This is a diagram showing an example of the distribution of each scene when there are three features. This is a diagram for explaining the operation of the scene discrimination device according to an embodiment. This is a block diagram showing an example of a scene discrimination device and a correlation coefficient calculation device according to another embodiment. This is a flowchart showing an example of a scene discrimination method according to another embodiment.

[0014] The embodiments will be described in detail below with reference to the drawings.

[0015] The embodiments described below are all general or specific examples. The numerical values, shapes, materials, components, arrangement and connection configurations of components, steps, and the order of steps shown in the following embodiments are examples only and are not intended to limit this disclosure.

[0016] (Embodiment) The following describes a scene discrimination device according to an embodiment.

[0017] Figure 1 is a block diagram showing an example of a scene discrimination device 1 according to an embodiment. In addition to the scene discrimination device 1, Figure 1 also shows a camera 100 and an output device 200.

[0018] Camera 100 captures an image (referred to as the input image) that is input to the scene discrimination device 1. Camera 100 is mounted on a vehicle such as an electric vehicle and acquires an image of the area around the vehicle, which it outputs to the scene discrimination device 1. Scene discrimination device 1 is a device that discriminates the scene captured in the input image acquired from camera 100. Scenes captured in the input image are, for example, scenes of bad weather such as fog or snow. Scene discrimination device 1 outputs the discrimination result to output device 200. Output device 200 is a display device mounted on a vehicle such as an electric vehicle, for example, an electronic mirror. Output device 200 corrects the input image based on the discrimination result of scene discrimination device 1, and displays an input image with improved visibility.

[0019] The scene discrimination device 1 comprises a feature extraction unit 11, a correlation coefficient calculation unit 12, a feature extraction unit 21, a distance calculation unit 22, and a scene discrimination unit 23. The scene discrimination device 1 is a computer including a processor (microprocessor) and memory. The memory is ROM (Read Only Memory) and RAM (Random Access Memory), and can store programs executed by the processor. The feature extraction unit 11, the correlation coefficient calculation unit 12, the feature extraction unit 21, the distance calculation unit 22, and the scene discrimination unit 23 are implemented by a processor that executes programs stored in memory.

[0020] The feature extraction unit 11 extracts typical image features from a group of typical images depicting the scene to be classified. The feature extraction unit 11 is an example of a typical image feature extraction unit. The group of typical images is a pre-prepared group of images depicting the scene to be classified. For example, the extracted typical image features may be a luminance histogram, luminance standard deviation, or luminance gradient. Two or more types of typical image features may also be extracted. For example, two types of features, such as a luminance histogram and luminance standard deviation, may be extracted; two types of features, such as a luminance histogram and luminance gradient, may be extracted; two types of features, such as luminance standard deviation and luminance gradient, may be extracted; or three types of features, such as a luminance histogram, luminance standard deviation, and luminance gradient, may be extracted. If there are multiple scenes to be classified, the feature extraction unit 11 extracts typical image features for each of the multiple scenes to be classified. In this case as well, two or more types of typical image features may be extracted for each of the multiple scenes to be classified.

[0021] The correlation coefficient calculation unit 12 calculates a correlation coefficient that shows the correlation between typical image features by performing a correlation analysis between the parameters that constitute the typical image features from the typical image features. The parameters that constitute the typical image features are, for example, the standard deviation of the luminance gradient and the luminance standard deviation. If there are multiple scenes to be classified, the correlation coefficient calculation unit 12 calculates a correlation coefficient for each of the multiple scenes to be classified.

[0022] The Mahalanobis-Taguchi (MT) method does not require complex calculations, allows for easy addition of parameters, and enables automatic analysis of trends in the training data. Therefore, as will be described later, the MT method may be applied to scene classification, for example. In this case, the correlation coefficient calculation unit 12 may generate a formula for calculating the Mahalanobis distance. For example, the Mahalanobis distance is an indicator that shows how far each point in the dataset is from the center.

[0023] The feature extraction unit 21 extracts input image features from the input image. The feature extraction unit 21 is an example of an input image feature extraction unit. The extracted input image features are features that correspond to typical image features. For example, if a luminance histogram is extracted as a typical image feature, a luminance histogram will be extracted as an input image feature. Also, if two or more types of typical image features are extracted, two or more types of input image features will be extracted. If there are multiple scenes to be classified, the feature extraction unit 21 extracts input image features for each of the multiple scenes to be classified.

[0024] The distance calculation unit 22 calculates discrimination distance data, which indicates the difference between the features of the input image and the features of the typical image group, based on the input image features and the correlation coefficient. For example, the discrimination distance data is the Mahalanobis distance, and the distance calculation unit 22 calculates the Mahalanobis distance from the input image features using the Mahalanobis distance calculation formula. If there are multiple scenes to be discriminated against, the distance calculation unit 22 calculates discrimination distance data for each of the multiple scenes to be discriminated against.

[0025] The scene discrimination unit 23 discriminates the scenes in the input image based on the discrimination distance data. For example, the scene discrimination unit 23 discriminates the scenes in the input image by comparing the calculated Mahalanobis distance with a threshold. By applying the MT method to scene discrimination, the scenes in the image can be discriminated at low cost and with high accuracy.

[0026] For example, the scene discrimination unit 23 may differentiate between scenes in the input image in stages by comparing discrimination distance data with multiple thresholds. For instance, by classifying a group of typical images showing foggy scenes according to the density of the fog and calculating a group of correlation coefficients for each density of fog, it is possible to determine not only whether the scene in the input image is a foggy scene, but also to detect the density of the fog.

[0027] If there are multiple scenes to be classified, the scene classification unit 23 classifies the scene corresponding to the smallest classification distance data among the multiple classification distance data calculated as the scene in the input image. This allows the system to classify the scene in the input image even when there are multiple scenes to be classified, eliminating the need to manually divide the multidimensional space into feature regions for each of the multiple scenes to be classified. The process of dividing the multidimensional space into feature regions will now be explained using Figures 2A and 2B.

[0028] Figure 2A shows an example of the distribution of each scene when there are two features.

[0029] Figure 2B shows an example of the distribution of each scene when there are three features.

[0030] Conventional discrimination methods analyze the distribution of two features (features A and B) for each scene, as shown in Figure 2A, to determine the distribution ranges for foggy scenes and sunny scenes. In this case, if the results of features A and B in the input image fall within the distribution range of a foggy scene, the scene in the input image can be determined to be a foggy scene. However, as shown in Figure 2B, when there are three features (features A, B, and C), it is necessary to determine the distribution range for each scene in three-dimensional space, and manually determining the distribution range is difficult. Although not shown in the figure, when there are four or more features, manually determining the distribution range becomes even more difficult. In contrast, the discrimination method disclosed herein does not require manually dividing the region in multi-dimensional space by feature for each of the multiple scenes to be discriminated against.

[0031] Next, the operation of the scene discrimination device 1 will be explained using Figure 3.

[0032] Figure 3 is a diagram illustrating the operation of the scene discrimination device 1 according to the embodiment.

[0033] For example, a large number of sample images depicting bad weather scenes are prepared as typical image sets, and a large number of sample images depicting foggy scenes are prepared as examples of bad weather scenes.

[0034] The feature extraction unit 11 extracts, for example, three types of typical image features (image feature A, image feature B, and image feature C) from a group of typical images depicting foggy scenes. Note that three types is just an example; there may be one, two, or four types. Furthermore, the scene discrimination device 1 makes it easy to add features to the scenes or images to be discriminated against. For example, as shown in Figure 3, a snow scene can be added as a scene to be discriminated against, and image feature N can be added as a typical image feature to be extracted.

[0035] The correlation coefficient calculation unit 12 analyzes the trends for each scene. Since the analysis of image features is automated in the correlation coefficient calculation unit 12, highly accurate analysis without subjectivity is possible.

[0036] The scene discrimination unit 23 can easily distinguish scenes simply by using the trends for each scene (specifically, the correlation coefficient for each scene). Because the computational load required to distinguish scenes is small, a high-performance CPU is not necessary. Furthermore, no separate device is required to distinguish scenes.

[0037] As explained above, for example, when manually deriving a discriminant formula or threshold for identifying scenes in an input image, it is necessary to analyze data equal to (type of feature) × (number of typical images) × (number of scenes to be identified) based on human subjectivity, and the identification accuracy decreases as the amount of data increases. In contrast, with the scene identification device 1, the analysis of features of typical images is performed automatically without relying on human subjectivity, and the correlation coefficient is calculated, so the decrease in identification accuracy can be suppressed even if the amount of data increases. Furthermore, since scenes in the input image can be identified simply by calculating the identification distance data based on the input image features and the correlation coefficient, a separate device and a high-performance CPU are not required, thus reducing costs. Therefore, scene identification in an image can be performed at low cost and with high accuracy. In addition, it is easy to add features to the scenes or images to be identified. For example, if features to be identified in a scene or image are added, it is only necessary to recalculate the correlation coefficient automatically.

[0038] The feature extraction unit 21 may extract input image features from each of the multiple partial images obtained by dividing the input image into multiple parts, the distance calculation unit 22 may calculate discrimination distance data for each of the multiple partial images, and the scene discrimination unit 23 may discriminate a scene for each of the multiple partial images.

[0039] For example, when the scene discrimination device 1 is applied to an in-vehicle camera 100, the mounting position of the in-vehicle camera is fixed, so there is a certain tendency in the distance to the subject in the input image. For example, there is a tendency for distant objects to be captured in the central part of the input image, and the sky to be captured in the upper part. More specifically, since fog has the characteristic that the further away it is, the central and upper parts of the input image are farther away from the camera 100 and are strongly affected by fog, while the left, right, and lower parts of the input image are closer to the camera and are less affected by fog. Therefore, by dividing the input image into multiple sub-images and setting thresholds for each of the sub-images according to this tendency, it is possible to discriminate a scene for each of the sub-images. In other words, if there is some tendency for each sub-image, the accuracy of scene discrimination can be improved by extracting features from each sub-image.

[0040] For example, the scene determination unit 23 may determine that the scene that appears most frequently among the scenes of each of the multiple determined partial images is the scene that appears in the input image. In this way, the scene can be determined by majority vote of the scene determination results of each of the multiple partial images. For example, even if the effect of fog on a part of the image is weak, the accuracy of scene determination can be improved.

[0041] For example, the scene discrimination unit 23 may determine a partial image corresponding to a discrimination distance data whose variation from the average value is greater than or equal to a predetermined value among the multiple discrimination distance data calculated as an excluded image, and then discriminate a scene for each of the multiple excluded partial images. If there are abnormal values ​​(for example, areas with dense fog) in only a part of the input image, such abnormal values ​​are excluded and the scene is determined, thereby improving the accuracy of scene discrimination.

[0042] For example, the scene discrimination unit 23 may perform weighting according to the position of the partial image corresponding to each of the calculated plurality of discrimination distance data. By setting weights according to the tendency of the partial images, the discrimination accuracy of the scene can be improved. For example, the weights are increased for portions (e.g., the central portion and the upper portion) where the influence of fog in the input image tends to be strong, and the weights are decreased for portions (e.g., the left and right portions and the lower portion) where the influence of fog tends to be weak, thereby improving the discrimination accuracy of the scene.

[0043] For example, the scene discrimination unit 23 may detect the vanishing point of the input image and determine the positions of a plurality of partial images according to the vanishing point. When the mounting position or posture of the camera 100 changes, the positions of the central portion, the upper portion, the left and right portions, and the lower portion in the input image will deviate from their original positions, and there is a possibility that a small weight is assigned to a portion where a large weight should be assigned, or a large weight is assigned to a portion where a small weight should be assigned. Therefore, by detecting the vanishing point of the input image, it is possible to determine the portions where large weights should be assigned and the portions where small weights should be assigned. Accordingly, even if the mounting position or posture of the camera 100 changes, a decrease in the discrimination accuracy of the scene can be suppressed.

[0044] (Other Embodiments) As described above, embodiments have been described as examples of the technology according to the present disclosure. However, the technology according to the present disclosure is not limited thereto, and is also applicable to embodiments in which appropriate changes, replacements, additions, omissions, etc. are made. For example, the following modifications are also included in one embodiment of the present disclosure.

[0045] For example, in the above embodiment, an example has been described in which the scene discrimination device 1 also has a function of calculating a correlation coefficient in addition to the function of discriminating a scene. However, the present disclosure is not limited to this. For example, the scene discrimination device 1 may not have a function of calculating a correlation coefficient, and the device for discriminating a scene and the device for calculating a correlation coefficient may be provided separately. This will be described using FIG. 4.

[0046] FIG. 4 is a block diagram showing an example of a scene discrimination device 20 and a correlation coefficient calculation device 10 according to another embodiment.

[0047] The correlation coefficient calculation device 10 is a device that calculates a correlation coefficient. Specifically, the correlation coefficient calculation device 10 includes a feature amount extraction unit 11 (typical image feature amount extraction unit) that extracts typical image feature amounts from a group of typical images in which the scene to be discriminated appears, and a correlation coefficient calculation unit 12 that calculates a correlation coefficient indicating the correlation between the typical image feature amounts from the typical image feature amounts.

[0048] The scene discrimination device 20 is a scene discrimination device that discriminates a scene appearing in an input image. Specifically, the scene discrimination device 20 includes a feature amount extraction unit 21 (input image feature amount extraction unit) that extracts input image feature amounts from the input image, and a distance calculation unit 22 that calculates discrimination distance data indicating the difference between the features of the input image and the features of the group of typical images based on the input image feature amounts and the correlation coefficient between the parameters constituting the typical image feature amounts calculated in advance from the typical image feature amounts extracted from the group of typical images in which the scene to be discriminated appears, and a scene discrimination unit 23 that discriminates the scene appearing in the input image based on the discrimination distance data.

[0049] For example, as preprocessing, a correlation coefficient is calculated in advance by a correlation coefficient calculation device 10 such as a PC (Personal Computer). That is, a processing unit that requires a large amount of calculation for calculating the correlation coefficient does not have to be mounted on a system such as a vehicle (for example, the scene discrimination device 20), and it is only necessary to apply the correlation coefficient calculated in the preprocessing to the scene discrimination device 20. Thereby, the cost of the scene discrimination device 20 can be reduced.

[0050] For example, the present disclosure can be realized not only as the scene discrimination device 1 but also as a scene discrimination method including steps (processes) performed by components constituting the scene discrimination device 1.

[0051] FIG. 5 is a flowchart showing an example of a scene discrimination method according to another embodiment.

[0052] The scene discrimination method is a method for discriminating a scene in an input image, and as shown in Figure 5, includes: a typical image feature extraction step (step S11) for extracting typical image features from a group of typical images in which the scene to be discriminated is depicted; a correlation coefficient calculation step (step S12) for calculating a correlation coefficient showing the correlation between typical image features from the typical image features; an input image feature extraction step (step S13) for extracting input image features from the input image; a distance calculation step (step S14) for calculating discrimination distance data showing the difference between the features of the input image and the features of the group of typical images based on the input image features and the correlation coefficient; and a scene discrimination step (step S15) for discriminating a scene in the input image based on the discrimination distance data.

[0053] This provides a scene discrimination method that can discriminate scenes in an image at low cost and with high accuracy. For example, the computationally intensive typical image feature extraction step and correlation coefficient calculation step can be performed on a PC (e.g., correlation coefficient calculation device 10), while the computationally intensive distance calculation step and scene discrimination step can be performed on a scene discrimination system (e.g., scene discrimination device 20), thereby reducing the cost of the scene discrimination system.

[0054] For example, this disclosure can be implemented as a program that causes a computer (processor) to execute the steps included in the scene determination method. Furthermore, this disclosure can be implemented as a non-temporary computer-readable recording medium, such as a CD-ROM, on which the program is recorded.

[0055] For example, if this disclosure is implemented in a program (software), each step is executed by the program using hardware resources such as the computer's CPU, memory, and input / output circuits. In other words, each step is executed by the CPU obtaining data from memory or input / output circuits, performing calculations, and outputting the calculation results to memory or input / output circuits.

[0056] In the above embodiment, each component included in the scene discrimination devices 1 and 20 and the correlation coefficient calculation device 10 may be implemented by dedicated hardware or by executing a software program suitable for each component. Each component may also be implemented by a program execution unit such as a CPU or processor reading and executing a software program recorded on a recording medium such as a hard disk or semiconductor memory.

[0057] Some or all of the functions of the scene discrimination devices 1 and 20 and the correlation coefficient calculation device 10 according to the above embodiment are typically implemented as an integrated circuit (LSI). These may be individually integrated on a single chip, or some or all of them may be integrated on a single chip. Furthermore, the implementation is not limited to an LSI, but may also be implemented using a dedicated circuit or a general-purpose processor. An FPGA (Field Programmable Gate Array) that can be programmed after LSI manufacturing, or a reconfigurable processor that can reconfigure the connections and settings of circuit cells inside the LSI may also be used.

[0058] Furthermore, if advances in semiconductor technology or other derived technologies lead to the emergence of integrated circuit technology that replaces LSIs, then naturally, that technology may be used to integrate each component included in the scene discrimination devices 1 and 20 and the correlation coefficient calculation device 10 into integrated circuits.

[0059] Furthermore, this disclosure also includes forms obtained by applying various modifications to the embodiments that a person skilled in the art could conceive, and forms realized by arbitrarily combining the components and functions of each embodiment without departing from the spirit of this disclosure.

[0060] (Note) The above description of embodiments discloses the following technology.

[0061] (Technical 1) A scene discrimination device for discriminating a scene in an input image, comprising: a typical image feature extraction unit for extracting typical image features from a group of typical images in which the scene to be discriminated is depicted; a correlation coefficient calculation unit for calculating a correlation coefficient showing the correlation between the typical image features from the typical image features; an input image feature extraction unit for extracting input image features from the input image; a distance calculation unit for calculating discrimination distance data showing the difference between the features of the input image and the features of the group of typical images based on the input image features and the correlation coefficient; and a scene discrimination unit for discriminating a scene in the input image based on the discrimination distance data.

[0062] For example, when manually deriving a discriminant formula or threshold for identifying scenes in an input image, it is necessary to analyze data equal to (type of feature) × (number of typical images) × (number of scenes to be identified) based on human subjectivity, and the identification accuracy decreases as the amount of data increases. In contrast, with the scene identification device of this disclosure, the analysis of features of typical images is performed automatically without relying on human subjectivity, and the correlation coefficient is calculated, so the decrease in identification accuracy can be suppressed even with a large amount of data. Furthermore, since scenes in an input image can be identified simply by calculating the identification distance data based on the input image features and the correlation coefficient, a separate device and a high-performance CPU are not required, thus reducing costs. Therefore, scene identification in an image can be performed at low cost and with high accuracy. In addition, it is easy to add features to the scenes or images to be identified. For example, if features to be identified in a scene or image are added, it is only necessary to recalculate the correlation coefficient automatically.

[0063] (Technology 2) The scene discrimination device according to Technology 1, wherein the scene discrimination unit compares the discrimination distance data with a plurality of thresholds to determine the scenes depicted in the input image in stages.

[0064] According to this method, for example, by classifying a typical set of images depicting foggy scenes according to the density of the fog and calculating a set of correlation coefficients for each level of fog density, it is possible not only to determine whether or not the scene in the input image is a foggy scene, but also to detect the density of the fog.

[0065] (Technical 3) A scene discrimination device according to Technical 1 or 2, wherein the typical image feature extraction unit extracts typical image features for each of the multiple scenes to be discriminated, the correlation coefficient calculation unit calculates the correlation coefficient for each of the multiple scenes to be discriminated, the input image feature extraction unit extracts the input image features for each of the multiple scenes to be discriminated, the distance calculation unit calculates the discrimination distance data for each of the multiple scenes to be discriminated, and the scene discrimination unit discriminates the scene corresponding to the smallest of the calculated discrimination distance data to be the scene captured in the input image.

[0066] According to this method, even if there are multiple scenes to be identified, the scenes captured in the input image can be determined. For example, there is no need to manually divide the region in the multidimensional space according to the features for each of the multiple scenes to be identified.

[0067] (Technical 4) The scene discrimination device according to any one of Technical 1 to 3, wherein the input image feature extraction unit extracts the input image feature for each of the multiple partial images obtained by dividing the input image into multiple parts, the distance calculation unit calculates the discrimination distance data for each of the multiple partial images, and the scene discrimination unit discriminates a scene for each of the multiple partial images.

[0068] According to this, for example, when the scene discrimination device of this disclosure is applied to an in-vehicle camera, since the mounting position of the in-vehicle camera is fixed, there is a certain tendency in the distance to the subject in the input image. For example, there is a tendency for distant objects to be captured in the central part of the input image, and the sky to be captured in the upper part. More specifically, the central and upper parts of the input image tend to be farther from the camera and strongly affected by fog, while the left, right, and lower parts of the input image tend to be closer to the camera and less affected by fog. Therefore, by dividing the input image into multiple sub-images and setting thresholds for each of the sub-images according to this tendency, it is possible to discriminate a scene for each of the sub-images. In other words, if there is a certain tendency for each sub-image, the accuracy of scene discrimination can be improved by extracting features from each sub-image.

[0069] (Technical 5) The scene discrimination device according to Technical 4, wherein the scene discrimination unit determines that the most frequent scene among the scenes of each of the plurality of discriminated partial images is the scene depicted in the input image.

[0070] According to this method, a scene can be identified by a majority vote of the scene identification results for each of the multiple partial images.

[0071] (Technical 6) The scene discrimination device according to Technical 4 or 5, wherein the scene discrimination unit determines a partial image corresponding to a discrimination distance data whose variation from the average value is greater than or equal to a predetermined value among the calculated plurality of discrimination distance data as an excluded image, and the excluded image determines a scene for each of the plurality of excluded partial images.

[0072] According to this method, if there are outliers (for example, areas with dense fog) in only a part of the input image, such outliers are excluded and the scene is identified, thereby improving the accuracy of scene identification.

[0073] (Technical 7) The scene discrimination device according to any one of Technical 4 to 6, wherein the scene discrimination unit weights each of the calculated plurality of discrimination distance data according to the position of the partial image corresponding to the discrimination distance data.

[0074] According to this method, the accuracy of scene discrimination can be improved by assigning weights to sub-images according to their characteristics. For example, by increasing the weight of areas in the input image where the fog effect tends to be strong (e.g., the central and upper parts) and decreasing the weight of areas where the fog effect tends to be weaker (e.g., the left, right, and lower parts), the accuracy of scene discrimination can be improved.

[0075] (Technical 8) The scene discrimination device according to Technical 7, wherein the scene discrimination unit detects the vanishing point of the input image and determines the positions of the plurality of partial images according to the vanishing point.

[0076] According to this, if the camera's mounting position or orientation changes, the positions of the central, upper, left-right, and lower parts of the input image will shift from their original positions. This could result in areas that should be heavily weighted being given small weights, or areas that should be lightly weighted being given large weights. Therefore, by detecting the vanishing points of the input image, it is possible to determine which areas should be heavily weighted and which should be lightly weighted. Consequently, even if the camera's mounting position or orientation changes, a decrease in scene discrimination accuracy can be suppressed.

[0077] (Technical 9) The scene discrimination device according to any one of Technical 1 to 8, wherein the discrimination distance data is the Mahalanobis distance.

[0078] According to this, by applying the MT method to scene discrimination, it is possible to discriminate scenes in images at low cost and with high accuracy.

[0079] (Technical 10) A correlation coefficient calculation device comprising: a typical image feature extraction unit that extracts typical image features from a group of typical images in which the scene to be discriminated is depicted; and a correlation coefficient calculation unit that calculates a correlation coefficient between the parameters constituting the typical image features from the typical image features.

[0080] According to this method, the analysis of typical image features is performed automatically and independently of human subjectivity, allowing for the calculation of correlation coefficients. This suppresses the decline in discrimination accuracy even with a large amount of data. Furthermore, it facilitates the addition of features to the scenes and images to be discriminated against.

[0081] (Technical 11) A scene discrimination device for discriminating a scene in an input image, comprising: an input image feature extraction unit for extracting input image features from the input image; a distance calculation unit for calculating discrimination distance data indicating the difference between the features of the input image and the features of the typical image group, based on the input image features and correlation coefficients between parameters constituting the typical image features, which are calculated in advance from the input image features and typical image features extracted from a group of typical images in which the scene to be discriminated is depicted; and a scene discrimination unit for discriminating a scene in the input image based on the discrimination distance data.

[0082] According to this method, since the scene depicted in the input image can be identified simply by calculating discrimination distance data based on the input image features and pre-calculated correlation coefficients, a separate device and a high-performance CPU are not required, thus reducing costs.

[0083] (Technical 12) A scene discrimination method for discriminating a scene in an input image, comprising: a typical image feature extraction step of extracting typical image features from a group of typical images in which the scene to be discriminated is depicted; a correlation coefficient calculation step of calculating a correlation coefficient showing the correlation between the typical image features from the typical image features; an input image feature extraction step of extracting input image features from the input image; a distance calculation step of calculating discrimination distance data showing the difference between the features of the input image and the features of the group of typical images based on the input image features and the correlation coefficient; and a scene discrimination step of discriminating a scene in the input image based on the discrimination distance data.

[0084] This method provides a scene discrimination method that can discriminate scenes in an image at low cost and with high accuracy. For example, by performing the computationally intensive typical image feature extraction step and correlation coefficient calculation step on a PC, and the computationally intensive distance calculation step and scene discrimination step on a system that performs scene discrimination, the cost of the system that performs scene discrimination can be reduced.

[0085] This disclosure can be applied to systems that identify scenes in images, etc.

[0086] 1, 20 Scene discrimination device 10 Correlation coefficient calculation device 11, 21 Feature extraction unit 12 Correlation coefficient calculation unit 22 Distance calculation unit 23 Scene discrimination unit 100 Camera 200 Output device

Claims

1. A scene discrimination device for discriminating a scene in an input image, comprising: a typical image feature extraction unit for extracting typical image features from a group of typical images in which the scene to be discriminated is depicted; a correlation coefficient calculation unit for calculating a correlation coefficient showing the correlation between the typical image features from the typical image features; an input image feature extraction unit for extracting input image features from the input image; a distance calculation unit for calculating discrimination distance data showing the difference between the features of the input image and the features of the group of typical images based on the input image features and the correlation coefficient; and a scene discrimination unit for discriminating a scene in the input image based on the discrimination distance data.

2. The scene discrimination device according to claim 1, wherein the scene discrimination unit compares the discrimination distance data with a plurality of thresholds to determine the scenes depicted in the input image in stages.

3. The scene discrimination device according to claim 1 or 2, wherein the typical image feature extraction unit extracts the typical image features for each of the multiple scenes to be discriminated, the correlation coefficient calculation unit calculates the correlation coefficient for each of the multiple scenes to be discriminated, the input image feature extraction unit extracts the input image features for each of the multiple scenes to be discriminated, the distance calculation unit calculates the discrimination distance data for each of the multiple scenes to be discriminated, and the scene discrimination unit discriminates the scene corresponding to the smallest of the calculated discrimination distance data to be the scene captured in the input image.

4. The scene discrimination device according to any one of claims 1 to 3, wherein the input image feature extraction unit extracts the input image feature for each of the multiple partial images obtained by dividing the input image into multiple parts, the distance calculation unit calculates the discrimination distance data for each of the multiple partial images, and the scene discrimination unit discriminates a scene for each of the multiple partial images.

5. The scene determination device according to claim 4, wherein the scene determination unit determines that the most frequent scene among the scenes of each of the plurality of determined partial images is the scene depicted in the input image.

6. The scene discrimination device according to claim 4 or 5, wherein the scene discrimination unit determines a partial image corresponding to a discrimination distance data whose variation from the average value is greater than or equal to a predetermined value among the calculated plurality of discrimination distance data as an excluded image, and the excluded image determines a scene for each of the plurality of excluded partial images.

7. The scene discrimination device according to any one of claims 4 to 6, wherein the scene discrimination unit weights each of the calculated plurality of discrimination distance data according to the position of the partial image corresponding to the discrimination distance data.

8. The scene determination device according to claim 7, wherein the scene determination unit detects the vanishing point of the input image and determines the positions of the plurality of partial images according to the vanishing point.

9. The scene discrimination device according to any one of claims 1 to 8, wherein the discrimination distance data is the Mahalanobis distance.

10. A correlation coefficient calculation device comprising: a typical image feature extraction unit that extracts typical image features from a group of typical images in which the scene to be classified is depicted; and a correlation coefficient calculation unit that calculates correlation coefficients between parameters constituting the typical image features from the typical image features.

11. A scene discrimination device for discriminating a scene in an input image, comprising: an input image feature extraction unit for extracting input image features from the input image; a distance calculation unit for calculating discrimination distance data indicating the difference between the features of the input image and the features of the typical image group, based on the input image features and correlation coefficients between parameters constituting the typical image features, which are calculated in advance from the input image features and typical image features extracted from a group of typical images in which the scene to be discriminated is depicted; and a scene discrimination unit for discriminating a scene in the input image based on the discrimination distance data.

12. A scene discrimination method for discriminating a scene in an input image, comprising: a typical image feature extraction step of extracting typical image features from a group of typical images in which the scene to be discriminated is depicted; a correlation coefficient calculation step of calculating a correlation coefficient showing the correlation between the typical image features from the typical image features; an input image feature extraction step of extracting input image features from the input image; a distance calculation step of calculating discrimination distance data showing the difference between the features of the input image and the features of the group of typical images based on the input image features and the correlation coefficient; and a scene discrimination step of discriminating a scene in the input image based on the discrimination distance data.

Images