Road monitoring device, road monitoring method, and road monitoring program

Abstract

To provide a road monitoring device capable of precisely detecting an anomaly of a road from images of the area picked up including the road.SOLUTION: A road monitoring device 1 includes an anomaly determination unit 15. The anomaly determination unit 15 determines an anomaly of a road based on information showing features common to multiple past images picked up under different environmental conditions by an imaging device that picks up images of the area including the road and information of input images newly captured by the imaging device.SELECTED DRAWING: Figure 5

Description

[Technical field]

[0001] The present disclosure relates to a road monitoring device, a road monitoring method, and a road monitoring program for determining abnormalities in a road. [Background technology]

[0002] Conventionally, many surveillance cameras are installed on roads such as expressways, and the images captured by each surveillance camera are monitored by monitors from road management companies or local governments, who judge whether there are any abnormalities on the road from the images captured by the surveillance cameras. Therefore, the more surveillance cameras there are, the greater the burden on the monitors.

[0003] Therefore, it is conceivable to use a technology for determining abnormalities in an area where an image is captured by an imaging device for monitoring roads. For example, Patent Document 1 discloses a technology for generating a difference image between each background image, which is an image captured in the past by an imaging device capturing an area including a road, and an input image newly captured by the imaging device, and detecting an abnormal area in the input image based on the difference image.

[0004] In the technology described in Patent Document 1, a weighting coefficient according to the similarity between the environmental conditions when each background image was captured and the environmental conditions when the input image was captured is accumulated for each pixel of the difference image to generate multiple weighted difference images. In the technology described in Patent Document 1, a weighted difference result image is generated by accumulating difference values ​​at the same pixel position between multiple weighted difference images, and a position where the accumulated difference value is equal to or greater than a threshold value for each pixel of the weighted difference result image is determined to be an abnormal position. [Prior art documents] [Patent documents]

[0005] [Patent Document 1] JP 2009-225010 A Summary of the Invention [Problem to be solved by the invention]

[0006] However, with the technique described in Patent Document 1, it is difficult to accurately detect an abnormal region in an input image captured under environmental conditions that are low in similarity to the environmental conditions under which each background image was captured.

[0007] The present disclosure has been made in consideration of the above, and has an object to provide a road monitoring device that can accurately detect abnormalities in a road from an image capturing an area including a road. [Means for solving the problem]

[0008] In order to solve the above problems and achieve the object, the road monitoring device of the present disclosure comprises: An abnormality in a road is judged based on information showing a common feature in a plurality of past images captured under different environmental conditions by an imaging device that captures an area including a road, and information on an input image newly captured by the imaging device. Abnormality determination section a common feature amount calculation unit that calculates a common feature amount that represents a common feature in units of one or more pixels based on a plurality of past images as information indicating the common feature; and a determination target feature amount calculation unit that calculates a determination target feature amount that is a feature amount in units of one or more pixels in an input image as information of the input image. Equipped with. The common feature includes a first common feature indicating a feature of one or more pixel units of regions in the multiple past images, and a second common feature indicating a degree of association of one or more pixel units between the multiple past images. The abnormality determination unit is When determining whether or not there is an abnormality in the road based on the common feature calculated by the common feature calculation unit and the judgment target feature calculated by the judgment target feature calculation unit, a difference between the judgment target feature and the first common feature in one or more pixel units is adjusted by a second common feature in one or more pixel units. Detect abnormalities on the road. Effect of the Invention

[0009] According to the present disclosure, it is possible to provide an advantage that abnormalities in a road can be detected with high accuracy from an image in which an area including a road is captured. [Brief description of the drawings]

[0010] [Figure 1] FIG. 1 is a diagram showing an example of a configuration of a monitoring system including a road monitoring device according to a first embodiment. [Diagram 2] FIG. 1 is a diagram showing an example of an area imaged by the imaging device according to the first embodiment; [Diagram 3] FIG. 1 is a diagram for explaining an example of a common feature amount calculation process executed by the road monitoring device according to the first embodiment; [Figure 4] FIG. 1 is a diagram for explaining an example of a determination target feature amount calculation process executed by a road monitoring device according to a first embodiment; [Diagram 5] FIG. 1 is a diagram showing an example of a configuration of a road monitoring device according to a first embodiment. [Figure 6] FIG. 1 is a diagram for explaining a common feature calculated by a common feature calculation unit according to the first embodiment; [Figure 7] FIG. 1 is a diagram for explaining an example of an abnormality determination process performed by an abnormality determination unit of the road monitoring device according to the first embodiment; [Figure 8] FIG. 1 is a diagram for explaining an example of a difference calculation process performed by an abnormality determination unit of the road monitoring device according to the first embodiment; [Figure 9] 1 is a flowchart showing an example of a common feature amount calculation process performed by the road monitoring device according to the first embodiment. [Figure 10] A flowchart showing an example of a process including an abnormality determination process performed by the road monitoring device according to the first embodiment. [Figure 11] FIG. 1 is a diagram showing an example of a hardware configuration of a road monitoring device according to a first embodiment. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0011] Hereinafter, a road monitoring device, a road monitoring method, and a road monitoring program according to embodiments will be described in detail with reference to the drawings.

[0012] Embodiment 1 1 is a diagram showing an example of a configuration of a monitoring system including a road monitoring device according to the first embodiment. As shown in FIG. 1, the monitoring system 100 according to the first embodiment includes a road monitoring device 1 and imaging devices 21, 22, . . . , 2 n It is equipped with.

[0013] Road monitoring device 1 and imaging devices 21, 22,...,2 n are connected to a network 3. The network 3 is, for example, a wide area network (WAN) such as the Internet or a local area network (LAN). The network 3 may include a mobile communication network.

[0014] Imaging devices 21, 22,...,2 n are fixedly disposed at different positions and capture different regions. The imaging device 21 captures an image of a region R1, the imaging device 22 captures an image of a region R2, and the imaging device 2 n is the region Rn The area R1, R2, . . . , R n Each of the imaging devices 21, 22, . . . , 2 n When each of the regions R1, R2, . . . , R n When each of these is referred to without distinguishing them individually, they may be referred to as region R.

[0015] Fig. 2 is a diagram showing an example of an area captured by the imaging device according to the embodiment 1. As shown in Fig. 2, an area R captured by the imaging device 2 includes a road 60, a mountain 61, trees 62, a sky 63, and the like.

[0016] Each imaging device 2 transmits image information of an image obtained by imaging the area R to the road monitoring device 1 via the network 3. The image obtained by imaging the area R is a moving image, but may also be a still image. The road monitoring device 1 receives the image information transmitted from each imaging device 2 via the network 3, and determines an abnormality in the road 60 included in each area R based on the received image information. The image information includes information on each pixel of the image obtained by imaging the area R.

[0017] The road monitoring device 1 calculates information indicating common features of the multiple past images based on image information of a group of past images including multiple past images, which are images of the region R captured by each imaging device 2 under different environmental conditions. The environmental conditions are, for example, conditions such as weather and time of day. The weather is, for example, sunny, cloudy, rainy, or snowy, and the time of day is, for example, morning, afternoon, or night. The road monitoring device 1 determines abnormalities in the road 60 in real time for each region R based on image information of an input image, which is an image of the region R newly captured by each imaging device 2, and information indicating common features of the multiple past images.

[0018] For example, the road monitoring device 1 executes a common feature amount calculation process for calculating a common feature amount that represents, in pixel units, a feature common to a plurality of past images captured under different environmental conditions by the imaging device 2 as information indicating the common feature. In the following, the plurality of past images captured under different environmental conditions by the imaging device 2 may be simply referred to as a plurality of past images.

[0019] The common feature includes, for example, a first common feature indicating an average value or a median value of pixel-based feature values ​​between the multiple past images, and a second common feature indicating a degree of association between the pixel-based feature values. The degree of association is, for example, a degree of variation in pixel-based feature values ​​between the multiple past images, and the smaller the degree of variation, the higher the degree of association, and can also be said to be a degree of similarity of feature values ​​between the multiple past images. Note that the common feature may be defined as an average value or a median value of pixel-based feature values ​​in a region R in the multiple past images for pixels that constitute the multiple past images and have an association degree equal to or greater than a threshold value.

[0020] 3 is a diagram for explaining an example of a common feature amount calculation process executed by the road monitoring device according to the first embodiment. As shown in FIG. 3, the road monitoring device 1 stores past images 41, 42, . . . , 4 k Based on image information of a past image group 5 including the image 41, 42, . ...3, 44, 45, 46, 47, 48, 49, 50, 51, 52, 53, 54, 55, 56, 57, 58, 59, 60, 61, 62, 63, k When referring to each of these images without distinguishing them individually, they may be referred to as past images 4.

[0021] Each time the road monitoring device 1 acquires an input image, which is an image of area R newly captured by each imaging device 2, it executes a target feature calculation process to calculate in real time for each area R, based on the information of the input image, the pixel-by-pixel features in the input image as target features.

[0022] 4 is a diagram for explaining an example of a determination target feature calculation process executed by the road monitoring device according to the embodiment 1. As shown in Fig. 4, the road monitoring device 1 calculates a determination target feature, which is a feature in pixel units in the input image 6, based on information of the input image 6, which is an image of the area R newly captured by the imaging device 2.

[0023] Then, the road monitoring device 1 judges in real time an abnormality in the road 60 for each area R based on the common feature amount and the judgment target feature amount. This allows the road monitoring device 1 to accurately detect an abnormality in the road 60 from an image captured of the area R.

[0024] For example, the road monitoring device 1 can determine in real time which pixels of the road 60 have an abnormality on the road 60, based on a value obtained by multiplying the difference between the first common feature and the determination target feature by the second common feature. Furthermore, when the first common feature is defined by an average value or median value of the features of pixels whose relevance is equal to or greater than a threshold value, among the multiple pixels constituting the past image, the road monitoring device 1 can determine in real time whether there is an abnormality on the road 60 for each region R, based on the difference between the first common feature and the determination target feature.

[0025] The following provides a more detailed explanation of the road monitoring device 1. For the sake of convenience, the following describes a process for detecting an abnormality in the road 60 in one area R, but the same process is performed for the other areas R as well.

[0026] Fig. 5 is a diagram showing an example of the configuration of the road monitoring device according to the embodiment 1. As shown in Fig. 5, the road monitoring device 1 includes a past image information acquisition unit 10, a common feature calculation unit 11, a common feature storage unit 12, an input image information acquisition unit 13, a determination target feature calculation unit 14, an abnormality determination unit 15, a region enhancement unit 16, an abnormality type determination unit 17, and an output processing unit 18.

[0027] The past image information acquisition unit 10 acquires image information of a past image group 5 including a plurality of past images 4, which are images of the region R captured in the past under different environmental conditions by the imaging device 2, and outputs the acquired image information of the past image group 5 to the common feature amount calculation unit 11. The past image group 5 includes a plurality of past images 4 captured under different environmental conditions by the imaging device 2. The past images 4 are images of the region R in normal times. The normal state is a state in which no abnormality of the road 60 to be detected by the road monitoring device 1 occurs in the region R. An abnormality of the road 60 is, for example, the fall of road equipment on the road 60, the collapse of the road 60, falling rocks on the road 60, or the inflow of soil and sand into the road 60 due to a landslide of the mountain 61. An abnormality of the road 60 also includes, for example, an abnormality of the road equipment or the vicinity of the road in the region R, such as peeling of the inner wall of a tunnel, damage to a road sign, or the collapse of a road slope. Peeling of the inner wall of a tunnel refers, for example, to a state in which the peeled off portion hangs down without falling onto the road 60. Damage to a road sign refers, for example, to a state in which the road sign is bent but has not fallen onto the road 60. Examples of road facilities include signs and tunnel ceilings.

[0028] The past image information acquisition unit 10 acquires image information of the past image group 5 from, for example, an external device (not shown) or a storage medium (not shown) via the network 3. The past image 4 is, for example, an image captured by the imaging device 2 that is judged by a person to be a normal image, and this allows the road monitoring device 1 to approximate the result of a person judging an abnormality in the road 60 in the area R. Note that the past image 4 is not limited to an image that is judged by a person to be a normal image, and may be, for example, an image of the area R captured by the imaging device 2 during a specific period in the past. The specific period is, for example, a period during which no abnormality in the area R has been reported.

[0029] The common feature amount calculation unit 11 calculates a common feature amount, which is a feature amount in pixel units, as information indicating a feature common to a plurality of past images 4 captured under different environmental conditions, based on the image information of the past image group 5 acquired by the past image information acquisition unit 10. The common feature amount calculation unit 11 calculates the common feature amount by machine learning, for example, with the past image group 5 as a correct image group. The common feature amount calculation unit 11 stores information on the calculated common feature amount in the common feature amount storage unit 12. Hereinafter, the past image 4 may be referred to as a correct image, and the past image group 5 may be referred to as a correct image group.

[0030] Machine learning is supervised learning that uses past images 4, which are images taken repeatedly under different environmental conditions under normal circumstances, as learning data, and analyzes the past images 4 taken under different environmental conditions to acquire information indicating common features of these past images 4 through learning. In supervised learning, for example, a neural network can be used. Note that instead of a neural network, machine learning using a learning algorithm such as linear regression or logistic regression may be used.

[0031] The common feature calculation unit 11 includes an extraction unit 20 that extracts features of each correct image based on information of the group of correct images acquired by the past image information acquisition unit 10, and a calculation unit 21 that calculates a common feature based on the features of each correct image extracted by the extraction unit 20.

[0032] The extraction unit 20 extracts, for example, the relevance to adjacent pixels and color, etc., on a pixel-by-pixel basis as the feature amount of each correct answer image. The relevance to adjacent pixels is, for example, color continuity, also called edge. The color is a color indicated by color information on a pixel-by-pixel basis included in the image information of the correct answer image, and is expressed, for example, by RGB (Red, Green, Blue) values ​​or HSV (Hue Saturation Value) values.

[0033] The calculation unit 21 calculates a common feature based on the feature of each correct answer image extracted by the extraction unit 20, and stores information on the calculated common feature in the common feature storage unit 12. The common feature calculated by the calculation unit 21 is a pixel-by-pixel feature in an image of the region R in a state where no abnormality occurs on the road 60, and includes a first common feature indicating the pixel-by-pixel feature of the region R in the plurality of correct answer images, and a second common feature indicating the degree of association between the pixel-by-pixel feature between the correct answer images.

[0034] The first common feature is expressed by, for example, a feature indicating a color, such as an RGB value or an HSV value, and a feature indicating a degree of association with adjacent pixels. In the following, the feature indicating a color feature may be referred to as a first feature, and the feature indicating an association with adjacent pixels may be referred to as a second feature. The feature indicating a color may be, for example, a feature indicating at least one of brightness, hue, and color, instead of an RGB value or an HSV value.

[0035] Fig. 6 is a diagram for explaining a common feature calculated by the common feature calculation unit according to the first embodiment. In Fig. 6, the vertical axis indicates a first feature, and the horizontal axis indicates a second feature. The first feature is, for example, a color, and the second feature is, for example, a relevance to adjacent pixels.

[0036] The first common feature is the center of gravity, average value, or median value of feature coordinates in an orthogonal coordinate system defined by the first feature and the second feature in pixel units in the group of correct images, and is shown as the common feature coordinates in Fig. 6. When the first common feature is represented by three or more feature amounts, the center of gravity, average value, or median value of the feature coordinates in a coordinate system with three or more axes is calculated as the first common feature.

[0037] In the following, the first common feature will be described as being represented by the first feature and the second feature, but the first common feature may include a feature other than the first feature and the second feature, or may be a feature other than the first feature and the second feature.

[0038] Moreover, the common feature amount calculation unit 11 calculates, for example, the distance between the common feature amount coordinates and the feature amount coordinates farthest from the common feature amount coordinates, which is the first common feature amount, among the feature amount coordinates of the correct image. Hereinafter, the distance between the common feature amount coordinates and the feature amount coordinates farthest from the common feature amount coordinates may be referred to as a threshold distance. The threshold distance is, for example, the Euclidean distance, the Mahalanobis distance, the Chebyshev distance, or the Manhattan distance. The threshold distance is information included in the above-mentioned common feature amount, and is the third common feature amount.

[0039] The common feature amount calculation unit 11 can calculate the threshold distance by, for example, distance learning. For example, the common feature amount calculation unit 11 performs distance learning by deep learning using a neural network. Note that the common feature amount calculation unit 11 can also perform distance learning by a learning algorithm such as linear regression or logistic regression.

[0040] The second common feature is a feature indicating the degree of relevance between the correct answer images on a pixel-by-pixel basis. The degree of relevance between the correct answer images on a pixel-by-pixel basis indicates, for example, the degree of variation between the correct answer images of the feature values ​​extracted by the extraction unit 20 on a pixel-by-pixel basis. For example, the degree of relevance between the correct answer images on a pixel-by-pixel basis indicates the degree of variation between the correct answer images in terms of color and relevance to adjacent pixels. The lower the degree of variation of the feature values ​​between the correct answer images, the higher the degree of relevance between the correct answer images on a pixel-by-pixel basis. For example, the degree of relevance may be a value between 0 and 1, with the lower the degree of variation being the value closer to 1, and the higher the degree of variation being the value closer to 0. Note that the degree of relevance is not limited to this value, and may be appropriately set according to a method for determining pixels indicating an abnormality in the road 60, which will be described later.

[0041] The color of the pixels in the correct image that correspond to the sky 63 varies greatly depending on whether or not the sky 63 in region R includes objects such as clouds, the sun, or the moon. Even if the sky 63 in region R does not include any objects, the color of the pixels in the correct image that correspond to the sky 63 can be blue, red, or black depending on the weather.

[0042] Therefore, the color of the pixel corresponding to the sky 63 in the correct answer image has a high degree of variation in pixel-by-pixel feature amount between the correct answer images. In this case, the calculation unit 21 determines that the pixel corresponding to the sky 63 in the correct answer image has a low degree of relevance between the correct answer images in pixel units.

[0043] In addition, although vehicles such as automobiles and motorcycles travel on the surface of road 60, the probability that a pixel in an image captured at a given moment is a pixel that represents part of a vehicle is low. In addition, the surface of road 60 is often achromatic or nearly achromatic regardless of the time of day.

[0044] Therefore, the color of the pixel corresponding to the road surface of the road 60 in the correct answer image has a low degree of variation in pixel-by-pixel feature amount between the correct answer images. In this case, the calculation unit 21 determines that the pixel corresponding to the road surface of the road 60 in the correct answer image has a high degree of pixel-by-pixel relevance between the correct answer images.

[0045] The input image information acquisition unit 13 acquires information on an input image 6 which is an image of a region R newly captured by the imaging device 2. The input image information acquisition unit 13 acquires, for example, information on the input image 6 output from the imaging device 2 via the network 3 in real time.

[0046] Each time image information of the input image 6 is acquired by the input image information acquisition unit 13, the judgment target feature calculation unit 14 calculates a judgment target feature, which is a pixel-by-pixel feature in the input image 6, based on the acquired image information of the input image 6, and outputs information on the calculated judgment target feature to the abnormality judgment unit 15.

[0047] The determination target feature output from the determination target feature calculation unit 14 to the anomaly determination unit 15 is a feature of the same type as the type of the feature of the correct image extracted by the extraction unit 20. The determination target feature is, for example, a relevance to adjacent pixels, a color, etc.

[0048] The abnormality determination unit 15 acquires information on common features including the first common feature, the second common feature, and the third common feature from the common feature storage unit 12. The abnormality determination unit 15 judges an abnormality of the road 60 in the region R based on the common features acquired from the common feature storage unit 12 and the judgment target feature output from the judgment target feature calculation unit 14. The abnormality determination unit 15 judges an abnormality of the road 60 every time the judgment target feature is output from the judgment target feature calculation unit 14.

[0049] The abnormality determination unit 15 determines an abnormality in the road 60 based on a result of adjusting a pixel-by-pixel difference between a first common feature and a determination target feature among the common features by a pixel-by-pixel second common feature.

[0050] The abnormality determination unit 15 includes a difference calculation unit 30 and a determination unit 31. The difference calculation unit 30 calculates a pixel-by-pixel difference between a first common feature among the common features and a determination target feature. The determination unit 31 adjusts the pixel-by-pixel difference calculated by the difference calculation unit 30 with a pixel-by-pixel second common feature among the common features, and determines whether or not each pixel of the input image 6 is a pixel indicating an abnormality in the road 60 based on the adjustment result.

[0051] For example, the determination unit 31 multiplies the pixel-by-pixel difference calculated by the difference calculation unit 30 by a second common feature in pixel units among the common features, and calculates the multiplication result as a dissimilarity in pixel units. The determination unit 31 determines a pixel indicating an abnormality in the road 60 based on the dissimilarity in pixel units. For example, the determination unit 31 determines that a pixel having a dissimilarity equal to or greater than a threshold distance is a pixel indicating an abnormality in the road 60. The threshold distance is the third common feature described above.

[0052] Fig. 7 is a diagram for explaining an example of an abnormality determination process by the abnormality determination unit of the road monitoring device according to the first embodiment. As shown in Fig. 7, the difference calculation unit 30 of the abnormality determination unit 15 performs a difference calculation process for calculating the difference between the first common feature and the determination target feature on a pixel-by-pixel basis. In the example shown in Fig. 7, pixels P1, P2, P3, . . . , Pm are included as pixels constituting the image of region R. m is an integer indicating the number of pixels constituting the image of region R.

[0053] Here, an example of the difference calculation process will be described. Fig. 8 is a diagram for explaining an example of the difference calculation process by the abnormality determination unit of the road monitoring device according to the first embodiment. In Fig. 8, the vertical axis indicates the first feature amount, and the horizontal axis indicates the second feature amount. The first feature amount is, for example, color, and the second feature amount is, for example, relevance to adjacent pixels.

[0054] Here, the determination unit 31 calculates the distance between common feature amount coordinates, which are coordinates defined by the first feature amount and the second feature amount included in the first common feature amount, and determination target feature amount coordinates, which are coordinates defined by the first feature amount and the second feature amount included in the determination target feature amount, as a difference between the first common feature amount and the determination target feature amount. The distance between the common feature amount coordinates and the determination target feature amount coordinates is, for example, Euclidean distance, Mahalanobis distance, Chebyshev distance, Manhattan distance, or the like.

[0055] 7, the description of the abnormality determination unit 15 will be continued. The determination unit 31 of the abnormality determination unit 15 performs a dissimilarity calculation process in which the determination unit 31 multiplies the difference calculated in the difference calculation process by the second common feature amount, and calculates the multiplication result as the dissimilarity between the input image 6 and the past image group 5 on a pixel-by-pixel basis.

[0056] Then, the determination unit 31 performs an abnormality determination process to determine an abnormality in the road 60 based on the dissimilarity calculated in the dissimilarity calculation process and the threshold distance. For example, the determination unit 31 determines that a pixel whose dissimilarity is equal to or greater than the threshold distance is a pixel that indicates an abnormality in the road 60, and determines that a pixel whose dissimilarity is less than the threshold distance is a pixel that does not indicate an abnormality in the road 60.

[0057] The determination unit 31 can also perform an abnormality determination process for determining a pixel indicating an abnormality in the road 60 based on the difference between the first common feature amount and the determination target feature amount of a plurality of identical pixels. For example, the determination unit 31 can determine that the pixel indicates an abnormality in the road 60 when the dissimilarity of the identical pixel is equal to or greater than the threshold distance with a preset frequency, and can otherwise determine that the pixel does not indicate an abnormality in the road 60. The determination unit 31 can also determine that the pixel indicates an abnormality in the road 60 when the number of times that the dissimilarity of the identical pixel calculated in the dissimilarity calculation process is consecutively equal to or greater than the threshold distance is equal to or greater than a preset threshold, and can otherwise determine that the pixel does not indicate an abnormality in the road 60.

[0058] Furthermore, the process by the determination unit 31 is not limited to the above-mentioned process. For example, the determination unit 31 may be configured to determine, as a determination target pixel, a pixel having a second common feature amount equal to or greater than a threshold distance among a plurality of pixels constituting the input image 6, and determine an abnormality in the road 60 based on a difference between the determination target feature amount and the first common feature amount in the determination target pixel.

[0059] In this case, the determination unit 31 treats the difference of the determination target pixel as the determination target pixel among the differences in multiple pixel units calculated by the difference calculation unit 30. The determination unit 31 determines that a pixel in which the difference between the determination target feature amount and the first common feature amount in the determination target pixel is equal to or greater than the threshold distance is a pixel that indicates an abnormality in the road 60, and determines that a pixel in which the difference in the determination target pixel is less than the threshold distance is a pixel that does not indicate an abnormality in the road 60.

[0060] In this case, the determination unit 31 can also perform an abnormality determination process for determining an abnormality in the road 60 based on a plurality of differences of pixels to be determined as identical. For example, the determination unit 31 can determine that the pixel to be determined is a pixel indicating an abnormality in the road 60 when the difference in the pixel to be determined is equal to or greater than the threshold distance, and can also determine that the pixel to be determined is not a pixel indicating an abnormality in the road 60 when the difference in the pixel to be determined as identical calculated by the difference calculation unit 30 is equal to or greater than the threshold distance with a preset frequency, and can also determine that the pixel to be determined is not a pixel indicating an abnormality in the road 60 when the difference in the pixel to be determined as identical calculated by the difference calculation unit 30 is equal to or greater than the threshold distance with a preset frequency, and can also determine that the pixel to be determined is not a pixel indicating an abnormality in the road 60 when the number of times that the difference in the pixel to be determined as identical calculated by the difference calculation unit 30 is equal to or greater than the threshold distance is equal to or greater than a preset threshold ....

[0061] Returning to FIG. 5, the road monitoring device 1 will be described further. The area highlighting unit 16 of the road monitoring device 1 highlights pixels in the input image 6 that indicate an abnormality in the road 60 determined by the abnormality determination unit 15. The area highlighting unit 16 highlights pixels that indicate an abnormality in the road 60, for example, by making the pixels that indicate an abnormality in the road 60 a specific color or by surrounding an area of ​​one or more pixels that indicate an abnormality in the road 60 with a thick line. The specific color is, for example, a color that has the fewest identical or similar colors among the colors of the pixels in the input image 6, but may also be a predetermined color. The predetermined color is, for example, red.

[0062] The area highlighting unit 16 can highlight pixels indicating an abnormality in the road 60 so that the difference and the first color approach the second color as the difference between the dissimilarity and the distance threshold increases or the difference from the distance threshold increases. The first color is, for example, blue, and the second color is, for example, red.

[0063] The abnormality type determination unit 17 of the road monitoring device 1 determines the type of abnormality of the road 60 based on image information of an image including pixels indicating an abnormality of the road 60 determined by the abnormality determination unit 15. The types of abnormality of the road 60 include, for example, a rock falling onto the surface of the road 60, a collapse of the surface of the road 60, an inflow of soil and sand onto the surface of the road 60, or road equipment falling onto the road 60.

[0064] The abnormality type determination unit 17 has, for example, a calculation model that receives image information as input and outputs information on the type of abnormality in the road 60, and inputs image information of an image from the input image 6 that includes pixels that indicate an abnormality in the road 60 into the calculation model, and determines the type of abnormality in the road 60 based on the information on the type of abnormality in the road 60 that is output from the calculation model.

[0065] The information on the type of abnormality of the road 60 is, for example, a score for each type of abnormality of the road 60, and the abnormality type determination unit 17 determines, among the multiple types of abnormality of the road 60, an abnormality of the road 60 having a score equal to or greater than a threshold as the type of abnormality of the road 60. The computational model is, for example, a computational model generated by deep learning, but may also be a computational model generated by a learning algorithm such as linear regression or logistic regression.

[0066] The output processing unit 18 outputs judgment result information including information on the input image 6 in which pixels indicating abnormalities in the road 60 are highlighted by the area highlighting unit 16, and information on the type of abnormality in the road 60 judged by the abnormality type judgment unit 17.

[0067] Next, a description will be given of a common feature calculation process performed by the road monitoring device 1 with reference to a flowchart. Fig. 9 is a flowchart showing an example of a common feature calculation process performed by the road monitoring device according to the first embodiment.

[0068] 9, the past image information acquisition unit 10 of the road monitoring device 1 acquires image information of the past image group 5 (step S10). Then, the common feature amount calculation unit 11 of the road monitoring device 1 extracts pixel-by-pixel feature amounts of each past image 4 based on the image information of the past image group 5 acquired in step S10 (step S11).

[0069] Next, the common feature calculation unit 11 calculates a pixel-by-pixel common feature of the region R based on the pixel-by-pixel feature of each past image 4 (step S12), and stores information on the calculated pixel-by-pixel common feature in the common feature storage unit 12 (step S13). When the process of step S13 ends, the road monitoring device 1 ends the process shown in FIG.

[0070] Next, a description will be given of a process including the abnormality determination process by the road monitoring device 1 with reference to a flowchart. Fig. 10 is a flowchart showing an example of a process including the abnormality determination process by the road monitoring device according to the first embodiment.

[0071] 10, the input image information acquisition unit 13 of the road monitoring device 1 acquires information of the input image 6 from the imaging device 2 (step S20). The determination target feature amount calculation unit 14 of the road monitoring device 1 extracts pixel-by-pixel feature amounts of the input image 6 as pixel-by-pixel determination target features based on the information of the input image 6 acquired by the input image information acquisition unit 13 (step S21).

[0072] Next, the abnormality determination unit 15 of the road monitoring device 1 acquires information on the common feature in pixel units from the common feature storage unit 12 (step S22). Then, the abnormality determination unit 15 calculates the difference between the determination target feature and the first common feature in pixel units (step S23).

[0073] Next, the abnormality determination unit 15 multiplies the difference calculated in step S23 by the second common feature to calculate a dissimilarity on a pixel-by-pixel basis (step S24).Then, the abnormality determination unit 15 determines an abnormality in the road 60 based on the dissimilarity on a pixel-by-pixel basis calculated in step S24 and a threshold distance (step S25).

[0074] Next, the area highlighting unit 16 of the road monitoring device 1 generates an input image 6 in which the area of ​​pixels indicating an abnormality in the road 60 is highlighted (step S26). Furthermore, the abnormality type determining unit 17 of the road monitoring device 1 determines the type of abnormality in the road 60 based on the image including the pixels indicating the abnormality in the road 60 (step S27).

[0075] Next, the output processing unit 18 of the road monitoring device 1 outputs judgment result information including information of the input image 6 in which pixels indicating abnormalities in the road 60 have been emphasized in step S26, and information on the type of abnormality in the road 60 judged in step S27 (step S28).

[0076] Next, when the process of step S28 is completed, the road monitoring device 1 judges whether or not to terminate the operation process (step S29). For example, when the power supply of the road monitoring device 1 is turned off or when an operation on an operation unit (not shown) is an operation to terminate the operation, the road monitoring device 1 judges to terminate the operation process.

[0077] If the road monitoring device 1 determines not to end the operation processing (step S29: No), it transitions to step S20, and if it determines that an operation end operation has been performed (step S29: Yes), it ends the processing shown in FIG. 10.

[0078] 11 is a diagram illustrating an example of a hardware configuration of the road monitoring device according to the embodiment 1. As illustrated in FIG. 11, the road monitoring device 1 includes a computer including a processor 101, a memory 102, a communication device 103, and an interface circuit 104.

[0079] The processor 101, the memory 102, the communication device 103, and the interface circuit 104 can transmit and receive information to and from each other via, for example, a bus 105. The common feature storage unit 12 is realized by the memory 102. The processor 101 executes functions of the past image information acquisition unit 10, the common feature calculation unit 11, the input image information acquisition unit 13, the judgment target feature calculation unit 14, the abnormality judgment unit 15, the area enhancement unit 16, the abnormality type judgment unit 17, and the output processing unit 18 by reading and executing a program stored in the memory 102. The processor 101 is, for example, an example of a processing circuit, and includes one or more of a CPU (Central Processing Unit), a DSP (Digital Signal Processor), and a system LSI (Large Scale Integration).

[0080] The memory 102 includes one or more of a RAM (Random Access Memory), a ROM (Read Only Memory), a flash memory, an EPROM (Erasable Programmable Read Only Memory), and an EEPROM (Electrically Erasable Programmable Read Only Memory). The memory 102 also includes a recording medium on which a computer-readable program is recorded. Such a recording medium includes one or more of a non-volatile or volatile semiconductor memory, a magnetic disk, a flexible memory, an optical disk, a compact disk, and a DVD (Digital Versatile Disc). The road monitoring device 1 may include an integrated circuit such as an ASIC (Application Specific Integrated Circuit) and an FPGA (Field Programmable Gate Array).

[0081] The road monitoring device 1 may be configured with a terminal device, may be configured with a server, or may be configured with a terminal device and a server. When the road monitoring device 1 is configured with two or more devices, each of the two or more devices has, for example, the hardware configuration shown in FIG. 11. Note that communication between the two or more devices is performed via a communication device 103. The road monitoring device 1 may also be configured with two or more servers. For example, the road monitoring device 1 may be configured with a processing server and a data server.

[0082] Alternatively, the imaging device 2 and a terminal device may be installed on-site, and the results detected by the terminal device may be transmitted to the server. In this case, the functions of the input image information acquisition unit 13, the determination target feature amount calculation unit 14, and the anomaly determination unit 15 are executed by the terminal device.

[0083] In the above example, the road monitoring device 1 calculates the common feature and the determination target feature in units of pixels, but it can also calculate the common feature and the determination target feature in units of multiple pixels. For example, the road monitoring device 1 can calculate the common feature and the determination target feature in units of 2×2, or 4 pixels. In this case, the road monitoring device 1 judges an abnormality in the road 60 based on the common feature in units of 4 pixels and the determination target feature in units of 4 pixels.

[0084] As described above, the road monitoring device 1 according to the first embodiment includes the abnormality determination unit 15. The abnormality determination unit 15 determines an abnormality in the road 60 based on information indicating a common feature in a plurality of past images 4 captured under different environmental conditions by the imaging device 2 that captures an area R including the road 60, and information on the input image 6 newly captured by the imaging device 2. This enables the road monitoring device 1 to accurately detect an abnormality in the road 60 from the image in which the area R including the road 60 is captured.

[0085] Moreover, the common feature is a feature in units of one or more pixels. This allows the road monitoring device 1 to accurately detect an abnormality in the road 60 in units of one or more pixels from an image of the area R including the road 60.

[0086] The road monitoring device 1 also includes a common feature amount calculation unit 11 and a judgment target feature amount calculation unit 14. The common feature amount calculation unit 11 calculates a common feature amount representing a common feature in units of one or more pixels based on a plurality of past images 4 as information indicating the common feature. The judgment target feature amount calculation unit 14 calculates a judgment target feature amount, which is a feature amount in units of one or more pixels in the input image 6, as information of the input image 6. The abnormality determination unit 15 determines an abnormality in the road 60 based on the common feature amount calculated by the common feature amount calculation unit 11 and the judgment target feature amount calculated by the judgment target feature amount calculation unit 14. This allows the road monitoring device 1 to accurately detect an abnormality in the road 60 from an image in which an area R including the road 60 is captured.

[0087] The common feature includes a first common feature indicating one or more pixel-unit feature of the region R in the multiple past images 4, and a second common feature indicating one or more pixel-unit relatedness between the multiple past images 4. The abnormality determination unit 15 determines an abnormality in the road 60 based on a result of adjusting one or more pixel-unit differences between the determination target feature and the first common feature by one or more pixel-unit second common features. This enables the road monitoring device 1 to more accurately detect an abnormality in the road 60 from an image in which the region R including the road 60 is captured.

[0088] In addition, the common feature calculation unit 11 calculates the common feature by machine learning, which enables the road monitoring device 1 to more accurately detect an abnormality in the road 60 from an image in which the area R including the road 60 is captured.

[0089] Moreover, the common feature amount calculation unit 11 includes an extraction unit 20 and a calculation unit 21. The extraction unit 20 extracts one or more pixel-based feature amounts from each of the multiple past images 4. The calculation unit 21 calculates a common feature amount based on the one or more pixel-based feature amounts from each of the multiple past images 4 extracted by the extraction unit 20. This enables the road monitoring device 1 to accurately detect abnormalities in the road 60 from an image in which an area R including the road 60 is captured.

[0090] The road monitoring device 1 also includes an area highlighting unit 16 that highlights an area in the input image 6 that indicates an abnormality in the road 60 determined by the abnormality determination unit 15. This allows the road monitoring device 1 to present the abnormality in the road 60 to, for example, a monitor monitoring the input image 6 in a manner that allows the monitor to more easily grasp the abnormality.

[0091] The road monitoring device 1 also includes an abnormality type determination unit 17 that determines the type of abnormality in the road 60 based on image information of the image of the area showing the abnormality in the road 60 determined by the abnormality determination unit 15. This allows the road monitoring device 1 to present the type of abnormality in the road 60 to, for example, a monitor monitoring the input image 6.

[0092] The configurations shown in the above embodiments are merely examples, and may be combined with other known technologies. Parts of the configurations may be omitted or modified without departing from the spirit of the invention. [Explanation of symbols]

[0093] 1 Road monitoring device, 2,21,22,...,2 n Imaging device, 3 Network, 4, 41, 42, 4 k Past image, 5 past image group, 6 input image, 10 past image information acquisition unit, 11 common feature calculation unit, 12 common feature memory unit, 13 input image information acquisition unit, 14 judgment target feature calculation unit, 15 abnormality judgment unit, 16 area enhancement unit, 17 abnormality type judgment unit, 18 output processing unit, 20 extraction unit, 21 calculation unit, 30 difference calculation unit, 31 judgment unit, 60 road, 61 mountain, 62 tree, 63 sky, 100 surveillance system.

Claims

1. an abnormality determination unit that determines an abnormality in the road based on information indicating a common feature in a plurality of past images captured under different environmental conditions by an imaging device that captures an area including the road and information on an input image newly captured by the imaging device; a common feature amount calculation unit that calculates a common feature amount that represents the common feature in units of one or more pixels based on the plurality of past images as information indicating the common feature; a determination target feature amount calculation unit that calculates a determination target feature amount, which is a feature amount of one or more pixels in the input image, as information of the input image, The common feature amount is a first common feature amount indicating a feature amount of one or more pixels of the region in the plurality of previous images, and a second common feature amount indicating a degree of association of one or more pixels between the plurality of previous images, The abnormality determination unit When judging an abnormality of the road based on the common feature calculated by the common feature calculation unit and the judgment target feature calculated by the judgment target feature calculation unit, the abnormality of the road is judged based on a result of adjusting one or more pixel-unit differences between the judgment target feature and the first common feature by one or more pixel-unit second common feature. A road monitoring device comprising:

2. The common feature is one or more pixel-level features.

2. The road monitoring device according to claim 1.

3. The common feature amount calculation unit Calculate the common features using machine learning 3. A road monitoring device according to claim 1 or 2.

4. The common feature amount calculation unit an extracting unit that extracts one or more pixel-unit feature amounts from each of the plurality of past images; a calculation unit that calculates the common feature amount based on one or more pixel-unit feature amounts in each of the plurality of past images extracted by the extraction unit.

4. A road monitoring device according to claim 1, wherein the road monitoring device is a vehicle.

5. 1. A computer implemented method for road surveillance, comprising: an abnormality determination step of determining an abnormality in the road based on information indicating a common feature in a plurality of past images captured under different environmental conditions by an imaging device that captures an area including the road and information on an input image newly captured by the imaging device; a common feature amount calculation step of calculating a common feature amount representing the common feature in units of one or more pixels based on the plurality of past images as information indicating the common feature; a determination target feature calculation step of calculating a determination target feature, which is a feature of one or more pixels in the input image, as information of the input image, The common feature amount is a first common feature amount indicating a feature amount of one or more pixels of the region in the plurality of previous images, and a second common feature amount indicating a degree of association of one or more pixels between the plurality of previous images, In the abnormality determination step, When judging an abnormality of the road based on the common feature calculated in the common feature calculation step and the judgment target feature calculated in the judgment target feature calculation step, the abnormality of the road is judged based on a result of adjusting one or more pixel-unit differences between the judgment target feature and the first common feature by one or more pixel-unit second common feature. A road monitoring method comprising:

6. an abnormality determination step of determining an abnormality in the road based on information indicating a common feature in a plurality of past images captured under different environmental conditions by an imaging device that captures an area including the road and information on an input image newly captured by the imaging device; a common feature amount calculation step of calculating a common feature amount representing the common feature in units of one or more pixels based on the plurality of past images as information indicating the common feature; a determination target feature calculation step of calculating a determination target feature, which is a feature of one or more pixels in the input image, as information of the input image, The common feature amount is a first common feature amount indicating a feature amount of one or more pixels of the region in the plurality of previous images, and a second common feature amount indicating a degree of association of one or more pixels between the plurality of previous images, In the abnormality determination step, When judging an abnormality of the road based on the common feature calculated in the common feature calculation step and the judgment target feature calculated in the judgment target feature calculation step, the computer is caused to execute the step of judging an abnormality of the road based on a result of adjusting one or more pixel-unit differences between the judgment target feature and the first common feature by one or more pixel-unit second common feature. A road monitoring program comprising:

Images