Information processing device, determination method, and program

Abstract

An information processing device, a determination method, and a program are provided that use images to determine whether snow removal is necessary. [Solution] In a system in which a vehicle equipped with an on-board camera and an information processing device are connected to a network, the information processing device has an input unit that inputs images of snowy roads, and a judgment unit that determines whether snow removal is necessary on snowy roads using output from a model that identifies rut ​​areas on snowy roads from images, and the judgment unit includes a semantic segmentation unit that divides the image into regions, and an estimation unit that determines whether snow removal is necessary using output from the semantic segmentation model.

Description

[Technical Field]

[0001] The present invention relates to a technology for determining whether snow removal is necessary using an image. [Background technology]

[0002] In areas with heavy snowfall in winter, road conditions deteriorate due to the accumulation of large amounts of snow and the formation of ruts caused by melting snow. In many cases, local governments have their employees patrol the roads and request snow removal vehicles from snow removal contractors based on visual judgment based on experience. However, this method incurs a lot of labor costs due to the patrols.

[0003] A conventional technique related to the automation of snow removal necessity determination is a method for evaluating snow accumulation conditions using images captured by a surveillance camera (Non-Patent Document 1). The technique disclosed in Non-Patent Document 1 quantitatively evaluates snow accumulation depth in images captured by a highway surveillance camera through regression using deep learning. [Prior art documents] [Non-patent literature]

[0004] [Non-Patent Document 1] Masato ABE, Koichi SUGISAKI, Kazuki NAKAMURA and Isao KAMIISHI.: SNOW DEPTH EVALUATION FROM HIGHWAY MONITORING CAMERA USING DEEP LEARNING, Artificial Intelligence and Data Science, 2021, Volume 2, Issue 1, Pages 26-29. [Non-patent document 2] Bowen Cheng, Ishan Misra, Alexander Kirillov, Alexander G. Schwing and Rohit Girdhar, Masked-attention Mask Transformer for Universal Image Segmentation, In CVPR, 2022 [Non-patent document 3] Jon Louis Bentley. "Multidimensional Binary Search Trees Used for Associative Searching". In: Communications of the ACM 18.9 (1975), pp. 509-517 Summary of the Invention [Problem to be solved by the invention]

[0005] However, the technology disclosed in Non-Patent Document 1 is considered to have low accuracy in determining whether snow removal is necessary, since the evaluation is based only on the snow depth on the road shoulder.

[0006] The present invention has been made in view of the above points, and aims to provide a technology that enables determination of the need for snow removal with higher accuracy than conventional technology. [Means for solving the problem]

[0007] According to the disclosed technology, a determination unit receives an image of a snowy road as an input, and determines whether snow removal is required on the snowy road using an output from a model that identifies rut ​​areas on the snowy road. An information processing device comprising: [Effects of the Invention]

[0008] The disclosed technology provides a technology that enables determination of the need for snow removal with higher accuracy than conventional technology. [Brief explanation of the drawings]

[0009] [Figure 1] 1 is a diagram illustrating an example of a system configuration according to an embodiment of the present invention. [Figure 2] 1 is a diagram illustrating an example of the configuration of an information processing device 100 according to a first embodiment and a second embodiment. [Figure 3] FIG. 2 is a diagram illustrating an example of the configuration of a determination unit 120 in the first embodiment. [Figure 4] FIG. 2 is a diagram illustrating an example of the configuration of a determination unit 120 in the first embodiment. [Figure 5] FIG. 10 is a diagram showing a processing flow during learning. [Figure 6] FIG. 10 is a diagram for explaining the depth of a rut. [Figure 7] FIG. 10 is a diagram showing a processing flow during inference. [Figure 8] FIG. 2 is a diagram illustrating an example of the configuration of a determination unit 120 in the first embodiment. [Figure 9] FIG. 10 is a diagram illustrating an example of the configuration of an information processing device 100 according to a third embodiment. [Figure 10] 1 is a diagram illustrating an example of a hardware configuration of an information processing device 100. FIG. DETAILED DESCRIPTION OF THE INVENTION

[0010] Hereinafter, embodiments of the present invention (present embodiments) will be described with reference to the drawings. The embodiments described below are merely examples, and the embodiments to which the present invention is applied are not limited to the following embodiments. First to third embodiments will be described below.

[0011] (System configuration example) Fig. 1 shows an example of the configuration of a system common to the first to third embodiments. As shown in Fig. 1, this system includes a vehicle 300 connected to a network 200 and an information processing device 100. The vehicle 300 is equipped with an on-board camera 310. Although Fig. 1 shows one vehicle 300 as the vehicle that captures images, it is assumed that in practice a plurality of vehicles will be used.

[0012] In this system, when a vehicle 300 travels on a road, an image of the road taken by an on-board camera 310 is transmitted to an information processing device 100 via a network 200. The information processing device 100 uses the image to determine whether snow removal is necessary. Note that this "image" may be a still image or an image (frame) that constitutes a moving image.

[0013] The vehicle 300 is assumed to be a general automobile having four tires, but is not limited to this. In the example of Fig. 1, the information processing device 100 is provided outside the vehicle 300, but this is just an example. For example, the information processing device 100 may be provided inside the vehicle 300.

[0014] The information processing device 100 may output the result of the determination of whether snow removal is necessary, or may automatically request the dispatch of a snowplow if snow removal is necessary. In either case, this system can automate the decision to request the dispatch of a snowplow, which contributes to reducing labor costs.

[0015] It should be noted that the images used by the information processing device 100 to determine whether snow removal is necessary are not limited to images taken by the in-vehicle camera 310. The images used by the information processing device 100 to determine whether snow removal is necessary may be, for example, images taken by a surveillance camera, images taken by a person with a camera such as a smartphone, images taken of a television screen, etc. However, by using images taken by the in-vehicle camera 310, evaluation can be performed without being limited to a specific area.

[0016] Hereinafter, first to third embodiments will be described as embodiments relating to the configuration and operation of the information processing device 100.

[0017] [First embodiment] (Overview of the configuration and operation of the information processing device 100) Fig. 2 shows an example of the functional configuration of the information processing device 100. As shown in Fig. 2, the information processing device 100 includes an input unit 110, a determination unit 120, an output unit 130, and a learning unit 140. Note that the configuration shown in Fig. 2 is a configuration assuming that the information processing device 100 performs learning of the determination unit 120. When using a trained model (determination unit 120) trained outside the information processing device 100, the learning unit 140 is not necessary.

[0018] The input unit 110 receives an image captured by an on-board camera and inputs the image to the determination unit 120. The determination unit 120 determines whether snow removal is necessary based on the image input from the input unit 110, and passes the determination result to the output unit 130. The output unit 130 outputs the determination result. An overview of the processing of the information processing device 100 is as follows.

[0019] On snow-covered roads (called snowy roads), ruts, which are the tracks left by tires, are created when vehicles pass over them. Ruts hinder the safe driving of vehicles, so the occurrence of ruts is an important basis for determining whether snow removal is necessary.

[0020] Therefore, the determination unit 120 explicitly detects ruts on snowy roads before determining whether snow removal is necessary. As a result, the technology according to this embodiment can perform a determination with higher accuracy than snow removal necessity determination technologies that do not involve rut detection. Furthermore, using ruts makes it easier for humans to interpret the basis for the determination.

[0021] In the technique disclosed in Non-Patent Document 1, a region of interest (ROI) in an image is manually set in advance. However, this method can result in unnecessary information being included in the region, or necessary information being omitted from the region.

[0022] Therefore, in this embodiment, a mechanism for extracting (segmenting) rut areas on a snowy road is introduced into the determination unit 120, making it possible to input efficient information to the estimation unit at the subsequent stage.

[0023] (Configuration example of the determination unit 120) Fig. 3 shows an example of the configuration of the determination unit 120. In this embodiment, a neural network model is used as the determination unit 120. Fig. 3 corresponds to the configuration of that model.

[0024] As shown in FIG. 3, the determination unit 120 of the first embodiment includes a semantic segmentation unit 121, an estimation unit 122, and a combination unit 123.

[0025] The semantic segmentation unit 121 is a model that performs semantic segmentation on an input image (original image) and outputs the segmentation result (classified mask image). Semantic segmentation may also be called region division.

[0026] Semantic segmentation is a method of assigning a label or category to each pixel in an image. Note that although semantic segmentation is used in this embodiment, a method other than semantic segmentation for performing region division may also be used.

[0027] As shown in FIG. 3, the mask image output from the semantic segmentation unit 121 and the original image are combined (concatenated) by the combining unit 123, and the missing image is input to the estimation unit 122. The estimation unit 122 determines whether snow removal is necessary from the image obtained by combining the original image and the mask image, and outputs the determination result. As the determination result, a score may be output, or the result of threshold processing performed on the score (e.g., snow removal necessary or snow removal not required) may be output. In this embodiment, the result of threshold processing is output.

[0028] The semantic segmentation unit 121 can be realized, for example, by the technique disclosed in Non-Patent Document 2. The estimation unit 122 can be realized, for example, by a convolutional neural network (CNN).

[0029] FIG. 4 shows the configuration of the determination unit 120 when Mask2Former disclosed in Non-Patent Document 2 is used as the semantic segmentation unit 121 and CNN is used as the estimation unit 122.

[0030] The semantic segmentation unit 121 (i.e., Mask2Former) shown in Figure 4 has a backbone 1 (e.g., ResNet) that extracts features from an image, a pixel decoder 2 that gradually upsamples low-resolution features to high resolution, and a transformer decoder 3 that performs final prediction.

[0031] (Learning process flow) The processing flow during learning will be explained according to the steps in the flowchart of FIG.

[0032] <S101(ステップ101)> In S101, classes are defined based on one or more or all of the following: the presence or absence of ruts, the depth of the ruts if present, and differences in snow quality. For example, if the focus is on snow accumulation in the rutted areas and the depth is considered to be "low" or "high," and the snow quality is considered to be "packed snow" or "other," the number of classes will be four. Figure 6 shows an image of rut depth. Note that this classification is just one example, and classification may be performed using methods other than this.

[0033] <s102> In S102, an image captured by an in-vehicle camera 310 installed in a vehicle 300 such as a bus or a taxi is acquired.

[0034] <s103> In S103, based on the definition in S101, annotations are applied to the images acquired in S102 to create learning data.

[0035] <s104> The learning unit 140 of the information processing device 100 uses the learning data created in S103 to train the semantic segmentation unit 121. Specifically, the learning unit 140 adjusts the parameters of the semantic segmentation unit 121 so that the classified mask image output for the input image to the semantic segmentation unit 121 becomes a mask image classified into the correct class.

[0036] By performing the above learning, it is possible to construct a semantic segmentation unit 121 that can infer rut areas within an image. More specifically, it is possible to construct a semantic segmentation unit 121 that can perform processing taking into account information such as rut ​​depth and snow quality.

[0037] The learning unit 140 also learns the estimation unit 122. For example, a learning image is input to the trained semantic segmentation unit 121, an image obtained by combining the mask image output from the trained semantic segmentation unit 121 and the original image is input to the estimation unit 122, and the parameters of the estimation unit 122 are adjusted so that the output from the estimation unit 122 becomes the correct output (correct determination result).

[0038] (Processing flow during inference) Next, the processing flow at the time of inference (at the time of data evaluation) will be explained according to the procedure in the flowchart of Fig. 7. Here, it is assumed that both the semantic segmentation unit 121 and the estimation unit 122 have already been trained.

[0039] <s201> In S201, data to be evaluated (an unknown image not included in the learning data) is input to the input unit 110. The image is input to the semantic segmentation unit 121. The semantic segmentation unit 121 outputs a mask image in which the rut region has been masked.

[0040] <s202> In S202, the combining unit 123 combines the original image and the mask image input in S201 in the channel direction. The combining unit 123 inputs the image obtained by the combination (combined image) to the estimation unit 122.

[0041] <s203> In S203, first, the estimation unit 122 performs regression processing on the combined image to generate a score. In this embodiment, a threshold is set for the regression score, and the estimation unit 122 determines whether snow removal is necessary by comparing the score with the threshold. For example, the estimation unit 122 has been trained to increase the score when snow removal is necessary, and outputs the snow removal necessity as a determination result when the score is greater than the set threshold. Note that a processing unit that performs threshold determination may be provided separately from the estimation unit 122.

[0042] [Second embodiment] Next, a second embodiment will be described. The second embodiment is partially different from the first embodiment. The following describes the differences between the second embodiment and the first embodiment.

[0043] In the first embodiment, a mask image (segmentation mask) was used as the output from the semantic segmentation unit 121 to be used as input to the estimation unit 122, but in the second embodiment, an intermediate output of the semantic segmentation unit 121 is used as the output from the semantic segmentation unit 121 to be used as input to the estimation unit 122.

[0044] An example of the configuration of the determination unit 120 in the second embodiment is shown in Fig. 8. As shown in Fig. 8, in the second embodiment, the feature amount of the original image obtained by the backbone 1 of the semantic segmentation unit 121 (specifically, Mask2Former) is output and input to the estimation unit 122. The estimation unit 122 in the second embodiment is, for example, an FNN (Feedforward Neural Network), but is not limited to this and any arbitrary neural network can be used.

[0045] The semantic segmentation unit 121 also outputs a mask image similar to that in the first embodiment. However, the mask image is not input to the estimation unit 122 but is output to the outside. The learning process in the second embodiment is similar to that in the first embodiment. The inference process is as follows.

[0046] First, data to be evaluated (an unknown image not included in the training data) is input to the input unit 110. The image is input to the semantic segmentation unit 121. The semantic segmentation unit 121 outputs a mask image in which the rut region has been masked, and feature amounts (intermediate output) obtained during the process. The intermediate output is input to the estimation unit 122.

[0047] The estimation unit 122 performs regression processing from the intermediate output to generate a score. In this embodiment, a threshold is set for the regression score, and the estimation unit 122 determines whether snow removal is necessary by comparing the score with the threshold. For example, the estimation unit 122 has been trained to increase the score when snow removal is necessary, and outputs the determination result that snow removal is necessary when the score is greater than the set threshold. Note that a processing unit that performs threshold determination may be provided separately from the estimation unit 122.

[0048] By using the intermediate output as in the second embodiment, it is possible to reduce processing costs compared to using the final output of the semantic segmentation unit 121. In addition, since a mask image is output, it is also possible for humans to interpret the basis for the determination.

[0049] [Third embodiment] Next, a third embodiment will be described. The adjustment unit 150 used in the third embodiment, which will be described later, may be applied to the determination unit 120 of the first embodiment, or may be applied to the determination unit 120 of the second embodiment. Hereinafter, the first and second embodiments will be collectively referred to as the "basic technology."

[0050] (Device configuration) The technology disclosed in Non-Patent Document 1 and elsewhere evaluates snow accumulation through fixed-point observation. Fixed-point observation provides insufficient information for determining the need for snow removal over a wide area, making it difficult to translate this into efficient snow removal decisions. This issue can also arise with the basic technology described above.

[0051] Therefore, in the third embodiment, an adjustment unit 150 is added to the information processing device 100 in the basic technology. Fig. 9 shows an example of the configuration of the information processing device 100 in the third embodiment.

[0052] The adjustment unit 150 adjusts the snow removal necessity judgment result output from the judgment unit 120 so that consistent snow removal judgment results are obtained for each certain section, taking into consideration the snow removal necessity judgment results in the area surrounding the location or time at which the judgment result was obtained.

[0053] The adjustment unit 150 may be provided outside the information processing device 100. In this case, the information processing device 100 and the adjustment unit 150 are connected via a network, and the snow removal necessity determination result obtained by the determination unit 120 of the information processing device 100 is transmitted to the adjustment unit 150, and the final determination result is output from the adjustment unit 150. When the adjustment unit 150 is provided outside the information processing device 100, the adjustment unit 150 may be called an adjustment device. Even when the adjustment unit 150 is provided outside the information processing device 100, the configuration including the determination unit 120 and the adjustment unit 150 may be called an "information processing device." Specific processing contents of the adjustment unit 150 are as follows.

[0054] (Operation of adjustment unit 150) The adjustment unit 150 stores the snow removal necessity determination results for various points (geographical locations) and various times output from the determination unit 120. For example, the adjustment unit 150 stores the snow removal necessity determination results for multiple points at each time (times at predetermined time intervals from the past to the present) obtained from multiple images taken by on-board cameras of multiple vehicles. In other words, the adjustment unit 150 stores the snow removal necessity determination results in association with the geographical information from which the determination results were obtained and the time the images were taken.

[0055] In the third embodiment, it is assumed that not only the image to be evaluated but also metadata including the image capture time and geographical information is input to the judgment unit 120, and that the metadata is output from the judgment unit 120 together with the judgment result and input to the adjustment unit 150.

[0056] The adjustment unit 150 adjusts the snow removal necessity determination result obtained by the determination unit 120 based on the snow removal necessity determination results of the geographical vicinity of the point where the snow removal necessity determination result was obtained.

[0057] More specifically, the adjustment unit 150 searches for snow removal necessity determination results at locations in the vicinity of the geographical location linked to the snow removal necessity determination result (referred to as determination result A) obtained by the determination unit 120, and obtains a set of snow removal necessity determination results whose image shooting time is within a certain period of time (within a predetermined time range) of the image shooting time of the image linked to determination result A.

[0058] Adjustment unit 150 compares each snow removal necessity determination result in the set with determination result A, and if the number of snow removal necessity determination results in the set that differ from determination result A is equal to or greater than a threshold, adjustment unit 150 changes determination result A to align it with the snow removal necessity determination results of the neighboring area. For example, if determination result A indicates that snow removal is necessary but most of the determination results in the neighboring area indicate that snow removal is not necessary, adjustment unit 150 changes determination result A to indicate that snow removal is not necessary.

[0059] Any known method, such as the technique disclosed in Non-Patent Document 3, can be used to search for the nearby region. Any criterion can be used to set the threshold. For example, the number of judgment results, the proportion of judgment results in the set, or any other value can be set as the threshold. The nearby region may also be a region that is within a predetermined distance from the target position.

[0060] By providing the adjustment unit 150 as in the third embodiment, it is possible to refer to the results of the snow removal necessity determination over a wide area, and to make an appropriate snow removal necessity determination at each location. In addition, it is possible to check the detected snow removal necessity determination results from a bird's-eye view in units of a certain number of blocks or roads, and to actually prioritize the snow removal locations after taking into consideration the temporal locality and spatial continuity of the locations where snow removal is required. The hardware configuration and other elements common to the first to third embodiments will be described below.

[0061] (Example of device hardware configuration) The information processing device 100 described in this embodiment can be realized, for example, by causing a computer to execute a program.

[0062] That is, the information processing device 100 can be realized by using hardware resources such as a CPU and memory built into a computer to execute a program corresponding to the processing performed by the information processing device 100. The program can be recorded on a computer-readable recording medium (such as a portable memory) and can be saved or distributed. The program can also be provided via a network such as the Internet or email.

[0063] Fig. 10 is a diagram showing an example of the hardware configuration of the computer. The computer in Fig. 10 includes a drive device 1000, an auxiliary storage device 1002, a memory device 1003, a CPU 1004, an interface device 1005, a display device 1006, an input device 1007, an output device 1008, and the like, all of which are interconnected via a bus B.

[0064] A program for realizing processing on the computer is provided by a recording medium 1001 such as a CD-ROM or a memory card. When the recording medium 1001 storing the program is set in the drive device 1000, the program is installed from the recording medium 1001 to the auxiliary storage device 1002 via the drive device 1000. However, the program does not necessarily have to be installed from the recording medium 1001, but may be downloaded from another computer via a network. The auxiliary storage device 1002 stores the installed program as well as necessary files, data, etc.

[0065] The memory device 1003 reads and stores the program from the auxiliary storage device 1002 when an instruction to start the program is received. The CPU 1004 realizes functions related to the information processing device 100 in accordance with the program stored in the memory device 1003. The interface device 1005 is used as an interface for connecting to a network, etc. The display device 1006 displays a GUI (Graphical User Interface) or the like according to the program. The input device 1007 is composed of a keyboard, mouse, buttons, a touch panel, etc., and is used to input various operation instructions. The output device 1008 outputs the results of calculations.

[0066] (Effects of the embodiment) As explained above, the technology according to this embodiment focuses on the rut area in the image, making it possible to determine the need for snow removal with higher accuracy than conventional technologies. Furthermore, conventional technologies such as Non-Patent Document 1 are based on the assumption that images are captured using a surveillance camera, which limits the locations that can be evaluated. However, the technology according to this embodiment does not limit the areas that can be evaluated, making it possible to evaluate the need for snow removal over a wide area. Furthermore, by using the technology of the third embodiment, it is possible to obtain highly reliable snow removal need determination results that are consistent in time and space.

[0067] The technology according to this embodiment is expected to automate the monitoring of snowy roads, reducing labor costs and improving the efficiency of snow removal request work.

[0068] The following additional notes are provided regarding the above-described embodiments.

[0069] <Additional Notes> (Additional note 1) A determination unit that receives an image of a snowy road as an input and determines whether snow removal is required on the snowy road using an output from a model that identifies rut ​​areas on the snowy road. An information processing device comprising: (Additional note 2) The model is a semantic segmentation model. Item 1. An information processing device according to item 1. (Additional note 3) the determination unit includes the model and an estimation unit; The estimation unit determines whether snow removal is necessary using a mask image output from the model and the image. Item 1. An information processing device according to item 1. (Additional note 4) the determination unit includes the model and an estimation unit; The estimation unit determines whether snow removal is necessary using the feature amount of the image, which is an intermediate output from the model. Item 1. An information processing device according to item 1. (Additional note 5) an adjustment unit that adjusts the specific snow removal necessity determination result output from the determination unit based on the snow removal necessity determination results at multiple locations and multiple times; Item 1. The information processing device according to item 1, further comprising: (Additional note 6) The adjustment unit adjusts the specific snow removal necessity determination result by comparing a set of snow removal necessity determination results in a vicinity area of ​​a point where an image linked to the specific snow removal necessity determination result was captured with the specific snow removal necessity determination result. Item 5. An information processing device according to item 5. (Additional note 7) A determination method executed by an information processing device, a determining step of determining whether snow removal is required on the snowy road using an image of the snowy road as an input and an output from a model that identifies rut ​​areas on the snowy road; A determination method comprising: (Additional note 8) A program for causing a computer to function as each unit in the information processing device according to any one of claims 1 to 6.

[0070] Although the present embodiment has been described above, the present invention is not limited to such a specific embodiment, and various modifications and changes are possible within the scope of the gist of the present invention described in the claims. [Explanation of symbols]

[0071] 100 Information processing device 110 Input section 120 Judgment section 121 Semantic Segmentation Department 1. Backbone 2 Pixel Decoder 3 Transformer Decoder 122 Estimation Department 123 Joint 130 Output section 140 Learning Department 150 Adjustment section 200 Network 300 vehicles 310 In-vehicle camera 1000 Drive Device 1001 Recording media 1002 Auxiliary storage device 1003 Memory device 1004 CPU 1005 Interface device 1006 Display device 1007 Input Device 1008 Output Device

Claims

1. A determination unit that receives an image of a snowy road as an input and determines whether snow removal is required on the snowy road using an output from a model that identifies rut ​​areas on the snowy road. An information processing device comprising:

2. The model is a semantic segmentation model. The information processing device according to claim 1 .

3. the determination unit includes the model and an estimation unit; The estimation unit determines whether snow removal is necessary by using a mask image output from the model and the image. The information processing device according to claim 1 .

4. the determination unit includes the model and an estimation unit; The estimation unit determines whether snow removal is necessary using the feature amount of the image, which is an intermediate output from the model. The information processing device according to claim 1 .

5. an adjustment unit that adjusts the specific snow removal necessity determination result output from the determination unit based on the snow removal necessity determination results at multiple locations and multiple times; The information processing device according to claim 1 , further comprising:

6. The adjustment unit adjusts the specific snow removal necessity determination result by comparing a set of snow removal necessity determination results in a vicinity area of ​​a point where an image linked to the specific snow removal necessity determination result was captured with the specific snow removal necessity determination result. The information processing device according to claim 5 .

7. A determination method executed by an information processing device, a determining step of determining whether snow removal is required on the snowy road using an image of the snowy road as an input and an output from a model that identifies rut ​​areas on the snowy road; A determination method comprising:

8. A program for causing a computer to function as each unit in the information processing device according to any one of claims 1 to 6.

Images