Information processing device

Abstract

To provide an information processing device capable of detecting a road sign in an image with high accuracy.SOLUTION: An information processing device includes: an image processing unit that detects a road sign from an image captured by an on-vehicle camera, and calculates a confidence indicating the likelihood of the road sign being of a specific type; and a determination processing unit that, when the reliability is less than a threshold value, determines the presence or absence of a road sign of the specific type in the image by referencing map information in which positions of road signs of the specific type on a map are indicated on the basis of the position at which the image was captured.SELECTED DRAWING: Figure 5

Description

[Technical Field]

[0001] The present invention relates to an information processing device. [Background technology]

[0002] For example, Patent Document 1 describes determining the road sign corresponding to the current position from either an image of the road sign or location information. [Prior art documents] [Patent documents]

[0003] [Patent Document 1] Japanese Patent Application Publication No. 2019-212188 Summary of the Invention [Problem to be solved by the invention]

[0004] For example, in automobile driving diagnosis, road signs within the driver's field of vision must be identified, so road signs are detected using object detection technology that utilizes artificial intelligence (AI) from images taken with an in-vehicle camera rather than using location information for road signs. However, depending on the performance of the in-vehicle camera and environmental conditions such as the weather at the time of shooting, there is a risk of false detection of road signs.

[0005] SUMMARY OF THE INVENTION The present invention has been made in view of the above-mentioned problems, and has as its object to provide an information processing device that can detect road signs in an image with high accuracy. [Means for solving the problem]

[0006] The information processing device of the present invention has an image processing unit that detects road signs from an image captured by an on-board camera and calculates a reliability indicating the likelihood that the road sign is of a specific type, and a determination processing unit that, if the reliability is less than a threshold, determines whether or not a road sign of the specific type is present in the image by referring to map information indicating the positions of road signs of the specific type on a map based on the position at which the image was captured. The determination processing unit estimates the traveling direction of the vehicle equipped with the on-board camera based on the change over time in the position where the image was taken, and if the reliability is less than a threshold, determines that the specific type of road sign is present in the image when the specific type of road sign is present on the map in the traveling direction, and determines that the specific type of road sign is not present in the image when the specific type of road sign is not present in the traveling direction. .

[0008] In the above information processing device, the image processing unit may detect the road sign from the image using a detection model that has been machine-learned through supervised learning, which receives the image and the type of the road sign as input and outputs the specific type of road sign.

[0009] In the information processing device, the determination processing unit may determine that the specific type of road sign is present in the image when the reliability is equal to or greater than a threshold value.

[0010] In the above-mentioned information processing device, if the judgment processing unit determines that the specific type of road sign is present in the image, the information processing device may have an evaluation processing unit that evaluates the driving of the vehicle equipped with the on-board camera based on the rules indicated by the specific type of road sign. [Effects of the Invention]

[0011] According to the present invention, road signs in an image can be detected with high accuracy. [Brief explanation of the drawings]

[0012] [Figure 1] FIG. 1 is a configuration diagram showing an example of a driving diagnosis system. [Figure 2] FIG. 2 is a diagram showing an example of a road sign detected from an image. [Figure 3] FIG. 3 is a diagram showing an example of an image and map information in which a "stop" road sign is present. [Figure 4]FIG. 4 is a diagram showing an example of an image and map information in which there is no "stop" road sign. [Figure 5] FIG. 5 is a flowchart showing an example of the operation of the driving diagnosis device. DETAILED DESCRIPTION OF THE INVENTION

[0013] (Configuration of driving diagnosis system) 1 is a configuration diagram showing an example of a driving diagnosis system 9. The driving diagnosis system 9 includes a driving diagnosis device 1 that evaluates the driving of a vehicle 90, and a map information server 91 that holds a map information database 910 that shows the positions of road signs. The driving diagnosis device 1 is an example of an information processing device, and is, for example, a computer such as a server.

[0014] The map information server 91 provides map information from a map information database (map information DB) 910 via the Internet NW. The map information server 91 provides map information according to the position of the vehicle 90 transmitted from the driving diagnosis device 1. The map information indicates the positions of road signs on the road. An example of a service provided by the map information server 91 is map matching. Map matching identifies a travel route on the road from the position of the vehicle 90.

[0015] The vehicle 90 is, for example, a hybrid vehicle or a gasoline engine vehicle, but is not limited to these. The vehicle 90 is equipped with a drive recorder 900, an on-board camera 901, and a GPS (Global Positioning System) 902.

[0016] The in-vehicle camera 901 captures images in the traveling direction of the vehicle 90. The GPS 902 detects the latitude and longitude as the position of the vehicle 90. The GPS 902 may be built into the drive recorder 900. The drive recorder 900 records the images captured by the in-vehicle camera 901 together with the position information detected by the GPS 902 in a recording medium 900a such as a memory card. The position information is an example of the position where the image was captured.

[0017] The driving diagnosis device 1 evaluates the driving of the vehicle 90 based on image data and position information recorded by the drive recorder 900. The driving diagnosis device 1 has a CPU (Central Processing Unit) 10, a ROM (Read Only Memory) 11, a RAM (Random Access Memory) 12, an HDD (Hard Disk Drive) 13, a communication port 14, and an I / O port 15. The CPU 10 is electrically connected to the ROM 11, the RAM 12, the HDD 13, the communication port 14, and the I / O port 15 via a bus 19 so as to be able to input and output signals to and from each of them.

[0018] The ROM 11 stores a program that drives the CPU 10. The RAM 12 functions as a working memory for the CPU 10. The communication port 14 is, for example, a communication circuit for a wireless LAN (Local Area Network), and performs communication processing with the CPU 10 via the Internet NW. The I / O port 15 is connected to a recording medium 900a such as a memory card, and performs input / output processing of information to and from the CPU 10.

[0019] When the CPU 10 reads a program from the ROM 11, it generates, as functions, an operation control unit 100, a data acquisition unit 101, an object detection unit 102, a sign determination unit 103, and a driving evaluation unit 104. In addition, the HDD 13 stores image data 130, vehicle position data 131, and evaluation data 132.

[0020] The operation control unit 100 controls the overall operation of the driving diagnosis device 1. The operation control unit 100 instructs the data acquisition unit 101, the object detection unit 102, the sign determination unit 103, and the driving evaluation unit 104 to perform various operations according to a predetermined sequence.

[0021] The data acquisition unit 101 acquires image data and location information from, for example, the recording medium 900a of the drive recorder 900 via the I / O port 15. The image data and location information are provided with shooting time information indicating the time when the image was taken. The data acquisition unit 101 stores the image as image data 130 and the location information as vehicle position data 131 in the HDD 13 for each shooting time.

[0022] The object detection unit 102 is an example of an image processing unit. The object detection unit 102 detects road signs from the image data 130 using an AI-based object detection technology, and calculates the reliability (score) of the detection. As an example, the object detection unit 102 calculates a reliability indicating the likelihood that the road sign is a "stop" sign. Note that the "stop" road sign is an example of a specific type of road sign, but is not limited to this, and other types of road signs may also be detected.

[0023] FIG. 2 is a diagram showing an example of a road sign detected from an image. The object detection unit 102 surrounds the object to be detected with a rectangular bounding box BOX. The object detection unit 102 calculates a reliability indicating the likelihood that the object within the bounding box BOX is a "stop" road sign. The reliability is calculated according to a method based on the object detection algorithm. Examples of the algorithm include, but are not limited to, R-CNN (Region-Convolutional Neural Network).

[0024] The object detection unit 102 detects road signs from images using a detection model that has been machine-learned through supervised learning, which takes an image and the type of road sign as input and outputs a "stop" road sign. Specifically, in the learning process for the detection model, the road sign to be detected is specified from the image of the training data, and the name of the road sign is annotated. The detection model is a neural network that mathematically models human brain function, and the weight coefficients of the activation function for the part corresponding to the neuron are determined based on machine learning. Because a detection model that has been machine-learned through supervised learning is used in this way, the object detection unit 102 can quickly detect road signs.

[0025] Referring again to FIG. 1, the sign determination unit 103 is an example of a determination processing unit. If the reliability is equal to or greater than a threshold, the sign determination unit 103 determines that a "stop" road sign is present in the image of the image data 130. Therefore, when the road sign is clearly a "stop" sign, as in the example image shown in FIG. 2, the sign determination unit 103 can determine that a "stop" road sign is present in the image based on the reliability calculated by the object detection unit 102. In this case, the driving evaluation unit 104 evaluates the driving on the premise that the "stop" road sign is within the field of view of the driver of the vehicle 90.

[0026] Furthermore, if the reliability is less than the threshold, the sign determination unit 103 accesses the map information server 91 via the communication port 14 and determines whether or not there is a "stop" road sign in the image by referring to the map information DB 910 based on the vehicle position data 131. Therefore, even if the reliability is lower than the threshold due to the performance of the in-vehicle camera 901 or environmental conditions such as the weather at the time of shooting, it is possible to determine with high accuracy whether or not there is a "stop" road sign in the image based on the position of the "stop" road sign indicated by the map information DB 910.

[0027] FIG. 3 is a diagram showing an example of an image Ga and map information Ma in which a "stop" road sign is present. In image Ga, which shows the area ahead of vehicle 90, a "stop" road sign is present at a T-junction. Object detection unit 102 detects the "stop" road sign from image data 130 and encloses it in a bounding box BOX. However, compared to the image in FIG. 2, the distance from vehicle 90 to the road sign is longer, and the weather is cloudy, making the type of road sign unclear, and object detection unit 102 calculates a reliability level that is lower than the threshold value.

[0028] Because the reliability is less than the threshold, the sign determination unit 103 obtains the position P of the vehicle 90 at the time the image Ga was captured from the vehicle position data 131. The sign determination unit 103 obtains map information Ma corresponding to the position P of the vehicle 90 from the map information DB 910. The map information Ma indicates the position S of a "stop" road sign on the map.

[0029] The sign determination unit 103 estimates the traveling direction d of the vehicle 90 from the change over time in the position P of the vehicle 90 shown in the image at each time. The sign determination unit 103 determines that the position S of a "stop" road sign is present in the traveling direction d of the vehicle 90 based on the map information Ma. Therefore, the sign determination unit 103 determines that the "stop" road sign is within the field of view of the driver of the vehicle 90, and determines that a "stop" road sign is present in the image. The sign determination unit 103 determines the presence of a road sign by performing geometric calculation processing from the position P of the vehicle 90, the traveling direction d, and the position S of the road sign.

[0030] FIG. 4 is a diagram showing an example of an image Gb and map information Mb in which a "stop" road sign is not present. In the image Gb, which shows the area ahead of the vehicle 90, a "left curve" road sign is present at a curve. The object detection unit 102 detects the "left curve" road sign from the image data 130 and encloses it in a bounding box BOX. However, compared to the image in FIG. 2, there is a patch of grass around the "left curve" road sign, making the type of road sign unclear, and the object detection unit 102 calculates a reliability that is lower than the threshold value.

[0031] Because the reliability is less than the threshold, the sign determination unit 103 obtains the position P of the vehicle 90 at the time the image Gb was captured from the vehicle position data 131. The sign determination unit 103 obtains map information Mb corresponding to the position P of the vehicle 90 from the map information DB 910. The map information Mb indicates the position S of a "stop" road sign on the map. In this example, it is assumed that the position S of the "stop" road sign is not within the field of view of the driver at the position P of the vehicle 90.

[0032] The sign determination unit 103 estimates the traveling direction d of the vehicle 90 from the change over time in the position P of the vehicle 90 shown in the image at each time. The sign determination unit 103 determines, based on the map information Mb, that the position S of the "stop" road sign does not exist in the traveling direction d of the vehicle 90. Therefore, the sign determination unit 103 determines that the "stop" road sign is not within the field of view of the driver of the vehicle 90, and determines that there is no "stop" road sign in the image Gb.

[0033] In this way, if a "stop" road sign is present on the map in the traveling direction d of the vehicle 90, the sign determination unit 103 determines that the road sign is present in the image Ga, and if a "stop" road sign is not present in the traveling direction d, the sign determination unit 103 determines that the road sign is not present in the image Gb. Therefore, the sign determination unit 103 can determine the presence or absence of a "stop" road sign in the images Ga and Gb from the change over time in the relative relationship between the position P of the vehicle 90 and the position S of the "stop" road sign. Note that in this example, the sign determination unit 103 references the map information DB 910 of the map information server 91 via the Internet NW, but is not limited to this. For example, the map information DB 910 may be stored in the HDD 13, and the sign determination unit 103 may reference the map information DB 910 in the HDD 13.

[0034] Referring again to FIG. 1 , if the sign determination unit 103 determines that a "stop" road sign is present in the image, the driving evaluation unit 104 evaluates the driving of the vehicle 90 based on the road sign. For example, the driving evaluation unit 104 calculates the speed and direction of movement of the vehicle 90 from the vehicle position data 131, and determines whether the vehicle 90 has sufficiently decelerated a certain distance or more before the "stop" road sign. The driving evaluation unit 104 records the determination result of the driving evaluation as evaluation data 132 in the HDD 13. The evaluation data 132 can be presented to the driver of the vehicle 90 as a report, for example, to contribute to improving safe driving.

[0035] (Operation of driving evaluation device) 5 is a flowchart showing an example of the operation of the driving diagnosis device 1. This operation is executed, for example, when the operation control unit 100 receives a driving evaluation instruction from a terminal (not shown) via the communication port 14.

[0036] First, the data acquisition unit 101 acquires the image data 130 and the vehicle position data 131 from the recording medium 900a of the drive recorder 900 and stores them in the HDD 13 (step St1). Note that the means for acquiring the image data 130 and the vehicle position data 131 is not limited to this, and may be, for example, wireless LAN communication with the vehicle 90 via the communication port 14.

[0037] Next, the object detection unit 102 detects road signs in the image at a predetermined time in the image data 130 using a bounding box BOX (step St2). Next, the object detection unit 102 calculates the reliability of the detected road signs (step St3). It is desirable for the object detection unit 102 to perform detection using AI as described above, but this is not limiting, and other image processing techniques may also be used to detect road signs.

[0038] Next, the sign determination unit 103 compares the number of pixels in the bounding box BOX with a predetermined value K to prevent image noise or objects other than road signs from being mistaken for road signs (step St4). The number of pixels is calculated from the positions of a pair of diagonal corners of the bounding box BOX. If the number of pixels is less than or equal to the predetermined value K (No in step St4), the operation of step St8, which will be described later, is executed.

[0039] If the number of pixels is greater than the predetermined value K (Yes in step St4), the sign determination unit 103 compares the reliability with a threshold value TH (step St5). If the reliability is greater than or equal to TH (Yes in step St5), the sign determination unit 103 determines that a "stop" road sign is present in the image (step St6). In other words, it is determined that a "stop" road sign was within the field of view of the driver of the vehicle 90 at the time the image was captured.

[0040] Next, the driving evaluation unit 104 evaluates the driving in accordance with the "stop" road sign from the position indicated by the vehicle position data 131 (step St7). For example, the driving evaluation unit 104 estimates the speed of the vehicle 90 from the change in the position of the vehicle 90 over time, and determines whether the speed has decelerated to less than a predetermined value by the time the vehicle 90 reaches a position a certain distance before the "stop" road sign. In this example, an evaluation example based on a "stop" road sign has been given, but the deceleration of the vehicle 90 may also be evaluated in a similar manner when it is determined that a road sign indicating, for example, "school, kindergarten, nursery school, etc. nearby" is present.

[0041] In this way, when the sign determination unit 103 determines that a "stop" road sign is present in the image, the driving evaluation unit 104 evaluates the driving of the vehicle 90 based on the rule indicated by the "stop" road sign. The driving evaluation unit 104 stores the evaluation result as evaluation data 132 in the HDD 13. Therefore, the driving diagnosis device 1 can contribute to improving the safe driving of the vehicle 90.

[0042] Next, the operation control unit 100 determines whether to continue the driving diagnosis of the vehicle 90 based on an instruction from a terminal (not shown) (step St8). When ending the driving diagnosis (No in step St8), the driving diagnosis device 1 ends this operation. When continuing the driving diagnosis (Yes in step St8), the driving diagnosis device 1 executes each operation after step St2 for the image at the next time.

[0043] Also, when the reliability is less than (No in step St5), the sign determination unit 103 refers to the map information DB910 based on the position of the vehicle 90 indicated by the vehicle position data 131 (step St9). Next, as described by referring to FIGS. 3 and 4, the sign determination unit 103 determines whether there is a "stop" road sign on the map in the traveling direction of the vehicle 90 based on the reference result of the map information DB910 (step St10).

[0044] When there is a "stop" road sign in the traveling direction of the vehicle 90 (Yes in step St10), the sign determination unit 103 determines that there is a "stop" road sign in the image (step St6). Thereafter, each operation after step St7 is executed.

[0045] When there is no "stop" road sign in the traveling direction of the vehicle 90 (No in step St10), the sign determination unit 103 determines that there is no "stop" road sign in the image (step St11). Thus, when the reliability is less than the threshold value, the sign determination unit 103 can detect the road sign in the image with high accuracy by referring to the map information DB910 based on the position of the vehicle 90. Thereafter, each operation after step St8 is executed. The driving diagnosis device 1 operates as described above.

[0046] As described above, in this example, the driving diagnosis device 1 is used and the sign determination unit 103 determines whether or not there is a "stop" road sign in the image, but this determination process is not limited to driving diagnosis and can also be applied to, for example, autonomous driving. In this case, a computer including the object detection unit 102 and the sign determination unit 103 is mounted on the vehicle 90, and the computer operates the vehicle 90 according to the determination result of the sign determination unit 103. In the case of autonomous driving, the operation of the driving diagnosis device 1 described above is changed as appropriate based on the specifications of the autonomous driving.

[0047] The above-described embodiment is a preferred example of the present invention, but the present invention is not limited to this and can be modified in various ways without departing from the spirit of the present invention. [Explanation of symbols]

[0048] 1 Driving diagnosis device, 10 CPU, 102 object detection unit (image processing unit), 103 sign determination unit (determination processing unit), 104 driving evaluation unit (evaluation unit), 90 vehicle, 900 drive recorder, 901 in-vehicle camera, 902 GPS, 910 map information database

Claims

1. an image processing unit that detects road signs from images captured by an in-vehicle camera and calculates a reliability indicating the likelihood that the road signs are of a specific type; a determination processing unit that, when the reliability is less than a threshold, determines whether or not there is a road sign of the specific type in the image by referring to map information indicating the positions of road signs of the specific type on a map based on the position at which the image was taken, The determination processing unit estimating a traveling direction of the vehicle equipped with the on-board camera based on a change over time in the position where the image was taken; If the reliability is less than a threshold, and if a road sign of the specific type exists on the map and in the traveling direction, it is determined that the road sign of the specific type exists in the image, and if a road sign of the specific type does not exist in the traveling direction, it is determined that the road sign of the specific type does not exist in the image. Information processing device.

2. the image processing unit receives the image and the type of the road sign as input and outputs the specific type of road sign, and detects the road sign from the image using a detection model that has been machine-learned through supervised learning. The information processing device according to claim 1 .

3. The determination processing unit determines that the specific type of road sign is present in the image when the reliability is equal to or greater than the threshold. The information processing device according to claim 1 .

4. and an evaluation processing unit that, when the determination processing unit determines that the specific type of road sign is present in the image, evaluates the driving of the vehicle equipped with the on-board camera based on the rule indicated by the specific type of road sign. The information processing device according to claim 1 .

Images