Information notification system

Abstract

To inhibit occurrence of a situation where an erroneous traffic rule concerning a travel road of a vehicle is notified to a user of the vehicle.SOLUTION: When a traffic sign is not recognized from image information, a traffic rule to be displayed on a display device is generated based on a travel road sign representing a traffic sign which is contained in map information and is installed on a travel road of a vehicle. When the traffic sign is recognized from the image information, the traffic rule is generated based on an image recognition sign representing a traffic sign contained in image information. Whether the travel road sign is correct is determined based on the image recognition sign. When it is determined that the travel road sign is erroneous, compared with when it is determined that the travel road sign is correct, a distance or time over which the traffic rule is kept being outputted to the display device is prolonged.SELECTED DRAWING: Figure 4

Description

Technical Field

[0001] The present disclosure relates to a device for notifying a user of a vehicle of traffic rules regarding a road (travel road) on which the vehicle travels.

Background Art

[0002] Japanese Unexamined Patent Application Publication No. 2022-096906 discloses a driving support system. This conventional system collates a road sign recognized from image information of a road on which a vehicle travels with a road sign provided from a driving support device, and outputs the collation result of these road signs to a display device. In the conventional system, when two types of road signs do not match, the road sign recognized from the image information is output as the collation result. When the two types of road signs match, the matching road sign is output as the collation result.

Prior Art Documents

Patent Documents

[0003]

Patent Document 1

Summary of the Invention

Problems to be Solved by the Invention

[0004] In the conventional system, it is not specified whether the road sign output to the display device when two types of road signs match is the one recognized from the image information or the one provided from the driving support device. Therefore, consider the case where the former is output. In this case, regardless of whether the two types of road signs match or not, the road sign recognized from the image information is output to the display device as the collation result. Therefore, when the road sign recognized from the image information is correct, the correct traffic rules regarding the road on which the vehicle travels can be notified to the driver of the vehicle.

[0005] However, in the conventional system, there is no mention of the time for continuously outputting the road signs recognized from the image information to the display device. Therefore, if the duration of outputting the road signs recognized from the image information is short, the road signs provided by the driving support device may be output from the display device as the duration ends. Then, if the road signs recognized from the image information are correct, it may notify the driver of the vehicle of incorrect traffic rules regarding the driving road of the vehicle, confusing the driver of the vehicle.

[0006] One object of the present disclosure is to provide an information notification device that suppresses a situation in which incorrect traffic rules regarding the driving road of a vehicle are notified to the user of the vehicle.

Means for Solving the Problem

[0007] The present disclosure is a device that notifies a user of a vehicle of traffic rules regarding the driving road of the vehicle, and has the following features. The device includes a camera, a storage device, and a processor. The camera is mounted on the vehicle. The camera also acquires image information around the driving road. Map information including traffic signs provided on the road is stored in the storage device. The processor outputs the traffic rules to a display device. When no traffic sign is recognized from the image information, the processor generates the traffic rules based on the driving road signs indicating the traffic signs provided on the driving road, which are traffic signs included in the map information. When a traffic sign is recognized from the image information, the processor generates the traffic rules based on the image recognition signs indicating the traffic signs recognized from the image information. When a traffic sign is recognized from the image information, the processor further determines whether the driving road sign is correct based on the image recognition sign, sets the distance or time for continuously outputting the traffic rules generated based on the image recognition sign to the display device, and when it is determined that the driving road sign is incorrect, extends the distance or time for continuously outputting the traffic rules to the display device compared to when it is determined that the driving road sign is correct.

Advantages of the Invention

[0008] According to the present disclosure, when a traffic sign is recognized from image information, it is determined whether the driving road sign is correct based on the image recognition sign. And when it is determined that the driving road sign is incorrect, the distance or time for continuously outputting the traffic rules to the display device is extended compared to the case where it is determined that the driving road sign is correct. That is, when it is determined that the driving road sign is incorrect, the traffic rules generated based on the image recognition sign are output to the display device over a longer distance or for a longer time. Therefore, compared with the case where the distance or time for continuously outputting the traffic rules to the display device is made equal between the case where it is determined that the driving road sign is incorrect and the case where it is determined that the driving road sign is correct, it is possible to suppress the situation where incorrect traffic rules regarding the driving road are notified to the user of the vehicle.

Brief Description of the Drawings

[0009]

Figure 1

Figure 2

Figure 3

Figure 4

Figure 5

Modes for Carrying Out the Invention

[0010] Hereinafter, embodiments of the present disclosure will be described with reference to the drawings. However, the structures and the like described in the following embodiments are not necessarily essential to the present disclosure, except when specifically stated or clearly specified by the principle.

[0011] 1. Configuration Example of Information Notification Device FIG. 1 is a conceptual diagram for explaining a configuration example of an information notification device according to an embodiment. The information notification device 2 notifies information to a user US of the vehicle 1. The user US is, for example, an operator of the vehicle 1. The operator of the vehicle 1 may be a driver riding in the vehicle 1 or a remote operator who remotely operates (remote driving, remote assistance) the vehicle 1. The vehicle 1 may be a manually driven vehicle or an autonomously driven vehicle. The information notification device 2 may be mounted on the vehicle 1 or may be included in a remote operator terminal operated by a remote operator. A part of the information notification device 2 may be included in a cloud server.

[0012] The information notification device 2 includes an information processing device 3, a display device 4, and a map DB (database) 5. The information processing device 3 includes at least one processor 31 and at least one storage device 32. The processor 31 executes various information processes. Examples of the processor 31 include a CPU (Central Processing Unit), a GPU (Graphics Processing Unit), ASICs (Application Specific Integrated Circuits), an FPGA (Field-Programmable Gate Array), and the like. The storage device 32 stores (stores) various information. Examples of the storage device 32 include a volatile memory, a non-volatile memory, an HDD (Hard Disk Drive), an SSD (Solid State Drive), and the like.

[0013] The information processing device 3 acquires image information IMG around the traveling road DR of the vehicle 1 from a camera 11 mounted on the vehicle 1. The information processing device 3 also recognizes a traffic sign RS included in the image information IMG by analyzing the image information IMG. When the traffic sign RS is recognized, the information processing device 3 outputs a traffic rule TRR indicated by the recognized traffic sign RS (hereinafter, also referred to as an “image recognition sign IRS”) to the display device 4.

[0014] The information processing device 3 acquires position information POS indicating the position (latitude and longitude) of the vehicle 1 from a GNSS (Global Navigation Satellite System) sensor 12 mounted on the vehicle 1. The information processing device 3 also outputs, to the display device 4, traffic rules TRR indicated by traffic signs RS (hereinafter, also referred to as "driving road signs DRS") provided on the driving road DR, based on the history of the position information POS and the map information MAP stored in the map DB5.

[0015] The display device 4 displays the traffic rules TRR received from the information processing device 3. The display device 4 may be a display mounted on the vehicle 1 or a head-up display (HUD), or may also be a display of a remote operator terminal. The traffic rules TRR are information on traffic rules regarding the driving road DR. Examples of the traffic rules TRR include an image of the traffic sign RS, an icon of the traffic sign RS, or the content indicated by the traffic sign RS. When the image recognition sign IRS and the driving road sign DRS are obtained, the display device 4 preferentially displays the traffic rules TRR indicated by the image recognition sign IRS. When only the driving road sign DRS is obtained, the display device 4 displays the traffic rules TRR indicated by the driving road sign DRS.

[0016] The map DB5 is formed in a predetermined storage device (for example, a non-volatile recording medium such as a hard disk or a flash memory). Map information MAP is stored in the map DB5. The map information MAP includes a general navigation map. Examples of the information included in the navigation map are information on the position of roads, information on the shape of roads (for example, types of curves and straight lines), and information on the positions of intersections and structures. The structures include the driving road signs DRS.

[0017] 2. Functional configuration example of the information processing device FIG. 2 is a block diagram for explaining a functional configuration example of the information processing apparatus 3 shown in FIG. 1. In the example shown in FIG. 2, the information processing apparatus 3 includes an IRS acquisition unit 33, a DRS acquisition unit 34, a difference determination unit 35, a temporary regulation identification unit 36, an error extraction unit 37, and a display control unit 38. These functional blocks are realized, for example, by the cooperation of the processor 31 and the storage device 32.

[0018] The IRS acquisition unit 33 acquires an image recognition label IRS based on the image information IMG. The IRS acquisition unit 33 recognizes the traffic sign RS from the image information IMG, for example, by using image recognition AI (Artificial Intelligence). The image recognition AI is generated in advance through a learning method such as deep learning. The image recognition AI is trained to detect various traffic signs RS from the image information IMG and further identify the types of the detected traffic signs RS. The IRS acquisition unit 33 may read the speed limit specified by the recognized speed limit sign by image analysis. When the image recognition label IRS is acquired, the IRS acquisition unit 33 outputs it to the difference determination unit 35.

[0019] The DRS acquisition unit 34 acquires a driving road sign DRS based on the position information POS and the map information MAP. The DRS acquisition unit 34 identifies the current position and the driving road DR of the vehicle 1 from the history of the position information POS, for example. When the current position and the driving road DR are identified, the DRS acquisition unit 34 acquires the traffic sign RS (that is, the driving road sign DRS) provided on the driving road DR by referring to the map information MAP using the current position and the driving road DR. When the driving road sign DRS is acquired, the DRS acquisition unit 34 outputs it to the difference determination unit 35 and the display control unit 38.

[0020] The differential determination unit 35 determines whether there is a difference between the image recognition sign IRS and the driving road sign DRS. For example, the differential determination unit 35 calculates the similarity between the image recognition sign IRS and the driving road sign DRS. When the similarity is less than the threshold value, it is determined that there is a difference between the image recognition sign IRS and the driving road sign DRS. When it is determined that there is a difference, the differential determination unit 35 outputs the image recognition sign IRS and the driving road sign DRS to the temporary regulation identification unit 36.

[0021] The temporary regulation identification unit 36 determines whether the image recognition sign IRS corresponds to a temporary regulation sign. For example, the temporary regulation identification unit 36 determines whether the image recognition sign IRS corresponds to an electric light sign by using, for example, image recognition AI. When it is determined that the image recognition sign IRS corresponds to an electric light sign, the temporary regulation identification unit 36 determines that the reason for the difference between the image recognition sign IRS and the driving road sign DRS is due to a temporary regulation sign. When it is determined otherwise, the temporary regulation identification unit 36 outputs the image recognition sign IRS and the driving road sign DRS to the error extraction unit 37.

[0022] When the image recognition sign IRS and the driving road sign DRS are input, the error extraction unit 37 determines that there is an error in the driving road sign DRS. In this case, information about the road (for example, road ID) where the driving road sign DRS determined to have an error is provided is recorded in the storage device 32 together with the cumulative number of determination times CN. When it is determined that there is an error, the error extraction unit 37 outputs the image recognition sign IRS to the display control unit 38 together with the error detection signal of the driving road sign DRS. When it is determined that there is no error, the error extraction unit 37 outputs the image recognition sign IRS to the display control unit 38.

[0023] The display control unit 38 performs control (display control) to display the traffic rule TRR on the display device 4. As already described, the traffic rule TRR is information on the traffic rules regarding the driving road DR. The image recognition sign IRS, the driving road sign DRS, and an error detection signal can be input to the display control unit 38. When only the driving road sign DRS is input, the display control unit 38 displays the traffic rule TRR indicated by the driving road sign DRS on the display device 4. When the image recognition sign IRS and the driving road sign DRS are input, the display control unit 38 displays the traffic rule TRR indicated by the image recognition sign IRS on the display device 4. When the error detection signal is input, the display control unit 38 extends the distance or time for which it continues to display the traffic rule TRR indicated by the image recognition sign IRS on the display device 4.

[0024] Figure 3 is a diagram for explaining the distance or time for which the traffic rule TRR is continuously displayed. In the upper part of Figure 3, case (i) where this distance or time is constant is depicted. In the lower part of Figure 3, case (ii) where this distance or time is extended is depicted. Note that in cases (i) and (ii), it is determined that the image recognition sign IRS (driving speed 50 km / s) is correct and there is an error in the driving road sign DRS (driving speed 60 km / s).

[0025] In case (i), that is, when the distance or time for which the traffic rule TRR indicated by the image recognition sign IRS is continuously displayed is constant, when the display of the correct information (that is, the traffic rule TRR indicated by the image recognition sign IRS) ends, the display of the incorrect information (that is, the traffic rule TRR indicated by the driving road sign DRS) is started. This switching of the display is also assumed in case (ii). However, in case (ii), the distance or time for which the correct information is continuously displayed is extended.

[0026] In this way, according to the embodiment, it is possible to suppress the situation in which incorrect information is notified to the user US.

[0027] 3. Information processing example Figures 4 and 5 are flowcharts showing processing examples particularly related to the embodiments. The flowcharts shown in FIGS. 4 and 5 are repeatedly executed by the processor 31 shown in FIG. 1 at a predetermined control period.

[0028] In the routine shown in FIG. 4, first, a driving road sign DRS is acquired (step S11). The driving road sign DRS is acquired based on the position information POS and the map information MAP. For example, the current position of the vehicle 1 and the driving road DR are specified from the history of the position information POS. Subsequently, the driving road sign DRS is acquired by referring to the map information MAP using this current position and the driving road DR.

[0029] Subsequent to the processing of step S11, it is determined whether an image recognition sign IRS has been acquired (step S12). The image recognition sign IRS is acquired when a traffic sign RS is recognized from the image information IMG. That is, when the traffic sign RS is not recognized, the image recognition sign IRS is not acquired.

[0030] When the determination result in step S12 is negative, display control based on the driving road sign DRS is performed (step S13). The fact that the determination result in step S12 is negative means that only the driving road sign DRS has been acquired. Therefore, when display control is performed in step S13, the traffic rule TRR indicating the driving road sign DRS is displayed on the display device 4.

[0031] When the determination result in step S12 is positive, it is determined whether there is a difference between the image recognition sign IRS and the driving road sign DRS (step S14). For example, when the similarity between the image recognition sign IRS and the driving road sign DRS is less than a threshold value, it is determined that there is a difference between these signs. When the determination result in step S14 is positive, it is determined whether the image recognition sign IRS is an electric light sign (step S15).

[0032] If the determination result in step S14 is negative, or if the determination result in step S15 is positive, display control based on the image recognition sign IRS is performed (step S16). The fact that the determination result in step S14 is negative means that there is no difference between the image recognition sign IRS and the driving road sign DRS. Therefore, when display control is performed in step S16, the traffic rule TRR indicating the image recognition sign IRS is displayed on the display device 4. The fact that the determination result in step S15 is positive means that the image recognition sign IRS is an electroluminescent sign. Therefore, when display control is performed in step S16, the traffic rule TRR indicating the electroluminescent sign is displayed on the display device 4. In the display control of this step S16, the normal value α is used for the distance or time to continue displaying the traffic rule TRR indicated by the image recognition sign IRS.

[0033] If the determination result in step S15 is negative, it is determined that there is an error in the driving road sign DRS (step S17). Then, display control based on the image recognition sign IRS is performed (step S18). In the display control of this step S18, the extended value β (>α) is used for the distance or time to continue displaying the traffic rule TRR indicated by the image recognition sign IRS.

[0034] In the routine shown in FIG. 5, first, error detection information of the driving road sign DRS is collected (step S21). The error detection information is specified, for example, by the presence or absence of an error detection signal generated when it is determined that there is an error in the driving road sign DRS.

[0035] Subsequent to the process of step S21, it is determined whether or not the cumulative number of times CN of this determination exceeds a threshold TH (for example, TH ≥ 2) for the same traveling road sign DRS for which an error has been determined (step S22). Here, when the traffic sign RS corresponds to an electric light sign, an error detection signal is not generated. Also, when the recognition of the traffic sign RS is incorrect, there is a possibility that an error detection signal is not generated. However, the fact that the cumulative number CN exceeds the threshold TH means that there is a high possibility of an error in the traveling road sign DRS even considering the case where an error detection signal is not generated.

[0036] Therefore, when the determination result of step S22 is affirmative, the traveling road sign DRS is updated (step S23). When the traveling road sign DRS is updated, the traveling road sign DRS included in the map information MAP is rewritten. The rewriting of the traveling road sign DRS is performed, for example, using the traffic rule TRR indicated by the image recognition sign IRS compared with this traveling road sign DRS.

[0037] In the process of step S23, instead of updating the traveling road sign DRS, the map information MAP regarding the information of the road on which the traveling road sign DRS is provided may be updated. In this case, for example, the information processing device 3 communicates with an external server to inquire about the presence or absence of the latest information regarding the road on which the traveling road sign DRS is provided. If there is the latest information, the latest map information MAP is acquired from the map DB possessed by the external server, and the map information MAP stored in the map DB5 is rewritten using this latest map information MAP.

[0038] When the map DB5 is provided in the vehicle 1 and the information processing device 3 cannot communicate with the external server, it is also possible to perform a process of displaying, on the display device 4, information prompting the user US to rewrite the traveling road sign DRS (or rewrite the information of the road on which the traveling road sign DRS is provided) in the process of step S23. As the information prompting the rewriting, guidance information of a rewriting service by the dealer of the vehicle 1 is exemplified.

Explanation of Signs

[0039] 1... Vehicle, 11... Camera, 12... GNSS Sensor, 2... Information Notification Device, 3... Information Processing Device, 4... Display Device, 5... Map DB, 31... Processor, 32... Memory Device, DR... Driving Road, RS... Traffic Sign, US... User, DRS... Driving Road Sign, IMG... Image Information, IRS... Image Recognition Sign, MAP... Map Information, POS... Position Information, TRR... Traffic Rule

Claims

1. An apparatus for notifying a user of a vehicle of traffic rules regarding a driving road of the vehicle, comprising: a camera mounted on the vehicle for acquiring image information around the driving road; a storage device storing map information including traffic signs provided on the road; a processor for outputting the traffic rules to a display device; wherein the processor when no traffic sign is recognized from the image information, generates the traffic rules based on a driving road sign which is a traffic sign included in the map information and provided on the driving road; when a traffic sign is recognized from the image information, generates the traffic rules based on an image recognition sign indicating the traffic sign recognized from the image information; the processor further when a traffic sign is recognized from the image information, determines whether the driving road sign is correct based on the image recognition sign; sets a distance or time for continuously outputting the traffic rules generated based on the image recognition sign to the display device; when it is determined that the driving road sign is incorrect, extends the distance or time for continuously outputting the traffic rules to the display device as compared with the case where it is determined that the driving road sign is correct An information notification apparatus characterized by the above.

2. The information notification apparatus according to claim 1, wherein the processor further determines whether the image recognition sign is a temporary traffic control sign; when it is determined that the image recognition sign is the temporary traffic control sign, does not perform the determination of the correctness of the driving road sign based on the image recognition sign An information notification apparatus characterized by the above.

3. The information notification apparatus according to claim 1 or 2, wherein the processor further when at least two determinations that the driving road sign is incorrect are obtained for the same driving road sign, updates the same driving road sign stored in the storage device based on the image recognition sign used for the determination of the correctness of the same driving road sign An information notification apparatus characterized by the above.

4. The information notification apparatus according to claim 1 or 2, wherein the vehicle is configured to be communicable with an external storage device provided outside the vehicle and storing map information including traffic signs provided on the road; the processor further When the determination that the driving road sign is incorrect is obtained at least twice for the same driving road sign, the latest map information including traffic signs provided on the road where the same driving road sign is provided is acquired from the external storage device. The map information stored in the storage device is updated using the latest map information. An information notification device characterized by the above.

5. The information notification device according to claim 1 or 2, wherein the processor further when the determination that the driving road sign is incorrect is obtained at least twice for the same driving road sign, outputs information prompting an update of the same driving road sign to a display device. An information notification device characterized by the above.

Images