Information notification system

Abstract

To notify a user of a vehicle of highly accurate information.SOLUTION: An information notification system notifies a user of a vehicle of notification information. First information is notification information obtained as a result of recognition processing based on an image captured by a camera mounted on the vehicle. Second information that is the notification information obtained independently of the recognition processing, is registered in advance in map information. Achievement information includes third information that is the notification information obtained as a result of past recognition processing and information reflecting freshness of the third information. When the first information is not acquired and priority conditions are satisfied, the information notification system notifies the user of not the second information but the third information. The priority conditions are based on the freshness of the third information, and at least includes that the third information is newer than the second information.SELECTED DRAWING: Figure 1

Description

Technical Field

[0001] The present disclosure relates to a technique for notifying information to a user of a vehicle.

Background Art

[0002] Patent Document 1 discloses a driving support system. The driving support system updates map information based on the driving trajectory information (probe data) acquired from a measurement vehicle (probe car). The map information includes information on road signs. Further, the driving support system provides the map information to vehicles that use the map information.

Prior Art Documents

Patent Documents

[0003]

Patent Document 1

Summary of the Invention

Problems to be Solved by the Invention

[0004] Notifying a user of a vehicle of information such as traffic signs is useful. For example, when map information in which information on traffic signs is registered in advance is available, it is conceivable to notify the user of the information on traffic signs using the map information. However, there are cases where the update of the available map information is delayed, and its accuracy is not necessarily high.

[0005] One object of the present disclosure is to provide a technique capable of notifying a user of a vehicle of highly accurate information.

Means for Solving the Problems

[0006] A first aspect relates to an information notification system that notifies notification information to a user of a vehicle. The information notification system includes one or more processors and one or more storage devices. One or more storage devices are configured to store first information, map information, and performance information. The first piece of information is notification information obtained as a result of recognition processing based on images captured by a camera mounted on the vehicle. Map information includes pre-registered second-party information, which is notification information obtained independently of recognition processing. The performance information includes third-party information, which is notification information obtained as a result of past recognition processing, and information that reflects the freshness of the third-party information. If the first information is not obtained and the first priority condition is met, one or more processors will notify the user of the third information instead of the second information. The first priority criterion is based on the freshness of the third information, and at a minimum includes the fact that the third information is newer than the second information.

[0007] The second perspective, in addition to the first perspective, has the following further characteristics: The performance information further includes the confidence level of the third piece of information. The first priority condition further includes the fact that the confidence level of the third piece of information is equal to or greater than the first threshold.

[0008] The third perspective has the following additional characteristics in addition to those of the first or second perspective: When the first piece of information is obtained, one or more processors notify the user of the obtained first piece of information.

[0009] The fourth perspective, in addition to the second perspective, has the following further characteristics: If the first information is obtained and the second priority condition is met, one or more processors will notify the user of the third information instead of the first information. The second priority condition includes the fact that the confidence level of the third piece of information is equal to or greater than the second threshold, and that the freshness of the third piece of information is equal to or greater than the third threshold.

[0010] The fifth perspective, in addition to the second perspective, has the following further characteristics: One or more processors obtain a confidence level of the first piece of information in the image-based recognition process. If the first information is obtained and the third priority condition is met, one or more processors will notify the user of the third information instead of the first information. The third priority condition includes the fact that the confidence level of the third piece of information is higher than the confidence level of the first piece of information. [Effects of the Invention]

[0011] According to this disclosure, performance data will be introduced. By utilizing this performance data, it will be possible to notify vehicle users of more up-to-date information than map data. In other words, it will be possible to notify vehicle users of more accurate information. [Brief explanation of the drawing]

[0012] [Figure 1] This is a conceptual diagram illustrating the outline of the information notification system according to the embodiment. [Figure 2] This is a block diagram showing an example configuration of an information notification system according to an embodiment. [Figure 3] This flowchart shows the processes related to the accumulation and updating of performance information according to the embodiment. [Figure 4] This is a flowchart showing a first example of the information notification process according to the embodiment. [Figure 5] This flowchart shows a second example of the information notification process according to the embodiment. [Figure 6] This is a flowchart showing a third example of the information notification process according to the embodiment. [Modes for carrying out the invention]

[0013] 1. Overview FIG. 1 is a conceptual diagram for explaining the outline of an information notification system 10 according to the present embodiment. The information notification system 10 notifies information to a user O of the vehicle 1. For example, the user O of the vehicle 1 is an operator of the vehicle 1. The operator of the vehicle 1 may be a driver riding in the vehicle 1, or may be a remote operator who remotely operates (remote driving, remote assistance) the vehicle 1. The user O of the vehicle 1 may be a passenger riding in the vehicle 1. The vehicle 1 may be an autonomous vehicle. The information notification system 10 may be mounted on the vehicle 1, or may be included in a remote operator terminal operated by a remote operator. A part of the information notification system 10 may be included in a cloud server.

[0014] The information notification system 10 includes a notification device for notifying the user O of information. For example, the notification device includes a display device, and the information notification system 10 displays the information on the display device. As another example, the notification device includes a speaker, and the information notification system 10 may output voice information from the speaker.

[0015] The information notified from the information notification system 10 to the user O is hereinafter referred to as "notification information". Typically, the notification information is information related to the road on which the vehicle 1 travels. For example, the notification information is information related to a traffic sign S installed on the road. Examples of the traffic sign S include a speed limit sign, a stop sign, a no-overtaking sign, etc. The information related to the traffic sign S is an image of the traffic sign S, an icon of the traffic sign S, or the content indicated by the traffic sign S. For example, when the traffic sign S is a speed limit sign, the notification information may be the speed limit of that road.

[0016] [[ID=​

[0017] In the following explanation, we consider, as an example, the case where the target is a traffic sign S and the notification information is traffic sign information related to traffic sign S. The same applies to other cases where the target is different.

[0018] Traffic signs S installed on the road can be recognized using a camera 2 mounted on a vehicle 1. More specifically, camera 2 photographs the surroundings of vehicle 1 and acquires an image IMG showing the surroundings of vehicle 1. The information notification system 10 acquires the image IMG obtained by camera 2 and recognizes the traffic sign S by analyzing the image IMG. The information notification system 10 then notifies the user O of the traffic sign information related to the recognized traffic sign S. For example, the information notification system 10 displays an image of the recognized traffic sign S or its icon on a display device. As another example, the information notification system 10 may read the speed limit specified by the recognized speed limit sign and notify the user O of that speed limit.

[0019] The recognition information 110 is real-time information obtained through recognition processing based on an image IMG captured by a camera 2 mounted on the vehicle 1. In particular, the real-time traffic sign information (notification information) obtained as a result of this recognition processing will be referred to below as "first information". The recognition information 110 includes first information.

[0020] Depending on the circumstances, it may be difficult to obtain the primary information. For example, in environments such as backlighting, darkness, heavy rain, heavy snow, or dense fog, the primary information may not be obtainable. Another example is when a traffic sign S is obscured by tree branches or leaves; in such cases, the primary information may not be obtainable.

[0021] As an alternative when the first information is unavailable, it is possible to use map information 120 in which traffic sign information is pre-registered. The traffic sign information (notification information) pre-registered in map information 120 will be referred to as "second information" below. Roads on which traffic signs S are installed are assigned a road ID, which is identification information. Map information 120 shows the second information for each road ID. Map information 120 (second information) is obtained independently of the recognition process in vehicle 1. For example, map information 120 is generated and updated based on probe data measured by a large number of probe cars. Map information 120 also includes information on the date and time it was generated or updated. Such map information 120 is distributed by the map management server and becomes available. However, updates to map information 120 may be delayed, and the accuracy of map information 120 (second information) is not always high.

[0022] Therefore, according to this embodiment, "performance information 130" is newly introduced as an alternative when the first information cannot be obtained. Performance information 130 is generated, stored, and updated based on the results of past recognition processing in vehicle 1. Traffic sign information (notification information) obtained as a result of past recognition processing in vehicle 1 is hereinafter referred to as "third information". Performance information 130 includes road ID and third information.

[0023] The performance information 130 further includes a freshness parameter, which reflects the "freshness" of the third information. For example, the freshness parameter is a timestamp indicating the date and time the third information was acquired. The newer the timestamp, the "fresher" the third information is. As another example, the freshness information may be the elapsed time since the third information was acquired, i.e., the elapsed time from the timestamp to the current time. The shorter the elapsed time, the "fresher" the third information is.

[0024] The performance information 130 may also include the "confidence level" of the third piece of information. The confidence level can be said to be the certainty that traffic sign S actually exists. The confidence level may be expressed in the range of 0 to 100%. There is a high probability that the same traffic sign S will be recognized each time vehicle 1 travels the same road. Each time the same traffic sign S on the same road is recognized, the confidence level of the third piece of information regarding that traffic sign S increases. On the other hand, if a traffic sign S that should exist is not recognized, the confidence level may decrease.

[0025] The performance information 130 may be stored in a predetermined storage device installed in the vehicle 1, or it may be stored in a cloud server outside the vehicle 1. In either case, the performance information 130 is prepared separately from the map information 120 described above.

[0026] Let's consider the information notification process when vehicle 1 is traveling on a road (target road). If the first information regarding the target road is not obtained, the information notification system 10 will notify user O of the second or third information equivalent to the first information instead. The third information regarding the target road is obtained from the actual information 130. On the other hand, the second information regarding the target road is obtained from the map information 120. However, as mentioned above, the map information 120 may not be updated regularly, so the accuracy of the second information is not always high.

[0027] Therefore, if the first information regarding the target road is not obtained, the information notification system 10 determines whether the "priority conditions" for prioritizing the third information are met. The priority conditions include at least "the third information regarding the target road is newer than the second information." The freshness of the third information is obtained from the freshness parameter included in the actual information 130. The freshness of the second information is obtained from the generation / update date and time of the map information 120. Based on the freshness of that information, the information notification system 10 can determine whether the priority conditions are met.

[0028] The priority condition may include, in addition to "the third information regarding the target road is newer than the second information," "the confidence level of the third information regarding the target road is equal to or greater than the first threshold TH-C1." The information notification system 10 can determine whether the priority condition is met based on the freshness and confidence level of the third information.

[0029] If the priority conditions are not met, the information notification system 10 notifies user O of the second information obtained from map information 120 instead of the first information. On the other hand, if the priority conditions are met, the information notification system 10 notifies user O of the third information obtained from actual information 130 instead of the second information obtained from map information 120. As a result, user O receives more up-to-date traffic sign information (notification information). In other words, it becomes possible to notify user O of vehicle 1 of information with higher accuracy. This also contributes to improving the accuracy of driving vehicle 1.

[0030] 2. Example Configuration Figure 2 is a block diagram showing an example configuration of the information notification system 10 according to this embodiment. The information notification system 10 includes a communication interface 20, a notification device 30, one or more processors 40 (hereinafter simply referred to as processor 40), and one or more storage devices 50 (hereinafter simply referred to as storage devices 50).

[0031] The communication interface 20 is an interface for communicating with the outside world of the information notification system 10.

[0032] The notification device 30 notifies the user O of vehicle 1 of various information. If user O is the driver in vehicle 1, the notification device 30 is installed in vehicle 1. If user O is a remote operator, the notification device 30 is included in the remote operator terminal. For example, the notification device 30 includes a display device. Examples of display devices include a display, a head-up display (HUD), etc. Another example is that the notification device 30 may include a speaker.

[0033] The processor 40 performs various processes. For example, the processor 40 includes a CPU (Central Processing Unit). The processor 40 may be included in the vehicle 1 or in a remote operator terminal. Multiple processors 40 may be distributed across multiple locations.

[0034] The storage device 50 stores various types of information. Examples of storage devices 50 include volatile memory, non-volatile memory, HDD (Hard Disk Drive), SSD (Solid State Drive), etc. Multiple storage devices 50 may be distributed in multiple locations. The location of the storage device 50 is not particularly limited as long as it is accessible to the processor 40.

[0035] The control program PROG is a computer program for controlling the information notification system 10. The processor 40 executes the control program PROG, thereby enabling various processes by the information notification system 10. The control program PROG is stored in the storage device 50. Alternatively, the control program PROG may be recorded on a computer-readable recording medium.

[0036] The processor 40 acquires an image (IMG) captured by a camera 2 mounted on the vehicle 1 via the communication interface 20. The processor 40 also acquires vehicle location information (POS), which indicates the position (latitude and longitude) of the vehicle 1, via the communication interface 20. The vehicle location information (POS) is acquired, for example, by a GNSS (Global Navigation Satellite System) sensor mounted on the vehicle 1. The image (IMG) and vehicle location information (POS) are stored in the storage device 50.

[0037] The storage device 50 is further configured to store recognition information 110, map information 120, and performance information 130.

[0038] The recognition information 110 is real-time information obtained by recognition processing based on an image IMG. For example, the processor 40 recognizes a traffic sign S from an image IMG by using an image recognition AI (Artificial Intelligence). The image recognition AI is pre-generated through a learning method such as deep learning. The image recognition AI is trained to detect various traffic signs S from the image IMG and to identify the type of traffic sign S detected. The processor 40 may also read the speed limit specified by the recognized speed limit sign by image analysis. The first information is information about the recognized traffic sign S. The information about the traffic sign S is an image of the traffic sign S, an icon of the traffic sign S, or the content indicated by the traffic sign S. For example, if the traffic sign S is a speed limit sign, the first information may be the speed limit. The recognition information 110 includes the first information.

[0039] Map information 120 includes a general navigation map. Furthermore, map information 120 has pre-registered secondary information for each road ID. In addition, map information 120 also includes information on the date and time it was generated or updated. Map information 120 is managed and distributed by a map management server. The processor 40 obtains map information 120 from the map management server via the communication interface 20.

[0040] Performance information 130 is generated, stored, and updated based on the results of past recognition processing in vehicle 1. Figure 3 is a flowchart showing the processes related to the storage and updating of performance information 130.

[0041] In step S10, the processor 40 recognizes the traffic sign S by recognition processing based on the image IMG.

[0042] In step S20, the processor 40 obtains the road ID of the target road on which vehicle 1 is currently traveling, based on the vehicle location information POS and map information 120.

[0043] In the performance information 130, an entry is created for each traffic sign S. In the example shown in Figure 1, one entry contains a combination of road ID, traffic sign information (third-party information), freshness parameter, and confidence level for one traffic sign S. Such an entry is created for each traffic sign S.

[0044] In step S30, the processor 40 determines whether an entry for the road ID and traffic sign S recognized in the above step has already been registered in the performance information 130. In other words, the processor 40 determines whether a new traffic sign S has been recognized.

[0045] If an entry for the recognized traffic sign S is not yet registered, that is, if a new traffic sign S is recognized (step S30; Yes), the process proceeds to step S40. In step S40, the processor 40 adds a new entry for the newly recognized traffic sign S to the history information 130. At this time, the timestamp is set to the current time, and the freshness parameter is set based on that timestamp. The confidence level is also set to a predetermined initial value.

[0046] On the other hand, if an entry for the recognized traffic sign S has already been registered, that is, if an existing traffic sign S is recognized (step S30; No), the process proceeds to step S50. In step S50, the processor 40 updates the existing entry for the existing traffic sign S. At this time, the timestamp is updated to the current time, and the freshness parameter is updated based on that timestamp. The confidence level is also updated to increase from the previous value. For example, a predetermined value is added to the previous value of the confidence level.

[0047] There is a high probability that the same traffic sign S will be recognized each time vehicle 1 travels along the same road. Each time the same traffic sign S on the same road is recognized, the confidence level of the third piece of information regarding that traffic sign S increases.

[0048] In addition to the above processing, the processor 40 may check whether the traffic sign S registered in the performance information 130 is actually recognized on the target road. If the traffic sign S registered in the performance information 130 is not recognized on the target road, the processor 40 may reduce the confidence level for the traffic sign S.

[0049] 3. Information notification processing The processor 40 controls the notification device 30 to notify the user O of vehicle 1 of traffic sign information (notification information). Various examples of this information notification process are described below.

[0050] 3-1. Example 1 Figure 4 is a flowchart illustrating the first example of information notification processing. Here, it is assumed that vehicle 1 is traveling on the target road. As described above, the road ID of the target road is obtained based on the vehicle location information POS and map information 120.

[0051] In step S100, the processor 40 acquires an image IMG captured by the camera 2 mounted on the vehicle 1.

[0052] In step S101, the processor 40 determines whether or not the traffic sign S has been recognized by the recognition process based on the image IMG. In other words, the processor 40 determines whether or not the first information has been acquired.

[0053] If the traffic sign S is recognized, that is, if the first information is acquired (step S101; Yes), the process proceeds to step S110. In step S110, the processor 40 selects the acquired first information and notifies the user O of the first information.

[0054] On the other hand, if the traffic sign S is not recognized, that is, if the first information is not acquired (step S101; No), the process proceeds to step S102. In step S102, the processor 40 acquires map information 120 and actual information 130 related to the target road based on the road ID of the target road. Furthermore, the processor 40 determines whether the "first priority condition" is met based on the map information 120 and actual information 130 related to the target road.

[0055] The first priority condition includes at least "the third information concerning the target road is newer than the second information." The freshness of the third information is obtained from the freshness parameter included in the actual information 130. The freshness of the second information is obtained from the generation / update date and time of the map information 120. The processor 40 can determine whether the first priority condition is met based on the freshness of that information.

[0056] The first priority condition may further include, in addition to "the third information concerning the target road is newer than the second information," "the confidence level of the third information concerning the target road is equal to or greater than the first threshold TH-C1." The first threshold TH-C1 may be higher than the initial confidence level. The processor 40 can determine whether the first priority condition is met based on the freshness and confidence level of the third information.

[0057] If the first priority condition is not met (step S102; No), the process proceeds to step S120. In step S120, the processor 40 selects second information regarding the target road obtained from the map information 120 and notifies the user O of the second information.

[0058] On the other hand, if the first priority condition is met (step S102; Yes), the process proceeds to step S130. In step S130, the processor 40 selects third information regarding the target road obtained from the actual information 130 and notifies the user O of the third information.

[0059] 3-2. Second Example Figure 5 is a flowchart of a second example of the information notification process. Explanations that overlap with the first example described above are omitted as appropriate. In the second example, if the traffic sign S is recognized, that is, if the first information is acquired (step S101; Yes), the process proceeds to step S103.

[0060] In step S103, the processor 40 obtains performance information 130 regarding the target road based on the road ID of the target road. Furthermore, the processor 40 determines whether the "second priority condition" is met based on the performance information 130 regarding the target road. The second priority condition includes "the confidence level of the third information regarding the target road is equal to or greater than the second threshold TH-C2" and "the freshness of the third information regarding the target road is equal to or greater than the third threshold TH-F". The second threshold TH-C2 may be higher than the initial value of the confidence level. The freshness of the third information being equal to or greater than the third threshold TH-F means that the elapsed time from the timestamp of the third information to the current time is less than or equal to a certain amount of time.

[0061] If the second priority condition is not met (step S103; No), the process proceeds to step S110. In this case, the processor 40 notifies the user O of the first information. On the other hand, if the second priority condition is met (step S103; Yes), the process proceeds to step S130. In this case, the processor 40 notifies the user O of the third information.

[0062] According to the second example, even when the recognition accuracy of the traffic sign S is low, it is possible to notify the user O of vehicle 1 with highly accurate information.

[0063] 3-3. Third Example In the third example, the processor 40 obtains the confidence level of the first information in the recognition process based on the image IMG. The confidence level may be expressed in the range of 0 to 100%. For example, the image recognition AI outputs the confidence level of the image recognition process, and this confidence level of the image recognition process is used as the confidence level of the first information. The confidence level of the first information depends on the environment in which the image IMG is taken. For example, the confidence level tends to decrease in environments such as backlighting, darkness, heavy rain, heavy snow, and dense fog. The confidence level of the first information may also depend on the complexity of the road structure. For example, the confidence level tends to decrease in Y-shaped roads.

[0064] Figure 6 is a flowchart of a third example of the information notification process. Explanations that overlap with the first example described above are omitted as appropriate. In the third example, if the traffic sign S is recognized, that is, if the first information is acquired (step S101; Yes), the process proceeds to step S104.

[0065] In step S104, the processor 40 obtains performance information 130 regarding the target road based on the road ID of the target road. Furthermore, the processor 40 determines whether the "third priority condition" is met based on the performance information 130 regarding the target road. The third priority condition includes "the confidence level of the third information is higher than the confidence level of the first information."

[0066] If the third priority condition is not met (step S104; No), the process proceeds to step S110. In this case, the processor 40 notifies the user O of the first information. On the other hand, if the third priority condition is met (step S104; Yes), the process proceeds to step S130. In this case, the processor 40 notifies the user O of the third information.

[0067] According to the third example, even when the recognition accuracy of the traffic sign S is low, it is possible to notify the user O of vehicle 1 with highly accurate information.

[0068] 4. Effects As described above, according to this embodiment, performance information 130 is introduced as a new concept. By utilizing this performance information 130, it becomes possible to notify the user O of more up-to-date information than the map information 120. In other words, it becomes possible to notify the user O of vehicle 1 of more accurate information. This also contributes to improving the accuracy of driving vehicle 1. [Explanation of symbols]

[0069] 1…Vehicle, 2…Camera, 10…Information notification system, 110…Recognition information, 120…Map information, 130…Performance information

Claims

1. An information notification system that notifies vehicle users of information, One or more processors that perform recognition processing based on images taken by a camera mounted on the vehicle and showing the surrounding conditions of the vehicle, One or more storage devices and Equipped with, Traffic sign information is information that shows the content of traffic signs installed on roads. The one or more storage devices are Map information including road identification information, second information which is traffic sign information obtained on the road independently of the recognition process, and information on the date and time when the second information was generated or updated. The record includes road identification information, third information which is traffic sign information obtained as a result of the past recognition process on the road, and a timestamp indicating the date and time the third information was obtained, or a freshness parameter indicating the elapsed time since the date and time the third information was obtained. It is configured to store, The one or more processors determine whether they were able to acquire the first information, which is real-time traffic sign information relating to the target road on which the vehicle is traveling, through the recognition process. If the first information cannot be obtained by the recognition process, the one or more processors, Based on the identification information of the target road, the map information and performance information relating to the target road are obtained from one or more storage devices. The date and time on which the third information was obtained from the freshness parameter is obtained, and it is determined whether a first priority condition is met, which at least includes the fact that the date and time on which the third information was obtained is newer than the date and time on which the second information was generated or updated. If the first priority condition is met, the user is notified of the third information instead of the second information. It is configured in such a way Information notification system.

2. An information notification system according to claim 1, The aforementioned performance information further includes the confidence level of the aforementioned third information, The first priority condition further includes that the confidence level of the third information is equal to or greater than the first threshold. Information notification system.

3. An information notification system according to claim 1 or 2, If the first information is obtained through the recognition process, the one or more processors notify the user of the obtained first information. Information notification system.

4. The information notification system according to claim 2, If the first information is obtained through the recognition process and the second priority condition is met, the one or more processors shall notify the user of the third information instead of the first information. The second priority condition includes that the confidence level of the third information is equal to or greater than the second threshold, and that the elapsed time from the date and time the third information was obtained to the present time is less than or equal to a certain period of time. Information notification system.

5. An information notification system according to Claim 1, The aforementioned traffic signs are speed limit signs, stop signs, or no-passing signs. Information notification system.

Images