IMAGE RECOGNITION DEVICE AND IMAGE RECOGNITION METHOD

Abstract

To provide an image recognition device and an image recognition method that can appropriately recognize a road sign.SOLUTION: An image recognition device 33 comprises: an image information acquisition unit 41 that acquires image information obtained by capturing an image of a traveling route on which the own vehicle is traveling; a sign recognition unit 42 that recognizes a road sign based on the image information; a sign specifying unit 45 that specifies a road sign around the own vehicle by referring to map information generated based on travel track information of a vehicle; a collation unit 46 that collates the road sign recognized by the sign recognition unit 42 with the road sign based on the map information, in which when these road signs match with each other, the collation unit defines the matched road sign as a collation result, whereas the collation unit selects one of the road signs as a collation result, or selects neither road sign when these road signs do not match; and an output unit 47 that outputs the collation result.SELECTED DRAWING: Figure 2

Description

[Technical field]

[0001] The present invention relates to an image recognition device and an image recognition method. [Background technology]

[0002] A technology for appropriately recognizing road signs on a road on which a vehicle is traveling based on a camera image is known (Patent Document 1). In Patent Document 1, the detected position of the road sign is compared with the position of the road sign stored in the map data to distinguish whether the recognized road sign is a fixed sign or a temporary sign. [Prior art documents] [Patent documents]

[0003] [Patent Document 1] JP 2012-164254 A Summary of the Invention [Problem to be solved by the invention]

[0004] In recent years, technology has been developed that enables a server to acquire driving locus information (probe data) from a vehicle via a communication network and generate a map (probe data map, hereinafter referred to as a PD map) from the driving locus information.

[0005] However, the PD map does not distinguish between fixed signs and temporary signs, and as a result, the installation and removal of temporary signs is not updated appropriately on the PD map, which can lead to misrecognition.

[0006] The present invention has been made in consideration of the above circumstances, and has as its main object to provide an image recognition device and an image recognition method that are capable of appropriately recognizing road signs. [Means for solving the problem]

[0007] The first means is an image recognition device that includes an image information acquisition unit that acquires image information of a road along which the vehicle is traveling, a sign recognition unit that recognizes road signs based on the image information, a sign identification unit that identifies road signs around the vehicle by referring to map information generated based on vehicle travel trajectory information, a matching unit that compares the road signs recognized by the sign recognition unit with road signs based on the map information and, if there is a match, sets the matching road sign as the matching result, while, if there is no match, selects one of the road signs as the matching result or selects neither, and an output unit that outputs the matching result.

[0008] The second means is an image recognition method executed by an image recognition device, comprising: an image information acquisition step of acquiring image information of an image of a road along which the vehicle is traveling; a sign recognition step of recognizing road signs based on the image information; a sign identification step of identifying road signs around the vehicle by referring to map information generated based on vehicle travel trajectory information; a comparison step of comparing the road signs recognized by the sign recognition step with road signs based on the map information, and if there is a match, using the matching road sign as the comparison result, whereas if there is no match, selecting one of them as the comparison result or selecting neither; and an output step of outputting the comparison result.

[0009] According to the first and second means, the road sign recognized from the image information is compared with the road sign based on the map information, and if they match, the matched road sign is set as the comparison result, while if they do not match, one of them or none of them is selected as the comparison result. Therefore, more accurate information can be provided compared to the case where only the recognition result based on the image information is displayed. Also, more accurate information can be provided when a temporary sign is installed or removed. [Brief description of the drawings]

[0010] [Figure 1] FIG. 1 is a schematic configuration diagram of a driving assistance system. [Diagram 2] FIG. 2 is a functional block diagram of a driving assistance device and an image recognition device. [Diagram 3] 4 is a flowchart of image recognition processing according to the first embodiment. [Figure 4] 4 is a flowchart of a map update process according to the first embodiment. [Diagram 5] 1 is a time chart showing transition of likelihood; [Figure 6] 1 is a time chart showing transition of likelihood; [Figure 7] 10 is a flowchart of an image recognition process according to a second embodiment. [Figure 8] 13 is a flowchart of a map generation process according to a third embodiment. [Figure 9] 13 is a flowchart of an image recognition process according to a third embodiment. [Figure 10] 13 is a flowchart of an image recognition process according to a fourth embodiment. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0011] Hereinafter, an embodiment of the driving assistance system will be described with reference to the drawings. In the following embodiments and modifications, the same reference numerals are used to designate the same or equivalent parts, and the description of the same reference numerals is incorporated herein by reference.

[0012] (First embodiment) A driving assistance system according to the present embodiment will be described with reference to the drawings. As shown in Fig. 1, the driving assistance system 100 includes a cloud server 101 as a driving assistance device, and a plurality of vehicles 102 that can communicate with the cloud server 101 via a network.

[0013] The cloud server 101 updates a probe data map (hereinafter, referred to as a PD map) as map information based on probe data, which is travel trajectory information acquired from a vehicle 102 as a measurement vehicle, and provides the PD map to a vehicle 102 as a user vehicle that uses the PD map. The cloud server 101 is a computer equipped with a CPU, a ROM, a RAM, an input / output port, a communication module, and the like. Specifically, the cloud server 101 includes a communication device 11 that transmits and receives the PD map and the probe data, a data storage device 12 that stores the PD map and the probe data, and a map generating device 13 that generates and updates the PD map. The communication device 11 is connected to the data storage device 12 and is configured to be capable of inputting and outputting the probe data and the PD map. The data storage device 12 is connected to the map generating device 13 and is configured to be capable of transmitting and receiving various data.

[0014] The cloud server 101 realizes various functions by using these devices. Fig. 2 shows a functional block diagram of the cloud server 101. As shown in Fig. 2, the cloud server 101 includes a map storage unit 21 that stores a PD map, a data receiving unit 22 that receives probe data from the vehicle 102, a data storage unit 23 that stores the probe data, a map update unit 24 that updates the PD map, a map transmission unit 25 that transmits the PD map, and the like. In this embodiment, the communication device 11 realizes the data receiving unit 22 and the map transmission unit 25, the data storage device 12 realizes the map storage unit 21 and the data storage unit 23, and the map generation device 13 realizes the map update unit 24. These various functions will be described later.

[0015] The various functions may be realized by hardware such as electronic circuits, or at least a part of the functions may be realized by software, i.e., processing executed on a computer. The cloud server 101 is installed outside the vehicle 102.

[0016] The vehicle 102 includes a monocular camera 31 as an imaging device (image sensor), a GPS (Global Positioning System) receiving device 32, an image recognition device 33, a map storage device 34, an in-vehicle communication device 35, a display device 36, and the like.

[0017] The camera 31 is a camera using an imaging element such as a CCD or a CMOS. The camera 31 is disposed, for example, near the upper end of the windshield of the vehicle, and captures (takes) an image of the surrounding environment including the road (travel path) ahead of the vehicle. The captured image by the camera 31 is output to the image recognition device 33. Note that a plurality of cameras (compound eye cameras) may be provided.

[0018] The GPS receiving device 32 has a function of receiving position information (the current position of the vehicle 102) by the positioning function of the GPS and outputting it to the image recognition device 33. The map storage device 34 is a device that stores the PD map provided by the cloud server 101, and outputs the PD map to the image recognition device 33. The in-vehicle communication device 35 communicates with the cloud server 101, transmits probe data, and receives the PD map. The display device 36 is connected to the image recognition device 33, and displays the recognition result output from the image recognition device 33.

[0019] The image recognition device 33 is a computer equipped with a CPU, a ROM, a RAM, an input / output port, and the like. The image recognition device 33 has various functions, and performs, for example, image recognition. The various functions are realized by executing a program stored in the ROM or the like of the image recognition device 33. FIG. 2 shows a functional block diagram of the image recognition device 33. As shown in FIG. 2, the image recognition device 33 has functions as an image information acquisition unit 41, a sign recognition unit 42, a map reception unit 43, a map storage unit 44, a sign identification unit 45, a matching unit 46, an output unit 47, and a data transmission unit 48. These various functions will be described later. The various functions may be realized by electronic circuits, which are hardware, or at least a part of them may be realized by software, that is, by processing executed on a computer.

[0020] Next, various functions of the cloud server 101 and the image recognition device 33 will be described with reference to Fig. 2. First, the functions of the image recognition device 33 will be described.

[0021] The image information acquisition unit 41 acquires the captured images (image information) captured by the camera 31 at a predetermined period. The sign recognition unit 42 identifies road signs from the captured images acquired by the image information acquisition unit 41. Specifically, the sign recognition unit 42 determines the type of road sign present ahead of the vehicle based on the captured images and dictionary information for identifying road signs prepared in advance. The dictionary information for identifying road signs is prepared individually for each type of road sign, such as guide signs, warning signs, regulatory signs, directional signs, and auxiliary signs, and is stored in advance in a memory such as a ROM.

[0022] The sign recognition unit 42 determines the type of road sign by comparing the captured image with dictionary information by pattern matching. The method of recognizing road signs is not limited to this and may be changed as desired. Furthermore, machine learning such as a deep neural network (DNN) may be used to recognize the type of road sign.

[0023] The sign recognition unit 42 also identifies position information of the road sign. Specifically, the sign recognition unit 42 identifies position information (longitude, latitude) at which the road sign is detected, based on the relative position of the road sign with respect to the vehicle and the vehicle position information (longitude, latitude) received from the GPS receiver 32.

[0024] The sign recognition unit 42 then outputs information on the type of road sign and location information (longitude, latitude) as sign data. At this time, the sign recognition unit 42 may include in the sign data a recognition confidence level when recognizing a road sign based on a captured image. The recognition confidence level indicates, for example, the likelihood that an actually set road sign is the type indicated in the sign data, and is calculated based on the degree of match when pattern matching is used to verify the road sign.

[0025] The map receiving unit 43 receives the PD map provided from the cloud server 101 via the in-vehicle communication device 35. The PD map is transmitted from the cloud server 101 at a predetermined timing. The map receiving unit 43 may also request the cloud server 101 to transmit the PD map at a predetermined timing. The map storage unit 44 stores the PD map received by the map receiving unit 43 in the map storage device 34.

[0026] The sign identification unit 45 refers to the PD map received by the map receiving unit 43, i.e., the PD map stored in the map storage device 34, and identifies road signs (position information and type of road signs, etc.) around the vehicle based on the position information (longitude, latitude) of the vehicle received from the GPS receiver 32.

[0027] The matching unit 46 matches the road sign recognized by the sign recognition unit 42 with the road sign identified by the sign identification unit 45 based on the PD map, and selects one of them as the matching result. Specifically, the matching unit 46 compares the position information and identifies the road sign closest to the position information of the road sign recognized by the sign recognition unit 42 among the road signs of the vehicle identified by the sign identification unit 45. Then, the matching unit 46 compares the type of the identified road sign with the type of the road sign recognized by the sign recognition unit 42, and determines whether or not they match. If they match, the matching unit 46 sets the type of the matching road sign as the matching result. On the other hand, if they do not match, the matching unit 46 sets the type of the road sign recognized by the sign recognition unit 42 as the matching result.

[0028] The output unit 47 outputs the matching result by the matching unit 46 to the display device 36, and causes the display device 36 to display the road signs as the matching result. The data transmission unit 48 transmits the probe data to the cloud server 101 via the in-vehicle communication device 35. In this embodiment, the probe data includes at least the recognition result by the sign recognition unit 42, i.e., sign data related to road signs.

[0029] The probe data may also include the vehicle's driving history (vehicle speed, longitudinal acceleration, lateral acceleration, etc.), position information of the vehicle 102, road shape, traffic light position information, time information (such as the time of detection), and the passing time from intersection to intersection.

[0030] Next, functions on the cloud server 101 side will be described. The map storage unit 21 stores the latest PD map. The data receiving unit 22 receives probe data from the vehicle 102 and outputs it to the data storage unit 23. The data storage unit 23 stores the probe data input from the data receiving unit 22. At that time, the data storage unit 23 stores at least sign data related to road signs included in the probe data in chronological order.

[0031] The map update unit 24 updates the PD map based on the sign data stored in the data storage unit 23, and stores the latest PD map in the map storage unit 21. At that time, the map update unit 24 refers to the sign data stored by the data storage unit 23, identifies points of change in the sign data stored in chronological order, and updates the PD map based on the points of change.

[0032] To explain in more detail, the map update unit 24 calculates the time series change in likelihood based on the number of stored sign data for each type of sign data whose location information corresponds to each other, and identifies the change points in the sign data based on the increase or decrease in the likelihood.

[0033] The sign data whose position information corresponds to each other is the sign data whose position information included therein differs within a predetermined range, for example, the sign data whose difference in longitude is within a predetermined range and whose difference in latitude is within a predetermined range.

[0034] The likelihood indicates the likelihood (degree of certainty) of an actual road sign type among the types of road signs indicated by sign data whose location information corresponds to each other. The likelihood is calculated for each type. The map update unit 24 calculates a higher likelihood the more sign data of the target type are stored in a given unit period. Note that even with the same number of stored data, the likelihood differs depending on the number of probe data data on which the data is generated, that is, the number of samples. In other words, even with the same number of stored data, when the number of probe data data is large, the likelihood is lower than when the number of probe data data is small.

[0035] For this reason, the map update unit 24 calculates the likelihood after normalizing it by the number of pieces of probe data that are the basis for generation. Specifically, the map update unit 24 calculates the likelihood based on the ratio of the number of stored pieces of probe data that are the basis for generation to the number of pieces of probe data that are the basis for generation in a predetermined unit period. The number of pieces of probe data that are the basis for generation may be the total number of pieces of probe data received (or stored) per unit period. Alternatively, it may be the total number of pieces of probe data received per unit period from vehicles 102 that were present within a predetermined range centered on the position information of the target road sign.

[0036] Furthermore, if each sign data includes a recognition confidence level when the road sign is recognized, the map update unit 24 may weight each sign data by the recognition confidence level and then calculate the likelihood for each type. For example, the map update unit 24 may weight each type of sign data by accumulating the recognition confidence levels of the target types of sign data in a predetermined unit period. In other words, weighting may be performed by accumulating a value obtained by multiplying the number of stored sign data pieces ("1") by the recognition confidence level.

[0037] Then, the map update unit 24 calculates the time series change in likelihood for each type of road sign for road signs indicated by sign data whose position information corresponds to each other as described above.The map update unit 24 then identifies a change point in likelihood based on the time series change in likelihood for each type of road sign.Specifically, the map update unit 24 identifies a change point in likelihood as a case where the likelihood increases by a first determination value or more or where the likelihood decreases by a second determination value or more during a predetermined period.

[0038] In other words, when a road sign is installed, the number of stored sign data indicating the type of the road sign tends to increase. Therefore, when the likelihood increases by more than the first judgment value during a predetermined period, the map update unit 24 judges that a road sign has been installed during the period, and updates the PD map.

[0039] On the other hand, when a road sign is removed, the number of stored sign data indicating the type of the road sign tends to decrease. Therefore, when the likelihood decreases by the second judgment value or more during a predetermined period, the map update unit 24 judges that the road sign has been removed during the period, and updates the PD map.

[0040] In addition, when a road sign is changed, it is equivalent to installing and removing the road sign at the same time. In other words, when a road sign is changed, the number of stored sign data indicating the type of road sign before the change tends to decrease, while the number of stored sign data indicating the type of road sign after the change tends to increase. Therefore, when the likelihood indicating the type of road sign before the change decreases by the second judgment value or more during a predetermined period, while the likelihood indicating the type of road sign after the change increases by the first judgment value or more, the map update unit 24 determines that the road sign has been changed during that period, and updates the PD map.

[0041] When the map update unit 24 updates the PD map, it stores the updated PD map as the latest PD map in the map storage unit 21. The map transmission unit 25 reads out the latest PD map from the map storage unit 21 at a predetermined timing and transmits it to the vehicle 102. For example, the map transmission unit 25 may transmit the PD map to the vehicle 102 at a predetermined time or on a predetermined date, or may transmit the PD map to the vehicle 102 based on a request from the vehicle 102.

[0042] Next, a flow of image recognition processing (image recognition method) executed when recognizing a road sign on the vehicle 102 side will be described with reference to Fig. 3. The image recognition processing is executed by the image recognition device 33 at predetermined intervals.

[0043] First, the image recognition device 33 acquires position information of the vehicle 102 (host vehicle) from the GPS receiver 32 (step S101). Next, the image recognition device 33 refers to the PD map stored in the map storage device 34, and identifies road signs around the host vehicle based on the position information (longitude, latitude) of the host vehicle received from the GPS receiver 32 (step S102). Through this step S102, the image recognition device 33 functions as a sign identification unit 45. Also, step S102 is a sign identification step.

[0044] Then, the image recognition device 33 acquires a captured image from the camera 31, and recognizes road signs based on the captured image (step S103). At this time, as described above, the image recognition device 33 recognizes the type and position information of the road sign. In addition, the image recognition device 33 also calculates the recognition confidence level. By the processing of step S103, the image recognition device 33 functions as the image information acquisition unit 41. In addition, the image recognition device 33 functions as the sign recognition unit 42. In addition, step S103 is an image information acquisition step and a sign recognition step.

[0045] Next, as described above, the image recognition device 33 compares the road sign recognized in step S103 with the road sign based on the PD map (step S104). Then, the image recognition device 33 determines whether the road sign recognized in step S103 is different from the road sign based on the PD map (step S105). If the result of this determination is positive, that is, if they are different, the image recognition device 33 sets the type of the road sign recognized in step S103 as the comparison result and outputs the comparison result to the display device 36 (step S106). That is, the display device 36 is caused to display the road sign based on the captured image as the comparison result.

[0046] In step S105, if there is no road sign in the PD map that corresponds to the position information of the road sign based on the captured image, the image recognition device 33 determines that the road sign is different (makes a positive determination). Similarly, if there is no road sign in the captured image that corresponds to the position information of the road sign based on the PD map, the image recognition device 33 determines that the road sign is different (makes a positive determination).

[0047] On the other hand, if the determination result in step S105 is negative, that is, if there is a match, the image recognition device 33 determines the matching type of road sign as the matching result, and outputs the matching result to the display device 36 (step S107). By steps S104 to S107, the image recognition device 33 functions as the matching unit 46. Moreover, steps S104 to S107 are a matching step. Moreover, by step S107, the image recognition device 33 functions as the output unit 47. Moreover, step S107 is an output step.

[0048] When the processes of steps S106 and S107 are completed, the image recognition device 33 creates probe data including the sign data recognized based on the captured image, and transmits the probe data to the cloud server 101 via the in-vehicle communication device 35 (step S108). By this step S108, the image recognition device 33 functions as the data transmission unit 48. Also, step S108 is a data transmission step.

[0049] Next, the processing (driving assistance method) on the cloud server 101 side will be described. First, the storage processing executed when the probe data is received will be described. When the cloud server 101 receives the probe data, it stores the probe data in chronological order. More specifically, when the communication device 11 receives the probe data, it outputs the probe data to the data storage device 12. The data storage device 12 stores the inputted probe data in chronological order. At that time, the sign data included in the probe data is also stored in chronological order. This storage processing corresponds to a data storage step.

[0050] Next, the flow of the map update process executed when updating the PD map will be described with reference to Fig. 4. The map update process is executed at a predetermined timing by the cloud server 101. The map update process corresponds to a map update step.

[0051] The cloud server 101 reads out the stored probe data (step S201). That is, the map generating device 13 reads out a plurality of pieces of probe data stored in chronological order from the data storage device 12. Next, the cloud server 101 reads out the latest stored PD map (step S202). That is, the map generating device 13 reads out the latest PD map from the data storage device 12.

[0052] Next, as described above, the cloud server 101 identifies the sign data whose position information corresponds to each other among the sign data included in the read probe data (step S203). That is, the map generating device 13 identifies the sign data whose position information corresponds to each other.

[0053] Then, the cloud server 101 calculates a time series change in likelihood based on the number of stored sign data for each type of sign data whose position information identified in step S203 corresponds to each other (step S204). That is, the map generating device 13 accumulates the number of sign data (i.e., the number of stored sign data) for each type of sign data whose position information identified in step S203 corresponds to each other. At this time, the map generating device 13 accumulates the number of stored sign data per predetermined unit period and arranges them in time series to obtain a transition in likelihood for each type. Note that, when the sign data includes a recognition confidence level, the map generating device 13 performs weighting by the recognition confidence level, accumulates the number of stored sign data, and calculates the likelihood for each type. At this time, the map generating device 13 also calculates the likelihood for each type by normalizing the number of stored sign data by the number of probe data data per unit period. That is, the map generating device 13 calculates the likelihood for each type based on the ratio of the number of stored sign data to the number of probe data data per unit period.

[0054] In this embodiment, normalization is performed based on the number of pieces of probe data, but normalization is not required. Weighting based on the recognition confidence level is not required. In other words, the map generating device 13 may simply calculate the transition of the number of memories per predetermined unit period.

[0055] Next, the cloud server 101 identifies a change point from the time series change in the likelihood for each type calculated in step S204 (step S205). For example, if the likelihood for each type decreases by more than the second judgment value in a predetermined inspection period (e.g., 2 unit periods to 100 unit periods), the map generating device 13 identifies that the change point occurred at time T2 when the likelihood decreased by more than the second judgment value (see FIG. 5). On the other hand, if the likelihood for each type increases by more than the first judgment value in a predetermined inspection period (e.g., 2 unit periods to 100 unit periods), the map generating device 13 identifies that the change point occurred at time T1 when the likelihood increased by more than the first judgment value (see FIG. 6).

[0056] In step S205, if multiple types of likelihoods are identified, the change point may be identified from the time series changes of one or two types of likelihoods with the largest number of memories. In addition, if multiple types of likelihoods are identified, the likelihoods or types of likelihoods with a memory count of a predetermined number or less may be ignored.

[0057] Then, the cloud server 101 determines whether or not a change point has been identified in step S205 (whether or not a change point has occurred) (step S206). If the determination result in step S206 is negative, the map update process ends.

[0058] On the other hand, if the determination result in step S206 is positive, the cloud server 101 updates the PD map (step S207). Specifically, when the map generating device 13 identifies a change point where the likelihood of a certain type increases by more than the first determination value, it determines that a road sign of that type has been installed, updates the PD map, and stores it in the data storage device 12. At that time, the map generating device 13 averages the position information of the road signs of that type from the sign data indicating that type, and sets the average as the position information of the road signs on the PD map.

[0059] On the other hand, when the map generating device 13 identifies a change point where the likelihood of a certain type decreases by the second judgment value or more, it determines that the road sign of that type has been removed, updates the PD map, and stores it in the data storage device 12. Then, the map update process ends.

[0060] According to the above embodiment, the following excellent effects can be obtained.

[0061] Once the sign data of a road sign is installed, the same data should be acquired unless it is erroneously recognized. In other words, when a road sign is newly installed, removed, or changed, the sign data of the road sign included in the probe data changes. Therefore, the cloud server 101 can appropriately update the PD map at an appropriate timing by identifying the change points of the sign data stored in a chronological order and updating the PD map based on the change points. In addition, since the change points of the sign data stored in a chronological order are identified, erroneous recognition can be suppressed compared to the case where the type of the road sign is determined by majority vote regardless of the acquisition timing.

[0062] When a road sign is newly installed, removed, or changed, there is often a noticeable change in the number of stored sign data for each type. Therefore, the cloud server 101 integrates the number of stored sign data of each type for each unit period in the sign data corresponding to the location information, and calculates the time-series change in likelihood based on the number of stored sign data. Therefore, the cloud server 101 can appropriately identify the change point of the sign data by monitoring the increase or decrease in likelihood from the time-series change.

[0063] Furthermore, when the vehicle recognizes a road sign, the cloud server 101 considers the possibility of erroneous recognition, performs weighting according to the recognition confidence level, and calculates the time series change in likelihood for each type. This makes it possible to appropriately identify the change point in the sign data while considering the possibility of erroneous recognition.

[0064] In addition, even if the number of stored sign data per unit period is the same, the likelihood differs depending on the number of probe data on which the data is generated, that is, the number of samples. For this reason, the cloud server 101 calculates the likelihood after normalizing it by the number of probe data on which the data is generated. This makes it possible to identify the change point more accurately.

[0065] The image recognition device 33 of the vehicle 102 using the PD map includes a matching unit 46 that matches road signs recognized based on a captured image with road signs based on the PD map and selects one of them as a matching result. This allows the vehicle 102 using the PD map to also recognize and match road signs, making it possible to output more accurate information about road signs.

[0066] When the road sign recognized based on the photographed image does not match the road sign based on the map information, the matching unit 46 uses the road sign recognized based on the photographed image as the matching result. That is, compared to the PD map, the road sign recognized based on the photographed image is more likely to have newer and more accurate information. For this reason, when the road signs do not match, the road sign recognized by the sign recognition unit is preferentially output.

[0067] The vehicle 102 using the PD map also serves as a measurement vehicle, and the sign data of the road signs recognized by the image recognition device 33 is included in the probe data and transmitted to the cloud server 101. This enables the vehicle 102 to output accurate road signs, and the cloud server 101 to collect more probe data according to the use of the vehicle 102.

[0068] The image recognition device 33 uses machine learning to recognize road signs, which allows for more accurate recognition of road signs.

[0069] Second embodiment The driving assistance system of the first embodiment may be partially modified as follows. Note that in the second embodiment described below, the description of the first embodiment is used for the parts that are the same as or equivalent to those of the first embodiment, and the description thereof will be omitted.

[0070] In the second embodiment, the image recognition device 33 of the vehicle 102 is configured to be able to perform scene determination for identifying a construction zone from a captured image. That is, the image recognition device 33 includes a scene determination unit for identifying a construction zone from a captured image. The image recognition device 33 as a scene determination unit identifies a construction zone based on whether or not an object indicating that construction is underway is present in the captured image. Specifically, the image recognition device 33 determines that a construction zone is present when a sign or billboard indicating that construction is underway can be recognized from the captured image, or when a certain number or more of objects specific to a construction zone (such as pylons, barricades, and construction vehicles) can be recognized. Note that the method for recognizing objects indicating that construction is underway may be the same as the method for recognizing road signs (such as pattern matching).

[0071] Next, the flow of image recognition processing in the second embodiment will be described with reference to Fig. 7. First, the image recognition device 33 performs the processing of steps S101 to S105 in the same manner as in the first embodiment.

[0072] If the determination result of step S105 is positive, that is, if the road signs based on the captured image and the road signs based on the PD map are different, the image recognition device 33 performs scene determination to identify a construction section from the captured image (step S301). Then, the image recognition device 33 determines whether or not the construction section has been identified from the captured image (step S302). Through steps S301 and S302, the image recognition device 33 functions as a scene determination unit. Furthermore, steps S301 and S302 are scene determination steps.

[0073] If the determination result in step S302 is positive, that is, if it has been recognized as a construction section, the image recognition device 33 determines the type of road sign recognized in step S103 as a matching result, and outputs the matching result to the display device 36 (step S106). That is, the display device 36 is caused to display the road sign based on the captured image as the matching result.

[0074] On the other hand, if the determination result in step S302 is negative, i.e., if it could not be recognized as a construction section, the image recognition device 33, if the recognition confidence level of the road sign based on the captured image is equal to or higher than a predetermined value, determines the type of road sign recognized in step S103 as a matching result and outputs the matching result to the display device 36, and if it is less than the predetermined value, determines that it is impossible to determine (step S303). Note that if it is impossible to determine, the image recognition device 33 does not display the matching result.

[0075] On the other hand, if the determination result in step S105 is negative, that is, if there is a match, the image recognition device 33 determines the matching type of road sign as the matching result, and outputs the matching result to the display device 36 (step S107).

[0076] When the processing of steps S106, S107, and S303 is completed, the image recognition device 33 creates probe data including the sign data recognized based on the captured image, and transmits the probe data to the cloud server 101 via the in-vehicle communication device 35 (step S108).

[0077] According to the second embodiment, when a road sign to be checked is installed in a construction zone where temporary road signs are frequently installed, and the road signs do not match, the image recognition device 33 preferentially outputs the road sign recognized from the captured image. This allows the road sign to be displayed more accurately.

[0078] Third embodiment The driving assistance system of the first embodiment may be partially modified as follows: In the third embodiment below, the description of the parts that are the same as or equivalent to those of the first embodiment will be omitted.

[0079] First, the storage process of the third embodiment will be described. When the cloud server 101 receives the probe data, it stores the probe data. More specifically, when the communication device 11 receives the probe data, it outputs the probe data to the data storage device 12. The data storage device 12 stores the input probe data. This storage process corresponds to a data storage step.

[0080] Next, the flow of the map generation process of the third embodiment will be described with reference to FIG. 8. The map generation process is executed by the cloud server 101 at a predetermined timing. The cloud server 101 reads out the stored probe data (step S401). That is, the map generation device 13 reads out a plurality of probe data stored from the data storage device 12. Next, as described above, the cloud server 101 identifies, from among the sign data included in the read probe data, sign data whose position information corresponds to each other (step S402). That is, the map generation device 13 identifies sign data whose position information corresponds to each other.

[0081] Then, the cloud server 101 performs a majority vote on the types of road signs for the sign data whose position information corresponds to each other in step S402, and determines the type of road sign with the largest number of memories as the road sign to be included in the PD map (step S403). Note that in step S403, if the largest number of memories is equal to or less than a predetermined number, it may be ignored as an erroneous recognition. This predetermined number may be set appropriately according to the total number of probe data.

[0082] Furthermore, the cloud server 101 averages the position information of the sign data identified in step S402, and determines the average as the position information of the road sign determined in step S403 (position information to be published on the PD map) (step S404).

[0083] Furthermore, the cloud server 101 calculates the number of pieces of data, the average value of the recognition confidence, the position variance value, and the type variance value from the sign data identified in step S402 (step S405).

[0084] The number of data is the total number of sign data (including probe data) identified in step S402. The average recognition confidence level is calculated by averaging the recognition confidence levels included in each sign data identified in step S402. The position variance value indicates the degree of dispersion of position information in which the road sign was detected, which is included in the sign data of the probe data that was the basis for generating the road sign determined in step S403. The position variance value is calculated based on the dispersion (variance) of the position information of the sign data identified in step S402. The larger the dispersion, the larger the position variance value, and the smaller the dispersion, the smaller the position variance value.

[0085] The type variance value indicates the degree of dispersion of road sign types contained in the sign data that was the basis for generating the road sign determined in step S403. The type variance value is calculated based on the dispersion (variance) of the sign data types identified in step S402. The larger the dispersion, the larger the type variance value will be, and the smaller the dispersion, the smaller the type variance value will be.

[0086] Then, the cloud server 101 generates and updates a PD map based on the information determined or calculated in steps S403 to S405 (step S406). That is, the map generating device 13 generates a PD map assuming that a road sign of the type determined in step S403 exists at the position indicated by the position information determined in step S404, and stores the PD map in the data storage device 12. When generating and updating the PD map, the map generating device 13 stores the number of data, the average recognition confidence level, the position variance value, and the type variance value calculated in step S405 in association with the type and position information of each road sign in the data storage device 12. Then, the map generation process ends.

[0087] The updated PD map is transmitted from the cloud server 101 to the vehicle 102 at a predetermined timing, similar to the above embodiment. When the PD map is transmitted, various information associated with the road signs (number of data, average value of recognition confidence, position variance value, type variance value) is also transmitted.

[0088] Next, the image recognition process in the third embodiment will be described with reference to Fig. 9. The image recognition process is executed by the image recognition device 33 of the vehicle 102 at predetermined intervals.

[0089] When the image recognition process starts, the image recognition device 33 performs the processes of steps S101 to S105, similarly to the first embodiment. If the determination result of step S105 is negative, the image recognition device 33 performs steps S107 and S108, and ends the image recognition process.

[0090] On the other hand, if the result of the determination in step S105 is positive, that is, if the type of the road sign based on the captured image and the type of the road sign based on the PD map are different, the image recognition device 33 determines whether the recognition confidence level of the road sign recognized in step S103 is equal to or lower than the first threshold value (step S501). If the result of this determination is positive, the image recognition device 33 determines the type of the road sign identified based on the PD map as the matching result, and outputs the matching result to the display device 36 (step S502).

[0091] On the other hand, if the determination result in step S501 is negative, the image recognition device 33 identifies the number of pieces of probe data that were used as the basis for generating the road sign to be matched based on the PD map, and determines whether the number of pieces of data is equal to or less than the second threshold (step S503). That is, the image recognition device 33 acquires the number of pieces of data associated with the road sign in the PD map, and determines whether the number of pieces of data is equal to or less than the second threshold. If the determination result is positive, the image recognition device 33 determines the type of road sign recognized based on the captured image in step S103 as the matching result, and outputs the matching result to the display device 36 (step S504).

[0092] Furthermore, if the determination result in step S503 is negative, the image recognition device 33 identifies the average value of the recognition reliability of the road sign to be matched based on the PD map, and determines whether or not the average value is equal to or less than the third threshold value (step S505). That is, the image recognition device 33 obtains the average value of the recognition reliability associated with the road sign in the PD map, and determines whether or not the average value is equal to or less than the third threshold value. If the determination result is positive, the image recognition device 33 proceeds to step S504, where it determines the type of the road sign recognized based on the captured image in step S103 as the matching result, and outputs the matching result to the display device 36.

[0093] On the other hand, if the determination result in step S505 is negative, the image recognition device 33 identifies the position variance value of the road sign to be matched based on the PD map, and determines whether the position variance value is equal to or greater than a fourth threshold value (step S506). That is, the image recognition device 33 acquires the position variance value associated with the road sign in the PD map, and determines whether the position variance value is equal to or greater than a fourth threshold value. If the determination result is positive, the image recognition device 33 proceeds to step S504, where it regards the type of road sign recognized based on the captured image in step S103 as the matching result, and outputs the matching result to the display device 36.

[0094] On the other hand, if the determination result in step S506 is negative, the image recognition device 33 identifies the type variance value of the road sign to be matched based on the PD map, and determines whether or not the type variance value is equal to or greater than the fifth threshold (step S507). That is, the image recognition device 33 obtains the type variance value associated with the road sign in the PD map, and determines whether or not the type variance value is equal to or greater than the fifth threshold. If the determination result is positive, the image recognition device 33 proceeds to step S504, where it determines the type of the road sign recognized based on the captured image in step S103 as the matching result, and outputs the matching result to the display device 36.

[0095] If the determination result in step S507 is negative, the image recognition device 33 proceeds to step S402, where it determines the type of road sign identified based on the PD map as the matching result, and outputs the matching result to the display device .

[0096] When the processes of steps S502 and S504 are completed, the image recognition device 33 creates probe data including the sign data recognized based on the captured image, and transmits the probe data to the cloud server 101 via the in-vehicle communication device 35 (step S108). Then, the image recognition process ends.

[0097] According to the third embodiment, the following effects can be obtained.

[0098] When the road signs do not match, and the recognition confidence level of the road sign recognized in step S103 based on the captured image is equal to or lower than the first threshold, the image recognition device 33 as the matching unit 46 does not select (decide) the road sign recognized based on the captured image as the matching result. This allows more reliable information to be selected and displayed as the matching result.

[0099] Furthermore, when the road signs do not match, the image recognition device 33 as the matching unit 46 identifies the number of pieces of sign data that were the basis for generating the road sign to be matched based on the PD map, and when the number of pieces of data is equal to or less than a second threshold, does not select (decide) the road sign on the PD map as the matching result. In other words, when the number of pieces of probe data that were the basis for generating the road sign is small, it is possible to take into consideration the high possibility of erroneous recognition, and to select more reliable information and display it as the matching result.

[0100] Furthermore, when the road signs do not match, the image recognition device 33 as the matching unit 46 specifies the average value of the recognition confidence level of the road signs to be matched based on the PD map, and when the average value is equal to or less than the third threshold value, does not select (decide) the road sign on the PD map as the matching result. In other words, when the average value of the recognition confidence level of the sign data on which the result was generated is small, it is possible to take into consideration the high possibility of erroneous recognition, and to select information with a higher reliability and display it as the matching result.

[0101] Furthermore, when the road signs do not match, the image recognition device 33 as the matching unit 46 specifies the position variance value of the road sign to be matched based on the PD map, and when the position variance value is equal to or greater than the fourth value, does not select (decide) the road sign on the PD map as the matching result. In other words, when there is a large variance in the position information of the sign data on which the data was generated, it is possible to take into consideration the high possibility of erroneous recognition, and to select information with a higher reliability and display it as the matching result.

[0102] Furthermore, when the road signs do not match, the image recognition device 33 as the matching unit 46 specifies the type variance value of the road sign to be matched based on the PD map, and when the type variance value is equal to or greater than the fifth value, does not select (decide) the road sign on the PD map as the matching result. In other words, when there is a large variation in the type of sign data that is the basis for generation, it is possible to take into consideration the high possibility of erroneous recognition, and to select information with a higher reliability and display it as the matching result.

[0103] (Fourth embodiment) The driving assistance system of the first embodiment may be partially modified as follows: In the following fourth embodiment, the description of the parts that are the same as or equivalent to those of the first embodiment will be omitted.

[0104] In the fourth embodiment, the storage process and the map generation process executed by the cloud server 101 are similar to those in the third embodiment, so the description of the third embodiment is reused and the description will be omitted.

[0105] Next, the image recognition process in the fourth embodiment will be described with reference to Fig. 10. The image recognition process is executed by the image recognition device 33 of the vehicle 102 at predetermined intervals.

[0106] When the image recognition process starts, the image recognition device 33 performs the processes of steps S101 to S105, similarly to the first embodiment. If the determination result of step S105 is negative, the image recognition device 33 performs steps S107 and S108.

[0107] On the other hand, if the determination result in step S105 is positive, that is, if the type of the road sign based on the captured image and the type of the road sign based on the PD map are different, the image recognition device 33 calculates the reliability of the road sign based on the captured image based on the recognition confidence of the road sign recognized in step S103 (step S601). In step S601, the recognition confidence of the road sign may be directly adopted as the reliability of the road sign based on the captured image, or the recognition confidence may be adjusted to take into account the road conditions (weather and time of day (whether it is nighttime, etc.)).

[0108] Next, the image recognition device 33 uses various information included in the PD map to calculate the reliability of the road sign based on the PD map (step S602). Specifically, the image recognition device 33 refers to the PD map to obtain an average value of the recognition reliability associated with the road sign to be matched, and sets the average value as a base value of the reliability of the road sign based on the PD map. Therefore, when the average value of the recognition reliability is high, the reliability is higher than when it is low.

[0109] Next, the image recognition device 33 refers to the PD map to identify the number of pieces of sign data (probe data) that are the basis for generating the road sign that is associated with the road sign to be matched, and adjusts the reliability according to the number of pieces of data. Specifically, when the number of pieces of sign data is large, the base value is adjusted so that the reliability is higher than when the number of pieces of data is small.

[0110] Furthermore, the image recognition device 33 refers to the PD map, acquires a position variance value associated with the road sign to be matched, and adjusts the reliability based on the position variance value. Specifically, when the position variance value is large, the base value (base value adjusted based on the number of data) is adjusted so that the reliability is lower than when the position variance value is small.

[0111] Similarly, the image recognition device 33 refers to the PD map to obtain a type variance value associated with the road sign to be matched, and adjusts the reliability based on the type variance value. Specifically, when the type variance value is large, the base value (base value adjusted based on the number of data and the position variance value) is adjusted so that the reliability is lower than when the type variance value is small.

[0112] After calculating the reliability in steps S601 and S602, the image recognition device 33 compares the reliability of the road sign based on the captured image with the reliability of the road sign based on the PD map, and selects (determines) the road sign with the higher reliability as the matching result (step S603).

[0113] Then, the image recognition device 33 determines the type of road sign determined in step S603 as the matching result, and outputs the matching result to the display device 36 (step S604). After completing the process of step S604, the image recognition device 33 creates probe data including the sign data recognized based on the captured image, and transmits the probe data to the cloud server 101 via the in-vehicle communication device 35 (step S108). Then, the image recognition process ends.

[0114] According to the fourth embodiment, the following effects can be obtained.

[0115] When the road signs do not match, the image recognition device 33 as the matching unit 46 calculates the reliability of the road signs based on the captured image and the reliability of the road signs based on the PD map, and compares them to select (decide) the matching result. Therefore, information with a higher reliability can be selected and displayed as the matching result.

[0116] The reliability of a road sign based on a captured image is calculated based on the recognition confidence of the road sign. The reliability of a road sign based on a PD map is adjusted based on the number of pieces of sign data (probe data) associated with the road sign, the average recognition confidence, the position variance, and the type variance. This allows more reliable information to be selected and displayed as a matching result.

[0117] In the fourth embodiment, the reliability of a road sign based on a PD map is calculated based on the number of data, the average recognition confidence level, the position variance value, and the type variance value, but it may be calculated based on any one or more of these values.

[0118] (Other embodiments) The configuration of the above embodiment may be partially modified. Hereinafter, another embodiment (modification) in which the configuration of the above embodiment is partially modified will be described.

[0119] In the second embodiment, the image recognition device 33 may include the result of scene determination by the image recognition device 33 (i.e., the result of determination as to whether or not the road is in a construction zone) in the probe data. The cloud server 101 may use the result of scene determination as a criterion for determining whether or not to update the PD map.

[0120] For example, the cloud server 101 may set the judgment values ​​(first judgment value and second judgment value) small when identifying a change point in the likelihood based on sign data whose location information corresponds to a construction zone. In other words, the judgment criteria for a change point may be lowered in consideration of the high frequency of temporary road signs being installed in construction zones. In other words, even if the change range is small, it may be determined to be a change point. This makes it possible to quickly update the PD map in response to temporary road signs installed in construction zones.

[0121] In this modified example, the probe data may include information related to objects indicating that construction is underway, and the cloud server 101 may perform scene determination to identify construction sections based on the probe data.

[0122] If the determination result in step S302 in the second embodiment is negative, that is, if it is not possible to recognize that the road is a construction zone, the image recognition device 33 may determine that the road is not identifiable regardless of the recognition confidence level. If the recognition confidence level of the road sign is less than a predetermined value, the road sign based on the PD map may be used as the matching result. In addition, the processes in steps S501 to S507 in the third embodiment and the processes in steps S601 to S604 in the fourth embodiment may be executed.

[0123] In the third embodiment, the processing order of steps S501 and S503 to S507 may be changed as desired. For example, the determinations of steps S503 to S507 may be performed before step S501, and if the determination result of step S507 is negative, step S501 may be executed.

[0124] In the third embodiment, the first threshold value to the fifth threshold value may be set to any arbitrary value.

[0125] In the third embodiment, if the result of the determination in step S501 is negative, it may be determined that the determination is impossible. In this case, the result does not have to be displayed.

[0126] In the above embodiment, the vehicle 102 serves both as a measurement vehicle that transmits measured probe data to the cloud server 101 and as a user vehicle that receives a PD map provided by the cloud server 101 and uses the PD map, but it may serve only one of these roles.

[0127] The control unit and the method described in the present disclosure may be realized by a special-purpose computer provided by configuring a processor and a memory programmed to execute one or more functions embodied in a computer program. Alternatively, the control unit and the method described in the present disclosure may be realized by a special-purpose computer provided by configuring a processor with one or more dedicated hardware logic circuits. Alternatively, the control unit and the method described in the present disclosure may be realized by one or more special-purpose computers configured by a combination of a processor and a memory programmed to execute one or more functions and a processor configured with one or more hardware logic circuits. In addition, the computer program may be stored in a computer-readable non-transitory tangible recording medium as instructions executed by a computer. [Explanation of symbols]

[0128] 23...data storage unit, 24...map update unit, 41...image information acquisition unit, 42...sign recognition unit, 45...sign identification unit, 46...matching unit, 47...output unit, 100...driving assistance system, 101...cloud server, 102...vehicle.

Claims

1. An image information acquisition unit (41) that acquires image information of a road on which the vehicle is traveling; a sign recognition unit (42) that recognizes road signs based on the image information; a sign identification unit (45) that identifies road signs around the vehicle by referring to map information generated based on vehicle travel trajectory information; a matching unit (46) for matching a road sign recognized by the sign recognition unit with a road sign based on the map information, and when they match, determining the matching road sign as a matching result, and when they do not match, selecting either one of them as a matching result or selecting neither; an output unit (47) that outputs the collation result, the matching unit, when the road signs do not match, calculates reliability of the road signs based on the image information and calculates reliability of the road signs based on the map information, compares them, and selects the road sign with a higher reliability as a matching result; the reliability of the road sign based on the image information is calculated based on a recognition confidence indicating, in terms of probability, the likelihood that the type of the actual road sign is the type of the road sign recognized by the sign recognition unit; The reliability of the road sign based on the map information is calculated based on the recognition confidence level associated with the road sign to be matched.

2. The map information is generated based on a plurality of pieces of travel path information, The map information is associated with each of the road signs with the number of data items of the travel path information that are the basis for generating the road signs, 2. The image recognition device according to claim 1, wherein the matching unit calculates a reliability of the road sign based on the map information to be higher when a number of data items of driving trajectory information that were used as a basis for generating the road sign to be matched is large, compared to when a number of data items of driving trajectory information that were used as a basis for generating the road sign to be matched is small.

3. The map information is generated based on a plurality of pieces of travel path information, an average value of a recognition confidence level that is set when the road sign is recognized by a vehicle is associated with the map information for each of the road signs; 3. The image recognition device according to claim 1, wherein the matching unit calculates the reliability of the road sign based on the map information to be higher when an average value of the recognition confidence of the road sign to be matched is high than when the average value is low.

4. The map information is generated based on a plurality of pieces of travel path information, In the map information, a position variance value indicating a degree of dispersion of position information at which the road sign is detected, the position variance value being included in the travel trajectory information on which the road sign is generated, is associated with each of the road signs; The image recognition device according to any one of claims 1 to 3, wherein the matching unit calculates a reliability of the road sign based on the map information to be lower when the position variance value of the road sign to be matched is large compared to when the position variance value is small.

5. The map information is generated based on a plurality of pieces of travel path information, the map information includes, for each of the road signs, a type variance value indicating a degree of dispersion of types of the road signs corresponding to position information of the road signs included in the travel trajectory information on which the road signs were generated; The image recognition device according to any one of claims 1 to 4, wherein the matching unit calculates a reliability of the road sign based on the map information to be lower when the type variance value of the road sign to be matched is large compared to when the type variance value is small.

6. 2. The image recognition device according to claim 1, wherein when the road signs do not match, and when a recognition confidence level of the road sign recognized by the sign recognition unit based on the image information is equal to or lower than a first threshold value, the matching unit does not select the road sign recognized by the sign recognition unit as a matching result.

7. The map information is generated based on a plurality of pieces of travel path information, The map information is associated with each of the road signs with the number of data items of the travel path information that are the basis for generating the road signs, 7. The image recognition device according to claim 1, wherein, when the road signs do not match, the matching unit identifies a number of pieces of data of the driving trajectory information that was used as a basis for generating the road sign to be matched based on the map information, and when the number of pieces of data is equal to or less than a second threshold, does not select the road sign of the map information as a matching result.

8. The map information is generated based on a plurality of pieces of travel path information, an average value of a recognition confidence level that is set when the road sign is recognized by a vehicle is associated with the map information for each of the road signs; 8. The image recognition device according to claim 1, wherein, when the road signs do not match, the matching unit specifies an average value of the recognition confidence of the road signs to be matched based on the map information, and when the average value is equal to or less than a third threshold, the matching unit does not select the road sign in the map information as the matching result.

9. The map information is generated based on a plurality of pieces of travel path information, In the map information, a position variance value indicating a degree of dispersion of position information at which the road sign is detected, the position variance value being included in the travel trajectory information on which the road sign is generated, is associated with each of the road signs; 9. The image recognition device according to claim 1, wherein the matching unit, when the road signs do not match, specifies a position variance value of the road sign to be matched based on the map information, and when the position variance value is equal to or greater than a fourth threshold, does not select the road sign in the map information as the matching result.

10. The map information is generated based on a plurality of pieces of travel path information, the map information includes, for each of the road signs, a type variance value indicating a degree of dispersion of types of road signs corresponding to position information of the road signs included in the traveling trajectory information on which the road signs were generated; 10. The image recognition device according to claim 1, wherein, when the road signs do not match, the matching unit specifies a type variance value of the road sign that is the target of matching based on the map information, and when the type variance value is equal to or greater than a fifth threshold, the matching unit does not select the road sign in the map information as the matching result.

11. An image recognition method executed by an image recognition device (33), An image information acquisition step (S103) of acquiring image information of an image of a road on which the host vehicle is traveling; A sign recognition step (S103) of recognizing a road sign based on the image information; A sign identification step (S102) of identifying road signs around the vehicle by referring to map information generated based on vehicle travel trajectory information; a comparison step (S104 to S107) of comparing the road signs recognized by the sign recognition step with road signs based on the map information, and if they match, the matched road signs are taken as the comparison result, whereas if they do not match, either one of them is selected as the comparison result or neither is selected; and an output step (S106, S107) of outputting the collation result, in the matching step, if the road signs do not match, a reliability of the road sign based on the image information is calculated, and a reliability of the road sign based on the map information is calculated, and these are compared, and a road sign with a higher reliability is selected as a matching result; the reliability of the road sign based on the image information is calculated based on a recognition confidence indicating, by a probability, the likelihood that the type of the actual road sign is the type of the road sign recognized by the sign recognition step; The reliability of the road sign based on the map information is calculated based on the recognition confidence level associated with the road sign to be matched.

Images