Driving support method, driving support device, and computer-readable recording medium

Abstract

This driving support method includes: detecting a pair of a regulation sign and an auxiliary sign from an image photographed by a camera mounted on a vehicle; generating first regulation information indicated by the auxiliary sign from characters included in the auxiliary sign; determining whether or not second regulation information indicated by the regulation sign forming a pair with the auxiliary sign is applied to the vehicle on the basis of the first regulation information; and changing the manner of presenting the second regulation information to the occupant of the vehicle according to the determined result.

Description

Driving assistance method, driving assistance device, and computer-readable recording medium

[0001] The present disclosure relates to a driving assistance method, a driving assistance device, and a computer-readable recording medium.

[0002] Patent document 1 describes a driving assistance device that detects road signs indicating specific zones, which indicate the boundaries of specific zones where speed limits are uniformly set, from a forward image captured in front of the vehicle, and notifies the driver that the vehicle is traveling in a specific zone.

[0003] Japanese Patent Application Laid-Open No. 2021-99539

[0004] Road signs indicating prohibitions, restrictions, or limitations on specific traffic methods, including speed limits (hereinafter referred to as "regulatory signs"), may be paired with supplementary signs that provide additional explanations such as the reason for the restriction, the time periods, days of the week, and types of vehicles to which the restriction applies. Whether or not the restriction indicated by the regulatory sign actually applies depends on the content of the supplementary signs, so regulatory signs alone may provide incorrect regulatory information to vehicle occupants.

[0005] An object of the present disclosure is to provide a driving assistance method, a driving assistance device, and a computer-readable recording medium that can appropriately notify vehicle occupants of regulatory information indicated by regulatory signs.

[0006] A driving assistance method according to one or more embodiments of the present disclosure detects pairs of regulatory signs and auxiliary signs from images captured by a camera mounted on a vehicle, generates first regulatory information indicated by the auxiliary sign from characters contained in the auxiliary sign, determines based on the first regulatory information whether second regulatory information indicated by the regulatory sign paired with the auxiliary sign applies to the vehicle, and changes the manner in which the second regulatory information is presented to the vehicle occupants depending on the result of the determination.

[0007] According to one or more embodiments of the present disclosure, the second regulation information indicated by the regulatory sign can be appropriately notified to the vehicle occupants.

[0008] FIG. 1 is a block diagram showing the configuration of a driving assistance system including a driving assistance device according to an embodiment. FIG. 2 is a flowchart showing a specific operation procedure of the input unit 1 when the on-board sensor 12 includes a depth camera. FIG. 3 is a flowchart showing a specific operation procedure of the input unit 1 when the on-board sensor 12 includes a digital camera. FIG. 4 is a flowchart showing a specific operation procedure of the input unit 1 using a point cloud acquired by a LiDAR as the on-board sensor 12. FIG. 5 is a flowchart showing a specific operation procedure of the detection unit 2. FIG. 6 is a diagram showing an example of a regulatory sign 201 and an auxiliary sign 501 in an image, surrounded by bounding boxes 301 and 302. FIG. 7 is a diagram showing an example of a layout of a sign group in which two auxiliary signs 501 and 502 are arranged vertically (in a column). FIG. 8 is a diagram showing a state in which the two auxiliary signs 501 and 502 in FIG. 7 are surrounded by a single bounding box 304. FIG. 9 is a diagram showing an example of a layout pattern of a sign group 401 consisting of multiple signs, each of which displays one sign. FIG. 10 is a diagram showing an example of a layout pattern of a sign group 402 in which multiple signs are displayed on a single sign (Part 1). FIG. 11 is a diagram showing an example of a layout pattern of a sign group 403 in which multiple signs are displayed on a single sign (Part 2). FIG. 12 is a flowchart showing a specific operation procedure of the recognition unit 3 (Part 1). FIG. 13 is a diagram showing a regulatory sign indicating a road closure for vehicles 219. FIG. 14 is a diagram showing a regulatory sign indicating a time-limited parking area 220. FIG. 15 is a diagram showing a regulatory sign indicating no travel outside of designated directions 221. FIG. 16 is a flowchart showing the detailed operation procedure of step S306 in FIG. 12. FIG. 17 is a diagram explaining a method for detecting the arrow and end points of a regulatory sign 222 using bounding boxes 315 to 319. FIG. 18 is a diagram explaining how to determine the angle of the arrow of the regulatory sign 222. FIG. 19 is a diagram explaining how to determine the angle of the end point (318) of the regulatory sign 222. FIG. 20 is a flowchart showing a specific operation procedure of the recognition unit 3 (Part 2). FIG. 21 is a diagram illustrating the procedure for extracting the text lines 102, 103, and 104 from the image of the auxiliary sign 508 using the bounding boxes 319 to 324.FIG. 22 is a flowchart showing specific operational procedures of the information generating unit 4. FIG. 23 is a flowchart showing detailed procedures of step S405 in FIG. 22. FIG. 24 is a diagram showing a regulatory sign 225 indicating a median strip. FIG. 25 is a flowchart showing detailed procedures of step S408 in FIG. 22. FIG. 26 is a diagram showing an example of an auxiliary sign 509 that is not in a single line configuration. FIG. 27 is a table showing an example of dictionary data D51. FIG. 28 is a flowchart showing detailed procedures of step S4807 in FIG. 25. FIG. 29 is a block diagram showing a vehicle model tree 224 indicating the configuration of vehicle models. FIG. 30 is a diagram showing source code FM01 in which a vehicle model T01 that corresponds to a regulatory sign and a vehicle model F01 that does not correspond to a regulatory sign are entered. FIG. 31 is a flowchart showing detailed procedures of step S4808 in FIG. 25. FIG. 32 is a diagram showing source code FM02 in which a date T02 that corresponds to a regulatory sign and dates F02-1 and F02-2 that do not correspond to a regulatory sign are entered. FIG. 33 is a flowchart showing the detailed procedure of step S4809 in FIG. 25. FIG. 34 is a diagram showing source code FM03 in which the day of the week T03 corresponding to a regulatory sign and the days of the week F03-1 and F03-2 not corresponding to a regulatory sign are entered. FIG. 35 is a flowchart showing the detailed procedure of step S4810 in FIG. 25. FIG. 36 is text 105 corresponding to an example of the first regulatory information (source code FM04) in FIG. 37. FIG. 37 is a diagram showing an example of first regulatory information (source code FM04) that specifies the vehicle type, date, day of the week, and time T04 extracted and integrated from the text 105 shown in FIG. 36 in steps S4807 to S4810 in FIG. 25. FIG. 38 is a diagram showing source code FM10 in which a vehicle type F10 not corresponding to a regulatory sign is entered. FIG. 39 is a diagram showing source code FM11 supplemented with decorative elements (left turn, straight ahead, roadside houses) F11-1 and F11-2 for each vehicle type. Fig. 40 shows source code FM20 in which only first control information specifying applicable and non-applicable vehicle models is described. Fig. 41 shows source code FM21 in which first control information T21 and F21 specifying the missing date, day of the week, and time are added to source code FM20 in Fig. 40.

[0009] The embodiments will be described with reference to the drawings. In the description of the drawings, the same parts are designated by the same reference numerals and the description thereof will be omitted.

[0010] FIG. 1 is a block diagram showing the configuration of a driving assistance system including a driving assistance device according to an embodiment. The driving assistance system is a system that appropriately notifies a driver of regulatory information (second regulatory information) indicated by regulatory signs shown in an image captured in the direction of travel of a vehicle. The vehicle is a vehicle that is the target of driving assistance. The driving assistance system includes a controller 11, an on-board sensor 12, a GPS receiver 13, a presentation unit 14, and a storage device 15. The controller 11 is connected to the other components (12, 13, 14, 15) via wired or wireless connections so as to be able to communicate data.

[0011] The driving assistance device according to the embodiment includes at least a controller 11. The controller 11 includes, for example, a microcontroller such as an ECU (Electronic Control Unit) mounted on a vehicle. The microcontroller includes, for example, a CPU (arithmetic processing unit), a memory (storage unit), an I / O port (input / output port), and the like, implemented on a single integrated circuit. The controller 11 may include the CPU, memory, I / O port, and the like, which are configured as separate electronic components.

[0012] Various computer programs required for operating the controller 11 as a driving assistance device are installed in the memory. The CPU executes these computer programs, causing the controller 11 to perform predetermined information processing in accordance with predetermined algorithms specified in the computer programs. This causes the controller 11 to constitute the information processing unit shown in FIG. 1 and operate as a driving assistance device. The operation of the controller 11 can realize a driving assistance method using the driving assistance device. The computer programs may be provided via any transmission medium capable of transmission. The computer programs may be provided by being recorded on a computer-readable recording medium, such as a non-transitory recording medium, such as a CD (compact disc) or a DVD (digital versatile disc).

[0013] The on-board sensor 12 is a sensor mounted on the vehicle and has a detection range extending at least in the direction of travel of the vehicle. The on-board sensor 12 includes an on-board camera capable of capturing an image in the direction of travel of the vehicle and generating an image, and a LiDAR (Light Detection and Ranging) sensor capable of measuring the three-dimensional position of an object around the vehicle. The on-board camera includes a digital camera capable of generating an RGB image and a camera with a depth sensor (hereinafter referred to as a "depth camera") capable of measuring the distance to an object. The on-board sensor 12 can be configured by combining an on-board camera and a LiDAR.

[0014] The GPS receiver 13 is mounted on the vehicle and receives signals from GPS (Global Positioning System) satellites. The GPS receiver 13 determines the position of the vehicle in the Earth coordinate system from the received signals and outputs GPS information indicating the position of the vehicle.

[0015] The presentation unit 14 is a user interface (UI) mounted on the vehicle. The presentation unit 14 presents the second regulation information indicated by the regulatory sign to the vehicle occupants. For example, the presentation unit 14 includes a display installed on the dashboard or instrument panel of the vehicle, a head-up display (HUD) that can be displayed on the windshield, and a speaker that outputs audio to the occupants.

[0016] The storage device 15 stores data necessary for the operation of the controller 11 and data generated during the operation. For example, the storage device 15 is configured with a hard disk drive (HDD) or a solid state drive (SSD). Data is written to and read from the storage device 15 in response to instructions from the controller 11. Dictionary data D51 is included in the data necessary for the operation of the controller 11. Details of the dictionary data D51 will be described later.

[0017] The controller 11 has an input unit 1, a detection unit 2, a recognition unit 3, an information generation unit 4, and an output unit 5 as an information processing unit.

[0018] The input unit 1 generates RGB images and depth images from driving scene frames acquired by an on-board camera. Driving scene frames are video frames consisting of multiple consecutive images captured periodically. Depth images are images that include distance information from the vehicle (depth camera) to objects in the images.

[0019] A number of operation examples of the input unit 1 will be described with reference to FIGS.

[0020] 2 is a flowchart showing a specific operation procedure of the input unit 1 when the on-board sensor 12 includes a depth camera. In step S101, the input unit 1 acquires an image captured by the depth camera as a traveling scene frame. In step S102, the input unit 1 acquires a residual RGB image obtained by removing depth information from the image captured by the depth camera. In step S103, the input unit 1 acquires a depth image having depth information from the image captured by the depth camera. In step S104, the input unit 1 acquires GPS information output from the GPS receiver 13. Specifically, the input unit 1 acquires GPS information of the vehicle at the time of image capture. The acquired location information is associated with the acquired RGB image and depth image and stored in memory.

[0021] 3 is a flowchart showing a specific operation procedure of the input unit 1 when the on-board sensor 12 includes a digital camera. The input unit 1 acquires RGB images captured by the digital camera as driving scene frames (steps S101 and S102). The input unit 1 estimates depth images from the RGB images using, for example, a machine learning model that has learned pairs of a large number of RGB images and associated depth images (step S105). The input unit 1 acquires the estimated depth images (step S103) and acquires GPS information (step S104), similar to the operation example of FIG. 2.

[0022] In step S105, disparity information from a stereo camera may be used instead of the machine learning model. That is, the distance to the object, i.e., a depth image, may be estimated from the disparity that occurs when the same object is imaged from different positions. By configuring a stereo camera using a digital camera, the distance to the object can be estimated from the disparity information.

[0023] FIG. 4 is a flowchart showing a specific operation procedure of the input unit 1 using a point cloud (point cloud: hereinafter referred to as a "LiDAR point cloud") acquired by the LiDAR as the on-board sensor 12. The input unit 1 acquires an RGB image captured by a digital camera as a driving scene frame (steps S101 and S102). The input unit 1 acquires the LiDAR point cloud from the on-board sensor 12 (step S106). The input unit 1 projects the LiDAR point cloud onto the RGB image (step S107). Note that, to accurately project the LiDAR point cloud onto the RGB image, calibration may be performed in advance, and the camera parameters obtained therefrom may be used for projection of the driving scene. The input unit 1 acquires the RGB image onto which the LiDAR point cloud is projected as a depth image (step S103). Similar to the operation example of FIG. 2, the input unit 1 acquires GPS information (step S104).

[0024] The detection unit 2 in FIG. 1 detects regulatory signs and auxiliary signs that supplement the regulatory signs from images captured by an on-board camera serving as the on-board sensor 12. FIG. 5 is a flowchart showing a specific operation procedure of the detection unit 2. The specific operation procedure of the detection unit 2 will be described with reference to FIG. 5. Here, a "regulatory sign" is a road sign that indicates a prohibition, regulation, or restriction on a specific traffic method. An "auxiliary sign" is a road sign that is paired with a regulatory sign and is intended to provide supplementary explanation of the reason for the regulation indicated by the regulatory sign, the time of day, day of the week, type of vehicle, etc. that the regulation applies to. Regulatory signs and auxiliary signs may be collectively referred to as "signs."

[0025] In step S201 of FIG. 5 , the detection unit 2 detects regulatory signs and auxiliary signs from the RGB image acquired by the input unit 1. For example, a machine learning model trained on a large amount of image data of regulatory signs or auxiliary signs can be used as the detection means. The detection unit 2 may not only detect regulatory signs and auxiliary signs, but also determine the category of the regulatory signs. For example, as shown in FIG. 6 , the detection unit 2 surrounds a regulatory sign 201 or auxiliary sign 501 in the image with a bounding box 301 or 302, respectively. The detection unit 2 outputs coordinate values ​​indicating the position of the bounding box within the entire image and the category of the regulatory sign. The coordinate values ​​of the bounding box include, for example, the coordinates ([xmin, ymin, xmax, ymax]) of the upper left and lower right vertices of a rectangular bounding box.

[0026] As the vehicle moves, the positions of the regulatory signs and auxiliary signs in the RGB image also change. Therefore, the process proceeds to step S202, where the detection unit 2 tracks the regulatory signs and auxiliary signs detected in step S201. Examples of tracking methods that can be used include FastMOT (Multi Object Tracking) and ByteTrack. The detection unit 2 outputs tracking information for the regulatory signs and auxiliary signs. The tracking information includes the bounding box ID, the regulatory sign category, and the updated coordinate values.

[0027] Proceeding to step S203, the detection unit 2 removes from the detected signs any signs that do not face the shooting direction of the on-board camera, i.e., the vehicle's traveling direction. For example, signs on roads that intersect with the road on which the vehicle is traveling do not face the vehicle's traveling direction. By removing these signs from the signs detected in step S201, signs that should be considered when the vehicle travels on the road can be extracted. Specifically, the aspect ratio of the bounding box surrounding the sign can be used as a criterion. If the ratio of the width to the height of the bounding box is smaller than a reference value, it can be determined that the sign does not face the vehicle's traveling direction. Alternatively, the variance value of the depth within a single sign from the depth image can be used as a criterion. If the variance value of the depth of a single sign is larger than a reference value, it can be determined that the sign does not face the vehicle's traveling direction.

[0028] In step S204, the detection unit 2 ranks the groups of signs detected in the image according to their distance from the vehicle. A "group of signs" refers to a group of signs whose relative distance in the image is equal to or less than a reference value. The groups of signs detected in the image are stored in the storage device 15 along with information about their distance from the vehicle and ranking information.

[0029] The layout of signs may differ from country to country or region to region. Therefore, in step S205, the detection unit 2 classifies the layout pattern of the sign group using GPS information indicating the country or region in which the vehicle is traveling. The detection unit 2 classifies the layout patterns of the sign groups 401 to 403 shown in Figures 9 to 11, for example, and outputs layout pattern information.

[0030] In Japan, there exists a sign group 401 consisting of multiple signs, each displaying one sign, as shown in Figure 9. The sign group 401 includes multiple regulatory signs 202-205 and multiple auxiliary signs 503-505.

[0031] Meanwhile, in China, as shown in Figures 10 and 11, there are layouts of sign groups 402 and 403 in which multiple signs are displayed on a single sign. In sign group 402 of Figure 10, regulatory signs 206-212 and auxiliary sign 506 are displayed on a single sign. In the layout of Figure 10, auxiliary sign 506 is located in the lower right corner of sign group 402 and forms a pair with regulatory sign 212, which is adjacent to it on the left. Regulatory sign 212 indicates that freight vehicles are prohibited from passing. Auxiliary sign 506 indicates that small freight vehicles are prohibited from passing only between 17:00 and 19:00, Monday through Friday.

[0032] Regulatory sign 206 indicates a speed limit. Regulatory sign 207 indicates that parking and stopping are prohibited. Regulatory sign 208 indicates that honking is prohibited. Regulatory sign 209 indicates that non-motorized vehicles (in Japan, this includes light vehicles such as bicycles and mopeds) must use the dedicated lane on the right. Regulatory sign 210 indicates that motorized vehicles (automobiles) must use the left side of the road. Regulatory sign 211 indicates that pedestrians must use the sidewalk on the right.

[0033] On the other hand, in sign group 403 in FIG. 11 , regulatory signs 213 to 218 and auxiliary sign 507 are written on a single signboard. In the layout in FIG. 11 , auxiliary sign 507 is located at the bottom center of sign group 403. Regulatory sign 213 indicates that freight vehicles are prohibited. Regulatory sign 214 indicates that tractors are prohibited. Regulatory sign 215 indicates that agricultural freight vehicles are prohibited. Regulatory sign 216 indicates that tricycles and low-speed freight vehicles are prohibited. Regulatory sign 217 indicates that animal-drawn vehicles are prohibited. Regulatory sign 218 indicates that human-powered tricycles (both those carrying cargo and those carrying passengers) are prohibited. Auxiliary sign 507 indicates that all regulatory signs 213 to 218 apply between 6:00 and 22:00.

[0034] Proceeding to step S206 of FIG. 5 , if the two auxiliary markers 501 and 502 detected in step S201 are arranged side by side in the vertical direction (column direction), the detection unit 2 merges the two or more auxiliary markers 501 and 502. In other words, the detection unit 2 recognizes the two or more auxiliary markers 501 and 502 as a single auxiliary marker. Specifically, the detection unit 2 determines whether the two auxiliary markers 501 and 502 are arranged side by side in the vertical direction (column direction) based on the coordinate values ​​of the bounding boxes 302 and 303 that surround the auxiliary markers 501 and 502, respectively. As shown in FIG. 8 , the auxiliary markers 501 and 502 that are arranged side by side in the vertical direction can be merged by surrounding the auxiliary markers 501 and 502 with a single bounding box 304.

[0035] The sign group layouts shown in Figures 7 and 8 are used in Japan. On the other hand, although not shown, in China, a layout pattern exists in which a single signboard has multiple regulatory signs arranged in a column on the left and multiple auxiliary signs arranged in a column on the right. In this layout pattern, adjacent regulatory signs and auxiliary signs form pairs. For this reason, two or more auxiliary signs arranged side by side in the vertical direction (column direction) are not merged. Thus, the decision in step S206 as to whether to merge auxiliary signs varies depending on the layout pattern of the sign group.

[0036] As described above, auxiliary signs are intended to provide supplementary information about the reason for the restriction indicated by the regulatory sign, the time and day of the week when the restriction applies, the type of vehicle, etc. Therefore, regulatory signs and auxiliary signs are generally paired. Therefore, proceeding to step S207 of FIG. 5 , the detection unit 2 detects pairs of regulatory signs and auxiliary signs using bounding boxes surrounding each of the regulatory signs and auxiliary signs, and outputs pair information (hereinafter, sometimes referred to as "pair IDs"). The pair IDs include information about the paired bounding boxes. Alternatively, the detection unit 2 may detect pairs of regulatory signs and auxiliary signs that are related to each other based on their respective positions. The "positions of the regulatory signs and auxiliary signs" include, for example, the distance from the vehicle to each sign and the vertical and horizontal distances between the regulatory and auxiliary signs.

[0037] For example, in the example of the Japanese sign group 401 shown in FIG. 9 , bounding boxes 305-311 are set to surround each of the multiple regulatory signs 202-205 and the multiple auxiliary signs 503-505. The detection unit 2 first calculates the center coordinates 306c, 307c, 308c, 309c, 310c, and 311c of the bounding boxes 305-311. This calculation can be performed using the coordinates of the upper left and lower right vertices of the rectangular bounding boxes 305-311 calculated in step S201. Next, the center coordinates of the bounding boxes of the regulatory signs are compared with the center coordinates of the bounding boxes of the auxiliary signs. That is, pairs of regulatory signs and auxiliary signs are extracted where the distance between the center coordinates in the horizontal direction (row direction) is within a certain threshold and the distance between the center coordinates in the vertical, horizontal, and lateral directions is shortest.

[0038] In the layout pattern of Figure 9, the horizontal (row) distance between the center coordinates 306c of the bounding box 306 and the center coordinates 308c of the bounding box 308 is within a certain threshold, and the vertical and horizontal distances between the center coordinates 306c and the center coordinates 308c are the shortest. Therefore, the detection unit 2 can detect a pair of the regulatory sign 203 surrounded by the bounding box 306 and the auxiliary sign 503 surrounded by the bounding box 308. Similarly, the detection unit 2 can detect a pair of the regulatory sign 204 surrounded by the bounding box 307 and the auxiliary sign 504 surrounded by the bounding box 309, and a pair of the regulatory sign 205 surrounded by the bounding box 310 and the auxiliary sign 505 surrounded by the bounding box 311. The last remaining regulatory sign (regulatory sign 202 in Figure 9) is paired with the expression "without auxiliary sign."

[0039] 10, the auxiliary sign 506 is located in the lower right corner of the sign group 402, and the other signs are all regulatory signs 206 to 212. For this layout pattern, the detection unit 2 uses the bounding boxes 312 and 313 to detect the pair of regulatory sign 212 and auxiliary sign 506 that are arranged side by side in the left-right direction. A specific detection method is to use the center coordinates of the bounding boxes 312 and 313 to replace the distance in the left-right direction (row direction) in FIG. 9 with the distance in the up-down direction (column direction).

[0040] 11, the auxiliary sign 507 is applied to all of the regulatory signs 213 to 218 included in the sign group 403. Therefore, the detection unit 2 detects pairs of each of the regulatory signs 213 to 218 and the auxiliary sign 507 surrounded by the bounding box 314.

[0041] In this way, the rule for pairing regulatory signs and auxiliary signs in step S207 depends on the layout pattern of the sign group. The detection unit 2 outputs the pair IDs of the regulatory signs and auxiliary signs that make pairs in each sign group.

[0042] The recognition unit 3 in Fig. 1 recognizes text from characters included in auxiliary signs. The recognition unit 3 also recognizes regulatory information indicated by regulatory signs (hereinafter referred to as "second regulatory information"). Fig. 12 is a flowchart showing a specific operation procedure of the recognition unit 3 (part 1). Fig. 20 is a flowchart showing a specific operation procedure of the recognition unit 3 (part 2). The specific operation procedure of the recognition unit 3 will be described with reference to the flowcharts in Figs. 12 and 20. Here, "text" refers to character data.

[0043] In step S301 of Fig. 12, the recognition unit 3 acquires the RGB image and the pair IDs of the regulatory sign and auxiliary sign detected by the detection unit 2. Proceeding to step S302, the recognition unit 3 extracts the portion of the sign surrounded by the bounding box from the RGB image. The recognition unit 3 then processes the regulatory sign and auxiliary sign individually. The recognition unit 3 processes the regulatory sign according to the flow from step S303 onwards shown in Fig. 12, and processes the auxiliary sign according to the flow from step S312 onwards shown in Fig. 20.

[0044] First, the processing of regulatory signs will be described with reference to FIGS.

[0045] In step S303, the recognition unit 3 acquires the extracted image of the regulatory sign and data indicating the category of the regulatory sign. In step S304, the recognition unit 3 determines whether further recognition of the content of the regulatory sign is necessary. For example, regulatory signs such as "Road Closed to Vehicles" 219 and "No Entry for Vehicles" shown in FIG. 13 do not require further recognition of their content. On the other hand, regulatory signs such as "Time-Limited Parking Zone" 220 shown in FIG. 14, "No Proceeding Except in Specified Direction" 221 shown in FIG. 15, and "No Parking or Stopping" require further recognition of their content. If further recognition of the content of the regulatory sign is necessary (YES in step S304), the recognition unit 3 proceeds to step S305. If further recognition of the content of the regulatory sign is not necessary (NO in step S304), the recognition unit 3 proceeds to step S308.

[0046] In step S305, the recognition unit 3 determines whether the category of the regulatory sign belongs to "No Proceeding Outside of Designated Direction." Note that "No Proceeding Outside of Designated Direction" refers to a regulatory sign that prohibits vehicles from entering an intersection in any direction other than the direction indicated by the arrow. "No Proceeding Outside of Designated Direction" signs are installed just before the target intersection, in the median strip, or adjacent to the traffic light associated with the intersection. If the category of the regulatory sign belongs to "No Proceeding Outside of Designated Direction" (YES in step S305), the process proceeds to step S306; if the category does not belong to "No Proceeding Outside of Designated Direction" (NO in step S305), the process proceeds to step S307.

[0047] In step S306, the recognition unit 3 organizes the contents indicated by the regulatory signs, i.e., the road shape. Details of step S306 will be described later with reference to the flowchart of FIG.

[0048] In step S307, the recognition unit 3 recognizes the content of the regulatory sign. Specifically, the recognition unit 3 identifies the content of the regulatory sign according to the category of the regulatory sign. For example, if the regulatory sign is a weight limit sign or a maximum speed sign, the recognition unit 3 recognizes the numbers in the sign using an optical character recognition / reader (OCR). The recognition unit 3 outputs text such as "Weight limit of ____ tons" or "Maximum speed of ____ km / h" as the recognition result, as the second regulatory information. For example, in the case of regulatory signs 212, 213-218 shown in FIGS. 10 and 11 , which indicate "authorized vehicles only" or "Road closure for vehicles (combined)," the recognition unit 3 recognizes an icon in the sign, detects the vehicle type associated with the icon, and outputs text such as "Authorized vehicles only for ____ vehicles" or "Road closure for ____ vehicles and ____ vehicles" as the second regulatory information.

[0049] In step S3061 of FIG. 16 , the recognition unit 3 detects an arrow from the cut-out image of the regulatory sign. Proceeding to step S3062, the recognition unit 3 detects a corner point from the cut-out image of the regulatory sign. In the context of "No travel outside the designated direction," "corner point" refers to an end point where there is no arrow. For example, as shown in FIG. 17 , the recognition unit 3 detects the arrow of the regulatory sign 222 using bounding boxes 315 to 317. Similarly, the recognition unit 3 detects the corner point of the regulatory sign 222 using bounding boxes 318 to 319.

[0050] Proceeding to step S3063, the recognition unit 3 identifies the lowest edge of the edge points included in the regulatory sign as the lane the vehicle is currently traveling in. This identification method can use the coordinate values ​​of the bounding boxes 318-319. For example, the recognition unit 3 identifies the edge point (319) with the lowest center coordinates of the bounding boxes 318-319 in Figure 17.

[0051] Proceeding to step S3064, the recognition unit 3 acquires the features of the arrow using the encoder, calculates the distance between the features and the features output by the same encoder in a template image of an arrow prepared in advance in 5-degree increments, and then outputs the angle of the template image with the closest distance as the angle of the arrow detected in step S3061.

[0052] The arrow angle may be calculated using a learning model that simultaneously estimates the arrow position and arrow angle from an image of the regulatory sign 222 shown in FIG. 17 . The learning model may be trained to output not only the arrow position but also the arrow angle by adding arrow angle information as an annotation to the image data of the arrow and training the model. As shown in FIG. 18 , the angle of the arrow (bounding box 320) is recorded as 0 degrees at the lowest, most extreme point (bounding box 321), with a clockwise angle of 101 degrees recorded from there. As shown in FIG. 19 , if a more extreme point (bounding box 318) other than the lowest, most extreme point is detected on the regulatory sign 222, the angle of the more extreme point is calculated using the same method as the arrow described above. Alternatively, as shown in the enlarged view of the bounding box 318 in FIG. 19 , the angle between the most extreme point and the bounding box 318 may be calculated by detecting corners 318a and 318b on both sides of the more extreme point and calculating the normal to the line segment connecting the corners 318a and 318b, thereby calculating the angle between the most extreme point and the bounding box 318. The recognition unit 3 creates a list of the coordinate values ​​of the bounding box surrounding the detected arrow and its end point, and the angles of the arrow and its end point, and outputs this as the second regulation information. In this way, the recognition unit 3 can calculate the angles of the arrow and its end point included in the regulation sign 222.

[0053] The operation of the recognition unit 3 proceeds to step S308 in FIG. 12 . The recognition unit 3 consolidates the second regulation information output in steps S306 and S307. The process proceeds to step S309, where noise contained in the second regulation information is removed using a time filter with a queue structure. The queue structure has a maximum length and follows the FIFO rule (first-in, first-out). Specifically, the recognition unit 3 accumulates consecutively recognized second regulation information in a time filter with a queue structure. When the number of identical second regulation information pieces accumulated exceeds a threshold (YES in step S310), the process proceeds to step S311, where the recognition unit 3 outputs the second regulation information. This allows for stable recognition of second regulation information on the time axis. The second regulation information is transmitted to the information generation unit 4 along with a pair ID with the auxiliary sign.

[0054] Next, the processing of auxiliary signs by the recognition unit 3 will be described with reference to FIGS.

[0055] After step S302 in Fig. 12, the process proceeds to step S312 in Fig. 20, where the recognition unit 3 acquires an image of the extracted auxiliary sign. In step S313, the recognition unit 3 extracts text images 102, 103, and 104 divided into lines from the image of the auxiliary sign 508, for example, as shown in Fig. 21, using bounding boxes 319 to 324. The extracted text images 102, 103, and 104 for each line may be referred to as "text lines." As a method for detecting text lines, for example, a text detection model trained on a large number of text lines may be used.

[0056] Proceeding to step S314, the recognition unit 3 uses OCR to recognize text from the extracted text lines 102, 103, and 104. As shown in FIG. 21 , the recognition unit 3 recognizes text such as "Large vehicle, etc.", "9-16," and "All day" from the text lines 102, 103, and 104. In this way, the recognition unit 3 recognizes multiple texts divided into lines from the characters included in the auxiliary sign. Note that information indicating the order of vertical placement is added to the multiple texts. Furthermore, if the auxiliary sign includes an icon, the recognition unit 3 detects the icon and recognizes the text. For example, the recognition unit 3 recognizes the text "two-wheel" from an icon of a motorcycle. The icon can be recognized using a known method, such as a pattern matching technique that compares the icon with a sample icon prepared in advance.

[0057] Proceeding to step S315, the recognition unit 3 removes noise contained in the text using a time filter with a queue structure. The queue structure has a maximum length and follows the FIFO rule (first in, first out). Specifically, the recognition unit 3 accumulates consecutively recognized text in a time filter with a queue structure, and when the same text accumulates to a threshold value or more (YES in step S316), proceeds to step S317 and outputs the text. This allows the recognition unit 3 to recognize the text of an auxiliary sign that is stable on the time axis. The text of the auxiliary sign is transmitted to the information generation unit 4 together with the pair ID with the regulatory sign.

[0058] The information generation unit 4 in Fig. 1 generates first restriction information indicated by an auxiliary sign by comparing the text with dictionary data D51. Fig. 22 is a flowchart showing a specific operation procedure of the information generation unit 4. Fig. 23 is a flowchart showing a detailed procedure of step S405 in Fig. 22. Fig. 25 is a flowchart showing a detailed procedure of step S408 in Fig. 22.

[0059] Fig. 28 is a flowchart showing the detailed procedure of step S4807 in Fig. 25. Fig. 31 is a flowchart showing the detailed procedure of step S4808 in Fig. 25. Fig. 33 is a flowchart showing the detailed procedure of step S4809 in Fig. 25. Fig. 35 is a flowchart showing the detailed procedure of step S4810 in Fig. 25. The specific operation procedure of the information generation unit 4 will be described with reference to the flowcharts in Figs. 22, 23, 25, 28, 31, 33 and 35.

[0060] In step S401 of FIG. 22, the information generation unit 4 acquires the pair information (pair ID) of the regulatory sign and auxiliary sign extracted in step S207 of FIG. 5, the second regulatory information indicated by the regulatory sign output in step S311 of FIG. 12, and the text of the auxiliary sign output in step S317 of FIG. 20.

[0061] In step S402, the information generator 4 determines whether the pair consists of only regulatory signs. If the pair consists of only regulatory signs (e.g., regulatory sign 202 in FIG. 9) (YES in S402), the information generator 4 proceeds only to step S403, where the information generator 4 processes only the regulatory signs. On the other hand, if the pair consists of a combination of a regulatory sign and an auxiliary sign (NO in S402), the information generator 4 proceeds to both steps S403 and S407, where the information generator 4 processes the regulatory sign and the auxiliary sign separately.

[0062] The information generating unit 4 processes the regulatory signs in steps S403 to S406.

[0063] In step S403, the information generation unit 4 acquires the second regulation information indicated by the regulation sign. In step S404, the information generation unit 4 determines whether the category of the regulation sign is "No travel outside the specified direction." If the category is "No travel outside the specified direction" (YES in step S404), the information generation unit 4 proceeds to step S405. If the category is not "No travel outside the specified direction" (NO in step S404), the information generation unit 4 proceeds to step S406.

[0064] In step S405, the information generating unit 4 identifies the road shape of the intersection corresponding to the regulatory sign. The detailed procedure of step S405 will be described with reference to FIG.

[0065] In step S4051 in FIG. 23, the information generating unit 4 acquires a list of the coordinate values ​​of the bounding box surrounding the arrow and the end point output in step S3064 in FIG. 16, and the angles of the arrow and the end point.

[0066] Proceeding to step S4052, the information generation unit 4 determines whether there is only one arrow and the position of the lowest point is higher than the position of the arrow. If the determination is affirmative (YES in S4052), the information generation unit 4 can determine that the sign is a regulatory sign 224 (see FIG. 24 ) prohibiting travel in any direction other than the designated direction, which is installed on a road with a median strip to prevent drivers from choosing the wrong lane. Therefore, in step S4053, the information generation unit 4 generates lane warning information indicating that the driver should travel in the lane to the left of the median strip as second regulatory information, and proceeds to step S406 in FIG. 22 .

[0067] On the other hand, if the determination is negative (NO in S4052), the process proceeds to step S4054, where the information generation unit 4 identifies the intersection closest to the vehicle's current position and driving direction as the intersection corresponding to the regulatory sign. The information generation unit 4 acquires an image of the intersection corresponding to the regulatory sign, for example, a satellite image or a map image. The process proceeds to step S4055, where the information generation unit 4 extracts multiple lanes intersecting at the intersection from the image of the intersection using semantic segmentation. The process proceeds to step S4056, where the information generation unit 4 maps the arrows and edges to the multiple lanes intersecting at the intersection based on the angles of the arrows and edges. The process proceeds to step S4057, where the information generation unit 4 generates lane information indicating lanes that can be traveled and lanes that cannot be traveled as second regulation information based on this mapping. The process then proceeds to step S406 of FIG. 22, where the second regulation information indicated by the regulatory sign is organized.

[0068] Meanwhile, the information generation unit 4 processes the auxiliary sign in steps S407 and S408. In step S407, the information generation unit 4 acquires the text of the auxiliary sign. In step S408, the information generation unit 4 organizes the regulation content indicated by the text (hereinafter referred to as "text logic"). The detailed procedure of step S408 will be described with reference to the flowchart in FIG. 25.

[0069] In step S4801 of Fig. 25, the information generation unit 4 acquires the coordinate values ​​of the multiple pieces of text divided into lines and the bounding boxes (see Fig. 21) that extract the multiple pieces of text, which were output in step S317 of Fig. 20. In step S4802, the information generation unit 4 analyzes the layout of the bounding boxes using the coordinate values ​​of the bounding boxes. In step S4803, the information generation unit 4 determines whether the bounding boxes are arranged in a single column. Specifically, if all of the bounding boxes 319 to 324 in the auxiliary sign 508 are arranged only vertically (in a column), as in the bounding boxes 319 to 324 shown in Fig. 21, the information generation unit 4 determines that the auxiliary sign 508 is arranged in a single column.

[0070] On the other hand, as shown in FIG. 26 , if the auxiliary sign 509 includes bounding boxes 327 to 329 arranged in the left-right direction (row direction), the information generator 4 determines that the auxiliary sign 509 is not arranged in a row. Specifically, the center coordinates of each bounding box can be compared. If the vertical distance between the center coordinates is smaller than a first reference value and the horizontal distance between the center coordinates is greater than a second reference value, the bounding boxes can be determined to be arranged in the left-right direction (row direction). For example, the auxiliary sign 509 shown in FIG. 26 includes bounding boxes 327 and 329 and bounding boxes 328 and 329 arranged in the left-right direction (row direction). Therefore, the auxiliary sign 509 shown in FIG. 26 is determined not to be arranged in a row.

[0071] If the text is not in a single column (NO in step S4803), the process proceeds to step S4804, where the information generation unit 4 uses bounding boxes surrounding each of the text to organize the multiple texts, divided into lines, into a single column. Specifically, in FIG. 26, the information generation unit 4 combines the horizontally arranged bounding boxes 327 and 329 and the horizontally arranged bounding boxes 328 and 329 into a single bounding box. The bounding box 329 is located midway between the bounding boxes 327 and 328 in the vertical direction. In this case, it is sufficient to add a single bounding box 329, which has only one line. As a result, the combined bounding box can enclose the text "Excluding Sundays and Holidays" and "Excluding December 29th to January 4th." As a result, the auxiliary sign 509 shown in Figure 26 is organized into multiple texts in a single column: "Vehicles with three or more wheels," "7:30-9:00," "Except Sundays and holidays," and "Except December 29th to January 4th."

[0072] The process proceeds to step S4805, where the multiple texts organized in a single column are compared with the dictionary data D51 row by row to divide the text into words and record the word types. Bidirectional segmentation using a bidirectional language model can be used as a division method. FIG. 27 shows an example of the dictionary data D51. The dictionary data D51 includes word types and examples of major words included in each word type.

[0073] In step S4806, the information generation unit 4 compares the multiple texts organized in a single column with the dictionary data D51 to combine two or more texts divided into multiple lines into a single line of text. Specifically, the multiple lines are combined into a single line using a rule base or a learning model. For example, if "date" is written in the first line and "until" is written in the second line, the "date" in the first line and the "until" in the second line are combined into a single line, "until the date." Furthermore, if no matching words can be extracted from the dictionary data D51 before or after an operator word such as "and" that indicates a parallel meaning, the lines above and below the line containing "and" are combined into a single line.

[0074] The information generation unit 4 then compares the text with the dictionary data D51 to generate first regulation information indicated by the auxiliary sign. The first regulation information includes information specifying at least one of the vehicle type, date, day of the week, and time. Here, the first regulation information is generated in the order of vehicle type, date, day of the week, time, and others. The first regulation information includes conditions that apply to the regulatory sign and conditions that do not apply. The information generation unit 4 organizes and outputs these conditions.

[0075] First, in step S4807, the information generating unit 4 extracts the first regulation information for identifying the vehicle type from the text. The detailed procedure of step S4807 will be described with reference to FIG.

[0076] In step S0701, the information generation unit 4 prepares an empty source code FM01 (see FIG. 30 ) in which vehicle types T01 corresponding to regulatory signs and vehicle types F01 not corresponding to regulatory signs are not entered. In step S0702, the information generation unit 4 compares the text with the dictionary data D51 to extract a "vehicle type" and an "operator word" from the text. If an operator word is present in the text (YES in step S0703), the information generation unit 4 proceeds to step S0704 to obtain the processing mode for the operator word. In step S0705, the information generation unit 4 searches the vehicle type tree 224, which shows the vehicle type configuration shown in FIG. 29 . Then, the information generation unit 4 searches for vehicle types corresponding to regulatory signs and vehicle types not corresponding to regulatory signs according to the identification rules indicated by the operator words, referring to the vehicle type tree 224. The data representing the vehicle type tree 224 in FIG. 29 is stored in the storage device 15, similar to the dictionary data D51.

[0077] For example, if the text "medium or larger" is recognized from the auxiliary sign, "medium" is extracted as the vehicle type, and "or larger" is extracted as the operator word. The information generation unit 4 refers to the vehicle type tree 224 shown in FIG. 29 and searches for vehicle types that correspond to the regulatory sign paired with the auxiliary sign and vehicle types that do not correspond to regulatory signs according to the rules for identifying "or larger." Specifically, as vehicle types T01 that correspond to regulatory signs, all vehicle types included in "medium" and "large, etc.", which are vehicle types larger than "medium," are searched for. All vehicle types included in "large, etc." include "extra large," "large," "large cargo," "large passenger," "tourist bus," and "routed bus." As vehicle types F01 that do not correspond to regulatory signs, "regular" and "two-wheeled" are searched for.

[0078] Proceeding to step S0706, the information generation unit 4 inputs the applicable vehicle model T01 and the inapplicable vehicle model F01 into the source code FM01, as shown in FIG. 30. The "True / False" of "obey" in FIG. 30 represents "applicable / inapplicable." "Vehicle type" indicates the vehicle model. In this way, by comparing the text with the dictionary data D51, the information generation unit 4 can combine the characters indicating the vehicle model with the operator word that identifies the vehicle model, thereby identifying the vehicle models to which the second regulation information indicated by the regulatory sign applies and the vehicle models to which the second regulation information does not apply.

[0079] Returning to FIG. 28 , if other lines of the same auxiliary sign contain other text identifying the vehicle type (YES in step S0707), the process proceeds to step S0708 to acquire the other text, and the process returns to step S0702 to execute the above-described process. For example, if the first line contains "large vehicles or larger" and the second line contains "excluding large vehicles," in the process of step S0706 for the text on the second line, "large vehicles," "tourist bus," and "routed bus" are deleted from the corresponding vehicle type T01 in FIG. 30 , and "large vehicles," "tourist bus," and "routed bus" are added to the corresponding vehicle type F01. On the other hand, if other lines of the same auxiliary sign do not contain other text identifying the vehicle type (NO in step S0707), the process proceeds to step S0709, and the information generation unit 4 outputs first regulation information identifying the vehicle type.

[0080] 25, the information generating unit 4 extracts the first restriction information for specifying the date from the text. The detailed procedure of step S4808 will be described with reference to FIG.

[0081] In step S0801, the information generation unit 4 prepares blank source code FM02 (see FIG. 32 ), which does not contain the date T02 corresponding to a regulatory sign or the dates F02-1 and F02-2 not corresponding to a regulatory sign. The process proceeds to step S0802, where the "date" and "operator word" are extracted from the text by comparing the text with dictionary data D51. If an operator word is present in the text (YES in step S0803), the process proceeds to step S0804, where the processing mode for the operator word is acquired. The information generation unit 4 then proceeds to step S0805, where it searches for dates corresponding to regulatory signs and dates not corresponding to regulatory signs according to the identification rule indicated by the operator word.

[0082] For example, if the text "From January 1st to March 31st" is recognized from an auxiliary sign, "January 1st" and "March 31st" are extracted as dates, and "from" and "until" are extracted as operator words. The information generation unit 4 searches for a date that corresponds to the regulatory sign that forms a pair with the auxiliary sign, according to the rules for identifying "from" and "until."

[0083] If another line of the same auxiliary sign contains other text specifying a date (YES in step S0806), the other text is acquired in step S0807. Proceeding to step S0812, the information generation unit 4 determines whether the text (the text on the first line) located above the other text (the text on the second line) is text specifying a date. If the determination is negative, the process returns to step S0801, and new, empty source code FM01 is prepared. On the other hand, if the determination is positive, the process returns to step S0802, and the text on the second line is processed based on the result of the text on the first line.

[0084] For example, if the first line is "January 1st to March 31st" and the second line is "February 1st to February 10th excluded," the second line is processed based on the processing result of the first line. The information generation unit 4 updates the corresponding date T02 to "January 1st to January 31st" and "February 11th to March 31st" (step S0805). Therefore, as shown in FIG. 32, the corresponding date T02 is updated to [{"start": "01-01", "end": "01-31"}, {"start": "02-11", "end": "03-31"}]. If year information is available, a "year" list is entered in source code FM02 in FIG. 32.

[0085] On the other hand, if the determination in step S0806 is negative, the process proceeds to step S0809, where it is determined whether or not there is first regulation information (step S4807) that identifies the vehicle model. If the determination is positive (YES in step S0809), the process proceeds to step S0810, where the information generation unit 4 integrates the first regulation information for vehicle model and date by adding the first regulation information for date within the range of the relevant vehicle model. Thereafter, the process proceeds to step S0811, where the information generation unit 4 complements the inapplicable dates F02-1 and F02-2 in the source code FM02, as shown in FIG. 32, and outputs the result (steps S0811 and S0812).

[0086] 25, the information generating unit 4 extracts the first restriction information specifying the day of the week from the text. The detailed procedure of step S4809 will be described with reference to FIG.

[0087] In step S0901, the information generation unit 4 prepares an empty source code FM03 (see FIG. 34 ) in which the days of the week T03 corresponding to regulatory signs and the days of the week F03-1 and F03-2 not corresponding to regulatory signs are not entered. In step S0902, the information generation unit 4 compares the text with dictionary data D51 to extract "days of the week" and "operator words" from the text. If an operator word is present in the text (YES in step S0903), the information generation unit 4 proceeds to step S0904 to acquire the processing mode for the operator word. In step S0905, the information generation unit 4 searches for days of the week corresponding to regulatory signs and days of the week not corresponding to regulatory signs according to the identification rule indicated by the operator word.

[0088] For example, if the text "Monday to Friday" is recognized from an auxiliary sign, "Monday" and "Friday" are extracted as days of the week, and "from" is extracted as an operator word. The information generation unit 4 searches for "Monday, Tuesday, Wednesday, Thursday, Friday" as days of the week that correspond to regulatory signs paired with the auxiliary sign, according to the rules for identifying "from." The information generation unit 4 may also search for "Saturday, Sunday" as days of the week that do not correspond to regulatory signs.

[0089] If another line of the same auxiliary sign contains other text specifying a day of the week (YES in step S0906), the other text is acquired in step S0907. Proceeding to step S0908, the information generation unit 4 determines whether the text (the text in the first line) located above the other text (the text in the second line) is text specifying a day of the week. If the determination is negative (NO in step S0908), the process returns to step S0901, and new, empty source code FM02 is prepared. On the other hand, if the determination is positive (YES in step S0908), the process returns to step S0902, and the information generation unit 4 processes the text in the second line based on the result of the text in the first line.

[0090] For example, if the first line is "Monday to Friday" and the second line is "excluding Wednesday," the second line is processed based on the processing result of the first line. That is, the information generating unit 4 updates the days of the week to which the regulatory sign applies to "Monday, Tuesday, Thursday, Friday" T03 as shown in Figure 34 (step S0905).

[0091] On the other hand, if the determination in step S0906 is negative, the process proceeds to step S0909, where the information generation unit 4 determines whether or not there is first regulation information (steps S4807 and S4808) that specifies at least one of the vehicle model and date. If the determination is positive (YES in step S0909), the process proceeds to step S0910, where the information generation unit 4 integrates the first regulation information for the vehicle model, date, and day of the week by adding the first regulation information for the corresponding vehicle model and day of the week within the date range. Thereafter, the process proceeds to step S0911, where the information generation unit 4 supplements the source code FM03 with the days of the week F03-1 and F03-2 for the non-corresponding vehicle model and date, as shown in FIG. 34 (step S0911), and outputs the result (step S0912).

[0092] 25, the information generating unit 4 extracts the first restriction information specifying the time from the text. The detailed procedure of step S4810 will be described with reference to FIG.

[0093] In step S1001, the information generation unit 4 prepares blank source code in which times corresponding to regulatory signs and times not corresponding to regulatory signs are not entered. Proceeding to step S1002, the information generation unit 4 extracts "time" and "operator word" from the text by comparing the text with dictionary data D51. For example, the text related to the time of an auxiliary sign has the start time and end time written before and after a "- (hyphen)," such as "8:00-10:30." Proceeding to step S1003, the information generation unit 4 searches for times corresponding to regulatory signs and times not corresponding to regulatory signs according to the identification rule indicated by the operator word.

[0094] In supplementary signs, the start and end times are always written before and after a hyphen (-). For example, if "8:00-10:30" is written, the corresponding time is recorded as [{"start_time": "08:00", "end_time": "10:30"}]. If it is for the whole day, it is recorded as [{"start_time": "00:00", "end_time": "24:00"}].

[0095] If another line of the same auxiliary sign contains other text specifying a time (YES in step S1004), the other text is acquired in step S1005. Proceeding to step S1006, the information generation unit 4 determines whether the text (the text in the first line) located above the other text (the text in the second line) is text specifying a time. If the determination is negative (NO in step S1006), the process returns to step S1001, where new, empty source code is prepared. On the other hand, if the determination is positive (YES in step S1006), the process returns to step S1002, where the information generation unit 4 processes the text in the second line based on the result of the text in the first line.

[0096] For example, if the first line is "8:00-10:30" and the second line is "17:00-19:00," the second line is processed based on the processing result of the first line. That is, the times at which the regulatory signs apply are updated to [{"start_time": "08:00", "end_time": "10:30"}, {"start_time": "17:00", "end_time": "19:00"}] (step S1003).

[0097] On the other hand, if the determination in step S1004 is negative, the process proceeds to step S1007, where the information generation unit 4 determines whether or not there is first regulation information (steps S4807, S4808, S4809) that specifies at least one of the vehicle type, date, and day of the week. If the determination is positive (YES in step S1007), the process proceeds to step S1008, where the information generation unit 4 integrates the first regulation information for the vehicle type, date, day of the week, and time by adding the first regulation information for the time within the range of the relevant vehicle type, date, and day of the week. Then, the process proceeds to step S1009, where the information generation unit 4 complements the time within the non-corresponding vehicle type, date, or day of the week (step S1009) and outputs the result (step S1010). Figure 37 shows an example (source code FM04) of first regulation information that specifies the vehicle type, date, day of the week, and time T04 extracted and integrated from the text 105 shown in Figure 36 in steps S4807 to S4810 of Figure 25.

[0098] Returning to step S4811 in FIG. 25 , the information generation unit 4 extracts other first regulation information indicated by the auxiliary sign, excluding the vehicle type, date, day of the week, and time. Because the other first regulation information is basically unique expressions, the information generation unit 4 detects unique expressions and records them as first regulation information. Specifically, the information generation unit 4 performs a closing analysis on text including the vehicle type. Specifically, if there is any decorative part of the vehicle type, the information generation unit 4 adds the decorative part to "vehicle_type_additional" according to the corresponding vehicle type.

[0099] For example, consider the text "Excluding two-wheeled vehicles and light vehicles turning left, and vehicles going straight that are related to roadside houses," in which case the information generating unit 4 complements the inapplicable vehicle type F10 shown in source code FM10 in Fig. 38 with decorative parts (turning left, going straight, roadside houses) for each vehicle type, as shown by reference numerals F11-1 and F11-2 in Fig. 39.

[0100] The process proceeds to step S4812 in Fig. 25, where the information generation unit 4 supplements the missing information on a rule basis. The information generation unit 4 supplements the source code FM20 in Fig. 40 with first restriction information T21, F21 that specifies the missing date, day of the week, and time, as shown in source code FM21 in Fig. 41.

[0101] The above procedure completes step S408 (organizing text logic) in Fig. 22. The information generator 4 outputs the first regulation information including the vehicle type, date, day of the week, time, and other information.

[0102] In step S409, the information generation unit 4 integrates the second regulation information indicated by the regulatory sign and the first regulation information indicated by the auxiliary sign for each pair. In step S410, the information generation unit 4 determines, based on the first regulation information, whether the second regulation information indicated by the regulatory sign paired with the auxiliary sign applies to the vehicle equipped with the driving assistance device of FIG. 1 . Specifically, the information generation unit 4 determines whether the second regulation information indicated by the regulatory sign paired with the auxiliary sign applies to the vehicle by comparing the make or model of the vehicle to be subjected to driving assistance, the current date, day of the week, and time with the first regulation information. Based on the determination result, the information generation unit 4 generates second regulation information to be notified to the user and outputs it to the output unit 5. Depending on the determination result, the information generation unit 4 changes the method of presenting the second regulation information to the vehicle occupants. For example, if the second regulation information applies to the vehicle, the information generation unit 4 presents the second regulation information to the occupants, including a notice that the second regulation information applies to the vehicle. On the other hand, if the second restriction information does not apply to the vehicle, the second restriction information is presented to the occupant, including a message that the second restriction information does not apply. In addition, as will be described below, the information generating unit 4 may change the method of presenting the second restriction information to the occupant of the vehicle depending on the determination result by changing the output form from the output unit 5.

[0103] The information generation unit 4 may calculate the progress status of an intersection, such as one-way or no-entry status, from satellite images obtained from GPS signals, and generate information notifying the user of lanes to be careful of when entering the intersection. The information generation unit 4 may calculate the priority of a planned driving route determined by a navigation system. For example, if the planned driving route includes a right turn at an intersection, the information generation unit 4 calculates a lower priority than oncoming vehicles traveling straight or turning left at the intersection. The information generation unit 4 may notify the user of lanes to be careful of based on the calculation results. For example, if the intersection is a three-way intersection and the road on the left is closed to vehicles, turning right has the highest priority. Therefore, the information generation unit 4 may notify the user that turning right at the intersection has the highest priority. This allows the occupants to drive the vehicle with peace of mind, regardless of the content of the signs.

[0104] The output unit 5 in FIG. 1 outputs information to be notified to the user to the presentation unit 14. For example, audio data output from a speaker (presentation unit 14) is output to the presentation unit 14. Alternatively, the output unit 5 may output the recognized second restriction information and text directly to the HUD (presentation unit 14). The output unit 5 may change the method of presenting the second restriction information to the vehicle occupants depending on the determination result. For example, the output unit 5 may output the text directly to the HUD (presentation unit 14) when the second restriction information is applied to the vehicle, and may output audio data indicating that the second restriction information is not applied to the vehicle to the speaker (presentation unit 14) when the second restriction information is not applied to the vehicle.

[0105] When the regulatory sign indicates that travel in any direction other than the designated direction is prohibited, the output unit 5 may render drivable lanes and non-drivable lanes on the HUD (presentation unit 14). For example, the output unit 5 may output information image data to the HUD (presentation unit 14) that displays drivable lanes in green and non-drivable lanes in red.

[0106] As described above, according to this embodiment, the following advantageous effects can be obtained.

[0107] The detection unit 2 detects regulatory signs and auxiliary signs that supplement the regulatory signs from images captured by a camera mounted on the vehicle. The detection unit 2 detects pairs of regulatory signs and auxiliary signs using bounding boxes that surround each of the regulatory signs and auxiliary signs. The recognition unit 3 recognizes text from characters included in the auxiliary signs. The information generation unit 4 generates first regulatory information indicated by the auxiliary signs by comparing the text with dictionary data D51. The information generation unit 4 determines, based on the first regulatory information, whether second regulatory information indicated by the regulatory sign paired with the auxiliary sign applies to the vehicle. The information generation unit 4 changes the method of presenting the second regulatory information to vehicle occupants depending on the determination result. By considering not only regulatory signs but also auxiliary signs, the second regulatory information indicated by the regulatory signs can be appropriately notified to the driver.

[0108] Recognizing text from characters included in the auxiliary sign includes recognizing multiple pieces of text divided into lines from the characters included in the auxiliary sign. The text can be correctly recognized from the characters included in the auxiliary sign.

[0109] The generating of the first restriction information indicated by the auxiliary sign further includes organizing the plurality of lines of text into a single column using a bounding box that surrounds each of the plurality of lines of text, thereby enabling the text to be correctly recognized from the characters included in the auxiliary sign.

[0110] Extracting the first restriction information indicated by the auxiliary sign further includes integrating two or more pieces of text divided into multiple lines into one line of text by comparing the multiple pieces of text organized in a single column with the dictionary data D51. This allows the text to be correctly recognized from the characters included in the auxiliary sign.

[0111] The first regulatory information indicated by the auxiliary sign includes information specifying at least one of the vehicle type, date, day of the week, and time. At least one of the vehicle type, date, day of the week, and time corresponding to the regulatory sign can be specified.

[0112] The information generation unit 4 compares the text with the dictionary data D51 to combine characters indicating the vehicle type, date, day of the week, or time with operator words specifying the range of the vehicle type, date, day of the week, or time. This allows the information generation unit 4 to identify the vehicle type, date, day of the week, and time to which the second regulation information applies, and the vehicle type, date, day of the week, and time to which the second regulation information does not apply. This allows the second regulation information indicated by the regulatory sign to be appropriately notified to the driver.

[0113] The first regulation information indicated by the auxiliary sign includes information specifying each of the vehicle type, date, day of the week, and time. The information generation unit 4 integrates the extracted information specifying each of the vehicle type, date, day of the week, and time. This makes it possible to accurately identify the vehicle type, date, day of the week, and time corresponding to the regulatory sign.

[0114] The information generating unit 4 compares the vehicle model, current date, day of the week, and time with the first regulatory information to determine whether the second regulatory information indicated by the regulatory sign paired with the auxiliary sign applies to the vehicle. By taking into account the first regulatory information indicated by the auxiliary sign, the second regulatory information indicated by the regulatory sign can be appropriately notified to the driver.

[0115] The recognition unit 3 accumulates consecutively recognized text in a time filter having a queue structure, and recognizes text when the same text accumulates more than a threshold value, thereby enabling the recognition unit 3 to recognize the text of auxiliary signs that are stable on the time axis.

[0116] If the regulatory sign belongs to the "No Travel Outside Designated Direction" category, the recognition unit 3 calculates the angle of the arrow and the end point included in the regulatory sign. The information generation unit 4 generates lane information indicating the lanes that can be traveled and the lanes that cannot be traveled as second regulatory information by mapping the arrow and the end point to multiple lanes that intersect at the intersection corresponding to the regulatory sign based on the angle of the arrow and the end point. The second regulatory information indicated by the "No Travel Outside Designated Direction" can be notified to the driver in an easy-to-understand manner.

[0117] The lowest edge of the edge points included in the regulatory sign is identified as the lane the vehicle is currently traveling in. The lane the vehicle is currently traveling in can be detected from the no-go sign.

[0118] The detection unit 2 detects pairs of regulatory signs and auxiliary signs that are related to each other based on the positions of the regulatory signs and auxiliary signs. The positions of the regulatory signs and auxiliary signs include the distance from the vehicle to each sign and the vertical and horizontal distances between the signs. This allows accurate detection of pairs of regulatory signs and auxiliary signs.

[0119] Although the embodiments of the present invention have been described above, the descriptions and drawings that form part of this disclosure should not be understood to limit the present invention. Various alternative embodiments, examples, and operating techniques will become apparent to those skilled in the art from this disclosure.

[0120] In the present embodiment, the controller 11 shown in FIG. 1 is mounted on a vehicle that is a target of driving assistance. However, this is not limiting. In another embodiment, the controller 11 may be located on a cloud. Signals output from the on-board sensors 12 and the GPS receiver 13 mounted on the vehicle are transmitted to the controller 11 on the cloud via a mobile communication network, and driving assistance information output from the controller 11 is output from the display unit 14 mounted on the vehicle via the mobile communication network and presented to the occupant. As described above, in another embodiment, the controller 11 may be configured as one or more virtualized servers, and a service (cloud server) may be provided in which the servers can be accessed via the Internet. In this case, the dictionary data D51 stored in the storage device 15 may also be located on the cloud, similar to the controller 11.

[0121] 1 Input unit 2 Detection unit 3 Recognition unit 4 Information generation unit 5 Output unit 11 Controller 12 On-board sensor 13 GPS receiver 14 Presentation unit 15 Storage device 101 Angle 102 to 104 Text line 105 Text 201 to 218, 222, 225 Regulatory sign 221 Do not proceed outside specified direction (regulatory sign) 224 Vehicle type tree 301 to 329 Bounding box 401 to 403 Sign group 501 to 509 Auxiliary sign D51 Dictionary data

Claims

1. A driving assistance method comprising: detecting regulatory signs and auxiliary signs that supplement the regulatory signs from an image captured by a camera mounted on a vehicle; detecting pairs of regulatory signs and auxiliary signs using bounding boxes that surround each of the regulatory signs and auxiliary signs; recognizing text from characters contained in the auxiliary signs; generating first regulatory information indicated by the auxiliary signs by comparing the text with dictionary data; determining, based on the first regulatory information, whether second regulatory information indicated by the regulatory sign that forms a pair with the auxiliary sign applies to the vehicle; and changing the manner in which the second regulatory information is presented to the vehicle occupants depending on the result of the determination.

2. A driving assistance method according to claim 1, wherein recognizing text from characters included in the auxiliary sign includes recognizing a plurality of texts divided into lines from the characters included in the auxiliary sign.

3. The driving assistance method according to claim 2, wherein generating the first regulation information indicated by the auxiliary sign includes organizing the plurality of texts, which have been divided into lines, into a single column using a bounding box that surrounds each of the plurality of texts.

4. A driving assistance method as claimed in claim 3, wherein generating the first regulation information indicated by the auxiliary sign further includes integrating two or more pieces of text divided into multiple lines into one line of text by comparing multiple pieces of text organized in a single column with the dictionary data.

5. A driving assistance method according to any one of claims 1 to 4, wherein the first regulation information indicated by the auxiliary sign includes information specifying at least one of the vehicle type, date, day of the week, and time.

6. A driving assistance method as described in claim 5, wherein by comparing the text with dictionary data, characters indicating the vehicle type, date, day of the week, or time are combined with an operator word specifying a range of the vehicle type, date, day of the week, or time period, to identify the vehicle type, date, day of the week, and time period to which the second regulation information applies, and the vehicle type, date, day of the week, and time period to which the second regulation information does not apply.

7. A driving assistance method according to any one of claims 1 to 6, wherein the first regulation information indicated by the auxiliary sign includes information identifying each of the vehicle type, date, day of the week, and time period, and the extracted information identifying each of the vehicle type, date, day of the week, and time period is integrated.

8. A driving assistance method according to any one of claims 1 to 7, wherein the vehicle type, current date, day of the week, and time are compared with the first regulatory information to determine whether the second regulatory information indicated by the regulatory sign paired with the auxiliary sign applies to the vehicle.

9. A driving assistance method according to any one of claims 1 to 8, further comprising: storing consecutively recognized texts in a time filter having a queue structure; and recognizing the text when the same texts are stored in a number equal to or greater than a threshold.

10. A driving assistance method according to any one of claims 1 to 9, wherein the regulatory sign is in a category of "no travel outside of a specified direction", the angle of an arrow and an end point included in the regulatory sign is calculated, and based on the angle of the arrow and the end point, the arrow and the end point are mapped to a plurality of lanes intersecting at an intersection corresponding to the regulatory sign, thereby generating lane information indicating the lanes that can be traveled and the lanes that cannot be traveled as the second regulatory information.

11. A driving assistance method according to claim 10, wherein the lowest edge of the edge points included in the regulatory sign is identified as the lane in which the vehicle is currently traveling.

12. A driving assistance method according to any one of claims 1 to 11, further comprising detecting a pair of regulatory signs and auxiliary signs that are related to each other based on the positions of the regulatory signs and auxiliary signs.

13. A driving assistance device comprising an arithmetic processing unit, a memory unit, and an input / output port, wherein the arithmetic processing unit: detects regulatory signs and auxiliary signs that supplement the regulatory signs from an image input to the input / output port, the image being taken by a camera mounted on a vehicle; detects pairs of regulatory signs and auxiliary signs using bounding boxes that surround the regulatory signs and auxiliary signs; recognizes characters contained in the auxiliary signs to extract text; extracts first regulatory information indicated by the auxiliary signs by comparing the text with dictionary data stored in the memory unit; determines, based on the first regulatory information, whether second regulatory information indicated by the regulatory sign that forms a pair with the auxiliary sign applies to the vehicle; and changes a method of presenting the second regulatory information to occupants of the vehicle depending on the result of the determination.

14. A computer-readable recording medium having recorded thereon a program for causing a computer to execute the following steps: detecting regulatory signs and auxiliary signs that supplement the regulatory signs from images captured by a camera mounted on a vehicle; detecting pairs of regulatory signs and auxiliary signs using bounding boxes that surround the regulatory signs and auxiliary signs; recognizing characters contained in the auxiliary signs and extracting text; extracting first regulatory information indicated by the auxiliary signs by comparing the text with dictionary data; determining, based on the first regulatory information, whether second regulatory information indicated by the regulatory sign that forms a pair with the auxiliary sign applies to the vehicle; and changing the method of presenting the second regulatory information to occupants of the vehicle depending on the result of the determination.

Images