Sign recognition device and sign recognition method

Abstract

Provided is a processing device capable of improving the performance of recognizing an object in an image such as a restriction lifting sign. This sign recognition device is characterized by comprising: a candidate sign recognition unit that recognizes a candidate sign of a predetermined type from an image; a paired sign search unit that associates, if a candidate for a first sign is recognized as the candidate sign of the predetermined type, the first sign with a second sign installed in correspondence with the first sign; a color information extraction unit that extracts, if there is a second sign installed in correspondence with the first sign, information on the color of the second sign associated with the first sign; and a threshold value determination unit that determines a threshold value for extracting information on the color of the candidate for the first sign on the basis of the reliability of the information on the color of the second sign extracted by the color information extraction unit.

Description

[Technical field]

[0001] The present invention relates to a sign recognition device and a sign recognition method used in an in-vehicle sensing device such as a stereo camera device. [Background technology]

[0002] In recent years, the demands for various recognition functions of in-vehicle sensing devices have become more sophisticated in order to realize advanced vehicle control according to the environment outside the vehicle, such as autonomous driving control and driving assistance control. A stereo camera device, which is one type of in-vehicle sensing device, is a device that simultaneously detects visual information from an image and distance information of objects in the image, so it can grasp the various environments around the vehicle (pedestrians, other vehicles, three-dimensional objects, road surfaces, road signs, signboards, etc.) in detail, greatly contributing to improving the safety of autonomous driving control and driving assistance control.

[0003] Some vehicles equipped with stereo camera devices use recognized traffic signs for vehicle acceleration / deceleration control. In addition, Euro NCAP, an evaluation index for advanced driver assistance systems, has also included evaluation items related to speed assistance systems (SAS, Speed ​​Assistance Systems), and their importance is increasing. In Japan, the importance of automatic speed control functions (ISA, Intelligent Speed ​​Assistance) that suppress speeding due to driver inattention or erroneous operation is also increasing, and it is necessary to improve the recognition accuracy of sign recognition functions that obtain speed limit information. Furthermore, it is necessary to recognize conditional speed limits such as valid sections, time limits, and vehicle type limits, so there is a demand to expand the number of recognizable traffic signs.

[0004] Due to the above circumstances, various technologies have been proposed for on-board camera devices that recognize the environment in front of the vehicle. For example, Patent Document 1 is a conventional technology that focuses on improving the recognition performance of deregulation signs (signs that lift restrictions such as speed limits and no overtaking), and the abstract of this document states that "We provide a camera device that can improve the detection accuracy of deregulation signs. The calculation processing unit (search unit) searches for diagonal line candidates from the image (diagonal line search 504). The calculation processing unit (selection unit) selects a diagonal line candidate of a deregulation sign that indicates the lifting of restrictions corresponding to this sign from the detected diagonal line candidates (diagonal line determination 505). The calculation processing unit (deregulation sign identification unit) identifies the deregulation sign from the image of the selected diagonal line candidate of the deregulation sign (identification process 506)." [Prior art documents] [Patent documents]

[0005] [Patent Document 1] International Publication No. 2020 / 071132 Summary of the Invention [Problem to be solved by the invention]

[0006] The camera device in Patent Document 1 identifies unrestricted signs from images of candidate diagonal lines, so if an image pattern like that shown in Figure 11(a) is present in part of the captured image, the diagonal line above the speed limit sign can be correctly identified as a unrestricted sign.

[0007] However, in the camera device of Patent Document 1, if part of the captured image contains an image pattern such as that shown in Figure 11(b), the length, thickness, and angle of the upper end of the pole extending above the speed limit sign are similar to the diagonal line portion of the non-restriction sign, and therefore there is a possibility that the pole above the sign will be mistakenly detected as a non-restriction sign.

[0008] The present invention has been made in consideration of the above-mentioned problems, and an object of the present invention is to provide a sign recognition device and a sign recognition method that suppresses the erroneous detection of image patterns such as pillars, utility poles, tree branches, etc. that are similar to the design of deregulation signs as non-existent deregulation signs. [Means for solving the problem]

[0009] In order to solve the above problems, a sign recognition device of the present invention has a sign candidate recognition unit that recognizes sign candidates of a predetermined type from an image; a paired sign search unit that, when a first sign candidate is recognized as a candidate for the predetermined type of sign, associates the first sign with a second sign that is installed in correspondence with the first sign; a color information extraction unit that, when a second sign installed in correspondence with the first sign exists, extracts color information of the second sign associated with the first sign; and a threshold determination unit that determines a threshold for use in extracting the color information of the first sign candidate, based on the reliability of the color information of the second sign extracted by the color information extraction unit. Effect of the Invention

[0010] According to the sign recognition device and sign recognition method of the present invention, it is possible to prevent image patterns of pillars, utility poles, tree branches, etc. that are similar in design to unrestricted signs from being mistakenly detected as non-existent unrestricted signs.

[0011] Other objects, configurations and advantages than those mentioned above will become apparent from the following description of the embodiments. [Brief description of the drawings]

[0012] [Figure 1] FIG. 1 is a functional block diagram showing a schematic configuration of a stereo camera device according to an embodiment. [Diagram 2] 4 is a flowchart of a basic process of the stereo camera device according to the embodiment. [Diagram 3] 4 is a flowchart of a sign recognition process of the sign recognition device according to an embodiment. [Figure 4] 4 is a flowchart of a circular object extraction process of the sign recognition device according to an embodiment. [Figure 5A] Speed ​​limit sign image pattern [Figure 5B] 6 is an explanatory diagram of a center estimation process and a radius estimation process for an edge image of a speed limit sign. [Figure 6] 4 is a flowchart of a valid line search process of the sign recognition device according to an embodiment. [Figure 7A] 7 is an explanatory diagram of the left and right edge searching process in FIG. 6; [Figure 7B] FIG. 7 is an explanatory diagram of the line detection process in FIG. 6. [Figure 7C] FIG. 7 is an explanatory diagram of the valid line determination process in FIG. 6. [Figure 8A] An image of the paired sign search process in Japan. [Figure 8B] An illustration of the paired sign search process in other countries. [Figure 9] 4 is a flowchart of a paired sign search process of the sign recognition device according to an embodiment. [Figure 10A] An illustration showing the locations of paired deregulation signs in Japan. [Figure 10B] An illustration showing the location of deregulation paired signs in other countries. [Figure 11] A diagram illustrating the design of deregulation signs and similar objects. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0013] Hereinafter, a sign recognition device according to an embodiment of the present invention will be described with reference to the drawings. In each drawing, parts having the same functions are designated by the same reference numerals, and repeated description may be omitted.

[0014] <Outline of the configuration of the stereo camera device> FIG. 1 is a functional block diagram showing a schematic configuration of a stereo camera device 100 according to an embodiment of the present invention. The stereo camera device 100 is a type of on-board sensing device mounted on a vehicle that executes automatic driving control or driving assistance control, and is a device that recognizes the vehicle exterior environment (e.g., white lines on a road, pedestrians, other vehicles, other three-dimensional objects, traffic signals, traffic signs, and lit lamps) based on image information of a shooting target area in front of the vehicle. The stereo camera device 100 is also a device that determines a control policy (such as braking and steering) of the vehicle according to the vehicle exterior environment, and outputs the control policy to an ECU (Electronic Control Unit) via an on-board network CAN (Controller Area Network). The ECU then controls the braking system and steering system of the vehicle according to the control policy of the stereo camera device 100, thereby safely decelerating the vehicle and allowing the vehicle to avoid obstacles.

[0015] As shown in Fig. 1, the stereo camera device 100 of this embodiment includes a camera 1 and a sign recognition device 2. The camera 1 is composed of a left camera 1L and a right camera 1R arranged side by side, and captures a pair of left and right images (left image P L , Right image P R ) to output.

[0016] The sign recognition device 2 recognizes traffic signs in an image captured by the camera 1. Specifically, the sign recognition device 2 is a single or multiple computer units including a processor such as a CPU (Central Processing Unit), and memories such as a ROM (Read Only Memory), a RAM (Random Access Memory), and an SSD (Solid State Drive). Each function of the sign recognition device 2 is realized by the processor executing a program stored in the ROM. The RAM and the SSD store intermediate data of calculations by the program and data such as images of the camera 1. As described above, the stereo camera device 100 is a device that can recognize the outside environment (white lines on the road, pedestrians, other vehicles, etc.) other than traffic signs. However, since the present invention focuses on the traffic sign recognition function of the stereo camera device 100, the device denoted by the reference symbol 2, which should originally be called an environment recognition device, is called the sign recognition device 2 in this embodiment.

[0017] As shown in the figure, the sign recognition device 2 has an image input interface 21, an image processing unit 22, an arithmetic processing unit 23, a storage unit 24, a CAN interface 25, a monitoring processing unit 26, and an internal bus 27 that interconnects these units. Each unit will be outlined below.

[0018] The image input interface 21 is an interface that controls the image sensor of the camera 1 and captures a captured image. The image captured by the image input interface 21 is transmitted to the image processing unit 22 and the calculation processing unit 23 via the internal bus 27.

[0019] The image processor 22 outputs the left image P captured by the imaging element of the left camera 1L. L and the right image P captured by the image sensor of the right camera 1R. RThe image processor 22 compares the left and right images, performs image correction such as device-specific deviation correction and noise interpolation caused by the imaging element on each image, and then stores the corrected left and right images in the storage unit 24. The image processor 22 also extracts corresponding points between the corrected left and right images, and calculates distance information for each pixel on the images based on the parallax between the corresponding points. The calculated distance information is then stored in the storage unit 24.

[0020] The arithmetic processing unit 23 recognizes various objects necessary for perceiving the environment around the vehicle, using the corrected left and right images and distance information stored in the memory unit 24. The various objects recognized here include, for example, pedestrians, other vehicles, other obstacles, traffic signals, traffic signs, vehicle taillights and headlights, etc. These recognition results and a part of intermediate calculation results are recorded in the memory unit 24. Furthermore, the arithmetic processing unit 23 determines a control policy for the vehicle using the recognition results of various objects.

[0021] The storage unit 24 stores data being processed or after being processed by the image processing unit 22 and the arithmetic processing unit 23. Various pieces of information required to recognize various objects are registered in advance in the storage unit 24. Specifically, the storage unit 24 is a memory such as the above-mentioned RAM or SSD, and includes an image buffer 24a and a classifier 24b, which will be described later.

[0022] The CAN interface 25 is an interface that transmits various objects recognized by the arithmetic processing unit 23 and the vehicle control policy determined by the arithmetic processing unit 23 to the in-vehicle network CAN. The ECU connected to the in-vehicle network CAN executes braking of the vehicle, warning of the driver, etc. based on the object recognition results and the control policy transmitted via the in-vehicle network CAN.

[0023] The monitoring processor 26 monitors whether the above-mentioned components are operating abnormally or whether errors are occurring during data transfer, and is a mechanism for preventing abnormal operations.

[0024] <Basic processing of stereo camera device> Here, the basic processing performed by the stereo camera device 100 of this embodiment will be described with reference to the flowchart of FIG.

[0025] First, in step S1L, the image processor 22 captures a left image P L Next, in step S2L, the image processor 22 captures the captured left image P L After applying image correction and other processing to absorb the inherent characteristics of the imaging element, the corrected left image P L is stored in the image buffer 24a of the storage unit 24.

[0026] In addition, in step S1R, in synchronization with steps S1L and S2L, the image processor 22 transmits a right image P to the right camera 1R. R Next, in step S2R, the image processor 22 captures the captured right image P R After applying image correction and other processing to absorb the inherent characteristics of the imaging element, the corrected right image P R is stored in the image buffer 24a of the storage unit 24.

[0027] In step S3, the image processor 22 converts the corrected left image P L and the right image P R The left and right images are compared and the parallax of the left and right images is calculated. The parallax of the left and right images makes it clear which points on the left and right images correspond to which points on the left and right images of a certain point on the target object, so that the distance to the target object can be calculated according to the principle of triangulation. The distance information calculated in this step is also stored in the memory unit 24.

[0028] In step S4, the calculation processing unit 23 recognizes various objects using the corrected left and right images and distance information stored in the image buffer 24a. Objects to be recognized include pedestrians, other vehicles, other three-dimensional objects, traffic signs, traffic signals, tail lights, etc. For various object recognition in this step, the classifier 24b is used as necessary. The classifier 24b stores and records, for example, the features of the objects to be recognized as machine learning data. Details of the sign recognition process (step S40) executed in this step will be described later.

[0029] In step S5, the arithmetic processing unit 23 determines a vehicle control policy taking into consideration the recognition results of various objects in step S4 and the state of the host vehicle (speed, steering angle, etc.). For example, if the host vehicle is exceeding the speed limit, a control policy is determined such as issuing a warning to the occupants or braking the host vehicle. In this embodiment, as described below, the arithmetic processing unit 23 determines the control of the host vehicle based on the information of the sign recognized in step S4.

[0030] In step S6, the CAN interface 25 outputs the various objects recognized in step S4 and the control policy of the vehicle determined in step S6 to the external ECU through the in-vehicle network CAN. As a result, the ECU can issue a warning to the occupants or brake the vehicle according to the control policy determined by the stereo camera device 100.

[0031] <Sign Recognition Processing> Next, a sign recognition process (step S40), which is one aspect of the various object recognition process (step S4) in FIG. 2, will be described with reference to the flowchart in FIG.

[0032] First, in step S41, the calculation processing unit 23 extracts an image pattern including a circular object from the corrected image in the image buffer 24a. This process is divided into an edge image generation process (step S41a), a center estimation process (step S41b), and a radius estimation process (step S41c), and each process will be described below in order with reference to Figs. 4 to 5B.

[0033] First, in the edge image generation process (step S41a), the calculation processing unit 23 extracts edge components of the corrected image in the image buffer 24a to generate an edge image P E1 For example, if a speed limit sign pattern as shown in FIG. 5A is captured in a portion of the corrected image, in this step, an edge image P E1 is generated.

[0034] Next, in the center estimation process (step S41b), the calculation processing unit 23 estimates the center of the circular object. Specifically, as shown in FIG. 5B, the calculation processing unit 23 estimates the center of the circular object. E1 A line segment Ln is drawn from each edge in the normal direction, and the point where a certain number of intersections of the line segment Ln overlap is estimated to be the center C. In the example of FIG. 5B, the center C is estimated to be between the numbers "11" and "0".

[0035] In the radius estimation process (step S41c), the calculation processing unit 23 calculates the edge image P E1 The radius of the circular object is estimated based on the histogram of the circular object. For example, in the case of estimating the radius of the circular object in FIG. 5B, a histogram is created with the horizontal axis representing the distance from the center C and the vertical axis representing the number of edges. Specifically, first, a circle with a radius r=0 is placed at the center C, and the radius r of this circle is gradually increased. Then, when this circle overlaps with an edge, the number of overlapping edges is counted as the number of edges at that distance. By continuing this process until the radius r reaches a predetermined value (a radius appropriate for a traffic sign), the histogram of the circular object in FIG. 5B can be generated. In the histogram based on FIG. 5B generated in this way, since many edges exist at two positions, the horizontal axis position corresponding to the distance from the center C to the first edge group E1 and the horizontal axis position corresponding to the distance from the center C to the second edge group E2, it can be inferred that both the first edge group E1 and the second edge group E2 are approximately annular. In this case, the calculation processing unit 23 estimates that the second edge group E2, which has a larger radius, is the outer diameter of the circular object.

[0036] Through the above-described processing, the calculation processing unit 23 can extract a circular object and its features (center position, radius) from the corrected image in the image buffer 24a.

[0037] In step S42, the calculation processing unit 23 acquires color information of the circular object extracted in step S41. This is for the purpose of using the color information of the actual sign as a reference for comparison when determining the authenticity of the deregulation sign in step S4b described later. Note that step S42 may be processed after step S43 described later, or may be processed in parallel with step S43.

[0038] In step S43, the arithmetic processing unit 23 identifies whether the circular object extracted in step S41 is a traffic sign candidate. Specifically, the arithmetic processing unit 23 executes an identification process using the classifier 24b on the image pattern of the corrected image in the image buffer 24a, which corresponds to the position of the circular object extracted in step S41. As a result, it is recognized whether the circular object extracted in step S41 is a traffic sign candidate, and if the circular object is a traffic sign candidate, the contents of the traffic sign (e.g., speed limit) are recognized. A first identification score indicating the likelihood of a traffic sign when identified as a traffic sign candidate is output in this identification process and stored in a memory. The traffic sign candidate identified in this step is hereinafter referred to as the "main sign." Note that if a circular object is not extracted in step S41, this step may be omitted.

[0039] In steps S44-45 and 47-4b, candidates for non-restrictive signs are searched for.

[0040] First, in step S44, the calculation processing unit 23 searches for a valid straight line that can be a candidate for a restriction release sign from the corrected image in the image buffer 24a. This valid straight line search process is divided into left and right edge search process (step S44a), straight line detection process (step S44b), and valid straight line determination process (step S44c) in detail, and each process will be described below in order with reference to Figs. 6 to 7C.

[0041] First, in the left and right edge search process (step S44a), the calculation processing unit 23 scans the pixels of the corrected image in the image buffer 24a from left to right for each row, as shown in FIG. L and the right edge E R If the search is successful, they are extracted as a pair of left and right edges. By performing this process on the entire area of ​​the corrected image, an edge image P E2 (Figure 7B is an excerpt of an edge image) can be generated.

[0042] Next, in the straight line detection process (step S44b), the calculation processing unit 23 detects the edge image P E2 The straight line portion is detected from the edge image P shown in FIG. E2 In the image, there are parts where pairs of left and right edges continue vertically for two or more rows, and parts where there is only one row. In this step, parts where pairs of left and right edges continue vertically for two or more rows are detected as straight lines.

[0043] In the valid straight line determination process (step S44c), the calculation processing unit 23 determines whether the straight line detected in step S44b is a valid candidate for a deregulation sign, and extracts valid straight lines that can be candidates. Deregulation signs in Japan have a circular sign with a prominent thick blue diagonal line (see the upper sign in FIG. 11(a)), and many foreign countries also have circular sign with prominent thick diagonal lines (e.g., Germany, Poland, Sweden, etc.). Therefore, in this step, straight line parts that can be candidates for deregulation signs are extracted from the corrected image, taking into account the design features of the diagonal line part of the deregulation sign.

[0044] When using the stereo camera device 100, the distance to the straight line portion detected in step S44b can be calculated, so that it is possible to roughly calculate the width and height of the diagonal line portion of the restriction release sign on the corrected image when it is assumed that the restriction release sign is located at the calculated distance. Therefore, when the width of the edge pair illustrated in Fig. 7B is too narrow or too wide, the straight line portion formed by the edge pair can be determined as an invalid straight line that cannot be a candidate for the restriction release sign, and can be excluded from the valid straight lines.

[0045] Similarly, as shown in Fig. 7C, if the height of the straight line portion is short and does not reach the lower limit of the valid range considered appropriate for the height of the diagonal line portion of a deregulation sign (Fig. 7C(b)), or if it is long and exceeds the upper limit (Fig. 7C(c)), then the straight line portion is determined to be an invalid straight line that cannot be a candidate for a deregulation sign, and is excluded from the valid straight lines. On the other hand, if the height of the straight line portion falls within the valid range (Fig. 7C(a)), then the straight line portion is extracted as a valid straight line.

[0046] By the processing of step S44 described above, image patterns of utility poles or pillars that have a shape significantly different from the diagonal line portion of a deregulation sign can be excluded from candidates for deregulation signs, and only image patterns of straight lines that may be deregulation signs can be extracted.

[0047] Next, in step S45, the calculation processing unit 23 determines whether or not a valid straight line has been extracted by the process of step S44. If a valid straight line has not been extracted, the process proceeds to step S46, and if a valid straight line has been extracted, the process proceeds to step S47.

[0048] In step S46, the calculation processing unit 23 tracks the real sign (traffic sign) candidate identified in step S43. Here, "tracking" refers to storing in a memory in association with shape information and position information of the sign candidate obtained in step S41 and color information obtained in step S42, which are acquired for each frame or for each image obtained at a predetermined time interval, in a time series. Specifically, the type of the real sign candidate identified in step S43 (e.g., a speed limit sign) is registered in a tracking list L1, which is a list for registering the type of traffic sign being imaged by the stereo camera device 100. Note that when the vehicle passes a real sign candidate registered in the tracking list L1 and is no longer able to image the real sign candidate, the information of the real sign candidate registered in the tracking list L1 is transferred to the recognized list L2.

[0049] On the other hand, in step S47, the arithmetic processing unit 23 calculates color information of the valid straight line extracted in step S44. For example, in the case of searching for a candidate for a Japanese deregulation sign (see FIG. 11(a)), the arithmetic processing unit 23 arranges a window in the image having a shape in which the valid straight line fits, and calculates the blue score in this window. Here, the blue score is calculated, for example, by the following method. That is, in the case where the color of each pixel in the image is defined by the three primary colors of red (R), green (G), and blue (B), the blue score is calculated by dividing the area of ​​the blue pixels in the window by the total area of ​​the window. The calculated blue score is stored in the memory. Note that, although blue is used as an example here, color information other than blue may be extracted depending on the sign design, and calculating the scores of these colors is collectively referred to as "obtaining color information."

[0050] As described above, for example, since deregulation signs in Japan have a design with prominent blue diagonal lines (see FIG. 11(a)), it is expected that the blue score in the window that actually contains the deregulation sign will be high. Therefore, by focusing on the level of the blue score of the window, it is possible to exclude, for example, effective straight lines that are clearly not blue, such as white straight lines and black straight lines, from candidates for deregulation signs, and to extract only effective straight lines in blue that may be candidates for deregulation signs.

[0051] However, when an unrestricted sign is imaged in the morning or evening sun, or when an unrestricted sign is imaged on a cloudy or rainy day, the color of the unrestricted sign in the image may be weakened, but in order not to exclude such unrestricted signs from the candidates, the threshold for comparison with the blue score in this step is set low. As a result, in this step, not only actual unrestricted signs but also effective straight lines that are not unrestricted signs (e.g., poles, utility poles, tree branches, etc.) may be extracted as candidates for unrestricted signs, and the reliability of the candidates for unrestricted signs at the time of this step is not sufficient.

[0052] In step S48, a candidate for a non-restricted sign is identified. This is performed in the same manner as the identification of the main sign in S42. Specifically, the calculation processing unit 23 identifies the candidate for a non-restricted sign using the classifier 24b. At this time, a second identification score representing the likelihood of the sign being a non-restricted sign is output in this identification process and stored in memory. The candidate for a non-restricted sign identified in this step is hereinafter referred to as a "non-restricted sign." Note that step S48 may be processed before step S47, or may be processed in parallel with step S47.

[0053] In step S49, the calculation processing unit 23 tracks the main sign identified in step S42 and the candidate deregulation sign extracted in step S48. Here, "tracking" has the same meaning as described in step S46. Specifically, the type of the main sign candidate identified in step S43 (e.g., speed limit sign) and the deregulation sign (candidate) extracted in step S48 are registered in a tracking list L1, which is a list for registering signs being imaged by the stereo camera device 100. Note that if the main sign registered in the tracking list L1 cannot be imaged because the vehicle passes the sign, the information of the main sign registered in the tracking list L1 is transferred to the recognized list L2.

[0054] In step S4a, the calculation processing unit 23 searches for a main sign (e.g., a speed limit sign, no overtaking) that pairs with the candidate for the deregulation sign extracted in step S48. The method of searching for a paired sign in this step differs from country to country. For example, in Japan, as shown in FIG. 8A, the main sign is placed below the deregulation sign, so the deregulation sign and the main sign registered at the same time in the tracking list L1 can be associated as paired signs. On the other hand, in other countries, as shown in FIG. 8B, the deregulation sign may be installed before the main sign, so the main sign registered in the recognized list L2 and the deregulation sign newly registered in the tracking list L1 can be associated as paired signs. By using any of the search methods, it is possible to search for a main sign that pairs with the deregulation sign regardless of the country.

[0055] Here, the paired marker search process in step S4a will be described in detail with reference to the flowchart in Fig. 9. This flowchart is divided into processes from step S4aa to step S4ad and processes from step S4af to step S4ah.

[0056] In steps S4aa to S4ad, the calculation processing unit 23 searches for paired signs based on the tracking list L1. This is a method for searching for paired signs in a country where a regular sign and a non-restricted sign are installed in the same place, as illustrated in FIG. 8A.

[0057] First, in step S4ab, the calculation processing unit 23 checks whether or not there is a main sign paired with the deregulation sign based on the tracking information (information for identifying the sign, such as the sign's setting position, size, type, tracking number, etc.) registered in the tracking list L1 in step S49. For example, as shown in Fig. 8A, if a deregulation sign and a speed limit sign are registered in the tracking list L1, these are extracted as candidates for paired signs.

[0058] Next, in step S4ac, the calculation processing unit 23 checks the arrangement of the deregulation sign and the paired sign candidate, and if the installation positions of the two are close, determines that the two are paired signs. For example, as shown in Fig. 10A, if a speed limit sign, which is a first paired sign candidate, is installed on the same pole as the deregulation sign, and a no overtaking sign, which is a second paired sign candidate, is installed on a pole far from the deregulation sign, the speed limit sign installed on the same pole as the deregulation sign is determined to be a paired sign, and the farther no overtaking sign is determined to be not a paired sign. Note that, if multiple main signs are installed on the same pole as the deregulation sign, the multiple main signs may be extracted as paired signs.

[0059] In step S4ad, the calculation processing unit 23 stores in the memory unit 24 the information related to the paired markers extracted in step 48c.

[0060] In step S4ae, the calculation processing unit 23 checks whether the pair marker has been registered. If the pair marker has been registered, the process of step S4a ends and the process proceeds to step S4b. On the other hand, if the pair marker has not been registered, the process proceeds to step S4af.

[0061] In steps S4af to S4ah, the calculation processing unit 23 searches for paired signs based on the tracking list L1 and the recognized list L2. This is a method for searching for paired signs in a country where the main sign and the non-restricted sign are installed at separate locations, as illustrated in FIG. 8B.

[0062] First, in step S4ag, the calculation processing unit 23 checks whether or not there is a main sign paired with the deregulation sign registered in the tracking list L1 based on the recognized information registered in the recognized list L2 (sign information previously recognized such as the setting position, size, type, and registration number of the sign). For example, as shown in FIG. 8B, if a speed limit sign imaged in the past is registered in the recognized list L2, it is extracted as a candidate for a paired sign. Since there is a possibility that a large amount of sign information previously recognized is registered in the recognized list L2, in this step, the main sign registered most recently is determined to be a candidate for a paired sign. For example, as shown in FIG. 10B, if a speed limit sign, which is the first paired sign candidate, is installed on a post in front of the deregulation sign, and a no overtaking sign, which is the second paired sign candidate, is installed on a post in front of the speed limit sign, the most recently registered speed limit sign is determined to be a paired sign, and the no overtaking sign registered before that is determined to be not a paired sign.

[0063] In FIG. 9, both the processing from step S4aa to step S4ad corresponding to the environment of FIG. 8A and the processing from step S4af to step S4ah corresponding to the environment of FIG. 8B are executed. However, if the country in which the vehicle is traveling can be identified using location information obtained from a GPS or the like, then it is also possible to select and execute only one of the processing from step S4aa to step S4ad or the processing from step S4af to step S4ah depending on the traffic laws of that country.

[0064] When the process of step S4a shown in Fig. 9 is completed, a main sign that pairs with the deregulation sign can be extracted regardless of the installation state of the deregulation sign (Figs. 8A and 8B). However, as described above, in step S45, there is a possibility that a valid straight line that is not actually a deregulation sign is extracted as a candidate, so in the subsequent process, the color information of the paired sign extracted in step S4a (i.e., the main sign that pairs with it) is also referenced to confirm whether the candidate extracted in step S48 is truly a deregulation sign. The color information of the main sign is acquired in step S48) and stored in memory.

[0065] In step S4b, the calculation processing unit 23 judges whether the time period is a time period in which the authenticity determination process (step S4c) of the candidate for the deregulation sign can be performed. As described later, in step S4c, the color information of the paired sign is used, and the color information of the paired sign is highly reliable if it is based on an image captured during the day, and is less reliable if it is based on an image captured during the night. Therefore, in this step, for example, based on the time information acquired from the built-in timer of the ECU or the GPS, it is judged that the environment is one in which highly reliable color information can be obtained during the day, and the process proceeds to step S4c, and if it is night, it is judged that the environment is one in which highly reliable color information is difficult to obtain, and the process proceeds to step S4d. Note that if the candidate for the deregulation sign is not extracted in step S43, the process of step S4c cannot be executed, and if the candidate for the deregulation sign is not extracted in step S48, the process of step S4c for determining the authenticity of the candidate is unnecessary in the first place, so in these cases, too, step S4c is avoided and the process proceeds to step S4c.

[0066] In step S4c, the calculation processing unit 23 judges the authenticity of the candidate for the deregulation sign extracted in step S48, and if the candidate is a fake deregulation sign (for example, if it is a support), the candidate is rejected. As described above, in step S48, the candidate for the deregulation sign is extracted based on the blue score of the valid straight line. However, in order not to exclude a deregulation sign illuminated with colored light or a deregulation sign in cloudy or rainy weather from the candidates, the threshold value for comparison with the blue score of the valid straight line is set low, and an effective straight line that is not actually a deregulation sign is allowed to be extracted as a candidate. Therefore, in this step, the threshold value used to judge the authenticity of the candidate for the deregulation sign is adjusted by referring to the color score of the real sign obtained in step S42, so that the authenticity of the candidate for the deregulation sign can be accurately judged regardless of the external environment.

[0067] For example, as shown in Figure 11, if the main sign is a speed limit sign with a red ring around the outside of the sign, the red score within the window shape that the red ring fits into is expected to be a high value. However, if the main sign is illuminated by colored light, if it is cloudy or rainy, if it is sunny, etc., the red score of the main sign (speed limit sign) is expected to change. In such cases, the blue score of the non-restriction sign is also expected to change to the same extent.

[0068] Therefore, in this step, if the red score of the main sign (speed limit sign) is low, the threshold of the blue score for determining the authenticity of the candidate for the deregulation sign is also set low, and conversely, if the red score is high, the threshold of the blue score is also set high. That is, the threshold of the color information for recognizing the main sign as a deregulation sign is determined using color information (e.g., the blue score (i.e., the confidence that it is blue)) acquired when recognizing the main sign. In addition, the threshold of the color information for recognizing the main sign as a deregulation sign may be determined based on the rate of change of the color information acquired when recognizing the main sign relative to the color information of the template image of the main sign, rather than the color information acquired when recognizing the main sign itself. The color information of the main sign and the rate of change of the color information of the main sign are collectively referred to as the "confidence" of the main sign.

[0069] By using the threshold value of color information (eg, blue score) adjusted in this manner, the authenticity of a candidate deregulation sign can be accurately determined regardless of the environment.

[0070] Finally, in step S4d, the calculation processing unit 23 confirms the result of identifying the sign by referring to the above-mentioned tracking list L1 or the recognized list L2. That is, if the process proceeds from step S45 to step S46, only the main sign candidate identified in step S43 is confirmed as the identification result as the sign. On the other hand, if the process proceeds from step S45 to step S47, for example, the main sign candidate identified in step S43 is identified as a paired sign of the deregulation sign candidate extracted in steps S48 and S4c. Then, if the paired sign is recognized in this step, a control policy is determined in the following step S5 to cancel the regulation indicated by the main sign. Here, "confirming" means that information obtained from the recognized sign is output to the subsequent control process (S5 in FIG. 2) as the recognition process result.

[0071] According to the stereo camera device 100 of this embodiment described above, not only can the deregulation sign shown in FIG. 11(a) be correctly recognized, but also it can suppress the misdetection of the upper end of the pole shown in FIG. 11(b), which has a similar angle and thickness to the diagonal line portion of the deregulation sign, as a deregulation sign, thereby improving the reliability of sign detection.

[0072] In the above embodiment, the stereo camera device 100 is configured with two cameras, but one camera or three or more cameras may be used. The above configurations, functions, processing units, processing means, etc. may be realized in hardware, partly or entirely, by designing them as integrated circuits, for example. The above configurations, functions, etc. may be realized in software by a processor interpreting and executing a program that realizes each function. Information such as programs, tables, files, etc. that realize each function can be stored in a memory, a storage device such as a hard disk, or a solid state drive (SSD), or a recording medium such as an IC card, an SD card, or a DVD. [Explanation of symbols]

[0073] 100... stereo camera device, 1... camera, 1L... left camera, 1R... right camera, 2... sign recognition device, 21... image input interface, 22... image processing unit, 23... calculation processing unit, 24... memory unit, 24a... image buffer, 24b... classifier, 25... CAN interface, 26... monitoring processing unit, 27... internal bus, CAN in-vehicle network

Claims

1. a sign candidate recognition unit that recognizes candidates of a predetermined type of sign from an image; a paired marker search unit that, when a candidate for a first marker is recognized as a candidate for the predetermined type of marker, associates the first marker with a second marker that is installed corresponding to the first marker; a color information extraction unit that extracts color information of a second sign associated with a first sign when a second sign is installed corresponding to the first sign; a threshold determination unit that sets a threshold for identifying the second mark candidate based on date and time and environment, and sets a threshold for determining authenticity of the second mark candidate based on color information acquired when recognizing the first mark; A sign recognition device comprising:

2. 2. The sign recognition device according to claim 1, Further, a sign candidate storage unit is provided for storing the recognition results of the sign candidate recognition unit in chronological order, The paired marker search unit detects whether the second marker is installed corresponding to the first marker. If the second marker is not associated with the first marker, A second sign to be associated with the first sign is extracted from the recognition results stored in the sign candidate storage unit. A sign recognition device characterized by outputting a signal.

3. 2. The sign recognition device according to claim 1, A sign recognition device characterized in that, when information on a plurality of first signs is registered in the tracking list, the first sign closest to the second sign is associated as a paired sign.

4. A step of recognizing candidates for a predetermined type of sign from an image captured by an in-vehicle camera; When a candidate for a first sign is recognized as a candidate for the predetermined type of sign, associating the first sign with a second sign installed corresponding to the first sign; When a second sign is installed corresponding to the first sign, extracting color information of the second sign associated with the first sign; A step of setting a threshold for identifying the second mark candidate based on date and time and environment, and setting a threshold for determining authenticity of the second mark candidate based on color information acquired when recognizing the first mark; The method for sign recognition is characterized in that the method is executed by a computer unit.

5. The sign recognition method according to claim 4, Further, a step of storing the recognition results in a time series is provided, A sign recognition method comprising: determining whether or not a second sign exists that is installed corresponding to the first sign; and, if the second sign is not associated with the first sign, extracting a second sign to be associated with the first sign from stored recognition results.

6. The sign recognition method according to claim 4, A sign recognition method characterized in that, when information on a plurality of first signs is registered in the tracking list, the first sign closest to the second sign is associated as a paired sign.

Images