Image recognition device

Abstract

To detect an installation abnormality of an on-vehicle camera even when a white line cannot be detected during vehicle travel.SOLUTION: An image recognition device 10 comprises: an image acquisition unit 11 that acquires an image of a scene ahead of a vehicle imaged by an on-vehicle camera 5; a first vanishing point calculation unit 13 that detects a white line from the image acquired by the image acquisition unit 11 and calculates a reference vanishing point in the image; a second vanishing point calculation unit 14 that calculates a vanishing point on the basis of movement of feature points within a predetermined region in the image when the first vanishing point calculation unit 13 cannot detect a white line; and an abnormality determination unit 15 that determines an installation abnormality of the on-vehicle camera 5 on the basis of calculation results of the first vanishing point calculation unit 13 and the second vanishing point calculation unit 14. The abnormality determination unit 15 determines that the on-vehicle camera 5 has an installation abnormality when a coordinate difference on the image between the reference vanishing point calculated by the first vanishing point calculation unit 13 and the vanishing point calculated by the second vanishing point calculation unit 14 exceeds a predetermined value.SELECTED DRAWING: Figure 1

Description

[Technical Field]

[0001] The present invention relates to an image recognition device. [Background technology]

[0002] There are image recognition devices that use forward images captured by an onboard camera to generate information necessary for controlling the distance between the vehicle and the vehicle ahead, recognizing obstacles, etc. Focus of Expansion (FOE) coordinates on the image are used to convert the distance of an object on the image to its actual distance. FOE coordinates are generally calculated based on the recognition of white lines on the road. [Prior art documents] [Patent documents]

[0003] [Patent Document 1] Japanese Patent Application Laid-Open No. 2011-227551 Summary of the Invention [Problem to be solved by the invention]

[0004] If the FOE coordinates are not in the correct position, an error may occur in the distance calculated on the image. FOE coordinate calculations are based on images captured by the in-vehicle camera at the initial mounting angle, so the in-vehicle camera must always be mounted at the correct angle on the vehicle. However, an external impact or other cause may cause the mounting angle of the in-vehicle camera to deviate from the initial mounting angle, resulting in an abnormality in the mounting angle. If such an abnormality occurs in the installation of the in-vehicle camera, the correct position of the FOE coordinates may not be calculated, and an error may occur in the distance calculated on the image.

[0005] For the reasons mentioned above, it is necessary to periodically calculate the FOE coordinates and check whether the FOE coordinates are in the correct position. However, there may be times when it is difficult to recognize the white lines on the road. For example, when it snows, the white lines on the road are covered with snow, making it difficult to recognize the white lines.

[0006] The present invention has been made in consideration of the above-mentioned circumstances, and its purpose is to provide an image recognition device that can detect abnormal installation of an on-board camera even when white lines cannot be detected while the vehicle is traveling. [Means for solving the problem]

[0007] In order to achieve the above-mentioned object, the image recognition device according to the present invention has the following features. an image acquisition unit that acquires an image of the area in front of the vehicle captured by an in-vehicle camera; a first vanishing point calculation unit that detects white lines from the image acquired by the image acquisition unit and calculates a vanishing point in the image; a second vanishing point calculation unit that calculates a vanishing point based on the movement of feature points within a predetermined area in the image when the first vanishing point calculation unit cannot detect a white line; an abnormality determination unit that determines an installation abnormality of the vehicle-mounted camera based on the calculation results of the first vanishing point calculation unit and the second vanishing point calculation unit; The abnormality determination unit determines that there is an installation abnormality in the vehicle-mounted camera when a coordinate difference on the image between the reference vanishing point calculated by the first vanishing point calculation unit and the vanishing point calculated by the second vanishing point calculation unit exceeds a predetermined value. Image recognition device. [Effects of the Invention]

[0008] According to the present invention, even if a white line cannot be detected while a vehicle is traveling, an installation abnormality of the on-board camera can be detected using the optical flow method, which calculates a vanishing point from the movement of feature points within a predetermined area in the image. This prevents the image recognition function from being unnecessarily stopped solely because a white line cannot be detected, or from continuing to use the image recognition function despite an installation abnormality.

[0009] The present invention has been briefly described above. The details of the present invention will become clearer by reading the following detailed description of the invention (hereinafter referred to as "embodiments") with reference to the accompanying drawings. [Brief explanation of the drawings]

[0010] [Figure 1] FIG. 1 is a block diagram of an image recognition system and an image recognition device according to an embodiment of the present invention. [Figure 2] FIG. 2 is a flowchart showing a procedure executed by the first vanishing point calculation unit to determine whether or not there is an installation abnormality in the vehicle-mounted camera. [Figure 3] FIG. 3 is a flowchart showing the procedure executed by the second vanishing point calculation unit and the abnormality determination unit to determine whether or not there is an installation abnormality in the vehicle-mounted camera. [Figure 4] FIG. 4 is a flowchart showing the procedure for setting a detection frame by the second vanishing point calculation unit. [Figure 5] Figure 5 is a conceptual diagram showing the state in which the second vanishing point calculation unit 14 has set a detection frame on an image frame in a video, where (A) shows a diagram of the image and detection frame at time t, and (B) shows a diagram of the image and detection frame at time t+Δt. [Figure 6] FIG. 6 is a flowchart showing the procedure of image recognition executed by the image recognition device. DETAILED DESCRIPTION OF THE INVENTION

[0011] Specific embodiments of the present invention will be described below with reference to the accompanying drawings.

[0012] 1 is a block diagram of an image recognition system 1 and an image recognition device 10 according to one embodiment of the present invention. The image recognition system 1 is mounted on a vehicle and analyzes images of the vehicle's surroundings. The image recognition system 1 includes an on-board camera 5, the image recognition device 10, a notification unit 21, an output unit 23, and an installation anomaly correction unit 25.

[0013] The vehicle-mounted camera 5 captures images of the vehicle's surroundings, including the area ahead of the vehicle, and generates image data. The vehicle-mounted camera 5 is attached to the front end, rear end, rearview mirror, etc. of the vehicle. The vehicle-mounted camera 5 is attached to the vehicle body at a predetermined mounting angle (posture), and it is assumed that the vehicle will capture images of the surroundings in this mounting state.

[0014] The mounting angle of the vehicle-mounted camera 5 may deviate from the initial mounting angle due to an external impact or the like, which may result in an installation abnormality. The image recognition device 10 is a device for determining whether or not there is an installation abnormality in such a vehicle-mounted camera 5. The image recognition device 10 includes an image acquisition unit 11 that acquires an image of the front of the vehicle captured by the vehicle-mounted camera 5, and a calculation unit 12. The calculation unit 12 includes a first vanishing point calculation unit 13, a second vanishing point calculation unit 14, and an abnormality determination unit 15.

[0015] The first vanishing point calculation unit 13 detects white lines on the road from the image acquired by the image acquisition unit 11, and calculates the first vanishing point in the image. If the first vanishing point calculation unit 13 cannot detect white lines, the second vanishing point calculation unit 14 calculates a vanishing point based on the movement of feature points within a predetermined area in the image.

[0016] The abnormality determination unit 15 determines an installation abnormality of the vehicle-mounted camera 5 based on the calculation results of the first vanishing point calculation unit 13 and the second vanishing point calculation unit 14. Furthermore, the abnormality determination unit 15 determines that an installation abnormality of the vehicle-mounted camera 5 exists when the coordinate difference on the image between the reference vanishing point calculated by the first vanishing point calculation unit 13 and the vanishing point calculated by the second vanishing point calculation unit 14 exceeds a predetermined value.

[0017] The notification unit 21 notifies a passenger such as a driver of the determination result of the calculation unit 12. The output unit 23 outputs the determination result of the calculation unit 12 to an external device. The installation abnormality correction unit 25 receives the determination result of the calculation unit 12 from the output unit 23, and corrects the installation position of the in-vehicle camera 5 if an installation abnormality of the in-vehicle camera 5 has occurred.

[0018] Next, the function of the first vanishing point calculation unit 13 will be described. The first vanishing point calculation unit 13 detects white lines on the road from the image acquired by the image acquisition unit 11 and calculates the vanishing point in the image. The vanishing point is also called FOE coordinates or point at infinity, and can be identified, for example, as the same position as the point where the extensions of a pair of white lines located on both sides of the host vehicle's lane on the road surface on which the host vehicle is to travel intersect. White lines are a general term for lines that extend on the road and define the host vehicle's travel lane, and the form of the lines may be solid, dashed, or the like, and also includes yellow lines, etc.

[0019] 2 is a flowchart showing the procedure executed by the first vanishing point calculation unit 13 to determine whether there is an installation abnormality in the vehicle-mounted camera 5. The first vanishing point calculation unit 13 acquires image data of the image from the image acquisition unit 11 (step S1), and determines the driving lane of the vehicle from the image data (step S2). The first vanishing point calculation unit 13 detects the left lane and the right lane of the driving lane (step S3). The detection method includes processes such as smoothing processing of the image data, screen division processing, edge detection, and binarization.

[0020] The first vanishing point calculation unit 13 calculates the FOE coordinates, that is, the vanishing point, from the detected intersection of the left lane and the right lane (step S4). The calculated vanishing point is stored in, for example, a storage unit (not shown).

[0021] The first vanishing point calculation unit 13 calculates a vanishing point at predetermined intervals, continues to monitor the deviation between a vanishing point that has previously been calculated and stored in the storage unit and the newly calculated vanishing point (step S5), and determines whether or not a deviation of the vanishing point of a predetermined number of pixels (e.g., 80 pixels) or more has occurred within the image data (step S6). If a deviation of the predetermined number of pixels (e.g., 80 pixels) or more has not occurred (step S6; No), it is assumed that no abnormality has occurred in the installation of the camera, and so the first vanishing point calculation unit 13 continues to monitor the deviation.

[0022] If a deviation of a predetermined number of pixels (e.g., 80 pixels) or more occurs (Step S6; Yes), it is estimated that an installation abnormality has occurred in the camera, and the first vanishing point calculation unit 13 determines that an installation abnormality has occurred due to a change in the mounting angle of the vehicle-mounted camera 5, etc. (Step S7). Such an installation abnormality can occur, for example, when an external impact is applied to the vehicle body or the vehicle-mounted camera 5. The first vanishing point calculation unit 13 transmits the installation abnormality of the vehicle-mounted camera 5 to the notification unit 21, and the notification unit 21 notifies the occupant of the installation abnormality (Step S8), and the image recognition device 10 stops image recognition (Step S9).

[0023] As described above, the first vanishing point calculation unit 13 can automatically detect installation abnormalities in the vehicle-mounted camera 5. In particular, if the vehicle comes into contact with an object while parking, an installation abnormality in the vehicle-mounted camera 5 may be difficult to notice and may be left unnoticed, but the function of the first vanishing point calculation unit 13 can prevent such a situation.

[0024] However, situations may arise in which it is difficult for the vehicle-mounted camera 5 to capture an image of the white lines on the road. For example, when it snows, the white lines on the road are covered with snow, so the vehicle-mounted camera 5 cannot capture an image of the white lines, making it difficult to recognize the white lines. In such cases, the first vanishing point calculation unit 13 cannot execute the calculation of the first vanishing point described above, and therefore cannot detect an installation abnormality in the vehicle-mounted camera 5.

[0025] In preparation for the above-mentioned situation, the second vanishing point calculation unit 14 calculates the vanishing point using a so-called optical flow method, which is different from the white line recognition method used by the first vanishing point calculation unit 13. Then, the abnormality determination unit 15 determines an installation abnormality of the in-vehicle camera 5 based on the previous calculation results of the first vanishing point calculation unit 13 and the calculation results of the second vanishing point calculation unit 14. This makes it possible for the image recognition device 10 to prevent situations in which the image recognition function is unnecessarily stopped or the image recognition function is continued despite the occurrence of an installation abnormality, based solely on the inability to detect white lines.

[0026] The optical flow method is a technique for calculating a vanishing point by capturing the movement of a specific point in an image. The forward image captured by the vehicle-mounted camera 5 changes from moment to moment, but a velocity vector (optical flow) can be obtained by drawing a straight line between the same corresponding points (feature points) in the image at time t and time t+Δt. All optical flows appear radially from a single point in the image, and this single point is the FOE coordinate, or vanishing point.

[0027] 3 is a flowchart showing the procedure for calculating the second vanishing point executed by the second vanishing point calculation unit 14 and the abnormality determination unit 15. The second vanishing point calculation unit 14 acquires video data (including a predetermined number of image frames) spanning a predetermined period from the image acquisition unit 11 (step S11). Furthermore, the second vanishing point calculation unit 14 acquires the first vanishing point that was calculated by the first vanishing point calculation unit 13 in the procedure of FIG. 2 and stored in the storage unit at a timing before the white line was recognized (step S12).

[0028] Next, the second vanishing point calculation unit 14 sets a detection frame, which is a predetermined area that is arranged on the image frame and detects the optical flow, and determines whether or not the optical flow can be calculated (step S13). The details of step S13 will be described later (see Figures 4 and 5). If the optical flow cannot be calculated (step S13; No), the second vanishing point cannot be calculated either, and it is presumed that an abnormal installation of the vehicle-mounted camera 5 has occurred, so the process proceeds to step S18 (determination of installation abnormality), which will be described later.

[0029] If the optical flow can be calculated (step S13; Yes), the second vanishing point calculation unit 14 calculates an estimated second vanishing point based on the first vanishing point acquired in step S12 and an extension of the optical flow detected from two temporally adjacent image frames (for example, the image frame at time t and the image frame at time t+Δt) (step S14). The details of step S14 will be described later (see FIGS. 4 and 5).

[0030] The second vanishing point calculation unit 14 calculates a predetermined number of estimated second vanishing points (for example, estimated second vanishing points obtained from 10 video frames), and calculates a final fixed second vanishing point from the average of the calculated multiple fixed second vanishing points (step S15).

[0031] Next, the abnormality determination unit 15 compares the first vanishing point acquired in step S12 with the calculated confirmed second vanishing point (step S16), and determines whether or not there is a deviation of a predetermined number of pixels or more, i.e., a coordinate difference on the image (step S17). If there is no deviation of a predetermined number of pixels or more (step S17; No), it is presumed that no abnormality in camera installation has occurred, so the process returns to step S13, and the second vanishing point calculation unit 14 repeats the subsequent processes.

[0032] If there is a deviation of a predetermined number of pixels or more (step S17; Yes), it is estimated that an installation abnormality in the camera has occurred, and therefore the abnormality determination unit 15 determines that an installation abnormality in the vehicle-mounted camera 5 has occurred (step S18). The abnormality determination unit 15 transmits the installation abnormality in the vehicle-mounted camera 5 to the notification unit 21, and the notification unit 21 notifies the occupant of the installation abnormality (step S19), and the image recognition device 10 stops image recognition (step S20).

[0033] Fig. 4 is a flowchart showing the procedure for setting a detection frame on an image frame by the second vanishing point calculation unit 14 in step 13 of Fig. 3. The second vanishing point calculation unit 14 sets a detection frame, which is a predetermined area on the image frame for detecting optical flow. The detection frame is rectangular, for example, as shown by detection frames FL and FR in Fig. 5.

[0034] The second vanishing point calculation unit 14 compares the first vanishing point acquired in step S12 with a reference point that serves as a setting reference for the detection frame and that is stored in a pre-created detection frame database (DB) (step S31). The reference point is, for example, the coordinates of the upper left corner of the detection frame to be set. Based on the comparison, the second vanishing point calculation unit 14 selects and determines, from among the multiple reference points, the reference point for the detection frame to be set on the image frame (step S32). The second vanishing point calculation unit 14 can, for example, select the reference point that is closest to the first vanishing point.

[0035] Next, the second vanishing point calculation unit 14 sets the size of the detection frame to be set and sets the detection frame on the image (step S33). The second vanishing point calculation unit 14 sets the size of the detection frame based on at least the vehicle's traveling speed and the time interval between capturing images. The time interval Δt is the interval between capturing images (image frames) in the video data acquired in step S11 of FIG. 3.

[0036] Next, the second vanishing point calculation unit 14 determines whether or not the optical flow can be calculated within the detection frame (step S34). If the optical flow can be calculated (step S34; Yes), the second vanishing point calculation unit 14 performs the processes from step S14 onwards in FIG.

[0037] If optical flow cannot be detected even after a predetermined time (e.g., 10 seconds) has elapsed (step S34; No), the optical method cannot be executed for that detection frame, and so second vanishing point calculation unit 14 determines whether setting and optical flow calculation have been performed for all detection frames in the detection frame database (step S35). If setting has not been completed for all detection frames (step S35; No), second vanishing point calculation unit 14 sets a reference point for a new detection frame that has not been set from the detection frame database, sets the detection frame on the image frame, and moves the detection frame (step S36). Thereafter, second vanishing point calculation unit 14 performs the processing of step S33 again.

[0038] If the setting is completed for all detection frames and the optical flow cannot be calculated (step S35; Yes), it is determined that the optical method cannot be executed even if all detection frames are used, and therefore it is estimated that an installation abnormality has occurred in the vehicle-mounted camera 5. Therefore, the abnormality determination unit 15 performs the processes from step S18 onwards in FIG.

[0039] 5A and 5B are conceptual diagrams showing a state in which the second vanishing point calculation unit 14 has set a detection frame on an image frame in a video, where (A) shows the image and detection frame at time t, and (B) shows the image and detection frame at time t+Δt. The second vanishing point calculation unit 14 sets the detection frame and calculates the optical flow according to the procedure in FIG.

[0040] As shown in Fig. 5(A), the second vanishing point calculation unit 14 sets a pair of detection frames FL and FR on the left and right (in the x-axis direction) of the vehicle's driving lane on the image frame of the first image captured at time t, which is the first time, as described in steps S31 to S33 of Fig. 4. The first vanishing point P1 calculated by the first vanishing point calculation unit 13 and stored in the storage unit using the procedure in Fig. 2 is also set. The first vanishing point P1 is the position of the vanishing point that is expected to be finally set, and the coordinates of the first vanishing point P1 in the image frame are (x, y).

[0041] Next, the second vanishing point calculation unit 14 determines feature points within each of the detection frames FL and FR on the image frame at time t. A feature point is, for example, a location within a very small area of ​​5x5 pixels where there is a change of a predetermined magnitude or more in the distribution of brightness, color, etc. (a location where there is a particularly sudden change is called an edge). In this example, object 1 is a feature point within the detection frame FL, and object 2 is a feature point within the detection frame FR. At time t, the coordinates of object 1 within the image frame are (x1, y1), and the coordinates of object 2 within the image frame are (x2, y2).

[0042] 5(B), the second vanishing point calculation unit 14 also sets a pair of detection frames FL and FR on the image frame captured at time t+Δt, which is a second time that is a time interval Δt after the first time t. The size and position of the detection frames FL and FR within the image frame are the same at time t and time t+Δt.

[0043] As the vehicle moves over time interval Δt, the positions of objects 1 and 2 move in the image frame. At time t+Δt, the coordinates of object 1 in the image frame are (x1+Δx1, y1+Δy1), and the coordinates of object 2 in the image frame are (x2+Δx2, y2+Δy2). Therefore, second vanishing point calculation unit 14 can calculate optical flow OP1 (Δx1, Δy1) corresponding to the movement of object 1 in detection frame FL, and optical flow OP2 (Δx2, Δy2) corresponding to the movement of object 2 in detection frame FR.

[0044] Furthermore, as described in step S14 of Fig. 3, the second vanishing point calculation unit 14 creates extension lines by extending each of the optical flows OP1 and OP2 toward the first vanishing point P1, and calculates the point where the two extension lines intersect as the estimated second vanishing point P2. Furthermore, as described in step S15 of Fig. 3, the second vanishing point calculation unit 14 calculates a predetermined number of estimated second vanishing points according to the method described above, and calculates a final, confirmed second vanishing point from the average of the multiple calculated estimated second vanishing points. The coordinates of the confirmed second vanishing point are (x', y').

[0045] The calculation of the second vanishing point assumes a situation in which it is impossible to detect a white line by the first vanishing point calculation unit 13. Therefore, it is assumed that the feature point is set, for example, not on the road surface but outside the roadside at a position in a predetermined height direction (y-axis direction) (such as a structure of a predetermined height). Based on this assumption, the positions of the pair of detection frames FL and FR are also set.

[0046] According to the image recognition device 10 of this embodiment, even if a white line cannot be detected while the vehicle is traveling, an installation abnormality of the on-board camera 5 can be detected using the optical flow method, which calculates a vanishing point from the movement of feature points within the detection frames FL and FR, which are predetermined areas in the image. This prevents the image recognition function from being unnecessarily stopped solely because a white line cannot be detected, or from continuing to function despite an installation abnormality.

[0047] Furthermore, the second vanishing point calculation unit 14 sets detection frames FL and FR, which are a pair of predetermined regions on the left and right of the vehicle's driving lane, and calculates the movements of objects 1 and 2, which are feature points in each detection frame FL and FR, i.e., the intersection of the extension lines of optical flows OP1 and OP2, as the second vanishing point. This ensures the necessary accuracy even when calculating vanishing points using the optical flow method.

[0048] Furthermore, second vanishing point calculation unit 14 calculates estimated second vanishing points based on the movements of feature points within detection frames FL and FR in multiple images, and sets the average of the multiple estimated second vanishing points as the confirmed second vanishing point, and abnormality determination unit 15 can determine an installation abnormality of in-vehicle camera 5 based on the coordinate difference on the image between the first vanishing point and the confirmed second vanishing point. This makes it possible to further improve the accuracy of vanishing point calculation when calculating vanishing points using the optical flow method.

[0049] As described above, the second vanishing point calculation unit 14 can set the size of the detection frames FL and FR based on at least the vehicle's traveling speed and the time interval between capturing images. For example, under the following conditions, the second vanishing point calculation unit 14 can set the width of the detection frame to approximately 1 / 10 of the width of the image frame, and the height of the detection frame to approximately 1 / 8.5 of the height of the image frame. Vehicle speed: 30~100km / h Time interval Δt: 600 ms Image frame resolution: 1920 x 1280 - Horizontal angle of view of vehicle camera: approx. 120 degrees

[0050] A large detection frame allows for a high probability of capturing feature points that move from the first time t to the second time t+Δt, thereby increasing the likelihood of calculating the optical flow in step S34 of Fig. 4. However, in order to detect feature points determined in the first image in the second image at the second time t+Δt, it is necessary to perform a process for the entire detection frame, in which the detection frame is divided into infinitesimal regions and feature points are found in each infinitesimal region. This process may increase the calculation load and processing time of the image recognition device 10.

[0051] Image recognition device 10 according to this embodiment sets detection frames FL, FR of appropriate sizes based on at least the vehicle traveling speed and the image capture time interval Δt. Therefore, even when, for example, in order to detect feature points within detection frames FL, FR, the detection frames FL, FR are divided into minimal regions and feature points are searched for in each minimal region, and this processing is performed on the entire detection frames FL, FR, it is possible to prevent excessive increases in the calculation load and processing time.

[0052] When setting a pair of detection frames FL and FR, the second vanishing point calculation unit 14 determines the positions and then the sizes, and the detection frames FL and FR can be placed at positions that are equally spaced apart on the left and right (x-axis direction) from the first vanishing point P1, as shown in Fig. 5. In other words, the second vanishing point calculation unit 14 can set a pair of detection frames FL and FR that are symmetrical on the left and right with respect to the position of an assumed vanishing point, such as the first vanishing point P1.

[0053] This increases the accuracy of the vanishing point position because the vanishing point is calculated from the average of the changes in the two feature points of the detection frames FL and FR. Furthermore, if the feature points of both detection frames FL and FR move in the same direction, it is highly likely that the vehicle is turning, so the calculation of the vanishing point can be stopped for a while, preventing a decrease in the accuracy of the vanishing point position.

[0054] If no feature points are detected within the set detection frames FL and FR, second vanishing point calculation unit 14 moves the detection frames FL and FR as in step S36 of Fig. 4, but can also move the detection frames a predetermined amount in the vertical direction (y-axis direction) and perform detection again. The predetermined amount is, for example, 100 pixels. In this case, the detection frame database holds reference points spaced apart at intervals of 100 pixels in the vertical direction.

[0055] This makes it possible to calculate the vanishing point even if the pitch angle of the vehicle-mounted camera 5 changes due to an external impact or the like.

[0056] 6 is a flowchart showing the steps of image recognition executed by the image recognition device 10. The image recognition device 10 determines whether or not white line detection is possible by the first vanishing point calculation unit 13 (step S41). If white line detection is possible (step S41; Yes), the first vanishing point calculation unit 13 calculates the first vanishing point based on the white line detection according to the procedure in FIG. 2 (step S42), and determines whether or not there is an installation abnormality in the vehicle-mounted camera 5 (step S43). If there is an installation abnormality (step S43; Yes), the notification unit 21 notifies the occupant of the installation abnormality (step S44), and the image recognition device 10 stops image recognition (step S45). If there is no installation abnormality (step S43; No), the process returns to step S41 and repeats.

[0057] If it is not possible to detect the white line in step S41 (step S41; No), the second vanishing point calculation unit 14 sets a detection frame (step S46) and determines whether or not an optical flow is detected (step S47). If an optical flow is not detected (step S47; No), the notification unit 21 notifies the occupant of an installation abnormality (step S44), and the image recognition device 10 stops image recognition (step S45).

[0058] If an optical flow is detected (step S47; Yes), the second vanishing point calculation unit 14 calculates the second vanishing point (step S48), and the abnormality determination unit 15, following the procedure of Fig. 3, compares the first vanishing point previously stored in the storage unit with the second vanishing point calculated in step S48 to determine whether or not there is an installation abnormality in the vehicle-mounted camera 5 (step S43). If there is an installation abnormality (step S43; Yes), the notification unit 21 notifies the occupant of the installation abnormality (step S44), and the image recognition device 10 stops image recognition (step S45). If there is no installation abnormality (step S43; No), the process returns to step S41 and repeats the process.

[0059] When an installation abnormality of the in-vehicle camera 5 is detected, the notification unit 21 notifies the occupant of the installation abnormality, so the occupant may themselves correct the mounting angle of the in-vehicle camera 5. However, the installation abnormality correction unit 25 may receive the judgment result of the calculation unit 12 from the output unit 23, learn the occurrence pattern of installation abnormalities of the in-vehicle camera 5 using AI (Artificial Intelligence), learn the necessary corrections, and present the correction method.

[0060] In addition, the image recognition device 10 of this embodiment can determine installation abnormalities of the vehicle-mounted camera 5 using only a single vehicle-mounted camera 5, so it is possible to reduce costs, installation space, and weight compared to using multiple vehicle-mounted cameras.

[0061] Here, the features of the embodiment of the image recognition device according to the present invention described above will be briefly summarized and listed below in [1] to [3].

[0062] [1] An image acquisition unit (image acquisition unit 11) that acquires an image of the area in front of the vehicle captured by an on-board camera (on-board camera 5); a first vanishing point calculation unit (first vanishing point calculation unit 13) that detects white lines from the image acquired by the image acquisition unit and calculates a first vanishing point in the image; a second vanishing point calculation unit (second vanishing point calculation unit 14) that calculates a second vanishing point based on the movement of feature points within a predetermined area in the image when the first vanishing point calculation unit cannot detect a white line; An abnormality determination unit (abnormality determination unit 15) that determines an installation abnormality of the vehicle-mounted camera based on the calculation results of the first vanishing point calculation unit and the second vanishing point calculation unit, The abnormality determination unit determines that there is an installation abnormality in the vehicle-mounted camera when a coordinate difference on the image between the first vanishing point calculated by the first vanishing point calculation unit and the second vanishing point calculated by the second vanishing point calculation unit exceeds a predetermined value. Image recognition device (image recognition device 10).

[0063] According to the image recognition device configured as described in [1] above, even if a white line cannot be detected while the vehicle is moving, an installation abnormality of the on-board camera can be detected using the optical flow method, which calculates a vanishing point from the movement of feature points within a predetermined area in the image. This prevents the image recognition function from being stopped unnecessarily based solely on the inability to detect a white line, or from continuing to use the image recognition function despite an installation abnormality.

[0064] [2] The second vanishing point calculation unit sets a pair of the predetermined areas on the left and right of the vehicle's driving lane, and calculates the intersection of extension lines of the movements of the feature points in each of the predetermined areas as the second vanishing point. The image recognition device according to [1] above.

[0065] According to the image recognition device having the configuration [2] above, it is possible to ensure the necessary accuracy even in calculating the vanishing point using the optical flow method.

[0066] [3] The second vanishing point calculation unit calculates estimated second vanishing points based on the movements of feature points within the predetermined area in each of a plurality of images, and determines an average of the estimated second vanishing points as a confirmed second vanishing point; the abnormality determination unit determines an installation abnormality of the vehicle-mounted camera based on a coordinate difference on the image between the first vanishing point and the determined second vanishing point. The image recognition device according to [2] above.

[0067] According to the image recognition device having the configuration described above in [3], it is possible to further improve the accuracy of calculating the vanishing point when calculating the vanishing point using the optical flow method.

[0068] The present invention is not limited to the above-described embodiments, and can be appropriately modified, improved, etc. Furthermore, the material, shape, size, number, location, etc. of each component in the above-described embodiments are arbitrary and not limited as long as they can achieve the present invention. [Explanation of symbols]

[0069] 1. Image Recognition System 5. In-car camera 10 Image recognition device 11 Image acquisition unit 12 Arithmetic section 13 First vanishing point calculation section 14 Second vanishing point calculation section (vanishing point calculation section) 15 Abnormality determination section 21 Notification Department 23 Output section 25 Installation error correction section

Claims

1. an image acquisition unit that acquires an image of the area in front of the vehicle captured by an in-vehicle camera; a first vanishing point calculation unit that detects white lines from the image acquired by the image acquisition unit and calculates a reference vanishing point in the image; a second vanishing point calculation unit that calculates a vanishing point based on movement of feature points within a predetermined area in the image when the first vanishing point calculation unit cannot detect a white line; an abnormality determination unit that determines an installation abnormality of the vehicle-mounted camera based on the calculation results of the first vanishing point calculation unit and the second vanishing point calculation unit, The abnormality determination unit determines that there is an installation abnormality in the vehicle-mounted camera when a coordinate difference on the image between the reference vanishing point calculated by the first vanishing point calculation unit and the vanishing point calculated by the second vanishing point calculation unit exceeds a predetermined value. Image recognition device.

2. the second vanishing point calculation unit sets a pair of the predetermined areas on the left and right of a vehicle's driving lane, and calculates an intersection of extension lines of movements of the feature points in each of the predetermined areas as a vanishing point. The image recognition device according to claim 1 .

3. the second vanishing point calculation unit calculates a vanishing point based on the movement of feature points within the predetermined area in each of a plurality of images, and determines an average of the plurality of vanishing points as a confirmed vanishing point; the abnormality determination unit determines an installation abnormality of the vehicle-mounted camera based on a coordinate difference on the image between the reference vanishing point and the determined vanishing point. The image recognition device according to claim 2 .

Images