Monitoring device

Abstract

To provide a monitoring device that can ensure a measurement precision for a distance from a camera to an object by suppressing an error in camera parameter.SOLUTION: A monitoring device 1 includes: a camera 10; a distortion correcting unit 21 that generates a distortion correction image, and obtains a focal distance and an FOE coordinate both corresponding to the distortion correction image; and an object detecting unit 22 that detects a position of an object photographed in the distortion correction image as an image object position. The monitoring device 1 also includes: a distance measuring unit 23 that obtains an actual distance to the object using the image object position, the focal distance, and the FOE coordinate; a mark detecting unit 24 that detects, as a land mark, a specific element on a road photographed in the distortion correction image; and a mark distance obtaining unit 25 that obtains a land mark distance. The monitoring device 1 further includes a parameter correcting unit 26 that corrects camera parameters including the focal distance and the FOE coordinate using a plurality of positions of the land mark in the distortion correction image and the plurality of land mark distances.SELECTED DRAWING: Figure 2

Description

Technical Field

[0001] The present disclosure relates to a monitoring device that monitors an area including a road on which a moving object moves as a monitoring area.

Background Art

[0002] Conventionally, by comparing the distance to a ground object obtained based on an image captured by an in-vehicle camera with the distance to the ground object obtained based on the position information of the ground object stored in high-precision map data, a device for detecting the optical axis deviation of the camera is known (see, for example, Patent Document 1).

Prior Art Documents

Patent Documents

[0003]

Patent Document 1

Summary of the Invention

Problems to be Solved by the Invention

[0004] By the way, the inventors of the present invention have been earnestly studying a monitoring device that monitors an area including a road on which a moving object moves as a monitoring area and monitors an object in the monitoring area. Specifically, it is being considered to obtain the distance from the camera to the object based on the position of the object in the captured image obtained by imaging the monitoring area with the camera, the focal length of the camera, and the FOE coordinates.

[0005] In this type of monitoring device, when monitoring an object in a state where the captured image is distorted, since it affects the monitoring result, it is desirable to generate a distortion-corrected image in which the distortion of the captured image is corrected using distortion correction parameters set based on a camera installation model assumed in advance.

[0006] However, in reality, there are differences between the installation model of the camera and the actual unit due to factors such as the inclination during camera installation and individual differences of the camera. As a result, errors occur in camera parameters such as the focal length and FOE coordinates corresponding to the distortion-corrected image. Such errors in camera parameters are not preferable because they cause a decrease in the measurement accuracy of the distance from the camera to the object. These were discovered after the inventors' intensive studies. Note that FOE is an abbreviation for Focus Of Expansion and is defined as the intersection of a uniaxial ray parallel to the road surface of the road and the image plane.

[0007] An object of the present disclosure is to provide a monitoring device capable of suppressing errors in camera parameters and ensuring the measurement accuracy of the distance from the camera to the object.

Means for Solving the Problems

[0008] The invention according to claim 1 is A monitoring device that monitors an area including a road on which a moving body moves as a monitoring area and monitors an object in the monitoring area, A camera (10) that images the monitoring area, A distortion correction unit (21) that generates a distortion-corrected image obtained by correcting the distortion of the captured image captured by the camera and obtains the focal length and FOE coordinates corresponding to the distortion-corrected image, An object detection unit (22) that detects the position of an object appearing in the distortion-corrected image as an image object position, A distance measurement unit (23) that obtains the actual distance from the camera to the object using the image object position, focal length, and FOE coordinates detected by the object detection unit, A mark detection unit (24) that detects a specific element on the road appearing in the distortion-corrected image as a landmark, A mark distance acquisition unit (25) that acquires a landmark distance, which is the distance from the camera to the landmark in a uniaxial direction parallel to the road surface of the road, A parameter correction unit (26) that corrects camera parameters including a focal length and FOE coordinates, using a plurality of pieces of mark information that are sets of a mark position, which is the position of a landmark in a distortion-corrected image, and a landmark distance obtained by a mark distance acquisition unit.

[0009] The distance measurement unit can estimate the distance from the camera to the landmark in a uniaxial direction parallel to the road surface based on the mark position in the distortion-corrected image. Since the estimated distance obtained by the distance measurement unit uses camera parameters, it is affected by errors in the camera parameters.

[0010] On the other hand, the landmark distance from the camera to the landmark in the uniaxial direction obtained by the mark distance acquisition unit is not affected by errors in the camera parameters, unlike the estimated distance obtained by the distance measurement unit.

[0011] Thus, the landmark distance and the estimated distance obtained by the distance measurement unit differ in terms of the presence or absence of the influence of errors on the camera parameters. Therefore, by using a plurality of pieces of mark information that are sets of a mark position and a landmark distance, the camera parameters including the focal length and FOE coordinates can be appropriately corrected.

[0012] Therefore, according to the monitoring device of the present disclosure, it is possible to suppress errors in the camera parameters and ensure the measurement accuracy of the distance from the camera to an object.

[0013] Note that the reference numerals in parentheses attached to each component etc. indicate an example of the correspondence relationship between the component etc. and the specific components etc. described in the embodiments described later.

Brief Description of the Drawings

[0014]

Figure 1

Figure 2

Figure 3

Figure 4

Figure 5

Figure 6

Figure 7

Figure 8

Figure 9

Figure 10

Figure 11

Figure 12

Figure 13

Figure 14

Figure 15

Figure 16

Figure 17

Figure 18

Figure 19

Figure 20

Figure 21

Figure 22

Embodiments for Carrying Out the Invention

[0015] Hereinafter, embodiments of the present disclosure will be described with reference to the drawings. In the following embodiments, parts that are the same as or equivalent to those described in the preceding embodiments may be given the same reference numerals, and the description thereof may be omitted. Also, in the embodiments, when only a part of the components is described, the components described in the preceding embodiments can be applied to the other parts of the components. The following embodiments can be partially combined with each other as long as there is no problem in the combination, even if not particularly specified.

[0016] (First Embodiment) This embodiment will be described with reference to FIGS. 1 to 13. In this embodiment, an example in which the monitoring device 1 of the present disclosure is applied to a traffic monitoring system TMS will be described. The monitoring device 1 is a device that monitors an area including a road on which a moving object moves as a monitoring area. Note that the "moving object" includes not only mechanical structures such as vehicles but also humans, animals, and the like.

[0017] As shown in FIG. 1, the traffic monitoring system TMS includes the monitoring device 1 and lighting devices LT such as streetlights. The traffic monitoring system TMS controls the lighting device LT according to the monitoring result of the monitoring area by the monitoring device 1. For example, when a vehicle in motion is detected by the monitoring device 1, the traffic monitoring system TMS alerts pedestrians walking near the road by turning on the streetlights or increasing the brightness of the streetlights.

[0018] The monitoring device 1 is attached to, for example, the columns of lighting devices LT, traffic lights, or dedicated columns so as to be able to monitor a monitoring area including a road. Specifically, as shown in FIG. 2, the monitoring device 1 includes a camera 10 that images the monitoring area, and a control unit 20 that monitors the monitoring area based on the captured image captured by the camera 10.

[0019] The camera 10, for example, as shown in FIG. 3, uses an area including an intersection where a plurality of roads intersect as the monitoring area and images the monitoring area. The camera 10 of the present embodiment is configured by an omnidirectional camera that images the entire area around itself. The camera 10 receives an omnidirectional view at once and outputs it as the captured omnidirectional images. The camera 10 is arranged substantially parallel to the road surface so that, for example, the imaging surface 121 is horizontal.

[0020] Specifically, the camera 10 includes a wide-angle lens 11 and an imaging unit 12. Conceptually, as shown in FIG. 4, the wide-angle lens 11 projects an image of an area imaged on the lens surface 111 of a hemispherical surface onto the imaging surface 121 passing through the center point RO of the lens surface 111. Thereby, the imaging unit 12 receives an omnidirectional view at once. The imaging unit 12 is configured by, for example, a camera including a photoelectric conversion element such as a CCD or a CMOS. Note that CCD is an abbreviation for Charge Coupled Device. CMOS is an abbreviation for Complementary Metal Oxide Semiconductor.

[0021] Here, the size of the image in the omnidirectional image obtained by the camera 10 decreases as the distance from the lens surface 111 of the wide-angle lens 11 increases. For this reason, the omnidirectional image becomes an image distorted in an arc shape, for example, as shown in FIG. 5.

[0022] The control unit 20 is composed of a computer including a processor and a memory and its peripheral devices. The memory stores programs, data, etc. for executing various control processes. The control unit 20 executes various programs stored in the memory. The control unit 20 executes a monitoring process for monitoring a moving object in the monitoring area based on the captured image captured by the camera 10. In addition, the control unit 20 executes a correction process for camera parameters. Note that the memory of the control unit 20 is composed of a non-transitory tangible storage medium.

[0023] The control unit 20 is communicably connected to external devices such as the camera 10, the position identification device PD, and the lighting device LT. The connection mode between the control unit 20 and the external devices may be either wireless or wired.

[0024] The control unit 20 functions as various functional units by executing various programs and the like. The control unit 20 of the present embodiment is configured to include a distortion correction unit 21, an object detection unit 22, a distance measurement unit 23, a mark detection unit 24, a mark distance acquisition unit 25, and a parameter correction unit 26 as various functional units.

[0025] As described above, the omnidirectional image is distorted from the actual scenery and is difficult to handle as it is. Therefore, in the control unit 20, the distortion of the omnidirectional image is corrected by the distortion correction unit 21. The distortion correction unit 21 generates a distortion-corrected image obtained by correcting the distortion of the captured image captured by the camera 10, and obtains the focal length FL and the FOE coordinates corresponding to the distortion-corrected image.

[0026] The distortion correction unit 21 of the present embodiment divides and extracts specific locations in the captured image obtained from the camera 10, and corrects the optical distortion of the captured image using the parameters for distortion correction processing stored in the parameter storage unit PM for the extracted image.

[0027] The skew correction unit 21 extracts, for example, as shown in FIG. 6, the division points div1, div3, div5, div7 corresponding to four roads near the intersection shown in the captured image and the distant division points div0, div2, div4, div6 connected thereto. After that, the skew correction unit 21 corrects the optical distortion of the divided images corresponding to the division points div0 to div7, and outputs, for example, eight skew-corrected images shown in FIG. 7. Note that the division points div0 to div7 are not limited to eight, and can be any number. Further, the division points div0 to div7 may be set by an operator, or may be automatically set based on the flow of vehicles traveling on the road.

[0028] In addition, the skew correction unit 21 obtains the focal length FL and the FOE coordinates corresponding to each skew-corrected image. The focal length FL is obtained based on the design value of the wide-angle lens 11 or the like. The FOE coordinates are obtained based on various parameters including the focal length FL and the installation height Hc of the camera 10. The FOE coordinates are the intersection of the extension line of the movement vector (so-called optical flow) of the moving object shown in the image and the image plane, and can also be obtained based on the intersections of a plurality of optical flows.

[0029] The object detection unit 22 detects the position of the object shown in each skew-corrected image as the image object position. When a vehicle is shown in the skew-corrected image, the object detection unit 22 detects the coordinates corresponding to a specific part of the vehicle in the skew-corrected image as the image object position. The object detection unit 22 of the present embodiment detects, for example, as shown in FIG. 8, the lower end coordinates corresponding to the lower end portion in the skew-corrected image as the image object position. Note that the object detection unit 22 may detect the coordinates of a part other than the lower end portion of the vehicle as the image object position. Further, the "object" includes not only mechanical structures such as vehicles, but also humans, animals, and the like.

[0030] The distance measurement unit 23 obtains the actual distance TL from the camera 10 to the object using the image object position, focal length FL, and FOE coordinates detected by the object detection unit 22. The actual distance TL obtained here is the distance from the camera 10 to the vehicle in the three-dimensional space. The distance measurement unit 23 obtains the actual distance TL from the camera 10 to the vehicle based on, for example, the lower end coordinates corresponding to the lower end part of the vehicle, the FOE coordinates, the focal length FL, and the installation height Hc of the camera 10, using the mathematical formula F1 shown in FIG. 9.

[0031] The mark detection unit 24 detects specific elements on the road shown in the distortion-corrected image as landmarks. The mark detection unit 24 of the present embodiment detects the road surface paint on the road shown in the distortion-corrected image as a landmark.

[0032] For example, as shown in FIG. 10, near an intersection, a zebra crossing with a striped pattern and a diamond mark for notifying the presence of the zebra crossing may be set as road surface paint. The mark detection unit 24 of the present embodiment detects the road surface paint of the zebra crossing and the diamond mark as landmarks. Note that the road surface paint includes not only legal markings such as regulatory markings and indication markings set on the road surface of the road, but also non-statutory markings indicating directions and guiding routes. Note that the diamond marks are set at positions 30 m and 50 m away from the zebra crossing.

[0033] In addition, the mark detection unit 24 detects feature points that are the features of the landmark from the mark area corresponding to the landmark. The mark detection unit 24 detects the corner points in the mark area as feature points. For the road surface paint, the mark detection unit 24 detects, for example, as shown in FIG. 11, the corner points of the diamond mark and the corner points of the vertical stripe part of the zebra crossing as feature points. For example, as shown in FIG. 12, each of the various-shaped landmarks (A) to (F) has a plurality of corner points. Therefore, the mark detection unit 24 detects each corner point as a feature point of the landmark.

[0034] The mark distance acquisition unit 25 acquires from the outside a landmark distance, which is the distance from the camera 10 to the landmark in a uniaxial direction parallel to the road surface. Note that the "uniaxial direction parallel to the road surface" is the direction in which a ray of light extending infinitely horizontally extends.

[0035] The mark distance acquisition unit 25 acquires the landmark distance obtained by using a high-precision position identification device PD such as a map database outside the monitoring device 1. The landmark distance can be obtained, for example, by using the latitude and longitude information of the feature points of the landmark included in the map database and the latitude and longitude information of the camera 10. Note that the function of calculating the landmark distance may be included in the control unit 20 or may be included in a device other than the control unit 20.

[0036] The parameter correction unit 26 corrects camera parameters including the focal length FL and the FOE coordinates by using a plurality of pieces of mark information in which the mark position, which is the position of the landmark in the distortion-corrected image, and the landmark distance acquired by the mark distance acquisition unit 25 are paired. Note that, as shown in the formula F1 in FIG. 9, the focal length FL and the FOE coordinates, which are camera parameters, are correlated with the landmark distance and the mark position in the distortion-corrected image. Therefore, the camera parameters can be corrected based on the landmark distance and the mark position in the distortion-corrected image.

[0037] Specifically, the parameter correction unit 26 corrects the camera parameters so that the difference between the estimated distance from the camera 10 to the landmark in the uniaxial direction estimated based on the mark position in the distortion-corrected image and the landmark distance acquired by the mark distance acquisition unit 25 becomes small. In other words, the parameter correction unit 26 uses the above-described estimated distance as a measured value and the landmark distance acquired by the mark distance acquisition unit 25 as a true value, and corrects the camera parameters so that the measured value approaches the true value.

[0038] Here, the above-mentioned estimated distance can be obtained based on, for example, the mark position specified by the mark detection unit 24, the focal length FL, the actual distance TL from the camera 10 to the landmark based on the FOE coordinates, and the installation height Hc of the camera 10. Note that the estimated distance is the distance from the camera 10 to the landmark in a single-axis direction, which is different from the actual distance.

[0039] Next, the correction process of the camera parameters executed by the control unit 20 of the monitoring device 1 will be described with reference to the flowchart shown in FIG. 13. The control routine shown in FIG. 13 is periodically or irregularly executed by the control unit 20 after the installation work of the camera 10 is completed.

[0040] As shown in FIG. 13, in step S100, the control unit 20 extracts the coordinates of the landmark from the distortion-corrected image. Specifically, the control unit 20 extracts the coordinates of the corner points of the landmark from the distortion-corrected image as the mark position.

[0041] Subsequently, in step S110, the control unit 20 obtains the estimated distance from the camera 10 to the landmark in a single-axis direction parallel to the road surface based on the mark position in the distortion-corrected image. Specifically, the control unit 20 calculates the estimated distance using the mark position in the distortion-corrected image, the focal length FL, the FOE image, and the installation height Hc of the camera 10. Then, in step S120, the control unit 20 obtains the landmark distance obtained by using the map database outside the monitoring device 1.

[0042] Subsequently, in step S130, the control unit 20 calculates the difference between the estimated distance obtained in step S110 and the landmark distance obtained in step S120 as an error. Then, in step S140, the control unit 20 determines whether the error is extremely small. Specifically, the control unit 20 determines whether the error obtained in step S130 is within a predetermined allowable threshold.

[0043] When the error exceeds the tolerance threshold, the control unit 20 proceeds to step S150 and finely adjusts the camera parameters. Specifically, the control unit 20 uses the focal length FL and the FOE coordinates stored in the memory as initial values, and stores in the memory, as the focal length FL and the FOE coordinates, those obtained by changing the initial values. Thereafter, the control unit 20 returns to step S110 and obtains the estimated distance from the camera 10 to the landmark based on the camera parameters set in step S150.

[0044] On the other hand, when the error is within the tolerance threshold, the control unit 20 proceeds to step S160 and sets the focal length FL and the FOE coordinates stored in the memory as camera parameters suitable for the installation environment of the camera 10.

[0045] As described above, the monitoring device 1 can obtain, at the distance measurement unit 23, the estimated distance from the camera 10 to the landmark in a uniaxial direction parallel to the road surface based on the mark position in the distortion-corrected image. Since the estimated distance obtained by the distance measurement unit 23 is obtained using camera parameters, it is affected by errors in the camera parameters.

[0046] On the other hand, unlike the estimated distance obtained by the distance measurement unit 23, the landmark distance from the camera 10 to the landmark in the uniaxial direction obtained by the mark distance acquisition unit 25 is not affected by errors in the camera parameters.

[0047] As described above, the landmark distance and the estimated distance obtained by the distance measurement unit 23 differ in terms of the influence of errors in the camera parameters. Therefore, by using a plurality of pieces of mark information with a set of the mark position and the landmark distance, the camera parameters including the focal length FL and the FOE coordinates can be appropriately corrected.

[0048] Therefore, according to the monitoring device 1 of the present embodiment, it is possible to suppress errors in the camera parameters and ensure the measurement accuracy of the distance from the camera 10 to the object.

[0049] In addition, the monitoring device 1 includes a parameter correction unit 26. With such a configuration, even if changes in the camera posture over time occur, there is an advantage that the camera parameters can be appropriately corrected.

[0050] Further, the monitoring device 1 has the following features. (1) The parameter correction unit 26 corrects the camera parameters so that the difference between the distance from the camera 10 to the landmark in the uniaxial direction estimated based on the mark position in the distortion-corrected image by the distance measurement unit 23 and the landmark distance becomes small. According to this, the error of the camera parameters including the focal length FL and the FOE coordinates can be reduced, and the measurement accuracy of the distance to the object can be improved.

[0051] (2) Here, as a method for correcting general camera parameters, there is a method that uses a special jig such as a calibration mark. According to this method, it is necessary to regulate the road within a wide monitoring area. This will hinder the spread of the monitoring device 1.

[0052] On the other hand, the mark detection unit 24 of the monitoring device 1 of the present embodiment detects the road surface paint on the road shown in the distortion-corrected image as a landmark, and detects the feature points that are the features of the landmark from the mark area corresponding to the landmark. According to this, the camera parameters can be corrected without regulating the road, so that the work can be simplified. This greatly contributes to the spread of the monitoring device 1.

[0053] (3) The mark detection unit 24 detects the corner points in the mark area as feature points. In this way, if the corner points of the landmark are used as feature points, there is an advantage that the feature points become unique points and it becomes easy to specify the landmark distance.

[0054] (Modification of the First Embodiment) As in the first embodiment, it is desirable that the monitoring device 1 corrects the camera parameters so that the difference between the estimated distance from the camera 10 to the landmark estimated based on the mark position in the distortion-corrected image and the landmark distance becomes small, but it is not limited to this. The monitoring device 1 may correct the camera parameters by referring to data or a map in which the relationship between the difference between the estimated distance from the camera 10 to the landmark estimated based on the mark position in the distortion-corrected image and the landmark distance and the camera parameters is defined in advance.

[0055] As in the first embodiment, the monitoring device 1 detects the road surface paint on the road shown in the distortion-corrected image as a landmark, and detects the feature points that are the features of the landmark from the mark area corresponding to the landmark, but it is not limited to this. The monitoring device 1 may be configured to detect, for example, road furniture including manholes, curbs, reflectors, etc. on the road shown in the distortion-corrected image as landmarks.

[0056] As in the first embodiment, it is desirable that the monitoring device 1 uses the corner points of the landmark as feature points, but it is not limited to this. For example, for a landmark without corner points, the monitoring device 1 may use the center position in the landmark as a feature point.

[0057] (Second Embodiment) Next, the second embodiment will be described with reference to FIGS. 14 to 18. In this embodiment, the parts different from the first embodiment will be mainly described.

[0058] As shown in FIG. 14, the mark detection unit 24 of the monitoring device 1 includes a candidate point selection unit 241 that selects a plurality of candidate points CP to be used when obtaining the distance to the landmark from among the feature points of the landmark.

[0059] The candidate point selection unit 241 selects, as candidate points CP, among the plurality of feature points of the landmark, those whose distance from the camera 10 is within a predetermined key monitoring range. The key monitoring range is, for example, a range including a road on which a vehicle moves and a sidewalk on which a pedestrian moves, and does not include areas where moving objects such as vehicles and pedestrians cannot move (for example, building walls, rivers, forests).

[0060] Here, if there is a bias in the positions of the candidate points CP, there is a risk that the camera parameters will become correction values that are overly adapted to a specific distance. For example, as shown in FIG. 15, when only the feature points of the diamond marks among the crosswalk and the diamond marks are used as the candidate points CP, there is a risk that the measurement accuracy of the distance near the crosswalk located at a position away from the candidate points CP cannot be ensured sufficiently.

[0061] In consideration of this, the candidate point selection unit 241 selects the candidate points CP from among the plurality of feature points so that, for example, as shown in FIG. 16, the distances from the camera 10 to the positions corresponding to the plurality of candidate points CP in the landmark are dispersed.

[0062] Specifically, when three or more feature points are detected, the candidate point selection unit 241 selects, as the candidate points CP, the combination of feature points among the three or more feature points for which the difference is the largest when comparing the distances from the camera 10. Also in this case, it is desirable that feature points outside the key monitoring range be excluded from the candidates. Note that the distance from the camera 10 may be obtained by either the distance measurement unit 23 or the mark distance acquisition unit 25.

[0063] Further, when there is no road surface paint in the distortion-corrected image or even if there is road surface paint, its position is extremely deviated, the number of candidate points CP used when obtaining the distance to the landmark may be insufficient.

[0064] In view of this, the control unit 20 of the present embodiment corrects the camera parameters using the position of the detected object on the image detected by the object detection unit 22 and the distance to the detected object.

[0065] The control unit 20, for example, temporarily arranges an object for correcting camera parameters at a specific position on the road, detects the position of the object in the image by the object detection unit 22, and acquires the distance from the camera 10 to the specific position using a surveying instrument or the like. Note that the control unit 20 may detect the position in the image when a person or vehicle having a GPS transmitter moves within the monitoring area by the object detection unit 22, and acquire the distance from the GPS transmitter to the camera 10 based on the position of the GPS transmitter as the object distance. Further, the control unit 20 may arrange an object for correcting camera parameters on the roof of a house or the like instead of on the road, detect the position of the object in the image by the object detection unit 22, and acquire the distance from the camera 10 to the specific position using a surveying instrument or the like.

[0066] Specifically, the control unit 20 of the present embodiment includes an object distance acquisition unit 27 that acquires the distance from the camera 10 to the detected object detected by the object detection unit 22 in a uniaxial direction as the object distance. The object distance acquisition unit 27 acquires the object distance from distance measuring means such as a surveying instrument or GPS, for example.

[0067] Further, when the number of candidate points CP selected from the feature points of the road surface paint is equal to or less than a predetermined reference number, the mark detection unit 24 of the present embodiment selects the candidate points CP including the image object position of the detected object detected by the object detection unit 22 as the feature points. Then, when the image object position of the detected object detected by the object detection unit 22 is included in the candidate points CP, the parameter correction unit 26 corrects the camera parameters using the mark position, landmark distance, image object position, and object distance obtained by the object distance acquisition unit 27 in the distortion-corrected image.

[0068] Next, the correction process of the camera parameters executed by the control unit 20 of the present embodiment will be described with reference to the flowchart shown in FIG. 17. Since the processes of steps S100, S110 to S160 shown in FIG. 17 are substantially the same as those in the first embodiment, the description thereof will be omitted.

[0069] As shown in FIG. 17, after the control unit 20 extracts the coordinates of the landmark from the distortion-corrected image in step 100, it proceeds to step S105 to select candidate points CP to be used when obtaining the distance to the landmark from among the feature points of the landmark. Hereinafter, the selection of the candidate points CP will be described with reference to the flowchart of FIG. 18.

[0070] As shown in FIG. 18, the control unit 20 extracts, in step S200, among the plurality of feature points in the landmark, those within the key monitoring range set within the monitoring area as candidate points CP.

[0071] Subsequently, in step S210, the control unit 20 selects, as candidate points CP, the combination of feature points with the largest difference when comparing the distances from the camera 10 such that the distances from the camera 10 to the positions corresponding to the plurality of candidate points CP in the landmark are dispersed.

[0072] Subsequently, in step S220, the control unit 20 determines whether the number of selected candidate points CP is insufficient. For example, the control unit 20 determines whether the number of candidate points CP selected from among the feature points of the road surface paint is less than or equal to a predetermined reference number.

[0073] When the number of candidate points CP is less than or equal to the reference number, the control unit 20 adds, in step S230, the corner points on the image of the detected object detected by the object detection unit 22 to the feature points of the landmark, and then returns to step S200. As a result, candidate points CP are extracted from among the feature points on the image of the road surface paint and the feature points on the image of the detected object.

[0074] On the other hand, when the number of candidate points CP exceeds the reference number, the control unit 20 determines, in step S240, those extracted in the previous processing as candidate points CP. Thereafter, the control unit 20 proceeds to the processing of step S110 in FIG. 17.

[0075] For other aspects, they are the same as those of the first embodiment. The monitoring device 1 of this embodiment can obtain the same effects as those achieved by the same or equivalent configurations as those of the first embodiment in the same manner as the first embodiment.

[0076] Also, the monitoring device 1 of this embodiment has the following features. (1) Among the plurality of feature points, the mark detection unit 24 selects a plurality of those whose distance from the camera 10 is within a predetermined key monitoring range as candidate points CP. In this way, if the ones within the key monitoring range among the plurality of feature points are preferentially selected as candidate points CP, the camera parameters can be appropriately corrected to improve the measurement accuracy of the distance to the object.

[0077] (2) The mark detection unit 24 selects candidate points CP from among the plurality of feature points so that the distances from the camera 10 to the positions corresponding to the plurality of candidate points CP in the landmark are dispersed. Also by this, the bias of the positions of the candidate points CP can be suppressed, so that it is possible to avoid the camera parameters becoming correction values that are overly adapted to a specific distance.

[0078] (3) When three or more feature points are detected, the mark detection unit 24 selects, as candidate points CP, the combination of feature points with the largest difference in the distances from the camera 10 among the three or more feature points. According to this, the bias of the positions of the candidate points CP can be suppressed, so that it is possible to avoid the camera parameters becoming correction values that are overly adapted to a specific distance.

[0079] (4) The monitoring device 1 includes an object distance acquisition unit 27 that acquires, as the object distance, the distance from the camera 10 in the uniaxial direction to the detected object detected by the object detection unit 22. When the number of candidate points CP selected from among a plurality of feature points is equal to or less than a predetermined reference number, the mark detection unit 24 selects the candidate points CP including the image object position of the detected object detected by the object detection unit 22 as feature points. When the image object position of the detected object detected by the object detection unit 22 is included in the candidate points CP, the parameter correction unit 26 corrects the camera parameters using the mark position, landmark distance, image object position, and object distance acquired by the object distance acquisition unit 27 in the distortion-corrected image. According to this, even in a situation where it is difficult to sufficiently secure the candidate points CP only with road paint, the number of candidate points CP can be secured and the correction of the camera parameters can be appropriately corrected.

[0080] (Modification of the Second Embodiment) As in the second embodiment, it is desirable that the monitoring device 1 selects the candidate points CP from among a plurality of feature points so that the distances from the camera 10 to the positions corresponding to the plurality of candidate points CP in the landmark are dispersed, but it is not necessary to be so.

[0081] As in the second embodiment, it is desirable that the monitoring device 1 selects the candidate points CP including the image object position of the detected object detected by the object detection unit 22 as feature points when the number of candidate points CP selected from among a plurality of feature points is equal to or less than the reference number, but it is not limited to this. The monitoring device 1 may select the candidate points CP including the image object position of the detected object detected by the object detection unit 22 as feature points even when the number of candidate points CP selected from among a plurality of feature points exceeds a predetermined reference number. Further, the monitoring device 1 may not perform the correction of the camera parameters when the number of candidate points CP selected from among a plurality of feature points is equal to or less than a predetermined reference number.

[0082] (Third Embodiment) Next, the third embodiment will be described with reference to FIGS. 19 and 20. In this embodiment, mainly the parts different from the second embodiment will be described.

[0083] As shown in FIG. 19, the control unit 20 of the present embodiment includes a landmark storage unit 28 that stores mark information that is a set of a mark position that is the position of a landmark in the distortion-corrected image and a landmark distance acquired by the mark position and mark distance acquisition unit 25.

[0084] Further, the control unit 20 includes a landmark tracking unit 29 that uses the mark position stored in the landmark storage unit 28 as a storage position and compares the storage position with the mark position in the distortion-corrected image obtained by correcting the captured image newly captured by the camera 10.

[0085] When the storage position stored in the landmark storage unit 28 and the mark position in the newly obtained distortion-corrected image are different, the landmark tracking unit 29 stores the mark position in the newly obtained distortion-corrected image in the landmark storage unit 28 as the storage position. That is, the landmark tracking unit 29 tracks the position of the landmark, and when the position of the landmark changes, updates the storage position stored in the landmark storage unit 28 to the position of the latest landmark.

[0086] However, when some object appears near the mark position in the distortion-corrected image, at least a part of the landmark may be hidden by the object, and there is a possibility that the position of the landmark in the distortion-corrected image cannot be appropriately detected.

[0087] In consideration of this, when there is a detected object detected by the object detection unit 22 in a predetermined region including the mark position in the distortion-corrected image, the landmark tracking unit 29 of the present embodiment does not update the storage position stored in the landmark storage unit 28.

[0088] Next, the landmark tracking process executed by the control unit 20 of the present embodiment will be described with reference to the flowchart shown in FIG. 20. The control routine shown in FIG. 20 is executed by the control unit 20, for example, when the correction process is executed or in response to a command signal from the outside.

[0089] As shown in FIG. 20, in step S300, the control unit 20 obtains, as the mark position, the position of the landmark in the newly obtained distortion-corrected image. Further, the control unit 20 detects an object in the newly obtained distortion-corrected image.

[0090] Subsequently, in step S310, the control unit 20 determines whether there is a detected object detected by the object detection unit 22 near the mark position in the newly obtained distortion-corrected image. For example, the control unit 20 sets, as a predetermined region, a region corresponding to the entire landmark in the distortion-corrected image, and determines whether a part of the predetermined region overlaps with the detected object. Note that the predetermined region may be a region larger than the landmark in the distortion-corrected image.

[0091] When there is no detected object near the newly obtained mark position, the control unit 20 determines, in step S320, whether the mark position and the storage position stored in the landmark storage unit 28 substantially match. In other words, the control unit 20 determines whether the difference between the mark position and the storage position is equal to or less than a predetermined determination threshold.

[0092] When the mark position and the storage position are different, the control unit 20 stores, in the landmark storage unit 28, the newly obtained mark position as the storage position in step S330. When the mark position and the storage position substantially match, the control unit 20 skips step S330 and exits this process.

[0093] On the other hand, when there is a detected object near the newly obtained mark position, the control unit 20 skips the processes of steps S320 and S330 and exits this process. That is, when there is a detected object in a predetermined region including the newly obtained mark position, the control unit 20 exits this process without updating the storage position.

[0094] Other aspects are the same as those of the second embodiment. The monitoring device 1 of this embodiment can obtain, in the same manner as the second embodiment, the effects resulting from the same configuration or an equivalent configuration as that of the second embodiment.

[0095] In addition, the monitoring device 1 of the present embodiment has the following features. (1) The control unit 20 includes a landmark storage unit 28 and a landmark tracking unit 29. When the storage position stored in the landmark storage unit 28 is different from the mark position in the newly obtained distortion-corrected image, the landmark tracking unit 29 stores the mark position in the newly obtained distortion-corrected image as the storage position in the landmark storage unit 28. According to this, for example, even if changes in the camera posture due to aging occur, there is an advantage that the camera parameters can be appropriately corrected using the appropriate landmark position.

[0096] (2) When the position of the detected object shown in the distortion-corrected image is close to the storage position of the landmark, there is a risk that the landmark may be blocked by the detected object or the features of the detected object may be misdetected as the features of the landmark.

[0097] In contrast, when there is a detected object detected by the object detection unit 22 in a predetermined region including the mark position in the distortion-corrected image, the landmark tracking unit 29 does not update the storage position stored in the landmark storage unit 28. According to this, inappropriate correction of the camera parameters due to interference between the landmark and the detected object can be avoided.

[0098] (Modification of the Third Embodiment) As in the third embodiment, it is desirable that the monitoring device 1 prohibits the update of the storage position stored in the landmark storage unit 28 when there is a detected object near the mark position in the distortion-corrected image, but it is not necessary to be like this.

[0099] (Fourth Embodiment) Next, the fourth embodiment will be described with reference to FIGS. 21 and 22. In this embodiment, the parts different from the third embodiment will be mainly described.

[0100] As shown in FIG. 21, the landmark tracking unit 29 of the present embodiment includes an abnormality determination unit 291 that determines the presence or absence of a correction abnormality indicating that it is difficult to appropriately perform correction of camera parameters based on the tracking result of the position of the landmark.

[0101] Here, if the mounting position of the camera 1 changes due to external factors such as a typhoon, or if the road surface paint fades or is repainted over time, the stored position of the landmark and the newly detected mark position may change significantly. Such a change in the mark position can be a factor that makes it difficult to appropriately perform correction of the camera parameters.

[0102] In consideration of this, the abnormality determination unit 291 of the present embodiment determines that there is a correction abnormality indicating that it is difficult to appropriately perform correction of the camera parameters when the difference between the stored position and the mark position in the newly obtained distortion-corrected image exceeds a predetermined abnormality threshold.

[0103] Further, when the landmark tracking unit 29 determines that there is a correction abnormality, it outputs a signal indicating the correction abnormality to a notification device 31 that notifies the outside of the correction abnormality. The notification device 31 is, for example, a device that notifies information to a management terminal held by the administrator of the monitoring device 1, and notifies the administrator, for example, that maintenance of the monitoring device 1 is necessary because a correction abnormality has occurred. The notification device 31 may be provided outside the monitoring device 1 or may be configured as a part of the monitoring device 1.

[0104] Next, the landmark tracking process executed by the control unit 20 of the present embodiment will be described with reference to the flowchart shown in FIG. 22. Since the processes of steps S300 to S320 and S330 shown in FIG. 22 are substantially the same as those of the first embodiment, the description thereof will be omitted.

[0105] As shown in FIG. 22, when the mark position and the stored position are different, the control unit 20 proceeds to step S325 to determine whether the difference between the mark position and the stored position exceeds an abnormal threshold value. The abnormal threshold value is set to a value larger than the determination threshold value used in the determination process of step S320.

[0106] When the difference between the mark position and the stored position does not exceed the abnormal threshold value, the control unit 20 proceeds to step S330, stores the newly obtained mark position as the stored position in the landmark storage unit 28, and exits this process.

[0107] On the other hand, when the difference between the mark position and the stored position exceeds the abnormal threshold value, the control unit 20 determines a correction abnormality at step S340 and outputs a signal indicating the correction abnormality to the notification device 31. As a result, it is conveyed to the administrator of the monitoring device 1 via the notification device 31 that maintenance of the monitoring device 1 is necessary.

[0108] For the rest, it is the same as the third embodiment. The monitoring device 1 of this embodiment can obtain the same effects as those achieved by the common configuration or equivalent configuration as the third embodiment in the same manner as the third embodiment.

[0109] In addition, the monitoring device 1 of this embodiment has the following features. (1) When the difference between the stored position and the mark position in the newly obtained distortion-corrected image exceeds a predetermined abnormal threshold value, the landmark tracking unit 29 determines a correction abnormality indicating that it is difficult to appropriately perform correction of the camera parameters. According to this, for example, inappropriate correction of the camera parameters due to changes in the mounting state of the camera 10, changes in the mark position, movement of the mark position, etc. can be avoided.

[0110] (2) When the landmark tracking unit 29 determines a correction abnormality, it outputs a signal indicating the correction abnormality to the notification device 31 that notifies the outside of the correction abnormality. In this way, if the correction abnormality is notified to the outside, it becomes possible to take measures for a situation where it is difficult to appropriately perform correction of the camera parameters.

[0111] (Modification of the Fourth Embodiment) As in the fourth embodiment, it is desirable that the monitoring device 1 outputs a signal indicating a correction abnormality to the notification device 31 that notifies the outside of the correction abnormality when a correction abnormality is determined, but it is not necessary for it to be configured in this way.

[0112] (Other Embodiments) Although the representative embodiments of the present disclosure have been described above, the present disclosure is not limited to the above-described embodiments, and can be variously modified as follows, for example.

[0113] In the above-described embodiment, the monitoring device 1 in which the camera 10 is configured as an omnidirectional camera has been illustrated, but the present disclosure is not limited to this. The camera 10 may be configured as a camera with a narrower viewing angle than an omnidirectional camera.

[0114] In the above-described embodiment, the monitoring device 1 in which the camera 10 and the control unit 20 are provided integrally has been illustrated, but the monitoring device 1 may be configured such that the camera 10 and the control unit 20 are separate and arranged at positions separated from each other.

[0115] In the above-described embodiment, the traffic monitoring system TMS combining the monitoring device 1 and the lighting device LT has been illustrated, but the traffic monitoring system TMS may be configured by combining devices other than the monitoring device 1 and the lighting device LT.

[0116] In the above-described embodiment, an example in which the monitoring device 1 of the present disclosure is applied to the traffic monitoring system TMS has been described, but the application target of the monitoring device 1 is not limited to the traffic monitoring system TMS. The monitoring device 1 is also applicable to monitoring systems in parking lots, sidewalks, factories, etc.

[0117] Needless to say, in the above-described embodiments, the elements constituting the embodiments are not necessarily essential, except when explicitly stated as being particularly essential and when considered to be clearly essential in principle.

[0118] In the above embodiments, when numerical values such as the number of components, numerical values, amounts, ranges, etc. of the embodiments are mentioned, unless otherwise specified as particularly essential or clearly limited to a specific number in principle, they are not limited to that specific number.

[0119] In the above embodiments, when referring to the shape, positional relationship, etc. of components, etc., unless otherwise specified as particularly essential or clearly limited to a specific shape, positional relationship, etc. in principle, they are not limited to that shape, positional relationship, etc.

[0120] The control unit and its method of the present disclosure may be implemented by a dedicated computer provided by configuring a processor and a memory programmed to execute one or more functions embodied by a computer program. The control unit and its method of the present disclosure may be implemented by a dedicated computer provided by configuring a processor with one or more dedicated hardware logic circuits. The control unit and its method of the present disclosure may be implemented by one or more dedicated computers configured by a combination of a processor and a memory programmed to execute one or more functions and a processor configured by one or more hardware logic circuits. Further, the computer program may be stored in a computer-readable non-transitory tangible recording medium as instructions to be executed by a computer.

Explanation of Reference Numerals

[0121] 1 Monitoring device 10 Camera 21 Distortion correction unit 22 Object detection unit 23 Distance measurement unit 24 Mark detection unit 25 Mark distance acquisition unit 26 Parameter correction unit

Claims

1. A monitoring device that monitors an area including a road on which a moving body moves as a monitoring area and monitors an object in the monitoring area, a camera (10) that images the monitoring area, a distortion correction unit (21) that generates a distortion-corrected image obtained by correcting the distortion of the captured image captured by the camera, and obtains a focal length and FOE coordinates corresponding to the distortion-corrected image, an object detection unit (22) that detects the position of the object shown in the distortion-corrected image as an image object position, a distance measurement unit (23) that obtains an actual distance from the camera to the object using the image object position detected by the object detection unit, the focal length, and the FOE coordinates, a mark detection unit (24) that detects a specific element on the road shown in the distortion-corrected image as a landmark, a mark distance acquisition unit (25) that acquires a landmark distance that is a distance from the camera to the landmark in a uniaxial direction parallel to the road surface of the road, a parameter correction unit (26) that corrects camera parameters including the focal length and the FOE coordinates using a plurality of pieces of mark information each including a mark position that is the position of the landmark in the distortion-corrected image and the landmark distance acquired by the mark distance acquisition unit, A monitoring device comprising:

2. The monitoring device according to claim 1, wherein the parameter correction unit corrects the camera parameters such that a difference between an estimated distance from the camera to the landmark in the uniaxial direction estimated based on the mark position in the distortion-corrected image by the distance measurement unit and the landmark distance acquired by the mark distance acquisition unit becomes small.

3. The monitoring device according to claim 1 or 2, wherein the mark detection unit detects a road surface paint on the road shown in the distortion-corrected image as the landmark, and detects a feature point that is a feature of the landmark from a mark area corresponding to the landmark.

4. The monitoring device according to claim 3, wherein the mark detection unit detects a corner point in the mark area as the feature point.

5. The monitoring device according to claim 3, wherein the mark detection unit selects a plurality of the feature points whose distances from the camera are within a predetermined key monitoring range as candidate points to be used when obtaining the distance from the camera to the landmark.

6. The monitoring device according to claim 5, wherein the mark detection unit selects the candidate points from among the plurality of feature points such that the distances from the camera to the positions corresponding to the plurality of candidate points in the landmark are dispersed.

7. The monitoring device according to claim 6, wherein when three or more feature points are detected, the mark detection unit selects, as the candidate points, a combination of the feature points among the three or more feature points for which the difference is the largest when the distances from the camera are compared.

8. Comprising an object distance acquisition unit (27) that acquires, as an object distance, the distance from the camera in the uniaxial direction to a detected object detected by the object detection unit. When the number of candidate points selected from among the plurality of feature points by the mark detection unit is equal to or less than a predetermined reference number, the mark detection unit selects the candidate points including the image object position of the detected object detected by the object detection unit as the feature points. The monitoring device according to claim 5, wherein when the image object position of the detected object detected by the object detection unit is included in the candidate points, the parameter correction unit corrects the camera parameters using the mark position, the landmark distance, the image object position, and the object distance acquired by the object distance acquisition unit in the distortion-corrected image.

9. A landmark storage unit (28) that stores the mark information. A landmark tracking unit (29) that compares the storage position, which is the mark position stored in the landmark storage unit, with the mark position in the distortion-corrected image obtained by correcting the captured image newly captured by the camera. When the storage position and the mark position in the newly obtained distortion-corrected image are different, the landmark tracking unit stores the mark position in the newly obtained distortion-corrected image as the storage position in the landmark storage unit. The monitoring device according to claim 1 or 2, wherein the parameter correction unit corrects the camera parameters using a plurality of the storage positions and the landmark distances stored in the landmark storage unit.

10. The monitoring device according to claim 9, wherein when there is a detected object detected by the object detection unit in a predetermined region including the mark position in the distortion-corrected image, the landmark tracking unit does not update the storage position stored in the landmark storage unit.

11. The monitoring device according to claim 9, wherein when the difference between the stored position and the mark position in the newly obtained distortion-corrected image exceeds a predetermined abnormality threshold, the landmark tracking unit determines a correction abnormality indicating that it is difficult to appropriately perform the correction of the camera parameters.

12. The monitoring device according to claim 11, wherein when the landmark tracking unit determines the correction abnormality, the landmark tracking unit outputs a signal indicating the correction abnormality to a notification device (31) that notifies the correction abnormality to the outside.

Images