Observation equipment

Abstract

To improve estimation accuracy of a distance.SOLUTION: An observation device 10 comprises: an imaging unit 17; a storage unit 18; an acquisition unit 19; and a control unit 20. The imaging unit 17 generates an image. The storage unit 18 stores distance information. The distance information is the relation between the image position and the information about the distance. The acquisition unit 19 acquires the position information about the spatial position of a movable body. The control unit 20 corrects the distance information using the actual measurement distance when a distance deviation between the actual measurement distance and the estimated distance is equal to or greater than a first threshold. The actual measurement distance corresponds to the position information. The estimated distance is acquired on the basis of the image position of the movable body in the image and the distance information.SELECTED DRAWING: Figure 2

Description

[Technical Field]

[0001] The present invention relates to an observation device. [Background technology]

[0002] Sensors that generate images by capturing an image of a predetermined area, such as a camera, are known. Various information can be generated from the image. For example, it has been proposed to calculate the distance in real space to a subject corresponding to the subject image on the ground based on the mounting orientation of the sensor relative to the ground, such as a road, and the position of the subject image on the ground within the image (see Patent Document 1). [Prior art documents] [Patent documents]

[0003] [Patent Document 1] Japanese Patent Application Publication No. 03-273500 Summary of the Invention [Problem to be solved by the invention]

[0004] For example, the application of a camera that generates images to an observation device installed on the side of a road has been considered. Before use, the observation device is calibrated using markers placed at predetermined distances from the observation device, so that distance information relative to the position on the image captured by the camera can be calibrated. However, with conventional calibration, it is difficult to calibrate deviations in distance information due to ground shape, such as undulations, or changes in the installation position over time, and the accuracy of distance estimation can sometimes be low.

[0005] Therefore, an object of the present disclosure, made in consideration of the above-described problems of the conventional technology, is to provide an observation device that improves the accuracy of distance estimation. [Means for solving the problem]

[0006] In order to solve the above-mentioned problems, an observation apparatus according to a first aspect of the present invention comprises: An observation device that estimates a distance to an observation target in a predetermined area, an imaging unit that generates an image by imaging a space including the predetermined area; a storage unit that stores distance information that is a relationship between an image position on an image captured by the imaging unit and information about the distance; an acquisition unit that acquires, from a moving object, position information relating to a spatial position of the moving object; The device is equipped with a control unit that corrects the distance information using the actual measured distance when the distance deviation between the actual measured distance corresponding to the position information and the estimated distance obtained based on the image position of the moving object in the image and the distance information is greater than or equal to a first threshold. [Effects of the Invention]

[0007] According to the observation device according to the present disclosure configured as described above, the accuracy of distance estimation is improved. [Brief explanation of the drawings]

[0008] [Figure 1] FIG. 1 is a diagram showing an example of the configuration of a communication system including an observation device according to an embodiment. [Figure 2] FIG. 2 is a block diagram showing a schematic configuration of the observation device of FIG. 1. [Figure 3] FIG. 2 is a diagram showing a fixed state of the observation device of FIG. 1. [Figure 4] 10 is a diagram illustrating how a distance error occurs due to a difference between an actual road surface and a virtual plane. FIG. [Figure 5] 3 is a first flowchart for explaining a correction process in an initial setting mode executed by the control unit of FIG. 2; [Figure 6] 10 is a second flowchart illustrating the correction process in the initial setting mode executed by the control unit in FIG. 2; [Figure 7] 3 is a first flowchart for explaining the correction process in the normal mode executed by the control unit of FIG. 2; [Figure 8] 10 is a second flowchart illustrating the correction process in the normal mode executed by the control unit in FIG. 2; [Figure 9] This figure shows the distance estimated based on the corrected distance information when the distance information corresponding to the included image position is corrected using the actual distance among multiple actual distances that is not the smallest difference from the actual distance corresponding to the newly calculated distance deviation. [Figure 10] This figure shows the distance estimated based on the corrected distance information when the distance information corresponding to the included image position is corrected using the actual measured distance among multiple actual measured distances that has the smallest difference from the actual measured distance corresponding to the newly calculated distance deviation. DETAILED DESCRIPTION OF THE INVENTION

[0009] Hereinafter, an embodiment of an observation device to which the present disclosure is applied will be described with reference to the drawings.

[0010] 1 shows an example of the configuration of a communication system 11 including an observation device 10 according to one embodiment. The communication system 11 is, for example, a safe driving support communication system for an Intelligent Transport Systems (ITS). The safe driving support communication system is also called a safe driving support system or a safe driving support wireless system.

[0011] The observation device 10 may be a device that observes an observation target such as a vehicle, object, or person on a road in a predetermined area. The observation device 10 may be, for example, a roadside device or a surveillance camera device. In this embodiment, the observation device 10 is a roadside device that is placed near an intersection where multiple roads 12 (roadways) intersect and observes the road surface. The observation device 10 may also be placed on the side of the road other than at an intersection.

[0012] In the communication system 11, the observation device 10 and a moving object 13, such as a car traveling on a road 12, may communicate wirelessly with each other. Multiple moving objects 13 may communicate wirelessly with each other. In the communication system 11, wireless communication may be performed including with an electronic device carried by a pedestrian 14. The electronic device may be, for example, a mobile terminal device such as a smartphone.

[0013] The observation device 10 may notify the mobile body 13 of support information to support the driver of the mobile body 13 in safe driving. The support information may include information about an object present on the road being observed. The information about the object may include whether or not the object is present, the type of object present, the distance to the observation device 10, the speed, and the direction of travel. The support information may include information about the lighting status of the traffic lights 15, information about road regulations, road alignment information indicating the shape of the intersection where the observation device 10 is located (the shape of the road 12), etc. The support information may include information notified from other mobile bodies 13. The observation device 10 may notify the support information to an electronic device carried by a pedestrian 14.

[0014] The mobile object 13 detects its spatial position in real space using a positioning device such as a Global Navigation Satellite System (GNSS). The spatial position of the mobile object 13 may be a position in a world coordinate system. The world coordinate system is a coordinate system set in a three-dimensional space defined by the GNSS.

[0015] The moving object 13 may periodically transmit moving object information including at least location information regarding the spatial position of the moving object 13 to the observation device 10, for example. The location information regarding the spatial position may be not only the spatial position itself but also other information that can identify the spatial position. The moving object information may further include information regarding the speed, blinkers, etc.

[0016] Mobile body 13 may acquire various information notified from observation device 10 or the like using an electronic device mounted thereon. The electronic device mounted on mobile body 13 may be, for example, a car navigation system. The electronic device mounted on mobile body 13 may support the driver in safe driving by notifying the driver of warnings or other notifications based on the support information notified from observation device 10. Notifications to the driver may include the type, position, speed, direction of travel, etc. of observed objects such as other mobile bodies 13 near the intersection where the driver is traveling, as well as the presence of pedestrians 14 on crosswalks 16 near the intersection.

[0017] As described above, the communication system 11 may assist the driver of the mobile object 13 in safe driving. The mobile object 13 is not limited to an automobile. For example, the mobile object 13 may include a motorcycle, a bus, a streetcar, or a bicycle.

[0018] The observation device 10 will be described in detail below. The observation device 10 estimates the distance to an observation target in a predetermined area. Here, the distance estimated by the observation device 10 is a distance acquired based on information indicating the relationship between the image position on an image acquired by the observation device 10 and information relating to the distance to the observation target, as will be described later. The distance to the observation target may be the distance from an arbitrary origin in real space. The arbitrary origin is, for example, the road surface at the position of the observation device 10 when viewing the three-dimensional real space from vertically above.

[0019] 2, the observation device 10 includes an imaging unit 17, a storage unit 18, a communication unit (acquisition unit) 19, and a control unit 20. The observation device 10 may further include a network IF 21.

[0020] The imaging unit 17 generates an image by capturing an image of a space including the predetermined area. The imaging unit 17 is, for example, a monocular camera. The monocular camera may be a visible light camera or an FIR camera. The imaging unit 17 may generate images at a predetermined frame rate, such as 30 fps.

[0021] A camera coordinate system may be defined for the imaging unit 17. The camera coordinate system may be a three-dimensional coordinate system with an origin at any position in the imaging unit 17 and three mutually perpendicular axes passing through the origin as coordinate axes. The origin of the camera coordinate system is, for example, the intersection of the detection axis of the imaging unit 17 and the imaging element. The camera coordinate system includes, for example, the optical axis as one of its coordinate axes. The optical axis may pass through the center of the detection range of the imaging element.

[0022] 3, the observation device 10 may be fixed to a structure 22 that is tall enough to capture an outdoor scene including the road 12, such as a traffic light, utility pole, or streetlight near an intersection where the road 12 to be observed intersects. The position and orientation of the image capture unit 17 relative to the structure 22 may be determined in advance.

[0023] The position of the imaging unit 17 means the origin of the camera coordinate system relative to a reference position in a surrounding space coordinate system defined around the observation device 10. The attitude of the imaging unit 17 means the inclination of the coordinate axes of the camera coordinate system relative to the coordinate axes of the surrounding space coordinate system. The surrounding space coordinate system may have axes in the vertical direction of the real space and in two directions parallel to the horizontal plane and perpendicular to each other.

[0024] The position and orientation of the imaging unit 17 may be determined so that the optical axis ox of the imaging unit 17 intersects with the surface of the road 12. The position and orientation of the imaging unit 17 may be determined so that the detection range of the imaging unit 17 includes a predetermined area defined for the observation device 10.

[0025] 2, the storage unit 18 includes any storage device, such as a RAM (Random Access Memory) and a ROM (Read Only Memory), etc. The storage unit 18 may store various programs that cause the control unit 20 to function and various information that the control unit 20 uses.

[0026] The memory unit 18 stores distance information. The distance information is information indicating the relationship between an image position on an image captured by the imaging unit 17 and information regarding the distance to the observation target. Specifically, the image position on the image may be the coordinate of a pixel in a two-dimensional image coordinate system when the pixel constituting the image acquired from the imaging unit 17 indicates an object on the road surface or floor surface. Specifically, the information regarding the distance to the observation target may be the distance from an arbitrarily determined origin in the surrounding space coordinate system. The origin in the surrounding space coordinate system is the road surface at the position of the observation device 10 when viewing the three-dimensional real space from vertically above.

[0027] The distance information may be a conversion formula or a conversion table for converting information on the distance from an image position on an image. The conversion formula or the conversion table may be created in advance based on the characteristics of the image sensor and optical system in the image capturing unit 17, and the position and orientation of the image capturing unit 17 relative to the road 12, and may further be calibrated using a target placed on the road surface, etc.

[0028] The storage unit 18 may store the origin position of the surrounding space coordinate system in the world coordinate system. The storage unit 18 may store the orientation of the surrounding space coordinate system in the world coordinate system. The orientation of the surrounding space coordinate system is the inclination of the coordinate axes of the surrounding space coordinate system from the coordinate axes of the world coordinate system. The origin position and orientation of the surrounding space coordinate system in the world coordinate system may be measured when the observation device 10 is installed and stored in the storage unit 18 by, for example, inputting it using an input device.

[0029] The communication unit 19 may be controlled by the control unit 20 to perform wireless communication with the mobile object 13. The communication unit 19 may be configured with a communication circuit and an antenna. The antenna may be, for example, an omnidirectional antenna. The communication unit 19 may perform wireless communication using, for example, the 700 MHz band allocated to ITS. The communication unit 19 may also perform wireless communication using, for example, a wireless LAN (Local Area Network).

[0030] The communication unit 19 may perform various processes such as amplification on the signal received by the antenna, and output the processed received signal to the control unit 20. The communication unit 19 may acquire, for example, information on the spatial position of the mobile object 13 from the mobile object 13. The control unit 20 may perform various processes on the input received signal to acquire information contained in the received signal. The communication unit 19 may perform various processes such as amplification on the information acquired from the control unit 20, and wirelessly transmit the processed transmission signal from the antenna.

[0031] The control unit 20 includes one or more processors and memories. The processor may include a general-purpose processor that loads a specific program to execute a specific function, and a dedicated processor specialized for a specific process. The dedicated processor may include an application-specific integrated circuit (ASIC). The processor may include a programmable logic device (PLD). The PLD may include a field-programmable gate array (FPGA). The control unit 20 may be either a system-on-a-chip (SoC) or a system in a package (SiP) in which one or more processors work together.

[0032] The control unit 20 may determine whether or not an observation target exists in a predetermined area based on an image acquired from the imaging unit 17. Specifically, the control unit 20 detects a subject image corresponding to the observation target subject by known image recognition such as pattern matching on the image or deep learning. The control unit 20 may determine whether or not an observation target exists by detecting the subject image.

[0033] When an observation target is present, the control unit 20 may identify the type of the observation target based on the observation results of the imaging unit 17. Specifically, the control unit 20 identifies the type of the observation target by detecting the subject image and recognizing the image.

[0034] When an observation target is present, the control unit 20 may acquire an estimated distance to the position where the observation target is present based on the image. The estimated distance is acquired, for example, by calculation by the control unit 20. Specifically, the control unit 20 may calculate the estimated distance to the observation target by converting the image position in a two-dimensional coordinate system of pixels of a part of the detected subject image that is in contact with the road surface into a surrounding space coordinate system using distance information.

[0035] The control unit 20 may identify the speed and traveling direction of the detected observation target based on a plurality of detection signals detected at successive times.

[0036] When at least one observation target is present, the control unit 20 may generate support information including the type of the observation target, the estimated distance, the speed, and the traveling direction. The control unit 20 may control the communication unit 19 to notify the generated support information to the mobile object 13 around the observation device 10.

[0037] When acquiring moving body information from the moving body 13, the control unit 20 may correct the distance information as described below. In correcting the distance information, the control unit 20 may extract position information of the moving body 13 included in the moving body information. The control unit 20 calculates the actual measured distance of the moving body 13 based on the position information of the moving body 13. Specifically, the control unit 20 first converts the position information of the moving body 13 corresponding to coordinates in the world coordinate system into position information of the surrounding space coordinate system based on the origin position and orientation of the surrounding space coordinate system in the world coordinate system. Next, the control unit 20 calculates the distance from the origin of the surrounding space coordinate system to the position of the moving body 13 in the surrounding space coordinate system as the actual measured distance.

[0038] Furthermore, the control unit 20 may detect a subject image of the moving body 13 from an image taken approximately at the same time as the moving body information by determining the presence or absence of the observation target and identifying the type of the observation target. An image taken approximately at the same time as the moving body information is, for example, an image taken at a time closest to the time when the spatial position of the moving body 13 was detected, among images taken continuously at a predetermined frame rate. However, strictly speaking, an image taken approximately at the same time as the moving body information does not need to be limited to an image taken at a time closest to the time when the spatial position of the moving body 13 was detected. An image taken approximately at the same time as the moving body information may be, for example, an image that can be considered to be an image taken at a time closest to the time when the spatial position of the moving body 13 was detected, such as an image acquired at the same time as the moving body information.

[0039] The control unit 20 calculates an estimated distance to the moving object 13 corresponding to the detected subject image based on the image. The control unit 20 may associate the calculated actual distance with the calculated estimated distance. When a single subject image of the moving object 13 is detected in a single image, the control unit 20 may associate the calculated estimated distance with the calculated actual distance. When multiple subject images of the moving object 13 are detected in a single image, the control unit 20 may stop associating the estimated distance with the calculated actual distance, or may stop correcting the distance information, or associate the estimated distance with the smallest difference from the actual distance with the actual distance.

[0040] The control unit 20 calculates a distance deviation, which is the difference between the associated measured distance and estimated distance. The control unit 20 compares the distance deviation with a first threshold value. The first threshold value is an allowable error when the estimated distance is included as support information. The first threshold value may be set manually or automatically based on the speed limit of the road 12 observed by the observation device 10, etc.

[0041] The control unit 20 may store in the memory unit 18 the calculated distance deviation, the measured distance from which the distance deviation was calculated, and the image position corresponding to the estimated distance from which the distance deviation was calculated, in association with each other.

[0042] As shown in Figure 4, when calculating the estimated distance using images from a monocular camera, the correspondence is determined on the assumption that the road surface, such as a road 12, in real space forms a single virtual plane vp. The image position corresponds to the direction of the observation device 10. The intersection of the virtual plane vp with a line extending from the observation device 10 in each direction corresponding to each image position corresponds to the estimated distance. Furthermore, the intersection of the real road surface rp of the actual road 12 with a line extending from the observation device 10 in each direction corresponding to each image position corresponds to the measured distance. If the vertical height of the real road surface rp of the actual road 12 differs from the height of the virtual plane vp, a distance discrepancy will occur between the estimated distance and the measured distance.

[0043] For example, in a first direction dr1 in which the vertical heights of the virtual plane vp and the actual road surface rp are the same, the estimated distance dis_v1 and the measured distance dis_r1 are the same, and the distance error is zero. For example, in a second direction dr2 in which the vertical heights of the virtual plane vp and the actual road surface rp are the same, the estimated distance dis_v2 and the measured distance dis_r2 are the same, and the distance error is zero. For example, in a third direction dr3 in which the vertical heights of the virtual plane vp and the actual road surface rp are different, the estimated distance dis_v3 and the measured distance dis_r3 are different, and the distance error Δdis3 is equal to or greater than the first threshold. For example, in a fourth direction dr4 in which the vertical heights of the virtual plane vp and the actual road surface rp are different, the estimated distance dis_v4 and the measured distance dis_r4 are different, and the distance error Δdis4 is equal to or greater than the first threshold. For example, in a fifth direction dr5 in which the vertical heights of the virtual plane vp and the actual road surface rp are different, the estimated distance dis_v5 and the measured distance dis_r5 are different, and the distance deviation Δdis5 is equal to or greater than the first threshold.

[0044] When the distance deviation at the point where the estimated distance was acquired is equal to or greater than a first threshold, the control unit 20 corrects the distance information using the measured distance. When the distance information is a conversion table, the correction may be performed by replacing information on the distance corresponding to the image position on the image corresponding to the point where the estimated distance was acquired in the conversion table with the measured distance. For example, when the control unit 20 calculates a distance deviation Δdis2 corresponding to the second direction dr2, the control unit 20 replaces information on the distance in the conversion table at the image position on the image corresponding to the second direction dr2 with the value of the measured distance. When the distance information is a conversion formula, the conversion formula may be corrected so that the measured distance is calculated at the image position where the estimated distance was calculated.

[0045] When a distance deviation is newly calculated, the control unit 20 may determine whether the image position associated with the measured distance stored in the storage unit 18 differs from the image position corresponding to the newly calculated distance deviation. The image position corresponding to the distance deviation is the image position from which the estimated distance used to calculate the distance deviation is calculated. When the image positions differ, the control unit 20 may determine whether at least one of the newly calculated distance deviation and the distance deviation corresponding to the stored measured distance is equal to or greater than a first threshold.

[0046] When at least one of the distances is equal to or greater than the first threshold, the control unit 20 may correct information about the distance in the distance information corresponding to an inclusive image position between two different image positions using two measured distances corresponding to the two image positions. The inclusive image position is a position on the image between the two different image positions. That is, it is a position on the image acquired by the imaging unit 17 at a point between the coordinates of the surrounding space coordinate system corresponding to the two image positions. Specifically, the control unit 20 may correct information about the distance corresponding to the inclusive image position in the distance information based on the difference between the two measured distances corresponding to the two image positions and the difference between the two estimated distances, the shorter measured distance of the two measured distances, and the difference between the shorter measured distance and the estimated distance corresponding to the inclusive image position.

[0047] For example, information about the distance defined by the distance information for an inclusive image position sandwiched between two positions, an image position corresponding to the second direction dr2 and an image position corresponding to the third direction dr3, is corrected using two measured distances corresponding to the second direction dr2 and the third direction dr3. The information about the distance for the inclusive image position may be distributed so that the image position corresponding to the second direction dr2 and the image position corresponding to the third direction dr3 each match the measured distance. For example, information about the distance for each inclusive image position may be set according to the distance in the image of a plurality of inclusive image positions (e.g., the distance between pixels).

[0048] For example, after storing the measured distance corresponding to the second direction dr2 in the memory unit 18, when the distance deviation Δdis3 corresponding to the third direction dr3 is calculated, the control unit 20 replaces the distance information dis_vx in the distance information corresponding to any direction between the second direction dr2 and the third direction dr3 with the corrected distance cdis_vx calculated using equation (1).

[0049]

number

[0050] When the distance information is a conversion table, the correction may be performed by replacing the distance information in the conversion table for image positions corresponding to any of a plurality of directions between the first direction dr1 and the second direction dr2 with the corrected distance cdis_vx according to the above formula. Alternatively, when the distance information is a conversion formula, the original conversion formula may be corrected by incorporating formula (1) into the original conversion formula in the range between the first direction dr1 and the second direction dr2.

[0051] When multiple measured distances are stored in the memory unit 18 and a new distance deviation is calculated, the control unit 20 may correct the distance information corresponding to the aforementioned included image position using the measured distance among the multiple measured distances that has the smallest difference from the measured distance corresponding to the newly calculated distance deviation.

[0052] For example, when the control unit 20 stores the measured distances dis_r2 and dis_r3 in the memory unit 18 and then calculates a new distance shift Δdis4, it corrects the distance information of the image position corresponding to any direction sandwiched between the third direction dr3 and the fourth direction dr4 using the measured distance dis_r3 that has the smallest difference from the measured distance dis_r4 corresponding to the distance shift Δdis4.

[0053] When there are multiple image positions where the distance deviation is less than the first threshold, the control unit 20 may correct the distance information corresponding to the included image position using the image position closest to the image position associated with the distance deviation that is greater than or equal to the first threshold.

[0054] For example, when the control unit 20 stores the measured distances dis_r1 and dis_r2 in the memory unit 18 and then newly calculates the distance shift Δdis3, it corrects the distance information of the image position corresponding to any direction sandwiched between the second direction dr2 and the third direction dr3 using the measured distance dis_r2 corresponding to the second direction dr2, which is the image position closest to the image position corresponding to the distance shift Δdis4.

[0055] If the distance deviation is equal to or greater than a second threshold, the control unit 20 may not correct the distance information, but may instead generate warning information to warn of an abnormality in the observation device 10. The second threshold may be greater than the first threshold. The control unit 20 may control the network IF 21 to send the warning information to a server of a user who maintains the observation device 10.

[0056] The control unit 20 may be able to switch the operation mode to at least an initial setting mode (initial setting) or a normal mode. The control unit 20 may correct the distance information described above in either the initial setting mode or the normal mode. The control unit 20 may be switched to the initial setting mode by input using an input device, for example, when the observation device 10 is installed. The control unit 20 may be switched to the normal mode after the initial setting mode, which will be described later, has ended.

[0057] The control unit 20 may correct the distance information described above in the initial setting mode. The control unit 20 may correct the distance information when a single subject image of the moving object 13 is detected in a single image in the initial setting mode. The control unit 20 may not perform association and distance information correction when multiple subject images of the moving object 13 are detected in a single image in the initial setting mode. The control unit 20 may correct the distance information without generating warning information when the distance deviation is equal to or greater than the second threshold in the initial setting mode.

[0058] When the distance information is sufficiently corrected in the initial setting mode, the control unit 20 may terminate the initial setting mode and transition to the normal mode. The condition that the distance information is sufficiently corrected may be, for example, that moving body information has been acquired from 10 moving bodies 13, in other words, that moving body information has been acquired multiple times from each of the 10 moving bodies 13.

[0059] The network IF 21 may include a communication module that connects to an external network. The observation device 10 may be connected to the external network via the network IF 21 and may communicate information with, for example, an external server.

[0060] Next, the correction process in the initial setting mode executed by the control unit 20 in this embodiment will be described using the flowcharts in Figures 5 and 6. The correction process in the initial setting mode is started by switching the operation mode using an input device after the observation device 10 is installed.

[0061] In step S100, the control unit 20 determines whether or not the mobile object information has been acquired. If not, the process returns to step S100. If acquired, the process proceeds to step S101.

[0062] In step S101, the control unit 20 extracts the position information from the moving body information determined to have been acquired in step S100. After the extraction, the process proceeds to step S102.

[0063] In step S102, the control unit 20 calculates the actual distance based on the position information extracted in step S101. After calculating the actual distance, the process proceeds to step S103.

[0064] In step S103, the control unit 20 detects a subject image of the moving object 13 from images taken at approximately the same time as the moving object information determined to have been acquired in step S100. After the subject image is detected, the process proceeds to step S104.

[0065] In step S104, the control unit 20 determines whether or not subject images of multiple moving objects 13 were detected in step S103. If subject images of multiple moving objects 13 were detected, the process returns to step S100. If subject images of multiple moving objects 13 were not detected but a single moving object 13 was detected, the process proceeds to step S105.

[0066] In step S105, the control unit 20 calculates an estimated distance from the image position of the subject image detected in step S103 using the distance information stored in the storage unit 18. After calculating the estimated distance, the process proceeds to step S106. Note that at the start of the initial setting mode, the storage unit 18 stores distance information that was generated on the assumption that the road surface observed by the observation device 10 is flat.

[0067] In step S106, the control unit 20 calculates a distance deviation as the difference between the actual distance calculated in step S102 and the estimated distance calculated in step S105. After calculating the distance deviation, the process proceeds to step S107.

[0068] In step S107, control unit 20 associates the actual distance calculated in step S102, the image position at which the estimated distance calculated in step S105 was calculated, and the distance deviation calculated in step S106. Furthermore, control unit 20 associates the actual distance, image position, and distance deviation with each other and stores them in storage unit 18. After the association and storage are completed, the process proceeds to step S108.

[0069] In step S108, the control unit 20 determines whether the distance deviation calculated in step S106 is equal to or greater than a first threshold. If it is equal to or greater than the first threshold, the process proceeds to step S109. If it is not equal to or greater than the first threshold, the process proceeds to step S110.

[0070] In step S109, the control unit 20 corrects the distance information corresponding to the specific image position based on the measured distance calculated in step S102. The specific image position is the image position of the subject image of the moving object 13 detected in step S103. After correcting the distance information, the process proceeds to step S110.

[0071] In step S110, the control unit 20 determines whether the image position stored in the storage unit 18 is different from the image position associated in step S107. If they are different, the process proceeds to step S111. If they are the same, the process proceeds to step S116.

[0072] In step S111, the control unit 20 determines whether there are multiple image positions stored in the storage unit 18 that are different from the image position associated in step S107. If there is only one image position, the process proceeds to step S112. If there are multiple image positions, the process proceeds to step S113.

[0073] In step S112, the control unit 20 associates the image position associated with the measured distance stored in the storage unit 18 with the image position associated in step S107. After the association, the process proceeds to step S114.

[0074] In step S113, the control unit 20 associates the image position associated with the actual distance that has the smallest difference from the actual distance calculated in step S107, among the multiple actual distances stored in the storage unit 18, with the image position associated in step S107. After the association, the process proceeds to step S114.

[0075] In step S114, the control unit 20 determines whether at least one of the distance deviations corresponding to the image positions associated in either step S112 or step S113 is equal to or greater than a first threshold. If at least one of the distance deviations is equal to or greater than the first threshold, the process proceeds to step S115. If both distance deviations are not equal to or greater than the first threshold, the process proceeds to step S116.

[0076] In step S115, the control unit 20 corrects the distance information corresponding to the specific range of image positions based on the measured distances corresponding to the image positions associated in step S112 or step S113. The specific range of image positions is the range sandwiched between the image positions associated in step S112 or step S113. After correcting the distance information, the process proceeds to step S116.

[0077] In step S116, the control unit 20 determines whether or not mobile object information has been acquired from the ten mobile objects 13. If not, the process returns to step S100. If acquired, the correction process in the initial setting mode ends.

[0078] Next, the correction process in the normal mode executed by the control unit 20 in this embodiment will be described using the flowcharts in Figures 7 and 8. The correction process in the normal mode starts when mobile object information is acquired after the initial setting mode ends.

[0079] In step S200, the control unit 20 extracts the location information from the newly acquired moving body information. After extraction, the process proceeds to step S201.

[0080] In steps S201 and S202, the control unit 20 performs the same control as in steps S102 and S103 of the correction process in the initial setting mode. After the subject image is detected in step S202, the process proceeds to step S203.

[0081] In step S203, the control unit 20 calculates an estimated distance from the image position of the subject image detected in step S202, using the distance information stored in the storage unit 18. After calculating the estimated distance, the process proceeds to step S204.

[0082] In step S204, the control unit 20 determines whether or not subject images of multiple moving objects 13 were detected in step S202. If subject images of multiple moving objects 13 were not detected but a single moving object 13 was detected, the process proceeds to step S205. If subject images of multiple moving objects 13 were detected, the process proceeds to step S206.

[0083] In step S205, the control unit 20 associates the actual distance calculated in step S201 with the estimated distance calculated in step S203. After the association, the process proceeds to step S207.

[0084] In step S206, the control unit 20 associates the estimated distance, among the estimated distances calculated in step S203 for each of the multiple subject images of the moving object 13 detected in step S202, with the actual measured distance calculated in step S201, which has the smallest difference from the estimated distance. After the association, the process proceeds to step S207.

[0085] In step S207, control unit 20 calculates a distance shift as the difference between the actual distance and the estimated distance associated in step S205 or step S206. Furthermore, control unit 20 associates the actual distance calculated in step S201, the image position at which the estimated distance associated with the actual distance was calculated in step S205 or S206, and the calculated distance shift in memory unit 18. After calculating and storing the distance shift, the process proceeds to step S208.

[0086] In step S208, the control unit 20 determines whether the distance deviation calculated in step S207 is equal to or greater than a second threshold. If it is equal to or greater than the second threshold, the process proceeds to step S209. If it is not equal to or greater than the second threshold, the process proceeds to step S210.

[0087] In step S209, the control unit 20 generates warning information. Furthermore, the control unit 20 controls the network IF 21 to transmit the generated warning information to an external server. After the generation and transmission, the correction process in the normal mode ends.

[0088] In step S210, the control unit 20 determines whether the distance deviation calculated in step S207 is equal to or greater than a first threshold. If it is equal to or greater than the first threshold, the process proceeds to step S211. If it is not equal to or greater than the first threshold, the process proceeds to step S212.

[0089] In step S211, the control unit 20 corrects the distance information corresponding to the specific image position based on the measured distance calculated in step S201. The specific image position is the image position of the subject image of the moving object 13 detected in step S202. After correcting the distance information, the process proceeds to step S212.

[0090] In step S212, the control unit 20 determines whether the image position stored in the storage unit 18 is different from the image position associated in step S205 or step S206. If they are the same, the process proceeds to step S213. If they are different, the correction process in the normal mode ends.

[0091] In step S213, the control unit 20 determines whether there are multiple image positions stored in the storage unit 18 that are different from the image positions associated in step S205 or step S206. If there is only one image position, the process proceeds to step S214. If there are multiple image positions, the process proceeds to step S215.

[0092] In step S214, the control unit 20 associates the image position associated with the measured distance stored in the storage unit 18 with the image position associated in step S205 or step S206. After the association, the process proceeds to step S216.

[0093] In step S215, the control unit 20 associates the image position associated with the actual distance that has the smallest difference from the actual distance calculated in step S205 or S206, among the multiple actual distances stored in the storage unit 18, with the image position associated in step S205 or S206. After the association, the process proceeds to step S216.

[0094] In step S216, the control unit 20 determines whether at least one of the distance deviations corresponding to the image positions associated in either step S214 or step S215 is equal to or greater than a first threshold. If at least one of the distance deviations is equal to or greater than the first threshold, the process proceeds to step S217. If both distance deviations are not equal to or greater than the first threshold, the correction process in normal mode ends.

[0095] In step S216, the control unit 20 corrects the distance information corresponding to the specific range of image positions based on the measured distances corresponding to the image positions associated in step S214 or step S215. The specific range of image positions is the range sandwiched between the image positions associated in step S214 or step S215. After correcting the distance information, the correction process in normal mode ends.

[0096] The observation device 10 of this embodiment, configured as described above, includes a memory unit 18 that stores distance information, which is the relationship between image position on an image captured by the imaging unit 17 and information related to distance; a communication unit (acquisition unit) 19 that acquires position information related to the spatial position of the moving object 13 from the moving object 13; and a control unit 20 that corrects the distance information using the measured distance when the distance discrepancy between the measured distance corresponding to the position information and the estimated distance estimated based on the image position of the moving object 13 in the image and the distance information is equal to or greater than a first threshold. With this configuration, the observation device 10 can calibrate differences in the shape of the actual road surface rp in the vertical direction due to factors such as the gradient. Therefore, the observation device 10 can improve the accuracy of distance estimation.

[0097] Furthermore, when the distance deviation between at least one of two different image positions is equal to or greater than a first threshold, the observation device 10 of this embodiment corrects the distance information corresponding to the image position contained between the two image positions using the two measured distances corresponding to the two image positions. With this configuration, the observation device 10 can calibrate the difference in the shape of the actual road surface rp not only for the two image positions corresponding to the calculated two measured distances, but also for the area between the two image positions. Therefore, the observation device 10 can improve the distance estimation accuracy over a wide range.

[0098] Furthermore, the observation device 10 of this embodiment corrects distance information corresponding to an inclusive image position using an image position closest to an image position where the distance deviation is equal to or greater than the first threshold, among image positions where the distance deviation is less than the first threshold. Correcting distance information corresponding to an image position where the distance deviation is less than the first threshold with a measured distance corresponding to another image position where the distance deviation is equal to or greater than the first threshold reduces the accuracy of distance estimation calculated from the image position where the distance deviation is less than the first threshold. In response to such an event, the observation device 10 having the above-described configuration can reduce image positions where the distance deviation is less than the first threshold from the range of inclusive image positions corrected with a measured distance corresponding to another image position where the distance deviation is equal to or greater than the first threshold. Therefore, the observation device 10 can reduce the range in which the accuracy of distance estimation decreases.

[0099] Furthermore, when multiple measured distances are stored in the storage unit 18 and a new distance deviation is calculated, the observation device 10 of this embodiment corrects the distance information corresponding to the aforementioned embodied image position using the measured distance that has the smallest difference from the newly calculated measured distance corresponding to the distance deviation. For example, as shown in FIG. 9, if eight measured distances dis_r are calculated for the actual road surface rp, and the measured distances are arranged in ascending order and the distance information is corrected using the measured distances at every other two locations, the distance is estimated based on the first virtual plane vp1, which has a relatively large distance deviation from the actual road surface rp. On the other hand, the observation device 10 having the above configuration can estimate the distance based on the second virtual plane vp2, which has a smaller distance deviation from the actual road surface rp than the first virtual plane vp1, as shown in FIG. 10. Therefore, the observation device 10 can further improve the distance estimation accuracy.

[0100] Furthermore, when an image includes multiple images of the moving object 13, the observation device 10 of this embodiment corrects distance information based on the actual measured distance and the estimated distance that has the smallest difference from the actual measured distance among the estimated distances corresponding to the multiple images of the moving object 13. With this configuration, when multiple subject images of the moving object 13 are detected from an image, the observation device 10 can appropriately determine which subject image to use to correct the distance information. Therefore, the observation device 10 can correct distance information even when multiple subject images of the moving object 13 are detected from an image.

[0101] Furthermore, when the distance deviation is equal to or greater than a second threshold value that is greater than the first threshold value, the observation device 10 of this embodiment generates warning information that warns of an abnormality in the observation device 10. With this configuration, the observation device 10 can warn of a state that is assumed to be an abnormality in the observation device 10 itself or an abnormality in the installation of the observation device 10.

[0102] Furthermore, if the distance deviation is equal to or greater than the second threshold during initial setup, the observation device 10 of this embodiment stops generating warning information and corrects the distance information. When correcting the distance information during initial setup, the distance deviation may become relatively large. In the event of such an event, the observation device 10 does not determine that an abnormality has occurred in the observation device 10 during initial setup, thereby preventing unnecessary warnings.

[0103] Although the embodiments of the present disclosure have been described based on the drawings and examples, it should be noted that those skilled in the art would easily be able to make various modifications or alterations based on the present disclosure. Therefore, it should be noted that these modifications and alterations are included within the scope of the present disclosure. For example, the functions included in each component or step can be rearranged so as not to be logically inconsistent, and multiple components or steps can be combined or divided into one.

[0104] In this disclosure, descriptions such as "first" and "second" are identifiers for distinguishing the configuration. In this disclosure, configurations distinguished by descriptions such as "first" and "second" can have their numbers swapped. For example, the first direction can have its identifiers "first" and "second" swapped with the second direction. The identifier swapping is performed simultaneously. The configurations remain distinguished even after the identifier swapping. Identifiers may be deleted. A configuration from which an identifier has been deleted is distinguished by a symbol. The identifiers "first" and "second" used in this disclosure should not be used solely to interpret the order of the configurations or to justify the existence of an identifier with a smaller number. [Explanation of symbols]

[0105] 10 Observation equipment 11. Communication Systems 12 Road 13 Mobile 14 Pedestrians 15 Traffic Lights 16. Crosswalk 17 Imaging unit 18 Memory section 19 Acquisition Department 20 Control Unit 21 Communications Department 22 Structures dr1 First direction dr2 Second direction dr3 Third direction dr4 fourth direction rp actual road surface ox optical axis vp virtual plane

Claims

1. An observation device, a storage unit that stores distance information that is a relationship between an image position on an image generated by capturing an image of a space including a predetermined area where an observation target exists and information about a distance to the observation target; a control unit that corrects the distance information corresponding to an included image position between the two different image positions using two measured distances corresponding to the two image positions, The control unit corrects the distance information corresponding to the included image position when there is a distance deviation between at least one of information on the distance to the observation device corresponding to the two image positions and actual measured distances corresponding to the two image positions. Observation equipment.

2. The control unit corrects the distance information corresponding to the included image position when the distance deviation is equal to or greater than a first threshold. The observation device according to claim 1 .

Images