Observation device

Abstract

To improve estimation accuracy of a distance.SOLUTION: An observation device 10 comprises: an imaging unit 17; a storage unit 18; an acquisition unit 19; and a control unit 20. The imaging unit 17 generates an image. The storage unit 18 stores distance information. The distance information is the relation between the image position and the information about the distance. The acquisition unit 19 acquires the position information about the spatial position of a movable body. The control unit 20 corrects the distance information using the actual measurement distance when a distance deviation between the actual measurement distance and the estimated distance is equal to or greater than a first threshold. The actual measurement distance corresponds to the position information. The estimated distance is acquired on the basis of the image position of the movable body in the image and the distance information.SELECTED DRAWING: Figure 2

Description

[Technical field]

[0001] The present invention relates to an observation device. [Background technology]

[0002] There is known a sensor that generates an image by capturing an image of a predetermined area, like a camera. Various information can be generated from the image. For example, it has been proposed to calculate the distance in real space to a subject corresponding to the subject image on the ground based on the mounting attitude with respect to the ground such as a road and the position of the subject image on the ground in the image (see Patent Document 1). [Prior art documents] [Patent documents]

[0003] [Patent Document 1] Japanese Patent Application Publication No. 03-273500 Summary of the Invention [Problem to be solved by the invention]

[0004] For example, it is being considered to apply a camera that generates images to an observation device installed on the roadside. Before starting to use the observation device, the distance information for the position on the image captured by the camera can be calibrated by performing calibration using markers placed at predetermined distances from the observation device. However, with conventional calibration, it is difficult to calibrate the deviation of the distance information caused by the shape of the ground such as undulations and changes in the mounting position over time, and the distance estimation accuracy may be low.

[0005] Therefore, an object of the present disclosure, made in consideration of the problems with the conventional technology as described above, is to provide an observation device that improves the accuracy of distance estimation. [Means for solving the problem]

[0006] In order to solve the above-mentioned problems, an observation apparatus according to a first aspect of the present invention comprises: An observation device that estimates a distance to an observation target in a predetermined area, comprising: an imaging unit that generates an image by imaging a space including the predetermined area; a storage unit that stores distance information that is a relationship between an image position on an image captured by the imaging unit and information about the distance; an acquisition unit that acquires, from a moving object, position information related to a spatial position of the moving object; The device is equipped with a control unit that corrects the distance information using the actual measured distance when a distance deviation between an actual measured distance corresponding to the position information and an estimated distance obtained based on the image position of the moving object in the image and the distance information is greater than or equal to a first threshold value. Effect of the Invention

[0007] According to the observation device according to the present disclosure configured as above, the accuracy of distance estimation is improved. [Brief description of the drawings]

[0008] [Figure 1] FIG. 1 is a diagram showing an example of the configuration of a communication system including an observation device according to an embodiment. [Diagram 2] FIG. 2 is a block diagram showing a schematic configuration of the observation device in FIG. [Diagram 3] FIG. 2 is a diagram showing a fixed state of the observation device in FIG. [Figure 4] 1 is a state diagram for explaining how a distance deviation occurs due to a deviation between an actual road surface and a virtual plane. [Diagram 5] 3 is a first flowchart for explaining a correction process in an initial setting mode executed by the control unit in FIG. 2; [Figure 6] 10 is a second flowchart for explaining the correction process in the initial setting mode executed by the control unit in FIG. 2; [Figure 7] 3 is a first flowchart for explaining a correction process in a normal mode executed by the control unit in FIG. 2; [Figure 8] 10 is a second flowchart for explaining the correction process in the normal mode executed by the control unit in FIG. [Figure 9] This figure shows the distance estimated based on the corrected distance information when the distance information corresponding to the included image position is corrected using an actual measured distance among multiple actual measured distances that has a non-smallest difference from the actual measured distance corresponding to the newly calculated distance shift. [Figure 10] This figure shows the distance estimated based on the corrected distance information when the distance information corresponding to the encapsulated image position is corrected using the actual measured distance among multiple actual measured distances that has the smallest difference from the actual measured distance corresponding to the newly calculated distance shift. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0009] Hereinafter, an embodiment of an observation device to which the present disclosure is applied will be described with reference to the drawings.

[0010] 1 shows an example of the configuration of a communication system 11 including an observation device 10 according to an embodiment. The communication system 11 is, for example, a safe driving support communication system of an intelligent transport system (ITS). The safe driving support communication system is also called a safe driving support system or a safe driving support wireless system.

[0011] The observation device 10 may be a device that observes an observation target such as a vehicle, an object, or a person on a road in a predetermined area. The observation device 10 is, for example, a roadside device, a surveillance camera device, or the like. In this embodiment, the observation device 10 is a roadside device that may be placed near an intersection where multiple roads 12 (carriages) intersect and observe the road surface. The observation device 10 may be placed on the roadside other than at the intersection.

[0012] In the communication system 11, the observation device 10 and a moving object 13, such as a car traveling on a road 12, may perform wireless communication with each other. A plurality of moving objects 13 may perform wireless communication with each other. In the communication system 11, wireless communication may be performed including an electronic device carried by a pedestrian 14. The electronic device is, for example, a mobile terminal device such as a smartphone.

[0013] The observation device 10 may notify the mobile body 13 of support information for supporting the driver of the mobile body 13 in safe driving. The support information may include information about an observation target present on the road being observed. The information about the observation target may include the presence or absence of the observation target, the type of observation target present, the distance to the observation device 10, the speed, and the direction of travel. The support information may include information about the illumination of the traffic lights 15, information about road regulations, road linear information indicating the shape of the intersection where the observation device 10 is located (the shape of the road 12), etc. The support information may include information notified from other mobile bodies 13. The observation device 10 may notify the support information to an electronic device carried by the pedestrian 14.

[0014] The moving object 13 detects its spatial position in real space using a positioning device such as a Global Navigation Satellite System (GNSS). The spatial position of the moving object 13 may be a position in a world coordinate system. The world coordinate system is a coordinate system set in a three-dimensional space defined by the GNSS.

[0015] The moving object 13 may transmit moving object information including at least position information related to the spatial position of the moving object 13 to the observation device 10, for example, periodically. The position information related to the spatial position may be not only the spatial position itself but also other information that can identify the spatial position. The moving object information may further include information related to the speed, blinkers, etc.

[0016] The mobile body 13 may acquire various information notified from the observation device 10 or the like by the electronic device mounted thereon. The electronic device mounted on the mobile body 13 may be, for example, a car navigation system. The electronic device mounted on the mobile body 13 may support the driver in safe driving by notifying the driver of a warning or the like based on the support information notified from the observation device 10. The notification to the driver may be the type, position, speed, traveling direction, etc. of observed objects such as other mobile bodies 13 near the intersection where the driver is proceeding, and the presence of pedestrians 14 on the crosswalk 16 near the intersection.

[0017] As described above, the communication system 11 may assist the driver of the mobile object 13 in driving safely. The mobile object 13 is not limited to an automobile. For example, the mobile object 13 may include a motorcycle, a bus, a streetcar, and a bicycle.

[0018] The observation device 10 will be described in detail below. The observation device 10 estimates the distance to an observation target in a predetermined area. Here, the distance estimated by the observation device 10 is a distance acquired based on information indicating the relationship between an image position on an image acquired by the observation device 10 and information on the distance to the observation target, as will be described later. The distance to the observation target may be a distance from an arbitrary origin in real space. The arbitrary origin is, for example, the road surface at the position of the observation device 10 when the three-dimensional real space is viewed vertically from above.

[0019] 2, the observation device 10 includes an imaging unit 17, a storage unit 18, a communication unit (acquisition unit) 19, and a control unit 20. The observation device 10 may further include a network IF21.

[0020] The imaging unit 17 generates an image by capturing an image of a space including the above-mentioned predetermined area. The imaging unit 17 is, for example, a monocular camera. The monocular camera may be a visible light camera or an FIR camera. The imaging unit 17 may generate an image at a predetermined frame rate such as 30 fps.

[0021] A camera coordinate system may be defined for the imaging unit 17. The camera coordinate system may be a three-dimensional coordinate system with an origin at any position in the imaging unit 17 and three mutually perpendicular axes passing through the origin. The origin of the camera coordinate system is, for example, the intersection of the detection axis of the imaging unit 17 and the imaging element. The camera coordinate system includes, for example, an optical axis as one of its coordinate axes. The optical axis may pass through the center of the detection range of the imaging element.

[0022] 3, the observation device 10 may be fixed to a structure 22 having a height capable of capturing an image of a scene including the road 12 outdoors, such as a traffic light, utility pole, or street light near an intersection where the road 12 to be observed intersects. The position and orientation of the imaging unit 17 with respect to the structure 22 may be determined in advance.

[0023] The position of the imaging unit 17 means the origin of the camera coordinate system relative to a reference position in an ambient space coordinate system defined around the observation device 10. The attitude of the imaging unit 17 means the inclination of the coordinate axes of the camera coordinate system relative to the coordinate axes of the ambient space coordinate system. The ambient space coordinate system may have axes along the vertical direction of the real space and two directions parallel to the horizontal plane and perpendicular to each other.

[0024] The position and orientation of the imaging unit 17 may be determined so that the optical axis ox of the imaging unit 17 intersects with the surface of the road 12. The position and orientation of the imaging unit 17 may be determined so that the detection range of the imaging unit 17 includes a predetermined area defined with respect to the observation device 10.

[0025] 2, the storage unit 18 includes any storage device such as a RAM (Random Access Memory) and a ROM (Read Only Memory). The storage unit 18 may store various programs that cause the control unit 20 to function and various information that the control unit 20 uses.

[0026] The storage unit 18 stores distance information. The distance information is information indicating the relationship between an image position on an image captured by the imaging unit 17 and information on the distance to the observation target. Specifically, the image position on the image may be the coordinates of a pixel constituting an image acquired from the imaging unit 17 in a two-dimensional image coordinate system when the pixel indicates an object on a road surface or floor surface. Specifically, the information on the distance to the observation target may be the distance from an origin that is arbitrarily determined in the surrounding space coordinate system. The origin in the surrounding space coordinate system is the road surface at the position of the observation device 10 when the three-dimensional real space is viewed vertically from above.

[0027] The distance information may be a conversion formula or a conversion table for converting information on the distance from an image position on an image. The conversion formula or the conversion table may be created in advance based on the characteristics of the imaging element and the optical system in the imaging unit 17, and the position and orientation of the imaging unit 17 with respect to the road 12, and may further be calibrated using a target or the like placed on the road surface.

[0028] The storage unit 18 may store the origin position of the surrounding space coordinate system in the world coordinate system. The storage unit 18 may store the orientation of the surrounding space coordinate system in the world coordinate system. The orientation of the surrounding space coordinate system is the inclination of the coordinate axes of the surrounding space coordinate system from the coordinate axes of the world coordinate system. The origin position and orientation of the surrounding space coordinate system in the world coordinate system may be measured when the observation device 10 is installed and stored in the storage unit 18 by, for example, inputting the same using an input device.

[0029] The communication unit 19 may be controlled by the control unit 20 to perform wireless communication with the mobile object 13. The communication unit 19 may be composed of a communication circuit and an antenna. The antenna may be, for example, an omnidirectional antenna. The communication unit 19 may perform wireless communication using, for example, the 700 MHz band allocated to ITS. The communication unit 19 may also perform wireless communication using, for example, a wireless LAN (Local Area Network).

[0030] The communication unit 19 may perform various processes such as amplification on the signal received by the antenna, and output the processed received signal to the control unit 20. The communication unit 19 may acquire information on the spatial position of the mobile body 13 from the mobile body 13, for example. The control unit 20 may perform various processes on the input received signal, and acquire information contained in the received signal. The communication unit 19 may perform various processes such as amplification on the information acquired from the control unit 20, and wirelessly transmit the processed transmission signal from the antenna.

[0031] The control unit 20 includes one or more processors and memories. The processor may include a general-purpose processor that loads a specific program to execute a specific function, and a dedicated processor specialized for a specific process. The dedicated processor may include an application specific integrated circuit (ASIC). The processor may include a programmable logic device (PLD). The PLD may include a field-programmable gate array (FPGA). The control unit 20 may be either a system-on-a-chip (SoC) or a system in a package (SiP) in which one or more processors work together.

[0032] The control unit 20 may determine whether or not an observation target exists in a predetermined area based on an image acquired from the imaging unit 17. Specifically, the control unit 20 detects a subject image corresponding to the subject of the observation target by known image recognition such as pattern matching for the image and deep learning. The control unit 20 may determine whether or not an observation target exists by detecting the subject image.

[0033] When an observation target is present, the control unit 20 may identify the type of the observation target based on the observation result of the imaging unit 17. Specifically, the control unit 20 identifies the type of the observation target by detecting the subject image and recognizing the image.

[0034] When an observation target exists, the control unit 20 may obtain an estimated distance to the position where the observation target exists based on the image. The estimated distance is obtained, for example, by calculation by the control unit 20. Specifically, the control unit 20 may calculate the estimated distance to the observation target by converting the image position in a two-dimensional coordinate system of the pixel of the part of the detected subject image that is in contact with the road surface into the surrounding space coordinate system using the distance information.

[0035] The control unit 20 may identify the speed and traveling direction of the detected observation target based on a plurality of detection signals detected at successive times.

[0036] When at least one observation target is present, the control unit 20 may generate support information including the type, estimated distance, speed, and traveling direction of the observation target. The control unit 20 may control the communication unit 19 to notify the mobile object 13 around the observation device 10 of the generated support information.

[0037] When acquiring the moving body information from the moving body 13, the control unit 20 may correct the distance information as described below. In correcting the distance information, the control unit 20 may extract the position information of the moving body 13 included in the moving body information. The control unit 20 calculates the actual distance of the moving body 13 based on the position information of the moving body 13. Specifically, the control unit 20 first converts the position information of the moving body 13 corresponding to the coordinates in the world coordinate system into position information of the surrounding space coordinate system based on the origin position and orientation of the surrounding space coordinate system in the world coordinate system. Next, the control unit 20 calculates the distance from the origin of the surrounding space coordinate system to the position of the moving body 13 in the surrounding space coordinate system as the actual distance.

[0038] Furthermore, the control unit 20 may detect the subject image of the moving body 13 from an image approximately contemporaneous with the moving body information by determining the presence or absence of the observation target and identifying the type of the observation target. An image approximately contemporaneous with the moving body information is, for example, an image captured at a time closest to the time when the spatial position of the moving body 13 is detected, among images captured continuously at a predetermined frame rate. However, strictly speaking, an image approximately contemporaneous with the moving body information does not need to be limited to an image captured at a time closest to the time when the spatial position of the moving body 13 is detected. An image approximately contemporaneous with the moving body information may be, for example, an image that can be regarded as an image captured at a time closest to the time when the spatial position of the moving body 13 is detected, such as an image acquired at the same time as the moving body information.

[0039] The control unit 20 calculates an estimated distance of the moving object 13 corresponding to the detected subject image based on the image. The control unit 20 may associate the calculated actual distance with the calculated estimated distance. When a single subject image of the moving object 13 is detected in a single image, the calculated estimated distance may be associated with the calculated actual distance. When multiple subject images of the moving object 13 are detected in a single image, the association and correction of the distance information may be stopped, or the estimated distance with the smallest difference from the actual distance may be associated with the actual distance.

[0040] The control unit 20 calculates a distance deviation, which is the difference between the corresponding actual distance and estimated distance. The control unit 20 compares the distance deviation with a first threshold. The first threshold is an allowable error when the estimated distance is included as support information. The first threshold may be set manually or automatically depending on the speed limit of the road 12 observed by the observation device 10, etc.

[0041] The control unit 20 may store in the storage unit 18 the calculated distance shift, the actual measured distance from which the distance shift was calculated, and the image position corresponding to the estimated distance from which the distance shift was calculated, in association with each other.

[0042] As shown in FIG. 4, in the calculation of the estimated distance using the image of the monocular camera, the correspondence is determined on the assumption that the road surface such as the road 12 in the real space forms a single virtual plane vp. The image position corresponds to the direction of the observation device 10. The intersection of the virtual plane vp with a straight line extending from the observation device 10 in each direction corresponding to each image position corresponds to the estimated distance. Also, the intersection of the real road surface rp of the actual road 12 with a straight line extending from the observation device 10 in each direction corresponding to each image position corresponds to the measured distance. If the vertical height of the real road surface rp of the actual road 12 differs from the height of the virtual plane vp, a distance discrepancy occurs between the estimated distance and the measured distance.

[0043] For example, for a first direction dr1 in which the vertical heights of the virtual plane vp and the real road surface rp are the same, the estimated distance dis_v1 and the measured distance dis_r1 are the same, and the distance deviation is zero. For example, for a second direction dr2 in which the vertical heights of the virtual plane vp and the real road surface rp are the same, the estimated distance dis_v2 and the measured distance dis_r2 are the same, and the distance deviation is zero. For example, for a third direction dr3 in which the vertical heights of the virtual plane vp and the real road surface rp are different, the estimated distance dis_v3 and the measured distance dis_r3 are different, and the distance deviation Δdis3 is equal to or greater than the first threshold. For example, for a fourth direction dr4 in which the vertical heights of the virtual plane vp and the real road surface rp are different, the estimated distance dis_v4 and the measured distance dis_r4 are different, and the distance deviation Δdis4 is equal to or greater than the first threshold. For example, for a fifth direction dr5 in which the vertical heights of the virtual plane vp and the actual road surface rp are different, the estimated distance dis_v5 and the measured distance dis_r5 are different, and the distance deviation Δdis5 is equal to or greater than the first threshold value.

[0044] When the distance deviation at the point where the estimated distance is acquired is equal to or greater than the first threshold, the control unit 20 corrects the distance information using the actual distance. When the distance information is a conversion table, the correction may be performed by replacing information on the distance corresponding to the image position on the image corresponding to the point where the estimated distance is acquired in the conversion table with the actual distance. For example, when the distance deviation Δdis2 corresponding to the second direction dr2 is calculated, the control unit 20 replaces information on the distance in the conversion table at the image position on the image corresponding to the second direction dr2 with the value of the actual distance. When the distance information is a conversion formula, the conversion formula may be corrected so that the actual distance is calculated at the image position where the estimated distance is calculated.

[0045] When a distance shift is newly calculated, the control unit 20 may determine whether or not the image position associated with the actual measured distance stored in the storage unit 18 is different from the image position corresponding to the newly calculated distance shift. The image position corresponding to the distance shift is an image position that causes the estimated distance used to calculate the distance shift to be calculated. When the image positions are different, the control unit 20 may determine whether or not at least one of the newly calculated distance shift and the distance shift corresponding to the stored actual measured distance is equal to or greater than a first threshold value.

[0046] When at least one of the distances is equal to or greater than the first threshold, the control unit 20 may correct information on the distance in the distance information that corresponds to the included image position between two different image positions, using two actual distances corresponding to the two image positions. The included image position is a position on the image between two different image positions. That is, it is a position on the image acquired by the imaging unit 17 at a point between the coordinates of the surrounding space coordinate system corresponding to the two image positions. Specifically, the control unit 20 may correct information on the distance in the distance information that corresponds to the included image position, based on the difference between the two actual distances corresponding to the two image positions and the difference between the two estimated distances, the shorter actual distance of the two actual distances, and the difference between the shorter actual distance and the estimated distance corresponding to the included image position.

[0047] For example, information on the distance defined by the distance information for an included image position sandwiched between two positions, an image position corresponding to the second direction dr2 and an image position corresponding to the third direction dr3, is corrected using two actual distances corresponding to the second direction dr2 and the third direction dr3. Information on the distance for the included image position may be distributed so that the image position corresponding to the second direction dr2 and the image position corresponding to the third direction dr3 each match the actual distance. For example, information on the distance for each included image position may be set according to the distance in the image of a plurality of included image positions (e.g., the distance between pixels).

[0048] For example, after storing the actual distance corresponding to the second direction dr2 in the memory unit 18, when the control unit 20 calculates the distance shift Δdis3 corresponding to the third direction dr3, the control unit 20 replaces the distance information dis_vx in the distance information corresponding to any direction between the second direction dr2 and the third direction dr3 with the corrected distance cdis_vx calculated using equation (1).

[0049]

number

[0050] When the distance information is a conversion table, the correction may be performed by replacing the distance information in the conversion table for image positions corresponding to any of a plurality of directions between the first direction dr1 and the second direction dr2 with the correction distance cdis_vx according to the above formula. Alternatively, when the distance information is a conversion formula, the original conversion formula may be corrected by incorporating formula (1) into the original conversion formula in the range between the first direction dr1 and the second direction dr2.

[0051] When multiple actual distances are stored in the memory unit 18 and a distance shift is newly calculated, the control unit 20 may correct the distance information corresponding to the aforementioned included image position using the actual distance among the multiple actual distances that has the smallest difference from the actual distance corresponding to the newly calculated distance shift.

[0052] For example, when the control unit 20 stores the measured distances dis_r2 and dis_r3 in the memory unit 18 and then calculates a new distance shift Δdis4, the control unit 20 corrects the distance information of the image position corresponding to any direction between the third direction dr3 and the fourth direction dr4 using the measured distance dis_r3 that has the smallest difference from the measured distance dis_r4 corresponding to the distance shift Δdis4.

[0053] When there are multiple image positions where the distance shift is less than a first threshold, the control unit 20 may correct the distance information corresponding to the included image position using the image position closest to the image position associated with a distance shift that is greater than or equal to the first threshold.

[0054] For example, when the control unit 20 stores the measured distances dis_r1 and dis_r2 in the memory unit 18 and then calculates a new distance shift Δdis3, the control unit 20 corrects the distance information of an image position corresponding to any direction between the second direction dr2 and the third direction dr3 using the measured distance dis_r2 corresponding to the second direction dr2, which is the image position closest to the image position corresponding to the distance shift Δdis4.

[0055] When the distance deviation is equal to or greater than a second threshold, the control unit 20 may not correct the distance information, but may generate warning information to warn of an abnormality in the observation device 10. The second threshold may be greater than the first threshold. The control unit 20 may control the network IF 21 to transmit the warning information to a server of a user who maintains the observation device 10.

[0056] The control unit 20 may be capable of switching the operation mode to at least an initial setting mode (initial setting) or a normal mode. The control unit 20 may correct the above-mentioned distance information in either the initial setting mode or the normal mode. The control unit 20 may be switched to the initial setting mode by, for example, input using an input device when the observation device 10 is installed. The control unit 20 may be switched to the normal mode after the initial setting mode described later is completed.

[0057] The control unit 20 may correct the distance information described above in the initial setting mode. The control unit 20 may correct the distance information when a single subject image of the moving body 13 is detected in a single image in the initial setting mode. The control unit 20 may not perform association and correction of distance information when multiple subject images of the moving body 13 are detected in a single image in the initial setting mode. The control unit 20 may not generate warning information and may correct the distance information when the distance deviation is equal to or greater than the second threshold value in the initial setting mode.

[0058] When the distance information is sufficiently corrected in the initial setting mode, the control unit 20 may end the initial setting mode and transition to the normal mode. The condition that the distance information is sufficiently corrected may be, for example, that moving body information has been acquired from 10 moving bodies 13, in other words, that moving body information has been acquired multiple times from each of the 10 moving bodies 13.

[0059] The network IF 21 may include a communication module that connects to an external network. The observation device 10 may be connected to the external network via the network IF 21 and may communicate information with, for example, an external server.

[0060] Next, the correction process in the initial setting mode executed by the control unit 20 in this embodiment will be described with reference to the flowcharts in Figures 5 and 6. The correction process in the initial setting mode is started by switching the operation mode using an input device after the observation device 10 is installed.

[0061] In step S100, the control unit 20 determines whether or not moving object information has been acquired. If not, the process returns to step S100. If acquired, the process proceeds to step S101.

[0062] In step S101, the control unit 20 extracts the position information from the moving object information determined to have been acquired in step S100. After the extraction, the process proceeds to step S102.

[0063] In step S102, the control unit 20 calculates the actual distance based on the position information extracted in step S101. After calculating the actual distance, the process proceeds to step S103.

[0064] In step S103, the control unit 20 detects a subject image of the moving object 13 from images taken at approximately the same time as the moving object information determined to have been acquired in step S100. After the subject image is detected, the process proceeds to step S104.

[0065] In step S104, the control unit 20 determines whether or not subject images of multiple moving objects 13 are detected in step S103. If subject images of multiple moving objects 13 are detected, the process returns to step S100. If subject images of multiple moving objects 13 are not detected but a subject image of a single moving object 13 is detected, the process proceeds to step S105.

[0066] In step S105, the control unit 20 calculates an estimated distance from the image position of the subject image detected in step S103, using the distance information stored in the storage unit 18. After the calculation of the estimated distance, the process proceeds to step S106. Note that, at the start of the initial setting mode, the storage unit 18 stores distance information generated on the assumption that the road surface observed by the observation device 10 is flat.

[0067] In step S106, the control unit 20 calculates a distance deviation as a difference between the actual distance calculated in step S102 and the estimated distance calculated in step S105. After calculating the distance deviation, the process proceeds to step S107.

[0068] In step S107, the control unit 20 associates the actual distance calculated in step S102, the image position at which the estimated distance calculated in step S105 was calculated, and the distance shift calculated in step S106. Furthermore, the control unit 20 associates the actual distance, the image position, and the distance shift with each other and stores them in the storage unit 18. After the association and storage are completed, the process proceeds to step S108.

[0069] In step S108, the control unit 20 determines whether the distance deviation calculated in step S106 is equal to or greater than a first threshold. If it is equal to or greater than the first threshold, the process proceeds to step S109. If it is not equal to or greater than the first threshold, the process proceeds to step S110.

[0070] In step S109, the control unit 20 corrects the distance information in the distance information corresponding to the specific image position based on the actual distance calculated in step S102. The specific image position is the image position of the subject image of the moving object 13 detected in step S103. After the distance information is corrected, the process proceeds to step S110.

[0071] In step S110, the control unit 20 determines whether the image position stored in the storage unit 18 is different from the image position associated in step S107. If they are different, the process proceeds to step S111. If they are the same, the process proceeds to step S116.

[0072] In step S111, the control unit 20 determines whether there are a plurality of image positions stored in the storage unit 18 that are different from the image position associated in step S107. If there is a single image position, the process proceeds to step S112. If there are a plurality of image positions, the process proceeds to step S113.

[0073] In step S112, the control unit 20 associates the image position associated with the measured distance stored in the storage unit 18 with the image position associated in step S107. After the association, the process proceeds to step S114.

[0074] In step S113, the control unit 20 associates the image position associated with the actual distance that has the smallest difference from the actual distance calculated in step S107 among the multiple actual distances stored in the storage unit 18 with the image position associated in step S107. After the association, the process proceeds to step S114.

[0075] In step S114, the control unit 20 determines whether or not at least one of the distance deviations corresponding to the image positions associated in either step S112 or step S113 is equal to or greater than a first threshold. If at least one of the distance deviations is equal to or greater than the first threshold, the process proceeds to step S115. If both distance deviations are not equal to or greater than the first threshold, the process proceeds to step S116.

[0076] In step S115, the control unit 20 corrects the distance information in the distance information corresponding to the specific range of image positions based on the measured distances corresponding to the image positions associated in step S112 or step S113. The specific range of image positions is the range sandwiched between the image positions associated in step S112 or step S113. After the correction of the distance information, the process proceeds to step S116.

[0077] In step S116, the control unit 20 determines whether or not the moving object information has been acquired from the ten moving objects 13. If not, the process returns to step S100. If acquired, the correction process in the initial setting mode ends.

[0078] Next, the correction process in the normal mode executed by the control unit 20 in this embodiment will be described with reference to the flowcharts of Figures 7 and 8. The correction process in the normal mode starts when acquiring moving body information after the initial setting mode ends.

[0079] In step S200, the control unit 20 extracts the position information from the newly acquired moving body information. After the extraction, the process proceeds to step S201.

[0080] In steps S201 and S202, the control unit 20 performs the same control as in steps S102 and S103 of the correction process in the initial setting mode. After the subject image is detected in step S202, the process proceeds to step S203.

[0081] In step S203, control unit 20 calculates an estimated distance from the image position of the subject image detected in step S202, using the distance information stored in storage unit 18. After calculating the estimated distance, the process proceeds to step S204.

[0082] In step S204, the control unit 20 determines whether or not subject images of multiple moving objects 13 were detected in step S202. If subject images of multiple moving objects 13 were not detected but a single moving object 13 was detected, the process proceeds to step S205. If subject images of multiple moving objects 13 were detected, the process proceeds to step S206.

[0083] In step S205, the control unit 20 associates the actual distance calculated in step S201 with the estimated distance calculated in step S203. After the association, the process proceeds to step S207.

[0084] In step S206, the control unit 20 associates the estimated distance, which has the smallest difference from the actual distance calculated in step S201, among the estimated distances calculated in step S203 for each of the multiple subject images of the moving object 13 detected in step S202 with the actual distance. After the association, the process proceeds to step S207.

[0085] In step S207, the control unit 20 calculates a distance shift as a difference between the actual distance and the estimated distance associated in step S205 or step S206. Furthermore, the control unit 20 associates the actual distance calculated in step S201, the image position at which the estimated distance associated with the actual distance in step S205 or S206 was calculated, and the calculated distance shift in the storage unit 18. After calculating and storing the distance shift, the process proceeds to step S208.

[0086] In step S208, the control unit 20 determines whether the distance deviation calculated in step S207 is equal to or greater than the second threshold. If it is equal to or greater than the second threshold, the process proceeds to step S209. If it is not equal to or greater than the second threshold, the process proceeds to step S210.

[0087] In step S209, the control unit 20 generates warning information. Furthermore, the control unit 20 controls the network IF 21 to transmit the generated warning information to an external server. After the generation and transmission, the correction process in the normal mode ends.

[0088] In step S210, the control unit 20 determines whether the distance deviation calculated in step S207 is equal to or greater than a first threshold. If it is equal to or greater than the first threshold, the process proceeds to step S211. If it is not equal to or greater than the first threshold, the process proceeds to step S212.

[0089] In step S211, the control unit 20 corrects the distance information in the distance information corresponding to the specific image position based on the actual distance calculated in step S201. The specific image position is the image position of the subject image of the moving object 13 detected in step S202. After the distance information is corrected, the process proceeds to step S212.

[0090] In step S212, the control unit 20 determines whether the image position stored in the storage unit 18 is different from the image position associated in step S205 or step S206. If they are the same, the process proceeds to step S213. If they are different, the correction process in the normal mode ends.

[0091] In step S213, the control unit 20 determines whether there are a plurality of image positions stored in the storage unit 18 that are different from the image positions associated in step S205 or step S206. If there is a single image position, the process proceeds to step S214. If there are a plurality of image positions, the process proceeds to step S215.

[0092] In step S214, the control unit 20 associates the image position associated with the measured distance stored in the storage unit 18 with the image position associated in step S205 or step S206. After the association, the process proceeds to step S216.

[0093] In step S215, the control unit 20 associates the image position associated with the actual distance that has the smallest difference from the actual distance calculated in step S205 or S206 among the multiple actual distances stored in the storage unit 18 with the image position associated in step S205 or S206. After the association, the process proceeds to step S216.

[0094] In step S216, the control unit 20 determines whether at least one of the distance deviations corresponding to the image positions associated in either step S214 or step S215 is equal to or greater than a first threshold. If at least one of the distance deviations is equal to or greater than the first threshold, the process proceeds to step S217. If both distance deviations are not equal to or greater than the first threshold, the correction process in the normal mode ends.

[0095] In step S216, the control unit 20 corrects the distance information in the distance information corresponding to the specific range of image positions based on the measured distances corresponding to the image positions associated in step S214 or step S215. The specific range of image positions is the range sandwiched between the image positions associated in step S214 or step S215. After the distance information is corrected, the correction process in the normal mode ends.

[0096] The observation device 10 of this embodiment configured as described above includes a storage unit 18 that stores distance information, which is the relationship between the image position on the image captured by the imaging unit 17 and information on distance, a communication unit (acquisition unit) 19 that acquires position information on the spatial position of the moving object 13 from the moving object 13, and a control unit 20 that corrects the distance information using the actual measured distance when the distance deviation between the actual measured distance corresponding to the position information and the estimated distance estimated based on the image position of the moving object 13 in the image and the distance information is equal to or greater than a first threshold. With this configuration, the observation device 10 can calibrate the difference in shape in the vertical direction due to the gradient of the actual road surface rp, etc. Therefore, the observation device 10 can improve the estimation accuracy of the distance.

[0097] In addition, when at least one distance deviation between two different image positions is equal to or greater than a first threshold value, the observation device 10 of this embodiment corrects the distance information corresponding to the included image position between the two image positions using the two actual measured distances corresponding to the two image positions. With this configuration, the observation device 10 can calibrate the difference in the shape of the real road surface rp not only for the two image positions corresponding to the two calculated actual measured distances, but also for the area between the two image positions. Therefore, the observation device 10 can improve the estimation accuracy of the distance over a wide range.

[0098] In addition, the observation device 10 of the present embodiment corrects the distance information corresponding to the included image position using the image position closest to the image position where the distance shift is equal to or greater than the first threshold among the image positions where the distance shift is less than the first threshold. When the distance information corresponding to the image position where the distance shift is less than the first threshold is corrected with the actual distance corresponding to another image position where the distance shift is equal to or greater than the first threshold, the estimation accuracy of the distance calculated from the image position where the distance shift is less than the first threshold decreases. In response to such an event, the observation device 10 having the above-described configuration can reduce the image positions where the distance shift is less than the first threshold from the range of included image positions corrected with the actual distance corresponding to another image position where the distance shift is equal to or greater than the first threshold. Therefore, the observation device 10 can reduce the range where the estimation accuracy of the distance decreases.

[0099] In addition, in the observation device 10 of the present embodiment, when a plurality of actual distances are stored in the storage unit 18 and a distance shift is newly calculated, the observation device 10 corrects the distance information corresponding to the above-mentioned embodied image position using the actual distance that has the smallest difference from the actual distance corresponding to the newly calculated distance shift among the plurality of actual distances. For example, as shown in FIG. 9, when eight actual distances dis_r are calculated for the actual road surface rp, the actual distances are arranged in order of shortest and the distance information is corrected using the actual distances of two locations every other, and the distance is estimated based on the first virtual plane vp1, which has a relatively large distance shift from the actual road surface rp. On the other hand, the observation device 10 having the above configuration can estimate the distance based on the second virtual plane vp2, which has a smaller distance shift from the actual road surface rp than the first virtual plane vp1, as shown in FIG. 10. Therefore, the observation device 10 can further improve the distance estimation accuracy.

[0100] Furthermore, when an image includes multiple images of the moving object 13, the observation device 10 of this embodiment corrects the distance information based on the actual measured distance and an estimated distance that has the smallest difference from the actual measured distance among the estimated distances corresponding to the multiple images of the moving object 13. With this configuration, when multiple subject images of the moving object 13 are detected from an image, the observation device 10 can appropriately determine which subject image to use to correct the distance information. Therefore, the observation device 10 can correct the distance information even when multiple subject images of the moving object 13 are detected from an image.

[0101] Furthermore, when the distance deviation is equal to or greater than a second threshold value that is greater than the first threshold value, the observation device 10 of this embodiment generates warning information that warns of an abnormality in the observation device 10. With this configuration, the observation device 10 can warn of a state that is assumed to be an abnormality in the observation device 10 itself or an installation abnormality in the observation device 10.

[0102] Furthermore, in the observation device 10 of this embodiment, if the distance deviation is equal to or greater than the second threshold during initial setting, the generation of warning information is stopped and the distance information is corrected. When correcting the distance information during initial setting, the distance deviation may become relatively large. For such an event, the observation device 10 does not determine that the observation device 10 is abnormal during initial setting, and therefore unnecessary warnings can be prevented.

[0103] Although the embodiments of the present disclosure have been described based on the drawings and examples, it should be noted that those skilled in the art can easily make various modifications or corrections based on the present disclosure. Therefore, it should be noted that these modifications or corrections are included in the scope of the present disclosure. For example, the functions included in each component or step can be rearranged so as not to be logically inconsistent, and multiple components or steps can be combined into one or divided.

[0104] In this disclosure, descriptions such as "first" and "second" are identifiers for distinguishing the configuration. In the configurations distinguished by descriptions such as "first" and "second" in this disclosure, the numbers in the configurations can be exchanged. For example, the first direction can exchange the identifiers "first" and "second" with the second direction. The exchange of identifiers is performed simultaneously. The configurations remain distinguished even after the exchange of identifiers. The identifiers may be deleted. A configuration from which an identifier has been deleted is distinguished by a code. The descriptions of identifiers such as "first" and "second" in this disclosure should not be used solely to interpret the order of the configurations or to justify the existence of an identifier with a smaller number. [Explanation of symbols]

[0105] 10 Observation equipment 11. Communications Systems 12 Road 13 Mobile 14 Pedestrians 15 Traffic lights 16 Crosswalk 17 Imaging section 18 Memory section 19 Acquisition Department 20 Control section 21 Communications Department 22 Structures dr1 First direction dr2 Second direction dr3 Third direction dr4 fourth direction rp Real road surface ox optical axis vp Virtual plane

Claims

1. a storage unit that stores distance information that is a relationship between an image position on an image generated by capturing an image of a space including a predetermined area in which an observation target exists and information regarding a distance to the observation target; and a control unit that corrects the distance information corresponding to an included image position between the two different image positions by using two actual measured distances corresponding to the two image positions. Observation equipment.

2. The control unit corrects the distance information corresponding to the encapsulated image position when a distance deviation between information on the distance acquired based on the two image positions and the distance information and an actual distance to the observation target corresponding to the two image positions is equal to or greater than a first threshold value. The observation device according to claim 1.

Images