Apparatus and method for estimating coordinates of road hazard

Abstract

The present specification relates to a computer vision and positioning technology. This method by which a moving body estimates the coordinates of an object existing in a space comprises: receiving a satellite signal so as to obtain satellite coordinates of the moving body; receiving an image of the space; calculating, on the basis of the relationship between floor coordinates in the image and pixel coordinates of a detection area, position information between the moving body and the object detected in the image; and calculating satellite coordinates of the object by using the satellite coordinates of the moving body and the position information between the moving body and the object.

Description

Device and method for estimating coordinates of road hazards

[0001] This specification relates to computer vision and positioning technology, and more particularly, to a device and method for estimating the coordinates of a hazardous material existing in a space where a vehicle is driving, such as a road, from an image taken of the space, and a recording medium recording the method.

[0002] On roads, there are hazards such as potholes (localized holes created by the collapse or collapse of a portion of the road surface), pavement cracks, and fallen objects. If an unexpected hazard suddenly appears while driving, it can be difficult for drivers to respond appropriately, such as slowing down or changing direction to avoid the hazard, increasing the risk of traffic accidents.

[0003] Recently, technologies have emerged that support drivers' safer driving by providing them with advance information about road hazards. Research is actively underway on technologies that collect images of road hazards and the location information of the locations where the images were captured, using vehicles equipped with information collection terminals. These images are then provided to road management agencies or shared with other drivers. Korean Patent No. 10-2147540, "Information Sharing Server for Enabling Sharing of Road Condition Information Based on Vehicle Location Information and Driving Condition Information and Operating Method Thereof," introduces a technology for sharing road condition information based on vehicle location information and driving condition information.

[0004] However, because the location information for hazards detected from high-speed vehicles is inaccurate, it has become difficult for authorities to take action against hazards or warn other drivers. Typically, the location of hazards is determined by the location of the camera installed in the vehicle (i.e., the vehicle itself), and vehicle movement can lead to errors in determining the location of the hazard itself.

[0005] Therefore, research is needed on a technology that can estimate the exact location of a hazardous material detected in images acquired through devices such as cameras and GPS sensors in a moving vehicle for analysis of the location of hazardous materials and follow-up measures.

[0006] The technical problems that the embodiments of the present specification seek to solve are to overcome the limitation of conventional road management technology in not being able to provide accurate location information of hazardous materials, and to resolve the problem that even when satellite signals are utilized, location errors occurring due to high-speed driving after receiving satellite signals are not taken into account, thereby eliminating inconveniences that arise in road hazardous material management or information sharing.

[0007] In order to solve the above technical problem, a method for estimating coordinates of an object existing in space according to one embodiment of the present specification includes the steps of: a mobile body receiving a satellite signal to obtain its own satellite coordinates; a step of the mobile body receiving an image of space; a step of the mobile body calculating positional information between the mobile body and an object detected in the image based on a relationship between a floor coordinate in the image and a pixel coordinate of a detection area; and a step of the mobile body calculating the satellite coordinates of the object using the satellite coordinates of the mobile body and the positional information between the mobile body and the object.

[0008] In a coordinate estimation method according to one embodiment, the step of calculating position information between the moving body and the object detected in the image may include the step of deriving a perspective transformation for the detection area from a relationship between the floor coordinates in the image and the pixel coordinates of the detection area based on the position of the image sensor that acquires the image and the movement information detected through the first inertial sensor; the step of calculating the floor coordinates corresponding to the object detected in the image using the derived perspective transformation; and the step of calculating the position information between the moving body and the object detected in the image using the floor coordinates calculated corresponding to the object and the distance to the object.

[0009] A coordinate estimation method according to one embodiment may further include a step of correcting an error caused by a difference between a movement measurement standard of the moving object and a movement measurement standard of the image sensor.

[0010] In a coordinate estimation method according to one embodiment, the step of calculating the satellite coordinates of the object may include the step of correcting the satellite coordinates of the moving object itself by considering a change in the position of the moving object from the time of receiving the satellite signal to the time of detecting the object; and the step of calculating the satellite coordinates of the object using the corrected satellite coordinates of the moving object itself and the distance and direction between the moving object and the object.

[0011] Furthermore, the following provides a computer-readable recording medium having recorded thereon a program for executing the coordinate estimation method described above on a computer.

[0012] In order to solve the above technical problem, a coordinate estimation device according to one embodiment of the present specification includes a receiver for receiving a satellite signal to obtain satellite coordinates of a moving object; an image sensor for receiving an image of a space; and a processor for estimating coordinates of an object existing in the space, wherein the processor calculates position information between the moving object and an object detected in the image based on a relationship between a floor coordinate in the image and pixel coordinates of a detection area, and executes a command for calculating satellite coordinates of the object using the satellite coordinates of the moving object and the position information between the moving object and the object.

[0013] In a coordinate estimation device according to one embodiment, the processor may perform a command to derive a perspective transformation for the detection area from a relationship between the floor coordinates in the image and the pixel coordinates of the detection area based on the position of the image sensor that acquires the image and the movement information detected through the first inertial sensor, to derive floor coordinates corresponding to an object detected in the image using the derived perspective transformation, and to derive position information between the moving body and the object detected in the image using the floor coordinates derived corresponding to the object and the distance to the object.

[0014] According to one embodiment, a coordinate estimation device further includes a first inertial sensor that is fixed in correspondence with the image sensor and sets a movement measurement standard; and a second inertial sensor that is fixed in correspondence with the moving object and sets a movement measurement standard, wherein the processor can perform a command to correct movement information detected through the first inertial sensor based on the second inertial sensor by using a difference between movement measurement standards set through each of the inertial sensors, thereby eliminating an error caused by adjustment of the orientation direction of the image sensor.

[0015] In a coordinate estimation device according to one embodiment, the processor may perform a command to correct the satellite coordinates of the mobile object by considering a change in the position of the mobile object from the time of receiving the satellite signal to the time of detecting the object, and to calculate the satellite coordinates of the object by using the corrected satellite coordinates of the mobile object and the distance and direction between the mobile object and the object.

[0016] Embodiments of the present specification can provide real-time satellite coordinates of road hazards in images acquired through a camera by first estimating the satellite coordinates of the point to which the vehicle has moved since receiving the last satellite signal and secondarily estimating the satellite coordinates of actual road hazards based on the current location of the vehicle, thereby eliminating positional errors due to vehicle driving after receiving the satellite signal, and consequently enabling effective road hazard management and accurate sharing of road hazard information.

[0017] Figure 1 is an exemplary diagram illustrating a situation in which the location of a hazardous material detected on a road is analyzed in which embodiments of the present specification are implemented.

[0018] FIG. 2 is a diagram showing a schematic process for estimating coordinates of road hazards proposed by embodiments of the present specification.

[0019] FIG. 3 is a flowchart illustrating a method for estimating coordinates of an object existing in space according to one embodiment of the present specification.

[0020] FIG. 4 is a flowchart illustrating in more detail the process of calculating position information between a moving object and an object in the embodiment of FIG. 3, which estimates the coordinates of an object existing in space.

[0021] FIG. 5 is a diagram for explaining a process of deriving a perspective transformation for a detection area and calculating the size of an object detected in an image in a coordinate estimation method according to one embodiment of the present specification.

[0022] FIG. 6 is a diagram illustrating a hardware configuration for correcting an error caused by a difference between a movement measurement standard of a moving object and a movement measurement standard of an image sensor in a coordinate estimation method according to another embodiment of the present specification.

[0023] FIG. 7 is a flowchart illustrating in more detail the process of calculating satellite coordinates of an object in the embodiment of FIG. 3, which estimates the coordinates of an object existing in space.

[0024] FIG. 8 is a drawing for explaining a process of identifying the location of a hazardous material based on a moving object and deriving the coordinates of the hazardous material in a coordinate estimation method according to one embodiment of the present specification.

[0025] FIG. 9 is a block diagram illustrating a device for estimating coordinates of an object existing in space according to one embodiment of the present specification.

[0026] <Explanation of symbols>

[0027] 900: Coordinate Estimation Device

[0028] 10: Receiver 50: Processor

[0029] 20: Image sensor 30: Inertial sensor

[0030] Hereinafter, embodiments of the present specification will be described in detail with reference to the drawings. However, detailed descriptions of well-known functions or components that may obscure the gist of the embodiments in the following description and the attached drawings will be omitted. Additionally, throughout the specification, the term "including" a component does not exclude other components, unless specifically stated otherwise, but rather implies the inclusion of other components.

[0031] Additionally, while terms such as "first" and "second" may be used to describe various components, these components should not be limited by these terms. These terms may be used to distinguish one component from another. For example, without departing from the scope of this specification, a first component may be referred to as a second component, and similarly, a second component may also be referred to as a first component.

[0032] The terminology used herein is for the purpose of describing specific embodiments only and is not intended to be limiting of the present disclosure. The singular expressions include plural expressions unless the context clearly dictates otherwise. In this application, it should be understood that the terms "comprise" or "have" indicate the presence of a described feature, number, step, operation, component, part, or combination thereof, but do not preclude the possibility of the presence or addition of one or more other features, numbers, steps, operations, components, parts, or combinations thereof.

[0033] Unless specifically defined otherwise, all terms used herein, including technical or scientific terms, have the same meaning as commonly understood by a person of ordinary skill in the art to which this specification pertains. Terms defined in commonly used dictionaries should be interpreted to have a meaning consistent with their meaning in the context of the relevant technology, and will not be interpreted in an idealized or overly formal sense unless explicitly defined herein.

[0034] Figure 1 is an exemplary diagram illustrating a situation in which the locations of hazardous objects detected on a road are analyzed in accordance with embodiments of the present disclosure. This illustrates a situation in which a moving object (which may be a vehicle) traveling on a road acquires an image of its surroundings and detects hazardous objects or road surface damage within the acquired image. For example, referring to Figure 1, a situation is shown in which a road image ahead is acquired through a camera installed on a vehicle and a pothole is detected on the road surface.

[0035] In these situations, conventional technologies typically match specific coordinates corresponding to the vehicle's location with images of hazardous materials and report them to the control center. The vehicle's location can be the vehicle's own satellite coordinates acquired via a GPS receiver installed in the vehicle, or specific location information on a separately identifiable map.

[0036] However, the satellite coordinates or location information at this time do not accurately indicate the location of the detected hazard. This is because, in the situation shown in Figure 1, there is no technical means to pinpoint the location of the pothole. Therefore, conventional techniques have the limitation that they can only consider the vehicle's position at the time the hazard was detected or a specific location nearby as the location of the hazard. In other words, the precise location of a specific object within the image cannot be determined.

[0037] To address these issues, the embodiments of the present disclosure described below propose a technical means for accurately estimating location information (particularly, satellite coordinates) of a specific object detected within an image, even when using an image-based object detection technique.

[0038] Figure 2 is a schematic diagram illustrating a process for estimating the coordinates of road hazards proposed by embodiments of the present disclosure. As illustrated, a vehicle is traveling along a road from location A to location B. Assume the vehicle is equipped with a receiver capable of receiving satellite signals to obtain GNSS satellite coordinates and an image sensor (camera) capable of capturing surrounding images.

[0039] A vehicle is driving while continuously receiving satellite signals, and when the vehicle reaches location B after receiving the latest satellite signal at location A, a hazard (e.g., a pothole) may be detected in the image in front of the vehicle acquired by the camera. That is, there is a slight time difference (0.900 s) between the time when the satellite signal was last received (system time: 15h 30m 30.150 s) and the time when the hazard was detected (system time: 15h 30m 31.050 s). Therefore, even though the vehicle has received the latest satellite signal, the GNSS satellite coordinates acquired from the satellite signal cannot be regarded as the current real-time location of the vehicle. This is because the vehicle is continuously moving even in a short period of time, and the position error may become larger as the vehicle speed increases. For example, assuming that GNSS information is updated every second, if a vehicle is driving at a speed of 100 km / h, the vehicle will move approximately 27.78 m even during the short 1 second period when the GNSS information is updated. Therefore, in order to eliminate errors occurring within the satellite signal reception period (1 second in the example situation) during such driving, it is necessary to derive the satellite coordinates of the current real-time location B from the satellite coordinates acquired at location A. To this end, the embodiments of the present specification attempted to estimate the satellite coordinates of the vehicle after the last satellite signal was received by using the moving speed and azimuth of the vehicle.

[0040] Even if we assume that the satellite coordinates of location B are derived, the satellite coordinates of the vehicle (location B) and the satellite coordinates of the hazardous material (location C) are also different. In this case, since the positional relationship between the vehicle and the hazardous material can only be analyzed through images acquired through a camera, it is very difficult to determine the satellite coordinates of the hazardous material. Therefore, a technical means is needed to estimate the satellite coordinates (location C) of the hazardous material from the satellite coordinates of the vehicle (location B) through an image-based analysis process, which is the ultimate goal that the embodiments of the present specification seek to achieve. To this end, the embodiments of the present specification attempted to estimate the distance and satellite coordinates between the vehicle and the actual hazardous material by utilizing the spatial correlation within the images captured with the vehicle as the reference.

[0041] As described above, the embodiments of the present specification ultimately obtain satellite coordinates of a hazardous material through a two-step satellite coordinate derivation process (estimating location B from location A, estimating location C from location B), and the series of processes are described in more detail below.

[0042] FIG. 3 is a flowchart illustrating a method for estimating coordinates of an object existing in space according to one embodiment of the present specification, which can be achieved by a coordinate estimation device performing a series of processing steps. Here, the coordinate estimation device is installed in a moving object (e.g., a moving vehicle) and acquires an image of the surroundings (e.g., the front) of the vehicle through an image sensor (e.g., a camera), and acquires the satellite coordinates of the moving object itself based on a signal received from a satellite.

[0043] At step S310, the mobile receives a satellite signal and obtains its own satellite coordinates.

[0044] At step S330, the mobile device receives an image of the space. The received image may include various objects, such as vehicles on the road, hazardous materials, or road damage.

[0045] In step S350, the mobile body calculates position information between the mobile body (e.g., a moving vehicle) and an object (e.g., a road hazard) detected in the image based on the relationship between the floor coordinates in the image and the pixel coordinates of the detection area. In this process, the distance between the mobile body and the object can be estimated, and the size of the object can be estimated. More specifically, if the installation height of the image sensor (camera) equipped on the mobile body and the pitch and roll of the image sensor are known, and the front direction of the image sensor is set as the reference axis, the relationship between the pixel coordinates of the image acquired through the image sensor and the floor coordinates can be determined. In addition, using the installation height, pitch, and roll of the image sensor, a perspective transformation matrix can be obtained through four corresponding points of the captured image corresponding to the vertices of the detection area in front of the image sensor (e.g., a 5 m × 5 m square spanning from 5 m to 10 m). Then, if the actual floor coordinates corresponding to two points on the image are obtained, the distance between the two points on the floor can be known.

[0046] Meanwhile, various artificial intelligence models can be adopted to detect objects within images, and various element technologies such as object detection, tracking, recognition, or classification can be utilized as needed.

[0047] In step S370, the mobile device calculates the satellite coordinates of the object using the satellite coordinates of the mobile device itself and the positional information between the mobile device and the object. As previously described, in this process, the location of the mobile device is first estimated by considering the change in position due to the movement of the mobile device after receiving the satellite signal, and the satellite coordinates of the object are estimated through the correlation between the mobile device and the detected object.

[0048] More specifically, in order to estimate the position of a mobile object, the most recent GNSS data that received a satellite signal is extracted, and the travel time corresponding to the difference between the detection time of the object in the image and the reception time of the satellite signal is multiplied by the current speed to calculate the travel distance. To this end, the Haversine Formula, which calculates the distance between the latitude and longitude coordinates of two points before and after the movement, can be utilized. In addition, the latitude and longitude of the new point after the movement can be calculated by considering the bearing between the two points to calculate the coordinates of the mobile object. Then, in order to estimate the satellite coordinates of the object in the image, the X, Y-axis distances between the image sensor equipped on the mobile object and the object can be calculated, and the X, Y-axis distances, the radius of the Earth, and the satellite coordinates of the mobile object can be used to correct the satellite coordinates of the object.

[0049] FIG. 4 is a flowchart illustrating in more detail the process (step S350) of calculating position information between a moving object and an object in the embodiment of FIG. 3 for estimating the coordinates of an object existing in space.

[0050] In step S351, a perspective transformation for the detection area can be derived from a relationship between the floor coordinates in the image and the pixel coordinates of the detection area based on the position of the image sensor that acquires the image and the movement information detected through the first inertial sensor. Here, the relationship between the floor coordinates in the image and the pixel coordinates of the detection area can be set based on the installation height of the image sensor and the pitch and roll information detected through the first inertial sensor corresponding to the image sensor. In addition, based on the installation height of the image sensor and the pitch and roll information detected through the first inertial sensor, the coordinates for the detection area in the image can be calculated, and the coordinates for the detection area rotated by the roll can be calculated to determine a perspective transformation matrix based on the image sensor.

[0051] More specifically, the pitch, which is the angle of inclination in the front direction of the position estimation device, and the roll, which is the angle of inclination in the side direction, can be measured with respect to the floor plane (road floor) by stopping the vehicle on a horizontal floor perpendicular to the direction of gravity, using the inertial sensor of the position estimation device. At this time, the inertial sensor (Inertial Measurement Unit, IMU) may be configured to include an acceleration sensor, a gyro sensor, a geomagnetic sensor, etc. In addition, if the installation height, pitch, and roll of the position estimation device are known, and the front direction of the image sensor is set to the Y-axis, the relationship between the pixel coordinates of the image acquired through the image sensor and the floor coordinates can be determined. Based on the information acquired above, the pitch and roll of the image sensor can be calculated using a 3D rotation matrix, vector operations, and trigonometry. Then, using the pitch, roll, and installation height of the image sensor, four corresponding points of the captured image corresponding to the vertices of the detection area in front of the image sensor (for example, a 5m × 5m square extending from 5m to 10m) can be estimated. Now, we can obtain the perspective transformation matrix using the four corresponding points.

[0052] In step S353, the floor coordinates corresponding to the object detected in the image can be calculated using the perspective transformation derived in step S351. That is, the actual floor coordinates corresponding to any image pixel coordinates can be known using the perspective transformation matrix. For example, the floor coordinates can be obtained by setting the front direction of the image sensor as the Y axis, the 90° right direction of the image sensor as the X axis, and setting the unit of distance as cm.

[0053] In step S355, the location information between the mobile object and the object detected in the image can be calculated using the calculated floor coordinates corresponding to the object and the distance to the object. If the actual floor coordinates corresponding to two points on the image are obtained, the distance between the two points on the floor can be known through a two-point distance calculation that calculates the distance between the latitude and longitude coordinates of the two points. For example, if the floor point where the location estimation device on the mobile object is installed is set as the origin, the floor distance to the object can be known by obtaining the actual location coordinates of the object.

[0054] FIG. 5 is a diagram for explaining a process of deriving a perspective transformation for a detection area and calculating the size of an object detected in an image in a coordinate estimation method according to one embodiment of the present specification.

[0055] In the illustrated drawing, the perspective transformation matrix can be obtained using the image pixel coordinates of the vertices of a 5m × 5m square between 5m and 10m in front.

[0056] In Fig. 5, angles a1, a2, b1, and b2 can be determined through trigonometry. (a1- Pitch) is the image vertical angle at a point 5 m in front of the image center, and (a2- Pitch) is the image vertical angle at a point 10 m in front of the image center. In addition, b1 is the horizontal angle at a point 2.5 m in the image horizontal direction from a point 5 m in front of the center, and b2 is the horizontal angle at a point 2.5 m in the image horizontal direction from a point 10 m in front of the center. Therefore, the following relationship holds.

[0057] Y (Pixels) / [Angle between two points above and below the image] = Image height (e.g., 1080) / [Vertical FOV (field of view)]

[0058] X (Pixels) / [Angle between two points on the left and right of the image] = Image width (e.g., 1920) / [Horizontal FOV]

[0059] According to the above relationship, the image coordinates of four vertices corresponding to 5m × 5m on the image without considering the roll angle of the image sensor can be calculated. In addition, the coordinates of the four vertices are obtained by rotating them by the roll angle about the camera center point, and a perspective transformation matrix whose origin is the installation position of the image sensor can be obtained by corresponding the four vertices to the actual coordinates (-2.5, 5)m, (2.5, 5)m, (2.5, 10)m, (-2.5, 10)m.

[0060] Meanwhile, Fig. 6 is a diagram illustrating a hardware configuration for correcting errors due to differences between the movement measurement standards of a moving object and the movement measurement standards of an image sensor in a coordinate estimation method according to another embodiment of the present specification. Fig. 6 (a) is a front view of a coordinate estimation device, and Fig. 6 (b) is a corresponding side view.

[0061] To measure the distance to an object by mounting an image sensor (camera) on a mobile device (vehicle), the sensor's three-axis pitch, roll, and yaw values ​​must be determined based on the measured motion. However, considering the need to adjust the camera's orientation after installation in a terminal device installed on the mobile device, it can be seen that there is a difference between the mobile device's motion measurement standards and the image sensor's motion measurement standards.

[0062] Furthermore, considering the typical case where the processor of a terminal device installed on a mobile device controls only one inertial sensor, the vertical / horizontal orientation of the mobile device cannot be determined depending on the angle adjustment of the image sensor. Therefore, after mounting the terminal device on the mobile device, it is necessary to read the inertial sensor measurement values ​​of the terminal device and compensate for the changes in measurement values ​​due to the angle adjustment of the image sensor.

[0063] Accordingly, the coordinate estimation device proposed in the embodiment of Fig. 6 is equipped with two inertial sensors, and adopts a method of reading and analyzing measurement values ​​by attaching them to the main PCB and the camera PCB, respectively.

[0064] Referring to Figure 6, when mounting the terminal device on the vehicle, IMU Sensor #2 is set to Zero Initial. This makes IMU Sensor #2 the 3-axis reference line. Then, the camera angle is turned in the direction to be measured, and the software reads the data from IMU Sensor #1 to measure the camera's 3-axis (pitch, roll, yaw) values. Now, the measured values ​​can be calibrated based on IMU Sensor #2 and provided to the object distance measurement algorithm.

[0065] In summary, a coordinate estimation method according to another embodiment of the present specification may further include a process for correcting an error due to a difference between a motion measurement standard of a moving object and a motion measurement standard of an image sensor. More specifically, in this process, a motion measurement standard is set through a second inertial sensor fixed in correspondence with the moving object, and a motion measurement standard is set through a first inertial sensor fixed in correspondence with the image sensor. At this time, the image sensor can adjust the orientation direction or angle. Then, the motion information detected through the first inertial sensor can be corrected based on the second inertial sensor by using the difference between the motion measurement standards set through each of the inertial sensors. That is, the view angle of the camera is corrected based on the behavior of the vehicle.

[0066] FIG. 7 is a flowchart illustrating in more detail the process of calculating satellite coordinates of an object (step S370) in the embodiment of FIG. 3 for estimating coordinates of an object existing in space.

[0067] In step S371, the satellite coordinates of the mobile object can be corrected by considering the change in the position of the mobile object from the time the satellite signal is received to the time the object is detected. In this process, the satellite coordinates of the mobile object at the time of object detection can be calculated from the satellite coordinates acquired immediately before by using the time difference between the time the satellite signal is received and the time the object is detected, as well as the moving speed and azimuth of the mobile object. Below, a method for calculating the distance between two points before and after movement and the meaning of the parameters used in the description are specifically introduced.

[0068] Parameter Meaning lat1 Latitude of the initial mobile location lat2 Latitude of the current mobile location Δlat Latitude difference between two points lon1 Longitude of the initial mobile location lon2 Longitude of the current mobile location Δlon Longitude difference between two points t0 Satellite signal reception time (location stamp) t1 Object detection time Δt Movement time (t1-t0)

[0069] The total distance traveled can be calculated by multiplying the travel time Δt and the current speed s.

[0070] The distance between the latitude and longitude coordinates of two points can be calculated using the Haversine formula in the following mathematical equations 1 and 2.

[0071]

[0072]

[0073] The process of calculating the latitude and longitude of a new point by considering the bearing between two points is as follows.

[0074] First, convert the azimuth to radians using mathematical formula 3.

[0075]

[0076] Then, the latitude (lat2) and longitude (lon2) of the new point are calculated using Equations 4 and 5, respectively.

[0077]

[0078]

[0079] Now, in step S373, the satellite coordinates of the object can be calculated using the satellite coordinates of the mobile object itself corrected through step S371, and the distance and direction between the mobile object and the object.

[0080] FIG. 8 is a drawing for explaining a process of identifying the location of a hazardous material based on a moving object and deriving the coordinates of the hazardous material in a coordinate estimation method according to one embodiment of the present specification.

[0081] The method for calculating the location of an object (hazardous material) within an image based on a moving object and the meaning of the parameters used in the description are specifically introduced as follows.

[0082] Parameter Meaning φ1 x-coordinate value of the position of the moving object (terminal) φ2 x-coordinate value of the position of the object (hazard rule) λ1 y-coordinate value of the position of the moving object λ2 y-coordinate value of the position of the object r radius of the Earth d x x-axis distance between the moving object and the object y Y-axis distance between the moving object and the object p1 Current position of the moving object p2 Position of the object

[0083] Using the following mathematical expressions 6 and 7, p2(x)(=φ2) can be obtained.

[0084]

[0085]

[0086] Using the following mathematical expressions 8 and 9, p2(y)(=λ2) can be obtained.

[0087]

[0088]

[0089] Now, we can correct the position p2(x,y) of the object (hazard) by combining p2(x) and p2(y).

[0090] FIG. 9 is a block diagram illustrating a device for estimating the coordinates of an object existing in space according to one embodiment of the present specification, and is a reconstruction of the coordinate estimation method of FIG. 3 from the perspective of hardware configuration. Therefore, to avoid redundancy in explanation, only an outline of the operation and function of each component is briefly described herein.

[0091] The coordinate estimation device (900) includes a receiver (10), an image sensor (20), and a processor (50). In addition, it may include at least one inertial sensor (30) adjacent to the image sensor (20).

[0092] The receiver (10) is configured to receive satellite signals and obtain the satellite coordinates of the moving object, and may be, for example, a GPS satellite signal receiver.

[0093] The image sensor (20) is configured to receive an image of space, and can be, for example, a camera.

[0094] The processor (50) is configured to estimate the coordinates of an object existing in space, and the processor calculates position information between the moving object and the object detected in the image based on the relationship between the floor coordinates in the image and the pixel coordinates of the detection area, and executes a command to calculate the satellite coordinates of the object using the satellite coordinates of the moving object itself and the position information between the moving object and the object.

[0095] In addition, the processor (50) may perform a command to derive a perspective transformation for the detection area from a relationship between the floor coordinates in the image and the pixel coordinates of the detection area based on the position of the image sensor that acquires the image and the movement information detected through the first inertial sensor, to derive floor coordinates corresponding to an object detected in the image using the derived perspective transformation, and to derive position information between the moving body and the object detected in the image using the floor coordinates derived corresponding to the object and the distance to the object.

[0096] Meanwhile, two inertial sensors may be utilized to correct errors due to differences between the movement measurement standards of a moving object and the movement measurement standards of an image sensor. To this end, the coordinate estimation device (900) may further include a first inertial sensor that is fixed in correspondence with the image sensor and sets a movement measurement standard, and a second inertial sensor that is fixed in correspondence with the moving object and sets a movement measurement standard. At this time, the processor (50) may perform a command to eliminate errors caused by adjustment of the orientation direction of the image sensor by correcting the movement information detected through the first inertial sensor based on the second inertial sensor by utilizing the difference between the movement measurement standards set through each of the inertial sensors.

[0097] Furthermore, the processor (50) may perform a command to correct the satellite coordinates of the mobile object by considering the change in the position of the mobile object from the time of receiving the satellite signal to the time of detecting the object, and to calculate the satellite coordinates of the object by using the corrected satellite coordinates of the mobile object and the distance and direction between the mobile object and the object.

[0098] Embodiments according to the present specification may be implemented by various means, for example, hardware, firmware, software, or a combination thereof. In the case of hardware implementation, an embodiment of the present specification may be implemented by one or more application specific integrated circuits (ASICs), digital signal processors (DSPs), digital signal processing devices (DSPDs), programmable logic devices (PLDs), field programmable gate arrays (FPGAs), processors, controllers, microcontrollers, microprocessors, etc. In the case of firmware or software implementation, an embodiment of the present specification may be implemented in the form of a module, procedure, function, etc. that performs the capabilities or operations described above. Software code may be stored in a memory and executed by a processor. The memory may be located inside or outside the processor and may exchange data with the processor by various means already known in the art.

[0099] Meanwhile, the embodiments of the present specification can be implemented as computer-readable codes on a computer-readable recording medium. Computer-readable recording media include all types of recording devices that store data that can be read by a computer system. Examples of computer-readable recording media include ROMs, RAMs, CD-ROMs, magnetic tapes, floppy disks, optical data storage devices, etc. In addition, the computer-readable recording media can be distributed across network-connected computer systems, so that the computer-readable codes can be stored and executed in a distributed manner. In addition, functional programs, codes, and code segments for implementing the embodiments can be easily inferred by programmers in the technical field to which the present specification pertains.

[0100] One or more non-transitory computer-readable media storing one or more instructions according to one embodiment, the one or more instructions being executable by one or more processors, wherein the one or more instructions estimate coordinates of an object existing in space, wherein the one or more instructions receive a satellite signal to obtain satellite coordinates of the mobile object itself, receive an image of the space, and calculate position information between the mobile object and an object detected in the image based on a relationship between a floor coordinate in the image and a pixel coordinate of a detection area, and calculate satellite coordinates of the object using the satellite coordinates of the mobile object itself and the position information between the mobile object and the object.

[0101] The present disclosure has been described above, focusing on various embodiments thereof. Those skilled in the art will appreciate that various embodiments may be modified without departing from the essential characteristics of the present disclosure. Therefore, the disclosed embodiments should be considered illustrative rather than restrictive. The scope of the present disclosure is set forth in the claims, not the foregoing description, and all differences within the scope equivalent thereto should be construed as being encompassed by the present disclosure.

[0102] According to the embodiments of the present specification described above, by first estimating the satellite coordinates of the point where the vehicle has moved since receiving the last satellite signal and secondly estimating the satellite coordinates of the actual hazard based on the current location of the vehicle, real-time satellite coordinates of the hazard on the road in the image acquired through the camera can be provided, positional errors due to vehicle driving after receiving the satellite signal are eliminated, and as a result, effective management of road hazards and accurate sharing of hazard information are possible.

Claims

1. A method for estimating the coordinates of an object existing in space, A step in which a mobile body receives a satellite signal and obtains its own satellite coordinates; A step in which the above moving body receives an image of space; A step of calculating position information between the mobile body and an object detected in the image based on the relationship between the floor coordinates in the image and the pixel coordinates of the detection area; and A coordinate estimation method, comprising a step of calculating satellite coordinates of the object using the satellite coordinates of the mobile object itself and the position information between the mobile object and the object.

2. In paragraph 1, The step of calculating position information between the above-mentioned moving object and the object detected in the image is as follows: A step of deriving a perspective transformation for the detection area from the relationship between the floor coordinates in the image and the pixel coordinates of the detection area based on the position of the image sensor that acquires the image and the movement information detected through the first inertial sensor; A step of calculating floor coordinates corresponding to an object detected in the image using the derived perspective transformation; and A coordinate estimation method, comprising a step of calculating position information between the moving object and the object detected in the image using the floor coordinates calculated in response to the object and the distance to the object.

3. In paragraph 2, The steps for deriving the above perspective transformation are: A coordinate estimation method for establishing a relationship between the floor coordinates in the image and the pixel coordinates of the detection area based on the installation height of the image sensor and the pitch and roll information detected through the first inertial sensor corresponding to the image sensor.

4. In paragraph 2, The steps for deriving the above perspective transformation are: A coordinate estimation method for calculating coordinates for a detection area within the image based on the installation height of the image sensor and pitch and roll information detected through the first inertial sensor, and calculating coordinates for the detection area rotated by roll to determine a perspective transformation matrix based on the image sensor.

5. In paragraph 1, A coordinate estimation method further comprising a step of correcting an error caused by a difference between a movement measurement standard of the above-mentioned moving object and a movement measurement standard of the above-mentioned image sensor.

6. In paragraph 5, The step of correcting the above error is: A step of setting a movement measurement standard through a second inertial sensor fixed in response to the above moving body; A step of setting a movement measurement standard through a first inertial sensor fixed in response to the image sensor; and A coordinate estimation method, comprising a step of correcting movement information detected through the first inertial sensor based on the second inertial sensor by using the difference between movement measurement criteria set through each of the inertial sensors.

7. In paragraph 1, The step of calculating the satellite coordinates of the above object is: A step of correcting the satellite coordinates of the mobile object by considering the change in the position of the mobile object from the time of receiving the satellite signal to the time of detecting the object; and A coordinate estimation method comprising a step of calculating satellite coordinates of the object using the corrected satellite coordinates of the mobile object itself, and the distance and direction between the mobile object and the object.

8. In paragraph 7, The step of correcting the satellite coordinates of the above-mentioned mobile body is as follows: A coordinate estimation method for calculating satellite coordinates of a moving object at the time of object detection from satellite coordinates acquired immediately before, using the time difference between the time of receiving the satellite signal and the time of detecting the object, and the moving speed and azimuth of the moving object.

9. In one or more non-transitory computer-readable media storing one or more instructions, The one or more instructions executable by one or more processors are configured to estimate coordinates of an object existing in space, Receive satellite signals to obtain the satellite coordinates of the mobile device, Input an image of the space, Based on the relationship between the floor coordinates in the image and the pixel coordinates of the detection area, position information between the moving object and the object detected in the image is calculated, A computer-readable medium for calculating satellite coordinates of an object by using the satellite coordinates of the mobile object itself and the position information between the mobile object and the object.

10. A receiver that receives satellite signals and obtains the satellite coordinates of the mobile device; An image sensor that receives an image of the space; and Includes a processor that estimates the coordinates of an object existing in space, The above processor, A coordinate estimation device that calculates position information between the mobile body and an object detected in the image based on the relationship between the floor coordinates in the image and the pixel coordinates of the detection area, and performs a command to calculate satellite coordinates of the object using the satellite coordinates of the mobile body itself and the position information between the mobile body and the object.

11. In paragraph 10, The above processor, Based on the position of the image sensor that acquires the image and the movement information detected through the first inertial sensor, a perspective transformation for the detection area is derived from the relationship between the floor coordinates in the image and the pixel coordinates of the detection area, Using the derived perspective transformation, the floor coordinates corresponding to the object detected in the image are calculated, A coordinate estimation device that executes a command to calculate position information between the moving object and the object detected in the image using the floor coordinates calculated in response to the object and the distance to the object.

12. In paragraph 10, A first inertial sensor that is fixed in response to the image sensor and sets a movement measurement standard; and Further comprising a second inertial sensor that is fixed in response to the above moving body and sets a movement measurement standard, The above processor, A coordinate estimation device that performs a command to eliminate an error caused by adjusting the orientation direction of the image sensor by correcting the motion information detected through the first inertial sensor based on the second inertial sensor by using the difference between the motion measurement standards set through each of the inertial sensors.

13. In paragraph 10, The above processor, Correct the satellite coordinates of the mobile object by considering the change in the position of the mobile object from the time of receiving the satellite signal to the time of detecting the object, A coordinate estimation device that executes a command to calculate the satellite coordinates of the object using the corrected satellite coordinates of the mobile object itself, the distance and direction between the mobile object and the object.

Images