Image processing training data generation method, image processing method and device

Abstract

Disclosed are a method for generating training data for image processing, an image processing method and an apparatus thereof. The image processing method and the image processing apparatus detect an object from a driving image of a vehicle, obtain information about the height difference between the vehicle and the object, and input the image domain coordinates of the object in the driving image and the information about the height difference into a neural network, and determine the world domain coordinates of the object.

Description

[0001] This application claims the priority benefit of Korean Patent Application No. 10-2018-0109024 filed on September 12, 2018, in the Korean Intellectual Property Office, the disclosure of which is incorporated herein by reference in its entirety. Technical Field

[0002] The disclosed embodiments relate to a training data generation method for image processing, an image processing method and an apparatus thereof. Background Art

[0003] Recognition and detection of objects for autonomous driving can be performed through the driving images of the vehicle. At this time, nonlinear transformation through homography operation can be used to reconstruct the two-dimensional (2D) image domain coordinates into three-dimensional (3D) world domain coordinates.

[0004] However, the conversion of 2D image domain coordinates to 3D world domain coordinates through homography operations may be incorrect, and large errors may occur when lanes and objects are detected and the position of the vehicle is estimated. Such errors lead to instability when the vehicle is driven. In particular, in light sections with highly varying road surfaces, the accuracy may be significantly degraded. Summary of the invention

[0005] According to one aspect of the disclosed embodiment, there is provided an image processing method, the image processing method comprising: detecting an object in a driving image; obtaining a height difference between a vehicle and the object; determining a world domain coordinate of the object by neural network processing of the image domain coordinates and the height difference of the object in the driving image; and controlling a vehicle on a road with respect to the object based on the world domain coordinates of the object.

[0006] The height difference may include: pitch information of the vehicle and vanishing line information in the driving image.

[0007] The image processing method may further include: tracking image domain coordinates of the object over time; and filtering the tracked image domain coordinates of the object and converting the type of the image domain coordinates of the object into a floating point.

[0008] The image processing method may further include performing scaling normalization on image domain coordinates of the object in the driving image based on vanishing line information in the driving image.

[0009] The objects include dynamic objects with mobility and stationary objects without mobility, and the neural network may include at least one of a first neural network for estimating world coordinates of the dynamic objects and a second neural network for estimating world coordinates of the stationary objects.

[0010] When the object is a dynamic object with mobility, the image processing method may further include: generating a real-time map corresponding to the dynamic object by using a result of converting the image domain coordinates of the dynamic object into world domain coordinates; and generating driving parameters of a vehicle for controlling a vehicle on a road with respect to the dynamic object by using the real-time map.

[0011] When the object is a stationary object without mobility, the image processing method may further include: generating a landmark map corresponding to the stationary object by using a result of converting the image domain coordinates of the stationary object into world domain coordinates; and determining at least one of a position and a route of a vehicle used to control a vehicle on a road with respect to the stationary object by using the landmark map.

[0012] The image processing method may further include: outputting world coordinates of the object.

[0013] The image processing method may further include obtaining a driving image captured by a camera installed in the vehicle.

[0014] According to one aspect of the disclosed embodiment, a training data generation method is provided, the training data generation method comprising: obtaining multiple image domain coordinates of multiple dynamic objects by tracking the multiple dynamic objects in a driving image; converting the image domain coordinates of a first dynamic object among the multiple dynamic objects into first world domain coordinates of the first dynamic object, wherein the first dynamic object is located within a predetermined matching distance from a vehicle; obtaining multiple second world domain coordinates of the multiple surrounding objects by tracking the multiple surrounding objects using a distance sensor; matching one of the multiple surrounding objects with the first dynamic object by comparing the first world domain coordinates with the multiple second world domain coordinates; and generating training data including the image domain coordinates of the first dynamic object and the second world domain coordinates of the matched surrounding object.

[0015] The step of converting the image domain coordinates of the first dynamic object into the first world domain coordinates may include: converting the initial image domain coordinates of the first dynamic object into the first world domain coordinates through a homography operation.

[0016] The training data generation method further includes: associating a first identifier (ID) with the first dynamic object; and associating a plurality of second IDs with the plurality of surrounding objects. The step of matching one of the plurality of surrounding objects with the first dynamic object may include: matching a second ID associated with one of the plurality of surrounding objects among the plurality of second IDs with the first ID associated with the first dynamic object.

[0017] The plurality of dynamic objects may include at least one of surrounding vehicles, pedestrians, and animals.

[0018] The training data generating method may further include: tracking the plurality of image domain coordinates of the plurality of dynamic objects over time; and converting the types of the plurality of image domain coordinates of the plurality of dynamic objects into floating points by filtering the tracked plurality of image domain coordinates of the plurality of dynamic objects.

[0019] The training data generating method may further include performing scaling normalization on the plurality of image domain coordinates of the plurality of dynamic objects in the driving image based on vanishing line information in the driving image.

[0020] According to one aspect of the disclosed embodiment, a training data generating method is provided, the training data generating method comprising: storing a plurality of image domain coordinates of a stationary object by tracking the stationary object from a driving image including a plurality of frames over time; converting an image domain coordinate of a current frame among the plurality of image domain coordinates into a first global world domain coordinate based on global positioning system (GPS) information; obtaining a plurality of second global world domain coordinates of a plurality of surrounding objects based on an output of a range sensor and the GPS information; matching one of the plurality of surrounding objects with the stationary object by comparing the first global world domain coordinate with the plurality of second global world domain coordinates; and generating a plurality of training data, wherein each of the plurality of training data comprises one of the plurality of stored image domain coordinates and a plurality of second world domain coordinates of the matched surrounding object.

[0021] The training data generation method further includes: associating a first ID with a stationary object; and associating a plurality of second IDs with the plurality of surrounding objects. The step of matching one of the plurality of surrounding objects with the stationary object may include: matching a second ID provided to one of the plurality of surrounding objects with a first ID provided to the stationary object.

[0022] The stationary object may include at least one of a building, a sign, a traffic light, a crosswalk, a stop line, and a driving lane included in the driving image.

[0023] The training data generating method may further include: tracking the plurality of image domain coordinates of the stationary object over time; and converting a type of the plurality of image domain coordinates of the stationary object into a floating point by filtering the tracked plurality of image domain coordinates of the stationary object.

[0024] The training data generating method may further include performing scaling normalization on the plurality of image domain coordinates of the stationary object in the driving image based on vanishing line information in the driving image.

[0025] The training data generating method may further include cumulatively storing the output of the distance sensor and the GPS information.

[0026] According to one aspect of the disclosed embodiment, there is provided an image processing device, comprising: a processor for detecting an object in a driving image; obtaining a height difference between a vehicle and the object; determining a world domain coordinate of the object by neural network processing of the image domain coordinates and the height difference of the object in the driving image; and controlling a vehicle on a road with respect to the object based on the world domain coordinates of the object.

[0027] The height difference may include: pitch information of the vehicle and vanishing line information in the driving image.

[0028] The processor tracks image-domain coordinates of the object over time and filters the tracked image-domain coordinates of the object to convert the type of the image-domain coordinates of the object to floating point.

[0029] The processor may perform scaling normalization on image domain coordinates of objects in the driving image based on vanishing line information in the driving image.

[0030] The objects may include dynamic objects with mobility and stationary objects without mobility, and the neural network may include at least one of a first neural network for estimating world coordinates of the dynamic objects and a second neural network for estimating world coordinates of the stationary objects.

[0031] When the object is a dynamic object with mobility, the processor generates a real-time map corresponding to the dynamic object by using the result of converting the image domain coordinates of the dynamic object into world domain coordinates, and can generate driving parameters of a vehicle for controlling the vehicle on the road for the dynamic object by using the real-time map.

[0032] When the object is a stationary object without mobility, the processor generates a landmark map corresponding to the stationary object by using the result of converting the image domain coordinates of the stationary object into world domain coordinates, and can determine at least one of the position and route of a vehicle used to control a vehicle on a road with respect to the stationary object by using the landmark map.

[0033] The processor may output world coordinates of the object corresponding to the object.

[0034] The image processing apparatus may further include a camera installed in the vehicle to capture the driving image.

[0035] According to one aspect of the disclosed embodiment, a training data generating device is provided, the training data generating device comprising: a processor for obtaining a plurality of image domain coordinates of a plurality of dynamic objects by tracking the plurality of dynamic objects in a driving image; converting the image domain coordinates of a first dynamic object located within a predetermined matching distance from a vehicle among the plurality of dynamic objects into first world domain coordinates of the first dynamic object; obtaining a plurality of second world domain coordinates of the plurality of surrounding objects by tracking the plurality of surrounding objects using a distance sensor; matching one of the plurality of surrounding objects with the first dynamic object by comparing the first world domain coordinates with the plurality of second world domain coordinates; and generating training data including the image domain coordinates of the first dynamic object and the second world domain coordinates of the matched surrounding object.

[0036] According to one aspect of the disclosed embodiment, a training data generating device is provided, the training data generating device comprising: a processor for storing a plurality of image domain coordinates of a stationary object by tracking the stationary object from a driving image including a plurality of frames over time; converting the image domain coordinates of a current frame among the plurality of image domain coordinates into a first global world domain coordinate based on global positioning system (GPS) information; obtaining a plurality of second global world domain coordinates of a plurality of surrounding objects based on an output of a range sensor and the GPS information; matching one of the plurality of surrounding objects with the stationary object by comparing the first global world domain coordinate with the plurality of second global world domain coordinates; and generating a plurality of training data, wherein each of the plurality of training data comprises one of the plurality of stored image domain coordinates and the second world domain coordinate of the matched surrounding object. BRIEF DESCRIPTION OF THE DRAWINGS

[0037] The disclosed embodiments will be more clearly understood from the following detailed description in conjunction with the accompanying drawings:

[0038] Figure 1 is a diagram illustrating a method of reconstructing two-dimensional (2D) image domain coordinates of a driving image into three-dimensional (3D) world domain coordinates according to a disclosed embodiment;

[0039] Figure 2 is a flowchart illustrating an image processing method according to a disclosed embodiment;

[0040] Figure 3A is a diagram showing information on height difference according to a disclosed embodiment;

[0041] Figure 3B is a diagram illustrating a method of obtaining information about a vanishing line according to a disclosed embodiment;

[0042] Figure 4 is a diagram showing a configuration of an image processing apparatus according to a disclosed embodiment;

[0043] Figure 5 is a diagram showing the structure of a neural network according to a disclosed embodiment;

[0044] Fig. 6A is a diagram showing a driving image including vanishing lines;

[0045] Figure 6B is a diagram illustrating scaling normalization of an X-axis according to a disclosed embodiment;

[0046] Fig. 7A is a diagram showing another driving image including vanishing lines;

[0047] Figure 7B is a diagram illustrating scaling normalization of the Y-axis according to a disclosed embodiment;

[0048] Figure 8 is a diagram illustrating conversion of floating points according to a disclosed embodiment;

[0049] Fig. 9 is a flowchart illustrating an image processing method according to a disclosed embodiment;

[0050] Fig.10 is a flow chart illustrating a method for generating training data based on coordinates of dynamic objects in a driving image according to a disclosed embodiment;

[0051] Fig.11 is a configuration diagram of a training data generating apparatus for a dynamic object according to a disclosed embodiment;

[0052] Fig. 12A is a diagram showing a driving image captured by a vehicle moving on a road;

[0053] Fig. 12B is a diagram illustrating a method of generating training data for dynamic objects in a driving image according to a disclosed embodiment;

[0054] Fig.13 is a diagram illustrating a method of cumulatively generating training data according to a disclosed embodiment;

[0055] Fig.14 is a flow chart illustrating a method for generating training data based on coordinates of stationary objects in a driving image according to a disclosed embodiment;

[0056] Fig.15 is a configuration diagram of a training data generating apparatus for a stationary object according to a disclosed embodiment;

[0057] Fig.16 is a diagram illustrating a method of generating training data about a stationary object in a driving image according to a disclosed embodiment; and

[0058] Fig.17 is a block diagram of an image processing apparatus according to a disclosed embodiment. DETAILED DESCRIPTION

[0059] The description of specific structures or functions disclosed in the current specification is provided in order to describe the embodiments according to the described concepts. The disclosed subject matter can be implemented in many different forms and should not be construed as limited to the embodiments set forth herein.

[0060] Although terms such as "first", "second", etc. may be used to describe various components, such components are not limited to the above terms. The above terms are only used to distinguish one component from another component. For example, in the absence of conflict, a first component may indicate a second component, or a second component may indicate a first component.

[0061] When a component is referred to as being "connected" to another component, the component may be directly connected to the other component. However, it is understood that different components may be inserted.

[0062] Unless the context otherwise defines, singular expressions include plural expressions. The terms "include" or "may include" used herein in various example embodiments may indicate the presence of a corresponding function, operation, or component, and do not limit one or more additional functions, operations, or components. It will also be understood that when the terms "include" and / or "comprise" are used in this specification, the terms "include" and / or "comprises" may be used to specify the presence of stated features, wholes, steps, operations, elements, and / or components, but do not exclude the presence or addition of one or more other features, wholes, steps, operations, elements, components, and / or groups thereof.

[0063] When a particular embodiment can be implemented differently, a particular processing order can be performed differently from the order described. For example, two consecutively described processes can be performed substantially simultaneously or in an order opposite to the order described. For example, variations in the shapes of the illustrations due to manufacturing techniques and / or tolerances are expected. Therefore, the disclosed embodiments should not be interpreted as limited to the specific shapes of the regions shown herein, but will include deviations in shapes caused by, for example, manufacturing.

[0064] The embodiments to be described below may be used to display lanes in an augmented reality navigation system (such as a smart vehicle) or to generate visual information for assisting in maneuvering an autonomous vehicle. In addition, the embodiments may be used to interpret visual information and assist in stable and comfortable driving in a device including an intelligent system such as a head-up display (HUD), which is provided in a vehicle for driving assistance or to complete autonomous driving. The embodiments may be used in autonomous vehicles, smart vehicles, smart phones, and mobile devices. Hereinafter, the embodiments will be described in detail with reference to the accompanying drawings. The same reference numerals denote the same components.

[0065] In the following, a "road" may be a highway, a state road, a local road or a national highway on which a vehicle is driven. A road may include one or more lanes. A "driving lane" may correspond to a lane used by a driven vehicle among a plurality of lanes. "Lanes" may be distinguished from each other by lane markings displayed on the road surface. Lanes may be defined by lane markings on the right and left sides of the road. "Road markings" on the road surface on which a vehicle is driven may include lane markings (such as center lines or taxiway lines), symbols (such as lane change lines, no left turns, forward direction guide lines or pedestrian crossings) or non-lane markings such as characters (such as child protection areas or speed reduction, etc.).

[0066] Figure 1 is a view illustrating a method of reconstructing two-dimensional (2D) image-domain coordinates of a driving image into three-dimensional (3D) world-domain coordinates according to a disclosed embodiment.

[0067] Reference Figure 1 , a 2D driving image 110 of a vehicle and a 3D image 130 corresponding to the 2D driving image 110 are shown. The 3D image 130 may be a top-view image on a world domain, and may include depth information.

[0068] The vehicle's detection system detects vehicles, people, traffic lights, signs, lanes, and road conditions. The vehicle avoids collisions by detecting surrounding vehicles, intersections, and road markings including lanes, and can perform route search and vertical and horizontal direction control by recognizing and / or detecting signs and traffic lights. Hereinafter, "vehicle" may include autonomous driving functions and / or advanced driver assistance (ADA) functions.

[0069] During road driving, the 2D driving image 110 may be captured by a capture device. The capture device may be mounted on the front of the vehicle, the side of the vehicle, the top of the vehicle, the bottom of the vehicle, the rear of the vehicle, or any one or any combination of all of the above. The 2D driving image 110 may include various surrounding objects, such as surrounding vehicles 111, lanes 113, street lights 115, and crosswalks 117.

[0070] The image processing device according to the embodiment may convert the image domain coordinates of the surrounding objects detected by analyzing the 2D driving image 110 into 3D world domain coordinates. The image processing device may employ a neural network to convert the image domain coordinates of the surrounding objects detected by analyzing the 2D driving image 110 into 3D world domain coordinates. Similar to their indications in the 2D driving image 110, surrounding vehicles 131, lanes 133, street lights 135, and crosswalks 137 may be displayed in the 3D image 130.

[0071] The image processing device can control the vehicle to maintain the distance between vehicles while driving by associating dynamic objects detected from the image domain of the 2D driving image 110 to the 3D world domain. In addition, the image processing device can control the vehicle to maintain the lane and generate a route while driving by converting stationary objects detected from the 2D image domain to the 3D world domain and estimating the position of the vehicle.

[0072] In addition, the image processing apparatus according to an embodiment may reduce the load of a neural network by tracking and detecting a partial object (eg, a vehicle, a road, a sign, etc.) as a target without processing the 2D driving image 110 .

[0073] Hereinafter, for convenience, “2D image domain (coordinates)” will be simply expressed as “image domain (coordinates)” and “3D world domain (coordinates)” will be simply expressed as “world domain (coordinates)”.

[0074] Figure 2 is a flowchart illustrating an image processing method according to a disclosed embodiment.

[0075] Reference Figure 2 In operation S210, the image processing device according to the embodiment detects an object from a driving image of a vehicle. The driving image may be obtained by a capture device installed in the vehicle during driving of the vehicle to capture a front view, a side view, and other views from the perspective of the vehicle. Optionally, the driving image may be acquired through a communication interface (refer to Fig.17 At least one external image among a front view and a side view of the vehicle received from an external source (eg, a traffic camera) by the communication interface 1770 .

[0076] Similar to Figure 1 2D driving image 110 shown in FIG. 1 , the driving image may include a road image including surrounding vehicles, lanes, curbs, sidewalks, and surrounding environment and / or road surface images. The driving image may include various images other than color images, such as infrared images, depth images, and stereo images. The driving image may include one frame, multiple frames, or a video.

[0077] The objects detected by the image processing device may be other vehicles, road vanishing points, road markings, pedestrians, traffic lights, signs, people, animals, plants, and buildings. However, the disclosed embodiments are not limited thereto. The objects may include dynamic objects with mobility and / or stationary objects without mobility. Dynamic objects may include various objects with mobility, such as surrounding vehicles, pedestrians, and animals. Stationary objects may include various objects without mobility, such as various lanes (such as crosswalks, stop lines, and driving lanes), road markings, curbs, buildings, signs, plants (trees), lights, and traffic lights.

[0078] In operation S210, the image processing apparatus according to an embodiment may detect an object from a driving image by using a convolutional neural network (CNN) pre-trained to recognize an object. In the CNN, for example, bounding boxes of lane displays and non-lane displays to be detected from the driving image and types of lane displays and non-lane displays to be detected from the driving image may be pre-trained.

[0079] In operation S220, the image processing apparatus obtains information about the height difference between the vehicle and the object. For example, the information about the height difference may include information about the pitch of the vehicle and information about the vanishing line in the driving image. Figure 3A and Figure 3B Details information about the height difference between the vehicle and the object.

[0080] In operation S230, the image processing device inputs the image domain coordinates of the object in the driving image and the information about the height difference to the neural network, and determines the world domain coordinates of the object. The neural network may determine the world domain coordinates of the object including both dynamic objects and stationary objects. Optionally, the neural network may include at least one of a first neural network for determining the world domain coordinates of the dynamic object and a second neural network for determining the world domain coordinates of the stationary object. Figure 4 and Figure 5 The configuration and operation of the image processing apparatus according to the embodiment are described in detail.

[0081] The image processing device may output the world coordinates corresponding to the object determined in operation S230. The image processing device may explicitly or implicitly output the world coordinates of the object. "Explicitly outputting the world coordinates of the object" may include, for example, displaying the world coordinates of the object on a screen (or map) and / or outputting the world coordinates of the object as audio. Alternatively, "implicitly outputting the world coordinates of the object" may include, for example, controlling a vehicle, determining a position of a vehicle, or setting or changing a route by using the world coordinates of the object.

[0082] According to an embodiment, the image processing apparatus may perform scaling normalization on the image domain coordinates of the object in the driving image based on the information about the vanishing lines in the driving image. 6A to 7B A method of performing scaling normalization performed by an image processing apparatus is described in detail.

[0083] Optionally, according to an embodiment, the image processing apparatus may track the image domain coordinates of the object over time. The image processing apparatus may convert the type of the image domain coordinates of the object into a floating point by filtering the image domain coordinates of the object being tracked over time. Figure 8 A method for converting the type of image domain coordinates into floating points performed by an image processing apparatus is described in detail.

[0084] According to an embodiment, the image processing apparatus determines whether the object detected in operation S210 is a dynamic object with mobility or a stationary object without mobility, and may perform different operations according to the determination result. Fig. 9 Embodiments of an image processing apparatus that distinguishes a dynamic object from a stationary object and detects the dynamic object and the stationary object are described in detail.

[0085] Figure 3A is a diagram illustrating information on a height difference according to a disclosed embodiment.

[0086] Reference Figure 3A , vehicle pitch information 310 and vanishing line information 320 are shown.

[0087] The vehicle pitch information 310 may correspond to information indicating a slope or height of the vehicle based on the ground. The vehicle pitch information 310 may be measured by, for example, an inertial measurement unit (IMU) sensor or a gyroscope sensor. The vehicle pitch information 310 may be represented as "p".

[0088] Vanishing line information 320 may indicate the height of the vanishing line of the object in front of the driving image or the vanishing point at the height where the objects in front of the driving image converge. Vanishing line information 320 may include the position of the vanishing point in the driving image (e.g., the y coordinate of the vanishing point). Vanishing line information 320 may be obtained from the driving image captured by a capture device (such as a camera).

[0089] The image processing device may obtain the vanishing line information 320 by recognizing the object in the driving image. The vanishing line information 320 may be represented as "vl".

[0090] Figure 3B is a diagram illustrating a method of obtaining information about vanishing lines according to a disclosed embodiment.

[0091] According to an embodiment, the image processing device may determine the vanishing line based on the highest point of the drivable road. Figure 3B , the image processing device extracts regions 341, 342, and 343 of a drivable road based on image processing (such as depth training), and may determine the y coordinate of the vanishing line 330 based on the highest point of the extracted region of the drivable road. When the y coordinate increases from the upper end of the image to the lower end of the image, the highest point of the extracted region may be a pixel having the smallest y coordinate among the pixels included in the extracted region. In addition, the drivable road region may be extracted based on neighboring vehicles.

[0092] According to an embodiment, the image processing device extracts a plurality of lanes, and may determine a point where the extension lines of the plurality of lanes intersect as a vanishing point. Figure 3B The image processing apparatus extracts two lanes 351 and 352 based on image processing (such as depth training), extends the two lanes 351 and 352, and may determine a vanishing point 335. The image processing apparatus may determine the y coordinate of the determined vanishing point 335 as the y coordinate of the vanishing line 330.

[0093] Figure 4 is a diagram showing a configuration of an image processing apparatus 400 according to a disclosed embodiment. Figure 5 is a diagram illustrating the structure of a neural network according to a disclosed embodiment.

[0094] Reference Figure 4 and Figure 5 , the image processing device 400 according to an embodiment may include a camera sensor 410, an IMU sensor 420, and a neural network 430.

[0095] The image processing device 400 may detect an object from the driving image of the vehicle captured by the camera sensor 410. In operation 415, the image processing device 400 tracks the detected object and may calculate the point coordinates (i x ,i y ) is input to the neural network 430. In addition, the image processing device 400 obtains vanishing line information vl in the driving image of the vehicle captured by the camera sensor 410, and may input the obtained vanishing line information to the neural network 430. The image processing device 400 may input the pitch information p of the vehicle sensed by the IMU sensor 420 to the neural network 430.

[0096] The neural network 430 receives the coordinates of a point in the image domain (i x ,i y ), the pitch information p of the current vehicle and the vanishing line information v1, estimate the world domain coordinates of the object based on the input information items, and output the point coordinates (i x ,i y ) corresponds to the world coordinates (W x , Wy ).like Figure 5 As shown in , neural network 430 may include fully connected layers.

[0097] Depending on the embodiment, the neural network 430 may be trained to distinguish between dynamic objects and stationary objects and estimate the world coordinates of the dynamic objects and the stationary objects, or the neural network 430 may be trained to estimate the world coordinates of all objects without distinguishing between dynamic objects and stationary objects.

[0098] The neural network 430 may determine information indicating a height difference between the vehicle and surrounding objects through pitch information and vanishing line information about the vehicle, and determine world coordinates of the objects.

[0099] FIG. 6A to FIG. 6B and FIG. 7A to FIG. 7B is a diagram illustrating scaling normalization according to a disclosed embodiment.

[0100] The image processing apparatus according to an embodiment may perform scaling normalization on image domain coordinates of an object in a driving image.

[0101] As described above, the vanishing line information may include the position of the vanishing point in the driving image. Scaling normalization may be performed in the x-axis direction and the y-axis direction based on the position of the vanishing point (or a specific area including the vanishing point). During scaling normalization in the x-axis direction, the x-coordinate of the vanishing point may be considered. During scaling normalization in the y-axis direction, the y-coordinate of the vanishing point may be considered.

[0102] According to an embodiment, the image processing apparatus may perform scaling normalization based on a predetermined result (e.g., an average vanishing point position) by considering the vanishing line information included in a plurality of frames as a whole. Alternatively, the image processing apparatus obtains the vanishing line information of each frame, and may perform scaling normalization based on the information about the vanishing line frame by frame.

[0103] In the following, reference is made to Fig. 6A and Figure 6B Describes how to perform scaling normalization in the x-axis direction, see Fig. 7A and Figure 7B Describes a method for performing scaling normalization in the y-axis direction. For example, Fig. 6A and Fig. 7A The driving image shown in may have a resolution of 1920 (width) x 1200 (length).

[0104] exist Fig. 6AIn the driving image of FIG. 6 , in a long-distance object located around the x-coordinate of the vanishing point 610, although the distance between pixels is very large on the image domain, the distance between pixels may decrease as the distance from the x-coordinate of the vanishing point 610 increases. Therefore, in one embodiment, the distance between pixels on the image domain in the long-distance object may be correctly represented by performing normalization in which a scaling ratio of an area close to the x-coordinate of the vanishing point 610 is different from a scaling ratio of an area far from the x-coordinate of the vanishing point 610.

[0105] In more detail, Fig. 6A As shown in , the vehicle is located at 960 pixels approximately in the middle of 1920 (width) pixels in the horizontal axis of the driving image, and the x-coordinate of the vanishing point 610 is located at 960 pixels. Figure 6B When image domain coordinates are represented as linearly increasing regardless of the distance between the distant object and the vanishing point 610, as in function 630 shown in FIG. 6 , the resolution representing the distance between the distant object and the vanishing point 610 may be reduced.

[0106] Therefore, according to the embodiment, Figure 6B As in the log scale function 650 shown in , the slope value of the scaling factor may increase toward the vanishing point 610 and may decrease as the distance from the vanishing point 610 increases; therefore, an area close to the vanishing point 610 and corresponding to a long distance may be represented in detail.

[0107] The image processing apparatus according to an embodiment may improve the resolution representing the distance between a distant object and a vanishing point 610 by performing scaling normalization on the x-coordinate of the image domain of the distant object in the driving image based on vanishing line information in the driving image, for example in the form of a logarithmic scaling function 650.

[0108] exist Fig. 7A In the driving image of FIG. 7 , based on 600 pixels as the y coordinate of the vanishing point 710, Fig. 7A The area above 600 pixels in the driving image corresponds to the sky. Since the sky will exist in the driving image regardless of the road or lane that significantly affects the driving of the vehicle, the sky may not be seriously considered when the coordinates of the object are switched. In one embodiment, the amount of calculation may be reduced by performing scaling normalization on the image domain coordinates of the object included in the area (600 pixels to 1200 pixels) other than the area corresponding to 0 pixels to 600 pixels based on the y coordinate of the vanishing point.

[0109] exist Fig. 7AIn the driving image of FIG. 7 , around 600 pixels of the y coordinate of the vanishing point 710, although the distance between pixels is very large in the image domain, the distance between pixels can decrease as the distance from the 600 pixels of the y coordinate of the vanishing point 710 increases. Without considering the above content, as Figure 7B When image domain coordinates are represented as linearly increasing regardless of the distance between a distant object and vanishing point 710, as in function 730 shown in FIG. 7 , the resolution representing the distance between the distant object and vanishing point 710 may be reduced.

[0110] Therefore, in one embodiment, if Figure 7B As in the logarithmic scaling function 750 shown in , the resolution representing the distance between distant objects and the vanishing point 710 can be improved by performing normalization, wherein the area close to the y-coordinate of the vanishing point 710 is scaled differently than the area as the distance from the y-coordinate of the vanishing point 710 increases.

[0111] Figure 8 is a diagram illustrating conversion of floating points according to a disclosed embodiment.

[0112] refer to Figure 8 , graphs 810 , 820 , and 830 representing image domain coordinates of a tracked object over time are shown.

[0113] For example, as shown in diagram 810, the image domain coordinates (i x0 ,i y0 ) may be in the form of an integer. When the image processing apparatus normalizes the image domain coordinates, the image domain coordinates in the form of an integer may be discretely represented.

[0114] In one embodiment, the type of the image-domain coordinates of the object may be converted to floating point by tracking the image-domain coordinates of the object over time and filtering the tracked image-domain coordinates of the object.

[0115] For example, as shown in FIG820, the image processing apparatus tracks the image domain coordinates of the object, and may calculate the image domain coordinates (i x1 ,i y1 ) for filtering (such as, ,in, ɑ is a filter coefficient). In addition, as shown in FIG830, the image processing device calculates the image domain coordinates (i x2 ,i y2 ) for filtering (such as, ), and the filtered image domain coordinates can be converted to floating point coordinates.

[0116] The image processing apparatus according to the embodiment performs scaling normalization on the image domain coordinates of the object converted into floating points based on vanishing line information in the driving image, and may input the scaling normalization-performed image domain coordinates into the neural network.

[0117] Fig. 9 is a flowchart illustrating an image processing method according to a disclosed embodiment.

[0118] Reference Fig. 9 In operation S910, the image processing apparatus according to the embodiment may detect an object from a driving image of a vehicle. In operation S920, the image processing apparatus may determine whether the detected object is a dynamic object with mobility or a stationary object without mobility. The image processing apparatus may determine whether the detected object is a dynamic object or a stationary object through various machine training or various neural networks.

[0119] In operation S930, when it is determined in operation S920 that the object is a dynamic object, the image processing apparatus may generate a real-time map corresponding to the dynamic object by using the first neural network (i.e., a result obtained by converting the image domain coordinates of the dynamic object into world domain coordinates by the first neural network).

[0120] In operation S940, the image processing device may generate driving parameters of the vehicle by using a real-time map or a dynamically updated map. The driving parameters may include a driving angle control parameter, an acceleration control parameter, a deceleration control parameter, and / or a turn signal control parameter. The driving parameters generated by the image processing device may be used to prevent the vehicle from colliding with another vehicle.

[0121] In operation S950, when it is determined in operation S920 that the object is a stationary object, the image processing apparatus may generate a landmark map corresponding to the stationary object by using the second neural network (i.e., a result obtained by converting the image domain coordinates of the stationary object into world domain coordinates by the second neural network).

[0122] In operation S960, the image processing device may determine at least one of a location and a global path of the vehicle by using a landmark map.

[0123] Hereinafter, a method of training a neural network performed by an image processing apparatus will be described.

[0124] The image processing device according to the embodiment can train a neural network (NN) via remote distance data obtained through a homography operation. In a homography operation, the coordinates of another plane are determined by a unified transformation relationship established between corresponding points of the projection when a plane is projected onto another plane. Generally, a homography operation has high reliability within a short distance. Therefore, the image processing device can use point coordinates of a 2D image domain matched within a short distance on a 3D world domain as initial training data.

[0125] Then, the image processing device may train the neural network while gradually increasing the acquisition distance of the training data. The acquisition distance of the training data is gradually increased to prevent the image domain coordinates of the 2D driving image (2D image domain) and the 3D world domain coordinates from being incorrectly matched.

[0126] When accumulating training data, the image processing apparatus according to an embodiment may correctly collect training data by providing an identifier (ID) to a dynamic object and / or a stationary object on an image domain of a 2D driving image, collecting sequential data, and matching the collected data with data collected by a distance sensor.

[0127] The image processing apparatus according to the embodiment may also perform automatic calibration of converting 2D image domain coordinates into 3D world domain coordinates by accumulating training data in real time and training a neural network using the accumulated training data, wherein the automatic calibration is used to calibrate the position or posture distortion of a camera sensor or a distance sensor.

[0128] Fig.10 is a flow chart illustrating a method of generating training data based on coordinates of dynamic objects in a driving image according to a disclosed embodiment.

[0129] Reference Fig.10 In operation S1010, the training data generating device according to the embodiment (hereinafter, referred to as "generating device") obtains the image domain coordinates of the dynamic object by tracking the dynamic object through analyzing the driving image. Fig.11 An example of a configuration of a training data generating device according to an embodiment is described in detail.

[0130] According to an embodiment, the training data generating device may track the image domain coordinates of the dynamic object over time. The training data generating device may convert the type of the image domain coordinates of the dynamic object into a floating point by filtering the tracked image domain coordinates of the dynamic object. Optionally, the training data generating device may perform scaling normalization on the image domain coordinates of the dynamic object in the driving image based on vanishing line information in the driving image.

[0131] In operation S1020, the training data generating device converts the image domain coordinates of the first dynamic object located within the predetermined matching distance among the multiple dynamic objects into the first world domain coordinates. In operation S1020, the training data generating device may convert the image domain coordinates of the first dynamic object into the first world domain coordinates through a homography operation. The matching distance may be, for example, 15m or 30m. The training data generating device may provide the first ID to the first dynamic object.

[0132] In operation S1030, the training data generating apparatus obtains the second world coordinates of the surrounding objects by tracking the surrounding objects using a distance sensor. The distance sensor may be, for example, a laser radar sensor or a radar sensor. At this time, the training data generating apparatus may provide the second ID to the surrounding objects.

[0133] In operation S1040, the training data generating apparatus matches one of the surrounding objects with the first dynamic object by comparing the first world coordinates with the second world coordinates. The training data generating apparatus compares and matches the second ID provided to one of the surrounding objects with the first ID provided to the first dynamic object.

[0134] In operation S1050, the training data generating apparatus generates training data including image domain coordinates of the first dynamic object and second world domain coordinates of the matched surrounding objects.

[0135] Will refer to Fig. 12A , Fig. 12B and Fig.13 A method of generating training data performed by a training data generating apparatus according to an embodiment is described in detail.

[0136] Fig.11 is a configuration diagram of a training data generating apparatus 1100 for a dynamic object according to a disclosed embodiment.

[0137] Reference Fig.11 According to an embodiment, the training data generating device 1100 may include a camera sensor 1110 , a distance sensor 1120 , an IMU sensor 1130 , and a processor 1170 .

[0138] The training data generating device 1100 may detect a dynamic object from a driving image of a vehicle captured by the camera sensor 1110. The training data generating device 1100 may obtain image domain coordinates of the dynamic object by tracking the dynamic object using the driving image (1140). The image domain coordinates may be (i x ,i y The training data generating apparatus 1100 may obtain the vanishing line information vl from the driving image captured by the camera sensor 1110.

[0139] The training data generating device 1100 may generate the image domain coordinates (i x ,i y ) is converted into a first world domain coordinate (1145). As described in detail below, the training data generating apparatus 1100 may convert the image domain coordinate (i ) of the first dynamic object located within the predetermined matching distance 1160 among the multiple dynamic objects through a homography operation. x ,i y ) is converted to first world domain coordinates. After the initial training of the neural network is completed, the training data generating device 1100 can increase the matching distance 1160 by using the previously trained neural network instead of the homography operation for converting the image domain coordinates.

[0140] At this time, the training data generating apparatus 1100 may provide an ID (eg, a first ID) to the first dynamic object.

[0141] In addition, the training data generating apparatus 1100 may obtain the second world domain coordinates of the surrounding objects by tracking the surrounding objects of the vehicle by using the distance sensor 1120 (1150). The second world domain coordinates may be (W x , W y ). At this time, the distance sensor 1120 may output the distance and angle between the vehicle and the surrounding objects. The training data generating device 1100 may provide an ID (eg, a second ID) to the tracked surrounding objects.

[0142] The training data generating device 1100 can generate training data 1177 by comparing the first world domain coordinates of the first dynamic object with the second world domain coordinates of the surrounding objects, and matching the first world domain coordinates of the first dynamic object with the second world domain coordinates of the surrounding objects within a short distance (e.g., within a matching distance) (1173).

[0143] At this time, the training data generating device 1100 may obtain the image domain coordinates (i x ,i y ) and vanishing line information v1 and obtain the second world domain coordinates (W ) of the surrounding objects from the distance sensor 1120 x , W y ), to accumulate training data. The training data generating device 1100 may obtain the pitch information p of the vehicle through the IMU sensor 1130, and may use the obtained pitch information p to generate training data 1177.

[0144] In the image domain coordinates of the first dynamic object (i x ,i yAfter initially training the neural network using training data generated by homography operations performed by the image domain, the training data generating device 1100 may generate training data while gradually increasing the matching distance to a long distance. For example, in one embodiment, after the initial training, when the image domain coordinates are converted to the first world domain coordinates, the previously trained neural network is used instead of the homography operation, and thus the matching distance may be increased. Fig. 12A and Fig. 12B A method of increasing the matching distance 1160 and generating training data performed by the training data generating apparatus 1100 according to an embodiment is described in detail.

[0145] Fig. 12A and Fig. 12B is a diagram illustrating a method of generating training data for dynamic objects in a driving image according to a disclosed embodiment.

[0146] Reference Fig. 12A , driving images 1210, 1220, and 1230 capturing a vehicle 1205 moving on a road are shown. Assume that the driving image 1210 is captured at a time point t=0 and the vehicle 1205 is located within the matching distance, and the driving image 1220 and the driving image 1230 are captured at a time point t=1 and a time point t=2, respectively, and the vehicle 1205 is located at a distance greater than the matching distance.

[0147] In addition, refer to Fig. 12B , by transforming the image domain coordinates (i x ,i y ) and the second world domain coordinates (W ) of the surrounding objects obtained by the distance sensor at the time point t=n, the time point t=n+1 and the time point t=n+2 x , W y ) The process of performing matching step by step to generate training data is shown.

[0148] For example, assuming that n=0, the training data generating device may convert the image domain coordinates (i x0 ,i y0 ) is converted to the first world coordinates (W x0 , W y0 ). At this time, the training data generating device can provide ID a to the vehicle 1205.

[0149] The training data generating device may obtain the second world domain coordinates of the surrounding objects obtained by the distance sensor at the time point t=0 ( ). At this time, the training data generating device may provide the ID to the surrounding objects. For example, the training data generating device may provide the ID b to the vehicle 1205 among the surrounding objects obtained by the distance sensor.

[0150] The training data generating device converts the second world domain coordinates of the surrounding objects ( ) and the first world domain coordinates (W x0 , W y0 ) and the closest world domain coordinate (W x0 , W y0 ) of surrounding objects (e.g., as a second world domain with coordinates ( ) of the surrounding objects)) is matched with the vehicle 1205. By performing the matching, the training data generating device determines that the dynamic object with ID=a tracked by the camera sensor is the same as the surrounding object with ID=b tracked by the distance sensor, and can generate training data ( ).

[0151] At this time, at the time point t=1 and the time point t=2, the matching performed at the time point t=0 within the matching distance may also be maintained as ID=a=b, and the vehicle 1205 is located within a distance greater than the matching distance. The training data generating device may generate the training data by using the second world domain coordinates ( ) and the first world domain coordinates (W x1 , W y1 ), to generate training data ( The training data generating device may generate training data at time point t=2 by the same method as that at time point t=1 ( ).

[0152] Although not shown in the drawings, according to an embodiment, the training data generating device may store the object tracking history of the camera sensor and the object tracking history of the distance sensor. In this case, the training data at the time point before the matching is performed can be additionally generated by using the object tracking history after the matching is performed.

[0153] Fig.13 is a diagram illustrating a method of cumulatively generating training data according to a disclosed embodiment.

[0154] Reference Fig.13 , training data items cumulatively generated by the training data generating apparatus according to the embodiment using different converters according to matching distances are shown.

[0155] The training data generating device according to the embodiment converts the image domain coordinates of the first dynamic object located within the first matching distance (e.g., 12m) among the multiple dynamic objects tracked through the driving image into the first world domain coordinates through the homography operator, and can generate training data 0 as the initial training data (iteration 0) based on the first world domain coordinates. Then, after training the neural network through training data 0, the training data generating device can cumulatively generate training data items 1 and 2 (e.g., training data 1 and training data 2 (iteration 1 and iteration 2)) while increasing the matching distance as the number of repetitions gradually increases (e.g., increases once and twice) and converting the image domain coordinates of the first dynamic object through the neural network. As shown in reference Fig. 12A and Fig. 12B As described above, after performing matching within the matching distance, training data can be generated despite deviation from the matching distance. Therefore, the neural network is trained to perform conversion at a matching distance greater than the first matching distance, and as iterations are performed, the matching accuracy within a long distance can gradually increase.

[0156] Fig.14 is a flow chart illustrating a method for generating training data based on coordinates of stationary objects in a driving image according to a disclosed embodiment. Fig.14 In operation S1410, the training data generating device (or the processor of the training data generating device) according to the embodiment stores the image domain coordinates of the stationary object by tracking the stationary object from the driving image including a plurality of frames over time. At this time, the training data generating device may provide the first ID to the tracked stationary object. According to the embodiment, in operation S1410, the training data generating device may track the image domain coordinates of the stationary object over time. The training data generating device may convert the type of the image domain coordinates of the stationary object into a floating point by filtering the image domain coordinates of the tracked stationary object. In addition, the training data generating device may perform scaling normalization on the image domain coordinates of the stationary object in the driving image based on the vanishing line information in the driving image.

[0157] In operation S1420, the training data generating apparatus converts image domain coordinates of a current frame among image domain coordinates into first global world domain coordinates based on global positioning system (GPS) information.

[0158] In operation S1430, the training data generating device obtains the second global world domain coordinates of the surrounding objects based on the output of the distance sensor and the GPS information. At this time, the training data generating device may provide the second ID to the surrounding objects. In addition, the training data generating device may cumulatively store the output of the distance sensor and the GPS information.

[0159] In operation S1440, the training data generating apparatus matches one of the surrounding objects with the stationary object by comparing the first global world domain coordinates with the second global world domain coordinates. The training data generating apparatus may match the second ID provided to one of the surrounding objects with the first ID provided to the stationary object.

[0160] In operation S1450, the training data generating device generates training data. At this time, each of the training data items may include a second global world-domain coordinate of a surrounding object matched with one of the image-domain coordinates stored in operation S1410.

[0161] Fig.15 is a configuration diagram of a training data generating apparatus 1500 for a stationary object according to a disclosed embodiment.

[0162] Reference Fig.15 , the training data generating device 1500 according to an embodiment may include a camera sensor 1510 , a distance sensor 1520 , a GPS sensor 1530 , an IMU sensor 1540 , and a processor 1560 .

[0163] The training data generating device 1500 may capture a driving image including a plurality of frames over time through the camera sensor 1510. The training data generating device 1500 tracks the image domain coordinates of the stationary object from the driving image over time (1550), and may store the tracked image domain coordinates. At this time, the training data generating device 1500 may provide an ID (e.g., a first ID) to the stationary object.

[0164] The training data generating device 1500 may convert the type of the image domain coordinates of the stationary object into floating points by filtering the image domain coordinates of the tracked stationary object. The training data generating device 1500 may perform scaling normalization on the image domain coordinates of the stationary object converted into floating points based on the vanishing line information in the driving image.

[0165] The training data generating device 1500 may obtain vanishing line information vl from a driving image captured by the camera sensor 1510.

[0166] The training data generating apparatus 1500 may convert the image domain coordinates of the current frame among the image domain coordinates of the stationary object into the first global world domain coordinates based on the GPS information sensed by the GPS sensor 1530 .

[0167] The training data generating apparatus 1500 may generate the training data based on the output (W) of the distance sensor 1520. x , W y ) and GPS information to obtain the second global world domain coordinates of the surrounding objects of the vehicle. At this time, the training data generating device can provide the second ID to the surrounding objects.

[0168] The training data generating device 1500 compares the first global world domain coordinates with the second global world domain coordinates, and may match the second ID provided to one of the surrounding objects with the first ID provided to the stationary object (1563). At this time, the training data generating device 1500 may match the second ID provided to one of the surrounding objects with the first ID provided to the stationary object.

[0169] The training data generating device 1500 may generate training data 1567 including second global world domain coordinates of surrounding objects matching one of the previously stored image domain coordinates. The training data generating device 1500 obtains pitch information p of the vehicle through the IMU sensor 1540 and may generate training data 1567 using the pitch information p.

[0170] Will refer to Fig.16 A method of generating training data by tracking image domain coordinates of a stationary object over time, performed by the training data generating apparatus 1500 according to an embodiment, is described in detail.

[0171] Fig.16 is a diagram illustrating a method of generating training data about a stationary object in a driving image according to a disclosed embodiment.

[0172] Reference Fig.16 , driving images 1610, 1620, and 1630 of a vehicle 1605 over time and graphs 1615, 1625, and 1635 showing GPS information about the vehicle 1605 and surrounding objects 1607 corresponding to the driving images 1610, 1620, and 1630 are shown. It is assumed that the stationary object 1603 included in the driving images 1610, 1620, and 1630 and the surrounding objects 1607 shown in the graphs 1615, 1625, and 1635 are the same object (e.g., a street lamp). In addition, it is assumed that the driving image 1610 is captured at a time point of t=n, and the driving image 1620 and the driving image 1630 are captured at a time point of t=n+1 and a time point of t=n+2, respectively.

[0173] The training data generating apparatus according to the embodiment may store the image domain coordinates (i ) of the stationary object 1603 by tracking the stationary object 1603 from the driving images 1610, 1620, and 1630 over time. x ,i y The training data generating device may cumulatively store the output of the distance sensor (eg, radar sensor / lidar sensor) over time ( ) and GPS information (GPS x , GPS y ).

[0174] Because the stationary object 1603 does not have mobility, its position is fixed. However, when the vehicle 1605 is far away from the stationary object 1603, the conversion of the image domain coordinates is incorrect, and therefore, matching is not performed. Therefore, after the training data generating device cumulatively stores the coordinates of the stationary object 1603 captured according to the movement of the vehicle 1605, when the vehicle 1605 is close enough to the surrounding object 1607 corresponding to the stationary object 1603 and thus matching is performed, training data on the cumulatively captured coordinates of the stationary object 1603 can be generated.

[0175] In more detail, the training data generating apparatus may obtain the image domain coordinates (i x0 ,i y0 ), and the obtained image domain coordinates (i x0 ,i y0 ). The training data generating device may provide ID=1 to the stationary object 1603.

[0176] The training data generating device may generate the image domain coordinates (i x0 ,i y0 ) is converted to the first world coordinates (W x0 , W y0 The training data generating device may be based on GPS information (GPS) obtained by a GPS sensor. x0 , GPS y0 ) The first world domain coordinates (W x0 , W y0 ) converted to first global world domain coordinates .

[0177] In addition, the training data generating device may be based on the output of the distance sensor at the time point t=n ( ) and GPS information (GPS x0 , GPS y0 ), to obtain the second global world domain coordinates of the surrounding object 1607 .

[0178] The training data generating device may convert the image domain coordinates (i x1 ,i y1 ) converted to first global world domain coordinates , and the image domain coordinates (i x2 ,i y2) converted to first global world domain coordinates .

[0179] In addition, the training data generating device may generate the training data based on the output of the distance sensor as in the diagram 1625 at the time point t=n+1 and the diagram 1635 at the time point t=n+2 in the same manner as at the time point t=n. ) and GPS information ((GPS x1 , GPS y1 ),GPS x2 , GPS y2 )), to obtain the second global world domain coordinates of the surrounding object 1607 ( , ). The training data generating device may cumulatively store the image domain coordinates as the output of the camera sensor, the output of the distance sensor, the GPS information, and the second global world domain coordinates of the surrounding objects 1607.

[0180] In the diagram 1615 corresponding to the time point t=n, the first global world domain coordinates of the stationary object 1603 obtained by the camera sensor are The second global world domain coordinates of the surrounding objects 1607 obtained by the distance sensor When the distance between the vehicle 1605 and the surrounding objects 1607 gradually decreases over time, the difference (error) decreases, and the difference (error) can be removed or substantially removed, so that the error is negligible at the time point t = n + 2.

[0181] The training data generating device compares the first global world domain coordinates of the stationary object 1603 at the time point t=n+2 when the vehicle 1605 is closest to the surrounding object 1607 (or the vehicle 1605 passes the stationary object 1603) with the second global world domain coordinates of the surrounding object 1607, and can match the stationary object 1603 with the surrounding object 1607. The training data generating device can match the stationary object 1603 with ID=1 with the surrounding object 1607. Through matching, the training data generating device can generate a training data set ( 、(i x1 ,i y1 , )、( )).

[0182] Fig.17 is a block diagram of an image processing apparatus 1700 according to a disclosed embodiment.

[0183] Reference Fig.17According to an embodiment, the image processing apparatus 1700 includes a processor 1730. The image processing apparatus 1700 may further include a sensor 1710, a memory 1750, a communication interface 1770, and a display 1790. The sensor 1710, the processor 1730, the memory 1750, the communication interface 1770, and the display 1790 may communicate with each other through a communication bus 1705.

[0184] Sensor 1710 may include, for example, a camera sensor, an image sensor, a visual sensor, an IMU sensor, a gyroscope sensor, an acceleration sensor, a GPS sensor, a geomagnetic sensor, a lidar sensor, a radar sensor, and an altitude measurement sensor. However, the disclosed embodiments are not limited thereto. The camera sensor, the image sensor, and / or the visual sensor may be installed in the vehicle and may capture a driving image of the vehicle. The IMU sensor, the gyroscope sensor, and / or the altitude measurement sensor may sense the pitch information of the vehicle. The lidar sensor and / or the radar sensor may sense the (local) world coordinates of an object. The GPS sensor may sense the global world coordinates of the vehicle.

[0185] The processor 1730 may execute the above Figures 1 to 16 At least one method described in the drawings or an algorithm corresponding to the at least one method described in the drawings. That is, under the control of the execution of the processor 1730, the various blocks shown in the drawings may be implemented as hardware or software. The processor 1730 may execute a program representing the various blocks shown in the drawings, and may control the image processing apparatus 1700. The program code executed by the processor 1730 may be stored in the memory 1750.

[0186] The processor 1730 may be formed of, for example, a central processing unit (CPU) or a graphic processing unit (GPU).

[0187] The memory 1750 may store information about the driving image and the height difference between the vehicle and the object. In addition, the memory 1750 may store the image domain coordinates of the object tracked over time. In addition, the memory 1750 may store the real-time map corresponding to the dynamic object and / or the landmark map corresponding to the static object generated by the processor 1730.

[0188] The memory 1750 may store the world coordinates of the object determined by the processor 1730 .

[0189] The world coordinates of the stationary object stored in the memory 1750 can be read from the memory 1750 to quickly grasp unchanged information (such as crosswalks, signs, lanes, and surrounding terrain) when the vehicle passes through the same area. Considering that when the vehicle is used for commuting, the vehicle moves on the same route by using the information pre-stored in the memory 1750, the image processing speed for determining the world coordinates of the stationary object can be improved and the processing load can be reduced. The memory 1750 can be a volatile memory or a non-volatile memory.

[0190] The communication interface 1770 may receive a driving image captured by an external source outside the image processing device 1700, such as a traffic camera or a camera mounted to another vehicle or structure, or receive information of various sensors and map information received from outside the image processing device 1700. According to an embodiment, the communication interface 1770 may transmit the world coordinates of the object determined by the processor 1730 to the outside of the image processing device 1700 or the display 1790.

[0191] The display 1790 may display the world coordinates of the object together with the driving image, or may display the world coordinates of the object separately. The display 1790 may display the world coordinates of the object as, for example, the position of the object in the map information or navigation image. For example, when the image processing device 1700 is embedded in a vehicle, the display 1790 may be formed by a head-up display (HUD) provided in the vehicle.

[0192] The disclosed embodiments may be implemented by hardware components, software components, and / or a combination of hardware components and software components. For example, the devices, methods, and components described in the embodiments may be implemented by using one or more general-purpose computers or special-purpose computers (e.g., processors, controllers, arithmetic logic units (ALUs), digital signal processors, microcomputers, field programmable gate arrays (FPGAs), programmable logic units (PLUs), microprocessors, or another specific device capable of executing and responding to instructions). The processing device may execute an operating system (OS) and one or more software applications executed on the OS. In addition, the processing device may access, store, manipulate, process, and generate data in response to the execution of the software. For simplicity, only one processing device is shown to be used. However, those skilled in the art may understand that the processing device may include multiple processing elements and / or multiple types of processing elements. For example, the processing device may include multiple processors or a processor and a controller. In addition, other processing configurations (such as parallel processors) are feasible.

[0193] Software may include computer programs, codes, instructions, or one or more combinations of computer programs, codes, and instructions that may configure a processing device to operate as desired or instruct a processing device, either individually or collectively. Software and / or data may be permanently or temporarily contained in a particular type of machine, component, physical device, virtual device, computer storage medium, or device, or permanently or temporarily contained in a transmitted signal wave to be interpreted by or provide instructions or data to a processing device. Software is distributed on computer systems connected via a network and may be stored and executed in a distributed manner. Software and data may be stored in one or more computer-readable recording media.

[0194] The method according to the embodiment can be implemented in the form of program instructions that can be executed by various computer units, and can be recorded in a computer-readable recording medium. The computer-readable recording medium may include program instructions, data files and data structures, or a combination of program instructions, data files and data structures. The program instructions recorded in the computer-readable recording medium are program instructions specially designed and configured for the embodiment, or may be program instructions known to software engineers. For example, the computer-readable recording medium may be: a magnetic medium (such as a hard disk, a floppy disk, or a tape); an optical medium (such as a compact disk read-only memory (CD-ROM) or a digital versatile disk (DVD)); a magneto-optical medium (such as an optical magnetic disk); or a hardware device (such as a ROM, a random access memory (RAM), or a flash memory) specially configured to store and execute program instructions. The program instructions may include a high-level language code that can be executed by a computer using an interpreter and a machine language code created by a compiler. The hardware device may be configured to operate as one or more software modules to perform the operation of the embodiment, and vice versa.

[0195] While the disclosed embodiments have been particularly shown and described, it will be understood that various changes in form and details may be made therein without departing from the spirit and scope of the claims.

Claims

1. An image processing method for controlling a vehicle on a road, the image processing method comprising: Detecting objects within driving images; Get the height difference between the vehicle and the object; Determine the world domain coordinates of the object by neural network processing of the image domain coordinates and height differences of the object in the driving image; as well as Control the vehicle on the road based on the object's world coordinates, The height difference includes vanishing line information in the driving image, and the vanishing line information includes the position of the vanishing point in the driving image. The image processing method further includes: performing scaling normalization on the image domain coordinates of the object in the driving image based on the vanishing line information in the driving image, wherein the slope value of the scaling factor increases toward the vanishing point and decreases as the distance from the vanishing point increases.

2. The image processing method according to claim 1, wherein: Height differences also include: Vehicle pitch information.

3. The image processing method according to claim 1, further comprising: Tracking the image domain coordinates of an object over time; as well as Filter the image domain coordinates of the object and convert the type of the image domain coordinates of the object to floating point.

4. The image processing method according to claim 1, wherein: The object is a dynamic object with mobility or a stationary object without mobility, and Wherein, the neural network comprises: A first neural network for estimating world coordinates of a dynamic object; and The second neural network is used to estimate the world coordinates of stationary objects.

5. The image processing method according to claim 1, wherein: The object is a dynamic object with mobility, and Wherein, the image processing method further includes: generating a real-time map corresponding to the dynamic object by using a result of converting the image domain coordinates of the dynamic object into the world domain coordinates; and By using the real-time map, driving parameters of a vehicle for controlling the vehicle on a road with respect to dynamic objects are generated.

6. The image processing method according to claim 1, wherein: The object is a stationary object with no mobility, and Wherein, the image processing method further includes: generating a landmark map corresponding to the stationary object by using a result of converting the image domain coordinates of the stationary object into the world domain coordinates; and By using the landmark map, at least one of a position and a route of a vehicle for controlling the vehicle on a road with respect to a stationary object is determined.

7. The image processing method according to claim 1, further comprising: Output the world coordinates of the object.

8. The image processing method according to claim 1, further comprising: A driving image captured by a camera installed in a vehicle is obtained.

9. A method for generating training data, comprising: Obtaining a plurality of image domain coordinates of the plurality of dynamic objects by tracking the plurality of dynamic objects in the driving image; converting image domain coordinates of a first dynamic object among the plurality of dynamic objects into first world domain coordinates of the first dynamic object, wherein the first dynamic object is located within a predetermined matching distance from the vehicle; obtaining a plurality of second world domain coordinates of the plurality of surrounding objects by tracking the plurality of surrounding objects using a range sensor; Matching one of the plurality of surrounding objects with a first dynamic object by comparing the first world coordinate with the plurality of second world coordinates; and generating training data comprising the image domain coordinates of the first dynamic object and second world domain coordinates of the matching surrounding objects, The training data generation method further comprises: performing scaling normalization on the multiple image domain coordinates of the multiple dynamic objects in the driving image based on vanishing line information in the driving image, The vanishing line information includes the position of the vanishing point in the driving image. The slope value of the scaling factor increases toward the vanishing point and decreases as the distance from the vanishing point increases.

10. The training data generation method according to claim 9, wherein: The step of converting the image domain coordinates includes: converting the initial image domain coordinates of the first dynamic object into the first world domain coordinates through a homography operation.

11. The training data generation method according to claim 9, further comprising: associating the first identifier with the first dynamic object; as well as associating a plurality of second identifiers with the plurality of surrounding objects, The step of matching one of the plurality of surrounding objects with the first dynamic object includes: matching a second identifier associated with one of the plurality of surrounding objects among the plurality of second identifiers with a first identifier associated with the first dynamic object.

12. The training data generation method according to claim 9, wherein: The plurality of dynamic objects include at least one of surrounding vehicles, pedestrians and animals.

13. The training data generation method according to claim 9, further comprising: tracking the plurality of image-domain coordinates of the plurality of dynamic objects over time; as well as By filtering the plurality of image domain coordinates of the tracked plurality of dynamic objects, types of the plurality of image domain coordinates of the plurality of dynamic objects are converted into floating points.

14. A method for generating training data, comprising: storing a plurality of image domain coordinates of a stationary object by tracking the stationary object over time from a driving image comprising a plurality of frames; Based on the global positioning system information, converting the image domain coordinates of the current frame among the plurality of image domain coordinates into a first global world domain coordinate; obtaining a plurality of second global world domain coordinates of a plurality of surrounding objects based on the output of the range sensor and the global positioning system information; matching a surrounding object of the plurality of surrounding objects with the stationary object by comparing the first global world domain coordinates with the plurality of second global world domain coordinates; and generating a plurality of training data, wherein each of the plurality of training data comprises one of the plurality of stored image domain coordinates and a second world domain coordinate of a matching surrounding object, The training data generation method further comprises: performing scaling normalization on the plurality of image domain coordinates of the stationary object in the driving image based on the vanishing line information in the driving image, The vanishing line information includes the position of the vanishing point in the driving image. The slope value of the scaling factor increases toward the vanishing point and decreases as the distance from the vanishing point increases.

15. The training data generation method according to claim 14, further comprising: associating the first identifier with the stationary object; as well as associating a plurality of second identifiers with the plurality of surrounding objects, The step of matching one of the plurality of surrounding objects with the stationary object comprises: matching a second identifier associated with one of the plurality of surrounding objects among the plurality of second identifiers with a first identifier associated with the stationary object.

16. The training data generation method according to claim 14, wherein: The stationary object includes at least one of a building, a sign, a traffic light, a crosswalk, a stop line, and a driving lane included in the driving image.

17. The training data generation method according to claim 14, further comprising: tracking the plurality of image-domain coordinates of the stationary object over time; as well as By filtering the plurality of image domain coordinates of the tracked stationary object, types of the plurality of image domain coordinates of the stationary object are converted into floating points.

18. The training data generation method according to claim 14, further comprising: The output of the distance sensor and the global positioning system information are cumulatively stored.

Images