Method and apparatus for annotating detected objects - Patents.com

Abstract

To provide a method and apparatus for annotating a detected object.SOLUTION: A method for annotating a detected object comprises: a step 610 of classifying a class of the object; a step 620 of generating, on the basis of the class of the object, a bounding box including a first quadrilateral and a second quadrilateral that share a side with respect to the object; and a step 630 of generating an annotation for the object in units of the bounding box.SELECTED DRAWING: Figure 6

Description

[Technical field]

[0001] The present invention relates to a method and apparatus for annotating detected objects. [Background technology]

[0002] Artificial intelligence (AI) refers to technology that artificially reproduces human learning and reasoning abilities using computer programs. In terms of AI, machine learning refers to learning that uses a model consisting of multiple parameters to optimize parameters with given data.

[0003] It is necessary to design training data for artificial intelligence training. Designing training data requires processing data, and annotation in particular means generating bounding boxes for objects contained in video data and inputting the necessary information. Summary of the Invention [Problem to be solved by the invention]

[0004] An object of the present disclosure is to provide a method and apparatus for annotating a detected object. The problem to be solved by the present disclosure is not limited to the problem mentioned above, and other problems and advantages of the present disclosure not mentioned can be understood by the following description and will be more clearly understood by the embodiments of the present disclosure. Furthermore, it will be understood that the problem to be solved and advantages of the present disclosure can be realized by the means and combinations thereof shown in the claims. [Means for solving the problem]

[0005] A first embodiment for solving the technical problem includes a step of classifying a class of object, a step of generating a bounding box for the object based on the class of the object, the bounding box including a first rectangle and a second rectangle that share one side, and a step of generating an annotation for the object on a bounding box basis.

[0006] The method may further include determining attributes of the object, and generating the annotations may include generating annotations for a class of the object and an attribute of the object on a bounding box basis.

[0007] In the method, the class of object may be any of a car, a van, a truck, a motorcycle, a pedestrian, an emergency vehicle, or others.

[0008] In the method, if the object is loaded with other vehicles as an automobile designed to carry cargo, the loaded other vehicles may not be classified as a class.

[0009] In the method, the step of generating a bounding box may comprise applying different criteria to generate a bounding box depending on a class of the classified object.

[0010] In the method, the first rectangle may be a rectangle and the second rectangle may be a trapezoid.

[0011] In the method, the first rectangle may correspond to a front or back surface of the object, and the second rectangle may correspond to a left or right surface of the object.

[0012] In the method, if a top surface of the object is exposed, the first rectangle or the second rectangle may include the top surface of the object.

[0013] In the method, if the object is a vehicle having wheels, the second quadrilateral may include line segments connecting wheel ground points.

[0014] In the method, if the class of the object is a two-wheeler or a pedestrian, the width of the first rectangle may be generated to be equal to the shoulder width of a person included in the object.

[0015] The method may further include determining not to generate the bounding box if the detected object is a vehicle and a proportion of the detected object seen in the video data is less than a critical proportion of a total size of the object.

[0016] In the method, the object attributes may include visibility, movement state, lane position, main state, object size, and subclass.

[0017] The method may further include controlling the ego-vehicle based on the generated annotations.

[0018] To solve the above technical problem, an apparatus according to a second embodiment of the present invention may provide an apparatus for annotating a detected object, the apparatus including: a memory having at least one program stored therein; and a processor operating by executing the at least one program, wherein the processor classifies a class of an object; generates a bounding box including a first rectangle and a second rectangle sharing a single valve for the object based on the class of the object; and generates an annotation for the object on a bounding box basis.

[0019] A method according to a third embodiment of the present invention for solving the above technical problem includes the steps of recognizing at least one object included in a first image acquired while driving and generating a first outer line for each recognized object, extracting the recognized object from the first image based on the generated first outer line, acquiring a first coordinate value of a second outer line of the extracted object as a result of inputting the extracted object into a first learning model, and merging the object to which the second outer line is applied into the first image based on the acquired first coordinate value.

[0020] In the method, the method may further include a step of calculating a second coordinate value for a position of the extracted object in the first image, and calculating a third coordinate value which is the coordinate of the second outline based on the second coordinate value, and the step of merging into the first image may further take into consideration the obtained third coordinate value to merge the object to which the second outline is applied into the first image.

[0021] In the method, the first coordinate value may be composed of seven coordinate values.

[0022] In the method, the first coordinate value may be composed of eight coordinate values.

[0023] In the method, the first outline may be in the form of a polygon.

[0024] In the method, the first outline may be in the form of a rectangle.

[0025] In the method, the first contour may be generated as a result of inputting the first image into a second learning model.

[0026] In the method, the second contour may be in the form of a cuboid.

[0027] In the method, the second outline may be in the form of a square.

[0028] In the method, the second outline may have a shape including two polygons.

[0029] In the method, one of the two polygons may represent the front or rear of the object, and the other may represent a side of the object.

[0030] In the method, the second outline may have a form in which two polygons are connected based on at least one common side.

[0031] In the method, the second outline may have a shape including a rectangle and a trapezoid.

[0032] In the method, the specifications of the first outer contour line may include the specifications of the second outer contour line.

[0033] An apparatus according to a fourth embodiment of the present invention for solving the above technical problem includes a memory in which at least one program is stored, and a processor that performs calculations by executing the at least one program, wherein the processor recognizes at least one object included in a first image acquired while driving, generates a first outer line for each recognized object, extracts the recognized object from the first image based on the generated first outer line, and as a result of inputting the extracted object into a first learning model, acquires a first coordinate value for a second outer line of the extracted object, and merges the object to which the second outer line is applied into the first image based on the acquired first coordinate value.

[0034] A method according to a fifth embodiment of the present invention for solving the above technical problem includes the steps of recognizing at least one object included in a first image acquired while driving, and generating a first outline for each recognized object, extracting first coordinate values ​​constituting the generated first outline in the first image, and acquiring cuboid coordinates of the recognized object based on the recognized object and the extracted first coordinate values.

[0035] In the method, the first contour may be a bounding box for the recognized object.

[0036] In the method, the cuboid may be a visual indicator of information regarding height, width, and depth of the recognized object.

[0037] In the method, the cuboid may include side points representing sides of the recognized object.

[0038] In the method, the side points may include two points arranged one above the other.

[0039] In the method, the cuboid may include a rear sign representing a rear face of the recognized object.

[0040] In the method, the cuboid may include an exposure indicator indicating the exposure of the recognized object.

[0041] In the method, the cuboid may include at least one of a side point representing a side of the recognized object, a rear mark representing a rear of the recognized object, and an exposure mark representing an exposure degree of the recognized object.

[0042] In the method, the acquired cuboid includes a rear marker indicating the rear of the recognized object, and the rear marker may be a line extending vertically through the center of the recognized object if the orientation of the recognized object is completely front-only.

[0043] In the method, the length of the rear marking may match the height of the first outline.

[0044] In the method, the acquired cuboid includes a rear marker indicating a rear of the recognized object, and the rear marker may coincide with the first outline if the orientation of the recognized object is a completely rear only orientation.

[0045] In the method, the acquired cuboid includes side points representing the sides of the recognized object and rear markers indicating the rear of the recognized object, and if the orientation of the recognized object is such that only the front and sides of the recognized object are visible, the length of the line connecting the side points is equal to the height of a first outer contour, and the rear markers may be limited to a line sharing at least a portion of the left or right side of the first outer contour.

[0046] In the method, the acquired cuboid includes side points representing the sides of the recognized object and rear markers indicating the rear of the recognized object, and if the orientation of the recognized object is such that only the sides and rear of the recognized object are visible, the length of the line connecting the side points is limited to the length of a portion of the left or right side of the first outer line, and the rear markers can be represented by a rectangle sharing the right side of the first outer line.

[0047] In the above method, in paragraph 1, the acquired cuboid includes side points representing the sides of the recognized object and rear markers indicating the rear of the recognized object, and if the orientation of the recognized object is a direction in which only the left or right side of the recognized object is visible, the length of the line connecting the side points is equal to the height of a first outer contour, and the rear markers can be limited to lines sharing the left or right side of the first outer contour.

[0048] An apparatus according to a sixth embodiment of the present invention for solving the above technical problem includes a memory having at least one program stored therein, and a processor that performs calculations by executing the at least one program, wherein the processor recognizes at least one object included in a first image acquired while driving, generates a first outline for each of the recognized objects, extracts first coordinate values ​​constituting the generated first outline from the first image, and obtains cuboid coordinates of the recognized object based on the recognized object and the extracted first coordinate values.

[0049] A method according to a seventh embodiment of the present invention for solving the above technical problem includes the steps of inputting an image acquired during driving into a learning model and generating a visual marker for the position and orientation of an object detected from the image, performing a first correction to move the position of the generated visual marker inside the outline of the object, and performing a second correction to change at least one of the position and shape of the visual marker moved inside the outline based on characteristics of the visual marker.

[0050] In the method, the outline may be a bounding box for the recognized object.

[0051] In the method, the visual indicator may be a cuboid that visually represents information regarding height, width, and depth of the recognized object.

[0052] In the method, the visual indicators may include side points representing sides of the recognized object.

[0053] In the method, the visual signs may include a rear sign representing a rear of the recognized object.

[0054] In the method, the visual indicator may include an exposure indicator representing an exposure degree of the recognized object.

[0055] In the method, the step of performing the first correction may include moving a position of the generated visual marker inside the outline of the object to grasp the direction of the recognized object, and the step of performing the second correction may further take into account the direction of the grasped object to perform the second correction.

[0056] In the method, the step of performing the first correction may include determining the direction of the recognized object and then moving the position of the generated visual marker to within the outline of the object, and the step of performing the second correction may perform the second correction by further taking into account the direction of the determined object.

[0057] In the method, the step of performing the first correction can move, among the generated visual markers, visual markers located outside the outline to inside the outline.

[0058] In the method, the step of performing the first correction can move the visual markers located outside the outline of the generated outline to a boundary of the outline of the generated outline.

[0059] In the method, the step of making the first correction can move a visual marker located outside the outline to a nearest point among a plurality of points constituting the outline.

[0060] In the above method, the second correction step involves changing the position and shape of the visual marker in a box corresponding to the width of the rear of the recognized object if the visual marker is a rear marker.

[0061] In the method, the step of performing the second correction may, if the visual marker is a rear marker, change the shape of the visual marker with a line corresponding to the height of the rear surface of the recognized object.

[0062] An apparatus according to an eighth embodiment of the present invention for solving the above technical problem includes a memory in which at least one program is stored, and a processor that performs calculations by executing the at least one program, wherein the processor inputs an image acquired during driving into a learning model, generates a visual marker for a position and orientation of an object detected from the image, performs a first correction to move a position of the generated visual marker inside an outline of the object, and performs a second correction to change at least one of a position and a shape of the visual marker moved inside the outline based on characteristics of the visual marker.

[0063] An embodiment of the present invention may provide a computer readable recording medium having stored thereon a program for performing at least one of the methods. Effect of the Invention

[0064] According to an embodiment of the present invention, it is possible to effectively annotate objects included in video data for artificial intelligence learning and vehicle driving control. In particular, it is possible to generate a bounding box in an appropriate manner according to the type of object.

[0065] Furthermore, by suggesting specific criteria for objects that may be difficult to annotate, errors for extraction of specific objects can be minimized.

[0066] According to the present invention, it is possible to accurately collect three-dimensional information (cuboid information) regarding the direction of objects in an image even in complex road environments, and to generate basic information for realizing a stable autonomous driving algorithm. [Brief description of the drawings]

[0067] [Figure 1] FIG. 1 is a diagram for explaining an automatic driving method according to an embodiment. [Diagram 2] FIG. 1 is a diagram for explaining an automatic driving method according to an embodiment. [Diagram 3] FIG. 1 is a diagram for explaining an automatic driving method according to an embodiment. [Figure 4] FIG. 1 is a diagram for explaining a configuration of a bounding box according to an embodiment of the present disclosure. [Figure 5A] FIG. 1 is a diagram illustrating an example of a bounding box generated for an object according to an embodiment of the present disclosure. [Figure 5B] FIG. 1 is a diagram illustrating an example of a bounding box generated for an object according to an embodiment of the present disclosure. [Figure 5C] FIG. 1 is a diagram illustrating an example of a bounding box generated for an object according to an embodiment of the present disclosure. [Figure 6] 1 is a flowchart of a method for annotating detected objects according to one embodiment. [Figure 7] FIG. 2 is a block diagram of an apparatus for annotating a detected object according to one embodiment. [Figure 8A] FIG. 2 is a diagram relating to a camera that captures images of the outside of a vehicle operating in autonomous driving mode. [Figure 8B] FIG. 2 is a diagram relating to a camera that captures images of the outside of a vehicle operating in autonomous driving mode. [Figure 9] FIG. 2 is a schematic diagram illustrating an object recognition method according to an embodiment. [Figure 10]11 is a diagram illustrating a process of generating a first outline in the operation of the object outline generating device. FIG. [Figure 11] 11 is a diagram illustrating a process in which an object outline generating device extracts an object surrounded by a first outline. FIG. [Figure 12A] 13 is a diagram illustrating a process in which an object outline generating device generates a second outline of an object included in a target frame and merges two different objects for which the second outlines have been generated into one. [Figure 12B] 13 is a diagram illustrating a process in which an object outline generating device generates a second outline of an object included in a target frame and merges two different objects for which the second outlines have been generated into one. [Figure 13] FIG. 13 is a diagram for illustrating a schematic example of an example in which a second outline is generated for each object and then repeatedly merged into the original image as many times as the number of objects. [Figure 14] 14 is a flowchart illustrating a method for generating an object contour according to the embodiment described with reference to FIGS. 10 to 13. FIG. [Figure 15] 1 is a block diagram of an object outline generating apparatus according to an embodiment; [Figure 16] 4A and 4B are schematic diagrams showing results when a method for obtaining cuboids of an object in an image according to the present invention is not applied and when it is applied to an image obtained while driving. [Figure 17] FIG. 2 is a conceptual diagram for explaining the cuboid obtaining process of the cuboid obtaining device according to the present invention. [Figure 18] 11 is a diagram illustrating an example of a cuboid of an object in the front left direction acquired by a cuboid acquisition device. FIG. [Figure 19] 11 is a diagram showing an example for explaining the cuboid of an object in the front direction acquired by the cuboid acquisition device. FIG. [Figure 20] 13 is a diagram illustrating an example of a cuboid of an object in the front right direction acquired by the cuboid acquisition device. FIG. [Figure 21] 13 is a diagram illustrating an example of a cuboid of an object on the left side acquired by the cuboid acquisition device. FIG. [Figure 22] 13 is a diagram showing an example for explaining the cuboid of an object on the right side acquired by the cuboid acquisition device. FIG. [Diagram 23] 13 is a diagram illustrating an example of a cuboid of an object in the rear left direction acquired by a cuboid acquisition device. FIG. [Figure 24] 13 is a diagram illustrating an example of a cuboid of an object in the rear right direction acquired by a cuboid acquisition device. FIG. [Diagram 25] 11 is a diagram showing an example for explaining a cuboid of an object in the rear direction acquired by a cuboid acquisition device. FIG. [Figure 26] FIG. 2 is a schematic diagram showing an example of a process for correcting rear marks by a cuboid acquisition device according to the present invention. [Figure 27] FIG. 13 is a schematic diagram showing another example of a process for correcting rear marks using a cuboid acquisition device according to the present invention. [Figure 28] FIG. 13 is a schematic diagram showing another example of a process for correcting rear marks using a cuboid acquisition device according to the present invention. [Figure 29] FIG. 2 is a flowchart showing an example of a method for obtaining a cuboid according to the present invention. [Diagram 30] FIG. 2 is a block diagram of a cuboid acquisition device according to one embodiment. [Diagram 31] FIG. 2 is a diagram conceptually illustrating the operation process of the visual landmark correction device for an object in an image according to the present invention; [Diagram 32] FIG. 2 is a schematic diagram illustrating an example of a first correction process performed by the visual sign correction device according to the present invention. [Diagram 33] FIG. 2 is a schematic diagram illustrating an example of a second correction process performed by the visual sign correction device according to the present invention. [Diagram 34]FIG. 2 is a schematic diagram showing another example of the second correction process performed by the visual sign correction device according to the present invention. [Diagram 35] FIG. 2 is a schematic diagram showing another example of the second correction process performed by the visual sign correction device according to the present invention. [Diagram 36] FIG. 2 is a flow chart illustrating an example method for visual landmark correction according to the present invention. [Figure 37] FIG. 1 is a block diagram of a visual landmark correction device according to one embodiment. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0068] The advantages and features of the present invention, and the methods for achieving them, will become apparent from the embodiments described in detail with the accompanying drawings. However, the present invention is not limited to the embodiments presented below, and may be realized in various different forms, and it should be understood that the present invention includes all modifications, equivalents, and alternatives within the spirit and technical scope of the present invention. The embodiments presented below are provided to fully disclose the present invention and to fully inform those skilled in the art to which the present invention pertains of the scope of the invention. In the description of the present invention, if a detailed description of a related known technology is deemed to obscure the gist of the present invention, the detailed description will be omitted.

[0069] The terms used in this application are merely used to describe certain embodiments and are not intended to limit the present invention. A singular expression includes a plural expression unless otherwise clearly indicated in the context. In this application, the terms "include" or "have" are intended to specify the presence of features, numbers, steps, operations, components, parts, or combinations thereof described herein, and should be understood not to preclude the presence or additional possibility of one or more other features, numbers, steps, operations, components, parts, or combinations thereof.

[0070] Some embodiments of the present disclosure may be represented by functional block configurations and various processing steps. Some or all of such functional blocks may be implemented in various numbers of hardware and / or software configurations that perform specific functions. For example, the functional blocks of the present disclosure may be implemented by one or more microprocessors, or may be implemented by circuit configurations for a given function. Also, for example, the functional blocks of the present disclosure may be implemented in various programming or scripting languages. The functional blocks may be implemented by algorithms executed by one or more processors. Also, the present disclosure may employ conventional techniques for electronic configuration, signal processing, and / or data processing, etc. Terms such as "mechanism," "element," "means," and "configuration" may be used broadly and are not limited to mechanical and physical configurations.

[0071] Furthermore, the connecting lines or connecting members between components shown in the figures are merely illustrative of functional and / or physical or circuit connections, and in an actual device the connections between components may be represented by various interchangeable or additional functional, physical or circuit connections.

[0072] Hereinafter, "vehicle" may mean any kind of transport means used to move people or goods having engine, such as a car, a bus, a motorcycle, a scooter, or a truck.

[0073] The present disclosure will now be described in detail with reference to the accompanying drawings.

[0074] Referring to FIG. 1, an autonomous driving device according to an embodiment of the present invention may be mounted on a vehicle to realize an autonomous vehicle 10. The autonomous driving device mounted on the autonomous vehicle 10 may include various sensors (including a camera) for collecting surrounding situation information. As an example, the autonomous driving device may detect the movement of a leading vehicle 20 traveling ahead through an image sensor and / or an event sensor attached to the front of the autonomous vehicle 10. The autonomous driving device may further include sensors for detecting other vehicles 30 traveling in adjacent lanes as well as the front of the autonomous vehicle 10, and pedestrians and the like around the autonomous vehicle 10.

[0075] At least one of the sensors for collecting situation information around the autonomous vehicle may have a predetermined field of view (FoV) as shown in Fig. 1. As an example, when a sensor mounted on the front of the autonomous vehicle 10 has a field of view (FoV) as shown in Fig. 1, information detected at the center of the sensor may have a relatively high importance. This may be because the information detected at the center of the sensor contains most of the information corresponding to the movement of the leading vehicle 20.

[0076] The autonomous driving device processes information collected by sensors of the autonomous vehicle 10 in real time to control the movement of the autonomous vehicle 10, while at least a portion of the information collected by the sensors can be stored in a memory device.

[0077] 2, the autonomous driving device 40 may include a sensor unit 41, a processor 46, a memory system 47, and a vehicle body control module 48. The sensor unit 41 includes a plurality of sensors (including cameras) 42-45, and the plurality of sensors 42-45 may include an image sensor, an event sensor, an illuminance sensor, a GPS device, an acceleration sensor, and the like.

[0078] The data collected by the sensors 42-45 may be transferred to the processor 46. The processor 46 may store the data collected by the sensors 42-45 in the memory system 47 and control the body control module 48 to determine the movement of the vehicle based on the data collected by the sensors 42-45. The memory system 47 may include two or more memory devices and a system controller for controlling the memory devices. Each of the memory devices may be provided on a single semiconductor chip.

[0079] In addition to the system controller of memory system 47, each memory device included in memory system 47 may include a memory controller, and the memory controller may include an artificial intelligence (AI) calculation circuit such as a neural network. The memory controller can generate calculation data by applying a predetermined weight to data received from sensors 42 to 45 or processor 46, and store the calculation data in a memory chip.

[0080] 3 is a diagram showing an example of video data acquired by a sensor (including a camera) of an autonomous vehicle equipped with an autonomous driving device. Referring to FIG. 3, video data 50 may be data acquired by a sensor mounted on the front of the autonomous vehicle. Therefore, the video data 50 may include a front part 51 of the autonomous vehicle, a preceding vehicle 52 in the same lane as the autonomous vehicle, vehicles 53 traveling around the autonomous vehicle, and a background 54.

[0081] 3, data of the area in which the front part 51 of the autonomous vehicle and the background 54 appear may be data that is unlikely to affect the operation of the autonomous vehicle. In other words, the front part 51 of the autonomous vehicle and the background 54 may be considered to be data with a relatively low importance.

[0082] Meanwhile, the distance to the preceding vehicle 52 and the lane-changing behavior of the moving vehicle 53 may be very important factors in the safe operation of the autonomous vehicle. Therefore, data of an area including the preceding vehicle 52 and the moving vehicle 53 in the video data 50 may have a relatively high importance in the operation of the autonomous vehicle.

[0083] The memory device of the autonomous driving apparatus can store image data 50 received from the sensor by assigning different weights to different areas. For example, a high weight can be assigned to data of an area including a leading vehicle 52 and a moving vehicle 53, and a low weight can be assigned to data of an area including a front part 51 of the autonomous driving vehicle and a background 54.

[0084] In the following, operations according to various embodiments may be understood to be performed by an automated driving device or a processor included in the automated driving device.

[0085] An annotation device for a detected object according to various embodiments of the present disclosure may be substantially identical to an autonomous driving device, may be included in the autonomous driving device, or may be a component implemented as part of the function performed by the autonomous driving device.

[0086] The apparatus for annotating a detected object of the present disclosure can classify a class of the detected object. The apparatus for annotating a detected object of the present disclosure can generate a bounding box for the object based on the classified class of the object. The apparatus for annotating a detected object of the present disclosure can generate an annotation for the object in units of the generated bounding box. The following describes in detail how the apparatus for annotating a detected object annotates the detected object.

[0087] The apparatus for annotating detected objects of the present disclosure can classify the class of the detected object.

[0088] In one embodiment, the device for annotating detected objects can receive input of data to classify the class of the object, for example, the input data can be video data collected by a camera, but can include various types of data such as images, text, audio, etc.

[0089] In one embodiment, the apparatus for annotating detected objects may include a classifier, which can calculate the probability that input data is classified into a particular class. Through the probability calculation of the classifier, the apparatus for annotating detected objects may classify the class of the object included in the input data. Meanwhile, the classifier may be trained by training data, and may be trained in a direction that reduces the error between the result output from the classifier and the actual correct answer.

[0090] In this disclosure, an apparatus for annotating detected objects can classify the detected objects as falling into one of a number of classes.

[0091] In one embodiment, the classes can be described via Table 1 below.

[0092] [Table 1]

[0093] Among the classes in Table 1, passenger cars may correspond to automobiles designed to transport a small number of people. For example, sedans, SUVs, taxis, etc. may be classified as passenger cars.

[0094] Among the classes in Table 1, a passenger car can correspond to a motor vehicle designed to transport a large number of people. For example, passenger cars and buses that can carry 16 or more people can be classified as passenger cars.

[0095] Among the classes in Table 1, a truck may correspond to a vehicle designed to carry cargo. If another vehicle, including a heavy equipment vehicle, is loaded on the vehicle designed to carry cargo, the vehicle designed to carry cargo and the other vehicle may be considered as one vehicle and classified as a truck. In other words, the device that annotates the detected object does not classify the class for the other vehicle loaded on the vehicle designed to carry cargo.

[0096] Among the classes in Table 1, motorcycles can correspond to two-wheeled vehicles and people riding on them. For example, motorcycles, scooters, mopeds, bicycles, and Segways can be classified as motorcycles. Strollers and hand trucks can also be classified as motorcycles.

[0097] Among the multiple classes in Table 1, pedestrians can correspond to people passing by on foot.

[0098] Among the classes in Table 1, emergency vehicles may correspond to vehicles that are used in emergency situations as vehicles designed for a special purpose. For example, vehicles equipped with sirens, such as police cars, fire engines, and tow trucks, may be classified as emergency vehicles. For example, if a vehicle that can be classified into another class, such as an undercover police car, is equipped with a siren, it may be classified as an emergency vehicle only if the siren is activated.

[0099] Among the classes in Table 1, "Other" can correspond to vehicles that are not classified into the remaining classes. For example, three-wheelers, forklifts, excavators, rickshaws, snowplows, bulldozers, road rollers, etc. can be classified as "Other."

[0100] However, Table 1 above is provided as an example and any suitable classification scheme may be applied.

[0101] In the present disclosure, an apparatus for annotating detected objects can apply different annotation methods depending on the class of the object.

[0102] The apparatus for annotating detected objects of the present disclosure can generate a bounding box for the detected object based on the class of the object.

[0103] In one embodiment, the bounding box may include two rectangles. Specifically, the device for annotating a detected object may generate a bounding box including a first rectangle and a second rectangle that share one side for the detected object based on a class of the object.

[0104] FIG. 4 is a diagram for explaining a configuration of a bounding box according to an embodiment of the present disclosure.

[0105] The bounding box 400 according to an embodiment of the present disclosure may include a first rectangle 410 and a second rectangle 420. When the bounding box 400 according to the present embodiment is used, it is possible to distinguish and identify the front, back, and side of an object displayed in the 2D image data.

[0106] In one embodiment, the first rectangle 410 may have a rectangular shape. That is, the first rectangle 410 may include four sides that are perpendicular to adjacent line segments. In one embodiment, two sides of the first rectangle 410 may be horizontal to the ground, and the remaining two sides of the first rectangle 410 may be perpendicular to the ground.

[0107] In one embodiment, the second rectangle 420 may have a trapezoidal shape, i.e., two of the four sides of the second rectangle 420 may be parallel to each other.

[0108] In one embodiment, the first rectangle 410 and the second rectangle 420 may share one side. In one embodiment, the side shared by the first rectangle 410 and the second rectangle 420 may be a side that is perpendicular to the ground.

[0109] In one embodiment, the first rectangle 410 may correspond to the front or rear of the vehicle. In one embodiment, the second rectangle 420 may correspond to the right or left side of the vehicle. On the other hand, if the top of the vehicle is exposed, the first rectangle 410 or the second rectangle 420 may be generated to include the top of the vehicle.

[0110] In the present disclosure, the bounding box including the first rectangle 410 and the second rectangle 420 can be appropriately generated according to various situations.

[0111] In one embodiment, the first rectangle 410 may be generated to exclude the vehicle's side mirrors.

[0112] In one embodiment, the second quadrilateral 420 may be generated to include line segments connecting the wheel ground points if the object is a vehicle with wheels.

[0113] In one embodiment, the first rectangle 410 may be generated to be wider than the front or rear of the vehicle, i.e. to include part of the side of the vehicle, but not beyond the point where the wheels contact the ground.

[0114] In one embodiment, the bounding box 400 may be generated to not include the vehicle doors when they are open.

[0115] In one embodiment, the bounding box 400 may be generated to include only that portion that corresponds to the body of the vehicle if the upper equipment changes size or length, such as if the vehicle is a ladder truck or crane, or if parts of the vehicle move separately.

[0116] In one embodiment, if the vehicle tows another object having wheels, such as a cargo trailer, a bounding box 400 may be generated for each of the vehicle and the towed object. For example, if the tow vehicle tows another vehicle, the tow vehicle may be classified as an emergency vehicle and the towed vehicle may be classified as a passenger vehicle, and a bounding box 400 may be generated for each of the tow vehicle and the other vehicle.

[0117] 5A to 5C are diagrams for explaining an example of a bounding box generated for an object according to an embodiment of the present disclosure.

[0118] Referring to FIG. 5A, an example of a bounding box generated for a generic object is shown.

[0119] In Figure 5A, a first rectangle may be generated to correspond to the rear of the vehicle and a width of the first rectangle may correspond to the width of the rear of the vehicle. In Figure 5A, a second rectangle may be generated to include a line segment connecting the points where the wheels contact the ground. In Figure 5A, the first rectangle and the second rectangle may be generated to share a line segment perpendicular to the ground and to include an image of the detected object.

[0120] In the present disclosure, the device for annotating detected objects can apply different criteria to generate bounding boxes depending on the class of the classified object.

[0121] In one embodiment, if the class of the object is a truck, motorcycle, or other, the height (length vertical to the ground) of the first rectangle and the width (length horizontal to the ground) of the second rectangle can be generated to include the object (e.g., luggage, cargo) carried by the object. In this embodiment, the width (length horizontal to the ground) of the first rectangle can be generated regardless of the object carried by the object.

[0122] 5B, for a vehicle that is a truck, a bounding box including a first rectangle and a second rectangle is generated. The width of the first rectangle, which corresponds to the front of the truck, can be generated regardless of the object being loaded. Meanwhile, the second rectangle is generated to include a line segment connecting the points where the wheels contact the ground, and the height of the first rectangle and the width of the second rectangle can be generated to include the object being loaded by the truck.

[0123] In one embodiment, when the class of the object is a motorcycle, the width of the first rectangle may be generated corresponding to a specific dimension of the body of a person included in the object. When the class of the object is a motorcycle, it is difficult to specify the width of the front or rear, or the width of the front or rear is very narrow. For example, the specific dimension of the body of a person may be the width of the shoulders. Specifically, when the class of the object is a motorcycle, the width of the first rectangle may be the same as the width of the shoulders of the person, and the height of the first rectangle may be the same as the height of the object including the vehicle and the person on the vehicle. In this case, the second rectangle may be generated to include the side of the object on the right or left side of the first rectangle.

[0124] Referring to FIG. 5C, a bounding box including a first rectangle and a second rectangle is generated for a bicycle with a person riding on it. The width of the first rectangle corresponding to the rear of the bicycle may be generated to be equal to the shoulder width of the person riding on it. Furthermore, the height of the first rectangle may be generated to be equal to the height of the object including the bicycle and the person. The second rectangle may be generated to include the side of the object to the left of the first rectangle.

[0125] In one embodiment, when the class of the object is a pedestrian, the width of the first rectangle may be generated corresponding to a specific dimension of the body of a person included in the object. When the class of the object is a pedestrian, as in the case of a motorcycle, it is difficult to specify the width of the front or rear surface, or the width of the front or rear surface is very narrow. For example, the specific dimension of the body of a person may be the width of the shoulders. Specifically, when the class of the object is a pedestrian, the width of the first rectangle may be the same as the width of the shoulders of the person, and the height of the first rectangle may be the same as the height of the person.

[0126] On the other hand, in one embodiment, if the class of object is a motorcycle but there is no rider, the width of the first rectangle 410 may be generated based on the stem or front fork.

[0127] In one embodiment, if the class of object is pedestrian, then walking sticks, brooms, handbags, umbrellas, etc. carried by a person may not be included in the bounding box.

[0128] In the present disclosure, an apparatus for annotating a detected object can determine whether to generate a bounding box according to a condition when the video data contains only a portion of the object.

[0129] In one embodiment, a decision may be made as to whether to generate a bounding box based on the percentage of the object that is seen in the video data. For example, if the percentage of the vehicle that is seen in the video data is less than a critical percentage (e.g., 30%) of the object's overall size, the device that annotates the detected object may decide not to generate a bounding box.

[0130] In one embodiment, whether to generate a bounding box may be determined based on the percentage of the object's part appearing in the video data. For example, if the percentage of the vehicle's wheels appearing in the video data is less than 50% of the vehicle's wheel size, the device for annotating the detected object may determine not to generate a bounding box.

[0131] In one embodiment, whether to generate a bounding box may be determined based on the area that a particular object occupies in the video data. For example, if an object classified as a van, truck, or other occupies more than 150px in the video data, the device for annotating detected objects may determine to generate a bounding box.

[0132] In the present disclosure, by determining whether to generate a bounding box, applying a different method for generating a bounding box, etc. depending on various conditions, annotations can be made for detected objects and safe and effective driving control of the vehicle can be performed based on the annotations.

[0133] The apparatus for annotating a detected object of the present disclosure can generate annotations for the object on a bounding box basis.

[0134] In one embodiment, the device for annotating detected objects can determine object attributes on a bounding box basis. In this disclosure, object attributes can refer to the nature of an object in relation to an ego-vehicle apart from the class of the object.

[0135] In one embodiment, the attributes of an object that may be determined may be described via Table 2 below.

[0136] [Table 2]

[0137] In one embodiment, the attribute of an object may include the degree to which the object is visible. In this disclosure, the degree of visibility may refer to an attribute related to the proportion of the object that is visible in the video data relative to the entire object.

[0138] The visibility level may be determined according to the percentage of the image data where the object is visible. Any suitable criteria may be applied to determine the visibility level. For example, if the percentage of the object visible is 0% or more and less than 20%, the visibility level may be determined as "level 1", if the percentage of the object visible is 20% or more and less than 40%, the visibility level may be determined as "level 2", if the percentage of the object visible is 40% or more and less than 60%, the visibility level may be determined as "level 3", if the percentage of the object visible is 60% or more and less than 80%, the visibility level may be determined as "level 4", if the percentage of the object visible is 80% or more and less than 100%, the visibility level may be determined as "level 5", and if the entire object is visible, the visibility level may be determined as "level 6".

[0139] In one embodiment, the attributes of an object may include a movement state of the object. In this disclosure, a movement state may refer to attributes related to the movement of an object.

[0140] The movement state may be determined depending on whether the object is moving. Any suitable criteria may be applied to determine the movement state. For example, if the object is moving, the movement state may be determined as "moving", if the object is stationary but can move, the movement state may be determined as "stationary", if the object is stationary and has no intention of moving, the movement state may be determined as "parked". In one embodiment, if an object is detected to include a person, the movement state of the object may be restricted from being determined as "parked".

[0141] Depending on the moving state of the object, the vehicle can change the route setting or driving control. For example, if the moving state of the object ahead is "stopped", the vehicle can be controlled to wait for a certain period of time, but if the moving state of the object ahead is "parked", the vehicle can be controlled to move while avoiding the object ahead.

[0142] In one embodiment, the attributes of the object may include a lane position. In this disclosure, a lane position may refer to information or attributes related to a positional relationship with the host vehicle.

[0143] In one embodiment, the lane position may be determined based on a relative position with respect to the host vehicle in units of lanes. For example, if the object belongs to the same lane as the host vehicle, the lane position of the object may be "0". For example, if the object belongs to one of the right lanes with respect to the front of the host vehicle, the lane position of the object may be a positive integer that increases corresponding to the lane being separated from the lane to which the host vehicle belongs. For example, if the object belongs to the first right lane with respect to the front of the host vehicle (i.e., the closest lane among the right lanes), the lane position of the object may be "+1", if the object belongs to the second right lane, the lane position may be "+2", and if the object belongs to the third right lane, the lane position of the object may be "+3". Conversely, for example, if the object belongs to the first left lane (i.e., the closest one of the left lanes) relative to the front of the vehicle, the object's lane position may be "-1", if it belongs to the second left lane, the object's lane position may be "-2", and if it belongs to the third left lane, the object's lane position may be "-3".

[0144] In one embodiment, the attribute of the object may include a main state of the object. In the present disclosure, the main state may refer to an attribute related to whether it affects the running of the host vehicle.

[0145] For example, if an object is an object immediately in front of the host vehicle and may directly affect the traveling of the host vehicle, the main state may be "caution". For example, if an object cannot affect the traveling of the host vehicle at all, the main state may be "no caution required". For example, if an object may affect the traveling of the host vehicle by suddenly changing lanes, the main state may be "caution required". For example, if the host vehicle belongs to a lane adjacent to a sidewalk, the main state of a pedestrian on the sidewalk may be "caution required".

[0146] Meanwhile, in the process of determining the primary state of an object, medians, delineators, etc. may be taken into account. For example, even if the object is a vehicle traveling next to you in the next lane, the primary state of the object may be "no attention required."

[0147] By determining the primary state of the object, it is possible to differentiate the driving control dependency on one or more detected objects. For example, an object whose primary state is "no attention required" may have no effect on driving control. For example, an object whose primary state is "attention required" may affect driving control, and the host vehicle may be controlled to be sensitive to the driving (e.g., speed, lane movement) of the object.

[0148] In one embodiment, the attributes of an object may include the size of the object.

[0149] For example, the size of the object may be determined based on the size of the object relative to the host vehicle. For example, the size of the object may be determined based on a size classification according to a predetermined vehicle type. For example, the size of the object may be either "small", "medium", or "large".

[0150] In one embodiment, an attribute of an object may include a subclass of the object. In this disclosure, a subclass of an object is a subclass within a class of an object and may refer to an attribute for distinguishing a particular class by fine-tuning it.

[0151] For example, if the class of object is a two-wheeler, the subclasses may include Rear_car, Electric_cart, Hand_truck, Three_wheels, etc. For example, if the class of object is a pedestrian, the subclasses may include "standing", "carrying luggage", "accompanied by animal", "holding umbrella", etc.

[0152] In the present disclosure, an apparatus for annotating detected objects can generate annotations by preserving object classes and object attributes on a bounding box basis.

[0153] In the present disclosure, by determining the attributes of an object, it is possible to evaluate the degree of danger based on the relationship with the host vehicle. Also, the above Table 2 is provided as an example, and any appropriate object attributes may be assigned.

[0154] Objects annotated by the device for annotating detected objects of the present disclosure can serve as the basis for controlling the driving of a vehicle and may be used for training an autonomous driving device.

[0155] FIG. 6 is a flowchart of a method for annotating detected objects according to one embodiment.

[0156] The operations shown in Fig. 6 may be performed by the apparatus for annotating the detected object described above. Specifically, the operations shown in Fig. 6 may be performed by a processor included in the apparatus for annotating the detected object described above.

[0157] In step 610, the device that annotates the detected object may classify the class of the object.

[0158] In one embodiment, the class of object may be any of a car, van, truck, motorcycle, pedestrian, emergency vehicle, or others.

[0159] In one embodiment, the device that annotates the detected object may not classify the class of the loaded other vehicles if the object is loaded as a car designed to carry cargo.

[0160] In step 620, the device for annotating the detected object may generate a bounding box including a first rectangle and a second rectangle that share one side for the object based on a class of the object.

[0161] In one embodiment, step 620 may be performed by applying different criteria depending on the class of the classified object.

[0162] In one embodiment, the first square may be a rectangle.

[0163] In one embodiment, the first rectangle may correspond to the front or back surface of the object.

[0164] In one embodiment, the second rectangle may be a trapezoid.

[0165] In one embodiment, the second rectangle may correspond to the left or right side of the object.

[0166] In one embodiment, if the top surface of the object is exposed, the first rectangle or the second rectangle may include the top surface of the object.

[0167] In one embodiment, if the object is a vehicle with wheels, the second rectangle may include line segments connecting the points where the wheels contact the ground.

[0168] In one embodiment, if the class of the object is a motorcycle or a pedestrian, the width of the first rectangle may be generated to be equal to the shoulder width of a person included in the object.

[0169] In step 630, the device for annotating the detected objects may generate annotations for the objects on a bounding box basis.

[0170] In one embodiment, the device for annotating the detected objects may determine attributes of the objects, and step 630 may include generating annotations for object classes and object attributes on a bounding box basis.

[0171] In one embodiment, the object attributes may include visibility, movement state, lane position, main state, object size, and subclass.

[0172] In one embodiment, a device that annotates a detected object may determine not to generate a bounding box if the detected object is a vehicle and the proportion of it seen in the video data is less than a critical proportion of the object's overall size.

[0173] In one embodiment, the device that annotates the detected objects can control the ego-vehicle based on the generated annotations.

[0174] FIG. 7 is a block diagram of an apparatus for acquiring a position of a stationary target according to one embodiment.

[0175] Referring to Fig. 7, an apparatus 700 for annotating a detected object may include a communication unit 710, a processor 720, and a DB 730. Only components related to the embodiment are shown in the apparatus 700 for annotating a detected object in Fig. 7. Therefore, it can be understood by a person skilled in the art that the apparatus 700 may further include other general-purpose components in addition to the components shown in Fig. 7.

[0176] The communication unit 710 may include one or more components that enable wired / wireless communication with an external server or device. For example, the communication unit 710 may include at least one of a short-range communication unit (not shown), a mobile communication unit (not shown), and a broadcast receiving unit (not shown).

[0177] The DB 730 is hardware that stores various data processed in the device 700 for annotating a detected object, and may store programs for processing and controlling the processor 720. The DB 730 may store payment information, user information, and the like.

[0178] DB730 may include dynamic random access memory (DRAM), random access memory (RAM), such as static random access memory (SRAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), CD-ROM, Blu-ray or other optical disk storage, hard disk drive (HDD), solid state drive (SSD), or flash memory.

[0179] The processor 720 controls the overall operation of the apparatus 700 for annotating a detected object. For example, the processor 720 may execute a program stored in the DB 730 to overall control an input unit (not shown), a display (not shown), a communication unit 710, the DB 730, etc. The processor 720 may execute a program stored in the DB 730 to overall control the operation of the apparatus 700 for annotating a detected object.

[0180] The processor 720 may control at least a portion of the operation of the apparatus 700 for annotating detected objects described above in FIGS.

[0181] Processor 720 may be implemented using at least one of ASICs (application specific integrated circuits), DSPs (digital signal processors), DSPDs (digital signal processing devices), programmable logic devices (PLDs), field programmable gate arrays (FPGAs), controllers, micro-controllers, microprocessors, and other electrical units for performing other functions.

[0182] In one embodiment, the device 700 for annotating a detected object may be an electronic device having mobility. For example, the device 700 for annotating a detected object may be implemented as a smartphone, a tablet PC, a PC, a smart TV, a personal digital assistant (PDA), a laptop, a media player, a navigation, a device equipped with a camera, and other mobile electronic devices. Furthermore, the device 700 for annotating a detected object may be realized as a wearable device such as a watch, glasses, a headband, and a ring with a communication function and a data processing function.

[0183] In another embodiment, the apparatus 700 for annotating a detected object may be an electronic device that is integrated into a vehicle, for example, the apparatus 700 for annotating a detected object may be an electronic device that is inserted into a vehicle through tuning after the manufacturing process.

[0184] Also, in another embodiment, the device 700 for annotating a detected object may be a server located outside the vehicle. The server may be implemented by a computer device or multiple computer devices that communicate over a network to provide instructions, code, files, content, services, etc. The server may receive data required to determine the vehicle's travel path from a device mounted on the vehicle, and may determine the vehicle's travel path based on the received data.

[0185] In another embodiment, the process performed in the device 700 for annotating detected objects may be performed at least in part by a mobile electronic device, an electronic device integrated in the vehicle, and a server located outside the vehicle.

[0186] 8A and 8B are diagrams relating to a camera that captures the exterior of a vehicle operating in autonomous driving mode.

[0187] The camera is mounted on the vehicle and can capture the outside of the vehicle. The camera can capture the front, side, rear, etc. of the vehicle. The object outline generating device according to the present invention can obtain a plurality of images captured by the camera. The plurality of images captured by the camera can include a plurality of objects.

[0188] The information about an object includes object type information and object attribute information. Here, the object type information is index information indicating the type of the object and is composed of a broad range of groups and a detailed range of classes. And the object attribute information indicates attribute information related to the current state of the object and includes movement information, rotation information, traffic information, color information, visibility information, etc.

[0189] In one embodiment, the groups and classes included in the object type information may be the same as, but are not limited to, those in Table 3 below.

[0190] [Table 3]

[0191] The motion information represents motion information of an object, and can be defined as stopping, parking, moving, etc. In the case of a vehicle, stopping, parking, and moving can be determined by the attribute information of the object, while in the case of a pedestrian, moving, stopping, and unknown can be determined by the attribute information of the object, and in the case of an object that cannot be moved, such as a traffic light, the attribute information of the object can be determined by the default value of stopping.

[0192] Rotation information represents rotation information of an object, and can be defined as front, rear, horizontal, vertical, side, etc. In the case of a vehicle, object attribute information can be defined for the front, rear, and side, and object attribute information can be defined for a horizontal or vertical traffic light, respectively.

[0193] Traffic information means traffic information of an object, and can be defined as traffic sign instructions, warnings, regulations, auxiliary signs, etc. Color information means color information of an object, and can represent the color of an object, the color of a traffic light, and the color of a traffic sign.

[0194] 8A, the object 811 may be a pedestrian. The image 810 may have a predetermined size. Multiple images 810 may contain the same object 811, but the relative position of the vehicle and the object 811 keeps changing as the vehicle travels along the road, and the object 811 also moves over time, which causes the same object 811 to change position in each image.

[0195] Using the entire image to determine what the same object is in each image requires significant data transfer and computational overhead, which makes it difficult to process using edge computing on vehicle-mounted devices and difficult to analyze in real time.

[0196] 8B shows a bounding box 821 included in an image 820. A bounding box is metadata for an object, and bounding box information may include object type information (group, class, etc.), position information on the image 820, size information, etc.

[0197] Referring to FIG. 8B, the bounding box information may include information that the object 811 corresponds to a pedestrian class, information that the top left vertex of the object 811 is located at (x, y) on the image, information about the width and height of the object 811, and current state information that the object 811 is moving (i.e., motion information).

[0198] FIG. 9 is a schematic diagram illustrating an object recognition method according to one embodiment.

[0199] The object outline generating device according to the present invention may obtain a plurality of frames by separating a video captured by a camera into frames. The plurality of frames may include a previous frame 910 and a current frame 920. The object outline generating device may primarily analyze a frame to recognize an object included in the frame, and secondarily generate a bounding box corresponding to the recognized object as described in Figs. 8A and 8B. In the present invention, the bounding box is a line formed on the outline of an object and surrounding the object, and may be referred to as either a first outline or a second outline in the context.

[0200] The object outline generator can recognize a first pedestrian object 911 in the previous frame 910 .

[0201] In one embodiment, the object outline generating device divides a frame into grids of equal size, predicts the number of bounding boxes specified in a predetermined form centered on the center of each grid for each grid, and calculates the reliability of the object based on the prediction. The object outline generating device determines whether the frame includes an object or only the background, and selects a position having a high object reliability to determine an object category, thereby recognizing the object. However, the method of recognizing an object in the present disclosure is not limited to this.

[0202] The object outline generating device may obtain first position information of a first pedestrian object 911 recognized in a previous frame 910. As described above in FIGs. 8A and 8B, the first position information may include coordinate information of any vertex (e.g., the vertex at the top left corner) of a bounding box corresponding to the first pedestrian object 911 on the previous frame 910, and horizontal and vertical length information.

[0203] In addition, the object outline generating device may obtain second position information of a second pedestrian object 921 recognized in the current frame 920 .

[0204] The object outline generation device can calculate the similarity between first position information of a first pedestrian object 911 recognized in the previous frame 910 and second position information of a second pedestrian object 921 recognized in the current frame 920.

[0205] 9, using the first position information and the second position information, the object outline generating device can calculate the intersection and union between the first pedestrian object 911 and the second pedestrian object 921. The object outline generating device calculates the value of the intersection area compared to the union area, and if the calculated value is equal to or greater than a threshold, can determine that the first pedestrian object 911 and the second pedestrian object 921 are the same pedestrian object.

[0206] However, the method of determining the identity between objects is not limited to the above method.

[0207] FIG. 10 is a diagram illustrating a process of generating a first outline of the operation of the object outline generating device.

[0208] Figure 10(a) shows one frame (hereinafter referred to as the "target frame") out of multiple frames that make up the video captured by an autonomous vehicle while it was traveling. The target frame in Figure 10(a) shows that a motorcycle is parked close to the camera lens, and a passenger car is parked behind the parked motorcycle.

[0209] 10(b) is a schematic diagram showing the generation of contours of a motorcycle and a car in a target frame. When the object contour generation device according to the present invention analyzes a frame and recognizes at least one object, data on the shape, color, and appearance of the object are input data for a learning model, and the contour of the object is output as output data.

[0210] Here, the learning model may be, but is not limited to, a model based on deep learning. The learning model is a model trained to receive an image as input data, recognize an object based on a specific point of the object included in the image, and generate an outline surrounding the entire object, and a target frame of an image captured by an autonomous vehicle is input as test data for the learning model that has completed learning, so that an outline can be generated for the object included in the target frame.

[0211] As an example, the contour line generated around the object may be a polygon. As another example, the contour line generated around the object may be a rectangle as shown in FIG. 10(b), and the contour line may be formed to completely include the entire object recognized as one object.

[0212] The information for forming the outline can be expressed by coordinates or coordinate values. The outline of a rectangle as shown in FIG. 10(b) can be formed using only the top left and bottom right coordinates of the rectangle. For example, if the top left coordinate is (x1, y1) and the bottom right coordinate is (x2, y2), the bottom left coordinate can be determined to be (x1, y2) and the top right coordinate can be determined to be (x2, y1) according to the shape characteristics of the rectangle. In other words, there are a total of two coordinates to be determined to generate the outline, and a total of four coordinate values ​​to be determined (x1, x2, y1, y2).

[0213] According to an embodiment, the outline of an object recognized in a target frame may be determined by a user's input. Specifically, if there is a target frame in which an object is recognized in a video captured by an autonomous vehicle, the object outline generating device may selectively extract only the target frame and output it to a user terminal used by a user, and if the user recognizes the object and provides an input for generating an outline, the outline may be determined based on the user's input.

[0214] Hereinafter, the first outline is considered to refer to an outline generated around an object in order to extract the object immediately after the object is recognized in the target frame, as shown in Fig. 10(b). That is, Fig. 10(b) can be said to show both the first outline 1010 of the motorcycle and the first outline 1030 of the car.

[0215] FIG. 11 is a diagram for explaining a process in which the object outline generating device extracts an object surrounded by a first outline.

[0216] FIG. 11 will be described with reference to FIG.

[0217] The object outline generating device can extract the motorcycle and the passenger car from the target frame based on the first outline 1010 of the motorcycle and the first outline 1030 of the passenger car in Fig. 10. The process in which the object outline generating device extracts the motorcycle and the passenger car from the target frame based on the first outline 1010 of the motorcycle and the first outline 1030 of the passenger car may be called a cropping process.

[0218] The extracted first object 1110 and the extracted second object 1130 have a resolution lower than the overall resolution of the target frame, and dummy data that reduces accuracy when recognizing features of objects in an automated process can be minimized. The extracted first object 1110 and the extracted second object 1130 can be input to a learning model that outputs a second outline of the first object and the second object as result data. The second outline is a different concept that is distinct from the first outline described above, and the second outline will be described later.

[0219] In the process of extracting the motorcycle and the passenger car from the target frame, the object outline generating device can grasp and store the size (width and height), absolute position information, and relative position information in the target frame of the motorcycle and the passenger car based on the coordinate values ​​of the first outline of the motorcycle and the passenger car. Examples of use of the coordinate values ​​of the first outline, the size of the object, the position information of the object, and the relative position information in the target frame will be described later.

[0220] Figures 12A and 12B are diagrams illustrating a process in which an object outline generation device generates a second outline for an object included in a target frame, and two different objects for which the second outlines have been generated are merged into one.

[0221] More specifically, Figure 12A is a schematic diagram showing an object outline generation device generating second outlines for each object included in a target frame, and Figure 12B is a diagram showing the result after second outlines have been generated for two different objects and then the objects are merged into one.

[0222] The object outline generating device can input the extracted first object 1110 and the extracted second object 1130 to a learning model and control so that second outlines are generated for the motorcycle and the passenger car as shown in Fig. 12A. Hereinafter, the motorcycle for which the second outline is generated and the passenger car for which the second outline is generated will be referred to as the first object 1110' for which the second outline is generated and the second object 1130' for which the second outline is generated, respectively.

[0223] Here, the learning model is a model different from the learning model that generates the first outline described above, and may be, for example, a deep learning model that receives an extracted object image as an input and outputs a second outline of the object as output data. Depending on the embodiment, the learning model that generates the second outline may be a model that uses other machine learning techniques instead of the deep learning method.

[0224] Here, the second outline means a kind of metadata that compressively expresses three-dimensional information of an object, considering that an object represented in two dimensions is actually represented as three-dimensional information. The self-driving car can perform distance and perspective calculations for objects that are getting closer to the self-driving car or getting farther away from the self-driving car, by recognizing the object surrounded by the second outline. In the present invention, the second outline can function as reference data for the self-driving car to perform distance and perspective calculations for various objects, as described above. The second outline also surrounds an object like the first outline, but has a feature of being composed of more coordinates (or coordinate values) due to its characteristic of compressively expressing three-dimensional information.

[0225] Various embodiments of the coordinate characteristics of the second contour and the shape characteristics of the second contour are described below.

[0226] As an example, the coordinate values ​​of the second outline may be composed of seven coordinate values. In Fig. 12B, the second outline 1210 of the motorcycle has a shape in which a rectangle representing the front of the motorcycle and a trapezoid representing the side of the motorcycle are combined along one common side. In this case, the vertex coordinates for forming the second outline 1210 of the motorcycle are (x1, y1) which is the upper left coordinate, (x2, y1) which is the upper center coordinate, (x3, y3) which is the upper right coordinate, (x1, y2) which is the lower left coordinate, (x2, y2) which is the lower center coordinate, and (x3, y4) which is the lower right coordinate, and the coordinate values ​​that are the minimum information required to form the second outline 1210 of the motorcycle are a total of seven (x1, x2, x3, y1, y2, y3, y4).

[0227] As another example, the coordinate values ​​of the second outline may be composed of eight coordinate values. Although not shown in FIG 12B, when an object is surrounded by a second outline in the form of a cuboid to enhance the perspective of the object, a total of eight coordinates are required to form the second outline of the object, and the total of eight required coordinate values ​​are (x1, x2, x3, x4, y1, y2, y3, y4).

[0228] The shape of the second outline may be a shape including two polygons, in particular in this embodiment, one of which may represent the front or rear of the object, and the other may represent the side of the object.

[0229] [Table 4]

[0230] Table 4 shows an example of an interpretation of object movement when the second outline is a form in which two polygons are connected horizontally based on one common side. The learning model receives an image of an object surrounded by the first outline as an input, and is trained by the training data to display the front and rear of the object as rectangles and the left and right sides as trapezoids with parallel sides of different lengths.

[0231] 12B, it can be seen that at the top of the second outer contour 1210 of the motorcycle, "Front-left" is written as object metadata, and below that, a second outer contour in the form of a combination of a rectangle and a trapezoid corresponding to the front and left side of the motorcycle is located, and a second outer contour 1230 of a passenger car can also be generated in the above manner. While generating the second outer contour for the object, the learning model calculates the coordinate values ​​of the second outer contour as numerical information for forming the second outer contour, and the calculated coordinate values ​​of the second outer contour can be used to determine the position of the object when the object to which the second outer contour is applied is merged into the original image in the next step.

[0232] As already explained, in one embodiment of the present invention, the shape of the second outline may be a cuboid shape, and in this case, a total of eight coordinate values ​​are required.

[0233] Although not listed in Table 4, if only the front or rear surface is captured in the image captured by the camera mounted on the autonomous vehicle, or only the side surface is captured, but not both the front and rear surfaces, the second outline may be in the form of a rectangle. In other words, the second outline of an object of which only the front or rear surface is captured may be in the form of a rectangle, and the second outline of an object of which only the side surface is captured, but not both the front or rear surfaces, may be a simple rectangle that does not include a right angle.

[0234] The second outline may also be in the form of two polygons connected based on at least one common side, which has been diagrammatically illustrated using the second outline 1210 of the motorcycle and the second outline 1230 of the passenger car in FIG. 12B.

[0235] The standard of the second outline may be included in the standard of the first outline. Specifically, since the learning model is trained to generate the second outline from the first outline based on an image with limited resolution, the standard of the second outline can be automatically adjusted to be equal to or smaller than the standard of the first outline.

[0236] The object outline generating device may generate a second outline for each of the extracted objects as shown in Fig. 12B, and then merge the objects 1210', 1230' to which the second outline is applied into the original image. For example, if there are four objects in a target frame, the process of merging the objects to which the second outline is applied into the original image may be repeated four times, and the four objects to which the second outline is applied may be displayed in the image at once along with a background in which no objects are displayed in the final image. In other words, if the number of objects is n, the process of generating the second outline and merging it into the original image may also be performed n times.

[0237] In the present invention, the coordinate values ​​for forming the second outer contour of the extracted object may be referred to as first coordinate values, the coordinate values ​​with respect to the center coordinates of the object in the original image for referring to the extracted object may be referred to as second coordinate values, and the coordinate values ​​for forming the second outer contour of the object in the original image may be referred to as third coordinate values. After the first coordinate values ​​for forming the second outer contour are determined, the object outer contour generating device checks the position of the object in the original image through the second coordinate values, and processes the second outer contour to be generated in the original image centered on the second coordinate values, thereby generating an effect of merging the object to which the second outer contour is applied into the original image.

[0238] According to the present invention, accurate annotation for each object is possible through a process of first clearly setting a first outline for each object, extracting the object, learning for generating a second outline for each object, and finally merging the objects. That is, according to the present invention, the annotation accuracy can be significantly improved compared to the conventional method of annotating the entire target frame in 3D at once.

[0239] FIG. 13 is a diagram for illustrating an example in which a second outline is generated for each object and then repeatedly merged into the original image as many times as the number of objects.

[0240] 13(a) and 13(b) show an original image calculated through a process in which four objects are individually merged into the original image four times repeatedly. In FIG. 13(a) and FIG. 13(b), the number of objects is four, but in some embodiments, the number of objects merged into the original image is not limited to four, and may be more than four or less than four.

[0241] 13(a) is a schematic diagram showing the result of merging the second outline of an object into the original image after it has entered a cuboid shape. The second outline 1310 of the first object in FIG. 13(a) shows that a 3D cuboid has been applied and that the object is moving in the same direction as the self-driving car that captured the image, starting from the right side of the self-driving car.

[0242] Also, FIG. 13(b) shows a schematic diagram of the result of merging the second outline of the object into the original image after it has been converted into a form in which two polygons are combined. In particular, the two polygons may be a rectangle and a trapezoid, and it can be seen that the upper left end of the second outline is additionally marked with "car," "bus," or "truck" as the class information of the object along with "Rear-Right" or "Rear-Right." The second outline 1330 of the second object in FIG. 13(b) shows that the form in which two polygons are combined is applied, and that the object moves in the same direction as the self-driving car that captured the image from the left side.

[0243] FIG. 14 is a flowchart showing a method for generating an outline of an object according to the embodiment described with reference to FIGS.

[0244] The method in FIG. 14 can be realized by the object contour line generating device described with reference to FIGS. 10 to 13, and therefore, in the following, description that overlaps with the contents described with reference to FIGS. 10 to 13 will be omitted.

[0245] First, the object outline generating device may recognize an object in a first image and process the object through a learning model so that a first outline may be generated for each object (S1410). The first outline may be in the form of a polygon.

[0246] Then, the object outline generating device may extract an object from the first image based on the first outline (S1430). More specifically, the extracted object may be an image limited to a size area of ​​the first outline with the first outline as a boundary.

[0247] The object outline generating device may obtain a first coordinate value of a second outline of the object (S1450). The first coordinate value may be composed of seven or eight coordinate values, depending on the embodiment.

[0248] The object outline generating device may merge the object to which the second outline is applied into the first image based on the first coordinate value (S1470). As an alternative embodiment of step S1470, the object outline generating device may calculate a second coordinate value for the position of the extracted object in the original image (or target frame), and calculate a third coordinate value that is a value for the coordinates of the second outline based on the second coordinate value, thereby realizing an effect of merging the object to which the second outline is applied into the original image by taking into account both the first coordinate value and the third coordinate value.

[0249] FIG. 15 is a block diagram of an object outline generating device according to one embodiment.

[0250] 15, an object outline generating device 1500 may include a communication unit 1510, a processor 1520, and a DB 1530. Only components related to the embodiment are shown in the object outline generating device 1500 in FIG. 15. Therefore, it can be understood by a person skilled in the art that other general-purpose components may be further included in addition to the components shown in FIG.

[0251] The communication unit 1510 may include one or more components that enable wired / wireless communication with an external server or device. For example, the communication unit 1510 may include at least one of a short-range communication unit (not shown), a mobile communication unit (not shown), and a broadcast receiving unit (not shown).

[0252] The DB 1530 is hardware that stores various data processed within the object outline generating device 1500, and can store programs for processing and controlling the processor 1520.

[0253] DB1530 may include dynamic random access memory (DRAM), random access memory (RAM), such as static random access memory (SRAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), CD-ROM, Blu-ray or other optical disk storage, hard disk drive (HDD), solid state drive (SSD), or flash memory.

[0254] The processor 1520 controls the overall operation of the object outline generating device 1500. For example, the processor 1520 may execute a program stored in the DB 1530 to overall control an input unit (not shown), a display (not shown), a communication unit 1510, the DB 1530, etc. The processor 1520 may control the operation of the object outline generating device 1500 by executing a program stored in the DB 1530.

[0255] The processor 1520 can control at least a part of the operation of the object outline generating device 1500 described above with reference to FIGS.

[0256] 10 to 14, the processor 1520 may recognize at least one object included in a first image acquired during driving, generate a first outline for each recognized object, extract an object recognized in the first image based on the generated first outline, and input the extracted object into a first learning model to acquire a first coordinate value for a second outline of the extracted object, and merge the object to which the second outline is applied based on the acquired first coordinate value into the first image. Depending on the embodiment, a final image generated by merging the object to which the second outline is applied into the first image may be called a second image to distinguish it from the first image.

[0257] The processor 1520 may be implemented using at least one of ASICs (application specific integrated circuits), DSPs (digital signal processors), DSPDs (digital signal processing devices), programmable logic devices (PLDs), field programmable gate arrays (FPGAs), controllers, micro-controllers, microprocessors, and other electrical units for performing other functions.

[0258] FIG. 16 is a schematic diagram showing results when the method for obtaining cuboids of an object in an image according to the present invention is not applied to an image obtained while driving, and when it is applied.

[0259] Hereinafter, the method for obtaining cuboids of an object in an image according to the present invention will be referred to as a cuboid obtaining method, and the apparatus for obtaining cuboids of an object in an image according to the present invention will be referred to as a cuboid obtaining apparatus.

[0260] First, FIG. 16(a) shows the result when the cuboid acquisition method was not applied. More specifically, FIG. 16(a) shows a schematic diagram of the cuboid of an object acquired when an image acquired during driving is input as input data to a learning model. In the present invention, a cuboid is a concept that expands the conventional dictionary definition of a rectangular parallelepiped, and means a sign (or information composed of coordinates in an image) that visually represents information on the height, width, and depth of a recognized object when an object is recognized in an image. In other words, a cuboid has the characteristic of representing three-dimensional information of an object in an image that is expressed in two dimensions.

[0261] The object recognized in the image of FIG. 16(a) is a truck carrying a container box, and the cuboid in FIG. 16(a) includes a total of three shapes. First, the side point 1610A may be an example of a cuboid for intuitively expressing the side of the recognized object. Furthermore, the exposure mark 1630A may be an example of an indicator showing whether the object is completely exposed in the image. Furthermore, the rear mark 1650A may be an example of a cuboid for intuitively expressing the position of the rear of the recognized object.

[0262] Typically, when an image is input to the learning model, the cuboid coordinates of an object included in the image can be acquired as shown in FIG. 16(a). The learning model can be an improved form of neural network such as deep learning, which learns the input image according to a preset loss function and outputs a result for annotation, and in some embodiments, a machine learning algorithm different from the neural network can be applied. At this time, the acquired side points 1610A, exposure markers 1630A, and rear markers 1650A have the characteristic of being independently learned without depending on the outline 1605A of the object recognized in FIG. 16(a), so that the coordinates constituting the outline 1605A do not affect the coordinate values ​​of the side points 1610A, exposure markers 1630A, and rear markers 1650A.

[0263] Meanwhile, according to the cuboid acquisition device of the present invention, a cuboid as shown in FIG. 16(b) can be acquired. Specifically, the cuboid coordinates acquired through the learning model have a characteristic that is dependent on the outline 1605B generated on the outline of the object after the object is primarily recognized. More specifically, the coordinates of the side point 1610B, the exposure mark 1630B, and the rear mark 1650B in FIG. 16(b) all share coordinate values ​​with the coordinates of the outline 1605B generated on the outline of the object in FIG. 16(b). Here, sharing coordinate values ​​means that the side point 1610B, the exposure mark 1630B, and the rear mark 1650B are located on at least one of the four sides forming the outline 1605B.

[0264] The definitions of the side points, exposure marks and rear marks included in the cuboid in the present invention will be specifically explained below.

[0265] A side point may represent the boundary between the front and side of an object recognized in an image. Here, the side means the side of an object that can be seen through an image, and may be either the left or right side of the object. A side point is composed of points arranged above and below and a connection line connecting the two points, and the points arranged above and below have the same x coordinate and only the y coordinate is different. As an example, the coordinates of the two points of a side point may be (x1, y1) and (x1, y2), respectively. Once the two points forming the side point are determined, the connection line can be determined immediately. Since the side point represents the boundary between the front and side of an object, the length of the connection line of the side point may represent the height of the front of the object.

[0266] The lateral sign may indicate the degree of exposure of an object recognized in an image, and may be represented by one large dot in the image, as shown in FIG. 16 (a) or (b). The lateral sign is an indicator of whether an object is fully exposed in an image. As an example, if the entire image of an object is completely captured in the image, the exposure sign is located at the very center of the lower end of the object's outline, and if only a portion of the object is captured in the image, the exposure sign is located at an appropriate position taking into account the portion of the object that is not captured. As another example, if only a portion of the object is captured in the image, the exposure sign may be omitted from the beginning. In this case, the cuboid of the object will include only the above-mentioned side points and rear signs.

[0267] A rear sign is an index that indicates the position of the rear of an object in an image. Depending on the embodiment, the rear sign may be in the form of a line or a box. Changes in the shape of the rear sign will be described later with reference to Figs. 18 to 25.

[0268] FIG. 17 is a conceptual diagram for explaining the cuboid obtaining process of the cuboid obtaining device according to the present invention.

[0269] Referring to FIG. 17, a cuboid acquisition device 1700 according to the present invention includes a pre-processing unit 1730 and a model learning unit 1740, and image data 1710 acquired while a car is running is input to the cuboid acquisition device 1700.

[0270] First, the video data 1710 is raw data of a video captured by a camera installed in the autonomous vehicle while the vehicle is traveling, and objects (such as people, two-wheeled vehicles, four-wheeled vehicles, and other moving objects) included in the video can be recognized by the model learning unit 1740. Since the method of recognizing objects in a video in the present invention is not limited to a particular type of method, not only currently known methods but also object recognition methods that will be completed in the near future may be used.

[0271] The first contour data 1721 refers to information about a contour generated for each object recognized as an object in an image. Specifically, information about the contour 1605A in FIG. 16(a) or the contour 1605B in FIG. 16(b) may be the first contour data 1721. In the present invention, the first contour surrounding an object may be a bounding box in the form of a rectangle that fits closely to the edge of the object, taking into account the protrusion of the object. The first contour data 1721 may also be formed during the learning process of the learning model of the model learning unit 1740, and in particular, the coordinates of the points forming the first contour may be reference information for finally determining the coordinates of a cuboid.

[0272] The cuboid data 1722 refers to information about a cuboid acquired for each object recognized as an object in an image. For example, if a first object and a second object are included in an image, the cuboid data 1722 may include the coordinates of the cuboid of the first object and the coordinates of the cuboid of the second object. In this case, the cuboid may include the above-mentioned side points, rear markers, and exposure markers, and some cuboids may have inaccurate characteristics because the cuboids were acquired without considering the first outline data 1721.

[0273] In the present invention, when preprocessed image data is input to a learning model, objects are recognized in the image data, a certain number of values ​​(e.g., 13) are extracted as predicted values ​​for each object, and then first contour data 1721 is calculated from some of the predicted values, and cuboid data 1722 is calculated in parallel from the remaining predicted values, and the cuboid data 1722 is corrected to be dependent data of the first contour data 1721. In other words, the first contour data 1721 can function as a kind of boundary for converting the cuboid data 1722 into final cuboid coordinates.

[0274] When the first outer contour is a bounding box, the minimum coordinates for forming the first outer contour may be a total of four coordinates. For example, the minimum coordinates of the first outer contour may be (x1, y1), (x1, y5), (x4, y1), and (x4, y5). When the coordinates of the first outer contour are determined as described above, the coordinate values ​​for forming the first outer contour are x1, y1, x5, and y5, and all coordinates that match at least one of the x-axis or y-axis coordinates of the coordinates of the first outer contour can be considered as the coordinates of the first outer contour. As an example, (x3, y1) or (x1, y3) can be considered as the coordinates of the first outer contour. If the number of objects recognized in the image is n, the number of first outer contours may also be n.

[0275] The pre-processing unit 1730 may pre-process the image data 1710. For example, the pre-processing unit 1730 may perform a function of embedding the image data 1710 so that the image data 1710 can be input to the learning model of the model learning unit 1740 without error and learned, and may perform a function of checking the integrity of the image data 1710. The pre-processing unit 1730 may be implemented in a form incorporated in the model learning unit 1740 depending on the embodiment.

[0276] The model learning unit 1740 can receive the preprocessed image data 1710 from the preprocessing unit 1730 and perform learning to obtain cuboids of objects included in the image.

[0277] As shown in FIG. 17, the cuboid acquisition device 1700 according to the present invention does not immediately acquire the cuboid of an object using only image data 1710, but instead reflects information about the first outline generated for each object in learning, thereby controlling the finally acquired cuboid for each object to include accurate 3D information of the object.

[0278] In particular, the learning model of the cuboid acquisition device 1700 can acquire the cuboid coordinates of the object by acquiring the coordinate values ​​of the first outline of the object, and then acquiring the coordinates of the side points, exposure marks, and rear marks for the object, and correcting the coordinates of the acquired side points, exposure marks, and rear marks taking into account the coordinate values ​​of the first outline.

[0279] For example, when the cuboid acquisition device 1700 acquires the coordinates of two points in the vertical direction that form a side point, it can correct the y coordinate values ​​of the two points in the vertical direction that form the side point, taking into account the coordinate value of the first outline. More specifically, after determining the direction of the recognized object, the cuboid acquisition device 1700 acquires the coordinates of two points in the vertical direction that form a side point, and then corrects the y coordinate values ​​of the two points according to the height of the first outline.

[0280] The direction of the recognized object may be one of a total of eight directions. For example, the direction of the recognized object may be one of the front left direction, the front direction, the front right direction, the completely left direction, the completely right direction, the rear left direction, the rear direction, and the rear right direction. When the direction of the object recognized in the image is determined, the cuboid obtaining device 1700 corrects the coordinates of the side point, the exposure mark, and the rear mark obtained in the cuboid of the object according to a preset formula for each determined direction, thereby controlling so that an accurate cuboid can be obtained. The correction methods applied differently to the eight directions of the object will be described later with reference to FIGS. 18 to 25.

[0281] FIG. 18 is a diagram illustrating an example of a cuboid of an object in the front left direction acquired by the cuboid acquisition device.

[0282] In FIG. 18, the object is a car in the front left direction, the car is surrounded by a first outline 1800, and side points 1810 and rear markers 1830 are represented diagrammatically as cuboids relative to the object.

[0283] The cuboid acquisition device may first analyze an object to detect whether it is in the front-left direction, and then secondarily acquire coordinates for a side point 1810 and a rear mark 1830 as a cuboid for the object. In some embodiments, the cuboid acquisition device may first acquire coordinates for a side point 1810 and a rear mark 1830 for an object, and then secondarily detect which direction the object is facing.

[0284] After detecting that the direction of the object is the front-left direction, the cuboid acquisition device may correct the y coordinates of two points in the upper and lower directions constituting the side point 1810 to the y coordinates of the most adjacent first outer boundary line 1800. For example, if the first outer boundary line 1800 includes a side formed from (x1, y5) to (x10, y5) and the coordinates of the upper end point of the side point 1810 are (x4, y4), the cuboid acquisition device may correct the coordinates of the upper end point of the side point 1810 to (x4, y5). As another example, if the first outer boundary line 1800 includes a side formed from (x1, y5) to (x10, y5) and the coordinates of the upper end point of the side point 1810 are (x4, y7), the cuboid acquisition device may correct the coordinates of the upper end point of the side point 1810 to (x4, y5). That is, in the present invention, if the coordinates of a point before correction are outside or inside the first outer contour 1800, the coordinates of the point are corrected to coordinates of a point for constituting the first outer contour 1800. In this case, it will be understood by those skilled in the art that between the coordinates (x1, y5) and (x10, y5) on the first outer contour 1800, values ​​between x1 and x10 can be applied to only the x coordinate while the y coordinate is kept the same, such as (x2, y5), (x3, y5), (x4, y5), etc.

[0285] As another example, the cuboid acquisition device may perform learning processing so that, after detecting that the direction of the object is the front left direction, a line connecting two points in the up and down direction constituting the rear marker 1830 is limited to a line that shares at least a part of the left side or right side of the first outline 1800. Specifically, the cuboid acquisition device may calculate the height of the rear surface of the object by taking into consideration the first outline 1800, the overall contour of the recognized object, and the direction of the detected object (front left), and set the length of the rear marker 1830 to a length equal to the calculated height. In this case, since the rear surface of the object is not observed in the front left direction, the rear marker 1830 may be expressed as a line that shares at least a part of the left side or right side of the first outline. Referring to FIG. 18, it can be seen that the cuboid acquisition device generates the rear marker 1830 so as to share the right side of the first outline 1800, but the length of the rear marker 1830 is limited to a part of the right side by considering the height of the rear surface of the recognized object, the start point of the rear surface, and the end point of the rear surface.

[0286] FIG. 19 is a diagram illustrating an example for explaining the cuboid of an object in the front direction acquired by the cuboid acquisition device.

[0287] In FIG. 19, the object is a car in the forward direction and is surrounded by a first outline 1900, with side points 1910 and rear markers 1930 represented diagrammatically as cuboids relative to the object.

[0288] The cuboid acquisition device may first analyze an object to detect whether it is completely in the front only direction, and then secondarily acquire coordinates for a side point 1910 and a rear mark 1930 as a cuboid for the object. In some embodiments, the cuboid acquisition device may first acquire coordinates for a side point 1910 and a rear mark 1930 for an object, and then secondarily detect which direction the object is facing.

[0289] After detecting that the object is facing forward, the cuboid acquisition device can set two points in the upper and lower directions constituting a side point 1910 to points located exactly in the center between the upper and lower ends of the first outline 1900. Then, a connecting line of the side point 1910 can be automatically set.

[0290] In addition, the cuboid acquisition device detects that the object is facing the front direction, and then acquires a line extending vertically by dividing the center of the recognized object as the rear marker 1930. Since the rear marker 1930 is a marker indicating the rear of the object, if the object is completely facing the front direction, it should not be displayed as described above. However, in the present invention, in a situation where neither the side nor the rear of the object is observed, the rear marker 1930 is displayed as a line located in the center of the front of the object as shown in Fig. 19, so that the rear marker 1930 can provide three-dimensional information of the object in a roundabout manner.

[0291] Referring to FIG. 19, when an object is detected to be completely in the front direction, it can be seen that the connecting line of the side point 1910 and the rear mark 1930 coincide, and in this case, the length of the connecting line of the side point 1910 and the rear mark 1930 is the same as the height of the first outer line 1900.

[0292] FIG. 20 is a diagram illustrating an example of the cuboid of an object on the right front side, which is acquired by the cuboid acquisition device.

[0293] In FIG. 20, the object is a car on the front right side, the car is surrounded by a first outline 2000, and a side point 2010 and a rear marker 2030 are shown diagrammatically as cuboids relative to the object.

[0294] The cuboid acquisition device may first analyze an object to detect whether it is in the front-right direction, and then secondarily acquire coordinates for a side point 2010 and a rear mark 2030 as a cuboid for the object. In some embodiments, the cuboid acquisition device may first acquire coordinates for a side point 2010 and a rear mark 2030 for an object, and then secondarily detect the direction of the object.

[0295] After detecting that the direction of the object is the front-right direction, the cuboid acquisition device may correct the y coordinates of two points in the upper and lower directions constituting the side point 2010 to the y coordinates of the most adjacent first outer boundary line 2000. For example, if the first outer boundary line 2000 includes a side formed from (x1, y5) to (x10, y5) and the coordinates of the upper end point of the side point 2010 are (x4, y4), the cuboid acquisition device may correct the coordinates of the upper end point of the side point 2010 to (x4, y5). As another example, if the first outer boundary line 2000 includes a side formed from (x1, y5) to (x10, y5) and the coordinates of the upper end point of the side point 2010 are (x4, y7), the cuboid acquisition device may correct the coordinates of the upper end point of the side point 2010 to (x4, y5). That is, in the present invention, if the coordinates of a point before correction are outside or inside the first outer contour 2000, the coordinates of the point are corrected to the coordinates of a point for constituting the first outer contour 2000. In this case, it will be understood by those skilled in the art that between the coordinates (x1, y5) and (x10, y5) on the first outer contour 2000, values ​​between x1 and x10 can be applied to only the x coordinate while the y coordinate is kept the same, such as (x2, y5), (x3, y5), (x4, y5), etc.

[0296] As another example, the cuboid acquisition device may perform learning processing so that, after detecting that the direction of the object is the front right direction, a line connecting two points in the up and down direction constituting the rear marker 2030 is limited to a line that shares at least a part of the left side or right side of the first outer contour 2000. Specifically, the cuboid acquisition device may calculate the height of the rear surface of the object by taking into consideration the first outer contour 2000, the overall contour of the recognized object, and the direction of the detected object (front right side) as a whole, and set the length of the rear marker 2030 to be equal to the height. In this case, since the rear surface of the object is not observed in the front right direction, the rear marker 2030 may be expressed as a line that shares at least a part of the left side of the first outer contour. Referring to FIG. 20, it can be seen that the cuboid acquisition device generates the rear marker 2030 so as to share the left side of the first outer contour 2000, but the length of the rear marker 2030 is limited to a part of the left side by taking into consideration the height of the rear surface of the recognized object, the start point of the rear surface, and the end point of the rear surface.

[0297] FIG. 21 is a diagram showing an example for explaining the cuboid of an object on the left side, acquired by the cuboid acquisition device.

[0298] In FIG. 21, the object is a car with only the left side observed in the image, the car is surrounded by a first outline 2100, and side points 2110 and rear marker 2130 are represented diagrammatically as cuboids relative to the object.

[0299] The cuboid acquisition device may first analyze an object to detect that only the left side is observed, and then secondarily acquire coordinates for a side point 2110 and a rear mark 2130 as a cuboid for the object. In some embodiments, the cuboid acquisition device may first acquire coordinates for a side point 2110 and a rear mark 2130 for an object, and then secondarily detect the direction of the object.

[0300] After detecting that only the left side of the object is observed in the image, the cuboid acquisition device may correct the y coordinates of two points in the upper and lower directions constituting the side point 2110 to the y coordinates of the most adjacent first outer boundary line 2100. For example, if the first outer boundary line 2100 includes a side formed from (x1, y5) to (x10, y5) and the coordinates of the upper end point of the side point 2110 are (x4, y4), the cuboid acquisition device may correct the coordinates of the upper end point of the side point 2110 to (x4, y5). As another example, if the first outer boundary line 2100 includes a side formed from (x1, y5) to (x10, y5) and the coordinates of the upper end point of the side point 2110 are (x4, y7), the cuboid acquisition device may correct the coordinates of the upper end point of the side point 2110 to (x4, y5). That is, in the present invention, if the coordinates of a point before correction are outside or inside the first outline 2100, the coordinates of the point are corrected to the coordinates of a point for constituting the first outline 2100.

[0301] As another example, the cuboid acquisition device can perform learning processing so that when only the left side of an object is observed, a line connecting two points in the up-down direction constituting the rear marker 2130 is limited to a line sharing the left side or right side of the first outline 2100. Specifically, the cuboid acquisition device can calculate the height of the rear surface of the object by taking into consideration the first outline 2100, the overall contour of the recognized object, and the direction of the observed object (left side), and set the length of the rear marker 2130 to the same length as the height. In this case, since the rear surface of the object is not observed in the left side direction of the object, the rear marker 2130 can be expressed as a line sharing at least a part of the left side or right side of the first outline. Referring to FIG. 21, it can be seen that the cuboid acquisition device generates the rear marker 2130 so as to share the right side of the first outline 2100.

[0302] FIG. 22 is a diagram illustrating an example of the cuboid of an object on the right side, acquired by the cuboid acquisition device.

[0303] In FIG. 22, the object is a car with only the right side observed in the image, the car is surrounded by a first outline 2200, and side points 2210 and rear marker 2230 are shown diagrammatically as cuboids relative to the object.

[0304] The cuboid acquisition device may first analyze an object to detect that only the right side is observed, and then secondarily acquire coordinates for a side point 2210 and a rear mark 2230 as a cuboid for the object. In some embodiments, the cuboid acquisition device may first acquire coordinates for a side point 2210 and a rear mark 2230 for an object, and then secondarily detect the direction of the object.

[0305] After detecting that only the left side of the object is observed in the image, the cuboid acquisition device may correct the y coordinates of two points in the upper and lower directions constituting the side point 2210 to the y coordinates of the most adjacent first outer boundary line 2200. For example, if the first outer boundary line 2200 includes a side formed from (x1, y5) to (x10, y5) and the coordinates of the upper end point of the side point 2210 are (x4, y4), the cuboid acquisition device may correct the coordinates of the upper end point of the side point 2210 to (x4, y5). For example, if the first outer boundary line 2200 includes a side formed from (x1, y5) to (x10, y5) and the coordinates of the upper end point of the side point 2210 are (x4, y7), the cuboid acquisition device may correct the coordinates of the upper end point of the side point 2210 to (x4, y5). That is, in the present invention, if the coordinates of a point before correction are outside or inside the first outer contour 2200, the coordinates of the point are corrected to the coordinates of a point for constituting the first outer contour 2200.

[0306] As another example, the cuboid acquisition device can perform learning processing so that when only the right side of the object is observed, the line connecting two points in the up-down direction constituting the rear marker 2230 is limited to a line sharing the left side or right side of the first outline 2200. Specifically, the cuboid acquisition device can calculate the height of the rear surface of the object by taking into consideration the first outline 2200, the overall contour of the recognized object, and the direction of the observed object (left side), and set the same length as the height to the length of the rear marker 2230. In this case, since the rear surface of the object is not observed in the right side direction of the object, the rear marker 2230 can be expressed as a line sharing at least a part of the left side or right side of the first outline. Referring to FIG. 22, it can be seen that the cuboid acquisition device generates the rear marker 2230 so as to share the left side of the first outline 2200.

[0307] FIG. 23 is a diagram illustrating an example of a cuboid of an object in the rear left direction, acquired by the cuboid acquisition device.

[0308] In FIG. 23, the object is a car in the rear left direction, the car is surrounded by a first outline 2300, and a side point 2310 and a rear marker 2330 are represented diagrammatically as cuboids relative to the object.

[0309] The cuboid acquisition device may first analyze an object to detect whether it is in the rear-left direction, and then secondarily acquire coordinates for a side point 2310 and a rear mark 2330 as a cuboid for the object. In some embodiments, the cuboid acquisition device may first acquire coordinates for a side point 2310 and a rear mark 2330 for an object, and then secondarily detect which direction the object is facing.

[0310] After detecting that the direction of the object is the rear left direction, the cuboid acquisition device may correct the y coordinates of two points in the upper and lower directions constituting the side point 2310 to the y coordinates of the most adjacent first outer boundary line 2300. For example, if the first outer boundary line 2300 includes a side formed from (x1, y5) to (x10, y5) and the coordinates of the upper end point of the side point 2310 are (x4, y4), the cuboid acquisition device may correct the coordinates of the upper end point of the side point 2310 to (x4, y5). As another example, if the first outer boundary line 2300 includes a side formed from (x1, y5) to (x10, y5) and the coordinates of the upper end point of the side point 2310 are (x4, y7), the cuboid acquisition device may correct the coordinates of the upper end point of the side point 2310 to (x4, y5).

[0311] Meanwhile, the cuboid acquisition device can determine the coordinates of the bottom end point of the side point 2310 using the length between the direction of the object and the start and end points that determine the height of the front of the object. The fact that the direction of the recognized object is the rear left direction means that the front of the object is not observed, so the x coordinate of the side point 2310 coincides with the x coordinate of the left side of the first outer contour line. Also, the fact that the direction of the object is the rear left direction means that the object is moving in a direction away from the observer, so the start point that determines the height of the front of the object has the y coordinate of the height of the top end of the first outer contour line, and the end point that determines the height of the front of the object has the y coordinate of the part where the left front tire of the object hits the ground. As a result, as shown in FIG. 23, the length of the line connecting the side points 2310 arranged at the upper and lower ends of the first outer contour line 2300 can be limited to the length of a part of the left side of the first outer contour line 2300.

[0312] As another example, the cuboid acquisition device may detect that the direction of the object is the rear left direction, and then perform learning processing so that the rear marker 2330 is limited to a rectangle that shares the right side of the first outline 2300. Referring to Fig. 23, it can be seen that the rear marker 2330 is displayed as a rectangle that shares the right side of the first outline 2300 and also shares a part of the upper side and the lower side of the first outline 2300. In Fig. 23, the width of the rear marker 2330 means the width of the rear of the object, and the height of the rear marker 2330 means the height of the rear of the object.

[0313] FIG. 24 is a diagram illustrating an example of a cuboid of an object in the rear right direction, acquired by the cuboid acquisition device.

[0314] In FIG. 24, the object is a car in the rear right direction, the car is surrounded by a first outline 2400, and side points 2410 and rear markers 2430 are represented diagrammatically as cuboids relative to the object.

[0315] The cuboid acquisition device may first analyze an object to detect that it is in the rear-right direction, and then secondarily acquire coordinates for a side point 2410 and a rear mark 2430 as a cuboid for the object. In some embodiments, the cuboid acquisition device may first acquire coordinates for a side point 2410 and a rear mark 2430 for an object, and then secondarily detect the direction of the object.

[0316] After detecting that the direction of the object is the rear right direction, the cuboid acquisition device may correct the y coordinates of two points in the upper and lower directions constituting the side point 2410 to the y coordinates of the most adjacent first outer boundary line 2400. For example, if the first outer boundary line 2400 includes a side formed from (x1, y5) to (x10, y5) and the coordinates of the upper end point of the side point 2410 are (x4, y4), the cuboid acquisition device may correct the coordinates of the upper end point of the side point 2410 to (x4, y5). As another example, if the first outer boundary line 2400 includes a side formed from (x1, y5) to (x10, y5) and the coordinates of the upper end point of the side point 2410 are (x4, y7), the cuboid acquisition device may correct the coordinates of the upper end point of the side point 2410 to (x4, y5).

[0317] Meanwhile, the cuboid acquisition device can determine the coordinates of the bottom end point of the side point 2410 using the length between the start point and the end point that determine the direction of the object and the height of the front of the object. The fact that the direction of the recognized object is the rear right direction means that the front of the object is not observed, so the x coordinate of the side point 2410 coincides with the x coordinate of the right side of the first outer contour 2400. Also, the fact that the direction of the object is the rear right direction means that the object is moving in a direction away from the observer, so the start point that determines the height of the front of the object has the y coordinate of the height of the top end of the first outer contour 2400, and the end point that determines the height of the front of the object has the y coordinate of the part where the right front tire of the object hits the ground. As a result, as shown in FIG. 24, the length of the line connecting the side points 2410 arranged at the upper and lower ends of the first outer contour 2400 can be limited to the length of a part of the right side of the first outer contour 2400.

[0318] As another example, the cuboid acquisition device may detect that the direction of the object is the rear right direction, and then perform learning processing so that the rear marker 2430 is limited to a rectangle that shares the left side of the first outline 2400. Referring to Fig. 24, it can be seen that the rear marker 2430 is displayed as a rectangle that shares the left side of the first outline 2400 and also shares parts of the upper and lower sides of the first outline 2400. In Fig. 24, the width of the rear marker 2430 means the width of the rear of the object, and the height of the rear marker 2430 means the height of the rear of the object.

[0319] FIG. 25 is a diagram illustrating an example of a cuboid of an object in the rear direction, acquired by the cuboid acquisition device.

[0320] In FIG. 25, the object is a car in the rearward direction, the car is surrounded by a first outline 2500, and side points 2510 and rear markers 2530 are represented diagrammatically as cuboids relative to the object.

[0321] If the direction of the object recognized in the image is completely rear only, the cuboid acquisition device can generate a side point 2510 by a line connecting two points located at the top and bottom ends of the very center of the object, as described in FIG. 19.

[0322] Furthermore, if the orientation of the recognized object is completely rearward, the cuboid acquisition device can match the rear mark 2530 with the first outline 2500, as shown in FIG.

[0323] As described with reference to Figs. 18 to 25, the cuboid acquisition device according to the present invention can train a learning model to generate a cuboid including a side point and a rear sign. Although the side point and rear sign acquired in this process have a very simple form, they can respectively indicate the direction, position, three-dimensional information, etc. of an object, and can ensure high efficiency and accuracy in recognizing and determining a moving object when implementing an automatic driving algorithm. In addition, the cuboid acquisition device according to the present invention can train a learning model to make the x-coordinates and y-coordinates of the side point and rear sign dependent on the first outline, and can generate accurate cuboid information.

[0324] FIG. 26 is a schematic diagram showing an example of a process for correcting rear markers using a cuboid acquisition device according to the present invention.

[0325] Referring to Fig. 26(a), it can be seen that a rear sign 11610A is shown inside the first outline of a passenger car (car). The passenger car in Fig. 26(a) is an object observed only on the right side, and the rear of the passenger car cannot be observed, so the rear sign must be a line sharing the entire left side of the first outline as in Fig. 26(b), not a square inside the first outline as in Fig. 26(a). The cuboid acquisition device according to the present invention can acquire a cuboid 2610B for the passenger car so that a result as in Fig. 26(b) is output, instead of a result as in Fig. 26(a).

[0326] FIG. 27 is a schematic diagram showing an example of a process for correcting rear markers using a cuboid acquisition device according to the present invention.

[0327] Referring to Fig. 27(a), it can be seen that a rear marker 2710A outside the first outline of the bus is shown at the rear of the bus. Since the bus in Fig. 27(a) is an object on the left side of the front and the rear of the bus cannot be observed, the rear marker should not be a rectangle outside the first outline as in Fig. 27(a) but a line 2710B sharing a part of the right side of the first outline as in Fig. 27(b). The cuboid acquisition device according to the present invention can acquire a cuboid for a bus so that a result as in Fig. 27(b) is output instead of a result as in Fig. 27(a).

[0328] FIG. 28 is a schematic diagram showing another example of a process for correcting rear markers using a cuboid acquisition device according to the present invention.

[0329] Referring to Fig. 28(a), it can be seen that a rear sign 2810A is shown at the rear of a passenger car (car) surrounding a part of the rear of the passenger car. Since the passenger car in Fig. 28(a) is an object in the rear left direction and the entire rear of the passenger car can be observed, the rear sign must be a rectangle 2810B sharing the right side of the first outline as in Fig. 28(b). The cuboid acquisition device according to the present invention can acquire a cuboid for the passenger car in the rear left direction so that a result as in Fig. 28(b) is output, instead of a result as in Fig. 28(a).

[0330] FIG. 29 is a flowchart showing an example of a method for obtaining a cuboid according to the present invention.

[0331] The method according to FIG. 29 can be realized by the cuboid obtaining apparatus 1700 described in FIG. 17, so in the following, a description that overlaps with the contents described in FIG. 17 to FIG. 28 will be omitted.

[0332] The cuboid acquisition device 1700 may recognize at least one object included in a first image acquired while driving, and generate a first outline for each recognized object (S2910).

[0333] The cuboid acquisition device 1700 may extract first coordinate values ​​constituting a first contour line generated from the first image (S2930).

[0334] The cuboid acquisition device 1700 can acquire the coordinates of the cuboid of the recognized object based on the recognized object and the extracted first coordinate value (S2950).

[0335] FIG. 30 is a block diagram of a cuboid acquisition device according to one embodiment.

[0336] Referring to Fig. 30, a cuboid acquisition device 3000 can include a communication unit 3010, a processor 3020, and a DB 3030. Only components related to the embodiment are shown in the cuboid acquisition device 3000 in Fig. 30. Therefore, it can be understood by a person skilled in the art that other general-purpose components can be further included in addition to the components shown in Fig. 30.

[0337] The communication unit 3010 may include one or more components that enable wired / wireless communication with an external server or device. For example, the communication unit 3010 may include at least one of a short-range communication unit (not shown), a mobile communication unit (not shown), and a broadcast receiving unit (not shown).

[0338] The DB 3030 is hardware that stores various data processed within the cuboid acquisition device 3000, and can store programs for processing and controlling the processor 3020.

[0339] DB3030 may include dynamic random access memory (DRAM), random access memory (RAM), such as static random access memory (SRAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), CD-ROM, Blu-ray or other optical disk storage, hard disk drive (HDD), solid state drive (SSD), or flash memory.

[0340] The processor 3020 controls the overall operation of the cuboid acquisition device 3000. For example, the processor 3020 can overall control an input unit (not shown), a display (not shown), a communication unit 3010, a DB 3030, etc. by executing a program stored in the DB 3030. The processor 3020 can control the operation of the cuboid acquisition device 3000 by executing a program stored in the DB 3030.

[0341] The processor 3020 can control at least a part of the operation of the cuboid obtaining device described above with reference to FIGS.

[0342] As an example, the processor 3020 may recognize at least one object included in a first image acquired while driving, generate a first outline for each recognized object, extract first coordinate values ​​constituting the generated first outline from the first image, and obtain cuboid coordinates of the recognized object based on the recognized object and the extracted first coordinate values.

[0343] The processor 3020 may be implemented using at least one of ASICs (application specific integrated circuits), DSPs (digital signal processors), DSPDs (digital signal processing devices), programmable logic devices (PLDs), field programmable gate arrays (FPGAs), controllers, micro-controllers, microprocessors, and electrical units for performing other functions.

[0344] FIG. 31 is a diagram conceptually illustrating the operation process of the visual landmark correction device for an object in an image according to the present invention.

[0345] More specifically, Fig. 31 is a diagram conceptually illustrating a process in which raw data is input to a visual marker correction device 3100 for an object in an image and is finally processed into data for which labelling has been completed. For the sake of simplicity, hereinafter, the device for correcting visual markers of an object in an image according to the present invention will be referred to as a "visual marker correction device."

[0346] Typically, a camera installed in an autonomous vehicle can capture scenery observed while the autonomous vehicle is traveling, and the image captured by the camera can become raw data in FIG. 31. The raw data can be a video composed of multiple frames, and each frame constituting the video may capture multiple objects. The raw data can be input to an artificial intelligence-based learning model such as a deep learning model and labeled. An auto-labelling process performed by the learning model can recognize objects in the video and obtain characteristic information of the recognized objects, but since the labeling result is not perfect, the first labeled data can be inspected by an inspector 3110. Data that has been corrected in the inspection process can be classified as training data for the learning model and as data for which labeling has been completed, and data that is determined not to require correction by the inspector 3110 in the inspection process can be immediately classified as data for which labeling has been completed without becoming training data for the learning model.

[0347] The present invention proposes a method for improving the speed and accuracy of labeling objects included in an image captured during driving by implementing an inspector 3110 who inspects the primarily labeled data with an automated device instead of a human. The visual sign correction device 3100 according to the present invention receives raw data, performs primary auto-labeling, inspects the results of the auto-labeling, corrects (modifies) parts where labeling is not complete, and classifies data where labeling is complete by repeating the process, thereby improving an artificial intelligence model included physically or logically.

[0348] As described with reference to Figs. 18 to 25, the visual sign correction device according to the present invention can train a learning model to generate a cuboid including a side point and a rear sign. Although the side point and rear sign acquired in this process have a very simple form, they can respectively indicate the direction, position, three-dimensional information, etc. of the object, and can ensure high efficiency and accuracy in recognizing and determining a moving object when implementing an autonomous driving algorithm. In addition, the visual sign correction device according to the present invention can train a learning model to make the x and y coordinates of the side point and rear sign dependent on the outline, and can generate accurate cuboid information.

[0349] FIG. 32 is a schematic diagram showing an example of a first correction process performed by the visual sign correction device according to the present invention.

[0350] 32(a) shows a schematic diagram of a video including an object being input to a learning model, and side points 3210, 3211 and a rear marker 3230 being generated as cuboids for the object. The upper end point 3210 and the lower end point 3211 of the side points 3210, 3211 in FIG. 32(a) are both located outside the outline 3200 of the object. The rear marker 3230 in FIG. 32(a) is a rectangle whose height is longer than its width, and the coordinates of some of the points constituting the rear marker 3230 are located outside the outline 3200.

[0351] When the primary labeled data as shown in Fig. 32(a) is generated, the visual sign correction device according to the present invention detects that the upper end point 3210 and the lower end point 3211 of the side points 3210, 3211 are located outside the outline 3200, and can perform a first correction to move the upper end point 3210 and the lower end point 3211 of the side points 3210, 3211 in the direction of the most adjacent outline point. Although not shown in Fig. 32(a), the first correction can also be applied in the same manner when only one of the upper end point and the lower end point constituting the side points is located outside the outline 3200.

[0352] Furthermore, the visual sign correction device can detect the shape of the rectangular rear sign 3230, delete the parts that are outside the outline 3200, and then correct the top and right sides of the outline 3200 to form the rear sign 3230.

[0353] Referring to FIG. 32(b), it can be seen that the upper end point 3210 and the lower end point 3211 of the side points 3210, 3211 located outside the outline 3200 in FIG. 32(a) have both been moved to the positions of the points that constitute the outline 3200, and that the part of the rear marker 3230 that is outside the outline 3200 has been deleted, and the area of ​​the rectangular rear marker 3230 has been reduced.

[0354] In Figures 32(a) and 32(b), the rear mark 3230 is corrected by deleting only the portion located outside the outline 3200. However, in some embodiments, in order to remove the portion of the rear mark 3230 located outside the outline 3200, the visual mark correction device may move the rear mark 3230 inside the outline 3200 while maintaining the width of the rear mark 3230 as it is.

[0355] FIG. 33 is a schematic diagram showing an example of a second correction process performed by the visual sign correction device according to the present invention.

[0356] For convenience of explanation, Fig. 33(a) is the same diagram as Fig. 32(b) arranged. In Fig. 33(a), the upper end point 3310 and the lower end point 3311 of the side points 3310, 3311 have been changed to be within the outline 3300.

[0357] The visual sign correction device according to the present invention detects the direction of the object in a state where a first correction is applied as in Fig. 33(a), and then changes the rear sign 3330 in Fig. 33(a) to the rear sign 3330 in Fig. 33(b) based on the detected direction and the height of the rear of the object. Specifically, when the visual sign correction device detects that the direction of the object is the front-left direction, it can perform a second correction to change the square-shaped rear sign 3330 in Fig. 33(a) to the linear rear sign 3330 in Fig. 33(b) to typically indicate that the rear of the object is not observed.

[0358] In the above-mentioned second correction process, the visual sign correction device can also move the position of the exposure mark 3350 to a point that forms the outline 3300. The exposure mark 3350 can indicate whether or not the image shows that no part of the object is missing and that the entire part has been photographed (exposed). Since the object is entirely surrounded by the outline 3300 in FIG. 33(b), the visual sign correction device can change the y coordinate of the exposure mark 3350 and move it to the position of the point that forms the lower edge of the outline 3300.

[0359] FIG. 34 is a schematic diagram showing another example of the second correction process performed by the visual sign correction device according to the present invention.

[0360] The visual landmark correction device according to the present invention detects the direction of an object when a first correction is applied as shown in FIG. 34(a), and then corrects the side point 3410 and rear landmark 3430 in FIG. 34(a) to the side point 3410 and rear landmark 3430 in FIG. 34(b) based on the detected direction and the height of the rear of the object.

[0361] Specifically, when the direction of the object is detected as being completely left-only, the visual marker correction device can perform a second correction to change the position of the side point 3410 in Fig. 34(a) to the position of the side point 3410 in Fig. 34(b) and to change the rectangular rear marker 3430 in Fig. 34(a) to the linear rear marker 3430 in Fig. 34(b) in order to show that both the front and rear of the object are not observed. When the second correction is completed, the side point 3410 coincides with the left side of the outline 3400, and the rear marker 3430 coincides with the right side of the outline 3400.

[0362] FIG. 35 is a schematic diagram showing another example of the second correction process performed by the visual sign correction device according to the present invention.

[0363] The visual landmark correction device according to the present invention detects the direction of an object when a first correction is applied as shown in FIG. 35(a), and then corrects the side point 3510 and rear landmark 3530 in FIG. 35(a) to the side point 3510 and rear landmark 3530 in FIG. 35(b) based on the detected direction and the height of the rear of the object.

[0364] Specifically, when the direction of the object is detected as being completely rear only, the visual marker correction device can perform a second correction to change the position of the side point 3510 in Fig. 35(a) to the position of the side point 3510 in Fig. 35(b) and to change the rectangular rear marker 3530 in Fig. 35(a) to the linear rear marker 3530 in Fig. 35(b) in order to show that both the front and side of the object are not observed. When the second correction is completed, the side point 3510 is represented by two points and a connecting line that separate the exact centers of the upper and lower edges of the outline 3500, and the rear marker 3530 coincides with the outline 3500.

[0365] FIG. 36 is a flow chart illustrating an exemplary method for visual landmark correction according to the present invention.

[0366] The method according to FIG. 36 can be realized by the visual sign correcting device 3100 described in FIG. 31, and therefore, in the following, a description that overlaps with the contents described in FIG. 31 to FIG. 35 will be omitted.

[0367] The visual sign correction device 3100 may input an image acquired while driving to a learning model and generate a visual sign for the position and direction of an object detected from the image (S3610). Alternatively, an object included in the image acquired while driving may be recognized, and the recognized object may be input to the learning model to generate a visual sign for the position and direction of the object.

[0368] The visual marker correction device 3100 may perform a first correction to move the position of the visual marker generated in step S3610 to inside the outline of the object (S3630).

[0369] The visual mark correction device 3100 may perform a second correction (S3650) to change at least one of the position and the shape of the visual mark that has been moved inside the outline based on the characteristics of the visual mark.

[0370] FIG. 37 is a block diagram of a visual landmark correction device according to one embodiment.

[0371] Referring to Fig. 37, the visual sign correction device 3700 may include a communication unit 3710, a processor 3720, and a DB 3730. Only components related to the embodiment are shown in the visual sign correction device 3700 in Fig. 37. Therefore, it can be understood by a person skilled in the art that the visual sign correction device 3700 may further include other general-purpose components in addition to the components shown in Fig. 37.

[0372] The communication unit 3710 may include one or more components that enable wired / wireless communication with an external server or device. For example, the communication unit 3710 may include at least one of a short-range communication unit (not shown), a mobile communication unit (not shown), and a broadcast receiving unit (not shown).

[0373] The DB 3730 is hardware that stores various data processed within the visual sign correction device 3700, and can store programs for processing and controlling the processor 3720.

[0374] The DB3730 may include dynamic random access memory (DRAM), random access memory (RAM), such as static random access memory (SRAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), CD-ROM, Blu-ray or other optical disk storage, hard disk drive (HDD), solid state drive (SSD), or flash memory.

[0375] The processor 3720 controls the overall operation of the visual sign correction device 3700. For example, the processor 3720 may execute a program stored in the DB 3730 to overall control an input unit (not shown), a display (not shown), a communication unit 3710, the DB 3730, etc. The processor 3720 may control the operation of the visual sign correction device 3700 by executing a program stored in the DB 3730.

[0376] Processor 3720 may control at least a portion of the operation of the visual landmark correction device described above in FIGS.

[0377] As an example, the processor 3720 can use images captured during driving as input for a learning model to generate visual markers for the position and orientation of an object detected from the images, perform a first correction to move the position of the generated visual marker inside the outline of the object, and perform a second correction to change at least one of the position and shape of the visual marker moved inside the outline based on the characteristics of the visual marker.

[0378] The processor 3720 may be implemented using at least one of ASICs (application specific integrated circuits), DSPs (digital signal processors), DSPDs (digital signal processing devices), programmable logic devices (PLDs), field programmable gate arrays (FPGAs), controllers, micro-controllers, microprocessors, and electrical units for performing other functions.

[0379] The embodiments according to the present invention may be implemented in the form of a computer program executable by various components on a computer, and such a computer program may be recorded on a computer-readable medium, which may include hardware devices specially configured to store and execute program instructions, such as magnetic media such as hard disks, floppy disks, and magnetic tapes, optical recording media such as CD-ROMs and DVDs, magneto-optical media such as floptical disks, ROMs, RAMs, flash memories, and the like.

[0380] On the other hand, the computer programs may be those specially designed and constructed for the present invention, or those well known and available to those skilled in the art of computer software. Examples of computer programs include not only machine code, such as produced by a compiler, but also high-level language code that can be executed by a computer using an interpreter, etc.

[0381] According to one embodiment, the methods according to various embodiments of the present disclosure may be provided in a computer program product. The computer program product may be traded between a seller and a buyer as a commodity. The computer program product may be distributed in the form of a machine-readable storage medium (e.g., a compact disc read only memory (CD-ROM)) or may be distributed through an application store (e.g., the Play Store, etc.). TM The computer program product may be distributed online (e.g., downloaded or uploaded) via a web browser, a mobile device, or directly between two user devices. In the case of online distribution, at least a portion of the computer program product may be at least temporarily stored or temporarily generated on a machine-readable storage medium such as a memory of a manufacturer's server, an application store's server, or an intermediary server.

[0382] Unless otherwise clearly stated or contrary to the order of steps constituting the method according to the present invention, the steps may be performed in any suitable order. The order of steps described above is not necessarily intended to limit the present invention. The use of all examples or exemplary terms (such as, for example, etc.) in the present invention is merely for the purpose of explaining the present invention in detail, and the scope of the present invention is not limited by the examples or exemplary terms unless otherwise limited by the claims. Furthermore, those skilled in the art will appreciate that various modifications, combinations, and changes can be made according to design conditions and factors within the scope of the appended claims or their equivalents.

[0383] Therefore, the spirit of the present invention should not be limited to the above-described embodiments, and all scopes equivalent to or modified equivalently to the scope of the claims, as well as the scope of the claims described below, fall within the scope of the idea of ​​the present invention.

Claims

1. Methods for annotating detected objects include: classifying the class of the object; generating a bounding box for the object based on a class of the object, the bounding box including a first rectangle and a second rectangle that share one side; determining attributes of the object; and generating annotations for classes of objects and attributes of objects on a bounding box basis; The attributes of the object are: The method includes visibility, movement state, lane position, main state, object size, and subclasses.

2. The class of the object is Whether it is a car, van, truck, motorcycle, pedestrian, emergency vehicle or other; The method of claim 1.

3. The method of claim 1 , wherein if the object is loaded with other vehicles as an automobile designed to carry cargo, the loaded other vehicles do not classify a class.

4. The step of generating a bounding box includes: The method of claim 1 , further comprising applying different criteria to generate bounding boxes depending on the class of the classified object.

5. the first quadrangle is a rectangle; The method of claim 1 , wherein the second rectangle is a trapezoid.

6. the first rectangle corresponds to a front or back face of the object; The method of claim 1 , wherein the second rectangle corresponds to a left or right side of the object.

7. When the top surface of the object is exposed, The method of claim 6 , wherein the first rectangle or the second rectangle comprises a top surface of the object.

8. If the object is a vehicle having wheels, The method of claim 1 , wherein the second quadrilateral includes line segments connecting wheel ground points.

9. If the class of the object is a motorbike or a pedestrian, The method of claim 4 , wherein the width of the first rectangle is generated to be equal to a shoulder width of a person included in the object.

10. The method comprises: The method of claim 1 , further comprising: determining not to generate the bounding box if the detected object is a vehicle and a percentage of the detected object seen in the video data is less than a critical percentage of the object's overall size.

11. The method comprises: The method of claim 1 , further comprising controlling an ego-vehicle based on the generated annotations.

12. 1. An apparatus for annotating a detected object, comprising: a memory having at least one program stored therein; a processor operable by executing said at least one program; The processor, classifying the class of the object; generating a bounding box for the object based on a class of the object, the bounding box including a first rectangle and a second rectangle that share one side; determining attributes of the object; generating annotations for the object class and object attributes for each of the bounding boxes; The attributes of the object are: The device includes visibility, movement state, lane position, main state, object size, and subclasses.

13. A computer-readable recording medium having recorded thereon a program for causing a computer to execute the method according to claim 1.

Images