Method and device for training object detection model

Abstract

The present invention provides an object detection training method and an object detection training device. The object detection training device determines the posture and size of an object and a bounding box at various viewpoints according to an input image based on an object detection model, and trains the object detection model based on loss.

Description

[0001] [CROSS-REFERENCE TO RELATED APPLICATIONS]

[0002] This application claims the benefit of Korean Patent Application No. 10-2018-0097170 filed on August 21, 2018, in the Korean Intellectual Property Office, the disclosure of which is incorporated herein by reference in its entirety for all purposes. Technical Field

[0003] The following description relates to a technique for training an object detection model. Background Art

[0004] Recently, in order to solve the problem of classifying input patterns into predetermined groups, research is being actively conducted on trying to apply effective pattern recognition methods of humans to actual computers. The research includes research on artificial neural networks (ANNs), which are obtained by modeling the characteristics of human biological neurons by mathematical expressions. In order to solve the above problems, ANNs use algorithms that simulate human learning abilities. ANNs use the algorithms to generate mappings between input patterns and output patterns, which indicates that ANNs have learning abilities. In addition, ANNs have a generalization ability to generate relatively correct outputs based on training results for input patterns that have not yet been used for training.

[0005] For example, an autonomous vehicle may use an ANN to detect objects in front of the autonomous vehicle. Summary of the invention

[0006] This Summary is provided to introduce a selection of concepts in a simplified form that are further described in the Detailed Description below. This Summary is not intended to identify key features or essential features of the claimed subject matter, nor is it intended to be used as an aid in determining the scope of the claimed subject matter.

[0007] In some embodiments provided herein, an object detection training method is provided. In some embodiments, the method includes: using an object detection model to estimate the pose and size of an object based on features extracted from an input image; calculating a three-dimensional (3D) bounding box according to the pose and the size; determining a first output bounding box corresponding to the object by projecting the 3D bounding box to a first projection image; determining a second output bounding box corresponding to the object by projecting the 3D bounding box to a second projection image; and training the object detection model based on the pose, the size, the first output bounding box, and the second output bounding box, the training including a fusion operation of the first output bounding box and the second output bounding box.

[0008] In some embodiments of the method, determining the first output bounding box comprises determining a bird's eye view bounding box corresponding to the object by projecting the 3D bounding box onto a bird's eye view projection image.

[0009] In some embodiments of the method, determining the second output bounding box comprises determining a perspective bounding box corresponding to the object by projecting the 3D bounding box onto a perspective projection image.

[0010] In some embodiments of the method, estimating the pose and the size of the object includes extracting features from i) a two-dimensional (2D) bounding box corresponding to the object detected from the input image and ii) a cropped image corresponding to the 2D bounding box.

[0011] In some embodiments of the method, training the object detection model includes: calculating a loss based on the pose, the size, the first output bounding box, and the second output bounding box; and training the object detection model based on the loss, the training including updating weights in a neural network.

[0012] In some embodiments of the method, calculating the loss includes: calculating a first loss based on an overlap degree between the first output bounding box and a first reference bounding box; and calculating a second loss based on an overlap degree between the second output bounding box and a second reference bounding box.

[0013] In some embodiments of the method, calculating the loss includes: determining a target frame based on a front frame indicating the front of the object; and calculating a loss of a perspective bounding frame based on an overlap degree between the target frame and a reference frame.

[0014] In some embodiments of the method, estimating the posture and the size of the object includes: estimating a position offset based on the features extracted from the input image; estimating an initial position based on the posture and the size; determining an output position by applying the position offset to the estimated initial position; and calculating the loss is performed based on the determined output position.

[0015] In some embodiments of the method, estimating the pose and the size comprises estimating a confidence level for each of the estimated pose and the estimated size based on the features extracted from the input image.

[0016] In some embodiments of the method, estimating the size of the object includes estimating a width, a length, and a height of the object.

[0017] An additional object detection training method is also provided, including: using an object detection model to estimate the pose of an object based on features extracted from an input image and a depth image; using the object detection model to estimate a three-dimensional (3D) bounding box corresponding to the object based on the features extracted from the input image and the depth image; determining an output bounding box corresponding to the object by projecting the 3D bounding box onto a projected image; and training the object detection model based on the pose, the 3D bounding box, and the output bounding box, the training including a fusion operation on the 3D bounding box and the output bounding box.

[0018] Determining the output bounding box includes determining a perspective bounding box corresponding to the object by projecting the 3D bounding box onto a perspective projection image.

[0019] In some embodiments of the additional object detection method, extracting the features from the input image and the depth image is performed based on regions computed by a region proposal network (RPN).

[0020] In some embodiments of the additional object detection method, training the object detection model includes: calculating a loss based on the pose, the 3D bounding box, and the output bounding box; and training the object detection model based on the loss, wherein the training includes updating weights in a neural network.

[0021] In some embodiments of the additional object detection method, calculating the loss includes calculating the loss of the output bounding box based on a degree of overlap between the output bounding box and a reference bounding box.

[0022] In some embodiments of the additional object detection method, calculating the loss includes: determining a target frame based on a front frame indicating the front of the object according to a perspective bounding frame corresponding to the object; and calculating the loss of the perspective bounding frame based on a degree of overlap between the target frame and a reference frame.

[0023] In some embodiments of the additional object detection method, estimating the pose and the 3D bounding box includes estimating positions of points that at least partially define the 3D bounding box.

[0024] In some embodiments of the additional object detection method, estimating the pose and the 3D bounding box is performed based on a depth image corresponding to a bird's eye view.

[0025] In some embodiments of the additional object detection method, the fusion operation of the first output bounding box and the second output bounding box includes extracting features by identifying features obtained by cropping the first output bounding box and by cropping the second output bounding box.

[0026] The present invention also provides an object detection training device, comprising: a memory configured to store an object detection model; and a processor configured to: use the object detection model to estimate the posture and size of an object based on features extracted from an input image, calculate a three-dimensional (3D) bounding box according to the posture and the size, determine a first output bounding box corresponding to the object by projecting the 3D bounding box to a first projection image, determine a second output bounding box corresponding to the object by projecting the 3D bounding box to a second projection image, and train the object detection model based on the posture, the size, the first output bounding box and the second output bounding box and further based on a fusion operation of the first output bounding box and the second output bounding box.

[0027] In one general aspect, an object detection training method includes: using an object detection model to estimate the pose and size of an object based on features extracted from an input image, determining a first output bounding box corresponding to the object by projecting a three-dimensional (3D) bounding box calculated based on the pose and the size onto a first projection image, determining a second output bounding box corresponding to the object by projecting the calculated 3D bounding box onto a second projection image, and training the object detection model based on the pose, the size, the first output bounding box, and the second output bounding box.

[0028] Determining the first output bounding box may include determining a bird's-eye view bounding box corresponding to the object by projecting the calculated 3D bounding box onto a bird's-eye view projection image.

[0029] Determining the second output bounding box may include determining a perspective bounding box corresponding to the object by projecting the calculated 3D bounding box onto a perspective projection image.

[0030] Estimating the pose and the size of the object may include extracting features from a two-dimensional (2D) bounding box corresponding to the object detected from the input image and a cropped image corresponding to the 2D bounding box based on the object detection model.

[0031] Training the object detection model may include calculating a loss based on the pose, the size, the first output bounding box, and the second output bounding box, and training the object detection model based on the calculated loss.

[0032] Calculating the loss may include calculating the loss of the first output bounding box based on an overlap degree between the first output bounding box and a first reference bounding box, and calculating the loss of the second output bounding box based on an overlap degree between the second output bounding box and a second reference bounding box.

[0033] Calculating the loss may include determining a target frame based on a front frame indicating the front of the object, and calculating the loss of the perspective bounding frame based on an overlap degree between the target frame and a reference frame.

[0034] Estimating the pose and the size of the object may include further estimating a position offset based on the estimated features, estimating an initial position based on the pose and the size, and determining an output position by applying the position offset to the estimated initial position. Calculating the loss may include further calculating the loss based on the determined output position.

[0035] Estimating the pose and the size of the object may include estimating a confidence level for each of the estimated pose and the estimated size as a function of the extracted features.

[0036] Estimating the pose and the size of the object may include estimating a width, a length, and a height of the object as the size.

[0037] In another general aspect, an object detection training method includes: using an object detection model to estimate the pose of an object and a 3D bounding box corresponding to the object based on features extracted from an input image and a depth image, determining an output bounding box corresponding to the object by projecting the 3D bounding box to a projection image, and training the object detection model based on the pose, the 3D bounding box, and the output bounding box.

[0038] Determining the output bounding box may include determining a perspective bounding box corresponding to the object by projecting the 3D bounding box onto a perspective projection image.

[0039] Estimating the pose and the 3D bounding box may include extracting features from the input image and the depth image based on regions computed by a region proposal network (RPN).

[0040] Training the object detection model may include calculating a loss based on the pose, the 3D bounding box, and the output bounding box, and training the object detection model based on the calculated loss.

[0041] Calculating the loss may include calculating the loss of the output bounding box based on a degree of overlap between the output bounding box and a reference bounding box.

[0042] Calculating the loss may include determining a target frame based on a front frame indicating a front of the object and a back frame indicating a back of the object according to a perspective bounding frame corresponding to the object, and calculating the loss of the perspective bounding frame based on a degree of overlap between the target frame and a reference frame.

[0043] Estimating the pose and the 3D bounding box may include estimating positions of points defining the 3D bounding box.

[0044] Estimating the pose and the 3D bounding box may include acquiring the depth image corresponding to a bird's eye view.

[0045] In another general aspect, an object detection training device includes: a memory configured to store an object detection model; and a processor configured to: use the object detection model to estimate the pose and size of an object based on features extracted from an input image, determine a first output bounding box corresponding to the object by projecting a 3D bounding box calculated based on the pose and the size onto a first projection image, determine a second output bounding box corresponding to the object by projecting the calculated 3D bounding box onto a second projection image, and train the object detection model based on the pose, the size, the first output bounding box, and the second output bounding box.

[0046] Other features and aspects will be apparent from the following detailed description, drawings, and claims. BRIEF DESCRIPTION OF THE DRAWINGS

[0047] Figure 1 Shows an example of a bounding box for object detection.

[0048] Figure 2 is a block diagram illustrating an example of an object detection training device.

[0049] Figure 3 is a flow chart illustrating an example of an object detection training method.

[0050] Figure 4 Shows an example of a neural network as an object detection model.

[0051] Figure 5 An example of the object detection training process is shown.

[0052] Figure 6 Examples of a perspective projection image and a bird's-eye view projection image are shown.

[0053] Figure 7 An example of intersection-over-union (IoU) is shown.

[0054] Figure 8 An example of a perspective bounding box is shown.

[0055] Fig. 9 is a block diagram illustrating another example of an object detection training device.

[0056] Fig.10 Another example of an object detection training process is shown.

[0057] Fig.11 is a flow chart illustrating another example of an object detection training method.

[0058] Fig.12 Another example of an object detection training device is shown.

[0059] Fig.13 An example of the feature extraction process is shown.

[0060] Fig.14 Another example of an object detection training process is shown.

[0061] Fig.15 is a block diagram showing an example of a configuration of an object detection training device.

[0062] Fig.16 is a block diagram showing an example of the configuration of an object detection device.

[0063] Throughout the drawings and detailed description, unless otherwise specified or provided, the same figure reference numerals will be understood to refer to the same elements, features and structures. The drawings may not be scaled, and the relative sizes, proportions and depictions of the elements in the drawings may be exaggerated for clarity, illustration and convenience. DETAILED DESCRIPTION

[0064] The following detailed description is provided to help the reader fully understand the methods, devices and / or systems described herein. However, various changes, modifications and equivalent forms of the methods, devices and / or systems described herein will be apparent after understanding the disclosure of the application. For example, the operation sequences described herein are merely examples and are not limited to those operation sequences described herein, but may be changed as will be apparent after understanding the disclosure of the application, except for operations that necessarily occur in a certain order. In addition, for the sake of clarity and simplicity, features known in the art may not be described in detail.

[0065] The features described herein may be implemented in different forms and should not be construed as being limited to the examples described herein. Rather, the examples described herein are provided merely to illustrate some of the many possible ways to implement the methods, devices and / or systems described herein that will be apparent after understanding the disclosure of the present application.

[0066] Various modifications can be made to the following examples. Here, the examples are not interpreted as being limited to the present invention, and should be understood to include all changes, equivalent forms and replacement forms within the concept and technical scope of the present invention.

[0067] The terms used herein are only used to illustrate specific examples, not to limit the examples. Unless the context clearly indicates otherwise, the singular forms used herein are also intended to include the plural forms. It should be further understood that the terms "comprises and / or comprising" when used in this specification indicate the presence of the features, integers, steps, operations, elements, components or combinations thereof, but do not exclude the presence or addition of one or more other features, integers, steps, operations, elements, components and / or groups thereof.

[0068] Unless otherwise defined herein, all terms used herein, including technical or scientific terms, have the same meaning as generally understood. Terms defined in commonly used dictionaries should be interpreted as having meanings that match the contextual meanings in the relevant technology, and unless otherwise defined herein, should not be interpreted as ideal or overly formal meanings.

[0069] Hereinafter, various examples will be described in detail with reference to the accompanying drawings, and the same reference numerals in the drawings refer to the same elements throughout.

[0070] Figure 1 Shows an example of a bounding box for object detection.

[0071] The object detection apparatus detects an object from the image 100 based on an object detection model. The object detection model is a model trained to output information associated with the position, size, and posture of an object based on the image 100, and will be further described below.

[0072] The object detection device uses a sensor to acquire an image 100. The sensor is configured to sense the intensity of a signal (e.g., light or electromagnetic wave) reflected from a background or object located around the object detection device (e.g., the front side of the object detection device). For example, the object detection device uses a sensor to acquire a color image, an infrared image, or a depth image. The color image includes a plurality of color channel images, and for example includes a red image indicating the intensity corresponding to the red wavelength of visible light, a green image indicating the intensity corresponding to the green wavelength, and a blue image indicating the intensity corresponding to the blue wavelength. The infrared image represents the intensity corresponding to the infrared light received by the sensor. The depth image represents the depth (e.g., distance) to an arbitrary target point. The depth image is acquired by, for example, a light detection and ranging (LiDAR) sensor.

[0073] Figure 1 The bounding box as a result of object detection is shown. Figure 1 In , a bounding box indicates an object detected from a two-dimensional (2D) image (e.g., a color image or a black and white image). Object detection is performed by feature extraction. For example, feature extraction can be used to identify features such as a vehicle's headlights, wheels, windshield, and doors, see Figure 1 Based on the extracted features, individual vehicles can be detected. A bounding box is a box that surrounds the vehicle detected in the image. For example, specific features of a vehicle can be identified and listed in a feature vector. An example of a feature vector is Fig.13 Item 1321 (hereinafter referred to as Fig.13 The smallest rectangular prism that contains all elements of the eigenvector is the 3D bounding box, see Figure 1 Item 110. The projection of a 3D bounding box onto a 2D plane is a 2D bounding box, see Figure 1 Items 121 and 122 are shown.

[0074] The object detection device estimates the size, position and pose (orientation) of a bounding box containing the object in a three-dimensional (3D) space based on the image 100. The 3D space is the physical environment that the object detection device exists in. For example, when the object detection device is installed in a vehicle, the 3D space is the space around the vehicle.

[0075] The bounding box is the box that contains the object. Figure 1 , 3D bounding box 110 has an arbitrary shape (e.g., a rectangular shape) and includes a space occupied by an object in 3D space. For example, 3D bounding box 110 is a minimum bounding box that is defined such that the size of 3D bounding box 110 is minimized while each of the faces of 3D bounding box 110 is in contact with a portion of the object. When the object is a vehicle, the front side and the rear side of the vehicle are in contact with the front face 121 and the rear face 122 of 3D bounding box 110, respectively. In addition, the top side and the bottom side of the vehicle may be in contact with the top surface and the bottom surface of 3D bounding box 110, respectively. The side of the vehicle may be in contact with the side of 3D bounding box 110.

[0076] In an example, the object detection device estimates the coordinates of each of the vertices of the 3D bounding box 110 in the 3D space based on the object detection model. The object detection model may be implemented using a neural network, see Figure 4In another example, the object detection device estimates the pose, size, and reference point (e.g., center point or any vertex) of the 3D bounding box 110. The size includes, for example, the height, width, and length of the object. In an example of autonomous driving or driving assistance, the aforementioned object may be mainly a vehicle, and the front 121 and the back 122 of the 3D bounding box 110 detected from the image 100 may be mainly used.

[0077] The object detection device detects objects from an image based on a fully trained object detection model. For example, the object detection device determines a bounding box (e.g., 3D bounding box 110) corresponding to the object. The object detection training device trains the object detection model based on the training data. Hereinafter, an example of the operation of the object detection training device for training the object detection model will be described.

[0078] Figure 2 is a block diagram illustrating an example of an object detection training device 200 .

[0079] The object detection training apparatus 200 includes a feature extractor 210, an estimator 220, a first projection calculator 231, a second projection calculator 232, a loss calculator 240, and a trainer 250. The object detection model 270 may include the feature extractor 210, the estimator 220, the first projection calculator 231, and the second projection calculator 232.

[0080] The feature extractor 210 may divide the image into a grid. The grid may consist of a plurality of cells. Within each cell, an object may appear as a simple geometric shape.

[0081] The feature extractor 210 extracts features from the image. For example, the feature extractor 210 includes at least one convolutional layer of a neural network. The feature extractor 210 extracts features obtained by abstracting the image from the bounding box and the cropped image 201. The input bounding box is a 2D bounding box and is obtained by the aforementioned operation of detecting the 2D bounding box of the object from the input image. The cropped image is a partial image of the input image corresponding to the 2D bounding box.

[0082] The estimator 220 estimates the pose and size 202 of the object based on the extracted features. For example, the estimator 220 may compare the extracted features from the feature extractor 210 with the information of the candidate vehicle. The estimator 220 estimates the pose and size 202 based on the extracted features using regression analysis. The size of the object is information indicating the size of the object, and includes, for example, the height, width, and length of the object. The estimator 220 estimates the height, width, and length of the 3D bounding box containing the object as the size of the object. In addition, the estimator 220 estimates the pose of the 3D bounding box containing the object as the pose of the object. The pose of the 3D bounding box is information indicating the orientation of the object, and is represented, for example, by the angular difference between the direction from the sensor toward the center part of the object and the moving direction of the object. However, the pose and size of the object are not limited to the above description.

[0083] In the present invention, the sensor includes, for example, a sensor for collecting training data and a sensor installed in an object detection device. For example, the above-mentioned direction from the sensor toward the center of the object is a direction from the sensor corresponding to the viewpoint from which the input image is acquired toward the center of the object. In the following description, the positional relationship associated with the sensor is the positional relationship between the object and the sensor corresponding to the viewpoint from which the input image or depth image is acquired (not the viewpoint from which training is performed).

[0084] The first projection calculator 231 calculates a first projection image 281 based on the posture and size 202. For example, the first projection calculator 231 determines a first output bounding box corresponding to the object by projecting the 3D bounding box calculated based on the posture and size 202 onto the first projection image 281. The first projection image 281 is a projection image representing the object observed from a first viewpoint. The first viewpoint is, for example, a bird's-eye view, and the first projection image 281 is a bird's-eye view projection image. Figure 6 Further elaborating on the bird's eye view projection image, the first output bounding box is a 2D bounding box of a portion of the first projection image 281 that is determined to correspond to the object.

[0085] For example, the first projection calculator 231 estimates the position of the 3D bounding box using an optimization scheme based on projection geometry. Projection geometry indicates that when the 3D bounding box of an object is projected onto a 2D image, the 3D bounding box needs to be aligned with the 2D bounding box (e.g., Figure 2 The input bounding box shown) fits closely.

[0086] The optimization scheme based on projective geometry is interpreted as a problem of determining four lines of a 2D bounding box using eight points of a 3D bounding box. When four lines are determined using eight points, a maximum of "84" combinations are obtained. However, since the vehicle (i.e., the object) is arranged along the roadside, it is assumed that the roll angle in the orientation of the object, the roll angle and pitch angle in the pitch angle, and the yaw angle are "0", and there may be a maximum of "64" combinations. Therefore, when an error occurs in the 2D bounding box detected from the input image, the error in the 3D bounding box increases.

[0087] The second projection calculator 232 determines a second output bounding box corresponding to the object by projecting the 3D bounding box of the object onto the second projection image 282. The second projection image 282 is a projection image when the object is observed from a second viewpoint. The second viewpoint is, for example, a perspective view, and the second projection image 282 is a perspective projection image. Figure 6 Further elaborating on the perspective projection image, the second output bounding box is a 2D bounding box of a portion of the second projection image 282 that is determined to correspond to the object.

[0088] For example, the object detection training apparatus 200 integrates the first projection calculator 231 and the second projection calculator 232 with the object detection model 270, and trains the object detection model 270 based on the results, such as the first projection image 281 and the second projection image 282. Therefore, the object detection training apparatus 200 reflects the physical position relationship of the projected object and the 3D information of the object to the object detection model 270. Therefore, the error in the 3D bounding box can be minimized.

[0089] The loss calculator 240 calculates a loss corresponding to each of the pose and size 202, the first projection image 281, and the second projection image 282. For example, the loss calculator 240 separately calculates a loss of size, a loss of pose, a loss of the first projection image 281, and a loss of the second projection image 282, integrates the losses, and calculates a total loss of the object detection model 270.

[0090] The trainer 250 trains the object detection model 270 based on the calculated loss. For example, the trainer 250 updates the parameters of the object detection model 270 so that the total loss converges to an arbitrary value. For example, the calculated loss can be calculated as a mean square error value compared with the reference data. When the mean square error value drops below the target minimum value, the object detection model 270 is considered to have converged. The trainer 250 updates the parameters of the object detection model 270 based on the design of an objective function indicating the total loss to minimize the total loss, however, the examples are not limited thereto. The following will refer to Figure 4 The training of the object detection model 270 and an example of the structure of the object detection model 270 are further described.

[0091] Figure 3 is a flow chart illustrating an example of an object detection training method.

[0092] Reference Figure 3 In operation 310, the object detection training device estimates the pose and size of the object based on features extracted from the input image using the object detection model. For example, the object detection training device extracts features from a 2D bounding box corresponding to the object detected from the input image and a cropped image corresponding to the 2D bounding box based on the object detection model.

[0093] In operation 320, the object detection training apparatus determines a first output bounding box corresponding to the object by projecting the 3D bounding box calculated according to the pose and size onto the first projection image. For example, the object detection training apparatus determines a bird's-eye view bounding box corresponding to the object by projecting the calculated 3D bounding box onto the bird's-eye view projection image.

[0094] In operation 330, the object detection training apparatus determines a second output bounding box corresponding to the object by projecting the calculated 3D bounding box to the second projection image. For example, the object detection training apparatus determines a perspective bounding box corresponding to the object by projecting the calculated 3D bounding box to the perspective projection image.

[0095] In operation 340, the object detection training device trains the object detection model based on the pose, size, first output bounding box, and second output bounding box. For example, the object detection training device calculates a loss based on the pose, size, first output bounding box, and second output bounding box. The object detection training device trains the object detection model based on the calculated loss.

[0096] Figure 4 An example of a neural network 400 as an object detection model is shown.

[0097] The object detection model includes, for example, a neural network 400. The present invention provides an object detection device for detecting an object based on the neural network 400, and provides a method and device for training an object detection model (e.g., the neural network 400) (hereinafter, referred to as an "object detection training device"). In the following description, recognition includes verification and identification. Verification is an operation of determining whether input data is true or false, and identification is an operation of determining which of a plurality of labels the input data indicates.

[0098] An example of the structure of neural network 400 is described below.

[0099] The neural network 400 includes a plurality of layers each including a plurality of nodes. In addition, the neural network 400 includes connection weights that connect a plurality of nodes included in one of the plurality of layers to nodes included in another layer. In an example, the object detection training device acquires the neural network 400 from an internal database (DB) stored in a memory, or receives the neural network 400 from an external server via a communicator and acquires the neural network 400.

[0100] For example, the neural network 400 is a model designed to perform an arbitrary task using a large number of nodes connected via edges. The neural network 400 is implemented, for example, by hardware or a combination of software and hardware. The neural network 400 includes nodes, and the nodes are connected to each other via edges with connection weights. The connection weight is a predetermined value of an edge and is called a "synaptic weight" or "connection strength".

[0101] The neural network 400 includes a plurality of layers. For example, the neural network 400 includes an input layer 410, a hidden layer 420, and an output layer 430. The input layer 410 receives an input for performing training or recognition and transmits the input to the hidden layer 420. The output layer 430 generates an output of the neural network 400 based on a signal received from the hidden layer 420. The hidden layer 420 is located between the input layer 410 and the output layer 430, and changes the training input of the training data received via the input layer 410 into a value that can be relatively more easily predicted.

[0102] Each of the input layer 410, the hidden layer 420, and the output layer 430 includes a plurality of nodes. The nodes included in the input layer 410 are referred to as "input nodes", the nodes included in the hidden layer 420 are referred to as "hidden nodes", and the nodes included in the output layer 430 are referred to as "output nodes".

[0103] The input nodes included in the input layer 410 and the hidden nodes included in the hidden layer 420 are connected to each other via edges having connection weights. In addition, the hidden nodes included in the hidden layer 420 and the output nodes included in the output layer 430 are connected to each other via edges having connection weights.

[0104] Although not shown, the neural network includes a plurality of hidden layers. A neural network including a plurality of hidden layers is referred to as a "deep neural network (DNN)". Training of a DNN is referred to as "deep learning". For example, when it is assumed that the hidden layer 420 includes a first hidden layer, a second hidden layer, and a third hidden layer, the output of the hidden node included in the first hidden layer is connected to the hidden node included in the second hidden layer. In addition, the output of the hidden node included in the second hidden layer is connected to the hidden node included in the third hidden layer.

[0105] For example, the object detection training device inputs the outputs of the previous hidden nodes included in the previous hidden layer to each hidden layer via an edge with a connection weight, and generates the outputs of the hidden nodes included in each hidden layer based on an activation function and a value obtained by applying the connection weights to the outputs of the previous hidden nodes. In the example, in order to transmit the output to the next hidden node, the result of the activation function needs to exceed the threshold of the current hidden node. In this example, the node does not transmit a signal to the next node until a predetermined threshold activation strength is reached by an input vector, and remains in an inactive state.

[0106] The object detection training device trains the neural network 400 by supervised learning. The object detection training device is implemented, for example, by a hardware module or a combination of a software module and a hardware module. Supervised learning is a scheme in which a training input of training data is input to the neural network 400 together with a training output corresponding to the training input, and the connection weights of the edges are updated so that the output data corresponding to the training output is output. The training data is data containing pairs of training inputs and training outputs. Although the structure of the neural network 400 is Figure 4 In the example, the neural network is expressed as a node structure, however, each example is not limited to a node structure. For example, a variety of data structures can be used to store the neural network in a memory storage area.

[0107] The object detection training device determines the parameters of the nodes through a gradient descent scheme based on the error propagated back to the neural network and based on the output values ​​of the nodes included in the neural network. For example, the object detection training device updates the connection weights between the nodes through error backpropagation learning. Error backpropagation learning is a scheme in which the error is estimated by forward calculation of given training data, the estimated error is backpropagated from the output layer to the hidden layer and the input layer, and the connection weights are updated to reduce the error. The neural network 400 is processed in the order of the input layer 410, the hidden layer 420, and the output layer 430, however, in the error backpropagation learning, the connection weights are updated in the order of the output layer 430, the hidden layer 420, and the input layer 410. For example, at least one processor processes the neural network in the desired direction using a buffer memory configured to store each layer or a series of calculation data.

[0108] The object detection training device defines an objective function for measuring the proximity of the currently set connection weights to the optimal value, continues to change the connection weights based on the result of the objective function, and repeatedly performs training. For example, the objective function is a loss function for calculating the loss between the expected value to be output and the actual output value based on the training input of the training data in the neural network 400. The object detection training device updates the connection weights by reducing the value of the loss function. Figure 5 Example illustrating the loss function.

[0109] Figure 5 An example of the object detection training process is shown.

[0110] The object detection model 570 may include a feature extractor 510 and estimators 521 , 522 , and 523 , and may perform a first projection calculation 531 and a second projection calculation 532 .

[0111] The feature extractor 510 extracts features from the input image. For example, the feature extractor 510 extracts features from a cropped image and a bounding box corresponding to an object appearing in the input image. The cropped image is resized in a format corresponding to an input layer of the feature extractor 510. The feature extractor 510 includes at least one convolutional layer.

[0112] In addition, the feature extractor 510 provides the extracted features to the estimators 521, 522, and 523. The feature extractor 510 is shared by the estimators 521, 522, and 523. The estimators 521, 522, and 523 are, for example, branch networks connected to the feature extractor 510, and have different regression targets.

[0113] The estimator 521 estimates the size 501 of the object according to the features extracted by the feature extractor 510. The estimator 521 may include at least one fully connected layer, which may be referred to as an FC layer.

[0114] The estimator 522 estimates the pose 502 of the object based on the features extracted by the feature extractor 510. The pose 502 is represented by, for example, an angle, a sine (sin) value, and a cosine (cos) value corresponding to the orientation of the object. The estimator 522 includes at least one fully connected layer and performs an L2 norm operation.

[0115] The estimator 523 estimates a confidence level 503 of each of the size 501 and the pose 502 estimated from the extracted features. The confidence level 503 is, for example, a score indicating the degree of confidence of each of the size 501 and the pose 502 estimated by the estimators 521 and 522 as a numerical value.

[0116] Figure 5 The estimators 521, 522, and 523 are shown as branch networks of the feature extractor 510, however, examples are not limited thereto. In addition, the number and type of layers included in the estimators 521, 522, and 523 are not limited.

[0117] The object detection training device obtains a bird's-eye view projection image 581 from the size 501 and the posture 502 through the first projection calculation 531. For example, the object detection training device obtains a 3D bounding box containing the object from the size 501 and the posture 502, and projects the obtained 3D bounding box to the bird's-eye view projection image 581. The object detection training device determines the size and position of the bird's-eye view bounding box projected to the bird's-eye view projection image 581. The first projection calculation 531 is, for example, a matrix operation that converts 3D coordinates into 2D coordinates on the first projection image, however, each example is not limited to this.

[0118] The object detection training device obtains a perspective projection image 582 from the size 501 and the posture 502 through the second projection calculation 532. For example, the object detection training device calculates a 3D bounding box from the size 501 and the posture 502, and projects the 3D bounding box to the perspective projection image 582. The object detection training device determines the size and position of the perspective bounding box projected to the perspective projection image 582.

[0119] Each of the bird's-eye view bounding box and the perspective bounding box is a 2D bounding box that appears in the projection image as a 2D image. For example, the object detection training device converts the 3D coordinates of each of the points of the 3D bounding box into the 2D coordinates of each in the projection image to determine the bird's-eye view bounding box and the perspective bounding box.

[0120] The object detection training apparatus 200 calculates outputs of each of the size 501, the pose 502, the bird's-eye view projection image 581, and the perspective projection image 582 from the input image based on the object detection model 570. When the training of the object detection model 570 is not completed, the above outputs are temporary outputs.

[0121] The object detection training apparatus calculates the loss of the object detection model 570 based on the temporary output calculated from the object detection model 570 .

[0122] The object detection training apparatus calculates the L2 loss as the loss 543 of the size 501. For example, the object detection training apparatus calculates the Euclidean distance between the temporary size and the reference size as the loss. The temporary size is calculated based on the input image (ie, the training input), and the reference size is the training output.

[0123] The object detection training device calculates a cosine loss as a loss 544 for the pose 502. For example, the object detection training device calculates a cosine loss corresponding to the angular difference between the temporary orientation and the reference orientation. The temporary orientation corresponds to the pose 502 estimated based on the input image (i.e., the training input), and the reference orientation is the training output.

[0124] The object detection training device calculates a softmax loss as the loss 545 of the confidence level 503 .

[0125] The object detection training apparatus calculates a bird's eye view (bev) intersection over union (IoU) loss as a loss 541 for a bird's eye view projection image 581. The object detection training apparatus calculates a perspective (persp) IoU loss as a loss 542 for a perspective projection image 582. The IoU loss is a loss indicating the similarity between a temporary bounding box calculated based on a training input and a reference bounding box corresponding to a training output. Figure 7 and Figure 8 Let’s further illustrate the example of IoU loss.

[0126] The object detection training apparatus calculates the loss of each of the above temporary outputs, integrates the losses, and determines the total loss of the object detection model 570. The object detection model 570 updates the parameters (e.g., connection weights) of the feature extractor 510 and the estimators 521, 522, and 523 until the total loss converges.

[0127] The object detection model 570 in which the total loss has converged is, for example, a fully trained model (e.g., a neural network). The object detection device performs object detection based on the fully trained object detection model 570. In addition, even when the training is completed, the object detection device additionally trains the object detection model 570 based on a new input image. For example, the object detection device determines a cropped image corresponding to a bounding box as a training input based on the input image, determines a bounding box corresponding to the object as a training output, and updates the parameters of the object detection model 570.

[0128] Figure 6 Examples of a perspective projection image and a bird's-eye view projection image are shown.

[0129] The object detection device is installed in the vehicle 690. The object detection device includes a sensor 691, and the sensor 691 is configured to obtain an image 610, which represents an object in front of the vehicle 690 including the object detection device. The object detection device generates or obtains information about various viewpoints to estimate the position of the vehicle 690. In an example, the object detection device generates a bird's-eye view projection image 620 and a perspective projection image 630 in front of the vehicle 690. In another example, the object detection device generates the bird's-eye view projection image 620 and the perspective projection image 630 based on a color image in front of the vehicle 690 and a depth image of a bird's-eye view.

[0130] The bird's-eye view projection image 620 is, for example, a projection image corresponding to an observation viewpoint (e.g., a top view) at an altitude above the height of the device including the sensor 691. The bird's-eye view projection image 620 includes a bird's-eye view bounding box 621 corresponding to objects located around the device including the sensor 691. The bird's-eye view bounding box 621 is Figure 6 , but each example is not limited thereto. For example, when there are multiple objects, multiple frames are detected.

[0131] The perspective projection image 630 is, for example, a projection image corresponding to a viewpoint when the moving direction of the device including the sensor 691 is observed at an altitude corresponding to the height of the device including the sensor 691. The perspective projection image 630 includes a perspective bounding frame 631 corresponding to an object located in the moving direction of the device including the sensor 691. The perspective bounding frame 631 is, for example, a 2D bounding frame and includes a front frame and a rear frame.

[0132] The object detection training device trains the object detection model based on the loss of the bird's-eye view bounding box 621 in the bird's-eye view projection image 620 and the loss of the perspective bounding box 631 in the perspective projection image 630 to generate an object detection model configured to accurately estimate the size and posture of the object. In general, the loss is determined by comparing the estimated object posture, size and position with reference data. Reference data may also be referred to as ground truth data. The object detection results obtained from various viewpoints are reflected to the object detection model, and thus the accuracy of the object detection model is improved.

[0133] Although the present invention mainly describes the projection images and bounding boxes in the bird's eye view and the perspective view, each example is not limited thereto. Therefore, the object detection model is trained based on the loss of the projection images and bounding boxes in other views. In addition, the object detection model is trained based on the loss of the projection images and bounding boxes in multiple viewpoints.

[0134] Figure 7 An example of IoU is shown.

[0135] The object detection training device calculates the loss of bounding boxes corresponding to objects detected from the projected images corresponding to each viewpoint based on IoU. The metric called IoU corresponds to the ratio of the intersection area to the union area, see Figure 7 The IoU is an indicator indicating the degree of overlap between the output bounding box and the reference bounding box, and represents, for example, a value obtained by dividing the area of ​​the intersection region 730 between the two regions by the area of ​​the union region.

[0136] For example, the object detection training device determines an output bounding box 710 corresponding to the object based on the input image relative to the projection image corresponding to an arbitrary viewpoint. The object detection training device compares the output bounding box 710 with the reference bounding box 720. The reference bounding box 720 is a training output paired with the input image (i.e., training input), and is, for example, benchmark data. The object detection training device determines the IoU loss based on a value obtained by dividing the area of ​​the intersection region 730 between the output bounding box 710 and the reference bounding box 720 by the area of ​​the union region. For example, the object detection training device updates the parameters of the object detection model so that the area of ​​the intersection region 730 converges to the area of ​​the union region. When the value of IoU is close to "1", the similarity between the output bounding box 710 and the reference bounding box 720 increases.

[0137] The object detection training apparatus calculates the loss of the first output bounding box based on the degree of overlap between the first output bounding box and the first reference bounding box. In addition, the object detection training apparatus calculates the loss of the second output bounding box based on the degree of overlap between the second output bounding box and the second reference bounding box. However, the temporary output is not limited to the first output bounding box and the second output bounding box, and at least one output bounding box 710 is used to calculate the loss.

[0138] Figure 8 An example of a perspective bounding box is shown.

[0139] The perspective bounding box is a 2D bounding box, and includes a front frame 810 and a rear frame 820. The perspective bounding box is, for example, a frame obtained by projecting a 3D bounding box onto a perspective projection image corresponding to a perspective view. For example, the perspective bounding box is a frame obtained by converting 3D coordinates of a 3D bounding box corresponding to an object into 2D coordinates of a perspective projection image. For example, the front frame 810 of the perspective bounding box is a frame obtained by converting 3D coordinates of four points corresponding to the front of the 3D bounding box into 2D coordinates on the perspective projection image. The rear frame 820 of the perspective bounding box is a frame obtained by converting 3D coordinates of four points corresponding to the rear of the 3D bounding box into 2D coordinates on the perspective projection image.

[0140] The object detection training device defines a new reference frame for IoU calculation to calculate the loss of the perspective bounding frame. For example, the object detection training device determines the target frame 830 based on the front frame 810 and the back frame 820 according to the perspective bounding frame obtained by the second projection calculator and corresponding to the object. The front frame 810 and the back frame 820 indicate the front and back of the object, respectively. The object detection training device determines the frame with the smallest area and including the front frame 810 and the back frame 820 as the target frame 830. The target frame 830 is used as the reference frame above. Figure 7 The output bounding box 710.

[0141] Therefore, the object detection training apparatus calculates the loss of the perspective bounding box based on the degree of overlap between the target box 830 and the reference box.

[0142] Fig. 9 is a block diagram illustrating an example of an object detection training device 900 .

[0143] remove Figure 2 In addition to the illustrated structure, the object detection training apparatus 900 further includes a location determiner 933. Therefore, the object detection model 970 further includes a branch network configured to determine a location.

[0144] The feature extractor 910 extracts abstract features from the bounding box and the cropped image 901. The estimator 920 estimates the pose and size 902 of the object based on the extracted features. The first projection calculator 931 determines a first projection image 981 based on the pose and size 902. The second projection calculator 932 determines a second projection image 982 based on the pose and size 902.

[0145] The position determiner 933 determines the position of the object based on the extracted features and the posture and size 902. For example, the position determiner 933 determines the relative distance from the sensor to the object. The position determiner 933 estimates a position offset based on the extracted features. The position offset is, for example, an offset representing an error predicted for the initial position. The position determiner 933 estimates the initial position based on the size and posture. The position determiner 933 determines the output position by applying the position offset to the initial position. Therefore, the position determiner 933 corrects the initial position result estimated based on the posture and size 902 of the object to determine a more accurate position of the object.

[0146] The loss calculator 940 calculates a loss corresponding to each of the pose and size 902 , the first projection image 981 , the second projection image 982 , and the output position.

[0147] The trainer 950 trains the object detection model 970 based on the calculated loss. For example, the trainer 950 updates the parameters of the object detection model 970 so that the calculated loss converges to an arbitrary value.

[0148] Fig.10 Another example of an object detection training process is shown.

[0149] and Figure 5 Similarly, the object detection model 1070 may include the feature extractor 1010 and the estimators 1021 , 1022 , and 1023 , and may perform the first projection calculation 1031 and the second projection calculation 1032 . In addition, the object detection model 1070 may further include a position estimator 1024 .

[0150] The feature extractor 1010 extracts features from the bounding box and the cropped image corresponding to the object appearing in the input image. The feature extractor 1010 includes at least one convolution layer. The feature extractor 1010 is connected to a branch network, such as a plurality of estimators and a position estimator 1024, and the branch network has different regression targets.

[0151] The estimator 1021 estimates the size 1001 of the object based on the features extracted by the feature extractor 1010, and may include a fully connected layer. The estimator 1022 estimates the pose 1002 of the object based on the extracted features. The estimator 1022 may include a fully connected layer and perform an L2 norm operation. An example of an L2 norm operation is an operation that calculates the distance between two vectors. Algebraically, for example, the scalar d is calculated as d=|(xy)|, where x and y are vectors, and "|()|" indicates a norm operation. The estimator 1023 estimates the confidence level 1003 of each of the size 1001 and the pose 1002, and includes a fully connected layer. In general, the confidence level may be the probability that the hypothesis is correct.

[0152] The object detection training apparatus obtains a bird's-eye view projection image 1081 from the size 1001 and the posture 1002 through a first projection calculation 1031 , and obtains a perspective projection image 1082 through a second projection calculation 1032 .

[0153] The position estimator 1024 estimates the position offset 1004 based on the extracted features. The object detection training apparatus estimates the initial position based on the size 1001 and the pose 1002, applies the position offset 1004 to the initial position, and determines the output position 1084.

[0154] The object detection training apparatus calculates a temporary output of each of the size 1001, the pose 1002, the bird's-eye view projection image 1081, the perspective projection image 1082, and the output position 1084 according to the input image based on the object detection model 1070. The object detection training apparatus calculates the loss of each temporary output separately, integrates the losses, and determines the total loss.

[0155] The object detection training apparatus calculates a bird's eye view IoU loss as a loss 1041 of the bird's eye view projection image 1081, calculates a perspective IoU loss as a loss 1042 of the perspective projection image 1082, calculates an L2 loss as a loss 1043 of the size 1001, calculates a cosine loss as a loss 1044 of the pose 1002, calculates a soft maximization loss as a loss 1045 of the confidence level 1003, and calculates an L1 loss or an L2 loss as a loss 1046 of the output position 1084. For example, the object detection training apparatus calculates, as the loss 1046 of the output position 1084, a Euclidean distance between a value obtained by adding the position offset 1004 to the initial position and a reference value.

[0156] The object detection training device updates the parameters (e.g., connection weights) of the feature extractor 1010 and the estimators 1021, 1022, and 1023 until the total loss of the object detection model 1070 converges. In addition, even when the training is completed, the object detection training device calculates the temporary output of the object detection model 1070 in response to acquiring a new input image, and re-updates the parameters of the object detection model 1070 so that the loss of the temporary output converges.

[0157] Therefore, the object detection training apparatus trains the object detection model 1070 on the composite effect of the position, posture, and size of the object on the form of the projected object when the 3D object is projected onto the projection image. In addition, the object detection training apparatus applies a position offset to the initial position value estimated as a discrete value to determine an output position indicating a continuous value.

[0158] Fig.11 is a flow chart illustrating another example of an object detection training method.

[0159] In operation 1110, the object detection training apparatus estimates the pose of the object and the 3D bounding box corresponding to the object based on the features extracted from the input image and the depth image using the object detection model. The object detection training apparatus extracts features from the input image and the depth image based on the regions calculated by the region proposal network (RPN). The RPN is a network that proposes regions where objects are expected to exist in a space corresponding to the field of view (FOV) of the sensor, and will be referred to below. Fig.13 Further explanation.

[0160] In operation 1120, the object detection training apparatus determines an output bounding box corresponding to the object by projecting the calculated 3D bounding box to the projection image. For example, the object detection training apparatus determines a perspective bounding box corresponding to the object by projecting the calculated 3D bounding box to the perspective projection image.

[0161] In operation 1130, the object detection training device trains the feature extractor and the estimator based on the pose, the 3D bounding box, and the output bounding box. The object detection training device calculates the loss based on the pose, the 3D bounding box, and the output bounding box. The object detection training device trains the object detection model based on the calculated loss. The training algorithm may use a gradient descent method.

[0162] Fig.12 An example of an object detection training apparatus 1200 is shown.

[0163] The object detection training apparatus 1200 includes a feature extractor 1210, an estimator 1220, a projection calculator 1230, a loss calculator 1240, and a trainer 1250. The object detection model 1270 may include the feature extractor 1210, the estimator 1220, and the projection calculator 1230.

[0164] The feature extractor 1210 extracts features from the input image and the depth image 1201. The input image is, for example, a color image acquired based on a camera sensor. The depth image is, for example, an image representing the distance to a target point corresponding to each pixel acquired based on a LiDAR sensor. The depth image corresponds to a bird's-eye view. Fig.13 The example of feature extractor 1210 is further explained.

[0165] The estimator 1220 estimates the pose corresponding to the object and the 3D bounding box 1202 according to the extracted features. For example, the estimator 1220 estimates the positions (eg, 3D coordinates) of points defining the 3D bounding box based on the extracted features.

[0166] The projection calculator 1230 determines the projection image 1280 by projecting the 3D bounding box at an arbitrary viewpoint. For example, the projection calculator 1230 determines the perspective bounding box by projecting the 3D bounding box to the projection image 1280 corresponding to the perspective view. The projection calculator 1230 includes a matrix operation that converts each 3D point of the 3D bounding box into a 2D point on the projection image 1280.

[0167] The loss calculator 1240 calculates a loss corresponding to each of the pose and 3D bounding box 1202 and the projected image 1280 .

[0168] The trainer 1250 trains the object detection model 1270 based on the calculated loss. For example, the trainer 1250 updates the parameters of the object detection model 1270 so that the calculated loss converges.

[0169] Fig.13 An example of the feature extraction process is shown.

[0170] Fig.13A portion of an object detection model 1300 is shown, and the object detection model 1300 includes a feature extractor 1350 and a branch network corresponding to an estimator 1360 configured to estimate a 3D bounding box.

[0171] The feature extractor 1350 extracts features from the input image 1311 and the depth image 1312. For example, the feature extractor 1350 includes the RPN 1330, and extracts features corresponding to the region suggested by the RPN 1330.

[0172] For example, the feature extractor 1350 calculates an input feature map 1321 through a convolution layer based on the input image 1311. The feature extractor 1350 calculates a depth feature map 1322 through a convolution layer based on the depth image 1312. The RPN 1330 suggests an area where the existence probability of the detected object is relatively high based on the input feature map 1321, the depth feature map 1322, and the 3D anchor grid 1313.

[0173] The input feature map 1321 is data obtained by abstracting visual features of the input image 1311. The depth feature map 1322 is data obtained by abstracting visual features of the depth image 1312.

[0174] The 3D anchor grid 1313 is, for example, a grid indicating a space corresponding to the FOV of a sensor (e.g., an image sensor or a LiDAR sensor). The 3D anchor grid 1313 is a grid with a predetermined resolution (e.g., 0.1 m). The 3D anchor grid 1313 includes an anchor box set to detect an object in a space corresponding to the FOV of the sensor. The anchor box is, for example, a bounding box template that expresses the probability of the existence of an object in the space corresponding to the FOV of the sensor. The anchor box is, for example, a 3D bounding box. For example, the anchor box is located in an area of ​​the 3D anchor grid 1313 where the user expects to observe the probability of the existence of an object. The shape, size, and posture of the anchor box and the number of anchor boxes vary depending on the design.

[0175] The RPN 1330 calculates the probability that an object detected from the input image 1311 or the depth image 1312 is contained in one of the plurality of anchor boxes arranged in the 3D anchor grid 1313. The RPN 1330 includes a convolutional layer, a cropping and resizing operation, a fully connected layer, and a non-maximum suppression (NMS) operation, such as Fig.13 As shown in , however, each example is not limited thereto. In addition, the RPN 1330 recommends anchor boxes with relatively high object existence probabilities in the 3D anchor grid 1313. The RPN 1330 recommends the first "K" anchor boxes 1331 in the order of object existence probability, and K is an integer greater than or equal to "1".

[0176] The feature extractor 1350 performs cropping and resizing to obtain a portion of the input feature map 1321 corresponding to the top "K" anchor boxes suggested by the RPN 1330. In addition, the feature extractor 1350 performs cropping and resizing to obtain a portion of the depth feature map 1322 corresponding to the top "K" anchor boxes suggested by the RPN 1330. The feature extractor 1350 extracts features by fusing the features obtained by cropping the input feature map 1321 with the features obtained by cropping the depth feature map 1322.

[0177] The estimator 1360 includes a fully connected layer and an NMS operation, and estimates the position of the 3D bounding box 1340 corresponding to the object according to the features extracted by the feature extractor 1350. The object detection training apparatus trains the object detection model 1300 so that the estimated position of the 3D bounding box 1340 converges to the position of the reference frame.

[0178] The object detection training apparatus classifies each anchor box of the 3D anchor grid 1313 into a positive anchor box and a negative anchor box based on the degree of overlap between the proposed anchor box and the reference 3D box in the bird's-eye view projection image. For example, the object detection training apparatus trains the RPN 1330 based on the degree of overlap between the anchor box proposed by the RPN 1330 and the reference box in the bird's-eye view. In this example, when the degree of overlap between the proposed anchor box and the reference box is greater than a first threshold, the object detection training apparatus trains the RPN 1330 so that the anchor box outputs a positive value (e.g., "1"). When the degree of overlap between the proposed anchor box and the reference box is less than a second threshold, the object detection training apparatus trains the RPN 1330 so that the anchor box outputs a negative value (e.g., "0").

[0179] Therefore, the object detection training apparatus classifies each of the anchor frames into a positive anchor frame and a negative anchor frame based on the degree of overlap between the reference frame and the estimated frame in the bird's-eye view and the degree of overlap between the reference frame and the estimated frame in the perspective view. The anchor frames correspond to training outputs that respond to training inputs (e.g., the input image 1311 and the depth image 1312). The object detection training apparatus trains the RPN 1330 to output given training outputs (e.g., outputs of positive anchor frames and negative anchor frames) in response to the inputs of the input image 1311 and the depth image 1312. The object detection training apparatus accurately classifies the anchor frames of the 3D anchor grid into positive anchor frames and negative anchor frames, and thus can reduce unnecessary region proposals and the number of operations.

[0180] The features extracted by the feature extractor 1350 are used to calculate the pose or perspective projection image of the object, which will be referred to below. Fig.14 This is further explained.

[0181] Fig.14 Another example of an object detection training process is shown.

[0182] The object detection model 1470 may include a feature extractor 1410 and estimators 1421 and 1422 , and may perform a projection calculation 1430 .

[0183] As above Fig.13 As described in the embodiment of the present invention, the feature extractor 1410 extracts features from the input image and the depth image. For example, the feature extractor 1410 extracts features corresponding to an anchor box (e.g., an anchor box determined to be a positive anchor box) predicted to contain an object in a space corresponding to the FOV of the sensor from the input image and the depth image. The feature extractor 1410 is connected to a branch network, for example, estimators 1421 and 1422.

[0184] The estimator 1421 estimates the 3D bounding box 1401 according to the features extracted by the feature extractor 1410. For example, the estimator 1421 recursively estimates the 3D coordinates of each of the points (eg, vertices) of the 3D bounding box 1401 corresponding to the object. The estimator 1421 includes at least one fully connected layer.

[0185] The estimator 1422 estimates the angle 1402 according to the features extracted by the feature extractor 1410. For example, the estimator 1422 estimates the angle 1402 corresponding to the orientation of the object. The estimator 1422 includes at least one fully connected layer.

[0186] The object detection training device obtains the perspective projection image 1480 from the 3D bounding box 1401 through projection calculation 1430. For example, the object detection training device determines the size and position of the perspective bounding box projected onto the perspective projection image 1480. The projection calculation 1430 includes a matrix operation to convert 3D coordinates into 2D coordinates on the perspective projection image 1480, however, each example is not limited thereto.

[0187] The object detection training apparatus calculates as output a 3D bounding box 1401, an angle 1402, and a perspective projection image 1480. These calculations are performed according to an object detection model 1470 based on the input image and the depth image.

[0188] The object detection training device calculates the total loss of the object detection model 1470 based on the above temporary output calculated according to the object detection model 1470. For example, the object detection training device calculates the L2 loss as the loss 1441 of the 3D bounding box 1401. In addition, the object detection training device calculates the L1 loss as the loss 1442 of the angle 1402. The object detection training device calculates the perspective IoU loss as the loss 1443 of the perspective projection image 1480. The object detection training device calculates the loss of the output bounding box based on the degree of overlap between the output bounding box and the reference bounding box. Figure 7 The IoU loss is explained and therefore will not be elaborated on again.

[0189] The object detection training apparatus calculates the loss of each of the above temporary outputs, integrates the losses, and determines the total loss of the object detection model 1470. The object detection model 1470 updates the parameters (e.g., connection weights) of the feature extractor 1410 and the estimators 1421 and 1422 so that the total loss converges.

[0190] Fig.15 15 is a block diagram showing an example of a configuration of an object detection training device 1500 .

[0191] The object detection training device 1500 includes a processor 1510 and a memory 1520 .

[0192] Processor 1510 estimates the pose and size of the object based on the features extracted from the input image using object detection model 1521. Processor 1510 determines a first output bounding box corresponding to the object by projecting a 3D bounding box calculated based on the pose and size onto the first projection image. Processor 1510 determines a second output bounding box corresponding to the object by projecting the calculated 3D bounding box onto the second projection image. Processor 1510 trains object detection model 1521 based on the pose, size, first output bounding box, and second output bounding box. The operation of processor 1510 is not limited thereto, and processor 1510 may perform the above-referenced method. Figures 1 to 14 At least one of the operations described.

[0193] The memory 1520 stores an object detection model 1521. The object detection model 1521 is a fully trained model and includes, for example, Figure 5 The object detection model 570 is shown. Fig.10 The object detection model 1070 shown, or Fig.14 Object detection model 1470 is shown. Processor 1510 updates the parameters of object detection model 1521 even when training is completed.

[0194] Memory 1520 further includes training data 1522. Training data 1522 includes pairs of training inputs and training outputs. In an example, when the training input is an input image, the training output includes a reference bounding box, size, position, and pose of an object appearing in the corresponding image. In another example, when the training input includes an input image and a depth image, the training output includes a reference 3D bounding box and a reference pose of the object.

[0195] Fig.16 16 is a block diagram showing an example of the configuration of the object detection device 1600 .

[0196] The object detection apparatus 1600 includes an image acquirer 1610 , a processor 1620 , and a memory 1630 .

[0197] The image acquirer 1610 includes a sensor. The sensor includes, for example, a camera sensor and a depth sensor. The depth sensor is, for example, a LiDAR sensor. When the object detection apparatus 1600 is installed in a vehicle, the image sensor and the LiDAR sensor are installed in the front side and the top side of the vehicle, respectively.

[0198] The processor 1620 detects an object located in front of the vehicle from at least one of the input image and the depth image based on the object detection model 1631. For example, the processor 1620 estimates the position and size of the bounding box corresponding to the object based on the fully trained object detection model 1631. The object detection model 1631 includes, for example, Figure 5 The object detection model 570 is shown. Fig.10 The object detection model 1070 shown, or Fig.14 The object detection model 1470 is shown. In addition, in response to acquiring at least one of a new input image and a new depth image, the processor 1620 retrains the object detection model 1631 based on the acquired image.

[0199] The memory 1630 stores the object detection model 1631. For example, the memory 1630 stores the fully trained object detection model 1631.

[0200] This article refers to Figure 2 , Fig. 9 , Fig.12 , Fig.15 and Fig.16The object detection training equipment 200, 900, 1200 and 1500, the object detection equipment 1600, and other devices, units, modules, devices and other components are implemented by hardware components. Examples of hardware components that can be used to perform the operations described in the present application include controllers, sensors, generators, drivers, memories, comparators, arithmetic logic units, adders, subtractors, multipliers, dividers, integrators, and any other electronic components configured to perform the operations described in the present application, where appropriate. In other examples, one or more of the hardware components that perform the operations described in the present application are implemented by computing hardware, such as by one or more processors or computers. A processor or computer can be implemented by one or more processing elements, such as by a logic gate array, a controller and an arithmetic logic unit, a digital signal processor, a microcomputer, a programmable logic controller, a field programmable gate array, a programmable logic array, a microprocessor, or any other device or combination of devices that are configured to respond to instructions in a defined manner and execute instructions to achieve the desired result. In one example, a processor or computer includes or is connected to one or more memories, and the one or more memories store instructions or software executed by the processor or computer. The hardware components implemented by the processor or computer can execute instructions or software (such as an operating system (OS) and one or more software applications running on the OS) to perform the operations described in this application. The hardware components can also access, manipulate, process, create and store data in response to the execution of instructions or software. For simplicity, the singular term "processor" or "computer" can be used when elaborating on the examples described in this application, but in other examples, multiple processors or computers can be used, or the processor or computer can include multiple processing elements or multiple types of processing elements or both. For example, a single hardware component or two or more hardware components can be implemented by a single processor, or two or more processors, or a processor and a controller. One or more hardware components can be implemented by one or more processors, or a processor and a controller, and one or more other hardware components can be implemented by one or more other processors, or another processor and another controller. One or more processors or a processor and a controller can implement a single hardware component or two or more hardware components.The hardware components may have any one or more of different processing configurations, examples of which include a single processor, independent processors, parallel processors, single-instruction single-data (SISD) multiprocessing, single-instruction multiple-data (SIMD) multiprocessing, multiple-instruction single-data (MISD) multiprocessing, and multiple-instruction multiple-data (MIMD) multiprocessing.

[0201] Perform the operations described in this application Figure 3 and Fig.11 The method shown is performed by computing hardware, for example, by one or more processors or computers, and the computing hardware is implemented as described above to execute instructions or software to perform the operations performed by the method as described in this application. For example, a single operation or two or more operations can be performed by a single processor, or two or more processors, or a processor and a controller. One or more operations can be performed by one or more processors, or a processor and a controller, and one or more other operations can be performed by one or more other processors, or another processor and another controller. One or more processors or a processor and a controller can perform a single operation or two or more operations.

[0202] The instructions or software for controlling a processor or computer to implement the hardware components and perform the method as described above are written as a computer program, a code segment, an instruction, or any combination thereof, for individually or collectively indicating or configuring the processor or computer to be used as a machine or a special-purpose computer for performing the operations performed by the hardware components and methods as described above. In one example, the instructions or software include machine code directly executed by the processor or computer, such as machine code generated by a compiler. In another example, the instructions or software include higher-level codes executed by the processor or computer using an interpreter. A programmer of ordinary skill in the art can easily write instructions or software based on the block diagrams and flow charts shown in the figure and the corresponding descriptions in the specification, and the block diagrams and flow charts and the corresponding descriptions in the specification disclose algorithms for performing the operations performed by the hardware components and methods as described above.

[0203] Instructions or software for controlling a processor or computer to implement hardware components and execute methods as described above, as well as any associated data, data files, and data structures, are recorded, stored, or fixed in or on one or more non-transitory computer-readable storage media. Examples of non-transitory computer-readable storage media include read-only memory (ROM), programmable read only memory (PROM), electrically erasable programmable read-only memory (EEPROM), random-access memory (RAM), dynamic random access memory (DRAM), static random access memory (SRAM), flash memory, nonvolatile memory, CD-ROM, CD-R, CD+R, CD-RW, CD+RW, DVD-ROM, DVD-R, DVD+R, DVD-RW, DVD+RW, DVD-RAM, BD-ROM, BD-R, BD-R LTH, BD-RE, Blu-ray or optical disk storage, hard disk drive (HDD), solid state drive (SSD), card-type memory (e.g., micro multimedia card or card (e.g., secure digital (SD) or extreme digital (SD) digital, XD)), magnetic tape, floppy disk, magneto-optical data storage device, optical data storage device, hard disk, solid-state disk, and any other device configured to store instructions or software and any associated data, data files and data structures in a non-temporary manner and provide the instructions or software and any associated data, data files and data structures to a processor or computer so that the processor or computer can execute the instructions.

[0204] Although the present invention includes specific examples, it will be apparent to those of ordinary skill in the art that various changes in form and detail can be made in these examples without departing from the spirit and scope of the claims and their equivalents. The examples described herein should be considered to be illustrative only and not limiting. The description of the features or aspects in each example will be considered to be applicable to similar features or aspects in other examples. If the described techniques are performed in a different order, and / or if the components in the described systems, architectures, devices, or circuits are combined in different ways and / or replaced or supplemented by other components or their equivalents, suitable results can be obtained. Therefore, the scope of the present invention is not defined by detailed description, but by the claims and their equivalents, and all changes within the scope of the claims and their equivalents will be interpreted as being included in the present invention.

Claims

1. An object detection training method, comprising: Use an object detection model to estimate the pose and size of an object based on features extracted from an input image. Calculating a three-dimensional bounding box according to the posture and the size; determining a first output bounding box corresponding to the object by projecting the three-dimensional bounding box onto a first projection image; determining a second output bounding box corresponding to the object by projecting the three-dimensional bounding box onto a second projection image; as well as training the object detection model based on the pose, the size, the first output bounding box, and the second output bounding box, wherein the training includes a fusion operation of the first output bounding box and the second output bounding box, Wherein training the object detection model comprises: calculating a loss based on the pose, the size, the first output bounding box, and the second output bounding box; and The calculation of the loss includes: calculating a first loss based on a degree of overlap between the first output bounding box and a first reference bounding box; and A second loss is calculated based on an overlap degree between the second output bounding box and a second reference bounding box.

2. The object detection training method of claim 1, wherein determining the first output bounding box comprises determining a bird's-eye view bounding box corresponding to the object by projecting the three-dimensional bounding box onto a bird's-eye view projection image. 3 . The object detection training method according to claim 1 , wherein determining the second output bounding box comprises determining a perspective bounding box corresponding to the object by projecting the three-dimensional bounding box onto a perspective projection image.

4. The object detection training method of claim 1 , wherein estimating the pose and the size of the object comprises extracting features from i) a two-dimensional bounding box corresponding to the object detected from the input image and ii) a cropped image corresponding to the two-dimensional bounding box.

5. The object detection training method according to claim 1, wherein training the object detection model comprises: The object detection model is trained based on the loss, the training comprising updating weights in a neural network.

6. The object detection training method according to claim 5, wherein calculating the loss comprises: determining a target frame based on a front frame indicating the front face of the object; as well as The loss of the perspective bounding box is calculated based on the degree of overlap between the target box and the reference box.

7. The object detection training method according to claim 5, wherein Estimating the pose and the size of the object includes: estimating a position shift based on the features extracted from the input image; estimating an initial position according to the posture and the size; determining an output position by applying the position offset to the estimated initial position; and Calculating the loss is performed based on the determined output position.

8. The object detection training method of claim 1, wherein estimating the pose and the size comprises estimating a confidence level for each of the estimated pose and the estimated size based on the features extracted from the input image. 9 . The object detection training method of claim 1 , wherein estimating the size of the object comprises estimating a width, a length, and a height of the object.

10. An object detection training method comprising: Using an object detection model to estimate the pose of an object based on features extracted from an input image and a depth image; estimating a three-dimensional bounding box corresponding to the object based on the features extracted from the input image and the depth image using the object detection model; determining an output bounding box corresponding to the object by projecting the three-dimensional bounding box onto a projection image; as well as training the object detection model based on the pose, the three-dimensional bounding box and the output bounding box, wherein the training includes a fusion operation of the three-dimensional bounding box and the output bounding box, Wherein training the object detection model comprises: calculating a loss based on the pose, the three-dimensional bounding box, and the output bounding box; and The calculation of the loss includes: The loss of the output bounding box is calculated based on the degree of overlap between the output bounding box and the reference bounding box. 11 . The object detection training method according to claim 10 , wherein determining the output bounding box comprises determining a perspective bounding box corresponding to the object by projecting the three-dimensional bounding box onto a perspective projection image.

12. The object detection training method according to claim 10, wherein the features are extracted from the input image and the depth image based on regions calculated by a region proposal network.

13. The object detection training method according to claim 10, wherein training the object detection model comprises: The object detection model is trained based on the loss, the training comprising updating weights in a neural network.

14. The object detection training method according to claim 13, wherein calculating the loss comprises: determining a target frame based on a front frame indicating a front face of the object according to a perspective bounding frame corresponding to the object; as well as The loss of the perspective bounding box is calculated based on the degree of overlap between the target box and the reference box.

15. The object detection training method of claim 10, wherein estimating the pose and the three-dimensional bounding box comprises estimating positions of points that at least partially define the three-dimensional bounding box.

16. The object detection training method according to claim 10, wherein estimating the pose and the three-dimensional bounding box is performed based on a depth image corresponding to a bird's-eye view.

17. The method of claim 1, wherein the fusion operation of the first output bounding box and the second output bounding box comprises extracting features by identifying features obtained by cropping the first output bounding box and by cropping the second output bounding box.

18. An object detection training device comprising: a memory configured to store an object detection model; as well as The processor is configured to: Using the object detection model to estimate the pose and size of the object based on features extracted from the input image, Calculating a three-dimensional bounding box based on the posture and the size, determining a first output bounding box corresponding to the object by projecting the three-dimensional bounding box onto a first projection image, determining a second output bounding box corresponding to the object by projecting the three-dimensional bounding box onto a second projection image, and training the object detection model based on the pose, the size, the first output bounding box, and the second output bounding box, and further based on a fusion operation of the first output bounding box and the second output bounding box, The processor further trains the object detection model by: calculating a loss based on the pose, the size, the first output bounding box, and the second output bounding box; and The processor further calculates the loss by: calculating a first loss based on a degree of overlap between the first output bounding box and a first reference bounding box; and A second loss is calculated based on an overlap degree between the second output bounding box and a second reference bounding box.

Images